My training loss has fluctuated widely. I assume this has happened because I have used dropout(0.7) twice in my Conv layers. Am i thinking right? I would appreciate your insights.
\n\n.png?generation=1579322623524904&alt=media)
My training loss has fluctuated widely. I assume this has happened because I have used dropout(0.7) twice in my Conv layers. Am i thinking right? I would appreciate your insights.
\n\n.png?generation=1579322623524904&alt=media)
There can be several reasons. It can be related to the dataset and/or loss function. I think it's normal when we use a small batch size for training. However, if you look closely, fluctuation of loss is actually decreasing and if you smooth it, it will show a decreasing pattern. Dropout may not be directly related to the fluctuation of loss, however, higher dropout may result in overall higher loss and underfitting of a model. Another reason for the fluctuating loss is by using a high learning rate, but I see your learning rate is reasonably small. So LR might not be the reason here.
", "rawMarkdown": "There can be several reasons. It can be related to the dataset and/or loss function. I think it's normal when we use a small batch size for training. However, if you look closely, fluctuation of loss is actually decreasing and if you smooth it, it will show a decreasing pattern. Dropout may not be directly related to the fluctuation of loss, however, higher dropout may result in overall higher loss and underfitting of a model. Another reason for the fluctuating loss is by using a high learning rate, but I see your learning rate is reasonably small. So LR might not be the reason here.", "votes": null }, { "id": "722162", "postDate": "01/18/2020 07:30:23", "content": "i think the main reason here is small batch size and batchnormalization,,i faced similar issue here! his model will start diverging after few more epoches like it is happening here with me!
", "rawMarkdown": "i think the main reason here is small batch size and batchnormalization,,i faced similar issue here! his model will start diverging after few more epoches like it is happening here with me!", "votes": null }, { "id": "722182", "postDate": "01/18/2020 08:24:27", "content": "I think a simpler explanation could be if you are using hocop or my kernel then train_loss are not averaged across epoch at the end . But **Val_Loss is averaged for the full epoch **. Therefore the train_loss you see at the end is the loss for last batch (if batch-size =1 , its the loss for one image) alone . Which could vary greatly based on how good or how bad that particular image is and not representative of the performance of complete epoch.
", "rawMarkdown": "I think a simpler explanation could be if you are using hocop or my kernel then **train_loss are not averaged** across epoch at the end . But **Val_Loss is averaged for the full epoch **. Therefore the train_loss you see at the end is the loss for last batch (if batch-size =1 , its the loss for one image) alone . Which could vary greatly based on how good or how bad that particular image is and not representative of the performance of complete epoch.", "votes": null }, { "id": "722561", "postDate": "01/18/2020 18:04:34", "content": "@phoenix9032 Thank you that makes lots of sense.
", "rawMarkdown": "phoenix9032 Thank you that makes lots of sense.", "votes": null }, { "id": "722563", "postDate": "01/18/2020 18:06:06", "content": "@sgalib thank you for your very insightful comment.
", "rawMarkdown": "sgalib thank you for your very insightful comment.", "votes": null } ], "comments": [ { "id": 722129, "author_name": "sgalib", "author_url": "", "post_date": "01/18/2020 06:23:08", "content": "There can be several reasons. It can be related to the dataset and/or loss function. I think it's normal when we use a small batch size for training. However, if you look closely, fluctuation of loss is actually decreasing and if you smooth it, it will show a decreasing pattern. Dropout may not be directly related to the fluctuation of loss, however, higher dropout may result in overall higher loss and underfitting of a model. Another reason for the fluctuating loss is by using a high learning rate, but I see your learning rate is reasonably small. So LR might not be the reason here.
", "votes": null, "replies": [ { "id": 722162, "author_name": "mobassir", "author_url": "", "post_date": "01/18/2020 07:30:23", "content": "i think the main reason here is small batch size and batchnormalization,,i faced similar issue here! his model will start diverging after few more epoches like it is happening here with me!
", "votes": null, "replies": [] }, { "id": 722563, "author_name": "subediaarjun", "author_url": "", "post_date": "01/18/2020 18:06:06", "content": "@sgalib thank you for your very insightful comment.
", "votes": null, "replies": [] } ] }, { "id": 722182, "author_name": "phoenix9032", "author_url": "", "post_date": "01/18/2020 08:24:27", "content": "I think a simpler explanation could be if you are using hocop or my kernel then train_loss are not averaged across epoch at the end . But **Val_Loss is averaged for the full epoch **. Therefore the train_loss you see at the end is the loss for last batch (if batch-size =1 , its the loss for one image) alone . Which could vary greatly based on how good or how bad that particular image is and not representative of the performance of complete epoch.
", "votes": null, "replies": [ { "id": 722561, "author_name": "subediaarjun", "author_url": "", "post_date": "01/18/2020 18:04:34", "content": "@phoenix9032 Thank you that makes lots of sense.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "722090": "My training loss has fluctuated widely. I assume this has happened because I have used dropout(0.7) twice in my Conv layers. Am i thinking right? I would appreciate your insights.\n\n.png?generation=1579322623524904&alt=media)", "722129": "There can be several reasons. It can be related to the dataset and/or loss function. I think it's normal when we use a small batch size for training. However, if you look closely, fluctuation of loss is actually decreasing and if you smooth it, it will show a decreasing pattern. Dropout may not be directly related to the fluctuation of loss, however, higher dropout may result in overall higher loss and underfitting of a model. Another reason for the fluctuating loss is by using a high learning rate, but I see your learning rate is reasonably small. So LR might not be the reason here.", "722162": "i think the main reason here is small batch size and batchnormalization,,i faced similar issue here! his model will start diverging after few more epoches like it is happening here with me!", "722182": "I think a simpler explanation could be if you are using hocop or my kernel then **train_loss are not averaged** across epoch at the end . But **Val_Loss is averaged for the full epoch **. Therefore the train_loss you see at the end is the loss for last batch (if batch-size =1 , its the loss for one image) alone . Which could vary greatly based on how good or how bad that particular image is and not representative of the performance of complete epoch.", "722561": "phoenix9032 Thank you that makes lots of sense.", "722563": "sgalib thank you for your very insightful comment." }, "source": "meta" }