{ "id": 300960, "title": "How to Handle When Learning Curve Become Unstable?", "url": "/competitions/tensorflow-great-barrier-reef/discussion/300960", "author_name": "", "post_date": "2022-01-15T07:00:42.064430200Z", "votes": 6, "comment_count": 16, "views": 0, "content": "

Note for those who follows the original topic: I fully rewrote this topic based on the actual case I encountered. I think this is better compared to discussing about the general case.

\n

Simptom

\n\n

Assumed Cause

\n\n
\n

As @hengck23 pointed out, the root cause might be data.
\nsee also this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201

\n

Assumed Remedy

\n\n
\n

Fig1:
\n\"Screen-Shot-2022-01-28-at-20-36-28\"

\n

Fig2:
\n\"Screen-Shot-2022-01-28-at-20-36-37\"

\n

Fig3:
\n\"Screen-Shot-2022-01-28-at-20-36-45\"

\n
\n

Update Note

\n", "messages": [ { "id": "1650606", "postDate": "01/15/2022 07:00:42", "content": "

Note for those who follows the original topic: I fully rewrote this topic based on the actual case I encountered. I think this is better compared to discussing about the general case.

\n

Simptom

\n\n

Assumed Cause

\n\n
\n

As @hengck23 pointed out, the root cause might be data.
\nsee also this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201

\n

Assumed Remedy

\n\n
\n

Fig1:
\n\"Screen-Shot-2022-01-28-at-20-36-28\"

\n

Fig2:
\n\"Screen-Shot-2022-01-28-at-20-36-37\"

\n

Fig3:
\n\"Screen-Shot-2022-01-28-at-20-36-45\"

\n
\n

Update Note

\n", "rawMarkdown": "**Note for those who follows the original topic:** I fully rewrote this topic based on the actual case I encountered. I think this is better compared to discussing about the general case.\n\n# Simptom\n\n* mAP suddenly drops at epoch 6 (see gray curve on Fig1)\n* both train/val loss get higher abruptly at this epoch (gray curve of Fig2-3)\n\n# Assumed Cause\n\n* the gradient vanishes/explodes during training\n\n------\n\nAs @hengck23 pointed out, the root cause might be data.\nsee also this thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201\n\n\n# Assumed Remedy\n\n* to decrease leaning rate\n* to increase momentum weight when you use SGD, Adam, AdamW optimizer\n\n---\n\n**Fig1:**\n\"Screen-Shot-2022-01-28-at-20-36-28\"\n\n**Fig2:**\n\"Screen-Shot-2022-01-28-at-20-36-37\"\n\n**Fig3:**\n\"Screen-Shot-2022-01-28-at-20-36-45\"\n\n---\n\n# Update Note\n\n* 2022/1/28: Fully updated topic based on the actual case", "votes": null }, { "id": "1653008", "postDate": "01/17/2022 07:00:40", "content": "

memo: from the forum topic for the related symptom

\n
\n

1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set.
\n 2) Check your data to make sure there is no NaN in it (training, validation, test)
\n 3) Check the gradients and the weights to make sure there is no NaN.
\n 4) Decrease the learning rate as you train to make sure it's not because of a sudden big update that stuck in a sharp minima.
\n 5) To make sure everything's right, check the predictions of your network so that your network is not making some constant, or repetitive predictions.
\n 6) Check if your data in your batch is balanced with respect to all classes.
\n 7) normalize your data to be zero mean unit variance. Initialize the weights likewise. It will assist the training.

\n
\n

[1] https://stats.stackexchange.com/questions/228920/sudden-accuracy-drop-when-training-lstm-or-gru-in-keras
\n[2] https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net

", "rawMarkdown": "memo: from the forum topic for the related symptom\n\n> 1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set.\n2) Check your data to make sure there is no NaN in it (training, validation, test)\n3) Check the gradients and the weights to make sure there is no NaN.\n4) Decrease the learning rate as you train to make sure it's not because of a sudden big update that stuck in a sharp minima.\n5) To make sure everything's right, check the predictions of your network so that your network is not making some constant, or repetitive predictions.\n6) Check if your data in your batch is balanced with respect to all classes.\n7) normalize your data to be zero mean unit variance. Initialize the weights likewise. It will assist the training.\n\n[1] https://stats.stackexchange.com/questions/228920/sudden-accuracy-drop-when-training-lstm-or-gru-in-keras\n[2] https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net", "votes": null }, { "id": "1666969", "postDate": "01/28/2022 07:53:33", "content": "

Tips: Decreasing Leaning Rate & Increasing Momentum Parameter

\n

In my case, the following parameter tuning was effective, at least in some cases:

\n\n

One of the reason that the training process gets unstable is learning rate is too high.
\nIf it is the case, either decreasing learning rate or increasing momentum parameter (or both) will mitigate this problem.

", "rawMarkdown": "# Tips: Decreasing Leaning Rate & Increasing Momentum Parameter\n\nIn my case, the following parameter tuning was effective, at least in some cases:\n\n* decreasing learning rate\n* increase beta1 for AdamW optimizer (which is similar to momentum for SGD)\n\nOne of the reason that the training process gets unstable is learning rate is too high.\nIf it is the case, either decreasing learning rate or increasing momentum parameter (or both) will mitigate this problem.", "votes": null }, { "id": "1666979", "postDate": "01/28/2022 08:07:07", "content": "

Look at the top image below.
\nThe gray curve shows a sudden drop in mAP at the sixth epoch.
\nAlso, in the middle figure, the learning loss is increased abruptly at this epoch.
\nThis indicates that something is wrong with the learning (e.g., the gradient vanished/exploded, the structure of the trained model is broken, etc.).
\nIn this case, you can try to reduce the learning rate or increase the momentum with your optimizer.
\nFor the blue curve, I reduced the learning rate (0.01 -> 0.003) and increased the momentum (0.937 -> 0.979).
\nCompared to the gray curve, the learning curve becomes more stable.

\n

\"Screen-Shot-2022-01-28-at-16-54-02\"
\n\"Screen-Shot-2022-01-28-at-16-54-17\"
\n\"Screen-Shot-2022-01-28-at-16-54-30\"

", "rawMarkdown": "Look at the top image below.\nThe gray curve shows a sudden drop in mAP at the sixth epoch.\nAlso, in the middle figure, the learning loss is increased abruptly at this epoch.\nThis indicates that something is wrong with the learning (e.g., the gradient vanished/exploded, the structure of the trained model is broken, etc.).\nIn this case, you can try to reduce the learning rate or increase the momentum with your optimizer.\nFor the blue curve, I reduced the learning rate (0.01 -> 0.003) and increased the momentum (0.937 -> 0.979).\nCompared to the gray curve, the learning curve becomes more stable.\n\n\"Screen-Shot-2022-01-28-at-16-54-02\"\n\"Screen-Shot-2022-01-28-at-16-54-17\"\n\"Screen-Shot-2022-01-28-at-16-54-30\"", "votes": null }, { "id": "1667080", "postDate": "01/28/2022 09:48:46", "content": "

how to get momentum value (i.e. 0.979)?

", "rawMarkdown": "how to get momentum value (i.e. 0.979)?", "votes": null }, { "id": "1667129", "postDate": "01/28/2022 10:41:38", "content": "

@yienngxiong

\n

From this equation. This means the momentum is weighed x3 times.

\n
>>> 1 - ((1 - 0.937) / 3)\n0.979\n
\n

Reference

\n", "rawMarkdown": "yienngxiong \n\nFrom this equation. This means the momentum is weighed x3 times.\n\n```\n>>> 1 - ((1 - 0.937) / 3)\n0.979\n```\n\n# Reference\n\n- Adam: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html\n- SGD: https://pytorch.org/docs/stable/generated/torch.optim.SGD.html", "votes": null }, { "id": "1667140", "postDate": "01/28/2022 10:46:31", "content": "

As I am using AdamW optimizer, I specified beta1, but it's equivalent to momentum in SGD.

", "rawMarkdown": "As I am using AdamW optimizer, I specified beta1, but it's equivalent to momentum in SGD.", "votes": null }, { "id": "1667167", "postDate": "01/28/2022 11:11:00", "content": "

In other words, decreasing learning rate x1/3 and increasing momentum weight x3 means making the gradient of current step contributes x1/9 to the network parameter updates.

\n

This is effective to weaken “too much learning from just looking at the current batch”, and making your leaning curve smoother.

", "rawMarkdown": "In other words, decreasing learning rate x1/3 and increasing momentum weight x3 means making the gradient of current step contributes x1/9 to the network parameter updates.\n\nThis is effective to weaken “too much learning from just looking at the current batch”, and making your leaning curve smoother.", "votes": null }, { "id": "1667177", "postDate": "01/28/2022 11:19:01", "content": "

I think this tuning is also effective to increase stability on small batch size. However because of high variance of BN statistics in small batch size, the effect must be limited.

", "rawMarkdown": "I think this tuning is also effective to increase stability on small batch size. However because of high variance of BN statistics in small batch size, the effect must be limited.", "votes": null }, { "id": "1667201", "postDate": "01/28/2022 11:49:48", "content": "

the real reason for fluctuating loss curve is data.

\n

when the current batch of data is different from the previous one (or previous ones), the loss will also be different.
\nin short, it is effects of outliers.

\n

let me give you a few eamples. assume we are training a dog vs cat image classifier:

\n
    \n

  1. if for one batch, there is only one class (e.g. batch of 256 images and all label are dog), then the model will learn to predict dog with much higher probability after the batch update. when the next batch contains cat images, the loss suddenly increases.

    2. \n

  2. for one batch, the label are 100% wrongly labelled., then the model will predict the opposite after the batch update.
    \nwhen the next batch contains correct label, the loss suddenly increases.

    3. \n
\n
\n

mometum, learning rate limiting gradient values prevents accidential wrong udpate, but they do do not solve the root cause of the problem. if the outlier ratio is low, you can use such methods.

\n

if you really want to find out the root cause, you have to think of a way to find out the outliers.

", "rawMarkdown": "the real reason for fluctuating loss curve is data.\n\nwhen the current batch of data is different from the previous one (or previous ones), the loss will also be different.\nin short, it is effects of outliers.\n\nlet me give you a few eamples. assume we are training a dog vs cat image classifier:\n1. if for one batch, there is only one class (e.g. batch of 256 images and all label are dog), then the model will learn to predict dog with much higher probability after the batch update. when the next batch contains cat images, the loss suddenly increases.\n\n2. for one batch, the label are 100% wrongly labelled., then the model will predict the opposite after the batch update.\nwhen the next batch contains correct label, the loss suddenly increases.\n\n----\n\nmometum, learning rate limiting gradient values prevents accidential wrong udpate, but they do do not solve the root cause of the problem. if the outlier ratio is low, you can use such methods.\n\nif you really want to find out the root cause, you have to think of a way to find out the outliers.", "votes": null }, { "id": "1667223", "postDate": "01/28/2022 12:09:09", "content": "

@hengck23 Thanks for the comment.

\n

Actually, the gray curve is obtained by adding 30% of negative samples (whereas the pink curve is 0% negative samples).

\n

The data is sampled with a certain probability where I get sample ratio neg:pos=0.3:1.0 (I used full image datasets, then sample per every epoch to make data of this ratio.).

\n

So, If I take it for granted what you say (the root cause is data), I guess the most probable cause of fluctuation is based on the randomness of sampling.

\n

I thought random sampling is better than fixed size sampling (because it can learns on better noise), but in this situation, do you think this strategy is bad?

\n

(Currently I don't know how to test if the randomness really matters, but let me think about this.)

\n
\n

Additional Note

\n

on gray curve, both positive and negative data are sampled with certain probability, and they are sampled with repeat=True (namely, the same data can be sampled more than once).
\nOn the contrary, in the pink curve, each positive set is sampled only once (which is equivalent to repeat=False). So I guess the variance of sample is even bigger for the gray curve compared to pink one.

", "rawMarkdown": "hengck23 Thanks for the comment.\n\nActually, the gray curve is obtained by adding 30% of negative samples (whereas the pink curve is 0% negative samples).\n\nThe data is sampled with a certain probability where I get sample ratio neg:pos=0.3:1.0 (I used full image datasets, then sample per every epoch to make data of this ratio.).\n\nSo, If I take it for granted what you say (the root cause is data), I guess the most probable cause of fluctuation is based on the randomness of sampling.\n\nI thought random sampling is better than fixed size sampling (because it can learns on better noise), but in this situation, do you think this strategy is bad?\n\n(Currently I don't know how to test if the randomness really matters, but let me think about this.)\n\n---\n\n## Additional Note\n\non gray curve, both positive and negative data are sampled with certain probability, and they are sampled with `repeat=True` (namely, the same data can be sampled more than once).\nOn the contrary, in the pink curve, each positive set is sampled only once (which is equivalent to `repeat=False`). So I guess the variance of sample is even bigger for the gray curve compared to pink one.", "votes": null }, { "id": "1667242", "postDate": "01/28/2022 12:26:41", "content": "

According to @hengck23 , tuning the learning rate and momentum won't solve the root cause.

\n

See also this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201

", "rawMarkdown": "According to @hengck23 , tuning the learning rate and momentum won't solve the root cause.\n\nSee also this thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201", "votes": null }, { "id": "1667257", "postDate": "01/28/2022 12:35:04", "content": "

i think there are some COTS that looks like background. this cuases confusion when learning.
\n(also there are some blur small COTS in the ground truth, if you enlarge them in scale augmentation, there are may become bad samples. The larger COTS in the ground truth are not blur)

\n

when you do negative background sampling, you may sample background that looks like COTS.
\nthe ratio affects the frequency of such occurrence, etc

", "rawMarkdown": "i think there are some COTS that looks like background. this cuases confusion when learning.\n(also there are some blur small COTS in the ground truth, if you enlarge them in scale augmentation, there are may become bad samples. The larger COTS in the ground truth are not blur)\n\nwhen you do negative background sampling, you may sample background that looks like COTS.\nthe ratio affects the frequency of such occurrence, etc", "votes": null }, { "id": "1667273", "postDate": "01/28/2022 12:42:05", "content": "

\"mAP suddenly drops at epoch 6 (see gray curve on Fig1)\"

\n

most kagglers will experience decrease up to 6,7,8 for using video 1 as validation.

\n

for me although the map raises again, it finally will fall back again as I uses more train iterations. I use specialised augmentation and EMA for this.

\n
\n

this can be explained as follows:

\n", "rawMarkdown": "\"mAP suddenly drops at epoch 6 (see gray curve on Fig1)\"\n\nmost kagglers will experience decrease up to 6,7,8 for using video 1 as validation.\n\nfor me although the map raises again, it finally will fall back again as I uses more train iterations. I use specialised augmentation and EMA for this.\n\n---\n\nthis can be explained as follows:\n- deep models learned general target object charateristics at first. (generalisation is good, loss curve falls rapidly)\n- then it starts to learn (and sometimes wrongly overfitted to) noise in the data (the loss of the good data becomes very low. only bad data has high loss and contribute more to backprop gradient)", "votes": null }, { "id": "1667279", "postDate": "01/28/2022 12:49:42", "content": "

I see. I guess the un-labeled positive COTS (which is discussed in other discussions) might also harm learning.

", "rawMarkdown": "I see. I guess the un-labeled positive COTS (which is discussed in other discussions) might also harm learning.", "votes": null }, { "id": "1667289", "postDate": "01/28/2022 13:01:38", "content": "

I see. It might be too fast judging the gray curve is failed.

\n

BTW, can I ask what EMA is?

", "rawMarkdown": "I see. It might be too fast judging the gray curve is failed.\n\nBTW, can I ask what EMA is?", "votes": null }, { "id": "1667413", "postDate": "01/28/2022 14:58:18", "content": "

Related Paper about Unlabeled Data/Noisy Labels

\n", "rawMarkdown": "# Related Paper about Unlabeled Data/Noisy Labels\n\n- [Positive-Unlabeled Data Purification in the Wild for Object Detection](https://openaccess.thecvf.com/content/CVPR2021/papers/Guo_Positive-Unlabeled_Data_Purification_in_the_Wild_for_Object_Detection_CVPR_2021_paper.pdf)\n- [A Simple Semi-Supervised Learning Framework for Object Detection](https://arxiv.org/pdf/2005.04757.pdf)\n- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://paperswithcode.com/paper/end-to-end-semi-supervised-object-detection)\n- [Unbiased Teacher for Semi-Supervised Object Detection](https://paperswithcode.com/paper/unbiased-teacher-for-semi-supervised-object-1)\n- [Few-Example Object Detection with Model Communication](https://arxiv.org/pdf/1706.08249.pdf)", "votes": null } ], "comments": [ { "id": 1653008, "author_name": "tatamikenn", "author_url": "", "post_date": "01/17/2022 07:00:40", "content": "

memo: from the forum topic for the related symptom

\n
\n

1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set.
\n 2) Check your data to make sure there is no NaN in it (training, validation, test)
\n 3) Check the gradients and the weights to make sure there is no NaN.
\n 4) Decrease the learning rate as you train to make sure it's not because of a sudden big update that stuck in a sharp minima.
\n 5) To make sure everything's right, check the predictions of your network so that your network is not making some constant, or repetitive predictions.
\n 6) Check if your data in your batch is balanced with respect to all classes.
\n 7) normalize your data to be zero mean unit variance. Initialize the weights likewise. It will assist the training.

\n
\n

[1] https://stats.stackexchange.com/questions/228920/sudden-accuracy-drop-when-training-lstm-or-gru-in-keras
\n[2] https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net

", "votes": null, "replies": [] }, { "id": 1666969, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 07:53:33", "content": "

Tips: Decreasing Leaning Rate & Increasing Momentum Parameter

\n

In my case, the following parameter tuning was effective, at least in some cases:

\n\n

One of the reason that the training process gets unstable is learning rate is too high.
\nIf it is the case, either decreasing learning rate or increasing momentum parameter (or both) will mitigate this problem.

", "votes": null, "replies": [ { "id": 1666979, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 08:07:07", "content": "

Look at the top image below.
\nThe gray curve shows a sudden drop in mAP at the sixth epoch.
\nAlso, in the middle figure, the learning loss is increased abruptly at this epoch.
\nThis indicates that something is wrong with the learning (e.g., the gradient vanished/exploded, the structure of the trained model is broken, etc.).
\nIn this case, you can try to reduce the learning rate or increase the momentum with your optimizer.
\nFor the blue curve, I reduced the learning rate (0.01 -> 0.003) and increased the momentum (0.937 -> 0.979).
\nCompared to the gray curve, the learning curve becomes more stable.

\n

\"Screen-Shot-2022-01-28-at-16-54-02\"
\n\"Screen-Shot-2022-01-28-at-16-54-17\"
\n\"Screen-Shot-2022-01-28-at-16-54-30\"

", "votes": null, "replies": [] }, { "id": 1667080, "author_name": "yienngxiong", "author_url": "", "post_date": "01/28/2022 09:48:46", "content": "

how to get momentum value (i.e. 0.979)?

", "votes": null, "replies": [] }, { "id": 1667129, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 10:41:38", "content": "

@yienngxiong

\n

From this equation. This means the momentum is weighed x3 times.

\n
>>> 1 - ((1 - 0.937) / 3)\n0.979\n
\n

Reference

\n", "votes": null, "replies": [] }, { "id": 1667140, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 10:46:31", "content": "

As I am using AdamW optimizer, I specified beta1, but it's equivalent to momentum in SGD.

", "votes": null, "replies": [] }, { "id": 1667167, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 11:11:00", "content": "

In other words, decreasing learning rate x1/3 and increasing momentum weight x3 means making the gradient of current step contributes x1/9 to the network parameter updates.

\n

This is effective to weaken “too much learning from just looking at the current batch”, and making your leaning curve smoother.

", "votes": null, "replies": [] }, { "id": 1667177, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 11:19:01", "content": "

I think this tuning is also effective to increase stability on small batch size. However because of high variance of BN statistics in small batch size, the effect must be limited.

", "votes": null, "replies": [] }, { "id": 1667242, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 12:26:41", "content": "

According to @hengck23 , tuning the learning rate and momentum won't solve the root cause.

\n

See also this thread:
\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201

", "votes": null, "replies": [] } ] }, { "id": 1667201, "author_name": "hengck23", "author_url": "", "post_date": "01/28/2022 11:49:48", "content": "

the real reason for fluctuating loss curve is data.

\n

when the current batch of data is different from the previous one (or previous ones), the loss will also be different.
\nin short, it is effects of outliers.

\n

let me give you a few eamples. assume we are training a dog vs cat image classifier:

\n
    \n

  1. if for one batch, there is only one class (e.g. batch of 256 images and all label are dog), then the model will learn to predict dog with much higher probability after the batch update. when the next batch contains cat images, the loss suddenly increases.

    2. \n

  2. for one batch, the label are 100% wrongly labelled., then the model will predict the opposite after the batch update.
    \nwhen the next batch contains correct label, the loss suddenly increases.

    3. \n
\n
\n

mometum, learning rate limiting gradient values prevents accidential wrong udpate, but they do do not solve the root cause of the problem. if the outlier ratio is low, you can use such methods.

\n

if you really want to find out the root cause, you have to think of a way to find out the outliers.

", "votes": null, "replies": [ { "id": 1667223, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 12:09:09", "content": "

@hengck23 Thanks for the comment.

\n

Actually, the gray curve is obtained by adding 30% of negative samples (whereas the pink curve is 0% negative samples).

\n

The data is sampled with a certain probability where I get sample ratio neg:pos=0.3:1.0 (I used full image datasets, then sample per every epoch to make data of this ratio.).

\n

So, If I take it for granted what you say (the root cause is data), I guess the most probable cause of fluctuation is based on the randomness of sampling.

\n

I thought random sampling is better than fixed size sampling (because it can learns on better noise), but in this situation, do you think this strategy is bad?

\n

(Currently I don't know how to test if the randomness really matters, but let me think about this.)

\n
\n

Additional Note

\n

on gray curve, both positive and negative data are sampled with certain probability, and they are sampled with repeat=True (namely, the same data can be sampled more than once).
\nOn the contrary, in the pink curve, each positive set is sampled only once (which is equivalent to repeat=False). So I guess the variance of sample is even bigger for the gray curve compared to pink one.

", "votes": null, "replies": [] }, { "id": 1667257, "author_name": "hengck23", "author_url": "", "post_date": "01/28/2022 12:35:04", "content": "

i think there are some COTS that looks like background. this cuases confusion when learning.
\n(also there are some blur small COTS in the ground truth, if you enlarge them in scale augmentation, there are may become bad samples. The larger COTS in the ground truth are not blur)

\n

when you do negative background sampling, you may sample background that looks like COTS.
\nthe ratio affects the frequency of such occurrence, etc

", "votes": null, "replies": [] }, { "id": 1667279, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 12:49:42", "content": "

I see. I guess the un-labeled positive COTS (which is discussed in other discussions) might also harm learning.

", "votes": null, "replies": [] } ] }, { "id": 1667273, "author_name": "hengck23", "author_url": "", "post_date": "01/28/2022 12:42:05", "content": "

\"mAP suddenly drops at epoch 6 (see gray curve on Fig1)\"

\n

most kagglers will experience decrease up to 6,7,8 for using video 1 as validation.

\n

for me although the map raises again, it finally will fall back again as I uses more train iterations. I use specialised augmentation and EMA for this.

\n
\n

this can be explained as follows:

\n", "votes": null, "replies": [ { "id": 1667289, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 13:01:38", "content": "

I see. It might be too fast judging the gray curve is failed.

\n

BTW, can I ask what EMA is?

", "votes": null, "replies": [] } ] }, { "id": 1667413, "author_name": "tatamikenn", "author_url": "", "post_date": "01/28/2022 14:58:18", "content": "

Related Paper about Unlabeled Data/Noisy Labels

\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1650606": "**Note for those who follows the original topic:** I fully rewrote this topic based on the actual case I encountered. I think this is better compared to discussing about the general case.\n\n# Simptom\n\n* mAP suddenly drops at epoch 6 (see gray curve on Fig1)\n* both train/val loss get higher abruptly at this epoch (gray curve of Fig2-3)\n\n# Assumed Cause\n\n* the gradient vanishes/explodes during training\n\n------\n\nAs @hengck23 pointed out, the root cause might be data.\nsee also this thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201\n\n\n# Assumed Remedy\n\n* to decrease leaning rate\n* to increase momentum weight when you use SGD, Adam, AdamW optimizer\n\n---\n\n**Fig1:**\n\"Screen-Shot-2022-01-28-at-20-36-28\"\n\n**Fig2:**\n\"Screen-Shot-2022-01-28-at-20-36-37\"\n\n**Fig3:**\n\"Screen-Shot-2022-01-28-at-20-36-45\"\n\n---\n\n# Update Note\n\n* 2022/1/28: Fully updated topic based on the actual case", "1653008": "memo: from the forum topic for the related symptom\n\n> 1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set.\n2) Check your data to make sure there is no NaN in it (training, validation, test)\n3) Check the gradients and the weights to make sure there is no NaN.\n4) Decrease the learning rate as you train to make sure it's not because of a sudden big update that stuck in a sharp minima.\n5) To make sure everything's right, check the predictions of your network so that your network is not making some constant, or repetitive predictions.\n6) Check if your data in your batch is balanced with respect to all classes.\n7) normalize your data to be zero mean unit variance. Initialize the weights likewise. It will assist the training.\n\n[1] https://stats.stackexchange.com/questions/228920/sudden-accuracy-drop-when-training-lstm-or-gru-in-keras\n[2] https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net", "1666969": "# Tips: Decreasing Leaning Rate & Increasing Momentum Parameter\n\nIn my case, the following parameter tuning was effective, at least in some cases:\n\n* decreasing learning rate\n* increase beta1 for AdamW optimizer (which is similar to momentum for SGD)\n\nOne of the reason that the training process gets unstable is learning rate is too high.\nIf it is the case, either decreasing learning rate or increasing momentum parameter (or both) will mitigate this problem.", "1666979": "Look at the top image below.\nThe gray curve shows a sudden drop in mAP at the sixth epoch.\nAlso, in the middle figure, the learning loss is increased abruptly at this epoch.\nThis indicates that something is wrong with the learning (e.g., the gradient vanished/exploded, the structure of the trained model is broken, etc.).\nIn this case, you can try to reduce the learning rate or increase the momentum with your optimizer.\nFor the blue curve, I reduced the learning rate (0.01 -> 0.003) and increased the momentum (0.937 -> 0.979).\nCompared to the gray curve, the learning curve becomes more stable.\n\n\"Screen-Shot-2022-01-28-at-16-54-02\"\n\"Screen-Shot-2022-01-28-at-16-54-17\"\n\"Screen-Shot-2022-01-28-at-16-54-30\"", "1667080": "how to get momentum value (i.e. 0.979)?", "1667129": "yienngxiong \n\nFrom this equation. This means the momentum is weighed x3 times.\n\n```\n>>> 1 - ((1 - 0.937) / 3)\n0.979\n```\n\n# Reference\n\n- Adam: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html\n- SGD: https://pytorch.org/docs/stable/generated/torch.optim.SGD.html", "1667140": "As I am using AdamW optimizer, I specified beta1, but it's equivalent to momentum in SGD.", "1667167": "In other words, decreasing learning rate x1/3 and increasing momentum weight x3 means making the gradient of current step contributes x1/9 to the network parameter updates.\n\nThis is effective to weaken “too much learning from just looking at the current batch”, and making your leaning curve smoother.", "1667177": "I think this tuning is also effective to increase stability on small batch size. However because of high variance of BN statistics in small batch size, the effect must be limited.", "1667201": "the real reason for fluctuating loss curve is data.\n\nwhen the current batch of data is different from the previous one (or previous ones), the loss will also be different.\nin short, it is effects of outliers.\n\nlet me give you a few eamples. assume we are training a dog vs cat image classifier:\n1. if for one batch, there is only one class (e.g. batch of 256 images and all label are dog), then the model will learn to predict dog with much higher probability after the batch update. when the next batch contains cat images, the loss suddenly increases.\n\n2. for one batch, the label are 100% wrongly labelled., then the model will predict the opposite after the batch update.\nwhen the next batch contains correct label, the loss suddenly increases.\n\n----\n\nmometum, learning rate limiting gradient values prevents accidential wrong udpate, but they do do not solve the root cause of the problem. if the outlier ratio is low, you can use such methods.\n\nif you really want to find out the root cause, you have to think of a way to find out the outliers.", "1667223": "hengck23 Thanks for the comment.\n\nActually, the gray curve is obtained by adding 30% of negative samples (whereas the pink curve is 0% negative samples).\n\nThe data is sampled with a certain probability where I get sample ratio neg:pos=0.3:1.0 (I used full image datasets, then sample per every epoch to make data of this ratio.).\n\nSo, If I take it for granted what you say (the root cause is data), I guess the most probable cause of fluctuation is based on the randomness of sampling.\n\nI thought random sampling is better than fixed size sampling (because it can learns on better noise), but in this situation, do you think this strategy is bad?\n\n(Currently I don't know how to test if the randomness really matters, but let me think about this.)\n\n---\n\n## Additional Note\n\non gray curve, both positive and negative data are sampled with certain probability, and they are sampled with `repeat=True` (namely, the same data can be sampled more than once).\nOn the contrary, in the pink curve, each positive set is sampled only once (which is equivalent to `repeat=False`). So I guess the variance of sample is even bigger for the gray curve compared to pink one.", "1667242": "According to @hengck23 , tuning the learning rate and momentum won't solve the root cause.\n\nSee also this thread:\nhttps://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/300960#1667201", "1667257": "i think there are some COTS that looks like background. this cuases confusion when learning.\n(also there are some blur small COTS in the ground truth, if you enlarge them in scale augmentation, there are may become bad samples. The larger COTS in the ground truth are not blur)\n\nwhen you do negative background sampling, you may sample background that looks like COTS.\nthe ratio affects the frequency of such occurrence, etc", "1667273": "\"mAP suddenly drops at epoch 6 (see gray curve on Fig1)\"\n\nmost kagglers will experience decrease up to 6,7,8 for using video 1 as validation.\n\nfor me although the map raises again, it finally will fall back again as I uses more train iterations. I use specialised augmentation and EMA for this.\n\n---\n\nthis can be explained as follows:\n- deep models learned general target object charateristics at first. (generalisation is good, loss curve falls rapidly)\n- then it starts to learn (and sometimes wrongly overfitted to) noise in the data (the loss of the good data becomes very low. only bad data has high loss and contribute more to backprop gradient)", "1667279": "I see. I guess the un-labeled positive COTS (which is discussed in other discussions) might also harm learning.", "1667289": "I see. It might be too fast judging the gray curve is failed.\n\nBTW, can I ask what EMA is?", "1667413": "# Related Paper about Unlabeled Data/Noisy Labels\n\n- [Positive-Unlabeled Data Purification in the Wild for Object Detection](https://openaccess.thecvf.com/content/CVPR2021/papers/Guo_Positive-Unlabeled_Data_Purification_in_the_Wild_for_Object_Detection_CVPR_2021_paper.pdf)\n- [A Simple Semi-Supervised Learning Framework for Object Detection](https://arxiv.org/pdf/2005.04757.pdf)\n- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://paperswithcode.com/paper/end-to-end-semi-supervised-object-detection)\n- [Unbiased Teacher for Semi-Supervised Object Detection](https://paperswithcode.com/paper/unbiased-teacher-for-semi-supervised-object-1)\n- [Few-Example Object Detection with Model Communication](https://arxiv.org/pdf/1706.08249.pdf)" }, "source": "meta" }