Among other things that worked that I will partly share soon, here are 7 things that for me did not work, they either give the same cv/public leaderboard result as without them or lower
\nEven if everybody wants to know what works, I think it's always helpful for the community to know what failed
\nOptimizers: AdamW and RangerLars did not provide better results than good old ADAM.
Very large architectures (Effnet B8, Effnet B7, Effnet B6). For these types of architectures, my limited hardware capabilities are forcing me to enter with a very low batch size (even using apex) which leads to noisy gradient backpropagation on a noisy set, which is a very bad combination. If you are having an advanced GPU which supports more than 11GB you may have better results.
So far the old data from the similar competition did not bring any improvements in my models. Here you can get very easily tricked on the OOF score if you are accumulating the data and then stratify them into folds. These kind of auxiliary data even if is coming from an almost identical competition is mandatory to not enter the validation set. Otherwise, it will not be representative and correlated to the leaderboard.
\nI know that this worked for some of you, so it is not completely out of my todo list
Image size smaller than 512 using random resize crop. Although, using random resized crop with default parameters (range of size of the origin size cropped: 0.08, 1.0 and range of aspect ratio of the origin aspect ratio cropped: 0.75, 1.33 ) decreasing the image size seems to not perform. One thing to try here is the find the optimum parameters of random resize crop methodology that will allow smaller images which will further allow higher batch sizes.
Too small arhitectures (Resnet18, Resnet 50, even Effnet B0) are underfiting comparative with B2's, B3's and B4's
Fmix did not give better results than Cutmix (On Bengali competition where this technique gave good results, it helped create new symbols by spiting core components(roots+vocals+consonant) that were not provided by the training data but found on the testing set.
OHEM loss, gamblers loss and also focal loss did not performed better than CE for me. Did not tried yet cosine focal loss
Among other things that worked that I will partly share soon, here are 7 things that for me did not work, they either give the same cv/public leaderboard result as without them or lower
\nEven if everybody wants to know what works, I think it's always helpful for the community to know what failed
\nOptimizers: AdamW and RangerLars did not provide better results than good old ADAM.
Very large architectures (Effnet B8, Effnet B7, Effnet B6). For these types of architectures, my limited hardware capabilities are forcing me to enter with a very low batch size (even using apex) which leads to noisy gradient backpropagation on a noisy set, which is a very bad combination. If you are having an advanced GPU which supports more than 11GB you may have better results.
So far the old data from the similar competition did not bring any improvements in my models. Here you can get very easily tricked on the OOF score if you are accumulating the data and then stratify them into folds. These kind of auxiliary data even if is coming from an almost identical competition is mandatory to not enter the validation set. Otherwise, it will not be representative and correlated to the leaderboard.
\nI know that this worked for some of you, so it is not completely out of my todo list
Image size smaller than 512 using random resize crop. Although, using random resized crop with default parameters (range of size of the origin size cropped: 0.08, 1.0 and range of aspect ratio of the origin aspect ratio cropped: 0.75, 1.33 ) decreasing the image size seems to not perform. One thing to try here is the find the optimum parameters of random resize crop methodology that will allow smaller images which will further allow higher batch sizes.
Too small arhitectures (Resnet18, Resnet 50, even Effnet B0) are underfiting comparative with B2's, B3's and B4's
Fmix did not give better results than Cutmix (On Bengali competition where this technique gave good results, it helped create new symbols by spiting core components(roots+vocals+consonant) that were not provided by the training data but found on the testing set.
OHEM loss, gamblers loss and also focal loss did not performed better than CE for me. Did not tried yet cosine focal loss
Thanks for sharing.
\n1) I agree Adam is working better than AdamW
\n2) Yes smaller architectures are underfitting atleast for me.
\n3) I haven't tried OHEM loss. Can you please share your implementation ?
What does OHEM loss do:
\nGiven a batch sized K, it performs regular forward propagation and computes per instance losses. Then, it finds M<K hard examples in the batch with high loss values and it only back-propagates the loss computed over the selected instances.
This is the implementation:
\ndef ohem_loss( rate, cls_pred, cls_target ):
\n batch_size = cls_pred.size(0)
\n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)
\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)
\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )
\n if keep_num < sorted_ohem_loss.size()[0]:
\n keep_idx_cuda = idx[:keep_num]
\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]
\n cls_loss = ohem_cls_loss.sum() / keep_num
\n return cls_loss
@tyqiangz
\nThis is the one:
\nhttps://www.kaggle.com/c/cassava-disease
Also for me, Cutmix works better than FMix.
", "rawMarkdown": "Also for me, Cutmix works better than FMix.", "votes": null }, { "id": "1120340", "postDate": "12/20/2020 18:48:29", "content": "Hey i have implemented gamblers loss
\nI trained effnet using CE loss and got 80% acc and then i use gamblers loss and for 10 epochs its accuracy was 20-30%.
\nAs you said gamblers loss did not worked out, By this you mean same or your accuracy is not this bad ?
\nCan you please have look to my loss function
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/205424
Just to verify implementation was correct.
", "rawMarkdown": "Hey i have implemented **gamblers loss** \n\nI trained effnet using CE loss and got 80% acc and then i use gamblers loss and for 10 epochs its accuracy was 20-30%.\n\nAs you said gamblers loss did not worked out, By this you mean same or your accuracy is not this bad ? \n\nCan you please have look to my loss function \nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/205424\n\nJust to verify implementation was correct.", "votes": null }, { "id": "1120353", "postDate": "12/20/2020 19:00:03", "content": "Something ive noticed is that i get very different CV scores on the same folds depending if i use original aspect ratio, or using resize or using centercrop. So ensembling these models could help!
", "rawMarkdown": "Something ive noticed is that i get very different CV scores on the same folds depending if i use original aspect ratio, or using resize or using centercrop. So ensembling these models could help!", "votes": null }, { "id": "1120618", "postDate": "12/21/2020 01:37:22", "content": "Compared to ADAM, does SGD with momentum perform better, except for training speed?
", "rawMarkdown": "Compared to ADAM, does SGD with momentum perform better, except for training speed?", "votes": null }, { "id": "1120941", "postDate": "12/21/2020 08:15:29", "content": "When you tried to train deeper nets (B7, B6) did you use Adam or SGD?
\nAdam is more memory consuming, so SGD could save you a bit of VRAM.
@altprof , @sjtuyxc Did not tried SGD yet. It is on my todo list.
", "rawMarkdown": "altprof , @sjtuyxc Did not tried SGD yet. It is on my todo list.", "votes": null }, { "id": "1121220", "postDate": "12/21/2020 13:05:49", "content": "Thanks~ I got it!
", "rawMarkdown": "Thanks~ I got it!", "votes": null }, { "id": "1121307", "postDate": "12/21/2020 14:34:45", "content": "Good information ! Great
", "rawMarkdown": "Good information ! Great", "votes": null }, { "id": "1122480", "postDate": "12/22/2020 13:24:11", "content": "I had a similar experience with effnets and image sizes.
\nI was using older data and saw some small accuracy increases. I'm curious for those who might have seen increases using these as validation of what their methods may have been.
", "rawMarkdown": "I had a similar experience with effnets and image sizes. \n\nI was using older data and saw some small accuracy increases. I'm curious for those who might have seen increases using these as validation of what their methods may have been.", "votes": null }, { "id": "1122532", "postDate": "12/22/2020 13:57:30", "content": "I have similar results for my experiments:
\n512x512 give me worse performanceEfficientNet B3-B6 are the best architectures i've found so far~0.01I use as data augmentation techniques CutMix and MixUp + random flips, crops and rotations, but maybe it's worth a try to implement snapmix and see if it gives any improvements.
\nI'm planning to use bi-tempered logistic loss, but up to now my results are the same as those obtained with standard cross-entropy (single fold model LB score 0.890)
@lazcoder you using CutMix and MixUp in the same time ?
", "rawMarkdown": "lazcoder you using CutMix and MixUp in the same time ?", "votes": null }, { "id": "1123540", "postDate": "12/23/2020 10:07:45", "content": "@rajan The accuracy was lower pretty significant with gambler loss. Personally, I would go with other losses for this competition
", "rawMarkdown": "rajan The accuracy was lower pretty significant with gambler loss. Personally, I would go with other losses for this competition", "votes": null }, { "id": "1123543", "postDate": "12/23/2020 10:09:07", "content": "Training resize or validation resize?
", "rawMarkdown": "Training resize or validation resize?", "votes": null }, { "id": "1123546", "postDate": "12/23/2020 10:13:43", "content": "No, I use either CutMix or MixUp randomly on each batch.
\nI use something like
\nif p > p_mix:\n do_mixup(batch)\nelse:\n do_cutmix(batch)\nYes, I agree, It's the best way to do it !
", "rawMarkdown": "Yes, I agree, It's the best way to do it !", "votes": null }, { "id": "1123556", "postDate": "12/23/2020 10:21:28", "content": "Actually I've read somewhere that mixup is harmful for this competition (even if in my experiments it seems not).
\nMaybe an idea that is worth to try is to use snapmix instead of mixup
If you use them both in the same time, use them separately like in your code piece
\nThe opportunity of using one or both of them in this competition is another discussion.
\nFor example, I am using just one of them
Normal SGD worked better than SGD with momentum of 0.9 for me (learning rate=0.001 on both cases).
", "rawMarkdown": "Normal SGD worked better than SGD with momentum of 0.9 for me (learning rate=0.001 on both cases).", "votes": null }, { "id": "1137023", "postDate": "01/03/2021 15:53:53", "content": "\n\n\n
\n- Image size smaller than 512 using random resize crop. Although, using random resized crop with default parameters (range of size of the origin size cropped: 0.08, 1.0 and range of aspect ratio of the origin aspect ratio cropped: 0.75, 1.33 ) decreasing the image size seems to not perform. One thing to try here is the find the optimum parameters of random resize crop methodology that will allow smaller images which will further allow higher batch sizes.
\n
(1) What do you mean by 'Image size smaller than 512' ? The image size in this competition is all 800x600?
\n(2) Maybe the cropped lower limit 0.08 is too small, we need to set it to 0.6 or 0.8 ? Thanks!
i have used
\nGetting 87.7 max accuracy
\nmy notebook
\nhttps://www.kaggle.com/rajanlagah/noob-programmer-s-code
Will appreciate your feedback
", "rawMarkdown": "i have used \n- LabelSmooth\n- cutmix\n- res50 \n- eff B3-b5\n- SCELoss\n- TTA\n- Ranger\nWhat else can i do ? I have seen many people with high accuracy suggesting some one of them. What it is that i am missing \n \nGetting 87.7 max accuracy \n\nmy notebook \nhttps://www.kaggle.com/rajanlagah/noob-programmer-s-code\n\nWill appreciate your feedback", "votes": null }, { "id": "1137677", "postDate": "01/04/2021 06:33:23", "content": "how to calculate training acc when using mixup? ( it becomes two labels, such as class_A=0.4 and class_B = 0.6)
", "rawMarkdown": "how to calculate training acc when using mixup? ( it becomes two labels, such as class_A=0.4 and class_B = 0.6)", "votes": null }, { "id": "1144909", "postDate": "01/08/2021 18:36:57", "content": "Why didn't you try it on Google Colab, it would have given you better compute.
", "rawMarkdown": "Why didn't you try it on Google Colab, it would have given you better compute.", "votes": null }, { "id": "1226651", "postDate": "03/04/2021 18:03:17", "content": "what about
\nThanks for sharing.
\n1) I agree Adam is working better than AdamW
\n2) Yes smaller architectures are underfitting atleast for me.
\n3) I haven't tried OHEM loss. Can you please share your implementation ?
What does OHEM loss do:
\nGiven a batch sized K, it performs regular forward propagation and computes per instance losses. Then, it finds M<K hard examples in the batch with high loss values and it only back-propagates the loss computed over the selected instances.
This is the implementation:
\ndef ohem_loss( rate, cls_pred, cls_target ):
\n batch_size = cls_pred.size(0)
\n ohem_cls_loss = F.cross_entropy(cls_pred, cls_target, reduction='none', ignore_index=-1)
\n sorted_ohem_loss, idx = torch.sort(ohem_cls_loss, descending=True)
\n keep_num = min(sorted_ohem_loss.size()[0], int(batch_size*rate) )
\n if keep_num < sorted_ohem_loss.size()[0]:
\n keep_idx_cuda = idx[:keep_num]
\n ohem_cls_loss = ohem_cls_loss[keep_idx_cuda]
\n cls_loss = ohem_cls_loss.sum() / keep_num
\n return cls_loss
What's the \"similar competition\" that you spoke about? Mind tagging a link here?
", "votes": null, "replies": [ { "id": 1120013, "author_name": "vladvdv", "author_url": "", "post_date": "12/20/2020 14:49:19", "content": "@tyqiangz
\nThis is the one:
\nhttps://www.kaggle.com/c/cassava-disease
Also for me, Cutmix works better than FMix.
", "votes": null, "replies": [] }, { "id": 1120340, "author_name": "rajanlagah", "author_url": "", "post_date": "12/20/2020 18:48:29", "content": "Hey i have implemented gamblers loss
\nI trained effnet using CE loss and got 80% acc and then i use gamblers loss and for 10 epochs its accuracy was 20-30%.
\nAs you said gamblers loss did not worked out, By this you mean same or your accuracy is not this bad ?
\nCan you please have look to my loss function
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/205424
Just to verify implementation was correct.
", "votes": null, "replies": [ { "id": 1123540, "author_name": "vladvdv", "author_url": "", "post_date": "12/23/2020 10:07:45", "content": "@rajan The accuracy was lower pretty significant with gambler loss. Personally, I would go with other losses for this competition
", "votes": null, "replies": [] }, { "id": 1137161, "author_name": "rajanlagah", "author_url": "", "post_date": "01/03/2021 18:16:24", "content": "i have used
\nGetting 87.7 max accuracy
\nmy notebook
\nhttps://www.kaggle.com/rajanlagah/noob-programmer-s-code
Will appreciate your feedback
", "votes": null, "replies": [] } ] }, { "id": 1120353, "author_name": "yannmajewski", "author_url": "", "post_date": "12/20/2020 19:00:03", "content": "Something ive noticed is that i get very different CV scores on the same folds depending if i use original aspect ratio, or using resize or using centercrop. So ensembling these models could help!
", "votes": null, "replies": [ { "id": 1123543, "author_name": "vladvdv", "author_url": "", "post_date": "12/23/2020 10:09:07", "content": "Training resize or validation resize?
", "votes": null, "replies": [] } ] }, { "id": 1120618, "author_name": "sjtuyxc", "author_url": "", "post_date": "12/21/2020 01:37:22", "content": "Compared to ADAM, does SGD with momentum perform better, except for training speed?
", "votes": null, "replies": [ { "id": 1125896, "author_name": "suryaaseran", "author_url": "", "post_date": "12/25/2020 06:50:54", "content": "Normal SGD worked better than SGD with momentum of 0.9 for me (learning rate=0.001 on both cases).
", "votes": null, "replies": [] } ] }, { "id": 1120941, "author_name": "altprof", "author_url": "", "post_date": "12/21/2020 08:15:29", "content": "When you tried to train deeper nets (B7, B6) did you use Adam or SGD?
\nAdam is more memory consuming, so SGD could save you a bit of VRAM.
@altprof , @sjtuyxc Did not tried SGD yet. It is on my todo list.
", "votes": null, "replies": [ { "id": 1121220, "author_name": "sjtuyxc", "author_url": "", "post_date": "12/21/2020 13:05:49", "content": "Thanks~ I got it!
", "votes": null, "replies": [] } ] }, { "id": 1121307, "author_name": "", "author_url": "", "post_date": "12/21/2020 14:34:45", "content": "Good information ! Great
", "votes": null, "replies": [] }, { "id": 1122480, "author_name": "crained", "author_url": "", "post_date": "12/22/2020 13:24:11", "content": "I had a similar experience with effnets and image sizes.
\nI was using older data and saw some small accuracy increases. I'm curious for those who might have seen increases using these as validation of what their methods may have been.
", "votes": null, "replies": [] }, { "id": 1122532, "author_name": "lazcoder", "author_url": "", "post_date": "12/22/2020 13:57:30", "content": "I have similar results for my experiments:
\n512x512 give me worse performanceEfficientNet B3-B6 are the best architectures i've found so far~0.01I use as data augmentation techniques CutMix and MixUp + random flips, crops and rotations, but maybe it's worth a try to implement snapmix and see if it gives any improvements.
\nI'm planning to use bi-tempered logistic loss, but up to now my results are the same as those obtained with standard cross-entropy (single fold model LB score 0.890)
@lazcoder you using CutMix and MixUp in the same time ?
", "votes": null, "replies": [] }, { "id": 1123546, "author_name": "lazcoder", "author_url": "", "post_date": "12/23/2020 10:13:43", "content": "No, I use either CutMix or MixUp randomly on each batch.
\nI use something like
\nif p > p_mix:\n do_mixup(batch)\nelse:\n do_cutmix(batch)\nYes, I agree, It's the best way to do it !
", "votes": null, "replies": [] }, { "id": 1123556, "author_name": "lazcoder", "author_url": "", "post_date": "12/23/2020 10:21:28", "content": "Actually I've read somewhere that mixup is harmful for this competition (even if in my experiments it seems not).
\nMaybe an idea that is worth to try is to use snapmix instead of mixup
If you use them both in the same time, use them separately like in your code piece
\nThe opportunity of using one or both of them in this competition is another discussion.
\nFor example, I am using just one of them
how to calculate training acc when using mixup? ( it becomes two labels, such as class_A=0.4 and class_B = 0.6)
", "votes": null, "replies": [] } ] }, { "id": 1137023, "author_name": "clwwlc", "author_url": "", "post_date": "01/03/2021 15:53:53", "content": "\n\n\n
\n- Image size smaller than 512 using random resize crop. Although, using random resized crop with default parameters (range of size of the origin size cropped: 0.08, 1.0 and range of aspect ratio of the origin aspect ratio cropped: 0.75, 1.33 ) decreasing the image size seems to not perform. One thing to try here is the find the optimum parameters of random resize crop methodology that will allow smaller images which will further allow higher batch sizes.
\n
(1) What do you mean by 'Image size smaller than 512' ? The image size in this competition is all 800x600?
\n(2) Maybe the cropped lower limit 0.08 is too small, we need to set it to 0.6 or 0.8 ? Thanks!
Why didn't you try it on Google Colab, it would have given you better compute.
", "votes": null, "replies": [] }, { "id": 1226651, "author_name": "abhiagwl", "author_url": "", "post_date": "03/04/2021 18:03:17", "content": "what about
\n