I recently competed in Twitter sentiment extraction competition and fell 20 places in LB because we didn't use a proper CV strategy. As GMs have pointed out, the number one thing you should do when entering a competition is building a great CV strategy.
\n\nFor this one, I'm using a stratified 5-fold split. Now, I compute the standard deviation over CV scores and find it is still quite high: +/- 0.01.
\n\nWhen switching from StratifiedKFold to KFold and using Soft Margin Focal Loss, I managed to decrease the standard deviation from 0.0187 to 0.01.
\n\nI now that increasing the number of folds will likely lower the standard deviation but there is an obvious computation cost to that. One epoch takes around 4 or 5 minutes using PyTorch and EfficientNetB1, and I don't think I can reduce this time any lower.
\n\nI think it would benefit the Kaggle community to actually learn more on how to properly validate a model.
", "messages": [ { "id": "896989", "postDate": "06/22/2020 14:55:47", "content": "I recently competed in Twitter sentiment extraction competition and fell 20 places in LB because we didn't use a proper CV strategy. As GMs have pointed out, the number one thing you should do when entering a competition is building a great CV strategy.
\n\nFor this one, I'm using a stratified 5-fold split. Now, I compute the standard deviation over CV scores and find it is still quite high: +/- 0.01.
\n\nWhen switching from StratifiedKFold to KFold and using Soft Margin Focal Loss, I managed to decrease the standard deviation from 0.0187 to 0.01.
\n\nI now that increasing the number of folds will likely lower the standard deviation but there is an obvious computation cost to that. One epoch takes around 4 or 5 minutes using PyTorch and EfficientNetB1, and I don't think I can reduce this time any lower.
\n\nI think it would benefit the Kaggle community to actually learn more on how to properly validate a model.
", "rawMarkdown": "I recently competed in Twitter sentiment extraction competition and fell 20 places in LB because we didn't use a proper CV strategy. As GMs have pointed out, the number one thing you should do when entering a competition is building a great CV strategy. \n\nFor this one, I'm using a stratified 5-fold split. Now, I compute the standard deviation over CV scores and find it is still quite high: **+/- 0.01**.\n\nWhen switching from StratifiedKFold to KFold and using Soft Margin Focal Loss, I managed to decrease the standard deviation from 0.0187 to 0.01. \n\nI now that increasing the number of folds will likely lower the standard deviation but there is an obvious computation cost to that. One epoch takes around 4 or 5 minutes using PyTorch and EfficientNetB1, and I don't think I can reduce this time any lower.\n\nI think it would benefit the Kaggle community to actually learn more on how to properly validate a model.", "votes": null }, { "id": "899096", "postDate": "06/24/2020 01:35:16", "content": "I am using a similar setup as yours GroupKFold(n_splits=5) + EfficientNetB1 using 256X256. I am getting an oof score of 0.87 and LB of .907. When you said standard deviation over CV score is 0.01, are you taking the standard deviation for the best round on validation data error across the 5 fold?
Nice point @rftexas !
\n\nHave you tried StratifiedGroupKFold? You might find it more useful to reduce std futher.
\n\nAlso, increasing and using 8 fold might be a problem when training for bigger size images. When you say 4 to 5 mins, what image size are you referring to? I assume 512 x 512? 256x256 takes around 2 mins on my V100.
", "rawMarkdown": "Nice point @rftexas ! \n\nHave you tried [StratifiedGroupKFold](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation)? You might find it more useful to reduce std futher.\n\nAlso, increasing and using 8 fold might be a problem when training for bigger size images. When you say 4 to 5 mins, what image size are you referring to? I assume 512 x 512? 256x256 takes around 2 mins on my V100.", "votes": null }, { "id": "900790", "postDate": "06/25/2020 04:16:37", "content": "512 only taking 2 minutes in single GPU? I am using four and it takes about 2 minutes using all four.
", "rawMarkdown": "512 only taking 2 minutes in single GPU? I am using four and it takes about 2 minutes using all four.", "votes": null }, { "id": "902070", "postDate": "06/25/2020 22:53:06", "content": "I'm using 224x224 size. And I have a 4-minute epoch using TPU (although not optimized since MXU is not getting higher than 8%...). I am very curious on how you achieve a 2-minute epoch with a single GPU.
\n\nBy the way, I'll try StratifiedGroupKFold, very useful thanks!
", "rawMarkdown": "I'm using 224x224 size. And I have a 4-minute epoch using TPU (although not optimized since MXU is not getting higher than 8%...). I am very curious on how you achieve a 2-minute epoch with a single GPU. \n\nBy the way, I'll try StratifiedGroupKFold, very useful thanks!", "votes": null }, { "id": "902071", "postDate": "06/25/2020 22:53:49", "content": "Indeed. I am taking the standard deviation of my 5 best CV scores, one for each fold.
", "rawMarkdown": "Indeed. I am taking the standard deviation of my 5 best CV scores, one for each fold.", "votes": null }, { "id": "902352", "postDate": "06/26/2020 05:16:13", "content": "I preprocessed and saved all my images before hand including resize. :)
", "rawMarkdown": "I preprocessed and saved all my images before hand including resize. :)", "votes": null }, { "id": "902437", "postDate": "06/26/2020 06:34:25", "content": "@anwarsadique if I understand correctly @aroraaman is using a V100 GPU are your 4 GPUs also V100?
", "rawMarkdown": "anwarsadique if I understand correctly @aroraaman is using a V100 GPU are your 4 GPUs also V100?", "votes": null } ], "comments": [ { "id": 899096, "author_name": "anwarsadique", "author_url": "", "post_date": "06/24/2020 01:35:16", "content": "I am using a similar setup as yours GroupKFold(n_splits=5) + EfficientNetB1 using 256X256. I am getting an oof score of 0.87 and LB of .907. When you said standard deviation over CV score is 0.01, are you taking the standard deviation for the best round on validation data error across the 5 fold?
Indeed. I am taking the standard deviation of my 5 best CV scores, one for each fold.
", "votes": null, "replies": [] } ] }, { "id": 899138, "author_name": "aroraaman", "author_url": "", "post_date": "06/24/2020 02:45:39", "content": "Nice point @rftexas !
\n\nHave you tried StratifiedGroupKFold? You might find it more useful to reduce std futher.
\n\nAlso, increasing and using 8 fold might be a problem when training for bigger size images. When you say 4 to 5 mins, what image size are you referring to? I assume 512 x 512? 256x256 takes around 2 mins on my V100.
", "votes": null, "replies": [ { "id": 900790, "author_name": "anwarsadique", "author_url": "", "post_date": "06/25/2020 04:16:37", "content": "512 only taking 2 minutes in single GPU? I am using four and it takes about 2 minutes using all four.
", "votes": null, "replies": [] }, { "id": 902070, "author_name": "rftexas", "author_url": "", "post_date": "06/25/2020 22:53:06", "content": "I'm using 224x224 size. And I have a 4-minute epoch using TPU (although not optimized since MXU is not getting higher than 8%...). I am very curious on how you achieve a 2-minute epoch with a single GPU.
\n\nBy the way, I'll try StratifiedGroupKFold, very useful thanks!
", "votes": null, "replies": [] }, { "id": 902352, "author_name": "aroraaman", "author_url": "", "post_date": "06/26/2020 05:16:13", "content": "I preprocessed and saved all my images before hand including resize. :)
", "votes": null, "replies": [] }, { "id": 902437, "author_name": "yuval6967", "author_url": "", "post_date": "06/26/2020 06:34:25", "content": "@anwarsadique if I understand correctly @aroraaman is using a V100 GPU are your 4 GPUs also V100?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "896989": "I recently competed in Twitter sentiment extraction competition and fell 20 places in LB because we didn't use a proper CV strategy. As GMs have pointed out, the number one thing you should do when entering a competition is building a great CV strategy. \n\nFor this one, I'm using a stratified 5-fold split. Now, I compute the standard deviation over CV scores and find it is still quite high: **+/- 0.01**.\n\nWhen switching from StratifiedKFold to KFold and using Soft Margin Focal Loss, I managed to decrease the standard deviation from 0.0187 to 0.01. \n\nI now that increasing the number of folds will likely lower the standard deviation but there is an obvious computation cost to that. One epoch takes around 4 or 5 minutes using PyTorch and EfficientNetB1, and I don't think I can reduce this time any lower.\n\nI think it would benefit the Kaggle community to actually learn more on how to properly validate a model.", "899096": "I am using a similar setup as yours `GroupKFold(n_splits=5)` + EfficientNetB1 using 256X256. I am getting an oof score of 0.87 and LB of .907. When you said standard deviation over CV score is 0.01, are you taking the standard deviation for the best round on validation data error across the 5 fold?", "899138": "Nice point @rftexas ! \n\nHave you tried [StratifiedGroupKFold](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation)? You might find it more useful to reduce std futher.\n\nAlso, increasing and using 8 fold might be a problem when training for bigger size images. When you say 4 to 5 mins, what image size are you referring to? I assume 512 x 512? 256x256 takes around 2 mins on my V100.", "900790": "512 only taking 2 minutes in single GPU? I am using four and it takes about 2 minutes using all four.", "902070": "I'm using 224x224 size. And I have a 4-minute epoch using TPU (although not optimized since MXU is not getting higher than 8%...). I am very curious on how you achieve a 2-minute epoch with a single GPU. \n\nBy the way, I'll try StratifiedGroupKFold, very useful thanks!", "902071": "Indeed. I am taking the standard deviation of my 5 best CV scores, one for each fold.", "902352": "I preprocessed and saved all my images before hand including resize. :)", "902437": "anwarsadique if I understand correctly @aroraaman is using a V100 GPU are your 4 GPUs also V100?" }, "source": "meta" }