Dear Kagglers,
\nI've shared two kernels showing how to do training and inference in this competition. I'm going to also summarize some skills here to help new coming kagglers.
\nWith 1 and 2, the model would train faster and you could use a larger model with effectively larger batch size.
\nTrain:
\nValid:
\nPublic LB
\nTrain:
\nValid:
\nPublic LB
\nValid:
\nPublic LB
\nFeel free to advise & ask questions :)
", "messages": [ { "id": "1084762", "postDate": "11/20/2020 11:10:22", "content": "Dear Kagglers,
\nI've shared two kernels showing how to do training and inference in this competition. I'm going to also summarize some skills here to help new coming kagglers.
\nWith 1 and 2, the model would train faster and you could use a larger model with effectively larger batch size.
\nTrain:
\nValid:
\nPublic LB
\nTrain:
\nValid:
\nPublic LB
\nValid:
\nPublic LB
\nFeel free to advise & ask questions :)
", "rawMarkdown": "Dear Kagglers,\n\nI've shared two kernels showing how to do training and inference in this competition. I'm going to also summarize some skills here to help new coming kagglers.\n\n#### **Train**: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug \n1. Use Pytorch built-in Automatic Mixed Precision (AMP)\n2. Batch accumulation\n3. Transfer learning with efficientnet b0\n4. Heavy train-time augmentation: h\\vflip, coarsedropout, ... to increase generalization\n\nWith 1 and 2, the model would train faster and you could use a larger model with effectively larger batch size.\n\n#### **Inference**: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-inference-tta\n1. Checkpoint result averaging\n2. Test time augmentation\n\n#### **[11/21 Update]**\n**Train**: \n1. Use adam and learning rate scheduler setting from https://www.kaggle.com/yasufuminakama/cassava-resnext50-32x4d-starter-training\n2. Use noisy-student pretrained efficientnet b3\n3. More augmentations\n\n**Valid**: \n1. Use last 4 epochs' checkpoints + TTA 3 times\n2. More augmentation for TTA\n3. Logloss\\Accuracy Improvements: 0.44460/0.84650 to 0.26788/0.91192\n\n**Public LB**\n1. Improve from 0.853 to 0.895\n\n#### **[11/24 Update]**\n**Train**: \n1. Scale up to test: Use image size=512, noisy-student pretrained efficientnet b4\n\n**Valid**: \n1. Logloss\\Accuracy Improvements: 0.26788/0.91192 to 0.23218/0.92804\n\n**Public LB**\n1. Improve from 0.895 to 0.899\n\n#### **[11/26 Update]**\n**Valid**: \n1. Use the last 4 epochs with 3 TTA\n2. Logloss\\Accuracy Improvements: 0.23218/0.92804 to 0.21773/0.93224\n\n**Public LB**\n1. Improve from 0.899 to 0.900\n\nFeel free to advise & ask questions :)", "votes": null }, { "id": "1110763", "postDate": "12/13/2020 03:52:49", "content": "What does \"Use the last 4 epochs with 3 TTA\" mean ?
", "rawMarkdown": "What does \"Use the last 4 epochs with 3 TTA\" mean ?", "votes": null }, { "id": "1110867", "postDate": "12/13/2020 06:41:41", "content": "In his code, he trained 10 epochs.
\n\"Use the last 4 epochs\" means he chose last 4 models (i.e. 6, 7, 8, 9th models) for inference.
\nAnd he applied \"TTA (Test Time Augmentation)\"
Please share any notebook that use log loss with pytorch. Thanks in advanced.
", "rawMarkdown": "Please share any notebook that use log loss with pytorch. Thanks in advanced.", "votes": null }, { "id": "1146739", "postDate": "01/10/2021 02:16:29", "content": "Thank you for your share.I feel confused that i got val/87.5,but the LB got only 82.7,how could it be such a difference
", "rawMarkdown": "Thank you for your share.I feel confused that i got val/87.5,but the LB got only 82.7,how could it be such a difference", "votes": null }, { "id": "1156101", "postDate": "01/16/2021 23:33:34", "content": "Thanks for your methods and notebook sharing. Do you apply the model from any paper ? Do you have the paper reference? Thanks,
", "rawMarkdown": "Thanks for your methods and notebook sharing. Do you apply the model from any paper ? Do you have the paper reference? Thanks,", "votes": null } ], "comments": [ { "id": 1110763, "author_name": "nishikattt", "author_url": "", "post_date": "12/13/2020 03:52:49", "content": "What does \"Use the last 4 epochs with 3 TTA\" mean ?
", "votes": null, "replies": [ { "id": 1110867, "author_name": "jinkyh", "author_url": "", "post_date": "12/13/2020 06:41:41", "content": "In his code, he trained 10 epochs.
\n\"Use the last 4 epochs\" means he chose last 4 models (i.e. 6, 7, 8, 9th models) for inference.
\nAnd he applied \"TTA (Test Time Augmentation)\"
Please share any notebook that use log loss with pytorch. Thanks in advanced.
", "votes": null, "replies": [] }, { "id": 1146739, "author_name": "cnruiwang", "author_url": "", "post_date": "01/10/2021 02:16:29", "content": "Thank you for your share.I feel confused that i got val/87.5,but the LB got only 82.7,how could it be such a difference
", "votes": null, "replies": [] }, { "id": 1156101, "author_name": "denisechendd", "author_url": "", "post_date": "01/16/2021 23:33:34", "content": "Thanks for your methods and notebook sharing. Do you apply the model from any paper ? Do you have the paper reference? Thanks,
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1084762": "Dear Kagglers,\n\nI've shared two kernels showing how to do training and inference in this competition. I'm going to also summarize some skills here to help new coming kagglers.\n\n#### **Train**: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug \n1. Use Pytorch built-in Automatic Mixed Precision (AMP)\n2. Batch accumulation\n3. Transfer learning with efficientnet b0\n4. Heavy train-time augmentation: h\\vflip, coarsedropout, ... to increase generalization\n\nWith 1 and 2, the model would train faster and you could use a larger model with effectively larger batch size.\n\n#### **Inference**: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-inference-tta\n1. Checkpoint result averaging\n2. Test time augmentation\n\n#### **[11/21 Update]**\n**Train**: \n1. Use adam and learning rate scheduler setting from https://www.kaggle.com/yasufuminakama/cassava-resnext50-32x4d-starter-training\n2. Use noisy-student pretrained efficientnet b3\n3. More augmentations\n\n**Valid**: \n1. Use last 4 epochs' checkpoints + TTA 3 times\n2. More augmentation for TTA\n3. Logloss\\Accuracy Improvements: 0.44460/0.84650 to 0.26788/0.91192\n\n**Public LB**\n1. Improve from 0.853 to 0.895\n\n#### **[11/24 Update]**\n**Train**: \n1. Scale up to test: Use image size=512, noisy-student pretrained efficientnet b4\n\n**Valid**: \n1. Logloss\\Accuracy Improvements: 0.26788/0.91192 to 0.23218/0.92804\n\n**Public LB**\n1. Improve from 0.895 to 0.899\n\n#### **[11/26 Update]**\n**Valid**: \n1. Use the last 4 epochs with 3 TTA\n2. Logloss\\Accuracy Improvements: 0.23218/0.92804 to 0.21773/0.93224\n\n**Public LB**\n1. Improve from 0.899 to 0.900\n\nFeel free to advise & ask questions :)", "1110763": "What does \"Use the last 4 epochs with 3 TTA\" mean ?", "1110867": "In his code, he trained 10 epochs. \n\"Use the last 4 epochs\" means he chose last 4 models (i.e. 6, 7, 8, 9th models) for inference.\nAnd he applied \"TTA (Test Time Augmentation)\"", "1112306": "Please share any notebook that use log loss with pytorch. Thanks in advanced.", "1146739": "Thank you for your share.I feel confused that i got val/87.5,but the LB got only 82.7,how could it be such a difference", "1156101": "Thanks for your methods and notebook sharing. Do you apply the model from any paper ? Do you have the paper reference? Thanks," }, "source": "meta" }