Congrats to the top solutions and a great competition.
\nOne learns new thing in every competition, a perfect way to follow the journey of AI world and also to contribute to important problems, thanks Kaggle, competition host and fellow kagglers.
The solution was a global ensemble of 29 different models including 4 local ensembles(2 own, 2 public), trained in different ways.
\nAll trained with dim 512 but the rest I tried using different techniques and parameters.
Own local trainings and ensembles
\n1 - FastAIv2 - 6 models
\nCommon for all training
\nUsed 1-fold of each model from random 10-fold training.
\nRanger or AdamW optimizer
\nLabelSmoothingCrossEntropy loss
\n1CycleScheduler
\nStarted all training with 3 epochs with only training last layers and then trained all layers 20 epochs.
\nIn between the trainings I used my own custom LR finder, which I posted in another competition.
\n“LR Finder alternative use.”
\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/218711
Models
\n2 tf_efficientnet_b5_ns – mixup training
\n2 seresnext50_32x4d – cutmix training
\n1 xception – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.
\n1 densenet121 – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.
I equal ensembled the models with 5 TTA each with changing the aug. to just resizing image not cropping do have a different image view contrary the other 3 ensembles and skipped all other augs. just random brightness, contrast etc.
\n2 - Pytorch - 13 models
\n8 tf_efficientnet_b1_ns
\n5 resnext50-32x4d
Common for all trainings
\nExtra training data
\nBiTemperedLoss with Random Noise settings described in the https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html but I used more narrow approach with 0.5/1.5.
\nOneCycleLRWithWarmup from Catalyst which also takes in count momentum and weight_decay in the cycles.
\nUsed a random mix with almost equal split of all different aug. techniques SnapMix, Fmix, CutMix,CutOut, Mixup and MWH(Mixup-Without-Hesitation).
\nAdamP optimizer https://github.com/clovaai/AdamP
Here I instead used stacking with 5 TTA instead of ensemble to create a different approach to the global ensemble.
\nFor the last 2 local ensembles I used 2 public trained models 5 B4ns and 5 resnext50-32x4d with 4 TTA for each model to have bigger ensemble and trainings to not get stuck in one’s own strategy.
\nhttps://www.kaggle.com/piantic/cassava-resnext50-32x4d-weights
\nhttps://www.kaggle.com/underwearfitting/moa-b4-baseline
\nThanks, and credit to your training and models.
All 4 local ensemble where then ensemble with equal weights.
\nThat’s it, thanks for reading the write-up.
\nThis gave my 3rd silver medal, hard work, lots of reading and testing pay off.
\nI have also made a personal competition and challenge this time, to contribute in 5 different Kaggle competition in parallel, trying to get good results in every one of them, equal distribution of time. Just for fun and the learning. 1st out finished yesterday and got a bronze medal and now silver in this one, three to go, and hopes for the best, the learning is a big part of the value.
\nHappy Kaggleing!
", "messages": [ { "id": "1210354", "postDate": "02/19/2021 11:21:15", "content": "Congrats to the top solutions and a great competition.
\nOne learns new thing in every competition, a perfect way to follow the journey of AI world and also to contribute to important problems, thanks Kaggle, competition host and fellow kagglers.
The solution was a global ensemble of 29 different models including 4 local ensembles(2 own, 2 public), trained in different ways.
\nAll trained with dim 512 but the rest I tried using different techniques and parameters.
Own local trainings and ensembles
\n1 - FastAIv2 - 6 models
\nCommon for all training
\nUsed 1-fold of each model from random 10-fold training.
\nRanger or AdamW optimizer
\nLabelSmoothingCrossEntropy loss
\n1CycleScheduler
\nStarted all training with 3 epochs with only training last layers and then trained all layers 20 epochs.
\nIn between the trainings I used my own custom LR finder, which I posted in another competition.
\n“LR Finder alternative use.”
\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/218711
Models
\n2 tf_efficientnet_b5_ns – mixup training
\n2 seresnext50_32x4d – cutmix training
\n1 xception – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.
\n1 densenet121 – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.
I equal ensembled the models with 5 TTA each with changing the aug. to just resizing image not cropping do have a different image view contrary the other 3 ensembles and skipped all other augs. just random brightness, contrast etc.
\n2 - Pytorch - 13 models
\n8 tf_efficientnet_b1_ns
\n5 resnext50-32x4d
Common for all trainings
\nExtra training data
\nBiTemperedLoss with Random Noise settings described in the https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html but I used more narrow approach with 0.5/1.5.
\nOneCycleLRWithWarmup from Catalyst which also takes in count momentum and weight_decay in the cycles.
\nUsed a random mix with almost equal split of all different aug. techniques SnapMix, Fmix, CutMix,CutOut, Mixup and MWH(Mixup-Without-Hesitation).
\nAdamP optimizer https://github.com/clovaai/AdamP
Here I instead used stacking with 5 TTA instead of ensemble to create a different approach to the global ensemble.
\nFor the last 2 local ensembles I used 2 public trained models 5 B4ns and 5 resnext50-32x4d with 4 TTA for each model to have bigger ensemble and trainings to not get stuck in one’s own strategy.
\nhttps://www.kaggle.com/piantic/cassava-resnext50-32x4d-weights
\nhttps://www.kaggle.com/underwearfitting/moa-b4-baseline
\nThanks, and credit to your training and models.
All 4 local ensemble where then ensemble with equal weights.
\nThat’s it, thanks for reading the write-up.
\nThis gave my 3rd silver medal, hard work, lots of reading and testing pay off.
\nI have also made a personal competition and challenge this time, to contribute in 5 different Kaggle competition in parallel, trying to get good results in every one of them, equal distribution of time. Just for fun and the learning. 1st out finished yesterday and got a bronze medal and now silver in this one, three to go, and hopes for the best, the learning is a big part of the value.
\nHappy Kaggleing!
", "rawMarkdown": "Congrats to the top solutions and a great competition. \nOne learns new thing in every competition, a perfect way to follow the journey of AI world and also to contribute to important problems, thanks Kaggle, competition host and fellow kagglers.\n\nThe solution was a global ensemble of 29 different models including 4 local ensembles(2 own, 2 public), trained in different ways.\nAll trained with dim 512 but the rest I tried using different techniques and parameters.\n\nOwn local trainings and ensembles\n\n 1 - FastAIv2 - 6 models\n\nCommon for all training \n\nUsed 1-fold of each model from random 10-fold training.\nRanger or AdamW optimizer\nLabelSmoothingCrossEntropy loss\n1CycleScheduler\nStarted all training with 3 epochs with only training last layers and then trained all layers 20 epochs.\nIn between the trainings I used my own custom LR finder, which I posted in another competition.\n“LR Finder alternative use.”\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/218711\n\nModels\n\n2 tf_efficientnet_b5_ns – mixup training\n2 seresnext50_32x4d – cutmix training\n1 xception – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.\n1 densenet121 – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.\n\nI equal ensembled the models with 5 TTA each with changing the aug. to just resizing image not cropping do have a different image view contrary the other 3 ensembles and skipped all other augs. just random brightness, contrast etc.\n\n2 - Pytorch - 13 models\n\n8 tf_efficientnet_b1_ns\n5 resnext50-32x4d\n\nCommon for all trainings \n\nExtra training data\nBiTemperedLoss with Random Noise settings described in the https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html but I used more narrow approach with 0.5/1.5.\nOneCycleLRWithWarmup from Catalyst which also takes in count momentum and weight_decay in the cycles.\nUsed a random mix with almost equal split of all different aug. techniques SnapMix, Fmix, CutMix,CutOut, Mixup and MWH(Mixup-Without-Hesitation).\nAdamP optimizer https://github.com/clovaai/AdamP\n\nHere I instead used stacking with 5 TTA instead of ensemble to create a different approach to the global ensemble.\n\nFor the last 2 local ensembles I used 2 public trained models 5 B4ns and 5 resnext50-32x4d with 4 TTA for each model to have bigger ensemble and trainings to not get stuck in one’s own strategy.\nhttps://www.kaggle.com/piantic/cassava-resnext50-32x4d-weights\nhttps://www.kaggle.com/underwearfitting/moa-b4-baseline\nThanks, and credit to your training and models.\n\nAll 4 local ensemble where then ensemble with equal weights.\n\nThat’s it, thanks for reading the write-up.\n\nThis gave my 3rd silver medal, hard work, lots of reading and testing pay off.\n\nI have also made a personal competition and challenge this time, to contribute in 5 different Kaggle competition in parallel, trying to get good results in every one of them, equal distribution of time. Just for fun and the learning. 1st out finished yesterday and got a bronze medal and now silver in this one, three to go, and hopes for the best, the learning is a big part of the value.\n\nHappy Kaggleing!", "votes": null }, { "id": "1210367", "postDate": "02/19/2021 11:33:41", "content": "Good job! And congrats on your silver medal. :)
\nwell done! @kirderf
Thanks! :)
", "rawMarkdown": "Thanks! :)", "votes": null } ], "comments": [ { "id": 1210367, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 11:33:41", "content": "Good job! And congrats on your silver medal. :)
\nwell done! @kirderf
Thanks! :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1210354": "Congrats to the top solutions and a great competition. \nOne learns new thing in every competition, a perfect way to follow the journey of AI world and also to contribute to important problems, thanks Kaggle, competition host and fellow kagglers.\n\nThe solution was a global ensemble of 29 different models including 4 local ensembles(2 own, 2 public), trained in different ways.\nAll trained with dim 512 but the rest I tried using different techniques and parameters.\n\nOwn local trainings and ensembles\n\n 1 - FastAIv2 - 6 models\n\nCommon for all training \n\nUsed 1-fold of each model from random 10-fold training.\nRanger or AdamW optimizer\nLabelSmoothingCrossEntropy loss\n1CycleScheduler\nStarted all training with 3 epochs with only training last layers and then trained all layers 20 epochs.\nIn between the trainings I used my own custom LR finder, which I posted in another competition.\n“LR Finder alternative use.”\nhttps://www.kaggle.com/c/jane-street-market-prediction/discussion/218711\n\nModels\n\n2 tf_efficientnet_b5_ns – mixup training\n2 seresnext50_32x4d – cutmix training\n1 xception – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.\n1 densenet121 – cutmix training – here I also used upsample with more upsample to class 4 which I visually noticed had more false classifications.\n\nI equal ensembled the models with 5 TTA each with changing the aug. to just resizing image not cropping do have a different image view contrary the other 3 ensembles and skipped all other augs. just random brightness, contrast etc.\n\n2 - Pytorch - 13 models\n\n8 tf_efficientnet_b1_ns\n5 resnext50-32x4d\n\nCommon for all trainings \n\nExtra training data\nBiTemperedLoss with Random Noise settings described in the https://ai.googleblog.com/2019/08/bi-tempered-logistic-loss-for-training.html but I used more narrow approach with 0.5/1.5.\nOneCycleLRWithWarmup from Catalyst which also takes in count momentum and weight_decay in the cycles.\nUsed a random mix with almost equal split of all different aug. techniques SnapMix, Fmix, CutMix,CutOut, Mixup and MWH(Mixup-Without-Hesitation).\nAdamP optimizer https://github.com/clovaai/AdamP\n\nHere I instead used stacking with 5 TTA instead of ensemble to create a different approach to the global ensemble.\n\nFor the last 2 local ensembles I used 2 public trained models 5 B4ns and 5 resnext50-32x4d with 4 TTA for each model to have bigger ensemble and trainings to not get stuck in one’s own strategy.\nhttps://www.kaggle.com/piantic/cassava-resnext50-32x4d-weights\nhttps://www.kaggle.com/underwearfitting/moa-b4-baseline\nThanks, and credit to your training and models.\n\nAll 4 local ensemble where then ensemble with equal weights.\n\nThat’s it, thanks for reading the write-up.\n\nThis gave my 3rd silver medal, hard work, lots of reading and testing pay off.\n\nI have also made a personal competition and challenge this time, to contribute in 5 different Kaggle competition in parallel, trying to get good results in every one of them, equal distribution of time. Just for fun and the learning. 1st out finished yesterday and got a bronze medal and now silver in this one, three to go, and hopes for the best, the learning is a big part of the value.\n\nHappy Kaggleing!", "1210367": "Good job! And congrats on your silver medal. :)\nwell done! @kirderf", "1210435": "Thanks! :)" }, "source": "meta" }