First of all, I would like to thank Kaggle and the organizers for hosting this great competitions. Also, I would like to thank my teammates - @ludovick and @woshifym . Due to the time difference, we can work 24 hours a day.
\nOur solution can be divided into 2 parts, pytorch part and tensorflow part.
\nFor Pytorch part I would like to thank @ammarali32 for the 3-step method first. All our models based on 3-step method. We trained models with different backbones and different image size to get better results.
\nAfter that we found Soft label method performs great in this competition. So we trained all the models above using this method to form our 4-step method.
\nMy teammates @ludovick did a great job using tensorflow. I would like to invite him to introduce his method in details later.
\nFor brief summary:
For ensemble, we use the OOF files to train a linear regression model. Then we use this linear model to get the weight for each model. We select our best CV and best public score in our final submissions. Unfortunately, we just missed out best ensemble which is the second best in public score.
\nFinal submission one
\nCV: 0.9692 Public LB: 0.971 Private LB: 0.973
Final submission two
\nCV: 0.9689 Public LB: 0.971 Private LB: 0.974
Thanks to TFRC team for providing us free TPU.
", "messages": [ { "id": 1244443, "postDate": "2021-03-19T02:04:22.113Z", "content": "First of all, I would like to thank Kaggle and the organizers for hosting this great competitions. Also, I would like to thank my teammates - @ludovick and @woshifym . Due to the time difference, we can work 24 hours a day.
\nOur solution can be divided into 2 parts, pytorch part and tensorflow part.
\nFor Pytorch part I would like to thank @ammarali32 for the 3-step method first. All our models based on 3-step method. We trained models with different backbones and different image size to get better results.
\nAfter that we found Soft label method performs great in this competition. So we trained all the models above using this method to form our 4-step method.
\nMy teammates @ludovick did a great job using tensorflow. I would like to invite him to introduce his method in details later.
\nFor brief summary:
For ensemble, we use the OOF files to train a linear regression model. Then we use this linear model to get the weight for each model. We select our best CV and best public score in our final submissions. Unfortunately, we just missed out best ensemble which is the second best in public score.
\nFinal submission one
\nCV: 0.9692 Public LB: 0.971 Private LB: 0.973
Final submission two
\nCV: 0.9689 Public LB: 0.971 Private LB: 0.974
Thanks to TFRC team for providing us free TPU.
", "rawMarkdown": "First of all, I would like to thank Kaggle and the organizers for hosting this great competitions. Also, I would like to thank my teammates - @ludovick and @woshifym . Due to the time difference, we can work 24 hours a day.\n\n# Solutions\nOur solution can be divided into 2 parts, pytorch part and tensorflow part.\n\n## Pytorch part\nFor Pytorch part I would like to thank @ammarali32 for the 3-step method first. All our models based on 3-step method. We trained models with different backbones and different image size to get better results.\n\n* resnet200d + image size: 600 CV: 0.9578\n* ecaresnet269d + image size: 600 CV: 0.9584\n* resnest200e + image size: 640 CV:0.9625\n\nAfter that we found [Soft label method](https://github.com/alipay/cvpr2020-plant-pathology) performs great in this competition. So we trained all the models above using this method to form our 4-step method.\n\n## Tensorflow part\nMy teammates @ludovick did a great job using tensorflow. I would like to invite him to introduce his method in details later.\nFor brief summary:\n\n* Efficient B7 + 1024 image size\n* Efficient L2 + 768 image size\n* Soft label method\n* Pesudo Label\n\n## Ensemble\nFor ensemble, we use the OOF files to train a linear regression model. Then we use this linear model to get the weight for each model. We select our best CV and best public score in our final submissions. Unfortunately, we just missed out best ensemble which is the second best in public score.\n\nFinal submission one \nCV: 0.9692 Public LB: 0.971 Private LB: 0.973\n* resnet200d + 600\n* ecaresnet269d + 600\n* resnest200e + 640\n* Efficient B7 + 1024 + soft label + PL\n* Efficient L2 + 768 + soft label\n\nFinal submission two\nCV: 0.9689 Public LB: 0.971 Private LB: 0.974\n* resnet200d + 600\n* resnest200e + 640\n* Efficient B7 + 1024 + soft label + PL\n* Efficient B7 + 1024 + PL\n* Efficient L2 + 768 + soft label\n\n## Tricks\n* Larger model\n* Higher resolution\n* Soft Label\n* Pesudo Label\n\n## Acknowledge\nThanks to TFRC team for providing us free TPU.", "votes": 30 }, { "id": 1245610, "postDate": "2021-03-20T00:15:24.033Z", "content": "Thank you to Kaggle and the organizers for hosting this competition.
\nregarding the Tensorflow part :
\nWe use for our solution only EfficientNet architectures for the TF solution. In fact, we are quite limited regarding the different strong models in TF, only inceptionresnetV2 was giving interesting results, but it was still lower than EfficientNet B6/B7/L2.
\nWe use 3 resolutions : 768, 1024, 1280 and lights data aug : rotation, translation, shear, zoom, color (contrast etc)
1st step : Train EfficientNetB6/B7 on image size of 1024 gives respectively a LB of 0.966 and 0.965. EfficientNetB7 on 1280 gives LB of 0.966.
\nEfficientNet L2 with an image size of 768 gives 0.967. It would have been interesting to increase the image size for EffL2 unfortunately it seems impossible with TPUv3.
\nHowever, EfficientNetB6 was not adding enought diversity in our ensembling (no big increase for CV/LB) so we discard it for the next steps.
2nd step : self distillation. We include the softlabel of the models trained in the 1st step, such as the softlabel represent 30% of the \"final label\".
\nFor example, the label of the image is 1.0 and the model predict a probability of 0.8. The final label will be equal to : 0.7*hard label + 0.3 * softlabel = 0.7 + 0.24 = 0.94
\nThat gives a boost of 0.001 and 0.002 depending of the model. The ensembling of these 3 models gives an LB of 0.970.
3rd step: This final step use the pseudo label from the NIH dataset. We average the prediction of our 3 models from steps 2 to build the pseudo label. We train only EfficientNetB7 on image size of 1024 as we were out of time. (We also trained an Xception model but the submission randomly failed the last day :/ )
\nThe results of the EffB7 was 0.969 on LB.
Our final submission was ensembling the EffB7 of Step3 and effL2 of step2 and the pytorch model described by @tonyxu
", "rawMarkdown": "Thank you to Kaggle and the organizers for hosting this competition. \n\nregarding the Tensorflow part :\n\nWe use for our solution only EfficientNet architectures for the TF solution. In fact, we are quite limited regarding the different strong models in TF, only inceptionresnetV2 was giving interesting results, but it was still lower than EfficientNet B6/B7/L2.\nWe use 3 resolutions : 768, 1024, 1280 and lights data aug : rotation, translation, shear, zoom, color (contrast etc)\n\n- 1st step : Train EfficientNetB6/B7 on image size of 1024 gives respectively a LB of 0.966 and 0.965. EfficientNetB7 on 1280 gives LB of 0.966. \nEfficientNet L2 with an image size of 768 gives 0.967. It would have been interesting to increase the image size for EffL2 unfortunately it seems impossible with TPUv3.\nHowever, EfficientNetB6 was not adding enought diversity in our ensembling (no big increase for CV/LB) so we discard it for the next steps.\n\n- 2nd step : self distillation. We include the softlabel of the models trained in the 1st step, such as the softlabel represent 30% of the \"final label\".\nFor example, the label of the image is 1.0 and the model predict a probability of 0.8. The final label will be equal to : 0.7*hard label + 0.3 * softlabel = 0.7 + 0.24 = 0.94\nThat gives a boost of 0.001 and 0.002 depending of the model. The ensembling of these 3 models gives an LB of 0.970.\n\n- 3rd step: This final step use the pseudo label from the NIH dataset. We average the prediction of our 3 models from steps 2 to build the pseudo label. We train only EfficientNetB7 on image size of 1024 as we were out of time. (We also trained an Xception model but the submission randomly failed the last day :/ )\nThe results of the EffB7 was 0.969 on LB.\n\nOur final submission was ensembling the EffB7 of Step3 and effL2 of step2 and the pytorch model described by @tonyxu ", "votes": 6 }, { "id": 1244491, "postDate": "2021-03-19T02:54:13.527Z", "content": "Congrats on our new Grandmaster!!! Really happy to work with you. Thanks so much for teaching me how to use TPU in GCP.
", "rawMarkdown": "Congrats on our new Grandmaster!!! Really happy to work with you. Thanks so much for teaching me how to use TPU in GCP.", "votes": 3 }, { "id": 1248639, "postDate": "2021-03-22T18:13:46.827Z", "content": "Congrats on 10th place @ludovick @woshifym.
\nAnd competitions GM! @tonyxu
It is amazing to achieve that 10th place with only the classification model.
\nIt seems 3-stage trainings.
\n1) Pytorch Part - It also has sub 3 or 4-stage training.
\n2) TF Part - Using 1) models and generating soft labels. And train student models (self-distillation)
\n3) Ensemble All
Could you explain more about 2)?
\n2-1) Generating soft labels using 1) models and retraining all of 1) models?
\n2-2) When is the right time to use the pseudo label? (already used in sub 3 or 4-stage training?)
In other competitions, the ensemble didn't work out well with distillation models together.
\nI am curious if the ensemble worked well in self-distillation using soft labels.
\n(Any tips when ensemble in this cases?)
p.s. cvpr2020-plant-pathology was my first competition in kaggle. It's nice to see great solution again in this competition.
\np.s2. My questions may be lacking because I am not participating deeply in the competition.
Thanks for sharing! :)
", "rawMarkdown": "Congrats on 10th place @ludovick @woshifym.\nAnd competitions GM! @tonyxu\n\nIt is amazing to achieve that 10th place with only the classification model.\n\nIt seems 3-stage trainings.\n**1) Pytorch Part** - It also has sub 3 or 4-stage training.\n**2) TF Part** - Using 1) models and generating soft labels. And train student models (self-distillation)\n**3) Ensemble All**\n\nCould you explain more about **2)**?\n**2-1)** Generating soft labels using 1) models and retraining all of 1) models?\n**2-2)** When is the right time to use the pseudo label? (already used in sub 3 or 4-stage training?)\n\nIn other competitions, the ensemble didn't work out well with distillation models together.\nI am curious if the ensemble worked well in self-distillation using soft labels.\n(Any tips when ensemble in this cases?)\n\np.s. cvpr2020-plant-pathology was my first competition in kaggle. It's nice to see great solution again in this competition.\np.s2. My questions may be lacking because I am not participating deeply in the competition. \n\nThanks for sharing! :)", "votes": 1, "replies": [ { "id": 1250060, "postDate": "2021-03-23T18:38:16.040Z", "content": "hello,
\nthe pytorch and TF part are completely independant, we merge the different models in the part 3) in order to boost our final submission.
\nFor the TF part it can be decomposed into 3 steps:
\nthe first step is a regular training using only light data augmentation such as rotation/translation/shear/color etc. We use EfficientNetB6/B7/L2 on different image size such as 768
the second step will use the softlabel from the first step to train new EfficientNet model (same architecture and same split of the data (5 folds) [ I give an exemple in an other reply in this discussion]
the third step will use Pseudo Label. PL is obtained from the NIH dataset and the soft prediction (probabilities) of all the models trained in step 2. We trained only one model for PL as we were out of time. (EfficientNet B7 with an image size of 1024)
Hope it answers your question :)
", "rawMarkdown": "hello,\n\nthe pytorch and TF part are completely independant, we merge the different models in the part 3) in order to boost our final submission.\n\nFor the TF part it can be decomposed into 3 steps:\n- the first step is a regular training using only light data augmentation such as rotation/translation/shear/color etc. We use EfficientNetB6/B7/L2 on different image size such as 768\n\n- the second step will use the softlabel from the first step to train new EfficientNet model (same architecture and same split of the data (5 folds) [ I give an exemple in an other reply in this discussion]\n\n- the third step will use Pseudo Label. PL is obtained from the NIH dataset and the soft prediction (probabilities) of all the models trained in step 2. We trained only one model for PL as we were out of time. (EfficientNet B7 with an image size of 1024)\n\nHope it answers your question :) ", "votes": 1 }, { "id": 1250094, "postDate": "2021-03-23T19:20:57.080Z", "content": "Thanks for your answer. :) @ludovick
", "rawMarkdown": "Thanks for your answer. :) @ludovick ", "votes": 1 } ] }, { "id": 1244920, "postDate": "2021-03-19T10:13:50.990Z", "content": "Congrats on becoming GM @tonyxu
", "rawMarkdown": "Congrats on becoming GM @tonyxu ", "votes": 1, "replies": [ { "id": 1245821, "postDate": "2021-03-20T07:42:57.493Z", "content": "Thank you. You can also be a GM sooner or later😃
", "rawMarkdown": "Thank you. You can also be a GM sooner or later😃" } ] }, { "id": 1244579, "postDate": "2021-03-19T04:51:26.097Z", "content": "Congratulation in Gold Win and to the new Grandmaster Candidate
\nI think PseudoLabelling were the main breakthrough for this competition.
Thanks. We didn't try segmentation method in our solution. Soft label and Pesudo Label is the most important tricks in out solution.
", "rawMarkdown": "Thanks. We didn't try segmentation method in our solution. Soft label and Pesudo Label is the most important tricks in out solution.", "votes": 1 } ] }, { "id": 1246811, "postDate": "2021-03-21T06:13:32.630Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1245610, "author_name": "Shiro", "author_url": "", "post_date": "2021-03-20T00:15:24.033000", "content": "Thank you to Kaggle and the organizers for hosting this competition.
\nregarding the Tensorflow part :
\nWe use for our solution only EfficientNet architectures for the TF solution. In fact, we are quite limited regarding the different strong models in TF, only inceptionresnetV2 was giving interesting results, but it was still lower than EfficientNet B6/B7/L2.
\nWe use 3 resolutions : 768, 1024, 1280 and lights data aug : rotation, translation, shear, zoom, color (contrast etc)
1st step : Train EfficientNetB6/B7 on image size of 1024 gives respectively a LB of 0.966 and 0.965. EfficientNetB7 on 1280 gives LB of 0.966.
\nEfficientNet L2 with an image size of 768 gives 0.967. It would have been interesting to increase the image size for EffL2 unfortunately it seems impossible with TPUv3.
\nHowever, EfficientNetB6 was not adding enought diversity in our ensembling (no big increase for CV/LB) so we discard it for the next steps.
2nd step : self distillation. We include the softlabel of the models trained in the 1st step, such as the softlabel represent 30% of the \"final label\".
\nFor example, the label of the image is 1.0 and the model predict a probability of 0.8. The final label will be equal to : 0.7*hard label + 0.3 * softlabel = 0.7 + 0.24 = 0.94
\nThat gives a boost of 0.001 and 0.002 depending of the model. The ensembling of these 3 models gives an LB of 0.970.
3rd step: This final step use the pseudo label from the NIH dataset. We average the prediction of our 3 models from steps 2 to build the pseudo label. We train only EfficientNetB7 on image size of 1024 as we were out of time. (We also trained an Xception model but the submission randomly failed the last day :/ )
\nThe results of the EffB7 was 0.969 on LB.
Our final submission was ensembling the EffB7 of Step3 and effL2 of step2 and the pytorch model described by @tonyxu
", "votes": 6, "replies": [] }, { "id": 1244491, "author_name": "LeoF", "author_url": "", "post_date": "2021-03-19T02:54:13.527000", "content": "Congrats on our new Grandmaster!!! Really happy to work with you. Thanks so much for teaching me how to use TPU in GCP.
", "votes": 3, "replies": [] }, { "id": 1248639, "author_name": "Heroseo", "author_url": "", "post_date": "2021-03-22T18:13:46.827000", "content": "Congrats on 10th place @ludovick @woshifym.
\nAnd competitions GM! @tonyxu
It is amazing to achieve that 10th place with only the classification model.
\nIt seems 3-stage trainings.
\n1) Pytorch Part - It also has sub 3 or 4-stage training.
\n2) TF Part - Using 1) models and generating soft labels. And train student models (self-distillation)
\n3) Ensemble All
Could you explain more about 2)?
\n2-1) Generating soft labels using 1) models and retraining all of 1) models?
\n2-2) When is the right time to use the pseudo label? (already used in sub 3 or 4-stage training?)
In other competitions, the ensemble didn't work out well with distillation models together.
\nI am curious if the ensemble worked well in self-distillation using soft labels.
\n(Any tips when ensemble in this cases?)
p.s. cvpr2020-plant-pathology was my first competition in kaggle. It's nice to see great solution again in this competition.
\np.s2. My questions may be lacking because I am not participating deeply in the competition.
Thanks for sharing! :)
", "votes": 1, "replies": [ { "id": 1250060, "author_name": "Shiro", "author_url": "", "post_date": "2021-03-23T18:38:16.040000", "content": "hello,
\nthe pytorch and TF part are completely independant, we merge the different models in the part 3) in order to boost our final submission.
\nFor the TF part it can be decomposed into 3 steps:
\nthe first step is a regular training using only light data augmentation such as rotation/translation/shear/color etc. We use EfficientNetB6/B7/L2 on different image size such as 768
the second step will use the softlabel from the first step to train new EfficientNet model (same architecture and same split of the data (5 folds) [ I give an exemple in an other reply in this discussion]
the third step will use Pseudo Label. PL is obtained from the NIH dataset and the soft prediction (probabilities) of all the models trained in step 2. We trained only one model for PL as we were out of time. (EfficientNet B7 with an image size of 1024)
Hope it answers your question :)
", "votes": 1, "replies": [] }, { "id": 1250094, "author_name": "Heroseo", "author_url": "", "post_date": "2021-03-23T19:20:57.080000", "content": "Thanks for your answer. :) @ludovick
", "votes": 1, "replies": [] } ] }, { "id": 1244920, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-03-19T10:13:50.990000", "content": "Congrats on becoming GM @tonyxu
", "votes": 1, "replies": [ { "id": 1245821, "author_name": "toxu", "author_url": "", "post_date": "2021-03-20T07:42:57.493000", "content": "Thank you. You can also be a GM sooner or later😃
", "votes": 0, "replies": [] } ] }, { "id": 1244579, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-19T04:51:26.097000", "content": "Congratulation in Gold Win and to the new Grandmaster Candidate
\nI think PseudoLabelling were the main breakthrough for this competition.
Thanks. We didn't try segmentation method in our solution. Soft label and Pesudo Label is the most important tricks in out solution.
", "votes": 1, "replies": [] } ] }, { "id": 1246811, "author_name": "", "author_url": "", "post_date": "2021-03-21T06:13:32.630000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1244443": "First of all, I would like to thank Kaggle and the organizers for hosting this great competitions. Also, I would like to thank my teammates - @ludovick and @woshifym . Due to the time difference, we can work 24 hours a day.\n\n# Solutions\nOur solution can be divided into 2 parts, pytorch part and tensorflow part.\n\n## Pytorch part\nFor Pytorch part I would like to thank @ammarali32 for the 3-step method first. All our models based on 3-step method. We trained models with different backbones and different image size to get better results.\n\n* resnet200d + image size: 600 CV: 0.9578\n* ecaresnet269d + image size: 600 CV: 0.9584\n* resnest200e + image size: 640 CV:0.9625\n\nAfter that we found [Soft label method](https://github.com/alipay/cvpr2020-plant-pathology) performs great in this competition. So we trained all the models above using this method to form our 4-step method.\n\n## Tensorflow part\nMy teammates @ludovick did a great job using tensorflow. I would like to invite him to introduce his method in details later.\nFor brief summary:\n\n* Efficient B7 + 1024 image size\n* Efficient L2 + 768 image size\n* Soft label method\n* Pesudo Label\n\n## Ensemble\nFor ensemble, we use the OOF files to train a linear regression model. Then we use this linear model to get the weight for each model. We select our best CV and best public score in our final submissions. Unfortunately, we just missed out best ensemble which is the second best in public score.\n\nFinal submission one \nCV: 0.9692 Public LB: 0.971 Private LB: 0.973\n* resnet200d + 600\n* ecaresnet269d + 600\n* resnest200e + 640\n* Efficient B7 + 1024 + soft label + PL\n* Efficient L2 + 768 + soft label\n\nFinal submission two\nCV: 0.9689 Public LB: 0.971 Private LB: 0.974\n* resnet200d + 600\n* resnest200e + 640\n* Efficient B7 + 1024 + soft label + PL\n* Efficient B7 + 1024 + PL\n* Efficient L2 + 768 + soft label\n\n## Tricks\n* Larger model\n* Higher resolution\n* Soft Label\n* Pesudo Label\n\n## Acknowledge\nThanks to TFRC team for providing us free TPU.", "1245610": "Thank you to Kaggle and the organizers for hosting this competition. \n\nregarding the Tensorflow part :\n\nWe use for our solution only EfficientNet architectures for the TF solution. In fact, we are quite limited regarding the different strong models in TF, only inceptionresnetV2 was giving interesting results, but it was still lower than EfficientNet B6/B7/L2.\nWe use 3 resolutions : 768, 1024, 1280 and lights data aug : rotation, translation, shear, zoom, color (contrast etc)\n\n- 1st step : Train EfficientNetB6/B7 on image size of 1024 gives respectively a LB of 0.966 and 0.965. EfficientNetB7 on 1280 gives LB of 0.966. \nEfficientNet L2 with an image size of 768 gives 0.967. It would have been interesting to increase the image size for EffL2 unfortunately it seems impossible with TPUv3.\nHowever, EfficientNetB6 was not adding enought diversity in our ensembling (no big increase for CV/LB) so we discard it for the next steps.\n\n- 2nd step : self distillation. We include the softlabel of the models trained in the 1st step, such as the softlabel represent 30% of the \"final label\".\nFor example, the label of the image is 1.0 and the model predict a probability of 0.8. The final label will be equal to : 0.7*hard label + 0.3 * softlabel = 0.7 + 0.24 = 0.94\nThat gives a boost of 0.001 and 0.002 depending of the model. The ensembling of these 3 models gives an LB of 0.970.\n\n- 3rd step: This final step use the pseudo label from the NIH dataset. We average the prediction of our 3 models from steps 2 to build the pseudo label. We train only EfficientNetB7 on image size of 1024 as we were out of time. (We also trained an Xception model but the submission randomly failed the last day :/ )\nThe results of the EffB7 was 0.969 on LB.\n\nOur final submission was ensembling the EffB7 of Step3 and effL2 of step2 and the pytorch model described by @tonyxu ", "1244491": "Congrats on our new Grandmaster!!! Really happy to work with you. Thanks so much for teaching me how to use TPU in GCP.", "1248639": "Congrats on 10th place @ludovick @woshifym.\nAnd competitions GM! @tonyxu\n\nIt is amazing to achieve that 10th place with only the classification model.\n\nIt seems 3-stage trainings.\n**1) Pytorch Part** - It also has sub 3 or 4-stage training.\n**2) TF Part** - Using 1) models and generating soft labels. And train student models (self-distillation)\n**3) Ensemble All**\n\nCould you explain more about **2)**?\n**2-1)** Generating soft labels using 1) models and retraining all of 1) models?\n**2-2)** When is the right time to use the pseudo label? (already used in sub 3 or 4-stage training?)\n\nIn other competitions, the ensemble didn't work out well with distillation models together.\nI am curious if the ensemble worked well in self-distillation using soft labels.\n(Any tips when ensemble in this cases?)\n\np.s. cvpr2020-plant-pathology was my first competition in kaggle. It's nice to see great solution again in this competition.\np.s2. My questions may be lacking because I am not participating deeply in the competition. \n\nThanks for sharing! :)", "1244920": "Congrats on becoming GM @tonyxu ", "1244579": "Congratulation in Gold Win and to the new Grandmaster Candidate\nI think PseudoLabelling were the main breakthrough for this competition.", "1246811": "" } }