![\"\"](\"https://i.ibb.co/7SvCkkC/4.png\")
\nWe started the competition pretraining an efficientnet B7 on NIH dataset, it gave a +0.002 boost to both CV and LB.
\nThe OOFs of this model were saved and used to apply self distillation to all our models.
I would like to thank kaggle and the hosts SIIM-FISABIO-RSNA for hosting such a special competition in this pandemic.
\nAlso, big thanks to my teammates @nicohrubec @xiaojin712 and @nickuzmenkov that have been working hard everyday for the past month trying to keep up with the final race and maintain our position in the top of the leaderboard.
| Task | \nCV | \npublic LB | \nprivate LB | \n
|---|---|---|---|
| Study level | \n0.398 | \n0.464 | \n0.428 | \n
| Image level | \n0.264 | \n0.282 | \n0.279 | \n
| Total | \n0.662 | \n0.644 | \n0.617 | \n
\nWe started the competition pretraining an efficientnet B7 on NIH dataset, it gave a +0.002 boost to both CV and LB.
\nThe OOFs of this model were saved and used to apply self distillation to all our models.
![\"\"](\"https://i.ibb.co/DVSwVSc/SIIM-COVID-writeup.png\")
Self distillation:
\nWe started using self distillation by blending the labels*0.7 + B7 OOFs*0.3. The idea is similar to label smoothing, introduce some noise to the labels instead of feeding one hot encoded representation to the model. We kept increasing the B7' OOF's coefficient and both CV and LB kept increasing until we reached the final labels*0.15 + B7 OOFs*0.85 with a total boost of +0.007 on LB.
Pseudolabeling:
\nNothing special about how we do this. We just take the publicly available part of the test set and label it with our predictions.
\nWe then add the test data as additional data to all the train folds with the predictions as soft labels. This was one of the first improvements we made to our pytorch pipeline. At the time it improved CV by +0.007 and LB by +0.012.
Segmentation as auxilliary Head:
\nThe aux head code we essentially copied from Heng. The aux head is attached after the 4th convolutional block. The loss is calculated as 0.7 LOVASZ + 0.3 BCE.
\nInterestingly, we found that AUX did indeed improve the performance, but only for models without pseudolabeling. We did not investigate the reason for this any further. One potential reason could be that the predicted masks for the test set are too noisy to be useful.
\nEffnet V2-M + aux head didn't work as good as it did for other teams, it didn't outperform our best single model trained with pseudolabels. Including it in the blend gave a boost to the ensemble.
Finetuning:
\nIn the beginning we were working with V2L and image size 512. At some point we switched to image size 384 with Efficientnet V2M.
\nAdditionally, we tuned tuned the learning rate. All these changes together resulted in an increase of CV and LB of about 0.01.
![\"\"](\"https://i.ibb.co/5TBGf3P/3.png\")
\nTrained the 2 classes (Opacity/None) with the same set up as the study level models. We got a +0.004 boost with respect to the 2 classes classifier public notebook.
![\"\"](\"https://i.ibb.co/NnFrPPS/2.png\")
Loss = BCE(labels) + MSE(unpooled features).Loss = BCE(labels) + MSE(unpooled features) + Lovasz(masks).The study level models ensemble gave us a +0.004 boost on public LB 0.460 -> 0.464, which helped us reach the final score of 0.644. It also boosted our CV from 0.393 -> 0.398. However, it didn't work on private LB. The ensemble scored lower than our best single model on private LB, which is the reason behind our shakedown.
", "messages": [ { "id": "1465020", "postDate": "08/10/2021 20:05:32", "content": "I would like to thank kaggle and the hosts SIIM-FISABIO-RSNA for hosting such a special competition in this pandemic.
\nAlso, big thanks to my teammates @nicohrubec @xiaojin712 and @nickuzmenkov that have been working hard everyday for the past month trying to keep up with the final race and maintain our position in the top of the leaderboard.
| Task | \nCV | \npublic LB | \nprivate LB | \n
|---|---|---|---|
| Study level | \n0.398 | \n0.464 | \n0.428 | \n
| Image level | \n0.264 | \n0.282 | \n0.279 | \n
| Total | \n0.662 | \n0.644 | \n0.617 | \n
\nWe started the competition pretraining an efficientnet B7 on NIH dataset, it gave a +0.002 boost to both CV and LB.
\nThe OOFs of this model were saved and used to apply self distillation to all our models.
Self distillation:
\nWe started using self distillation by blending the labels*0.7 + B7 OOFs*0.3. The idea is similar to label smoothing, introduce some noise to the labels instead of feeding one hot encoded representation to the model. We kept increasing the B7' OOF's coefficient and both CV and LB kept increasing until we reached the final labels*0.15 + B7 OOFs*0.85 with a total boost of +0.007 on LB.
Pseudolabeling:
\nNothing special about how we do this. We just take the publicly available part of the test set and label it with our predictions.
\nWe then add the test data as additional data to all the train folds with the predictions as soft labels. This was one of the first improvements we made to our pytorch pipeline. At the time it improved CV by +0.007 and LB by +0.012.
Segmentation as auxilliary Head:
\nThe aux head code we essentially copied from Heng. The aux head is attached after the 4th convolutional block. The loss is calculated as 0.7 LOVASZ + 0.3 BCE.
\nInterestingly, we found that AUX did indeed improve the performance, but only for models without pseudolabeling. We did not investigate the reason for this any further. One potential reason could be that the predicted masks for the test set are too noisy to be useful.
\nEffnet V2-M + aux head didn't work as good as it did for other teams, it didn't outperform our best single model trained with pseudolabels. Including it in the blend gave a boost to the ensemble.
Finetuning:
\nIn the beginning we were working with V2L and image size 512. At some point we switched to image size 384 with Efficientnet V2M.
\nAdditionally, we tuned tuned the learning rate. All these changes together resulted in an increase of CV and LB of about 0.01.
\nTrained the 2 classes (Opacity/None) with the same set up as the study level models. We got a +0.004 boost with respect to the 2 classes classifier public notebook.
Loss = BCE(labels) + MSE(unpooled features).Loss = BCE(labels) + MSE(unpooled features) + Lovasz(masks).The study level models ensemble gave us a +0.004 boost on public LB 0.460 -> 0.464, which helped us reach the final score of 0.644. It also boosted our CV from 0.393 -> 0.398. However, it didn't work on private LB. The ensemble scored lower than our best single model on private LB, which is the reason behind our shakedown.
", "rawMarkdown": "I would like to thank kaggle and the hosts SIIM-FISABIO-RSNA for hosting such a special competition in this pandemic.\nAlso, big thanks to my teammates @nicohrubec @xiaojin712 and @nickuzmenkov that have been working hard everyday for the past month trying to keep up with the final race and maintain our position in the top of the leaderboard. \n\n### Final scores\n\n| **Task** | **CV** | **public LB** | **private LB** |\n| --- | --- |\n| **Study level**| 0.398 | 0.464 | 0.428 |\n| **Image level** | 0.264 | 0.282 | 0.279 |\n| **Total** | 0.662 | 0.644 | 0.617 |\n\n\n\n## 1- Pretraining\n\nWe started the competition pretraining an efficientnet B7 on NIH dataset, it gave a +0.002 boost to both CV and LB.\nThe OOFs of this model were saved and used to apply self distillation to all our models.\n* Pretrain on [NIH Chest X-Ray Dataset](https://www.kaggle.com/nih-chest-xrays/data).\n* [EfficientNet B7 training notebook](https://www.kaggle.com/nickuzmenkov/29-place-solution-siim-covid-tf-study-training).\n\n\n## 2- Study level classification\n\n* **Self distillation:** \nWe started using self distillation by blending the `labels*0.7 + B7 OOFs*0.3`. The idea is similar to label smoothing, introduce some noise to the labels instead of feeding one hot encoded representation to the model. We kept increasing the B7' OOF's coefficient and both CV and LB kept increasing until we reached the final `labels*0.15 + B7 OOFs*0.85` with a total boost of **+0.007** on LB.\n\n\n* **Pseudolabeling:**\nNothing special about how we do this. We just take the publicly available part of the test set and label it with our predictions.\nWe then add the test data as additional data to all the train folds with the predictions as soft labels. This was one of the first improvements we made to our pytorch pipeline. At the time it improved CV by **+0.007** and LB by **+0.012**.\n\n* **Segmentation as auxilliary Head:**\nThe aux head code we essentially copied from Heng. The aux head is attached after the 4th convolutional block. The loss is calculated as `0.7 LOVASZ + 0.3 BCE`.\nInterestingly, we found that AUX did indeed improve the performance, but only for models without pseudolabeling. We did not investigate the reason for this any further. One potential reason could be that the predicted masks for the test set are too noisy to be useful.\nEffnet V2-M + aux head didn't work as good as it did for other teams, it didn't outperform our best single model trained with pseudolabels. Including it in the blend gave a boost to the ensemble.\n\n* **Finetuning:**\nIn the beginning we were working with V2L and image size 512. At some point we switched to image size 384 with Efficientnet V2M.\nAdditionally, we tuned tuned the learning rate. All these changes together resulted in an increase of CV and LB of about 0.01.\n\n## 3- Opacity/None classification\n\nTrained the 2 classes (Opacity/None) with the same set up as the study level models. We got a +0.004 boost with respect to the 2 classes classifier public notebook.\n\n## 4- Image level detection\n\n* Yolov5x was trained with image size 512x512.\n* Yolov5x6 and YoloV5L6 were trained with image size 1024x1024.\n* Pseudolabeling improved the score a little bit, not as much as with study level models.\n* None class was blended with the negative for pneumonia class from the study level ensemble.\n* Ensemble (WBF) of 3 YoloV5 models + None class gave a final score of 0.282 on LB.\n\n## What worked for us:\n* Heavy augmentations (even very destructive ones).\n* Self distillation.\n* Pseudolabelling.\n\n## What didn't work for us:\n* Larger image sizes.\n* Knowledge distillation: \n`Loss = BCE(labels) + MSE(unpooled features).`\n* Knowledge distillation + Aux loss: \n`Loss = BCE(labels) + MSE(unpooled features) + Lovasz(masks).`\n* TTA (For study level).\n* EfficientNetV2 in TensorFlow.\n* ResNet models (seResNet50, seResNet-101, ResNet200D...)\n* Crossentropy with label smoothing.\n\n\n\n## Final thoughts\nThe study level models ensemble gave us a +0.004 boost on public LB **0.460 -> 0.464**, which helped us reach the final score of 0.644. It also boosted our CV from **0.393 -> 0.398**. However, it didn't work on private LB. The ensemble scored lower than our best single model on private LB, which is the reason behind our shakedown.", "votes": null }, { "id": "1465219", "postDate": "08/11/2021 00:24:12", "content": "Iām very happy to team up with you and @nicohrubec @nickuzmenkov and hope to win the gold medal next competition. ššš
\nThanks @amiiiney sharing our approach, å
å¼ä»¬å ę²¹ļ¼ļ¼ļ¼
Thank you @xiaojin712 :) It was a pleasure teaming up with you! I learned a lot from you. I hope next time we get shaken up and not down š
", "rawMarkdown": "Thank you @xiaojin712 :) It was a pleasure teaming up with you! I learned a lot from you. I hope next time we get shaken up and not down š", "votes": null }, { "id": "1466316", "postDate": "08/11/2021 12:09:54", "content": "Hi @xiaojin712 congratulations for your achievements, may I ask if you think kaggle medals will help us compete against other candidates when pursuing AI jobs, if yes, to what extent?
", "rawMarkdown": "Hi @xiaojin712 congratulations for your achievements, may I ask if you think kaggle medals will help us compete against other candidates when pursuing AI jobs, if yes, to what extent?", "votes": null }, { "id": "1466399", "postDate": "08/11/2021 12:52:23", "content": "Thank you @plugin1689 ļ¼I don't think that the kaggle medal will improve my work competitiveness, but through this platform I learned excellent solutions in object detection and image classification, and I can directly use these approaches in my work to meet customer needs. Mostly for the purpose of learning.
", "rawMarkdown": "Thank you @plugin1689 ļ¼I don't think that the kaggle medal will improve my work competitiveness, but through this platform I learned excellent solutions in object detection and image classification, and I can directly use these approaches in my work to meet customer needs. Mostly for the purpose of learning.", "votes": null }, { "id": "1471196", "postDate": "08/14/2021 03:26:11", "content": "Do you use only image size 384x384 in classification phase ?
", "rawMarkdown": "Do you use only image size 384x384 in classification phase ?", "votes": null }, { "id": "1471403", "postDate": "08/14/2021 07:25:04", "content": "Yes, in our setup it gave the best CV scores. In the end we tried to add an EffnetV2L trained with image size 512 to the ensemble, but that did not help as well.
", "rawMarkdown": "Yes, in our setup it gave the best CV scores. In the end we tried to add an EffnetV2L trained with image size 512 to the ensemble, but that did not help as well.", "votes": null } ], "comments": [ { "id": 1465219, "author_name": "xiaojin712", "author_url": "", "post_date": "08/11/2021 00:24:12", "content": "Iām very happy to team up with you and @nicohrubec @nickuzmenkov and hope to win the gold medal next competition. ššš
\nThanks @amiiiney sharing our approach, å
å¼ä»¬å ę²¹ļ¼ļ¼ļ¼
Thank you @xiaojin712 :) It was a pleasure teaming up with you! I learned a lot from you. I hope next time we get shaken up and not down š
", "votes": null, "replies": [] }, { "id": 1466316, "author_name": "plugin1689", "author_url": "", "post_date": "08/11/2021 12:09:54", "content": "Hi @xiaojin712 congratulations for your achievements, may I ask if you think kaggle medals will help us compete against other candidates when pursuing AI jobs, if yes, to what extent?
", "votes": null, "replies": [] }, { "id": 1466399, "author_name": "xiaojin712", "author_url": "", "post_date": "08/11/2021 12:52:23", "content": "Thank you @plugin1689 ļ¼I don't think that the kaggle medal will improve my work competitiveness, but through this platform I learned excellent solutions in object detection and image classification, and I can directly use these approaches in my work to meet customer needs. Mostly for the purpose of learning.
", "votes": null, "replies": [] } ] }, { "id": 1471196, "author_name": "atom1231", "author_url": "", "post_date": "08/14/2021 03:26:11", "content": "Do you use only image size 384x384 in classification phase ?
", "votes": null, "replies": [ { "id": 1471403, "author_name": "nicohrubec", "author_url": "", "post_date": "08/14/2021 07:25:04", "content": "Yes, in our setup it gave the best CV scores. In the end we tried to add an EffnetV2L trained with image size 512 to the ensemble, but that did not help as well.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1465020": "I would like to thank kaggle and the hosts SIIM-FISABIO-RSNA for hosting such a special competition in this pandemic.\nAlso, big thanks to my teammates @nicohrubec @xiaojin712 and @nickuzmenkov that have been working hard everyday for the past month trying to keep up with the final race and maintain our position in the top of the leaderboard. \n\n### Final scores\n\n| **Task** | **CV** | **public LB** | **private LB** |\n| --- | --- |\n| **Study level**| 0.398 | 0.464 | 0.428 |\n| **Image level** | 0.264 | 0.282 | 0.279 |\n| **Total** | 0.662 | 0.644 | 0.617 |\n\n\n\n## 1- Pretraining\n\nWe started the competition pretraining an efficientnet B7 on NIH dataset, it gave a +0.002 boost to both CV and LB.\nThe OOFs of this model were saved and used to apply self distillation to all our models.\n* Pretrain on [NIH Chest X-Ray Dataset](https://www.kaggle.com/nih-chest-xrays/data).\n* [EfficientNet B7 training notebook](https://www.kaggle.com/nickuzmenkov/29-place-solution-siim-covid-tf-study-training).\n\n\n## 2- Study level classification\n\n* **Self distillation:** \nWe started using self distillation by blending the `labels*0.7 + B7 OOFs*0.3`. The idea is similar to label smoothing, introduce some noise to the labels instead of feeding one hot encoded representation to the model. We kept increasing the B7' OOF's coefficient and both CV and LB kept increasing until we reached the final `labels*0.15 + B7 OOFs*0.85` with a total boost of **+0.007** on LB.\n\n\n* **Pseudolabeling:**\nNothing special about how we do this. We just take the publicly available part of the test set and label it with our predictions.\nWe then add the test data as additional data to all the train folds with the predictions as soft labels. This was one of the first improvements we made to our pytorch pipeline. At the time it improved CV by **+0.007** and LB by **+0.012**.\n\n* **Segmentation as auxilliary Head:**\nThe aux head code we essentially copied from Heng. The aux head is attached after the 4th convolutional block. The loss is calculated as `0.7 LOVASZ + 0.3 BCE`.\nInterestingly, we found that AUX did indeed improve the performance, but only for models without pseudolabeling. We did not investigate the reason for this any further. One potential reason could be that the predicted masks for the test set are too noisy to be useful.\nEffnet V2-M + aux head didn't work as good as it did for other teams, it didn't outperform our best single model trained with pseudolabels. Including it in the blend gave a boost to the ensemble.\n\n* **Finetuning:**\nIn the beginning we were working with V2L and image size 512. At some point we switched to image size 384 with Efficientnet V2M.\nAdditionally, we tuned tuned the learning rate. All these changes together resulted in an increase of CV and LB of about 0.01.\n\n## 3- Opacity/None classification\n\nTrained the 2 classes (Opacity/None) with the same set up as the study level models. We got a +0.004 boost with respect to the 2 classes classifier public notebook.\n\n## 4- Image level detection\n\n* Yolov5x was trained with image size 512x512.\n* Yolov5x6 and YoloV5L6 were trained with image size 1024x1024.\n* Pseudolabeling improved the score a little bit, not as much as with study level models.\n* None class was blended with the negative for pneumonia class from the study level ensemble.\n* Ensemble (WBF) of 3 YoloV5 models + None class gave a final score of 0.282 on LB.\n\n## What worked for us:\n* Heavy augmentations (even very destructive ones).\n* Self distillation.\n* Pseudolabelling.\n\n## What didn't work for us:\n* Larger image sizes.\n* Knowledge distillation: \n`Loss = BCE(labels) + MSE(unpooled features).`\n* Knowledge distillation + Aux loss: \n`Loss = BCE(labels) + MSE(unpooled features) + Lovasz(masks).`\n* TTA (For study level).\n* EfficientNetV2 in TensorFlow.\n* ResNet models (seResNet50, seResNet-101, ResNet200D...)\n* Crossentropy with label smoothing.\n\n\n\n## Final thoughts\nThe study level models ensemble gave us a +0.004 boost on public LB **0.460 -> 0.464**, which helped us reach the final score of 0.644. It also boosted our CV from **0.393 -> 0.398**. However, it didn't work on private LB. The ensemble scored lower than our best single model on private LB, which is the reason behind our shakedown.", "1465219": "Iām very happy to team up with you and @nicohrubec @nickuzmenkov and hope to win the gold medal next competition. ššš\nThanks @amiiiney sharing our approach, å å¼ä»¬å ę²¹ļ¼ļ¼ļ¼", "1465653": "Thank you @xiaojin712 :) It was a pleasure teaming up with you! I learned a lot from you. I hope next time we get shaken up and not down š", "1466316": "Hi @xiaojin712 congratulations for your achievements, may I ask if you think kaggle medals will help us compete against other candidates when pursuing AI jobs, if yes, to what extent?", "1466399": "Thank you @plugin1689 ļ¼I don't think that the kaggle medal will improve my work competitiveness, but through this platform I learned excellent solutions in object detection and image classification, and I can directly use these approaches in my work to meet customer needs. Mostly for the purpose of learning.", "1471196": "Do you use only image size 384x384 in classification phase ?", "1471403": "Yes, in our setup it gave the best CV scores. In the end we tried to add an EffnetV2L trained with image size 512 to the ensemble, but that did not help as well." }, "source": "meta" }