\n
![\"pipeline\"](\"https://pbs.twimg.com/media/EwqG7gdVcAEOmHS?format=jpg&name=large\")
\n(n=7)
First of all, I would like to express deep gratitude to organizers and all the teams for making this competition possible. Also, I would like to thank my teammates - @yasufuminakama and @raddar .
\nCode will be available at: https://github.com/analokmaus/kaggle-ranzcr-clip-public
\nHere is the overview of our pipeline:
\n
\n(n=7)
Input image size had a good correlation with model performance. We used 768 px input for all models, considering the training / inference time. For data augmentation, we used some common transformations such as horizontal flip, affine transform, random brightness / gamma, optical distortion etc.
\nWe used UNet-CNN architecture. Output of UNet was segmentation mask consisting of three channels - ETT, NGT, and CVC+SGC. UNet was supposed to play the role of spatial supervision. Thus, compared to normal CNN-only architecture it showed more robust performance and higher CV / LB correlation.
\nAs for backbone, big model in general performed better. We used ResNet200d, EfficientNet-b7, and NFNet-F1.
\nSince there wasn’t ResNet200d encoder for segmentation_models_pytorch, we implemented it. Implementation
\nFor UNet-CNN model, we trained UNet on additional annotations provided by competition host. And then trained the whole network on all images. Since we were not able to build a stable NFNet UNet encoder, we trained NFNet in the same way as three stage training by @yasufuminakama.
\nUtilising external CXR dataset was another key to improve model performance. We pseudo-labelled NIH and MIMIC CXR dataset using our best performing model, and sampled a subset with balanced class distribution from each dataset in order to reduce data size.
\nWe trained new student model with pseudo labels on external dataset. New student model was supposed to have a different feature extractor from teacher model. We finetuned student model on original dataset, and observed better performance than teacher model (see this discussion by @ammarali32 ).
\nWe did also test conventional single stage pseudo labelling. While we observed slight uplift in performance but we gave it up because it converged quite slow.
", "messages": [ { "id": 1241588, "postDate": "2021-03-17T05:36:49.527Z", "content": "First of all, I would like to express deep gratitude to organizers and all the teams for making this competition possible. Also, I would like to thank my teammates - @yasufuminakama and @raddar .
\nCode will be available at: https://github.com/analokmaus/kaggle-ranzcr-clip-public
\nHere is the overview of our pipeline:
\n
\n(n=7)
Input image size had a good correlation with model performance. We used 768 px input for all models, considering the training / inference time. For data augmentation, we used some common transformations such as horizontal flip, affine transform, random brightness / gamma, optical distortion etc.
\nWe used UNet-CNN architecture. Output of UNet was segmentation mask consisting of three channels - ETT, NGT, and CVC+SGC. UNet was supposed to play the role of spatial supervision. Thus, compared to normal CNN-only architecture it showed more robust performance and higher CV / LB correlation.
\nAs for backbone, big model in general performed better. We used ResNet200d, EfficientNet-b7, and NFNet-F1.
\nSince there wasn’t ResNet200d encoder for segmentation_models_pytorch, we implemented it. Implementation
\nFor UNet-CNN model, we trained UNet on additional annotations provided by competition host. And then trained the whole network on all images. Since we were not able to build a stable NFNet UNet encoder, we trained NFNet in the same way as three stage training by @yasufuminakama.
\nUtilising external CXR dataset was another key to improve model performance. We pseudo-labelled NIH and MIMIC CXR dataset using our best performing model, and sampled a subset with balanced class distribution from each dataset in order to reduce data size.
\nWe trained new student model with pseudo labels on external dataset. New student model was supposed to have a different feature extractor from teacher model. We finetuned student model on original dataset, and observed better performance than teacher model (see this discussion by @ammarali32 ).
\nWe did also test conventional single stage pseudo labelling. While we observed slight uplift in performance but we gave it up because it converged quite slow.
", "rawMarkdown": "First of all, I would like to express deep gratitude to organizers and all the teams for making this competition possible. Also, I would like to thank my teammates - @yasufuminakama and @raddar .\n\nCode will be available at: https://github.com/analokmaus/kaggle-ranzcr-clip-public\n\nHere is the overview of our pipeline: \n\n(n=7)\n\n## Preprocessing \n\nInput image size had a good correlation with model performance. We used 768 px input for all models, considering the training / inference time. For data augmentation, we used some common transformations such as horizontal flip, affine transform, random brightness / gamma, optical distortion etc. \n\n## Architectures \n\nWe used **UNet-CNN** architecture. Output of UNet was segmentation mask consisting of three channels - ETT, NGT, and CVC+SGC. UNet was supposed to play the role of spatial supervision. Thus, compared to normal CNN-only architecture it showed more robust performance and higher CV / LB correlation. \n\n\n\nAs for backbone, big model in general performed better. We used ResNet200d, EfficientNet-b7, and NFNet-F1. \n\nSince there wasn’t ResNet200d encoder for segmentation_models_pytorch, we implemented it. [Implementation](https://www.kaggle.com/analokamus/resnet200d-encoder-for-segmentation-models-pytorch?scriptVersionId=56975633)\n\n## Training procedure \n\nFor UNet-CNN model, we trained UNet on additional annotations provided by competition host. And then trained the whole network on all images. Since we were not able to build a stable NFNet UNet encoder, we trained NFNet in the same way as [three stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577) by @yasufuminakama.\n\n## External dataset \n\nUtilising external CXR dataset was another key to improve model performance. We pseudo-labelled NIH and MIMIC CXR dataset using our best performing model, and sampled a subset with balanced class distribution from each dataset in order to reduce data size. \n\nWe trained new student model with pseudo labels on external dataset. New student model was supposed to have a different feature extractor from teacher model. We finetuned student model on original dataset, and observed better performance than teacher model (see [this discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) by @ammarali32 ).\n\nWe did also test conventional single stage pseudo labelling. While we observed slight uplift in performance but we gave it up because it converged quite slow.", "votes": 86 }, { "id": 1243409, "postDate": "2021-03-18T07:38:29.113Z", "content": "Code is now available at https://github.com/analokmaus/kaggle-ranzcr-clip-public !
\nNote that the repository is WIP, and there will be some updates.
Wish I had spent more time exploring the unet stuff. I wrote it up but never really trained it to extensively or with strong backbones and high resolution. Used it in a similar way as additional channels, but didnt see any benefit from it.
", "rawMarkdown": "Wish I had spent more time exploring the unet stuff. I wrote it up but never really trained it to extensively or with strong backbones and high resolution. Used it in a similar way as additional channels, but didnt see any benefit from it. ", "votes": 4, "replies": [ { "id": 1243309, "postDate": "2021-03-18T06:13:26.553Z", "content": "Our team even. we planned to do something simple
", "rawMarkdown": "Our team even. we planned to do something simple" } ] }, { "id": 1244855, "postDate": "2021-03-19T09:05:54.200Z", "content": "I like the \"n-th Place Solution\" placeholder in the pipeline schema. 😄
\nWell done!
Congrats on 7th place!
\nThank you for sharing elegant solution.
", "rawMarkdown": "Congrats on 7th place!\n\nThank you for sharing elegant solution.", "votes": 1 }, { "id": 1242672, "postDate": "2021-03-17T18:28:29.477Z", "content": "Congratulations. Eagerly waiting for your UNet training notebook and your sudo-label making technique.
", "rawMarkdown": "Congratulations. Eagerly waiting for your UNet training notebook and your sudo-label making technique. ", "votes": 1 }, { "id": 1242668, "postDate": "2021-03-17T18:27:28.797Z", "content": "Congratz !
\nInteresting to see NFNets models are viable, I was skeptical when I first saw the paper but glad to see it held the test of Kaggle :)
@analokamus Impressive Solution Pipeline . Congratulations on Gold Finish
", "rawMarkdown": "@analokamus Impressive Solution Pipeline . Congratulations on Gold Finish ", "votes": 1 }, { "id": 1241844, "postDate": "2021-03-17T08:34:00.390Z", "content": "Congrats team for 7th place !
\nExcited to see your Unet training notebook.
Amazing work. You and your team not just scored high but also helped a lot in datasets and implementations during the competition. Big thanks ))
", "rawMarkdown": "Amazing work. You and your team not just scored high but also helped a lot in datasets and implementations during the competition. Big thanks )) ", "votes": 2 }, { "id": 1241694, "postDate": "2021-03-17T06:52:53.343Z", "content": "Congrats on 7th place! @analokamus, @yasufuminakama and @raddar.
\nThe clean explanation is impressive.
Congrats @analokamus and team on 7th place and thanks for the writeup
", "rawMarkdown": "Congrats @analokamus and team on 7th place and thanks for the writeup", "votes": 2 }, { "id": 1254829, "postDate": "2021-03-28T07:01:39.280Z", "content": "Thank you @analokamus for sharing this experience. This is a good knowledge base.
", "rawMarkdown": "Thank you @analokamus for sharing this experience. This is a good knowledge base." }, { "id": 1246814, "postDate": "2021-03-21T06:16:40.960Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1243216, "postDate": "2021-03-18T05:05:51.003Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": 1 }, { "id": 1241648, "postDate": "2021-03-17T06:18:21.627Z", "content": "Elegant pipeline. Thanks for sharing.
", "rawMarkdown": "Elegant pipeline. Thanks for sharing.", "votes": 1 } ], "comments": [ { "id": 1243409, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-18T07:38:29.113000", "content": "Code is now available at https://github.com/analokmaus/kaggle-ranzcr-clip-public !
\nNote that the repository is WIP, and there will be some updates.
Wish I had spent more time exploring the unet stuff. I wrote it up but never really trained it to extensively or with strong backbones and high resolution. Used it in a similar way as additional channels, but didnt see any benefit from it.
", "votes": 4, "replies": [ { "id": 1243309, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2021-03-18T06:13:26.553000", "content": "Our team even. we planned to do something simple
", "votes": 0, "replies": [] } ] }, { "id": 1244855, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-03-19T09:05:54.200000", "content": "I like the \"n-th Place Solution\" placeholder in the pipeline schema. 😄
\nWell done!
Congrats on 7th place!
\nThank you for sharing elegant solution.
", "votes": 1, "replies": [] }, { "id": 1242672, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-03-17T18:28:29.477000", "content": "Congratulations. Eagerly waiting for your UNet training notebook and your sudo-label making technique.
", "votes": 1, "replies": [] }, { "id": 1242668, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-03-17T18:27:28.797000", "content": "Congratz !
\nInteresting to see NFNets models are viable, I was skeptical when I first saw the paper but glad to see it held the test of Kaggle :)
@analokamus Impressive Solution Pipeline . Congratulations on Gold Finish
", "votes": 1, "replies": [] }, { "id": 1241844, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-03-17T08:34:00.390000", "content": "Congrats team for 7th place !
\nExcited to see your Unet training notebook.
Amazing work. You and your team not just scored high but also helped a lot in datasets and implementations during the competition. Big thanks ))
", "votes": 2, "replies": [] }, { "id": 1241694, "author_name": "Heroseo", "author_url": "", "post_date": "2021-03-17T06:52:53.343000", "content": "Congrats on 7th place! @analokamus, @yasufuminakama and @raddar.
\nThe clean explanation is impressive.
Congrats @analokamus and team on 7th place and thanks for the writeup
", "votes": 2, "replies": [] }, { "id": 1254829, "author_name": "Olusesi Adebisi", "author_url": "", "post_date": "2021-03-28T07:01:39.280000", "content": "Thank you @analokamus for sharing this experience. This is a good knowledge base.
", "votes": 0, "replies": [] }, { "id": 1246814, "author_name": "", "author_url": "", "post_date": "2021-03-21T06:16:40.960000", "content": "", "votes": 1, "replies": [] }, { "id": 1243216, "author_name": "nbswords", "author_url": "", "post_date": "2021-03-18T05:05:51.003000", "content": "Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 1241648, "author_name": "Atharva Phatak", "author_url": "", "post_date": "2021-03-17T06:18:21.627000", "content": "Elegant pipeline. Thanks for sharing.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1241588": "First of all, I would like to express deep gratitude to organizers and all the teams for making this competition possible. Also, I would like to thank my teammates - @yasufuminakama and @raddar .\n\nCode will be available at: https://github.com/analokmaus/kaggle-ranzcr-clip-public\n\nHere is the overview of our pipeline: \n\n(n=7)\n\n## Preprocessing \n\nInput image size had a good correlation with model performance. We used 768 px input for all models, considering the training / inference time. For data augmentation, we used some common transformations such as horizontal flip, affine transform, random brightness / gamma, optical distortion etc. \n\n## Architectures \n\nWe used **UNet-CNN** architecture. Output of UNet was segmentation mask consisting of three channels - ETT, NGT, and CVC+SGC. UNet was supposed to play the role of spatial supervision. Thus, compared to normal CNN-only architecture it showed more robust performance and higher CV / LB correlation. \n\n\n\nAs for backbone, big model in general performed better. We used ResNet200d, EfficientNet-b7, and NFNet-F1. \n\nSince there wasn’t ResNet200d encoder for segmentation_models_pytorch, we implemented it. [Implementation](https://www.kaggle.com/analokamus/resnet200d-encoder-for-segmentation-models-pytorch?scriptVersionId=56975633)\n\n## Training procedure \n\nFor UNet-CNN model, we trained UNet on additional annotations provided by competition host. And then trained the whole network on all images. Since we were not able to build a stable NFNet UNet encoder, we trained NFNet in the same way as [three stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577) by @yasufuminakama.\n\n## External dataset \n\nUtilising external CXR dataset was another key to improve model performance. We pseudo-labelled NIH and MIMIC CXR dataset using our best performing model, and sampled a subset with balanced class distribution from each dataset in order to reduce data size. \n\nWe trained new student model with pseudo labels on external dataset. New student model was supposed to have a different feature extractor from teacher model. We finetuned student model on original dataset, and observed better performance than teacher model (see [this discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) by @ammarali32 ).\n\nWe did also test conventional single stage pseudo labelling. While we observed slight uplift in performance but we gave it up because it converged quite slow.", "1243409": "Code is now available at https://github.com/analokmaus/kaggle-ranzcr-clip-public !\nNote that the repository is WIP, and there will be some updates.", "1241626": "Wish I had spent more time exploring the unet stuff. I wrote it up but never really trained it to extensively or with strong backbones and high resolution. Used it in a similar way as additional channels, but didnt see any benefit from it. ", "1244855": "I like the \"n-th Place Solution\" placeholder in the pipeline schema. 😄\nWell done!", "1243890": "Congrats on 7th place!\n\nThank you for sharing elegant solution.", "1242672": "Congratulations. Eagerly waiting for your UNet training notebook and your sudo-label making technique. ", "1242668": "Congratz !\nInteresting to see NFNets models are viable, I was skeptical when I first saw the paper but glad to see it held the test of Kaggle :)", "1242026": "@analokamus Impressive Solution Pipeline . Congratulations on Gold Finish ", "1241844": "Congrats team for 7th place !\nExcited to see your Unet training notebook.", "1241960": "Amazing work. You and your team not just scored high but also helped a lot in datasets and implementations during the competition. Big thanks )) ", "1241694": "Congrats on 7th place! @analokamus, @yasufuminakama and @raddar.\nThe clean explanation is impressive.\n", "1241593": "Congrats @analokamus and team on 7th place and thanks for the writeup", "1254829": "Thank you @analokamus for sharing this experience. This is a good knowledge base.", "1246814": "", "1243216": "Thanks for sharing.", "1241648": "Elegant pipeline. Thanks for sharing." } }![\"model\"](\"https://pbs.twimg.com/media/EwqHtBDVgAIorDd?format=jpg&name=medium\")