{ "id": 226595, "title": "16th Place Solution", "url": "/competitions/ranzcr-clip-catheter-line-classification/writeups/hotwater-16th-place-solution", "author_name": "", "post_date": "2021-03-17T07:03:01.543Z", "votes": 28, "comment_count": 15, "views": 0, "content": "

Thanks to Kaggle and hosts for this very interesting competition with a annotations. This has been a great collaborative effort and please also give your upvotes to @syxuming, @fanwenping, @chanyanyuese. Congrats to Winners

\n

Funny thing we had CV more than 1st Place Winner but lower in LB 😂😂

\n

TLDR

\n

We had a simple approach . here is the basic diagram,
\n\"\"

\n

First of All I would like to thank @ammarali32 and @ttahara for the starting points and notebooks. @ttahara your model in CV scored 0.9767 LB 0.972 (which is our best single) and Staged training proposed by @hengck23 and kernel provided by @yasufuminakama and followed by finetuning with multiple datasets

\n

CV strategy : we all had had almost different CV split but same algorithm. as proposed by @underwearfitting

\n

Models

\n

We basically used 3 backbones with 4 heads( but only 2 of them were experimented) into submission:

\n\n

Our Strategy

\n

We Started with the idea proposed by @hengck23 and @ttahara as starting.
\nWe train the model as 3 stages as @hengck23. When we trained Resnet200d with GeM with soft label(which is created by stage 3 model) on our competition set itself ( we thought it as same as knowledge distillation). and pretrain the model. this model showed us a CV: 0.97 and LB: wasn't tested. We found this soft labelling helps a lot. And we continue to do this on NIH , PadChest , VinBigData external dataset.
\ndoing this CV: 0.971/ PublicLB: 0.970/ PrivateLB: 0.971 for 3 staged model
\nand CV: 0.9767/PublicLB: 0.970/ PrivateLB: 0.972 for Multi Head Attention
\nWe first soft labelled only NIH and pretrained and finetuned and then PadChest pretrained and finetuned. We trained every staged model into 5 folds and only 1 fold for Multi Stage. We also used @ammarali32 's public high scoring weights and pretrained and finetuned to give into the ensemble.

\n

Ensembling

\n

We used simple averaging

\n

Other Points

\n

Augmentation

\n
RandomResizedCrop(CFG.img_size, CFG.img_size, scale=(0.9, 1), p=1), \nHorizontalFlip(p=0.5), ShiftScaleRotate(p=0.5),\nHueSaturationValue(hue_shift_limit=10, sat_shift_limit=10, val_shift_limit=10, p=0.7),\nRandomBrightnessContrast(brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2), p=0.7),\nCLAHE(clip_limit=(1,4), p=0.5),\nOneOf([ \nOpticalDistortion(distort_limit=1.0), GridDistortion(num_steps=5, distort_limit=1.),\nElasticTransform(alpha=3), ], p=0.2),\nOneOf([ GaussNoise(var_limit=[10, 50]),\nGaussianBlur(),\nMotionBlur(),\nMedianBlur(), ], p=0.2),\nResize(CFG.img_size, CFG.img_size), OneOf([ JpegCompression(), Downscale(scale_min=0.1, scale_max=0.15), ], p=0.2), IAAPiecewiseAffine()\n
\n

Logging: Neptune.ai

\n

Things we didn't had time to do

\n\n

Things didn't worked

\n\n

Thanks to @philippsinger for his writeup where I got how to write a solution writeup (since this is my First writeup)

\n

Thanks to All we learned a lot Team HotWater

", "messages": [ { "id": "1241461", "postDate": "03/17/2021 03:52:19", "content": "

Thanks to Kaggle and hosts for this very interesting competition with a annotations. This has been a great collaborative effort and please also give your upvotes to @syxuming, @fanwenping, @chanyanyuese. Congrats to Winners

\n

Funny thing we had CV more than 1st Place Winner but lower in LB 😂😂

\n

TLDR

\n

We had a simple approach . here is the basic diagram,
\n\"\"

\n

First of All I would like to thank @ammarali32 and @ttahara for the starting points and notebooks. @ttahara your model in CV scored 0.9767 LB 0.972 (which is our best single) and Staged training proposed by @hengck23 and kernel provided by @yasufuminakama and followed by finetuning with multiple datasets

\n

CV strategy : we all had had almost different CV split but same algorithm. as proposed by @underwearfitting

\n

Models

\n

We basically used 3 backbones with 4 heads( but only 2 of them were experimented) into submission:

\n\n

Our Strategy

\n

We Started with the idea proposed by @hengck23 and @ttahara as starting.
\nWe train the model as 3 stages as @hengck23. When we trained Resnet200d with GeM with soft label(which is created by stage 3 model) on our competition set itself ( we thought it as same as knowledge distillation). and pretrain the model. this model showed us a CV: 0.97 and LB: wasn't tested. We found this soft labelling helps a lot. And we continue to do this on NIH , PadChest , VinBigData external dataset.
\ndoing this CV: 0.971/ PublicLB: 0.970/ PrivateLB: 0.971 for 3 staged model
\nand CV: 0.9767/PublicLB: 0.970/ PrivateLB: 0.972 for Multi Head Attention
\nWe first soft labelled only NIH and pretrained and finetuned and then PadChest pretrained and finetuned. We trained every staged model into 5 folds and only 1 fold for Multi Stage. We also used @ammarali32 's public high scoring weights and pretrained and finetuned to give into the ensemble.

\n

Ensembling

\n

We used simple averaging

\n

Other Points

\n

Augmentation

\n
RandomResizedCrop(CFG.img_size, CFG.img_size, scale=(0.9, 1), p=1), \nHorizontalFlip(p=0.5), ShiftScaleRotate(p=0.5),\nHueSaturationValue(hue_shift_limit=10, sat_shift_limit=10, val_shift_limit=10, p=0.7),\nRandomBrightnessContrast(brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2), p=0.7),\nCLAHE(clip_limit=(1,4), p=0.5),\nOneOf([ \nOpticalDistortion(distort_limit=1.0), GridDistortion(num_steps=5, distort_limit=1.),\nElasticTransform(alpha=3), ], p=0.2),\nOneOf([ GaussNoise(var_limit=[10, 50]),\nGaussianBlur(),\nMotionBlur(),\nMedianBlur(), ], p=0.2),\nResize(CFG.img_size, CFG.img_size), OneOf([ JpegCompression(), Downscale(scale_min=0.1, scale_max=0.15), ], p=0.2), IAAPiecewiseAffine()\n
\n

Logging: Neptune.ai

\n

Things we didn't had time to do

\n\n

Things didn't worked

\n\n

Thanks to @philippsinger for his writeup where I got how to write a solution writeup (since this is my First writeup)

\n

Thanks to All we learned a lot Team HotWater

", "rawMarkdown": "**Thanks to Kaggle and hosts for this very interesting competition with a annotations. This has been a great collaborative effort and please also give your upvotes to @syxuming, @fanwenping, @chanyanyuese. Congrats to Winners**\n\nFunny thing we had CV more than 1st Place Winner but lower in LB 😂😂\n\n### TLDR\nWe had a simple approach . here is the basic diagram,\n![](https://i.ibb.co/f2j1XwB/Blank-diagram-6.png)\n\nFirst of All I would like to thank @ammarali32 and @ttahara for the starting points and notebooks. @ttahara your model in CV scored 0.9767 LB 0.972 (which is our best single) and Staged training proposed by @hengck23 and kernel provided by @yasufuminakama and followed by finetuning with multiple datasets\n\n**CV strategy** : we all had had almost different CV split but same algorithm. as proposed by @underwearfitting\n### Models\nWe basically used 3 backbones with 4 heads( but only 2 of them were experimented) into submission:\n- Backbones\n--. Resnet200d\n--. EfficientNetB7\n-- Resnet50d\n- Head\n-- Multi head Attention\n-- GeM\n-- Simple Global Pooling\n-- AdaptiveConcatPooling\n### Our Strategy\nWe Started with the idea proposed by @hengck23 and @ttahara as starting. \nWe train the model as 3 stages as @hengck23. When we trained Resnet200d with GeM with soft label(which is created by stage 3 model) on our competition set itself ( we thought it as same as knowledge distillation). and pretrain the model. this model showed us a CV: 0.97 and LB: wasn't tested. We found this soft labelling helps a lot. And we continue to do this on NIH , PadChest , VinBigData external dataset. \ndoing this CV: 0.971/ PublicLB: 0.970/ PrivateLB: 0.971 for 3 staged model\nand CV: 0.9767/PublicLB: 0.970/ PrivateLB: 0.972 for Multi Head Attention\nWe first soft labelled only NIH and pretrained and finetuned and then PadChest pretrained and finetuned. We trained every staged model into 5 folds and only 1 fold for Multi Stage. We also used @ammarali32 's public high scoring weights and pretrained and finetuned to give into the ensemble.\n\n### Ensembling \nWe used simple averaging\n### Other Points\n#### Augmentation \n```\nRandomResizedCrop(CFG.img_size, CFG.img_size, scale=(0.9, 1), p=1), \nHorizontalFlip(p=0.5), ShiftScaleRotate(p=0.5),\nHueSaturationValue(hue_shift_limit=10, sat_shift_limit=10, val_shift_limit=10, p=0.7),\nRandomBrightnessContrast(brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2), p=0.7),\nCLAHE(clip_limit=(1,4), p=0.5),\nOneOf([ \nOpticalDistortion(distort_limit=1.0), GridDistortion(num_steps=5, distort_limit=1.),\nElasticTransform(alpha=3), ], p=0.2),\nOneOf([ GaussNoise(var_limit=[10, 50]),\nGaussianBlur(),\nMotionBlur(),\nMedianBlur(), ], p=0.2),\nResize(CFG.img_size, CFG.img_size), OneOf([ JpegCompression(), Downscale(scale_min=0.1, scale_max=0.15), ], p=0.2), IAAPiecewiseAffine()\n```\n**Logging**: Neptune.ai\n\n### Things we didn't had time to do\n- Segmentation based learning ( that made us apart from other top candidates)\n- Retrieval based learning\n- GeM Pooling and AdaptiveConcatPool ( we prepared didnt experimented )\n### Things didn't worked \n- Multi Staged Ensembling ( because it was hurting too much )\n- Dynamic Temperature for pseudo labelling\n\nThanks to @philippsinger for his [writeup](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220563) where I got how to write a solution writeup (since this is my First writeup)\n\nThanks to All we learned a lot Team **HotWater**", "votes": null }, { "id": "1241471", "postDate": "03/17/2021 04:06:47", "content": "

Congratulation guys good work ))

", "rawMarkdown": "Congratulation guys good work ))", "votes": null }, { "id": "1241473", "postDate": "03/17/2021 04:09:00", "content": "

Thanks

", "rawMarkdown": "Thanks", "votes": null }, { "id": "1241510", "postDate": "03/17/2021 04:36:48", "content": "

Congratulations! Thank you for sharing your solution. Great work!

", "rawMarkdown": "Congratulations! Thank you for sharing your solution. Great work!", "votes": null }, { "id": "1241524", "postDate": "03/17/2021 04:48:31", "content": "

Congrats great work!! 🔥 @morizin
\nI guess training all these model would have required a lot of hardware.

", "rawMarkdown": "Congrats great work!! 🔥 @morizin \nI guess training all these model would have required a lot of hardware.", "votes": null }, { "id": "1241536", "postDate": "03/17/2021 04:55:08", "content": "

Sure. We had 8x V100 GPU(cloud) and 3x 3090 GPU

", "rawMarkdown": "Sure. We had 8x V100 GPU(cloud) and 3x 3090 GPU", "votes": null }, { "id": "1241537", "postDate": "03/17/2021 04:55:41", "content": "

Thanks, also Congratulation on your achievement

", "rawMarkdown": "Thanks, also Congratulation on your achievement", "votes": null }, { "id": "1241609", "postDate": "03/17/2021 05:54:54", "content": "

Congrats on 16th place @morizin and team, good job

", "rawMarkdown": "Congrats on 16th place @morizin and team, good job", "votes": null }, { "id": "1241616", "postDate": "03/17/2021 05:57:51", "content": "

Thanks

", "rawMarkdown": "Thanks", "votes": null }, { "id": "1241913", "postDate": "03/17/2021 09:19:08", "content": "

Thank you for sharing. Great Work!

\n

I'm very happy to hear that my model is the your best single model 😎

", "rawMarkdown": "Thank you for sharing. Great Work!\n\nI'm very happy to hear that my model is the your best single model 😎", "votes": null }, { "id": "1241917", "postDate": "03/17/2021 09:21:07", "content": "

Thanks for your MultiHead Attention
\nAnd Congratulations on your score!

", "rawMarkdown": "Thanks for your MultiHead Attention\nAnd Congratulations on your score!", "votes": null }, { "id": "1242028", "postDate": "03/17/2021 10:58:23", "content": "

@morizin Great work and Congratulations on Silver Finish

", "rawMarkdown": "morizin Great work and Congratulations on Silver Finish", "votes": null }, { "id": "1242684", "postDate": "03/17/2021 18:32:58", "content": "

Congratz on the nice finish & thanks for the write-up !

", "rawMarkdown": "Congratz on the nice finish & thanks for the write-up !", "votes": null }, { "id": "1242701", "postDate": "03/17/2021 18:42:01", "content": "

I think it was a collabrative effort

", "rawMarkdown": "I think it was a collabrative effort", "votes": null }, { "id": "1243480", "postDate": "03/18/2021 08:38:33", "content": "

Thank you

", "rawMarkdown": "Thank you", "votes": null }, { "id": "1244572", "postDate": "03/19/2021 04:48:13", "content": "

Sorry, I have a Change in the architecture
\nThe Multi Head Attention were also 3 stages.

", "rawMarkdown": "Sorry, I have a Change in the architecture\nThe Multi Head Attention were also 3 stages.", "votes": null } ], "comments": [ { "id": 1241471, "author_name": "ammarali32", "author_url": "", "post_date": "03/17/2021 04:06:47", "content": "

Congratulation guys good work ))

", "votes": null, "replies": [ { "id": 1241473, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 04:09:00", "content": "

Thanks

", "votes": null, "replies": [] } ] }, { "id": 1241510, "author_name": "yoshitaka1105", "author_url": "", "post_date": "03/17/2021 04:36:48", "content": "

Congratulations! Thank you for sharing your solution. Great work!

", "votes": null, "replies": [ { "id": 1241537, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 04:55:41", "content": "

Thanks, also Congratulation on your achievement

", "votes": null, "replies": [] } ] }, { "id": 1241524, "author_name": "rsinda", "author_url": "", "post_date": "03/17/2021 04:48:31", "content": "

Congrats great work!! 🔥 @morizin
\nI guess training all these model would have required a lot of hardware.

", "votes": null, "replies": [ { "id": 1241536, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 04:55:08", "content": "

Sure. We had 8x V100 GPU(cloud) and 3x 3090 GPU

", "votes": null, "replies": [] } ] }, { "id": 1241609, "author_name": "duykhanh99", "author_url": "", "post_date": "03/17/2021 05:54:54", "content": "

Congrats on 16th place @morizin and team, good job

", "votes": null, "replies": [ { "id": 1241616, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 05:57:51", "content": "

Thanks

", "votes": null, "replies": [] } ] }, { "id": 1241913, "author_name": "ttahara", "author_url": "", "post_date": "03/17/2021 09:19:08", "content": "

Thank you for sharing. Great Work!

\n

I'm very happy to hear that my model is the your best single model 😎

", "votes": null, "replies": [ { "id": 1241917, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 09:21:07", "content": "

Thanks for your MultiHead Attention
\nAnd Congratulations on your score!

", "votes": null, "replies": [] } ] }, { "id": 1242028, "author_name": "usharengaraju", "author_url": "", "post_date": "03/17/2021 10:58:23", "content": "

@morizin Great work and Congratulations on Silver Finish

", "votes": null, "replies": [ { "id": 1243480, "author_name": "morizin", "author_url": "", "post_date": "03/18/2021 08:38:33", "content": "

Thank you

", "votes": null, "replies": [] } ] }, { "id": 1242684, "author_name": "theoviel", "author_url": "", "post_date": "03/17/2021 18:32:58", "content": "

Congratz on the nice finish & thanks for the write-up !

", "votes": null, "replies": [ { "id": 1242701, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 18:42:01", "content": "

I think it was a collabrative effort

", "votes": null, "replies": [] } ] }, { "id": 1244572, "author_name": "morizin", "author_url": "", "post_date": "03/19/2021 04:48:13", "content": "

Sorry, I have a Change in the architecture
\nThe Multi Head Attention were also 3 stages.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1241461": "**Thanks to Kaggle and hosts for this very interesting competition with a annotations. This has been a great collaborative effort and please also give your upvotes to @syxuming, @fanwenping, @chanyanyuese. Congrats to Winners**\n\nFunny thing we had CV more than 1st Place Winner but lower in LB 😂😂\n\n### TLDR\nWe had a simple approach . here is the basic diagram,\n![](https://i.ibb.co/f2j1XwB/Blank-diagram-6.png)\n\nFirst of All I would like to thank @ammarali32 and @ttahara for the starting points and notebooks. @ttahara your model in CV scored 0.9767 LB 0.972 (which is our best single) and Staged training proposed by @hengck23 and kernel provided by @yasufuminakama and followed by finetuning with multiple datasets\n\n**CV strategy** : we all had had almost different CV split but same algorithm. as proposed by @underwearfitting\n### Models\nWe basically used 3 backbones with 4 heads( but only 2 of them were experimented) into submission:\n- Backbones\n--. Resnet200d\n--. EfficientNetB7\n-- Resnet50d\n- Head\n-- Multi head Attention\n-- GeM\n-- Simple Global Pooling\n-- AdaptiveConcatPooling\n### Our Strategy\nWe Started with the idea proposed by @hengck23 and @ttahara as starting. \nWe train the model as 3 stages as @hengck23. When we trained Resnet200d with GeM with soft label(which is created by stage 3 model) on our competition set itself ( we thought it as same as knowledge distillation). and pretrain the model. this model showed us a CV: 0.97 and LB: wasn't tested. We found this soft labelling helps a lot. And we continue to do this on NIH , PadChest , VinBigData external dataset. \ndoing this CV: 0.971/ PublicLB: 0.970/ PrivateLB: 0.971 for 3 staged model\nand CV: 0.9767/PublicLB: 0.970/ PrivateLB: 0.972 for Multi Head Attention\nWe first soft labelled only NIH and pretrained and finetuned and then PadChest pretrained and finetuned. We trained every staged model into 5 folds and only 1 fold for Multi Stage. We also used @ammarali32 's public high scoring weights and pretrained and finetuned to give into the ensemble.\n\n### Ensembling \nWe used simple averaging\n### Other Points\n#### Augmentation \n```\nRandomResizedCrop(CFG.img_size, CFG.img_size, scale=(0.9, 1), p=1), \nHorizontalFlip(p=0.5), ShiftScaleRotate(p=0.5),\nHueSaturationValue(hue_shift_limit=10, sat_shift_limit=10, val_shift_limit=10, p=0.7),\nRandomBrightnessContrast(brightness_limit=(-0.2,0.2), contrast_limit=(-0.2, 0.2), p=0.7),\nCLAHE(clip_limit=(1,4), p=0.5),\nOneOf([ \nOpticalDistortion(distort_limit=1.0), GridDistortion(num_steps=5, distort_limit=1.),\nElasticTransform(alpha=3), ], p=0.2),\nOneOf([ GaussNoise(var_limit=[10, 50]),\nGaussianBlur(),\nMotionBlur(),\nMedianBlur(), ], p=0.2),\nResize(CFG.img_size, CFG.img_size), OneOf([ JpegCompression(), Downscale(scale_min=0.1, scale_max=0.15), ], p=0.2), IAAPiecewiseAffine()\n```\n**Logging**: Neptune.ai\n\n### Things we didn't had time to do\n- Segmentation based learning ( that made us apart from other top candidates)\n- Retrieval based learning\n- GeM Pooling and AdaptiveConcatPool ( we prepared didnt experimented )\n### Things didn't worked \n- Multi Staged Ensembling ( because it was hurting too much )\n- Dynamic Temperature for pseudo labelling\n\nThanks to @philippsinger for his [writeup](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/220563) where I got how to write a solution writeup (since this is my First writeup)\n\nThanks to All we learned a lot Team **HotWater**", "1241471": "Congratulation guys good work ))", "1241473": "Thanks", "1241510": "Congratulations! Thank you for sharing your solution. Great work!", "1241524": "Congrats great work!! 🔥 @morizin \nI guess training all these model would have required a lot of hardware.", "1241536": "Sure. We had 8x V100 GPU(cloud) and 3x 3090 GPU", "1241537": "Thanks, also Congratulation on your achievement", "1241609": "Congrats on 16th place @morizin and team, good job", "1241616": "Thanks", "1241913": "Thank you for sharing. Great Work!\n\nI'm very happy to hear that my model is the your best single model 😎", "1241917": "Thanks for your MultiHead Attention\nAnd Congratulations on your score!", "1242028": "morizin Great work and Congratulations on Silver Finish", "1242684": "Congratz on the nice finish & thanks for the write-up !", "1242701": "I think it was a collabrative effort", "1243480": "Thank you", "1244572": "Sorry, I have a Change in the architecture\nThe Multi Head Attention were also 3 stages." }, "source": "meta" }