![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1796795%2Fb1fac286781793dc0ee4674cb04f3859%2F2%20(1).JPG?generation=1657688615932601&alt=media\")
Congratulations to the winners. It was an honor to compete with brilliant minds.
\nI'll first summarize the history of my public scores.
\npseudo-labeling->training as pseudo training): 0.972This is the model I used.
\n
Now I'll go into details of the 3 main problems that I faced and the solutions for each of them.
\nWe are given >2048x2048 resolution images, and we lose information when we downsample. So like many observed, increasing the resolution boosts the score quite much.
\nI wanted to use 2048x2048 resolution, but it was too large to fit the computer. So I used one convolutonal layer to downsample the image from 2048x2048 to 1024x1024, then used regular CNN. GPU memory usage was almost identical to when using plain 1024x1024 image as input. It was inspired by @ekydna 's method at https://www.kaggle.com/c/understanding_cloud_organization/discussion/118255
\nTo be precise, I used following pseudo code. I concatenated avgpool-ed input with downconv-ed input.
\ndef __init__():\n self.avgpool = nn.AvgPool2d(2)\n self.downconv = nn.Sequential(\n nn.Conv2d(1, 7, kernel_size=5, stride=2, padding=2, bias=False),\n nn.BatchNorm2d(7),\n nn.ReLU()\n )\ndef forward(x):\n x = torch.cat((self.avgpool(x), self.downconv(x)), dim=1)\n features = CNN(x)\nWe are given additional catheter position annotations for some of the image, so it was natural to find a way to leverage this information. Segmentation models came to mind first, so from the start, I processed the annotations to segmentation masks and used UNet architecture.
\nAt first, I tried to use multi-task learning, but I couldn't make it work. I suspect that it is difficult for the model to squeeze out the classification loss, when there is also segmentation loss.
\nThen, I tried pre-training approach which worked. I pretrained UNet with segmentation loss only, used pretrained UNet encoder to train for the classification. You need to increase pos weight for the segmentation bceloss to make it work.
\nAlso, I tried concatenating mask predictions to the image input, but couldn't make it work to increase CV.
\nThere are a lot of external datasets, like listed in https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873 , which we could use for the competition(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865). But the thing is, they are not labeled. So the question becomes how to utilize unlabeled data. I tried two approaches; pre-training and pseudo-training.
\nRecently, there are a lot of research on self-supervised learning to make use of large unlabeled data, especially contrastive learning in computer vision. I experimented with SWAV and SIMSIAM. I spent 2 weeks on these contrastive pretraining approach, but unfortunately it didn't boost the score.
\nOn the other hand, pseudo-training worked. I trained 5fold model with labeled dataset, used them to predict unlabeled dataset, selected images that have max prob > 0.5, appended it to the original dataset, and trained model.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1796795%2F67cd710a790111a4324a39672d2de966%2F3%20(1).JPG?generation=1657688656901232&alt=media\")
Hard augmentation prevents the model from overfitting. I used albumentations library for augmentations.
\ntransforms = albu.Compose([\n albu.RandomResizedCrop(cfg.resolution, cfg.resolution, scale=(0.9, 1), p=1),\n albu.OneOf([\n albu.MotionBlur(blur_limit=(3, 5)),\n albu.MedianBlur(blur_limit=5),\n albu.GaussianBlur(blur_limit=(3, 5)),\n albu.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n albu.OneOf([\n albu.OpticalDistortion(distort_limit=1.0),\n albu.GridDistortion(num_steps=5, distort_limit=1.),\n albu.ElasticTransform(alpha=3),\n ], p=0.7),\n albu.CLAHE(clip_limit=4.0, p=0.7),\n albu.IAAPiecewiseAffine(p=0.2),\n albu.IAASharpen(p=0.2),\n albu.RandomGamma(gamma_limit=(70, 130), p=0.3),\n albu.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.75),\n albu.OneOf([\n albu.ImageCompression(),\n albu.Downscale(scale_min=0.7, scale_max=0.95),\n ], p=0.2),\n albu.CoarseDropout(max_holes=8, max_height=int(cfg.resolution * 0.1),\n max_width=int(cfg.resolution * 0.1), p=0.5),\n albu.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n albu.Normalize(mean=0.482288, std=0.22085)\n])\nI didn't use HorizontalFlip since I thought the position of the catheter endpoint matters, and it didn't improve CV.
\nWhen only training with original dataset, 5fold ensemble boosted the score quite much. However, when training with pseudo labeled dataset, 5fold ensemble didn't boost the score much. I suspect the reason is; 1. with additional data, the data 5fold models see overlaps more 2. soft pseudo labels, which has more implicit informations than hard labels, forces the models to converge to certain point thus models lose diversity.
\nEnsembling different model architectures also didn't help much in pseudo training stage.
", "messages": [ { "id": "1241186", "postDate": "03/17/2021 00:01:52", "content": "Congratulations to the winners. It was an honor to compete with brilliant minds.
\nI'll first summarize the history of my public scores.
\npseudo-labeling->training as pseudo training): 0.972This is the model I used.
\n
Now I'll go into details of the 3 main problems that I faced and the solutions for each of them.
\nWe are given >2048x2048 resolution images, and we lose information when we downsample. So like many observed, increasing the resolution boosts the score quite much.
\nI wanted to use 2048x2048 resolution, but it was too large to fit the computer. So I used one convolutonal layer to downsample the image from 2048x2048 to 1024x1024, then used regular CNN. GPU memory usage was almost identical to when using plain 1024x1024 image as input. It was inspired by @ekydna 's method at https://www.kaggle.com/c/understanding_cloud_organization/discussion/118255
\nTo be precise, I used following pseudo code. I concatenated avgpool-ed input with downconv-ed input.
\ndef __init__():\n self.avgpool = nn.AvgPool2d(2)\n self.downconv = nn.Sequential(\n nn.Conv2d(1, 7, kernel_size=5, stride=2, padding=2, bias=False),\n nn.BatchNorm2d(7),\n nn.ReLU()\n )\ndef forward(x):\n x = torch.cat((self.avgpool(x), self.downconv(x)), dim=1)\n features = CNN(x)\nWe are given additional catheter position annotations for some of the image, so it was natural to find a way to leverage this information. Segmentation models came to mind first, so from the start, I processed the annotations to segmentation masks and used UNet architecture.
\nAt first, I tried to use multi-task learning, but I couldn't make it work. I suspect that it is difficult for the model to squeeze out the classification loss, when there is also segmentation loss.
\nThen, I tried pre-training approach which worked. I pretrained UNet with segmentation loss only, used pretrained UNet encoder to train for the classification. You need to increase pos weight for the segmentation bceloss to make it work.
\nAlso, I tried concatenating mask predictions to the image input, but couldn't make it work to increase CV.
\nThere are a lot of external datasets, like listed in https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873 , which we could use for the competition(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865). But the thing is, they are not labeled. So the question becomes how to utilize unlabeled data. I tried two approaches; pre-training and pseudo-training.
\nRecently, there are a lot of research on self-supervised learning to make use of large unlabeled data, especially contrastive learning in computer vision. I experimented with SWAV and SIMSIAM. I spent 2 weeks on these contrastive pretraining approach, but unfortunately it didn't boost the score.
\nOn the other hand, pseudo-training worked. I trained 5fold model with labeled dataset, used them to predict unlabeled dataset, selected images that have max prob > 0.5, appended it to the original dataset, and trained model.
\n
Hard augmentation prevents the model from overfitting. I used albumentations library for augmentations.
\ntransforms = albu.Compose([\n albu.RandomResizedCrop(cfg.resolution, cfg.resolution, scale=(0.9, 1), p=1),\n albu.OneOf([\n albu.MotionBlur(blur_limit=(3, 5)),\n albu.MedianBlur(blur_limit=5),\n albu.GaussianBlur(blur_limit=(3, 5)),\n albu.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n albu.OneOf([\n albu.OpticalDistortion(distort_limit=1.0),\n albu.GridDistortion(num_steps=5, distort_limit=1.),\n albu.ElasticTransform(alpha=3),\n ], p=0.7),\n albu.CLAHE(clip_limit=4.0, p=0.7),\n albu.IAAPiecewiseAffine(p=0.2),\n albu.IAASharpen(p=0.2),\n albu.RandomGamma(gamma_limit=(70, 130), p=0.3),\n albu.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.75),\n albu.OneOf([\n albu.ImageCompression(),\n albu.Downscale(scale_min=0.7, scale_max=0.95),\n ], p=0.2),\n albu.CoarseDropout(max_holes=8, max_height=int(cfg.resolution * 0.1),\n max_width=int(cfg.resolution * 0.1), p=0.5),\n albu.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n albu.Normalize(mean=0.482288, std=0.22085)\n])\nI didn't use HorizontalFlip since I thought the position of the catheter endpoint matters, and it didn't improve CV.
\nWhen only training with original dataset, 5fold ensemble boosted the score quite much. However, when training with pseudo labeled dataset, 5fold ensemble didn't boost the score much. I suspect the reason is; 1. with additional data, the data 5fold models see overlaps more 2. soft pseudo labels, which has more implicit informations than hard labels, forces the models to converge to certain point thus models lose diversity.
\nEnsembling different model architectures also didn't help much in pseudo training stage.
", "rawMarkdown": "Congratulations to the winners. It was an honor to compete with brilliant minds.\n\nI'll first summarize the history of my public scores.\n\n1. efficientnet-b5 baseline: **0.956**\n2. downconv: **0.959**\n3. segmentation pretrain: **0.963**\n4. other optimizations: **0.966**\n5. 5fold: **0.970**\n6. pseudo training (*For brevity, I'll call `pseudo-labeling->training` as `pseudo training`*): **0.972**\n7. 5fold & ensemble with b4, b6: **0.972**\n\nThis is the model I used.\n\n.JPG?generation=1657688615932601&alt=media)\n\nNow I'll go into details of the 3 main problems that I faced and the solutions for each of them.\n\n## 1. How to utilize high resolution? - Downconv\n\nWe are given >2048x2048 resolution images, and we lose information when we downsample. So like many observed, increasing the resolution boosts the score quite much. \n\nI wanted to use 2048x2048 resolution, but it was too large to fit the computer. So I used one convolutonal layer to downsample the image from 2048x2048 to 1024x1024, then used regular CNN. GPU memory usage was almost identical to when using plain 1024x1024 image as input. It was inspired by @ekydna 's method at https://www.kaggle.com/c/understanding_cloud_organization/discussion/118255\n\nTo be precise, I used following pseudo code. I concatenated avgpool-ed input with downconv-ed input.\n\n```python\ndef __init__():\n\tself.avgpool = nn.AvgPool2d(2)\n self.downconv = nn.Sequential(\n nn.Conv2d(1, 7, kernel_size=5, stride=2, padding=2, bias=False),\n nn.BatchNorm2d(7),\n nn.ReLU()\n )\ndef forward(x):\n\tx = torch.cat((self.avgpool(x), self.downconv(x)), dim=1)\n features = CNN(x)\n```\n\n## 2. How to utilize catheter position annotations? - Pre-training\n\nWe are given additional catheter position annotations for some of the image, so it was natural to find a way to leverage this information. Segmentation models came to mind first, so from the start, I processed the annotations to segmentation masks and used UNet architecture.\n\nAt first, I tried to use multi-task learning, but I couldn't make it work. I suspect that it is difficult for the model to squeeze out the classification loss, when there is also segmentation loss.\n\nThen, I tried pre-training approach which worked. I pretrained UNet with segmentation loss only, used pretrained UNet encoder to train for the classification. You need to increase pos weight for the segmentation bceloss to make it work.\n\nAlso, I tried concatenating mask predictions to the image input, but couldn't make it work to increase CV.\n\n## 3. How to utilize unlabeled data? - Pseudo-training\n\nThere are a lot of external datasets, like listed in https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873 , which we could use for the competition(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865). But the thing is, they are not labeled. So the question becomes how to utilize unlabeled data. I tried two approaches; pre-training and pseudo-training.\n\nRecently, there are a lot of research on self-supervised learning to make use of large unlabeled data, especially contrastive learning in computer vision. I experimented with SWAV and SIMSIAM. I spent 2 weeks on these contrastive pretraining approach, but unfortunately it didn't boost the score.\n\nOn the other hand, pseudo-training worked. I trained 5fold model with labeled dataset, used them to predict unlabeled dataset, selected images that have max prob > 0.5, appended it to the original dataset, and trained model.\n\n\n.JPG?generation=1657688656901232&alt=media)\n\n\n## Other points\n\n### Extensive augmentations\n\nHard augmentation prevents the model from overfitting. I used albumentations library for augmentations.\n\n```python\ntransforms = albu.Compose([\n albu.RandomResizedCrop(cfg.resolution, cfg.resolution, scale=(0.9, 1), p=1),\n albu.OneOf([\n albu.MotionBlur(blur_limit=(3, 5)),\n albu.MedianBlur(blur_limit=5),\n albu.GaussianBlur(blur_limit=(3, 5)),\n albu.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n albu.OneOf([\n albu.OpticalDistortion(distort_limit=1.0),\n albu.GridDistortion(num_steps=5, distort_limit=1.),\n albu.ElasticTransform(alpha=3),\n ], p=0.7),\n albu.CLAHE(clip_limit=4.0, p=0.7),\n albu.IAAPiecewiseAffine(p=0.2),\n albu.IAASharpen(p=0.2),\n albu.RandomGamma(gamma_limit=(70, 130), p=0.3),\n albu.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.75),\n albu.OneOf([\n albu.ImageCompression(),\n albu.Downscale(scale_min=0.7, scale_max=0.95),\n ], p=0.2),\n albu.CoarseDropout(max_holes=8, max_height=int(cfg.resolution * 0.1),\n max_width=int(cfg.resolution * 0.1), p=0.5),\n albu.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n albu.Normalize(mean=0.482288, std=0.22085)\n])\n```\n\nI didn't use HorizontalFlip since I thought the position of the catheter endpoint matters, and it didn't improve CV.\n\n### Ensemble\n\nWhen only training with original dataset, 5fold ensemble boosted the score quite much. However, when training with pseudo labeled dataset, 5fold ensemble didn't boost the score much. I suspect the reason is; 1. with additional data, the data 5fold models see overlaps more 2. soft pseudo labels, which has more implicit informations than hard labels, forces the models to converge to certain point thus models lose diversity.\n\nEnsembling different model architectures also didn't help much in pseudo training stage.", "votes": null }, { "id": "1241192", "postDate": "03/17/2021 00:06:42", "content": "Thanks for sharing and congrats!
", "rawMarkdown": "Thanks for sharing and congrats!", "votes": null }, { "id": "1241198", "postDate": "03/17/2021 00:10:53", "content": "Conratulation on Solo Gold :)
\n윤수님 보면서 많이 배우고 있습니다
\n축하드려요
Thanks a lot! 감사합니다😀
", "rawMarkdown": "Thanks a lot! 감사합니다😀", "votes": null }, { "id": "1241220", "postDate": "03/17/2021 00:35:11", "content": "Good job. Congrats on solo gold medal and 11th place @harangdev
", "rawMarkdown": "Good job. Congrats on solo gold medal and 11th place @harangdev", "votes": null }, { "id": "1241226", "postDate": "03/17/2021 00:40:14", "content": "Amazing win and with with impressive solution
", "rawMarkdown": "Amazing win and with with impressive solution", "votes": null }, { "id": "1241246", "postDate": "03/17/2021 00:54:49", "content": "Thanks. Congratulations for your impressive finish too
", "rawMarkdown": "Thanks. Congratulations for your impressive finish too", "votes": null }, { "id": "1241248", "postDate": "03/17/2021 00:55:57", "content": "Great writeup. I really like this part.
\n I concatenated avgpool-ed input with downconv-ed input.\nCongrats!! Using DownConv to utilize the full resolution image-info is impressive. I wonder about your GPU specs
", "rawMarkdown": "Congrats!! Using DownConv to utilize the full resolution image-info is impressive. I wonder about your GPU specs", "votes": null }, { "id": "1241267", "postDate": "03/17/2021 01:11:50", "content": "I had access to 4 x RTX3090, which I really appreciate ;)
", "rawMarkdown": "I had access to 4 x RTX3090, which I really appreciate ;)", "votes": null }, { "id": "1241298", "postDate": "03/17/2021 01:48:27", "content": "congrats @harangdev
", "rawMarkdown": "congrats @harangdev", "votes": null }, { "id": "1241307", "postDate": "03/17/2021 01:52:48", "content": "Thanks I forgot to upvote 😂😂
", "rawMarkdown": "Thanks I forgot to upvote 😂😂", "votes": null }, { "id": "1241317", "postDate": "03/17/2021 01:59:03", "content": "Congrats on solo gold medal and great write-up!
\n축하드립니다~ㅎㅎ @harangdev
Big congrats! 축하드려요! 혹시 pseudo training은 어떤 논문/discussion을 참고하셨는지 알려주실수 있나요?ㅎㅎ
", "rawMarkdown": "Big congrats! 축하드려요! 혹시 pseudo training은 어떤 논문/discussion을 참고하셨는지 알려주실수 있나요?ㅎㅎ", "votes": null }, { "id": "1241356", "postDate": "03/17/2021 02:30:23", "content": "Thanks ;) 감사합니다 ㅎㅎ
", "rawMarkdown": "Thanks ;) 감사합니다 ㅎㅎ", "votes": null }, { "id": "1241362", "postDate": "03/17/2021 02:34:07", "content": "@harangdev Congratulations on Solo Gold Finish . Highly impressive and detailed writeup . Thanks for sharing
", "rawMarkdown": "harangdev Congratulations on Solo Gold Finish . Highly impressive and detailed writeup . Thanks for sharing", "votes": null }, { "id": "1241453", "postDate": "03/17/2021 03:45:15", "content": "You asked for some references for pseudo training. Actually, if you search with the keyword pseudo labeling, you can find a lot of resources here in kaggle or at google. Previous SOTA of Imagenet, noisy-students (https://arxiv.org/abs/1911.04252) also uses pseudo labeling.
\n축하 감사합니다 :)
Congrats! Thanks for sharing well explained solution.
", "rawMarkdown": "Congrats! Thanks for sharing well explained solution.", "votes": null }, { "id": "1241480", "postDate": "03/17/2021 04:13:53", "content": "Thanks a lot! Rooting for your last gold for GM ;)
", "rawMarkdown": "Thanks a lot! Rooting for your last gold for GM ;)", "votes": null }, { "id": "1241499", "postDate": "03/17/2021 04:26:52", "content": "Congratulations. Great work
", "rawMarkdown": "Congratulations. Great work", "votes": null }, { "id": "1241555", "postDate": "03/17/2021 05:05:03", "content": "Congrats. Keep up the good work.
\nI asked many candidates that how not to use Resize function on an image and yet to capture all information in interviews. This is one of my favourite questions when it comes to taking an interview with the candidate. Yet, I could not think of applying this to this competition. What an irony. LOL
", "rawMarkdown": "Congrats. Keep up the good work.\n\nI asked many candidates that how not to use Resize function on an image and yet to capture all information in interviews. This is one of my favourite questions when it comes to taking an interview with the candidate. Yet, I could not think of applying this to this competition. What an irony. LOL", "votes": null }, { "id": "1241556", "postDate": "03/17/2021 05:11:56", "content": "Congrats on your solo gold medal and thanks for your sharing.
", "rawMarkdown": "Congrats on your solo gold medal and thanks for your sharing.", "votes": null }, { "id": "1241571", "postDate": "03/17/2021 05:23:12", "content": "Thank you for sharing your impressive solution. Congratulations!
", "rawMarkdown": "Thank you for sharing your impressive solution. Congratulations!", "votes": null }, { "id": "1241751", "postDate": "03/17/2021 07:28:34", "content": "Congratulations. Really impressive solution and lot of hard works. Thanks for sharing with us.
", "rawMarkdown": "Congratulations. Really impressive solution and lot of hard works. Thanks for sharing with us.", "votes": null }, { "id": "1241878", "postDate": "03/17/2021 08:59:27", "content": "Really nice solution ! I'll definitely re-use the downconv trick.
\nThanks for sharing and congratz on the solo gold !
Yeah, I probably wouldn't have thought of it too, if I didn't see Andrey Kiryasov's solution. Thanks!
", "rawMarkdown": "Yeah, I probably wouldn't have thought of it too, if I didn't see Andrey Kiryasov's solution. Thanks!", "votes": null }, { "id": "1241950", "postDate": "03/17/2021 09:53:29", "content": "Yeah, downconv is easily implemented, so I also think it doesn't hurt trying it. Thanks!
", "rawMarkdown": "Yeah, downconv is easily implemented, so I also think it doesn't hurt trying it. Thanks!", "votes": null }, { "id": "1241998", "postDate": "03/17/2021 10:32:24", "content": "Congrats, I'm just wondering from your write up it seems that input of UnetEncoder is (1024, 1024, 8) but didn't we need 3 channels for using ImageNet weights of EfficientNet ? @harangdev
You just need to replace first conv layer. For pytorch code, you can refer to https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/encoders/_utils.py#L5 .
", "rawMarkdown": "You just need to replace first conv layer. For pytorch code, you can refer to https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/encoders/_utils.py#L5 .", "votes": null }, { "id": "1242079", "postDate": "03/17/2021 11:44:46", "content": "Really great solution and thanks for the detailed write-up! Congrats!
", "rawMarkdown": "Really great solution and thanks for the detailed write-up! Congrats!", "votes": null }, { "id": "1242129", "postDate": "03/17/2021 12:23:09", "content": "@harangdev any idea how to do that on tensorflow ?
I don't know how to do it with tensorflow. Maybe there is such code at https://github.com/qubvel/segmentation_models
", "rawMarkdown": "I don't know how to do it with tensorflow. Maybe there is such code at https://github.com/qubvel/segmentation_models", "votes": null }, { "id": "1245123", "postDate": "03/19/2021 13:29:58", "content": "This paper that appeared on arXiv 2 days ago shares a lot of commonalities with the downsizing part of your solution 🙌 https://arxiv.org/abs/2103.09950
", "rawMarkdown": "This paper that appeared on arXiv 2 days ago shares a lot of commonalities with the downsizing part of your solution 🙌 https://arxiv.org/abs/2103.09950", "votes": null }, { "id": "1245220", "postDate": "03/19/2021 15:14:59", "content": "Actually I noticed this paper today, too. Planning on reading it :)
", "rawMarkdown": "Actually I noticed this paper today, too. Planning on reading it :)", "votes": null } ], "comments": [ { "id": 1241192, "author_name": "datafan07", "author_url": "", "post_date": "03/17/2021 00:06:42", "content": "Thanks for sharing and congrats!
", "votes": null, "replies": [] }, { "id": 1241198, "author_name": "deepbluebird", "author_url": "", "post_date": "03/17/2021 00:10:53", "content": "Conratulation on Solo Gold :)
\n윤수님 보면서 많이 배우고 있습니다
\n축하드려요
Thanks a lot! 감사합니다😀
", "votes": null, "replies": [] } ] }, { "id": 1241220, "author_name": "duykhanh99", "author_url": "", "post_date": "03/17/2021 00:35:11", "content": "Good job. Congrats on solo gold medal and 11th place @harangdev
", "votes": null, "replies": [] }, { "id": 1241226, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 00:40:14", "content": "Amazing win and with with impressive solution
", "votes": null, "replies": [ { "id": 1241246, "author_name": "harangdev", "author_url": "", "post_date": "03/17/2021 00:54:49", "content": "Thanks. Congratulations for your impressive finish too
", "votes": null, "replies": [] }, { "id": 1241307, "author_name": "morizin", "author_url": "", "post_date": "03/17/2021 01:52:48", "content": "Thanks I forgot to upvote 😂😂
", "votes": null, "replies": [] } ] }, { "id": 1241248, "author_name": "underwearfitting", "author_url": "", "post_date": "03/17/2021 00:55:57", "content": "Great writeup. I really like this part.
\n I concatenated avgpool-ed input with downconv-ed input.\nCongrats!! Using DownConv to utilize the full resolution image-info is impressive. I wonder about your GPU specs
", "votes": null, "replies": [ { "id": 1241267, "author_name": "harangdev", "author_url": "", "post_date": "03/17/2021 01:11:50", "content": "I had access to 4 x RTX3090, which I really appreciate ;)
", "votes": null, "replies": [] } ] }, { "id": 1241298, "author_name": "mylinh97", "author_url": "", "post_date": "03/17/2021 01:48:27", "content": "congrats @harangdev
", "votes": null, "replies": [] }, { "id": 1241317, "author_name": "piantic", "author_url": "", "post_date": "03/17/2021 01:59:03", "content": "Congrats on solo gold medal and great write-up!
\n축하드립니다~ㅎㅎ @harangdev
Thanks ;) 감사합니다 ㅎㅎ
", "votes": null, "replies": [] } ] }, { "id": 1241332, "author_name": "hihunjin", "author_url": "", "post_date": "03/17/2021 02:08:57", "content": "Big congrats! 축하드려요! 혹시 pseudo training은 어떤 논문/discussion을 참고하셨는지 알려주실수 있나요?ㅎㅎ
", "votes": null, "replies": [ { "id": 1241453, "author_name": "harangdev", "author_url": "", "post_date": "03/17/2021 03:45:15", "content": "You asked for some references for pseudo training. Actually, if you search with the keyword pseudo labeling, you can find a lot of resources here in kaggle or at google. Previous SOTA of Imagenet, noisy-students (https://arxiv.org/abs/1911.04252) also uses pseudo labeling.
\n축하 감사합니다 :)
@harangdev Congratulations on Solo Gold Finish . Highly impressive and detailed writeup . Thanks for sharing
", "votes": null, "replies": [] }, { "id": 1241472, "author_name": "songwonho", "author_url": "", "post_date": "03/17/2021 04:07:23", "content": "Congrats! Thanks for sharing well explained solution.
", "votes": null, "replies": [ { "id": 1241480, "author_name": "harangdev", "author_url": "", "post_date": "03/17/2021 04:13:53", "content": "Thanks a lot! Rooting for your last gold for GM ;)
", "votes": null, "replies": [] } ] }, { "id": 1241499, "author_name": "ammarali32", "author_url": "", "post_date": "03/17/2021 04:26:52", "content": "Congratulations. Great work
", "votes": null, "replies": [] }, { "id": 1241555, "author_name": "urvishp80", "author_url": "", "post_date": "03/17/2021 05:05:03", "content": "Congrats. Keep up the good work.
\nI asked many candidates that how not to use Resize function on an image and yet to capture all information in interviews. This is one of my favourite questions when it comes to taking an interview with the candidate. Yet, I could not think of applying this to this competition. What an irony. LOL
", "votes": null, "replies": [ { "id": 1241944, "author_name": "harangdev", "author_url": "", "post_date": "03/17/2021 09:50:04", "content": "Yeah, I probably wouldn't have thought of it too, if I didn't see Andrey Kiryasov's solution. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 1241556, "author_name": "chenlongwang", "author_url": "", "post_date": "03/17/2021 05:11:56", "content": "Congrats on your solo gold medal and thanks for your sharing.
", "votes": null, "replies": [] }, { "id": 1241571, "author_name": "yoshitaka1105", "author_url": "", "post_date": "03/17/2021 05:23:12", "content": "Thank you for sharing your impressive solution. Congratulations!
", "votes": null, "replies": [] }, { "id": 1241751, "author_name": "durbin164", "author_url": "", "post_date": "03/17/2021 07:28:34", "content": "Congratulations. Really impressive solution and lot of hard works. Thanks for sharing with us.
", "votes": null, "replies": [] }, { "id": 1241878, "author_name": "theoviel", "author_url": "", "post_date": "03/17/2021 08:59:27", "content": "Really nice solution ! I'll definitely re-use the downconv trick.
\nThanks for sharing and congratz on the solo gold !
Yeah, downconv is easily implemented, so I also think it doesn't hurt trying it. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 1241998, "author_name": "awsaf49", "author_url": "", "post_date": "03/17/2021 10:32:24", "content": "Congrats, I'm just wondering from your write up it seems that input of UnetEncoder is (1024, 1024, 8) but didn't we need 3 channels for using ImageNet weights of EfficientNet ? @harangdev
You just need to replace first conv layer. For pytorch code, you can refer to https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/encoders/_utils.py#L5 .
", "votes": null, "replies": [] }, { "id": 1242129, "author_name": "awsaf49", "author_url": "", "post_date": "03/17/2021 12:23:09", "content": "@harangdev any idea how to do that on tensorflow ?
I don't know how to do it with tensorflow. Maybe there is such code at https://github.com/qubvel/segmentation_models
", "votes": null, "replies": [] } ] }, { "id": 1242079, "author_name": "anjum48", "author_url": "", "post_date": "03/17/2021 11:44:46", "content": "Really great solution and thanks for the detailed write-up! Congrats!
", "votes": null, "replies": [] }, { "id": 1245123, "author_name": "anjum48", "author_url": "", "post_date": "03/19/2021 13:29:58", "content": "This paper that appeared on arXiv 2 days ago shares a lot of commonalities with the downsizing part of your solution 🙌 https://arxiv.org/abs/2103.09950
", "votes": null, "replies": [ { "id": 1245220, "author_name": "harangdev", "author_url": "", "post_date": "03/19/2021 15:14:59", "content": "Actually I noticed this paper today, too. Planning on reading it :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1241186": "Congratulations to the winners. It was an honor to compete with brilliant minds.\n\nI'll first summarize the history of my public scores.\n\n1. efficientnet-b5 baseline: **0.956**\n2. downconv: **0.959**\n3. segmentation pretrain: **0.963**\n4. other optimizations: **0.966**\n5. 5fold: **0.970**\n6. pseudo training (*For brevity, I'll call `pseudo-labeling->training` as `pseudo training`*): **0.972**\n7. 5fold & ensemble with b4, b6: **0.972**\n\nThis is the model I used.\n\n.JPG?generation=1657688615932601&alt=media)\n\nNow I'll go into details of the 3 main problems that I faced and the solutions for each of them.\n\n## 1. How to utilize high resolution? - Downconv\n\nWe are given >2048x2048 resolution images, and we lose information when we downsample. So like many observed, increasing the resolution boosts the score quite much. \n\nI wanted to use 2048x2048 resolution, but it was too large to fit the computer. So I used one convolutonal layer to downsample the image from 2048x2048 to 1024x1024, then used regular CNN. GPU memory usage was almost identical to when using plain 1024x1024 image as input. It was inspired by @ekydna 's method at https://www.kaggle.com/c/understanding_cloud_organization/discussion/118255\n\nTo be precise, I used following pseudo code. I concatenated avgpool-ed input with downconv-ed input.\n\n```python\ndef __init__():\n\tself.avgpool = nn.AvgPool2d(2)\n self.downconv = nn.Sequential(\n nn.Conv2d(1, 7, kernel_size=5, stride=2, padding=2, bias=False),\n nn.BatchNorm2d(7),\n nn.ReLU()\n )\ndef forward(x):\n\tx = torch.cat((self.avgpool(x), self.downconv(x)), dim=1)\n features = CNN(x)\n```\n\n## 2. How to utilize catheter position annotations? - Pre-training\n\nWe are given additional catheter position annotations for some of the image, so it was natural to find a way to leverage this information. Segmentation models came to mind first, so from the start, I processed the annotations to segmentation masks and used UNet architecture.\n\nAt first, I tried to use multi-task learning, but I couldn't make it work. I suspect that it is difficult for the model to squeeze out the classification loss, when there is also segmentation loss.\n\nThen, I tried pre-training approach which worked. I pretrained UNet with segmentation loss only, used pretrained UNet encoder to train for the classification. You need to increase pos weight for the segmentation bceloss to make it work.\n\nAlso, I tried concatenating mask predictions to the image input, but couldn't make it work to increase CV.\n\n## 3. How to utilize unlabeled data? - Pseudo-training\n\nThere are a lot of external datasets, like listed in https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/220873 , which we could use for the competition(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644#1231865). But the thing is, they are not labeled. So the question becomes how to utilize unlabeled data. I tried two approaches; pre-training and pseudo-training.\n\nRecently, there are a lot of research on self-supervised learning to make use of large unlabeled data, especially contrastive learning in computer vision. I experimented with SWAV and SIMSIAM. I spent 2 weeks on these contrastive pretraining approach, but unfortunately it didn't boost the score.\n\nOn the other hand, pseudo-training worked. I trained 5fold model with labeled dataset, used them to predict unlabeled dataset, selected images that have max prob > 0.5, appended it to the original dataset, and trained model.\n\n\n.JPG?generation=1657688656901232&alt=media)\n\n\n## Other points\n\n### Extensive augmentations\n\nHard augmentation prevents the model from overfitting. I used albumentations library for augmentations.\n\n```python\ntransforms = albu.Compose([\n albu.RandomResizedCrop(cfg.resolution, cfg.resolution, scale=(0.9, 1), p=1),\n albu.OneOf([\n albu.MotionBlur(blur_limit=(3, 5)),\n albu.MedianBlur(blur_limit=5),\n albu.GaussianBlur(blur_limit=(3, 5)),\n albu.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n albu.OneOf([\n albu.OpticalDistortion(distort_limit=1.0),\n albu.GridDistortion(num_steps=5, distort_limit=1.),\n albu.ElasticTransform(alpha=3),\n ], p=0.7),\n albu.CLAHE(clip_limit=4.0, p=0.7),\n albu.IAAPiecewiseAffine(p=0.2),\n albu.IAASharpen(p=0.2),\n albu.RandomGamma(gamma_limit=(70, 130), p=0.3),\n albu.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.75),\n albu.OneOf([\n albu.ImageCompression(),\n albu.Downscale(scale_min=0.7, scale_max=0.95),\n ], p=0.2),\n albu.CoarseDropout(max_holes=8, max_height=int(cfg.resolution * 0.1),\n max_width=int(cfg.resolution * 0.1), p=0.5),\n albu.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n albu.Normalize(mean=0.482288, std=0.22085)\n])\n```\n\nI didn't use HorizontalFlip since I thought the position of the catheter endpoint matters, and it didn't improve CV.\n\n### Ensemble\n\nWhen only training with original dataset, 5fold ensemble boosted the score quite much. However, when training with pseudo labeled dataset, 5fold ensemble didn't boost the score much. I suspect the reason is; 1. with additional data, the data 5fold models see overlaps more 2. soft pseudo labels, which has more implicit informations than hard labels, forces the models to converge to certain point thus models lose diversity.\n\nEnsembling different model architectures also didn't help much in pseudo training stage.", "1241192": "Thanks for sharing and congrats!", "1241198": "Conratulation on Solo Gold :)\n윤수님 보면서 많이 배우고 있습니다\n축하드려요", "1241216": "Thanks a lot! 감사합니다😀", "1241220": "Good job. Congrats on solo gold medal and 11th place @harangdev", "1241226": "Amazing win and with with impressive solution", "1241246": "Thanks. Congratulations for your impressive finish too", "1241248": "Great writeup. I really like this part.\n```\n I concatenated avgpool-ed input with downconv-ed input.\n```", "1241250": "Congrats!! Using DownConv to utilize the full resolution image-info is impressive. I wonder about your GPU specs", "1241267": "I had access to 4 x RTX3090, which I really appreciate ;)", "1241298": "congrats @harangdev", "1241307": "Thanks I forgot to upvote 😂😂", "1241317": "Congrats on solo gold medal and great write-up!\n축하드립니다~ㅎㅎ @harangdev", "1241332": "Big congrats! 축하드려요! 혹시 pseudo training은 어떤 논문/discussion을 참고하셨는지 알려주실수 있나요?ㅎㅎ", "1241356": "Thanks ;) 감사합니다 ㅎㅎ", "1241362": "harangdev Congratulations on Solo Gold Finish . Highly impressive and detailed writeup . Thanks for sharing", "1241453": "You asked for some references for pseudo training. Actually, if you search with the keyword `pseudo labeling`, you can find a lot of resources here in kaggle or at google. Previous SOTA of Imagenet, noisy-students (https://arxiv.org/abs/1911.04252) also uses pseudo labeling.\n축하 감사합니다 :)", "1241472": "Congrats! Thanks for sharing well explained solution.", "1241480": "Thanks a lot! Rooting for your last gold for GM ;)", "1241499": "Congratulations. Great work", "1241555": "Congrats. Keep up the good work.\n\nI asked many candidates that how not to use Resize function on an image and yet to capture all information in interviews. This is one of my favourite questions when it comes to taking an interview with the candidate. Yet, I could not think of applying this to this competition. What an irony. LOL", "1241556": "Congrats on your solo gold medal and thanks for your sharing.", "1241571": "Thank you for sharing your impressive solution. Congratulations!", "1241751": "Congratulations. Really impressive solution and lot of hard works. Thanks for sharing with us.", "1241878": "Really nice solution ! I'll definitely re-use the downconv trick.\nThanks for sharing and congratz on the solo gold !", "1241944": "Yeah, I probably wouldn't have thought of it too, if I didn't see Andrey Kiryasov's solution. Thanks!", "1241950": "Yeah, downconv is easily implemented, so I also think it doesn't hurt trying it. Thanks!", "1241998": "Congrats, I'm just wondering from your write up it seems that input of **UnetEncoder** is `(1024, 1024, 8)` but didn't we need `3 channels` for using `ImageNet` weights of **EfficientNet** ? @harangdev", "1242010": "You just need to replace first conv layer. For pytorch code, you can refer to https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/encoders/_utils.py#L5 .", "1242079": "Really great solution and thanks for the detailed write-up! Congrats!", "1242129": "harangdev any idea how to do that on `tensorflow` ?", "1242158": "I don't know how to do it with tensorflow. Maybe there is such code at https://github.com/qubvel/segmentation_models", "1245123": "This paper that appeared on arXiv 2 days ago shares a lot of commonalities with the downsizing part of your solution 🙌 https://arxiv.org/abs/2103.09950", "1245220": "Actually I noticed this paper today, too. Planning on reading it :)" }, "source": "meta" }