First of all, I would like to thank you for organizing this very interesting competition.
\nAlso, I can't thank IanPan enough. After we teamed up, our score improved very quickly, as if by magic. I have learned a lot from him through this competition.
The following is my part.
\n・Swin Transformer
\nIn my environment, Swin Transformer was the best. cait was not bad, but aux loss did not work well with cait, so I chose Swin Transformer.
\nI left the resolution at 384. I tried 768, but it was slightly worse. I also did some cropping, so 384 seemed to be sufficient.
・Pre training was done with chexpert data.
\nThis is Ian's idea. The data is quite different from ImageNet. we can get a good initial set of weights by pre training.
\nIn addition, further pre-training with the RSNA 2018 competition data before training with chexpert resulted in a slight improvement in CV over using chexpert only.
\nI used crop, which I will write about later, and did not use aux loss.
・5classes
\nI included 'none' in the class.
\n・Cropping with detection and segmentation
\nBy using the predictions of the detection for cropping, I was able to apply the wisdom of the detection model to the classification model, and I also used the lung Segmentator trained by VinBigData datasets. I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.
\nIf the box is used as-is, for example, a pleural effusion may result in an image that is difficult to distinguish, The box has been padded with 100 pixels.
\n・mixup
\nMixup brought about a small improvement. In the end, for my part, I used two different models, one with Mixup and one without.
\nAt the stage of not using aux loss, resize mix was the best, but in the end, mixup was better.
\n・aux loss
\nI used binary_cross_entropy_with_logits.
\nIn one of the two models, I set the negative image loss to zero.
\n・No pseudo label
\nI tried everything, but the results were slightly worse.
\n・The average of the 5fold model for the four classes of mAP was 0.397.
・YOLOV5
\nI used l6&l.
・Pre training was done with RSNA 2018 data.
\nThe same reason as for study level.
\n・mixup: 0.5
\nThere was a slight improvement.
\n・Use none predictions
\nFor each image, I found the following improved the score the most
\ndection_conf = dection_conf * (1-image none prediction)**0.4
I used predictions of classification models and detection models like this.
\nnone_pred = cl_none_pred0.7 + (1 - image_conf_max)0.3
The following is a message from Ian.
\nI would like to thank Yuji for agreeing to team with me. I reached out later in the competition when he was already doing really well without me. I learned a lot from him, and this competition reminded me how fun and enlightening it is to team up on Kaggle.
\nFor classification, I trained a 5-class classifier: 4 study labels + none at image-level (vs. opacity). We only trained on 1 image/study. Since the vast majority of studies only had a single unique image, this was not really an issue.
\nFirst, I pretrained the classifier on the 3 classes from the RSNA 2018 Pneumonia Detection Challenge. Then, I fine-tuned from these weights on this competition's data. This significantly increased performance.
\nI trained a hybrid classification-segmentation model. One architecture used EfficientNet-B6 and DeepLabV3+ while the other used Swin Transformer and FPN. These were trained using the awesome Segmentation Models PyTorch library (https://github.com/qubvel/segmentation_models.pytorch).
\nThe models were trained using AdamW optimizer, 512x512 images, and data augmentation with contrast/brightness adjustment, random noise, and scale/shift/rotate using albumentations.
\nI used BCE loss for both classification (even though you could use cross-entropy, I found BCE worked better for the metric) and segmentation, weighted 1:1. The segmentation part of the model only had 1 class for opacity. The segmentation label was generated using the bounding box labels. I turned the bounding box into an ellipse and used that as the training label.
\nI also used mixup, which showed significant improvements. 5-fold ensemble of the EfficientNet-B6/DeepLabV3+ and Swin/FPN models (10 models total) gave 0.408 LB for study-label classification.
\nI trained detection models using the awesome EfficientDet library (https://github.com/rwightman/efficientdet-pytorch) and MMDetection (https://github.com/open-mmlab/mmdetection).
\nI also pretrained these models on the RSNA 2018 dataset (images with opacity only) using the bounding box labels. This significantly increased performance for detection as well. I trained 5-fold EfficientDet-D7X on 512x512 images (positives only). I also trained 5-fold EfficientDet-D6 on 640x640 images (positives only).
\nTo add diversity to the ensemble, I trained models in MMDetection as well. I used Swin Transformer with RepPoints, borrowing the code from https://github.com/SwinTransformer/Swin-Transformer-Object-Detection to use Swin Transformer as a backbone for object detection. These models were trained using multi-scale training (512x512 to 768x768) and during inference 3x multi-scale TTA was used.
\nI contributed 15 detection models (EffDet-D7X, EffDet-D6, Swin-RepPoints) for the ensemble.
\nThe following is the part of the ensemble.
\nIn the end, we used a simple averaging.
\nI tried Nelder-Mead, ridge regression using predictions from detection models, etc., but all of them did not work because they had overfiit.
\nBy averaging the predictions of my two models with those of Ian's two models, we made significant improvements in both CV/LB. Since we both developed our models using each other's pipelines, there was a lot of diversity in the models, and by mixing our predictions at the end, the final predictions were more robust.
\nOur final study level LB was about 0.413.
I mixed my yolov5 with Ian's effdet and mmdet Swin Transformer models in wbf. I saw a big improvement. I used the same weights for all the models as I did not see any improvement with different weights. Also, iou threshold 0.6 was the best.
\ntrain: https://github.com/yujiariyasu/siim_covid19_detection
\ninference: https://www.kaggle.com/yujiariyasu/fork-of-siim-covid-19-full-pipeline-v2
I would like to thank all the participants for their hard work in this competition!
", "messages": [ { "id": "1462646", "postDate": "08/10/2021 00:29:21", "content": "First of all, I would like to thank you for organizing this very interesting competition.
\nAlso, I can't thank IanPan enough. After we teamed up, our score improved very quickly, as if by magic. I have learned a lot from him through this competition.
The following is my part.
\n・Swin Transformer
\nIn my environment, Swin Transformer was the best. cait was not bad, but aux loss did not work well with cait, so I chose Swin Transformer.
\nI left the resolution at 384. I tried 768, but it was slightly worse. I also did some cropping, so 384 seemed to be sufficient.
・Pre training was done with chexpert data.
\nThis is Ian's idea. The data is quite different from ImageNet. we can get a good initial set of weights by pre training.
\nIn addition, further pre-training with the RSNA 2018 competition data before training with chexpert resulted in a slight improvement in CV over using chexpert only.
\nI used crop, which I will write about later, and did not use aux loss.
・5classes
\nI included 'none' in the class.
\n・Cropping with detection and segmentation
\nBy using the predictions of the detection for cropping, I was able to apply the wisdom of the detection model to the classification model, and I also used the lung Segmentator trained by VinBigData datasets. I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.
\nIf the box is used as-is, for example, a pleural effusion may result in an image that is difficult to distinguish, The box has been padded with 100 pixels.
\n・mixup
\nMixup brought about a small improvement. In the end, for my part, I used two different models, one with Mixup and one without.
\nAt the stage of not using aux loss, resize mix was the best, but in the end, mixup was better.
\n・aux loss
\nI used binary_cross_entropy_with_logits.
\nIn one of the two models, I set the negative image loss to zero.
\n・No pseudo label
\nI tried everything, but the results were slightly worse.
\n・The average of the 5fold model for the four classes of mAP was 0.397.
・YOLOV5
\nI used l6&l.
・Pre training was done with RSNA 2018 data.
\nThe same reason as for study level.
\n・mixup: 0.5
\nThere was a slight improvement.
\n・Use none predictions
\nFor each image, I found the following improved the score the most
\ndection_conf = dection_conf * (1-image none prediction)**0.4
I used predictions of classification models and detection models like this.
\nnone_pred = cl_none_pred0.7 + (1 - image_conf_max)0.3
The following is a message from Ian.
\nI would like to thank Yuji for agreeing to team with me. I reached out later in the competition when he was already doing really well without me. I learned a lot from him, and this competition reminded me how fun and enlightening it is to team up on Kaggle.
\nFor classification, I trained a 5-class classifier: 4 study labels + none at image-level (vs. opacity). We only trained on 1 image/study. Since the vast majority of studies only had a single unique image, this was not really an issue.
\nFirst, I pretrained the classifier on the 3 classes from the RSNA 2018 Pneumonia Detection Challenge. Then, I fine-tuned from these weights on this competition's data. This significantly increased performance.
\nI trained a hybrid classification-segmentation model. One architecture used EfficientNet-B6 and DeepLabV3+ while the other used Swin Transformer and FPN. These were trained using the awesome Segmentation Models PyTorch library (https://github.com/qubvel/segmentation_models.pytorch).
\nThe models were trained using AdamW optimizer, 512x512 images, and data augmentation with contrast/brightness adjustment, random noise, and scale/shift/rotate using albumentations.
\nI used BCE loss for both classification (even though you could use cross-entropy, I found BCE worked better for the metric) and segmentation, weighted 1:1. The segmentation part of the model only had 1 class for opacity. The segmentation label was generated using the bounding box labels. I turned the bounding box into an ellipse and used that as the training label.
\nI also used mixup, which showed significant improvements. 5-fold ensemble of the EfficientNet-B6/DeepLabV3+ and Swin/FPN models (10 models total) gave 0.408 LB for study-label classification.
\nI trained detection models using the awesome EfficientDet library (https://github.com/rwightman/efficientdet-pytorch) and MMDetection (https://github.com/open-mmlab/mmdetection).
\nI also pretrained these models on the RSNA 2018 dataset (images with opacity only) using the bounding box labels. This significantly increased performance for detection as well. I trained 5-fold EfficientDet-D7X on 512x512 images (positives only). I also trained 5-fold EfficientDet-D6 on 640x640 images (positives only).
\nTo add diversity to the ensemble, I trained models in MMDetection as well. I used Swin Transformer with RepPoints, borrowing the code from https://github.com/SwinTransformer/Swin-Transformer-Object-Detection to use Swin Transformer as a backbone for object detection. These models were trained using multi-scale training (512x512 to 768x768) and during inference 3x multi-scale TTA was used.
\nI contributed 15 detection models (EffDet-D7X, EffDet-D6, Swin-RepPoints) for the ensemble.
\nThe following is the part of the ensemble.
\nIn the end, we used a simple averaging.
\nI tried Nelder-Mead, ridge regression using predictions from detection models, etc., but all of them did not work because they had overfiit.
\nBy averaging the predictions of my two models with those of Ian's two models, we made significant improvements in both CV/LB. Since we both developed our models using each other's pipelines, there was a lot of diversity in the models, and by mixing our predictions at the end, the final predictions were more robust.
\nOur final study level LB was about 0.413.
I mixed my yolov5 with Ian's effdet and mmdet Swin Transformer models in wbf. I saw a big improvement. I used the same weights for all the models as I did not see any improvement with different weights. Also, iou threshold 0.6 was the best.
\ntrain: https://github.com/yujiariyasu/siim_covid19_detection
\ninference: https://www.kaggle.com/yujiariyasu/fork-of-siim-covid-19-full-pipeline-v2
I would like to thank all the participants for their hard work in this competition!
", "rawMarkdown": "First of all, I would like to thank you for organizing this very interesting competition.\nAlso, I can't thank IanPan enough. After we teamed up, our score improved very quickly, as if by magic. I have learned a lot from him through this competition.\n\nThe following is my part.\n\n---\n\n### study level:\n・Swin Transformer \nIn my environment, Swin Transformer was the best. cait was not bad, but aux loss did not work well with cait, so I chose Swin Transformer.\nI left the resolution at 384. I tried 768, but it was slightly worse. I also did some cropping, so 384 seemed to be sufficient.\n\n##### 1st stage:\n・Pre training was done with chexpert data.\nThis is Ian's idea. The data is quite different from ImageNet. we can get a good initial set of weights by pre training.\nIn addition, further pre-training with the RSNA 2018 competition data before training with chexpert resulted in a slight improvement in CV over using chexpert only.\nI used crop, which I will write about later, and did not use aux loss.\n\n##### 2nd stage:\n・5classes\nI included 'none' in the class.\n・Cropping with detection and segmentation\nBy using the predictions of the detection for cropping, I was able to apply the wisdom of the detection model to the classification model, and I also used the lung Segmentator trained by VinBigData datasets. I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.\nIf the box is used as-is, for example, a pleural effusion may result in an image that is difficult to distinguish, The box has been padded with 100 pixels.\n・mixup\nMixup brought about a small improvement. In the end, for my part, I used two different models, one with Mixup and one without.\nAt the stage of not using aux loss, resize mix was the best, but in the end, mixup was better.\n・aux loss\nI used binary_cross_entropy_with_logits.\nIn one of the two models, I set the negative image loss to zero.\n・No pseudo label\nI tried everything, but the results were slightly worse.\n・The average of the 5fold model for the four classes of mAP was 0.397.\n\n### image level\n・YOLOV5\nI used l6&l.\n##### opacity:\n・Pre training was done with RSNA 2018 data.\nThe same reason as for study level.\n・mixup: 0.5\nThere was a slight improvement.\n・Use none predictions\nFor each image, I found the following improved the score the most\ndection_conf = dection_conf * (1-image none prediction)**0.4\n\n##### none:\nI used predictions of classification models and detection models like this.\nnone_pred = cl_none_pred*0.7 + (1 - image_conf_max)*0.3\n\n---\n\nThe following is a message from Ian.\n\n---\nI would like to thank Yuji for agreeing to team with me. I reached out later in the competition when he was already doing really well without me. I learned a lot from him, and this competition reminded me how fun and enlightening it is to team up on Kaggle. \n\n### Classification\n\nFor classification, I trained a 5-class classifier: 4 study labels + none at image-level (vs. opacity). We only trained on 1 image/study. Since the vast majority of studies only had a single unique image, this was not really an issue. \n\nFirst, I pretrained the classifier on the 3 classes from the RSNA 2018 Pneumonia Detection Challenge. Then, I fine-tuned from these weights on this competition's data. This significantly increased performance. \n\nI trained a hybrid classification-segmentation model. One architecture used EfficientNet-B6 and DeepLabV3+ while the other used Swin Transformer and FPN. These were trained using the awesome Segmentation Models PyTorch library (https://github.com/qubvel/segmentation_models.pytorch). \n\nThe models were trained using AdamW optimizer, 512x512 images, and data augmentation with contrast/brightness adjustment, random noise, and scale/shift/rotate using albumentations. \n\nI used BCE loss for both classification (even though you could use cross-entropy, I found BCE worked better for the metric) and segmentation, weighted 1:1. The segmentation part of the model only had 1 class for opacity. The segmentation label was generated using the bounding box labels. I turned the bounding box into an ellipse and used that as the training label. \n\nI also used mixup, which showed significant improvements. 5-fold ensemble of the EfficientNet-B6/DeepLabV3+ and Swin/FPN models (10 models total) gave 0.408 LB for study-label classification. \n\n### Detection \n\nI trained detection models using the awesome EfficientDet library (https://github.com/rwightman/efficientdet-pytorch) and MMDetection (https://github.com/open-mmlab/mmdetection). \n\nI also pretrained these models on the RSNA 2018 dataset (images with opacity only) using the bounding box labels. This significantly increased performance for detection as well. I trained 5-fold EfficientDet-D7X on 512x512 images (positives only). I also trained 5-fold EfficientDet-D6 on 640x640 images (positives only). \n\nTo add diversity to the ensemble, I trained models in MMDetection as well. I used Swin Transformer with RepPoints, borrowing the code from https://github.com/SwinTransformer/Swin-Transformer-Object-Detection to use Swin Transformer as a backbone for object detection. These models were trained using multi-scale training (512x512 to 768x768) and during inference 3x multi-scale TTA was used. \n\nI contributed 15 detection models (EffDet-D7X, EffDet-D6, Swin-RepPoints) for the ensemble. \n\n### What Didn't Work For Me\n- Training classifier with detection as auxiliary loss instead of augmentation\n- Mixup during detection\n- Training a multiclass detector (using study labels, instead of just training on single-class opacity)\n- In my experiments, pseudolabeling did not improve CV\n- Training a multiclass segmentation model and using that to help refine study-label predictions\n- Incorporating bounding box predictions into input for classification models\n\n---\n\nThe following is the part of the ensemble.\n\n### study level \nIn the end, we used a simple averaging.\nI tried Nelder-Mead, ridge regression using predictions from detection models, etc., but all of them did not work because they had overfiit.\nBy averaging the predictions of my two models with those of Ian's two models, we made significant improvements in both CV/LB. Since we both developed our models using each other's pipelines, there was a lot of diversity in the models, and by mixing our predictions at the end, the final predictions were more robust.\nOur final study level LB was about 0.413.\n\n### image level\nI mixed my yolov5 with Ian's effdet and mmdet Swin Transformer models in wbf. I saw a big improvement. I used the same weights for all the models as I did not see any improvement with different weights. Also, iou threshold 0.6 was the best.\n\n###code\ntrain: https://github.com/yujiariyasu/siim_covid19_detection\ninference: https://www.kaggle.com/yujiariyasu/fork-of-siim-covid-19-full-pipeline-v2\n \nI would like to thank all the participants for their hard work in this competition!", "votes": null }, { "id": "1462649", "postDate": "08/10/2021 00:36:33", "content": "Congratulations！🎉
", "rawMarkdown": "Congratulations！🎉", "votes": null }, { "id": "1462654", "postDate": "08/10/2021 00:40:31", "content": "Congrats on getting gold 👍🏻
", "rawMarkdown": "Congrats on getting gold 👍🏻", "votes": null }, { "id": "1462655", "postDate": "08/10/2021 00:40:58", "content": "Congrats! May I ask your detection score separately without study results and how much it improved by ensemble. Thanks!👀
", "rawMarkdown": "Congrats! May I ask your detection score separately without study results and how much it improved by ensemble. Thanks!👀", "votes": null }, { "id": "1462658", "postDate": "08/10/2021 00:44:25", "content": "Amazing solution, congratz on the results guys!
", "rawMarkdown": "Amazing solution, congratz on the results guys!", "votes": null }, { "id": "1462659", "postDate": "08/10/2021 00:44:58", "content": "congrats and thanks for the nice write-up!
\ncan I confirm the following?
\nfinal public submission 0.650, which break down to
\nstudy: 0.413.
\nimage: 0.237
do you have break down of:
\nCongratulations！
", "rawMarkdown": "Congratulations！", "votes": null }, { "id": "1462673", "postDate": "08/10/2021 00:56:46", "content": "Thanks! In fact, at the end of the competition, we didn't submit study level only. But I think it's probably 0.413. Maybe it's 0.414. We don't know much about LB since we hardly paid attention to it after working with Ian in the first place.
", "rawMarkdown": "Thanks! In fact, at the end of the competition, we didn't submit study level only. But I think it's probably 0.413. Maybe it's 0.414. We don't know much about LB since we hardly paid attention to it after working with Ian in the first place.", "votes": null }, { "id": "1462675", "postDate": "08/10/2021 00:57:47", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "1462676", "postDate": "08/10/2021 00:58:14", "content": "So do you! Congratulations!
", "rawMarkdown": "So do you! Congratulations!", "votes": null }, { "id": "1462681", "postDate": "08/10/2021 01:00:29", "content": "Thanks @yujiariyasu for the awesome write-up- I had a great time working with you!
", "rawMarkdown": "Thanks @yujiariyasu for the awesome write-up- I had a great time working with you!", "votes": null }, { "id": "1462689", "postDate": "08/10/2021 01:05:23", "content": "CV opacity ap was 0.5623. single best was 0.52~0.53. I'm not sure how much lb.
", "rawMarkdown": "CV opacity ap was 0.5623. single best was 0.52~0.53. I'm not sure how much lb.", "votes": null }, { "id": "1462690", "postDate": "08/10/2021 01:05:50", "content": "Thanks! Ian was amazing!
", "rawMarkdown": "Thanks! Ian was amazing!", "votes": null }, { "id": "1462691", "postDate": "08/10/2021 01:06:00", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "1462692", "postDate": "08/10/2021 01:06:52", "content": "Thanks to you too! I'll try my best to be like you one day:)
", "rawMarkdown": "Thanks to you too! I'll try my best to be like you one day:)", "votes": null }, { "id": "1463622", "postDate": "08/10/2021 09:03:38", "content": "Nice work for both of you. There are some things that I am curious about:
\nWhat is the goal of \"I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.\" Do you use cropping to preprocess image before study level prediction or to create additional training data? (Yuji part)
congratulations @yujiariyasu
", "rawMarkdown": "congratulations @yujiariyasu", "votes": null }, { "id": "1464010", "postDate": "08/10/2021 12:20:29", "content": "Congrats 🎉 @yujiariyasu & @vaillant. You guys did great.
\nBtw will you guys be nominating yourselves for the Student Prize ?
I used cropping to preprocess the images before making study level predictions.
\nthe prediction results of detection will change the crop, so the wisdom of the detection model can be conveyed to the classification model.
thank you!
", "rawMarkdown": "thank you!", "votes": null }, { "id": "1464341", "postDate": "08/10/2021 14:38:42", "content": "@vaillant is a doctor already as far as I remember, so I think your team won the prize ^^
", "rawMarkdown": "vaillant is a doctor already as far as I remember, so I think your team won the prize ^^", "votes": null }, { "id": "1464364", "postDate": "08/10/2021 14:45:28", "content": "congrats @yujiariyasu 🎉🎉
", "rawMarkdown": "congrats @yujiariyasu 🎉🎉", "votes": null }, { "id": "1464401", "postDate": "08/10/2021 14:58:56", "content": "haha, I wish we do. One of the things that inspired us throughout this competition 😅
", "rawMarkdown": "haha, I wish we do. One of the things that inspired us throughout this competition 😅", "votes": null }, { "id": "1464691", "postDate": "08/10/2021 17:03:21", "content": "@theoviel perhaps @yujiariyasu will claim it.. Maybe.. Let's see 😁😁
", "rawMarkdown": "theoviel perhaps @yujiariyasu will claim it.. Maybe.. Let's see 😁😁", "votes": null }, { "id": "1465083", "postDate": "08/10/2021 21:14:09", "content": "Congrats @yujiariyasu :)
\nWhere did you connect the aux loss to the swin transformer? I tried to do it the same way as with efficienNet but I kept running into shape mismatch errors. Would you mind sharing your segmentation head code for the swin transformer?
I graduated med school last year so we aren't eligible.
\nCongrats on your finish!
", "rawMarkdown": "I graduated med school last year so we aren't eligible. \n\nCongrats on your finish!", "votes": null }, { "id": "1465360", "postDate": "08/11/2021 03:09:23", "content": "like this
\ndef forward(self, x):\n mask = self.forward_features(x)\n\n x = self.head(mask)\n return x, mask\nAnd you need to resize the true mask.
", "rawMarkdown": "like this\n```\ndef forward(self, x):\n mask = self.forward_features(x)\n\n x = self.head(mask)\n return x, mask\n```\nAnd you need to resize the true mask.", "votes": null }, { "id": "1465519", "postDate": "08/11/2021 04:42:39", "content": "Congrats @yujiariyasu and thanks for sharing.
\nDo you plan to publish the source code?
\nI want to learn how to optimize params for Swin Transformer in your source code
Congrats on your second solo win! You deserve even better than GM.
\nI don't plan to publish it, but I can answer any questions you have here.
Thank you for your reply! I mean to which block/layer of the swin did you attach the aux loss?
", "rawMarkdown": "Thank you for your reply! I mean to which block/layer of the swin did you attach the aux loss?", "votes": null }, { "id": "1465703", "postDate": "08/11/2021 06:30:55", "content": "The features just before I put it in the head is the input mask for the aux criterion. And head is just nn.Linear().
", "rawMarkdown": "The features just before I put it in the head is the input mask for the aux criterion. And head is just nn.Linear().", "votes": null }, { "id": "1465704", "postDate": "08/11/2021 06:31:12", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "1466145", "postDate": "08/11/2021 10:32:52", "content": "Congrats @yujiariyasu , what optimizer and learning rate strategy did you use when training swin???
", "rawMarkdown": "Congrats @yujiariyasu , what optimizer and learning rate strategy did you use when training swin???", "votes": null }, { "id": "1466177", "postDate": "08/11/2021 10:50:56", "content": "adam & CosineAnnealingWarmRestarts.
\nThe schedulers were all pretty much the same.
Congrats bro, @yujiariyasu, hope that you will get the first place on other competitions
", "rawMarkdown": "Congrats bro, @yujiariyasu, hope that you will get the first place on other competitions", "votes": null }, { "id": "1467700", "postDate": "08/12/2021 05:38:09", "content": "Congrats, you are on the way to get Grandmaster rank.
", "rawMarkdown": "Congrats, you are on the way to get Grandmaster rank.", "votes": null }, { "id": "1468064", "postDate": "08/12/2021 08:41:58", "content": "Thanks! I hope so too.
", "rawMarkdown": "Thanks! I hope so too.", "votes": null }, { "id": "1468065", "postDate": "08/12/2021 08:42:33", "content": "Thanks! I'm going to proceed little by little.
", "rawMarkdown": "Thanks! I'm going to proceed little by little.", "votes": null } ], "comments": [ { "id": 1462649, "author_name": "blueboy97", "author_url": "", "post_date": "08/10/2021 00:36:33", "content": "Congratulations！🎉
", "votes": null, "replies": [ { "id": 1462675, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 00:57:47", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1462654, "author_name": "benihime91", "author_url": "", "post_date": "08/10/2021 00:40:31", "content": "Congrats on getting gold 👍🏻
", "votes": null, "replies": [ { "id": 1462676, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 00:58:14", "content": "So do you! Congratulations!
", "votes": null, "replies": [] } ] }, { "id": 1462655, "author_name": "southsakura", "author_url": "", "post_date": "08/10/2021 00:40:58", "content": "Congrats! May I ask your detection score separately without study results and how much it improved by ensemble. Thanks!👀
", "votes": null, "replies": [ { "id": 1462689, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 01:05:23", "content": "CV opacity ap was 0.5623. single best was 0.52~0.53. I'm not sure how much lb.
", "votes": null, "replies": [] } ] }, { "id": 1462658, "author_name": "arc144", "author_url": "", "post_date": "08/10/2021 00:44:25", "content": "Amazing solution, congratz on the results guys!
", "votes": null, "replies": [ { "id": 1462690, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 01:05:50", "content": "Thanks! Ian was amazing!
", "votes": null, "replies": [] } ] }, { "id": 1462659, "author_name": "hengck23", "author_url": "", "post_date": "08/10/2021 00:44:58", "content": "congrats and thanks for the nice write-up!
\ncan I confirm the following?
\nfinal public submission 0.650, which break down to
\nstudy: 0.413.
\nimage: 0.237
do you have break down of:
\nThanks! In fact, at the end of the competition, we didn't submit study level only. But I think it's probably 0.413. Maybe it's 0.414. We don't know much about LB since we hardly paid attention to it after working with Ian in the first place.
", "votes": null, "replies": [] } ] }, { "id": 1462661, "author_name": "seongwook93", "author_url": "", "post_date": "08/10/2021 00:46:20", "content": "Congratulations！
", "votes": null, "replies": [ { "id": 1462691, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 01:06:00", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1462681, "author_name": "vaillant", "author_url": "", "post_date": "08/10/2021 01:00:29", "content": "Thanks @yujiariyasu for the awesome write-up- I had a great time working with you!
", "votes": null, "replies": [ { "id": 1462692, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 01:06:52", "content": "Thanks to you too! I'll try my best to be like you one day:)
", "votes": null, "replies": [] } ] }, { "id": 1463622, "author_name": "namgalielei", "author_url": "", "post_date": "08/10/2021 09:03:38", "content": "Nice work for both of you. There are some things that I am curious about:
\nWhat is the goal of \"I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.\" Do you use cropping to preprocess image before study level prediction or to create additional training data? (Yuji part)
I used cropping to preprocess the images before making study level predictions.
\nthe prediction results of detection will change the crop, so the wisdom of the detection model can be conveyed to the classification model.
congratulations @yujiariyasu
", "votes": null, "replies": [ { "id": 1464160, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/10/2021 13:18:28", "content": "thank you!
", "votes": null, "replies": [] } ] }, { "id": 1464010, "author_name": "awsaf49", "author_url": "", "post_date": "08/10/2021 12:20:29", "content": "Congrats 🎉 @yujiariyasu & @vaillant. You guys did great.
\nBtw will you guys be nominating yourselves for the Student Prize ?
@vaillant is a doctor already as far as I remember, so I think your team won the prize ^^
", "votes": null, "replies": [] }, { "id": 1464401, "author_name": "awsaf49", "author_url": "", "post_date": "08/10/2021 14:58:56", "content": "haha, I wish we do. One of the things that inspired us throughout this competition 😅
", "votes": null, "replies": [] }, { "id": 1464691, "author_name": "zaber666", "author_url": "", "post_date": "08/10/2021 17:03:21", "content": "@theoviel perhaps @yujiariyasu will claim it.. Maybe.. Let's see 😁😁
", "votes": null, "replies": [] }, { "id": 1465109, "author_name": "vaillant", "author_url": "", "post_date": "08/10/2021 21:36:24", "content": "I graduated med school last year so we aren't eligible.
\nCongrats on your finish!
", "votes": null, "replies": [] } ] }, { "id": 1464364, "author_name": "jarupula", "author_url": "", "post_date": "08/10/2021 14:45:28", "content": "congrats @yujiariyasu 🎉🎉
", "votes": null, "replies": [ { "id": 1465704, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/11/2021 06:31:12", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1465083, "author_name": "amiiiney", "author_url": "", "post_date": "08/10/2021 21:14:09", "content": "Congrats @yujiariyasu :)
\nWhere did you connect the aux loss to the swin transformer? I tried to do it the same way as with efficienNet but I kept running into shape mismatch errors. Would you mind sharing your segmentation head code for the swin transformer?
like this
\ndef forward(self, x):\n mask = self.forward_features(x)\n\n x = self.head(mask)\n return x, mask\nAnd you need to resize the true mask.
", "votes": null, "replies": [] }, { "id": 1465664, "author_name": "amiiiney", "author_url": "", "post_date": "08/11/2021 06:12:04", "content": "Thank you for your reply! I mean to which block/layer of the swin did you attach the aux loss?
", "votes": null, "replies": [] }, { "id": 1465703, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/11/2021 06:30:55", "content": "The features just before I put it in the head is the input mask for the aux criterion. And head is just nn.Linear().
", "votes": null, "replies": [] } ] }, { "id": 1465519, "author_name": "nguyenbadung", "author_url": "", "post_date": "08/11/2021 04:42:39", "content": "Congrats @yujiariyasu and thanks for sharing.
\nDo you plan to publish the source code?
\nI want to learn how to optimize params for Swin Transformer in your source code
Congrats on your second solo win! You deserve even better than GM.
\nI don't plan to publish it, but I can answer any questions you have here.
Congrats @yujiariyasu , what optimizer and learning rate strategy did you use when training swin???
", "votes": null, "replies": [ { "id": 1466177, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/11/2021 10:50:56", "content": "adam & CosineAnnealingWarmRestarts.
\nThe schedulers were all pretty much the same.
Congrats bro, @yujiariyasu, hope that you will get the first place on other competitions
", "votes": null, "replies": [ { "id": 1468064, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/12/2021 08:41:58", "content": "Thanks! I hope so too.
", "votes": null, "replies": [] } ] }, { "id": 1467700, "author_name": "kingabzpro", "author_url": "", "post_date": "08/12/2021 05:38:09", "content": "Congrats, you are on the way to get Grandmaster rank.
", "votes": null, "replies": [ { "id": 1468065, "author_name": "yujiariyasu", "author_url": "", "post_date": "08/12/2021 08:42:33", "content": "Thanks! I'm going to proceed little by little.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1462646": "First of all, I would like to thank you for organizing this very interesting competition.\nAlso, I can't thank IanPan enough. After we teamed up, our score improved very quickly, as if by magic. I have learned a lot from him through this competition.\n\nThe following is my part.\n\n---\n\n### study level:\n・Swin Transformer \nIn my environment, Swin Transformer was the best. cait was not bad, but aux loss did not work well with cait, so I chose Swin Transformer.\nI left the resolution at 384. I tried 768, but it was slightly worse. I also did some cropping, so 384 seemed to be sufficient.\n\n##### 1st stage:\n・Pre training was done with chexpert data.\nThis is Ian's idea. The data is quite different from ImageNet. we can get a good initial set of weights by pre training.\nIn addition, further pre-training with the RSNA 2018 competition data before training with chexpert resulted in a slight improvement in CV over using chexpert only.\nI used crop, which I will write about later, and did not use aux loss.\n\n##### 2nd stage:\n・5classes\nI included 'none' in the class.\n・Cropping with detection and segmentation\nBy using the predictions of the detection for cropping, I was able to apply the wisdom of the detection model to the classification model, and I also used the lung Segmentator trained by VinBigData datasets. I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.\nIf the box is used as-is, for example, a pleural effusion may result in an image that is difficult to distinguish, The box has been padded with 100 pixels.\n・mixup\nMixup brought about a small improvement. In the end, for my part, I used two different models, one with Mixup and one without.\nAt the stage of not using aux loss, resize mix was the best, but in the end, mixup was better.\n・aux loss\nI used binary_cross_entropy_with_logits.\nIn one of the two models, I set the negative image loss to zero.\n・No pseudo label\nI tried everything, but the results were slightly worse.\n・The average of the 5fold model for the four classes of mAP was 0.397.\n\n### image level\n・YOLOV5\nI used l6&l.\n##### opacity:\n・Pre training was done with RSNA 2018 data.\nThe same reason as for study level.\n・mixup: 0.5\nThere was a slight improvement.\n・Use none predictions\nFor each image, I found the following improved the score the most\ndection_conf = dection_conf * (1-image none prediction)**0.4\n\n##### none:\nI used predictions of classification models and detection models like this.\nnone_pred = cl_none_pred*0.7 + (1 - image_conf_max)*0.3\n\n---\n\nThe following is a message from Ian.\n\n---\nI would like to thank Yuji for agreeing to team with me. I reached out later in the competition when he was already doing really well without me. I learned a lot from him, and this competition reminded me how fun and enlightening it is to team up on Kaggle. \n\n### Classification\n\nFor classification, I trained a 5-class classifier: 4 study labels + none at image-level (vs. opacity). We only trained on 1 image/study. Since the vast majority of studies only had a single unique image, this was not really an issue. \n\nFirst, I pretrained the classifier on the 3 classes from the RSNA 2018 Pneumonia Detection Challenge. Then, I fine-tuned from these weights on this competition's data. This significantly increased performance. \n\nI trained a hybrid classification-segmentation model. One architecture used EfficientNet-B6 and DeepLabV3+ while the other used Swin Transformer and FPN. These were trained using the awesome Segmentation Models PyTorch library (https://github.com/qubvel/segmentation_models.pytorch). \n\nThe models were trained using AdamW optimizer, 512x512 images, and data augmentation with contrast/brightness adjustment, random noise, and scale/shift/rotate using albumentations. \n\nI used BCE loss for both classification (even though you could use cross-entropy, I found BCE worked better for the metric) and segmentation, weighted 1:1. The segmentation part of the model only had 1 class for opacity. The segmentation label was generated using the bounding box labels. I turned the bounding box into an ellipse and used that as the training label. \n\nI also used mixup, which showed significant improvements. 5-fold ensemble of the EfficientNet-B6/DeepLabV3+ and Swin/FPN models (10 models total) gave 0.408 LB for study-label classification. \n\n### Detection \n\nI trained detection models using the awesome EfficientDet library (https://github.com/rwightman/efficientdet-pytorch) and MMDetection (https://github.com/open-mmlab/mmdetection). \n\nI also pretrained these models on the RSNA 2018 dataset (images with opacity only) using the bounding box labels. This significantly increased performance for detection as well. I trained 5-fold EfficientDet-D7X on 512x512 images (positives only). I also trained 5-fold EfficientDet-D6 on 640x640 images (positives only). \n\nTo add diversity to the ensemble, I trained models in MMDetection as well. I used Swin Transformer with RepPoints, borrowing the code from https://github.com/SwinTransformer/Swin-Transformer-Object-Detection to use Swin Transformer as a backbone for object detection. These models were trained using multi-scale training (512x512 to 768x768) and during inference 3x multi-scale TTA was used. \n\nI contributed 15 detection models (EffDet-D7X, EffDet-D6, Swin-RepPoints) for the ensemble. \n\n### What Didn't Work For Me\n- Training classifier with detection as auxiliary loss instead of augmentation\n- Mixup during detection\n- Training a multiclass detector (using study labels, instead of just training on single-class opacity)\n- In my experiments, pseudolabeling did not improve CV\n- Training a multiclass segmentation model and using that to help refine study-label predictions\n- Incorporating bounding box predictions into input for classification models\n\n---\n\nThe following is the part of the ensemble.\n\n### study level \nIn the end, we used a simple averaging.\nI tried Nelder-Mead, ridge regression using predictions from detection models, etc., but all of them did not work because they had overfiit.\nBy averaging the predictions of my two models with those of Ian's two models, we made significant improvements in both CV/LB. Since we both developed our models using each other's pipelines, there was a lot of diversity in the models, and by mixing our predictions at the end, the final predictions were more robust.\nOur final study level LB was about 0.413.\n\n### image level\nI mixed my yolov5 with Ian's effdet and mmdet Swin Transformer models in wbf. I saw a big improvement. I used the same weights for all the models as I did not see any improvement with different weights. Also, iou threshold 0.6 was the best.\n\n###code\ntrain: https://github.com/yujiariyasu/siim_covid19_detection\ninference: https://www.kaggle.com/yujiariyasu/fork-of-siim-covid-19-full-pipeline-v2\n \nI would like to thank all the participants for their hard work in this competition!", "1462649": "Congratulations！🎉", "1462654": "Congrats on getting gold 👍🏻", "1462655": "Congrats! May I ask your detection score separately without study results and how much it improved by ensemble. Thanks!👀", "1462658": "Amazing solution, congratz on the results guys!", "1462659": "congrats and thanks for the nice write-up!\n\ncan I confirm the following?\n\nfinal public submission 0.650, which break down to\nstudy: 0.413.\nimage: 0.237\n\ndo you have break down of:\n - individual public submission map of each study class (negative, typical, indterminate, atypical)\n - map of none and map of box opacity (I guess is about 0.82 and 0.60 respectively?)", "1462661": "Congratulations！", "1462673": "Thanks! In fact, at the end of the competition, we didn't submit study level only. But I think it's probably 0.413. Maybe it's 0.414. We don't know much about LB since we hardly paid attention to it after working with Ian in the first place.", "1462675": "Thank you!", "1462676": "So do you! Congratulations!", "1462681": "Thanks @yujiariyasu for the awesome write-up- I had a great time working with you!", "1462689": "CV opacity ap was 0.5623. single best was 0.52~0.53. I'm not sure how much lb.", "1462690": "Thanks! Ian was amazing!", "1462691": "Thank you!", "1462692": "Thanks to you too! I'll try my best to be like you one day:)", "1463622": "Nice work for both of you. There are some things that I am curious about:\nWhat is the goal of \"I crop the image only in the area where the detection box (conf > 0.3) or segmentation mask exists.\" Do you use cropping to preprocess image before study level prediction or to create additional training data? (Yuji part)", "1463660": "congratulations @yujiariyasu", "1464010": "Congrats 🎉 @yujiariyasu & @vaillant. You guys did great.\nBtw will you guys be nominating yourselves for the **Student Prize** ?", "1464159": "I used cropping to preprocess the images before making study level predictions.\nthe prediction results of detection will change the crop, so the wisdom of the detection model can be conveyed to the classification model.", "1464160": "thank you!", "1464341": "vaillant is a doctor already as far as I remember, so I think your team won the prize ^^", "1464364": "congrats @yujiariyasu 🎉🎉", "1464401": "haha, I wish we do. One of the things that inspired us throughout this competition 😅", "1464691": "theoviel perhaps @yujiariyasu will claim it.. Maybe.. Let's see 😁😁", "1465083": "Congrats @yujiariyasu :) \nWhere did you connect the aux loss to the swin transformer? I tried to do it the same way as with efficienNet but I kept running into shape mismatch errors. Would you mind sharing your segmentation head code for the swin transformer?", "1465109": "I graduated med school last year so we aren't eligible. \n\nCongrats on your finish!", "1465360": "like this\n```\ndef forward(self, x):\n mask = self.forward_features(x)\n\n x = self.head(mask)\n return x, mask\n```\nAnd you need to resize the true mask.", "1465519": "Congrats @yujiariyasu and thanks for sharing.\nDo you plan to publish the source code?\nI want to learn how to optimize params for Swin Transformer in your source code", "1465577": "Congrats on your second solo win! You deserve even better than GM.\nI don't plan to publish it, but I can answer any questions you have here.", "1465664": "Thank you for your reply! I mean to which block/layer of the swin did you attach the aux loss?", "1465703": "The features just before I put it in the head is the input mask for the aux criterion. And head is just nn.Linear().", "1465704": "Thank you!", "1466145": "Congrats @yujiariyasu , what optimizer and learning rate strategy did you use when training swin???", "1466177": "adam & CosineAnnealingWarmRestarts.\nThe schedulers were all pretty much the same.", "1467311": "Congrats bro, @yujiariyasu, hope that you will get the first place on other competitions", "1467700": "Congrats, you are on the way to get Grandmaster rank.", "1468064": "Thanks! I hope so too.", "1468065": "Thanks! I'm going to proceed little by little." }, "source": "meta" }