*submitted kernel: https://www.kaggle.com/jihunlorenzopark/siim-covid-sub-mmdet?scriptVersionId=70842068
\n*full training scripts on github: https://github.com/lorenzo-park/siim-covid-kaggle-model-sub
Thank you for hosting the competition to Kaggle, SIIM, FISABIO, RSNA. I am so happy to get the first gold for two years of Kaggling. I and my teammates keep working hard until the last night and finally the last day's submission ranked at 8th on private leaderboard!
\nOur model consists of three parts as most of the people have used:
\nStudy-level prediction, two class prediction and opacity bounding box detection.
CV Strategy Stratified Grouped 5 Fold, using PatientID for grouping
Optimizer AdamW with CosineAnnealingWarmRestarts
Model
\nLoss
\nAugmentation
A.Compose([\n A.Resize(img_size,img_size),\n A.HorizontalFlip(p=0.5),\n RandomBrightness(limit=0.1, p=0.75),\n A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=30, border_mode=0, p=0.75),\n A.Cutout(max_h_size=int(img_size * 0.3), max_w_size=int(img_size * 0.3), num_holes=1, p=0.75),\n ToTensorV2(p=1.0),\n ])\nEnsemble of 3 YOLO models and 1 CascadedRCNN model
\nuse YOLOv5 Hyperparameter Evolution
\nEnsemble method : weighted boxes fusion
\nFine-tuned by replacing the last layer of Study-Level 4-class Classification
Sharing the exact same train/valid fold of 4-class training stage to prevent potential leakage
Ensemble Unet++_384x384 + Swin Transformer_384x384
Tools
\nIdea threads
\nKernels
\nExternal datasets
\nNIH Chest X-rays: https://www.kaggle.com/nih-chest-xrays/data
Thank you for reading and congratulations to all winners.
\nThere could be missing details in the solution. I will keep update the solution as I get the missing details in my mind back.
*20210811: Add data duplication in Others part. 5th solution makes me remind this detail.
\n*20210811: Update used GPUs in Reproducilibity part*
\n*20210813: Add Image opacity detection part solution written by @normalkim0
\n*20210817: Attach submitted kernel link
\n*20210818: Add reproducible github repository link
*submitted kernel: https://www.kaggle.com/jihunlorenzopark/siim-covid-sub-mmdet?scriptVersionId=70842068
\n*full training scripts on github: https://github.com/lorenzo-park/siim-covid-kaggle-model-sub
Thank you for hosting the competition to Kaggle, SIIM, FISABIO, RSNA. I am so happy to get the first gold for two years of Kaggling. I and my teammates keep working hard until the last night and finally the last day's submission ranked at 8th on private leaderboard!
\nOur model consists of three parts as most of the people have used:
\nStudy-level prediction, two class prediction and opacity bounding box detection.
CV Strategy Stratified Grouped 5 Fold, using PatientID for grouping
Optimizer AdamW with CosineAnnealingWarmRestarts
Model
\nLoss
\nAugmentation
A.Compose([\n A.Resize(img_size,img_size),\n A.HorizontalFlip(p=0.5),\n RandomBrightness(limit=0.1, p=0.75),\n A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=30, border_mode=0, p=0.75),\n A.Cutout(max_h_size=int(img_size * 0.3), max_w_size=int(img_size * 0.3), num_holes=1, p=0.75),\n ToTensorV2(p=1.0),\n ])\nEnsemble of 3 YOLO models and 1 CascadedRCNN model
\nuse YOLOv5 Hyperparameter Evolution
\nEnsemble method : weighted boxes fusion
\nFine-tuned by replacing the last layer of Study-Level 4-class Classification
Sharing the exact same train/valid fold of 4-class training stage to prevent potential leakage
Ensemble Unet++_384x384 + Swin Transformer_384x384
Tools
\nIdea threads
\nKernels
\nExternal datasets
\nNIH Chest X-rays: https://www.kaggle.com/nih-chest-xrays/data
Thank you for reading and congratulations to all winners.
\nThere could be missing details in the solution. I will keep update the solution as I get the missing details in my mind back.
*20210811: Add data duplication in Others part. 5th solution makes me remind this detail.
\n*20210811: Update used GPUs in Reproducilibity part*
\n*20210813: Add Image opacity detection part solution written by @normalkim0
\n*20210817: Attach submitted kernel link
\n*20210818: Add reproducible github repository link
Congratulation @jihunlorenzopark Thanks for sharing.
\nCan you tell me how to use the Swin Transformer.
Thanks!
\nI used the one in timm library. It provides you the module and pretrained models with few lines of code. You can check the example usage on the kernel I mentioned in the article or my solution github repository model/timm.py.
Congrats on the gold medal!
\nI obtained similar public LB score but significantly lower private LB score. May I ask how you performed \"early stopping\"?
Thank you!
\nI used pytorch lightning and it has the early stopping callback.
\nIf you are asking how early stopping works, we set patience and if validation loss is not decreased for more epochs than the patience, then we stop the training and use the checkpoint at minimum validation loss.
Also, the best private LB is come with 0.635 public LB submission which uses WBF for different object detection models. So WBF with different kind of model can be the reason to relatively small decrease in private LB.
", "rawMarkdown": "Also, the best private LB is come with 0.635 public LB submission which uses WBF for different object detection models. So WBF with different kind of model can be the reason to relatively small decrease in private LB." }, { "id": 1466865, "postDate": "2021-08-11T16:58:22.583Z", "content": "Thanks for your reply!
\nI agree with you that ensemble more models can make the prediction more generalized and therefor lower decrease in private LB. 💯 In our case, we only trained one pipeline with yolov5 and Effnet v2, thus drop from 0.636 public LB to 0.611 private LB. 😛
\nAs you explained for early stopping, we actually applied it similarly. After you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!
", "rawMarkdown": "Thanks for your reply!\n\nI agree with you that ensemble more models can make the prediction more generalized and therefor lower decrease in private LB. 💯 In our case, we only trained one pipeline with yolov5 and Effnet v2, thus drop from 0.636 public LB to 0.611 private LB. 😛 \n\nAs you explained for early stopping, we actually applied it similarly. After you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!\n\n", "votes": 1 }, { "id": 1467027, "postDate": "2021-08-11T18:25:48.557Z", "content": "One more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use? I am curious to see how its extracted lung masks compared to the DL model.
\nSorry for my so many questions. 😀
", "rawMarkdown": "One more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use? I am curious to see how its extracted lung masks compared to the DL model. \n\nSorry for my so many questions. 😀", "votes": 1 }, { "id": 1467468, "postDate": "2021-08-12T02:20:17.917Z", "content": "\n\nAfter you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!
\n
For this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV.
", "rawMarkdown": "> After you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!\n\nFor this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV. ", "replies": [ { "id": 1468786, "postDate": "2021-08-12T15:12:25.117Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1467472, "postDate": "2021-08-12T02:22:34.783Z", "content": "\n\nOne more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use?
\n
This one is done with the series of cv2 and skimage functions. Here is the snippet link dataset.py file (line71-line84) in my repo.
\nI think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.
I'd love to share my thoughts on any questions. Feel free to do even more :)
", "rawMarkdown": "> One more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use?\n\nThis one is done with the series of cv2 and skimage functions. Here is the snippet link [dataset.py file (line71-line84)](https://github.com/lorenzo-park/kaggle-solution-siim-fisabio-rsna-covid19-detection/blob/master/dataset.py#L71) in my repo.\nI think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.\n\nI'd love to share my thoughts on any questions. Feel free to do even more :)" }, { "id": 1468830, "postDate": "2021-08-12T15:31:05.237Z", "content": "\n\nFor this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV.
\n
I see, thanks for your explanation! To obtain the 5 checkpoints from the 5 folds, I guess you use a constant LR during training, right?
\n\n\nThis one is done with the series of cv2 and skimage functions. Here is the snippet link dataset.py file (line71-line84) in my repo.
\n
\n I think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.
Thanks for sharing! I will be happy to try your method and compare with DL. 🤓
", "rawMarkdown": "> For this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV. \n\nI see, thanks for your explanation! To obtain the 5 checkpoints from the 5 folds, I guess you use a constant LR during training, right?\n\n> This one is done with the series of cv2 and skimage functions. Here is the snippet link dataset.py file (line71-line84) in my repo.\nI think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.\n\nThanks for sharing! I will be happy to try your method and compare with DL. 🤓\n\n", "votes": 1 }, { "id": 1469451, "postDate": "2021-08-13T00:10:47.817Z", "content": "I used LR scheduler, so LR is changing during the training for each fold, but the LR should be the same across all fold.
", "rawMarkdown": "I used LR scheduler, so LR is changing during the training for each fold, but the LR should be the same across all fold." } ] }, { "id": 1464143, "postDate": "2021-08-10T13:13:07.633Z", "content": "Thanks for the detailed solution and congrats.
", "rawMarkdown": "Thanks for the detailed solution and congrats.", "votes": 1 }, { "id": 1463401, "postDate": "2021-08-10T07:19:12.970Z", "content": "Congratulations, And thanks a lot for your share,
\nCan you tell me what is Segmentation auxiliary loss?
Thanks! You can check this thread out. Basically, it is multi-task training. Model gets the ability to extract more important feature for classification task by training with segmentation decoder together using segmentation auxiliary loss.
", "rawMarkdown": "Thanks! You can check [this thread](https://www.kaggle.com/c/siim-covid19-detection/discussion/240233) out. Basically, it is multi-task training. Model gets the ability to extract more important feature for classification task by training with segmentation decoder together using segmentation auxiliary loss. ", "votes": 1 }, { "id": 1464763, "postDate": "2021-08-10T17:40:29.143Z", "content": "Sorry for the late response,
\nThank you @jihunlorenzopark sir for your answer,
\nI've understood it very well
Congratulation @jihunlorenzopark Thanks for sharing.
\nCan you tell me how to use the Swin Transformer.
Congratulation @jihunlorenzopark Thanks for sharing.
\nCan you tell me how to use the Swin Transformer.
Thanks!
\nI used the one in timm library. It provides you the module and pretrained models with few lines of code. You can check the example usage on the kernel I mentioned in the article or my solution github repository model/timm.py.
Congrats on the gold medal!
\nI obtained similar public LB score but significantly lower private LB score. May I ask how you performed \"early stopping\"?
Thank you!
\nI used pytorch lightning and it has the early stopping callback.
\nIf you are asking how early stopping works, we set patience and if validation loss is not decreased for more epochs than the patience, then we stop the training and use the checkpoint at minimum validation loss.
Also, the best private LB is come with 0.635 public LB submission which uses WBF for different object detection models. So WBF with different kind of model can be the reason to relatively small decrease in private LB.
", "votes": 0, "replies": [] }, { "id": 1466865, "author_name": "Kuan Zhang", "author_url": "", "post_date": "2021-08-11T16:58:22.583000", "content": "Thanks for your reply!
\nI agree with you that ensemble more models can make the prediction more generalized and therefor lower decrease in private LB. 💯 In our case, we only trained one pipeline with yolov5 and Effnet v2, thus drop from 0.636 public LB to 0.611 private LB. 😛
\nAs you explained for early stopping, we actually applied it similarly. After you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!
", "votes": 1, "replies": [] }, { "id": 1467027, "author_name": "Kuan Zhang", "author_url": "", "post_date": "2021-08-11T18:25:48.557000", "content": "One more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use? I am curious to see how its extracted lung masks compared to the DL model.
\nSorry for my so many questions. 😀
", "votes": 1, "replies": [] }, { "id": 1467468, "author_name": "Jihun Lorenzo Park", "author_url": "", "post_date": "2021-08-12T02:20:17.917000", "content": "\n\nAfter you obtained the checkpoint by early stopping, did you perform a fune-tuning with smaller lr? Thanks!
\n
For this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV.
", "votes": 0, "replies": [ { "id": 1468786, "author_name": "", "author_url": "", "post_date": "2021-08-12T15:12:25.117000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1467472, "author_name": "Jihun Lorenzo Park", "author_url": "", "post_date": "2021-08-12T02:22:34.783000", "content": "\n\nOne more question is on the \"Lung mask obtained by this heuristics and opacity bbox mask))\". Is there a link for the tool available to use?
\n
This one is done with the series of cv2 and skimage functions. Here is the snippet link dataset.py file (line71-line84) in my repo.
\nI think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.
I'd love to share my thoughts on any questions. Feel free to do even more :)
", "votes": 0, "replies": [] }, { "id": 1468830, "author_name": "Kuan Zhang", "author_url": "", "post_date": "2021-08-12T15:31:05.237000", "content": "\n\nFor this part, we don't to re-train the model. The 5 checkpoints for each fold obtained by early stopping is used to submission (averaging results of 5 models). If you meant training with NIH pretrained weight, we tried with 0.1*(original LR) to encoder but using the same LR gives better CV.
\n
I see, thanks for your explanation! To obtain the 5 checkpoints from the 5 folds, I guess you use a constant LR during training, right?
\n\n\nThis one is done with the series of cv2 and skimage functions. Here is the snippet link dataset.py file (line71-line84) in my repo.
\n
\n I think the quality of the lung mask would be not good as DL based method, but the advantage is clear - No need additional trainings for Lung mask segmentation DL model.
Thanks for sharing! I will be happy to try your method and compare with DL. 🤓
", "votes": 1, "replies": [] }, { "id": 1469451, "author_name": "Jihun Lorenzo Park", "author_url": "", "post_date": "2021-08-13T00:10:47.817000", "content": "I used LR scheduler, so LR is changing during the training for each fold, but the LR should be the same across all fold.
", "votes": 0, "replies": [] } ] }, { "id": 1464143, "author_name": "Zabir Al Nazi Nabil", "author_url": "", "post_date": "2021-08-10T13:13:07.633000", "content": "Thanks for the detailed solution and congrats.
", "votes": 1, "replies": [] }, { "id": 1463401, "author_name": "Ifti", "author_url": "", "post_date": "2021-08-10T07:19:12.970000", "content": "Congratulations, And thanks a lot for your share,
\nCan you tell me what is Segmentation auxiliary loss?
Thanks! You can check this thread out. Basically, it is multi-task training. Model gets the ability to extract more important feature for classification task by training with segmentation decoder together using segmentation auxiliary loss.
", "votes": 1, "replies": [] }, { "id": 1464763, "author_name": "Ifti", "author_url": "", "post_date": "2021-08-10T17:40:29.143000", "content": "Sorry for the late response,
\nThank you @jihunlorenzopark sir for your answer,
\nI've understood it very well
Congratulation @jihunlorenzopark Thanks for sharing.
\nCan you tell me how to use the Swin Transformer.