![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6439446%2Fdd8e2019007298da32e43ffe92e1fe32%2Fsingle_model.png?generation=1691082591623984&alt=media\")
\nThanks to the organizers for this compelling competition. We present the 4th place solution from the GoN team at SDSRV.AI. It emphasizes the importance of dataset, validation, ensemble and post processing method. Special thanks to @ducmanvo, an amazing teammate.
\nOur solution employs a combination of 4 instance segmentation models comprising CascadeRCNN (ResNeXt, Regnet), MaskRCNN (swint), HybridTaskCascade (Re2Net), and 1 object detection model, Yolov6m. Each model is trained with a dataset consisting of 1549 images (ds1+ds2), with 84 images from dataset1 utilized for validation. During the inference process, the Weighted Boxes Fusion (WBF) technique is employed to ensemble RPN boxes and ROI boxes, while the masks generated by the ensemble of models utilize the mean average approach. For post-processing, we filter out small instances and refine instance score using mask scores.
\nTrain-test split
\n - Training: 1549 images (1211 images from ds2 + 338 images from ds1)
\n - Validation: 84 images from ds1
\n - During the initial stages of approaching the problem (in the final month of the competition), we initially utilized K-fold cross-validation. However, upon recognizing significant differences in distribution, label assignment methodologies between dataset1 and dataset2, as well as variations in label assignments among Whole Slide Images (WSI), we decided to split the data into two train-validation sets. The validation data was exclusively taken from dataset1 (20% of dataset1). Another reason for this data split was the scarcity of dataset1 images in the training set, which was a concern given that the private test data was exclusively sourced from dataset1 (with only 422 images available).
\n - To determine the dissimilarities in the label distribution between dataset1 and dataset2, as well as variations among different Whole Slide Images (WSI), we trained a base model, MaskRCNN R50, on the training data from dataset1 (fold1) and validated it on dataset1, dataset2 (fold1). Our observations revealed that the model's performance scored considerably higher on dataset1 compared to dataset2. Similar procedures were applied when comparing performance across various Whole Slide Images (WSI).
\nPreprocessing: remove duplicate annotations
\n - Approximately 4.5% of training data was duplicated. We directly removed these duplicate labels to ensure data integrity and improve model performance.
During training and validation, we specifically focused on the \"blood_vessel\" and \"unsure\" classes. This decision was influenced by the significant size difference between glomerulus and the other two classes, with glomerulus being much larger. Surprisingly, when training with all three classes, the model's performance on glomerulus remained exceptionally high. Additionally, the competition organizers informed us that during the testing phase, we could utilize glomerulus labels to eliminate false positives. Consequently, we concluded that training with additional glomerulus data was unnecessary. After removing glomerulus labels from the training dataset, we observed an improvement in the performance score for the \"blood_vessel\" class. Therefore, we made the decision to exclude glomerulus labels from all subsequent experiments.
\n2 stages model: MaskRCNN, CascadeRCNN, HybridTaskCascade
\nYolov6m: Two-stage models like MaskRCNN, CascadeRCNN, and HybridTaskCascade exhibit good Mean Average Precision (MAP) at a certain intersection-over-union (IoU) threshold range [0.5:0.95]. However, their overall MAP [0.5:0.95] scores may not reach high values. To address this limitation, YOLO series models offer a solution. Specifically, YOLOv6, with its default training configurations, achieves a significantly higher Mean Average Recall (MAR) of 55.1, compared to the 44.x obtained by the two-stage models. This improvement in MAR highlights the effectiveness of YOLO series models in overcoming the mentioned drawback.
\nDetail training config
Focusing on optimizing the validation set with dataset1 exclusively, fine-tuning the model for a few epochs using dataset1 in the training set improved performance on the validation data.
\n\nEnsemble
\n![\"Ensemble](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6439446%2F5a2d33de596e95e08f227e514cc49350%2Fensemble.png?generation=1691081285365346&alt=media\")
PostProcessing
\nscore_instance = score_bbox * mean(mask[mask > 0.5])Some detail experiments:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6439446%2Ffabaef83a462c8b1d1f54d8ad11389ea%2Ffinal%20experiments.png?generation=1691081498124640&alt=media\")
\nWe observed a strong correlation between the local validation and the public leaderboard (LB) scores. Therefore, we only made submissions when there was a significant improvement in the validation score.
Keywords: Instance segmentation, ensemble instance segmentation, re-ranking instance, refine instance score.
", "messages": [ { "id": "2372420", "postDate": "08/03/2023 16:57:36", "content": "Thanks to the organizers for this compelling competition. We present the 4th place solution from the GoN team at SDSRV.AI. It emphasizes the importance of dataset, validation, ensemble and post processing method. Special thanks to @ducmanvo, an amazing teammate.
\nOur solution employs a combination of 4 instance segmentation models comprising CascadeRCNN (ResNeXt, Regnet), MaskRCNN (swint), HybridTaskCascade (Re2Net), and 1 object detection model, Yolov6m. Each model is trained with a dataset consisting of 1549 images (ds1+ds2), with 84 images from dataset1 utilized for validation. During the inference process, the Weighted Boxes Fusion (WBF) technique is employed to ensemble RPN boxes and ROI boxes, while the masks generated by the ensemble of models utilize the mean average approach. For post-processing, we filter out small instances and refine instance score using mask scores.
\nTrain-test split
\n - Training: 1549 images (1211 images from ds2 + 338 images from ds1)
\n - Validation: 84 images from ds1
\n - During the initial stages of approaching the problem (in the final month of the competition), we initially utilized K-fold cross-validation. However, upon recognizing significant differences in distribution, label assignment methodologies between dataset1 and dataset2, as well as variations in label assignments among Whole Slide Images (WSI), we decided to split the data into two train-validation sets. The validation data was exclusively taken from dataset1 (20% of dataset1). Another reason for this data split was the scarcity of dataset1 images in the training set, which was a concern given that the private test data was exclusively sourced from dataset1 (with only 422 images available).
\n - To determine the dissimilarities in the label distribution between dataset1 and dataset2, as well as variations among different Whole Slide Images (WSI), we trained a base model, MaskRCNN R50, on the training data from dataset1 (fold1) and validated it on dataset1, dataset2 (fold1). Our observations revealed that the model's performance scored considerably higher on dataset1 compared to dataset2. Similar procedures were applied when comparing performance across various Whole Slide Images (WSI).
\nPreprocessing: remove duplicate annotations
\n - Approximately 4.5% of training data was duplicated. We directly removed these duplicate labels to ensure data integrity and improve model performance.
During training and validation, we specifically focused on the \"blood_vessel\" and \"unsure\" classes. This decision was influenced by the significant size difference between glomerulus and the other two classes, with glomerulus being much larger. Surprisingly, when training with all three classes, the model's performance on glomerulus remained exceptionally high. Additionally, the competition organizers informed us that during the testing phase, we could utilize glomerulus labels to eliminate false positives. Consequently, we concluded that training with additional glomerulus data was unnecessary. After removing glomerulus labels from the training dataset, we observed an improvement in the performance score for the \"blood_vessel\" class. Therefore, we made the decision to exclude glomerulus labels from all subsequent experiments.
\n2 stages model: MaskRCNN, CascadeRCNN, HybridTaskCascade
\nYolov6m: Two-stage models like MaskRCNN, CascadeRCNN, and HybridTaskCascade exhibit good Mean Average Precision (MAP) at a certain intersection-over-union (IoU) threshold range [0.5:0.95]. However, their overall MAP [0.5:0.95] scores may not reach high values. To address this limitation, YOLO series models offer a solution. Specifically, YOLOv6, with its default training configurations, achieves a significantly higher Mean Average Recall (MAR) of 55.1, compared to the 44.x obtained by the two-stage models. This improvement in MAR highlights the effectiveness of YOLO series models in overcoming the mentioned drawback.
\nDetail training config
Focusing on optimizing the validation set with dataset1 exclusively, fine-tuning the model for a few epochs using dataset1 in the training set improved performance on the validation data.
\n\nEnsemble
\nPostProcessing
\nscore_instance = score_bbox * mean(mask[mask > 0.5])Some detail experiments:
\n
\nWe observed a strong correlation between the local validation and the public leaderboard (LB) scores. Therefore, we only made submissions when there was a significant improvement in the validation score.
Keywords: Instance segmentation, ensemble instance segmentation, re-ranking instance, refine instance score.
", "rawMarkdown": "Thanks to the organizers for this compelling competition. We present the 4th place solution from the GoN team at SDSRV.AI. It emphasizes the importance of dataset, validation, ensemble and post processing method. Special thanks to @ducmanvo, an amazing teammate.\n\n# Summary\n\nOur solution employs a combination of 4 instance segmentation models comprising CascadeRCNN (ResNeXt, Regnet), MaskRCNN (swint), HybridTaskCascade (Re2Net), and 1 object detection model, Yolov6m. Each model is trained with a dataset consisting of 1549 images (ds1+ds2), with 84 images from dataset1 utilized for validation. During the inference process, the Weighted Boxes Fusion (WBF) technique is employed to ensemble RPN boxes and ROI boxes, while the masks generated by the ensemble of models utilize the mean average approach. For post-processing, we filter out small instances and refine instance score using mask scores.\n\n# Cross-Validation and Preprocessing\n**Train-test split**\n - Training: 1549 images (1211 images from ds2 + 338 images from ds1)\n - Validation: 84 images from ds1\n - During the initial stages of approaching the problem (in the final month of the competition), we initially utilized K-fold cross-validation. However, upon recognizing significant differences in distribution, label assignment methodologies between dataset1 and dataset2, as well as variations in label assignments among Whole Slide Images (WSI), we decided to split the data into two train-validation sets. The validation data was exclusively taken from dataset1 (20% of dataset1). Another reason for this data split was the scarcity of dataset1 images in the training set, which was a concern given that the private test data was exclusively sourced from dataset1 (with only 422 images available).\n - To determine the dissimilarities in the label distribution between dataset1 and dataset2, as well as variations among different Whole Slide Images (WSI), we trained a base model, MaskRCNN R50, on the training data from dataset1 (fold1) and validated it on dataset1, dataset2 (fold1). Our observations revealed that the model's performance scored considerably higher on dataset1 compared to dataset2. Similar procedures were applied when comparing performance across various Whole Slide Images (WSI).\n**Preprocessing**: remove duplicate annotations\n - Approximately 4.5% of training data was duplicated. We directly removed these duplicate labels to ensure data integrity and improve model performance.\n\n# Training & valid with blood_vessel & unsure only\nDuring training and validation, we specifically focused on the \"blood_vessel\" and \"unsure\" classes. This decision was influenced by the significant size difference between glomerulus and the other two classes, with glomerulus being much larger. Surprisingly, when training with all three classes, the model's performance on glomerulus remained exceptionally high. Additionally, the competition organizers informed us that during the testing phase, we could utilize glomerulus labels to eliminate false positives. Consequently, we concluded that training with additional glomerulus data was unnecessary. After removing glomerulus labels from the training dataset, we observed an improvement in the performance score for the \"blood_vessel\" class. Therefore, we made the decision to exclude glomerulus labels from all subsequent experiments.\n\n# Models\n**2 stages model**: MaskRCNN, CascadeRCNN, HybridTaskCascade\n- When training with the base model (MaskRCNN-R50), we noticed that the model struggled to converge using the default mmdet configurations. Consequently, in subsequent experiments, we reduced the utilization of augmentation methods, retaining only multiscale training. Additionally, we adopted a larger backbone (ResNeXt-101) and replaced SGD with AdamW optimizer. To further enhance model diversity during ensemble, we incorporated CascadeRCNN and HybridTaskCascade into our approach. These modifications aimed to improve convergence and overall performance of the models.\n\n**Yolov6m**: Two-stage models like MaskRCNN, CascadeRCNN, and HybridTaskCascade exhibit good Mean Average Precision (MAP) at a certain intersection-over-union (IoU) threshold range [0.5:0.95]. However, their overall MAP [0.5:0.95] scores may not reach high values. To address this limitation, YOLO series models offer a solution. Specifically, YOLOv6, with its default training configurations, achieves a significantly higher Mean Average Recall (MAR) of 55.1, compared to the 44.x obtained by the two-stage models. This improvement in MAR highlights the effectiveness of YOLO series models in overcoming the mentioned drawback.\n**Detail training config**\n - Optimizer: AdamW, warmup 3 epoch\n - Select best model use MAP[0.5:0.95] blood_vessel class\n - Data augmentation: Multi scale training: [(512x512), (640,640), (768, 768), (896, 896), (1024, 1024)]\n\n# Training: 2 stages\n- Stage 1: training all 1549 images, 2 classes\n- Stage2: training 338 images ds1, 2 classes\n\nFocusing on optimizing the validation set with dataset1 exclusively, fine-tuning the model for a few epochs using dataset1 in the training set improved performance on the validation data.\n\n\nCongratulations!
\nIn your Ensemble Diagram, you WBF the RPN head output of Detection (HTC, cascade, mask r cnn). Then, it was used as an input for ROI align + Box head. How is this possible in mmdetection?? Is there a notebook or github I can refer to?
", "rawMarkdown": "Congratulations!\n\nIn your Ensemble Diagram, you WBF the RPN head output of Detection (HTC, cascade, mask r cnn). Then, it was used as an input for ROI align + Box head. How is this possible in mmdetection?? Is there a notebook or github I can refer to?", "votes": null }, { "id": "2372805", "postDate": "08/04/2023 01:17:57", "content": "I think there is currently no public code that does that. You need to customize your inference flow. My training and inference code will be prepared and made public soon!
", "rawMarkdown": "I think there is currently no public code that does that. You need to customize your inference flow. My training and inference code will be prepared and made public soon!", "votes": null }, { "id": "2372806", "postDate": "08/04/2023 01:19:41", "content": "Looking forward to it thank you!!
", "rawMarkdown": "Looking forward to it thank you!!", "votes": null }, { "id": "2372994", "postDate": "08/04/2023 05:12:47", "content": "Great job, Looking forward to your training and inference code.
", "rawMarkdown": "Great job, Looking forward to your training and inference code.", "votes": null }, { "id": "2373307", "postDate": "08/04/2023 07:56:26", "content": "@damtrongtuyen The reason why you use 2 step wbf. In your architecture, I check that you only use yolo like a detection model for ensemble wbf after RPN head, the yolo model is't any effect in segmentation mask, right?
", "rawMarkdown": "damtrongtuyen The reason why you use 2 step wbf. In your architecture, I check that you only use yolo like a detection model for ensemble wbf after RPN head, the yolo model is't any effect in segmentation mask, right?", "votes": null }, { "id": "2373359", "postDate": "08/04/2023 08:34:06", "content": "The bbox output of rpn head and box head are different. Yes, we only use mask head from 2 stage models.
", "rawMarkdown": "The bbox output of rpn head and box head are different. Yes, we only use mask head from 2 stage models.", "votes": null }, { "id": "2373396", "postDate": "08/04/2023 08:52:45", "content": "But how did you ensemble RPN bbox with yolo bbox. I think that yolo bbox will be like box head bbox. They have difference size, shape, conf score,…
", "rawMarkdown": "But how did you ensemble RPN bbox with yolo bbox. I think that yolo bbox will be like box head bbox. They have difference size, shape, conf score,...", "votes": null }, { "id": "2373423", "postDate": "08/04/2023 09:07:11", "content": "Great finding! I drew the wrong arrow endpoint. It should be the ensemble bounding box after the boxhead, instead of the rpnhead! Thanks
", "rawMarkdown": "Great finding! I drew the wrong arrow endpoint. It should be the ensemble bounding box after the boxhead, instead of the rpnhead! Thanks", "votes": null }, { "id": "2378417", "postDate": "08/07/2023 16:23:37", "content": "Thanks @damtrongtuyen and ChatGPT for this great report :)
", "rawMarkdown": "Thanks @damtrongtuyen and ChatGPT for this great report :)", "votes": null }, { "id": "2380200", "postDate": "08/08/2023 13:17:13", "content": "Code training yolov6m and 2 stages model : mmdet maskrcnn, cascade, htc
", "rawMarkdown": "Code training [yolov6m](https://www.kaggle.com/code/damtrongtuyen/training-hubmap-yolov6m) and [2 stages model ](https://www.kaggle.com/damtrongtuyen/training-hubmap-2-stages-model): mmdet maskrcnn, cascade, htc", "votes": null }, { "id": "2381125", "postDate": "08/09/2023 02:29:04", "content": "Inference code for ensemble both rpn head and roi head(boxhead)
", "rawMarkdown": "[Inference code for ensemble both rpn head and roi head(boxhead)] (https://github.com/amirassov/kaggle-imaterialist/blob/f1ae37100801203500d20119b9de7e19b0d89a1c/mmdetection/mmdet/models/detectors/ensemble_htc.py)", "votes": null } ], "comments": [ { "id": 2372780, "author_name": "devchopin", "author_url": "", "post_date": "08/04/2023 00:30:48", "content": "Congratulations!
\nIn your Ensemble Diagram, you WBF the RPN head output of Detection (HTC, cascade, mask r cnn). Then, it was used as an input for ROI align + Box head. How is this possible in mmdetection?? Is there a notebook or github I can refer to?
", "votes": null, "replies": [ { "id": 2372805, "author_name": "damtrongtuyen", "author_url": "", "post_date": "08/04/2023 01:17:57", "content": "I think there is currently no public code that does that. You need to customize your inference flow. My training and inference code will be prepared and made public soon!
", "votes": null, "replies": [ { "id": 2372806, "author_name": "devchopin", "author_url": "", "post_date": "08/04/2023 01:19:41", "content": "Looking forward to it thank you!!
", "votes": null, "replies": [] } ] } ] }, { "id": 2372994, "author_name": "allenwpr", "author_url": "", "post_date": "08/04/2023 05:12:47", "content": "Great job, Looking forward to your training and inference code.
", "votes": null, "replies": [] }, { "id": 2373307, "author_name": "hidngnguyna", "author_url": "", "post_date": "08/04/2023 07:56:26", "content": "@damtrongtuyen The reason why you use 2 step wbf. In your architecture, I check that you only use yolo like a detection model for ensemble wbf after RPN head, the yolo model is't any effect in segmentation mask, right?
", "votes": null, "replies": [ { "id": 2373359, "author_name": "damtrongtuyen", "author_url": "", "post_date": "08/04/2023 08:34:06", "content": "The bbox output of rpn head and box head are different. Yes, we only use mask head from 2 stage models.
", "votes": null, "replies": [ { "id": 2373396, "author_name": "hidngnguyna", "author_url": "", "post_date": "08/04/2023 08:52:45", "content": "But how did you ensemble RPN bbox with yolo bbox. I think that yolo bbox will be like box head bbox. They have difference size, shape, conf score,…
", "votes": null, "replies": [ { "id": 2373423, "author_name": "damtrongtuyen", "author_url": "", "post_date": "08/04/2023 09:07:11", "content": "Great finding! I drew the wrong arrow endpoint. It should be the ensemble bounding box after the boxhead, instead of the rpnhead! Thanks
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2378417, "author_name": "vslaykovsky", "author_url": "", "post_date": "08/07/2023 16:23:37", "content": "Thanks @damtrongtuyen and ChatGPT for this great report :)
", "votes": null, "replies": [] }, { "id": 2380200, "author_name": "damtrongtuyen", "author_url": "", "post_date": "08/08/2023 13:17:13", "content": "Code training yolov6m and 2 stages model : mmdet maskrcnn, cascade, htc
", "votes": null, "replies": [] }, { "id": 2381125, "author_name": "damtrongtuyen", "author_url": "", "post_date": "08/09/2023 02:29:04", "content": "Inference code for ensemble both rpn head and roi head(boxhead)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2372420": "Thanks to the organizers for this compelling competition. We present the 4th place solution from the GoN team at SDSRV.AI. It emphasizes the importance of dataset, validation, ensemble and post processing method. Special thanks to @ducmanvo, an amazing teammate.\n\n# Summary\n\nOur solution employs a combination of 4 instance segmentation models comprising CascadeRCNN (ResNeXt, Regnet), MaskRCNN (swint), HybridTaskCascade (Re2Net), and 1 object detection model, Yolov6m. Each model is trained with a dataset consisting of 1549 images (ds1+ds2), with 84 images from dataset1 utilized for validation. During the inference process, the Weighted Boxes Fusion (WBF) technique is employed to ensemble RPN boxes and ROI boxes, while the masks generated by the ensemble of models utilize the mean average approach. For post-processing, we filter out small instances and refine instance score using mask scores.\n\n# Cross-Validation and Preprocessing\n**Train-test split**\n - Training: 1549 images (1211 images from ds2 + 338 images from ds1)\n - Validation: 84 images from ds1\n - During the initial stages of approaching the problem (in the final month of the competition), we initially utilized K-fold cross-validation. However, upon recognizing significant differences in distribution, label assignment methodologies between dataset1 and dataset2, as well as variations in label assignments among Whole Slide Images (WSI), we decided to split the data into two train-validation sets. The validation data was exclusively taken from dataset1 (20% of dataset1). Another reason for this data split was the scarcity of dataset1 images in the training set, which was a concern given that the private test data was exclusively sourced from dataset1 (with only 422 images available).\n - To determine the dissimilarities in the label distribution between dataset1 and dataset2, as well as variations among different Whole Slide Images (WSI), we trained a base model, MaskRCNN R50, on the training data from dataset1 (fold1) and validated it on dataset1, dataset2 (fold1). Our observations revealed that the model's performance scored considerably higher on dataset1 compared to dataset2. Similar procedures were applied when comparing performance across various Whole Slide Images (WSI).\n**Preprocessing**: remove duplicate annotations\n - Approximately 4.5% of training data was duplicated. We directly removed these duplicate labels to ensure data integrity and improve model performance.\n\n# Training & valid with blood_vessel & unsure only\nDuring training and validation, we specifically focused on the \"blood_vessel\" and \"unsure\" classes. This decision was influenced by the significant size difference between glomerulus and the other two classes, with glomerulus being much larger. Surprisingly, when training with all three classes, the model's performance on glomerulus remained exceptionally high. Additionally, the competition organizers informed us that during the testing phase, we could utilize glomerulus labels to eliminate false positives. Consequently, we concluded that training with additional glomerulus data was unnecessary. After removing glomerulus labels from the training dataset, we observed an improvement in the performance score for the \"blood_vessel\" class. Therefore, we made the decision to exclude glomerulus labels from all subsequent experiments.\n\n# Models\n**2 stages model**: MaskRCNN, CascadeRCNN, HybridTaskCascade\n- When training with the base model (MaskRCNN-R50), we noticed that the model struggled to converge using the default mmdet configurations. Consequently, in subsequent experiments, we reduced the utilization of augmentation methods, retaining only multiscale training. Additionally, we adopted a larger backbone (ResNeXt-101) and replaced SGD with AdamW optimizer. To further enhance model diversity during ensemble, we incorporated CascadeRCNN and HybridTaskCascade into our approach. These modifications aimed to improve convergence and overall performance of the models.\n\n**Yolov6m**: Two-stage models like MaskRCNN, CascadeRCNN, and HybridTaskCascade exhibit good Mean Average Precision (MAP) at a certain intersection-over-union (IoU) threshold range [0.5:0.95]. However, their overall MAP [0.5:0.95] scores may not reach high values. To address this limitation, YOLO series models offer a solution. Specifically, YOLOv6, with its default training configurations, achieves a significantly higher Mean Average Recall (MAR) of 55.1, compared to the 44.x obtained by the two-stage models. This improvement in MAR highlights the effectiveness of YOLO series models in overcoming the mentioned drawback.\n**Detail training config**\n - Optimizer: AdamW, warmup 3 epoch\n - Select best model use MAP[0.5:0.95] blood_vessel class\n - Data augmentation: Multi scale training: [(512x512), (640,640), (768, 768), (896, 896), (1024, 1024)]\n\n# Training: 2 stages\n- Stage 1: training all 1549 images, 2 classes\n- Stage2: training 338 images ds1, 2 classes\n\nFocusing on optimizing the validation set with dataset1 exclusively, fine-tuning the model for a few epochs using dataset1 in the training set improved performance on the validation data.\n\n\n