We’d like to thank Kaggle and the competition host for this challenge. We’d also like to thank all the other competitors who shared their wisdom, knowledge, code and critiques. We appreciate you.
\nThanks to my partner @mrinath for his expertise and hard work in putting the whole project together.
\nOur approach is simple and consisted of Object Detection and Classification. We used YOLOv5 for OD and EfficientNet 2D and 3D backbones for classification.
\nWe used only the T1wCE series in the axial plane for object detection and 2D networks. The basic strategy was ..
\nWe tried OD on sagittal and coronal planes but got a better mAP with axials. Our axial mAP was around .68.
\nTraining:
\nWe exported all images into each plane for a total of 12 datasets. Next, we used the YOLO OD models to find images with tumors, which were added to training datasets. The idea was to eliminate duplicate and non-tumor images. We tried about 20 different backbones on each 'sampled' dataset. We found the best results overall (which still weren't very good) were on T1wCE axials and EffNet B3.. so we stuck with that combination with the intent to ensemble the lesser performing models later.
\nPrediction:
\nTraining:
\nFor the 3D model, we knew having pretrained models as we use for other competitions would be beneficial. But most 3D models we were training from MONAI had no pre-trained weights. So we decided to use imagenet pre-trained models for 3D classification. We knew these models are for 2D images, so how can we classify 3D data? In most 2D challenges we have seen people change the last layers and add their own fully connected layer at the end of a pretrained model and change it according to their own. We changed the front part of the model.
\nEffNet B1 was the best backbone model for us.
Augmentations include CentreCropping and adding noise to the 3D data.
SAM optimizer with base Adam optimizer with cosine_with_hard_restarts_schedule_with_warmup
5 fold stratified k-fold.
\nPrediction:
We used our best performing fold for predicting the test, the best validation AUC was 0.6936.
Here's a cleaned up, quick save a version of our final infer notebook. The fully documented notebook will be shared when it's complete.
\nInfer Notebook -> https://www.kaggle.com/mrinath/infer-refactored
\nModel dataset -> https://www.kaggle.com/mrinath/rsna-winning-models
Our CV metrics and LB scores were not improving after a month of experiments. We started to realize the task isn't really possible. Other teams started reporting the same issue, which confirmed the difficulty for us.
\nOur overall assessment is that due to the small dataset, we probably trained on noise or features unrelated to MGMT status .. and some amount of randomness happened to put us in the medal zone. The result isn't clinically useful itself. But we do think the collective knowledge gained in this competition will benefit the community. This is how we learn things.
", "messages": [ { "id": 1550814, "postDate": "2021-10-20T03:42:59.793Z", "content": "We’d like to thank Kaggle and the competition host for this challenge. We’d also like to thank all the other competitors who shared their wisdom, knowledge, code and critiques. We appreciate you.
\nThanks to my partner @mrinath for his expertise and hard work in putting the whole project together.
\nOur approach is simple and consisted of Object Detection and Classification. We used YOLOv5 for OD and EfficientNet 2D and 3D backbones for classification.
\nWe used only the T1wCE series in the axial plane for object detection and 2D networks. The basic strategy was ..
\nWe tried OD on sagittal and coronal planes but got a better mAP with axials. Our axial mAP was around .68.
\nTraining:
\nWe exported all images into each plane for a total of 12 datasets. Next, we used the YOLO OD models to find images with tumors, which were added to training datasets. The idea was to eliminate duplicate and non-tumor images. We tried about 20 different backbones on each 'sampled' dataset. We found the best results overall (which still weren't very good) were on T1wCE axials and EffNet B3.. so we stuck with that combination with the intent to ensemble the lesser performing models later.
\nPrediction:
\nTraining:
\nFor the 3D model, we knew having pretrained models as we use for other competitions would be beneficial. But most 3D models we were training from MONAI had no pre-trained weights. So we decided to use imagenet pre-trained models for 3D classification. We knew these models are for 2D images, so how can we classify 3D data? In most 2D challenges we have seen people change the last layers and add their own fully connected layer at the end of a pretrained model and change it according to their own. We changed the front part of the model.
\nEffNet B1 was the best backbone model for us.
Augmentations include CentreCropping and adding noise to the 3D data.
SAM optimizer with base Adam optimizer with cosine_with_hard_restarts_schedule_with_warmup
5 fold stratified k-fold.
\nPrediction:
We used our best performing fold for predicting the test, the best validation AUC was 0.6936.
Here's a cleaned up, quick save a version of our final infer notebook. The fully documented notebook will be shared when it's complete.
\nInfer Notebook -> https://www.kaggle.com/mrinath/infer-refactored
\nModel dataset -> https://www.kaggle.com/mrinath/rsna-winning-models
Our CV metrics and LB scores were not improving after a month of experiments. We started to realize the task isn't really possible. Other teams started reporting the same issue, which confirmed the difficulty for us.
\nOur overall assessment is that due to the small dataset, we probably trained on noise or features unrelated to MGMT status .. and some amount of randomness happened to put us in the medal zone. The result isn't clinically useful itself. But we do think the collective knowledge gained in this competition will benefit the community. This is how we learn things.
", "rawMarkdown": "We’d like to thank Kaggle and the competition host for this challenge. We’d also like to thank all the other competitors who shared their wisdom, knowledge, code and critiques. We appreciate you.\n\nThanks to my partner @mrinath for his expertise and hard work in putting the whole project together.\n\nOur approach is simple and consisted of Object Detection and Classification. We used YOLOv5 for OD and EfficientNet 2D and 3D backbones for classification.\n\nWe used only the T1wCE series in the axial plane for object detection and 2D networks. The basic strategy was ..\n\n### 1. Extract all images into the same plane\n- Resample coronal and sagittals to axial. Thanks to @boojum for his nifty resampling method. -> https://www.kaggle.com/boojum/connecting-voxel-spaces\n- Eliminate empty or mostly empty images (mean pix value < 40)\n- Choose seven equally spaced slices out of the series.\n- Find the center slice, two slices inferior and three slices superior to center (since most of the brain is above the center slice in the axial plane).\n\n### 2. Find slices with tumors.\n- We trained a YOLO Object Detection model on ~400 hand-annotated images from the RSNA-MICCIA training dataset and used it to find slices with tumors.\nOD Train Notebook -> https://www.kaggle.com/davidbroberts/brain-tumor-yolo-od-train\nOD Train Dataset -> https://www.kaggle.com/davidbroberts/brain-tumor-object-detection-datasets\n\nWe tried OD on sagittal and coronal planes but got a better mAP with axials. Our axial mAP was around .68.\n\n### 3. 2D Classification\n\nTraining:\n\nWe exported all images into each plane for a total of 12 datasets. Next, we used the YOLO OD models to find images with tumors, which were added to training datasets. The idea was to eliminate duplicate and non-tumor images. We tried about 20 different backbones on each 'sampled' dataset. We found the best results overall (which still weren't very good) were on T1wCE axials and EffNet B3.. so we stuck with that combination with the intent to ensemble the lesser performing models later.\n\n- Train-Val random split with 25%data in val set. \n- Using optimizer AdamW and used cosine_schedule_with_warmup as the LR scheduler.\n- Augmentations include : CenterCrop,CLAHE,RandomRotate90,HorizontalFlip,VerticalFlip,RandomContrast and Cutmix\n\nPrediction:\n\n- Images that YOLO detected tumours on were passed to EffNet classifiers (1-7 images per study).\n- If multiple slices from the same case are detected, the first and the last slice is used.\n- The best validation AUC was 0.6829.\n- We used 4x TTA and power ensembling for the final prediction, it was better than plain averaging.\n- If after TTA we had 4 prediction sets y1, y2, y3, y4 then y_final = (y1² +y2² +y3² +y4²)/4\nFor studies with no usable T1wCE images or no tumours detected by OD, we globbed all the series together and passed them to 3D network.\n\n### 4. 3D Classification\n\nTraining:\n\nFor the 3D model, we knew having pretrained models as we use for other competitions would be beneficial. But most 3D models we were training from MONAI had no pre-trained weights. So we decided to use imagenet pre-trained models for 3D classification. We knew these models are for 2D images, so how can we classify 3D data? In most 2D challenges we have seen people change the last layers and add their own fully connected layer at the end of a pretrained model and change it according to their own. We changed the front part of the model.\n\n- EffNet B1 was the best backbone model for us.\n- Augmentations include CentreCropping and adding noise to the 3D data.\n- SAM optimizer with base Adam optimizer with cosine_with_hard_restarts_schedule_with_warmup\n\n- 5 fold stratified k-fold.\nPrediction:\n\n- We used our best performing fold for predicting the test, the best validation AUC was 0.6936.\n\n### Tools/Frameworks:\n- Hardware used: RTX 3090, Google Colab Pro and of course, Kaggle.\n- Model metric tracking: Weights & Biases (https://wandb.ai)\n- Framework: PyTorch\n- Pretrained models: Timm\n- Object detection annotation: https://makesense.ai\n\n### Things that didn't work (or help):\n\n- Object detection on T2w series\n- Mixup augmentation\nYOLO classification (likely because of a small dataset of ~400 images)\n- Contrast filtering ( hist EQ, manual LUT etc)\n- Aux Loss\nWe were inspired by the COVID-19 Detection winning solutions where many people worked with aux loss. We created an extra head in our 2D models to segment the tumour. Although it was promising during training, the scores were not good on the Public LB.\n- Using 3D models with no pretrained weights\n\n### Notebook:\n\nHere's a cleaned up, quick save a version of our final infer notebook. The fully documented notebook will be shared when it's complete.\n\nInfer Notebook -> https://www.kaggle.com/mrinath/infer-refactored\nModel dataset -> https://www.kaggle.com/mrinath/rsna-winning-models\n\nOur CV metrics and LB scores were not improving after a month of experiments. We started to realize the task isn't really possible. Other teams started reporting the same issue, which confirmed the difficulty for us.\n\nOur overall assessment is that due to the small dataset, we probably trained on noise or features unrelated to MGMT status .. and some amount of randomness happened to put us in the medal zone. The result isn't clinically useful itself. But we do think the collective knowledge gained in this competition will benefit the community. This is how we learn things.", "votes": 33 }, { "id": 1556700, "postDate": "2021-10-25T05:26:25.473Z", "content": "@davidbroberts Congrats!
\nI agree with you that the results are probably not useful clinically.
\nWhy did you decide to go for the detection approach, when most other people used segmentation?
We also experimented with segmentation like you said, as an auxiliary loss, but it was not giving fruitful results in the Public leaderboard.
\nIt was based on the ideas of the winners of covid detection competition.
\nAlthough we saw, we had to regularize very much like higher dropout rates and a very small learning rate to make the segmentation work.
Congrats @davidbroberts and @mrinath
", "rawMarkdown": "Congrats @davidbroberts and @mrinath ", "votes": 1 }, { "id": 1551669, "postDate": "2021-10-20T20:01:00.913Z", "content": "@davidbroberts thank you for your insights and the description!
", "rawMarkdown": "@davidbroberts thank you for your insights and the description! ", "votes": 1 }, { "id": 1550861, "postDate": "2021-10-20T05:29:49.847Z", "content": "If you are thinking about why there are so many 0.25 in the submission CSV, this is because we are only inferring on 3 test studies when committing, but when we submit it checks for all the test cases.
\nWe did this because the YOLO detection was taking very much time(even on only public data it took almost 45 minutes) so we did this
\nIn the infer notebook, we have done this like
if( len(test_studies) > 87):\n for study in test_studies:\n print(\"TESTING STUDY: \", study)\n test_study(study)\nelse:\n test_studies = test_studies[:3]\n for study in test_studies:\n test_study(study)\nGood to see your team here, you guys worked hard from the start of the competition. Congrats @davidbroberts and @mrinath and thanks for sharing solution.
", "rawMarkdown": "Good to see your team here, you guys worked hard from the start of the competition. Congrats @davidbroberts and @mrinath and thanks for sharing solution.", "votes": 2 }, { "id": 1550918, "postDate": "2021-10-20T06:40:59.673Z", "content": "Congratulations and thanks for the many shares from you.
", "rawMarkdown": "Congratulations and thanks for the many shares from you.", "votes": 2 }, { "id": 2243426, "postDate": "2023-05-02T22:32:55.743Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1557374, "postDate": "2021-10-25T16:11:23.137Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1553292, "postDate": "2021-10-22T03:55:32.870Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1556700, "author_name": "Pranshu15", "author_url": "", "post_date": "2021-10-25T05:26:25.473000", "content": "@davidbroberts Congrats!
\nI agree with you that the results are probably not useful clinically.
\nWhy did you decide to go for the detection approach, when most other people used segmentation?
We also experimented with segmentation like you said, as an auxiliary loss, but it was not giving fruitful results in the Public leaderboard.
\nIt was based on the ideas of the winners of covid detection competition.
\nAlthough we saw, we had to regularize very much like higher dropout rates and a very small learning rate to make the segmentation work.
Congrats @davidbroberts and @mrinath
", "votes": 1, "replies": [] }, { "id": 1551669, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-20T20:01:00.913000", "content": "@davidbroberts thank you for your insights and the description!
", "votes": 1, "replies": [] }, { "id": 1550861, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-10-20T05:29:49.847000", "content": "If you are thinking about why there are so many 0.25 in the submission CSV, this is because we are only inferring on 3 test studies when committing, but when we submit it checks for all the test cases.
\nWe did this because the YOLO detection was taking very much time(even on only public data it took almost 45 minutes) so we did this
\nIn the infer notebook, we have done this like
if( len(test_studies) > 87):\n for study in test_studies:\n print(\"TESTING STUDY: \", study)\n test_study(study)\nelse:\n test_studies = test_studies[:3]\n for study in test_studies:\n test_study(study)\nGood to see your team here, you guys worked hard from the start of the competition. Congrats @davidbroberts and @mrinath and thanks for sharing solution.
", "votes": 2, "replies": [] }, { "id": 1550918, "author_name": "atfujita", "author_url": "", "post_date": "2021-10-20T06:40:59.673000", "content": "Congratulations and thanks for the many shares from you.
", "votes": 2, "replies": [] }, { "id": 2243426, "author_name": "", "author_url": "", "post_date": "2023-05-02T22:32:55.743000", "content": "", "votes": 0, "replies": [] }, { "id": 1557374, "author_name": "", "author_url": "", "post_date": "2021-10-25T16:11:23.137000", "content": "", "votes": 1, "replies": [] }, { "id": 1553292, "author_name": "", "author_url": "", "post_date": "2021-10-22T03:55:32.870000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1550814": "We’d like to thank Kaggle and the competition host for this challenge. We’d also like to thank all the other competitors who shared their wisdom, knowledge, code and critiques. We appreciate you.\n\nThanks to my partner @mrinath for his expertise and hard work in putting the whole project together.\n\nOur approach is simple and consisted of Object Detection and Classification. We used YOLOv5 for OD and EfficientNet 2D and 3D backbones for classification.\n\nWe used only the T1wCE series in the axial plane for object detection and 2D networks. The basic strategy was ..\n\n### 1. Extract all images into the same plane\n- Resample coronal and sagittals to axial. Thanks to @boojum for his nifty resampling method. -> https://www.kaggle.com/boojum/connecting-voxel-spaces\n- Eliminate empty or mostly empty images (mean pix value < 40)\n- Choose seven equally spaced slices out of the series.\n- Find the center slice, two slices inferior and three slices superior to center (since most of the brain is above the center slice in the axial plane).\n\n### 2. Find slices with tumors.\n- We trained a YOLO Object Detection model on ~400 hand-annotated images from the RSNA-MICCIA training dataset and used it to find slices with tumors.\nOD Train Notebook -> https://www.kaggle.com/davidbroberts/brain-tumor-yolo-od-train\nOD Train Dataset -> https://www.kaggle.com/davidbroberts/brain-tumor-object-detection-datasets\n\nWe tried OD on sagittal and coronal planes but got a better mAP with axials. Our axial mAP was around .68.\n\n### 3. 2D Classification\n\nTraining:\n\nWe exported all images into each plane for a total of 12 datasets. Next, we used the YOLO OD models to find images with tumors, which were added to training datasets. The idea was to eliminate duplicate and non-tumor images. We tried about 20 different backbones on each 'sampled' dataset. We found the best results overall (which still weren't very good) were on T1wCE axials and EffNet B3.. so we stuck with that combination with the intent to ensemble the lesser performing models later.\n\n- Train-Val random split with 25%data in val set. \n- Using optimizer AdamW and used cosine_schedule_with_warmup as the LR scheduler.\n- Augmentations include : CenterCrop,CLAHE,RandomRotate90,HorizontalFlip,VerticalFlip,RandomContrast and Cutmix\n\nPrediction:\n\n- Images that YOLO detected tumours on were passed to EffNet classifiers (1-7 images per study).\n- If multiple slices from the same case are detected, the first and the last slice is used.\n- The best validation AUC was 0.6829.\n- We used 4x TTA and power ensembling for the final prediction, it was better than plain averaging.\n- If after TTA we had 4 prediction sets y1, y2, y3, y4 then y_final = (y1² +y2² +y3² +y4²)/4\nFor studies with no usable T1wCE images or no tumours detected by OD, we globbed all the series together and passed them to 3D network.\n\n### 4. 3D Classification\n\nTraining:\n\nFor the 3D model, we knew having pretrained models as we use for other competitions would be beneficial. But most 3D models we were training from MONAI had no pre-trained weights. So we decided to use imagenet pre-trained models for 3D classification. We knew these models are for 2D images, so how can we classify 3D data? In most 2D challenges we have seen people change the last layers and add their own fully connected layer at the end of a pretrained model and change it according to their own. We changed the front part of the model.\n\n- EffNet B1 was the best backbone model for us.\n- Augmentations include CentreCropping and adding noise to the 3D data.\n- SAM optimizer with base Adam optimizer with cosine_with_hard_restarts_schedule_with_warmup\n\n- 5 fold stratified k-fold.\nPrediction:\n\n- We used our best performing fold for predicting the test, the best validation AUC was 0.6936.\n\n### Tools/Frameworks:\n- Hardware used: RTX 3090, Google Colab Pro and of course, Kaggle.\n- Model metric tracking: Weights & Biases (https://wandb.ai)\n- Framework: PyTorch\n- Pretrained models: Timm\n- Object detection annotation: https://makesense.ai\n\n### Things that didn't work (or help):\n\n- Object detection on T2w series\n- Mixup augmentation\nYOLO classification (likely because of a small dataset of ~400 images)\n- Contrast filtering ( hist EQ, manual LUT etc)\n- Aux Loss\nWe were inspired by the COVID-19 Detection winning solutions where many people worked with aux loss. We created an extra head in our 2D models to segment the tumour. Although it was promising during training, the scores were not good on the Public LB.\n- Using 3D models with no pretrained weights\n\n### Notebook:\n\nHere's a cleaned up, quick save a version of our final infer notebook. The fully documented notebook will be shared when it's complete.\n\nInfer Notebook -> https://www.kaggle.com/mrinath/infer-refactored\nModel dataset -> https://www.kaggle.com/mrinath/rsna-winning-models\n\nOur CV metrics and LB scores were not improving after a month of experiments. We started to realize the task isn't really possible. Other teams started reporting the same issue, which confirmed the difficulty for us.\n\nOur overall assessment is that due to the small dataset, we probably trained on noise or features unrelated to MGMT status .. and some amount of randomness happened to put us in the medal zone. The result isn't clinically useful itself. But we do think the collective knowledge gained in this competition will benefit the community. This is how we learn things.", "1556700": "@davidbroberts Congrats!\nI agree with you that the results are probably not useful clinically. \nWhy did you decide to go for the detection approach, when most other people used segmentation?", "1561438": "Congrats @davidbroberts and @mrinath ", "1551669": "@davidbroberts thank you for your insights and the description! ", "1550861": "If you are thinking about why there are so many 0.25 in the submission CSV, this is because we are only inferring on 3 test studies when committing, but when we submit it checks for all the test cases.\nWe did this because the YOLO detection was taking very much time(even on only public data it took almost 45 minutes) so we did this\nIn the infer notebook, we have done this like\n\n```\nif( len(test_studies) > 87):\n for study in test_studies:\n print(\"TESTING STUDY: \", study)\n test_study(study)\nelse:\n test_studies = test_studies[:3]\n for study in test_studies:\n test_study(study)\n```", "1551176": "Good to see your team here, you guys worked hard from the start of the competition. Congrats @davidbroberts and @mrinath and thanks for sharing solution.", "1550918": "Congratulations and thanks for the many shares from you.", "2243426": "", "1557374": "", "1553292": "" } }