{ "id": 281347, "title": "1st place solution with very simple code", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/281347", "author_name": "Firas Baba", "post_date": "2021-10-24T16:48:52.059000", "votes": 103, "comment_count": 27, "views": 0, "content": "

First of all congratulations to all the winners! Thanks as well to Kaggle and RSNA for hosting this competition and giving us the chance to work on such an interesting problem.

\n

On the contrary of what top solutions look like, my final solution was one of the very first baselines I started with. There was no model ensembling, no complex/big models, and no sophisticated training techniques.

\n

Team name secret:

\n

As many of you have seen, the team name was set to “I hate this competition” during the last month and I will explain below the reasons that made me hate spending time on this competition.

\n\n

What I concluded after a whole week doing EDA:

\n\n

Very slow validation strategy:

\n

Training the same model with the same everything (parameters, hardware, fixing all random seeds), with the same training data and the same validation set will give different aucroc scores. I remember that when I trained a EF-b0 more than 100 times (same everything) I got CV scores between 0.53 and 0.62 with a high std (can’t remember it). Training 5 folds 100 times, reduced drastically the std of the scores and I got scores (after averaging the 5 folds) between 0.52 and 0.56.
\nGuess what? To judge any model/idea/approach I used to train 100 models (Every model is trained 20 times X 5folds). I ranked the experiments based on the scores average. Then, I selected the top 5 ideas+models and I ran 250 models per idea (Every model is trained 50 times X 5folds. The folds in the second stage are different from the first stage folds). After this, I re-ranked the ideas based on the average score for the 250 models (I do the oof of every idea/model and I average the 50 oofs).
\nPlease note that some of what I called models in this section represent 4 trained models (on \"FLAIR\", \"T1w\", \"T1wCE\", \"T2w”). Whenever I want to try an idea I apply it to the CNN model using all the 4 types data and I train 4 different models and I average the 4 models. I have excluded the “T2w” data in most of the final models (I think that more than 50% of the models were not using the “T2w” data). Anyway, let’s skip this part because the ideas and models I was trying were totally random. In fact, most of the ideas I tried were just ideas I get when I close my eyes to sleep. I have to be honest, this competition ruined many of my nights and made me feel like a stupid loser.

\n

What was the top one idea after the 2 stage ranking?

\n

I am not sure but I remember that I started with 8 simple ideas and then I tried some sophisticated and customized models (around 20).
\nThe top 2 models after applying the 2 stage validation filtering were using the same model and same training techniques but the top 1 was using all of the 4 different structural multi-parametric MRI data (FLAIR, t1w, t1wce, t2w) while the second top model was just using the “T1wCE” data.

\n

The final model:

\n\n

The best central image trick:

\n

Each independent case has a different number of images for all the MRI scans. Using all the scans will confuse the model to learn the spatial dependence of the brain pixels.
\nFor example, Let’s assume that case_1 has 80 T1wCE scans and case_2 has 500 T1wCE scans and that we will use 40 images as an input for the model.
\nWhat people did, is that they take the central image (aka image number 40 (80//2) for case_1 and image number 250 (500//2) in case_2) and then they build the 3d images this way:
\ncase_1: from image number 20 to image number 60
\ncase_2: from image number 480 to image number 520.
\nIn this example, we will end up with 2 3d images that don’t represent the same portion of the brain. Thus, the model will not only fail to learn the tumor pattern but will also start learning some spatial patterns that are not useful in our case.
\nWhat I wanted to do and failed is to select a fixed starting and ending point for all the brains and train with the same information for all the cases. But, I couldn't find a way to successfully make it work.
\nAfter many failures, I found that using the biggest image as a central image (the image that contains the largest brain cutaway view) will slightly improve my local CV (improvements were between 0.01 and 0.02). I think that this was the only 100% successful experiment I did in this competition. It is not exactly what I wanted, but it kinda worked.

\n

What did not work:

\n\n

One funny fact:

\n

I forgot to select my best submissions since I am on vacation and I forgot about the last day of the competition. The cool thing is that my best 2 models had the best public LB score and they were automatically selected as my final submissions :D

\n

One quick final thought:

\n

I did not enjoy making submissions in this competition as much as I enjoyed the private LB results which explains the fact that the team name was changed from “I hate this competition” to “I love this competition”. The data was too small and the models were barely learning, I am sure that if everyone re-runs his best model again we will still have a small shakeup in the top teams. Finally, I feel super happy to have had my first first-place finish and my first solo gold and I apologize for not interacting with the community in the discussion forum. See you in the next competitions :D

\n

Code

\n

Training code is available here
\nThe winner inference notebook can be found here

", "messages": [ { "id": 1556179, "postDate": "2021-10-24T16:48:52.060Z", "content": "

First of all congratulations to all the winners! Thanks as well to Kaggle and RSNA for hosting this competition and giving us the chance to work on such an interesting problem.

\n

On the contrary of what top solutions look like, my final solution was one of the very first baselines I started with. There was no model ensembling, no complex/big models, and no sophisticated training techniques.

\n

Team name secret:

\n

As many of you have seen, the team name was set to “I hate this competition” during the last month and I will explain below the reasons that made me hate spending time on this competition.

\n\n

What I concluded after a whole week doing EDA:

\n\n

Very slow validation strategy:

\n

Training the same model with the same everything (parameters, hardware, fixing all random seeds), with the same training data and the same validation set will give different aucroc scores. I remember that when I trained a EF-b0 more than 100 times (same everything) I got CV scores between 0.53 and 0.62 with a high std (can’t remember it). Training 5 folds 100 times, reduced drastically the std of the scores and I got scores (after averaging the 5 folds) between 0.52 and 0.56.
\nGuess what? To judge any model/idea/approach I used to train 100 models (Every model is trained 20 times X 5folds). I ranked the experiments based on the scores average. Then, I selected the top 5 ideas+models and I ran 250 models per idea (Every model is trained 50 times X 5folds. The folds in the second stage are different from the first stage folds). After this, I re-ranked the ideas based on the average score for the 250 models (I do the oof of every idea/model and I average the 50 oofs).
\nPlease note that some of what I called models in this section represent 4 trained models (on \"FLAIR\", \"T1w\", \"T1wCE\", \"T2w”). Whenever I want to try an idea I apply it to the CNN model using all the 4 types data and I train 4 different models and I average the 4 models. I have excluded the “T2w” data in most of the final models (I think that more than 50% of the models were not using the “T2w” data). Anyway, let’s skip this part because the ideas and models I was trying were totally random. In fact, most of the ideas I tried were just ideas I get when I close my eyes to sleep. I have to be honest, this competition ruined many of my nights and made me feel like a stupid loser.

\n

What was the top one idea after the 2 stage ranking?

\n

I am not sure but I remember that I started with 8 simple ideas and then I tried some sophisticated and customized models (around 20).
\nThe top 2 models after applying the 2 stage validation filtering were using the same model and same training techniques but the top 1 was using all of the 4 different structural multi-parametric MRI data (FLAIR, t1w, t1wce, t2w) while the second top model was just using the “T1wCE” data.

\n

The final model:

\n\n

The best central image trick:

\n

Each independent case has a different number of images for all the MRI scans. Using all the scans will confuse the model to learn the spatial dependence of the brain pixels.
\nFor example, Let’s assume that case_1 has 80 T1wCE scans and case_2 has 500 T1wCE scans and that we will use 40 images as an input for the model.
\nWhat people did, is that they take the central image (aka image number 40 (80//2) for case_1 and image number 250 (500//2) in case_2) and then they build the 3d images this way:
\ncase_1: from image number 20 to image number 60
\ncase_2: from image number 480 to image number 520.
\nIn this example, we will end up with 2 3d images that don’t represent the same portion of the brain. Thus, the model will not only fail to learn the tumor pattern but will also start learning some spatial patterns that are not useful in our case.
\nWhat I wanted to do and failed is to select a fixed starting and ending point for all the brains and train with the same information for all the cases. But, I couldn't find a way to successfully make it work.
\nAfter many failures, I found that using the biggest image as a central image (the image that contains the largest brain cutaway view) will slightly improve my local CV (improvements were between 0.01 and 0.02). I think that this was the only 100% successful experiment I did in this competition. It is not exactly what I wanted, but it kinda worked.

\n

What did not work:

\n\n

One funny fact:

\n

I forgot to select my best submissions since I am on vacation and I forgot about the last day of the competition. The cool thing is that my best 2 models had the best public LB score and they were automatically selected as my final submissions :D

\n

One quick final thought:

\n

I did not enjoy making submissions in this competition as much as I enjoyed the private LB results which explains the fact that the team name was changed from “I hate this competition” to “I love this competition”. The data was too small and the models were barely learning, I am sure that if everyone re-runs his best model again we will still have a small shakeup in the top teams. Finally, I feel super happy to have had my first first-place finish and my first solo gold and I apologize for not interacting with the community in the discussion forum. See you in the next competitions :D

\n

Code

\n

Training code is available here
\nThe winner inference notebook can be found here

", "rawMarkdown": "First of all congratulations to all the winners! Thanks as well to Kaggle and RSNA for hosting this competition and giving us the chance to work on such an interesting problem.\n\nOn the contrary of what top solutions look like, my final solution was one of the very first baselines I started with. There was no model ensembling, no complex/big models, and no sophisticated training techniques.\n\n### **Team name secret:**\nAs many of you have seen, the team name was set to “I hate this competition” during the last month and I will explain below the reasons that made me hate spending time on this competition.\n- In every competition, I always start with a very simple model and then submit it. After this, I try to use different validation strategies in order to find the best strategy that really reflects the model performance. What made me upset about this competition is that the CV score was almost random in my early models and that no strategy worked to validate the performance of the model.\n- Bigger models were giving noisy CV scores (almost random) and I couldn’t fine-tune any of them. I only limited my models to resnet10-50 and b0-b3 during all my experiments (large resents, densents and efficientnets failed).\n- The batch norm layer in this Efficientnet implementation [here](https://github.com/lukemelas/EfficientNet-PyTorch) was giving non-sense output when I used 3 spatial inputs for the CNN. I found that other people suffered from this problem after searching about this issue and I felt disappointed since many of my experiments were just a total failure. I switched to **monai** library at that time.\n- Ensembling was not really improving the score. Once I change the folds, the scores completely change and the std of the scores was so high.\n- Seeing people getting 0.8+ on the public LB while my models were stuck at around 0.5-0.6 was annoying for me.\n- I was certain and 100% sure that most of the public LB scores were just random scores and that all of the scores will drop in the private LB.\n- I saw someone in the discussion forum who shared that he/she got 0.7+ in the public LB by just using random predictions.\nI was reading the discussions every day and I was really amazed by how many people were targeting 0.7 and 0.8+ scores. This seemed like an impossible mission for me. My utmost ambition was getting a model that scores 0.6+ on validation and public LB (which I failed to get even after the competition ended). \n\n### **What I concluded after a whole week doing EDA:**\n- Public LB does make sense if you don’t overfit your validation set.\n- Most of the public models fail to learn any useful patterns.\n- Many teams are just focusing on improving their Public scores and didn’t even take into consideration the shakeup that could happen at the end of the competition.\n- There will be no escape from the shakeup.\n- It will be very easy to get a top100 position in the private LB with a simple model because many many competitors only focus on the public score.\n- Teaming up (in order to ensemble different models) won’t help to improve any score.\n- I should not invest more than 2 weeks working on this competition because the risk is so high and I cannot guarantee to get any medal. \n- The chances to be in the top 1% or in the middle of the LB (500-700) are almost the same.\n\n### **Very slow validation strategy:**\nTraining the same model with the same everything (parameters, hardware, fixing all random seeds), with the same training data and the same validation set will give different aucroc scores. I remember that when I trained a EF-b0 more than 100 times (same everything) I got CV scores between 0.53 and 0.62 with a high std (can’t remember it). Training 5 folds 100 times, reduced drastically the std of the scores and I got scores (after averaging the 5 folds) between 0.52 and 0.56.\nGuess what? To judge any model/idea/approach I used to train 100 models (Every model is trained 20 times X 5folds). I ranked the experiments based on the scores average. Then, I selected the top 5 ideas+models and I ran 250 models per idea (Every model is trained 50 times X 5folds. The folds in the second stage are different from the first stage folds). After this, I re-ranked the ideas based on the average score for the 250 models (I do the oof of every idea/model and I average the 50 oofs).\nPlease note that some of what I called models in this section represent 4 trained models (on \"FLAIR\", \"T1w\", \"T1wCE\", \"T2w”). Whenever I want to try an idea I apply it to the CNN model using all the 4 types data and I train 4 different models and I average the 4 models. I have excluded the “T2w” data in most of the final models (I think that more than 50% of the models were not using the “T2w” data). Anyway, let’s skip this part because the ideas and models I was trying were totally random. In fact, most of the ideas I tried were just ideas I get when I close my eyes to sleep. I have to be honest, this competition ruined many of my nights and made me feel like a stupid loser.\n\n### **What was the top one idea after the 2 stage ranking?**\n\nI am not sure but I remember that I started with 8 simple ideas and then I tried some sophisticated and customized models (around 20).\nThe top 2 models after applying the 2 stage validation filtering were using the same model and same training techniques but the top 1 was using all of the 4 different structural multi-parametric MRI data (FLAIR, t1w, t1wce, t2w) while the second top model was just using the “T1wCE” data.\n\n### **The final model:**\n- 3D CNN\n- Resnet10\n- BCE loss\n- Adam optimizer\n- 15 epochs\n- LR: epoch 1->10; lr = 0.0001 | epoch 10 to 15 lr=0.00005\n- Image size: 256x256\n- Batch size: 8 (the bigger bs I use the worse CV I get, I was alternating between bs=4 and bs=8)\n- No mixed-precision is used.\n- Used a small trick to build the 3D images. Let’s call it “The best central image trick”.\n- One epoch takes around 1minute and 20 seconds using an RTX 3090.\n\n### **The best central image trick:**\nEach independent case has a different number of images for all the MRI scans. Using all the scans will confuse the model to learn the spatial dependence of the brain pixels.\nFor example, Let’s assume that case_1 has 80 T1wCE scans and case_2 has 500 T1wCE scans and that we will use 40 images as an input for the model.\nWhat people did, is that they take the central image (aka image number 40 (80//2) for case_1 and image number 250 (500//2) in case_2) and then they build the 3d images this way:\ncase_1: from image number 20 to image number 60\ncase_2: from image number 480 to image number 520.\nIn this example, we will end up with 2 3d images that don’t represent the same portion of the brain. Thus, the model will not only fail to learn the tumor pattern but will also start learning some spatial patterns that are not useful in our case.\nWhat I wanted to do and failed is to select a fixed starting and ending point for all the brains and train with the same information for all the cases. But, I couldn't find a way to successfully make it work.\nAfter many failures, I found that using the biggest image as a central image (the image that contains the largest brain cutaway view) will slightly improve my local CV (improvements were between 0.01 and 0.02). I think that this was the only 100% successful experiment I did in this competition. It is not exactly what I wanted, but it kinda worked.\n\n### **What did not work:**\n- 2d CNNs\n- 4 backbones 3D CNNs (each for one structural multi-parametric MRI folder)\n- Ensembling\n- Pretrained on brain images\n- Using the metadata from the DCIM images\n- Stacking the output from the different CNNs using based tree models.\n- Deep CNNs\n- Some tricks I did to normalize the voxels in some consistent ways. (It has been more than a month since I did that and I now think that what I did seems completely stupid lol)\n\n### **One funny fact:**\nI forgot to select my best submissions since I am on vacation and I forgot about the last day of the competition. The cool thing is that my best 2 models had the best public LB score and they were automatically selected as my final submissions :D\n\n### **One quick final thought:**\nI did not enjoy making submissions in this competition as much as I enjoyed the private LB results which explains the fact that the team name was changed from “I hate this competition” to “I love this competition”. The data was too small and the models were barely learning, I am sure that if everyone re-runs his best model again we will still have a small shakeup in the top teams. Finally, I feel super happy to have had my first first-place finish and my first solo gold and I apologize for not interacting with the community in the discussion forum. See you in the next competitions :D \n\n### **Code**\nTraining code is available [here](https://github.com/FirasBaba/rsna-resnet10)\nThe winner inference notebook can be found [here](https://www.kaggle.com/rinnqd/monai-simple-prediction-from-flair)", "votes": 103 }, { "id": 1556663, "postDate": "2021-10-25T04:27:16.193Z", "content": "

@rinnqd Congrats!
\nDid you check any GradCAM activations of your CNNs to see on what basis it is making the decisions? I am curious as even trained radiologists don't know what to look for to find MGMT promoter.

", "rawMarkdown": "@rinnqd Congrats! \nDid you check any GradCAM activations of your CNNs to see on what basis it is making the decisions? I am curious as even trained radiologists don't know what to look for to find MGMT promoter.", "votes": 5, "replies": [ { "id": 1568458, "postDate": "2021-11-02T19:11:01.730Z", "content": "

I didn't try to explain the model to be honest. I think that explaining the output from a 3D matrix will take forever which is not worth it.

", "rawMarkdown": "I didn't try to explain the model to be honest. I think that explaining the output from a 3D matrix will take forever which is not worth it. " } ] }, { "id": 1565902, "postDate": "2021-10-31T06:47:35.200Z", "content": "

Awesome approach and write up, Congratulations @rinnqd for your solo gold medal!

", "rawMarkdown": "Awesome approach and write up, Congratulations @rinnqd for your solo gold medal!", "votes": 1 }, { "id": 1563198, "postDate": "2021-10-28T05:59:08.247Z", "content": "

Very well explained. This was a well formulated approach. Congrats

", "rawMarkdown": "Very well explained. This was a well formulated approach. Congrats", "votes": 1 }, { "id": 1561435, "postDate": "2021-10-27T14:28:34.613Z", "content": "

Congrats @rinnqd

", "rawMarkdown": "Congrats @rinnqd ", "votes": 1 }, { "id": 1561149, "postDate": "2021-10-27T11:43:01.610Z", "content": "

Congratulations on the competition! Thanks for not only sharing what worked for you but also what you tried that didn't work so well.

", "rawMarkdown": "Congratulations on the competition! Thanks for not only sharing what worked for you but also what you tried that didn't work so well.", "votes": 1 }, { "id": 1559443, "postDate": "2021-10-27T02:45:26.943Z", "content": "

Nice Approach, Congratulations @rinnqd

", "rawMarkdown": "Nice Approach, Congratulations @rinnqd ", "votes": 1 }, { "id": 1558094, "postDate": "2021-10-26T03:55:53.680Z", "content": "

Congrats, Baba 🙌

", "rawMarkdown": "Congrats, Baba 🙌", "votes": 1 }, { "id": 1557956, "postDate": "2021-10-25T23:54:46.490Z", "content": "

Congrats!! I enjoyed this read, it is SO frustrating when you're trying all these different methods to improve your score and yet they all make it worse I feel that.

", "rawMarkdown": "Congrats!! I enjoyed this read, it is SO frustrating when you're trying all these different methods to improve your score and yet they all make it worse I feel that.", "votes": 1 }, { "id": 1556851, "postDate": "2021-10-25T07:10:23.357Z", "content": "

Congratulations.

", "rawMarkdown": "Congratulations. ", "votes": 1 }, { "id": 1556659, "postDate": "2021-10-25T04:23:08.463Z", "content": "

congratulations

", "rawMarkdown": "congratulations", "votes": 1 }, { "id": 1556631, "postDate": "2021-10-25T03:51:48.937Z", "content": "

Great approach! Nice and simple.

", "rawMarkdown": "Great approach! Nice and simple.", "votes": 1 }, { "id": 1556598, "postDate": "2021-10-25T02:58:31.877Z", "content": "

@rinnqd You are Awesome,we are really proud of you,
\nbut what is most intersting is the simplicity of the models ,i mean, since the start i was expecting from the first winners a terrifying ensemble of 300 models that no one hear of ,which didn't hapen i think that due to the poor weird small dataset (it's sad , espacily if it can literally save many lives) also i want to refer to the using of \"The best central image trick\" that was a cleaver and yet simple .
\nThis is surely one of the most interesting competitions i have ever wittnessed and your solution was even more interesting.
\nGood Job again !

", "rawMarkdown": "@rinnqd You are Awesome,we are really proud of you,\nbut what is most intersting is the simplicity of the models ,i mean, since the start i was expecting from the first winners a terrifying ensemble of 300 models that no one hear of ,which didn't hapen i think that due to the poor weird small dataset (it's sad , espacily if it can literally save many lives) also i want to refer to the using of \"The best central image trick\" that was a cleaver and yet simple .\nThis is surely one of the most interesting competitions i have ever wittnessed and your solution was even more interesting.\nGood Job again !", "votes": 1 }, { "id": 1563616, "postDate": "2021-10-28T13:08:05.107Z", "content": "

Not only do you have machine learning skills but great writing skills as well! this is really well written!
\nThank you for the post! your way of thinking really insightful.

\n

If I was to compete I would for sure be one of those LB overfitters.

\n

Congrats!

", "rawMarkdown": "Not only do you have machine learning skills but great writing skills as well! this is really well written! \nThank you for the post! your way of thinking really insightful. \n\nIf I was to compete I would for sure be one of those LB overfitters.\n\nCongrats! ", "votes": 2 }, { "id": 1559562, "postDate": "2021-10-27T05:29:41.773Z", "content": "

Nice Approach, Congratulations @rinnqd

", "rawMarkdown": "Nice Approach, Congratulations @rinnqd", "votes": 2 }, { "id": 1558913, "postDate": "2021-10-26T14:52:37.073Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": 2 }, { "id": 2894261, "postDate": "2024-06-28T11:25:24.417Z", "content": "

\n

@rinnqd You're absolutely right about your assumption regarding public and private performance, as can be seen in the upper plot, where I scattered public vs. private.

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1813930%2F3d45c899871c66755ec84186a3dc1453%2Fdownload_challenge.png?generation=1719573789980168&alt=media)\n\n@rinnqd You're absolutely right about your assumption regarding public and private performance, as can be seen in the upper plot, where I scattered public vs. private." }, { "id": 2859058, "postDate": "2024-06-06T19:04:18.843Z", "content": "

Could anyone clarify what EDA is, exactly?

", "rawMarkdown": "Could anyone clarify what EDA is, exactly?" }, { "id": 2792002, "postDate": "2024-05-04T02:44:15.270Z", "content": "

<3 <3 legend

", "rawMarkdown": "<3 <3 legend" }, { "id": 1977818, "postDate": "2022-10-08T09:36:56.277Z", "content": "

Have you tried group normalization? as the MRI images generally benefit from using channel type normalization.

", "rawMarkdown": "Have you tried group normalization? as the MRI images generally benefit from using channel type normalization." }, { "id": 1962753, "postDate": "2022-09-30T00:27:16.427Z", "content": "

I think it's a good enlightenment for me. I join some competition for the first time, and before I knew it, I forgot about EDA and focused only on LB. I wondered what it was all about…
\nI think I realize once again that a good model is built on solid EDA.
\nThank you very much. @rinnqd 🙏

", "rawMarkdown": "I think it's a good enlightenment for me. I join some competition for the first time, and before I knew it, I forgot about EDA and focused only on LB. I wondered what it was all about...\nI think I realize once again that a good model is built on solid EDA.\nThank you very much. @rinnqd 🙏" }, { "id": 1851274, "postDate": "2022-07-11T06:00:06.970Z", "content": "

Very insightful !

", "rawMarkdown": "Very insightful !" }, { "id": 1581414, "postDate": "2021-11-13T18:20:37.237Z", "content": "

Have you tried to use voxel normalization like shown here?

", "rawMarkdown": "Have you tried to use voxel normalization like shown [here](https://www.kaggle.com/ren4yu/normalized-voxels-align-planes-and-crop)?" }, { "id": 1557353, "postDate": "2021-10-25T16:01:37.700Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1564015, "postDate": "2021-10-28T19:20:21.837Z", "content": "

Thanks for sharing this!!!:))

", "rawMarkdown": "Thanks for sharing this!!!:))", "votes": 1 }, { "id": 1556805, "postDate": "2021-10-25T06:41:28.193Z", "content": "

Thanks for sharing! Congrats!

", "rawMarkdown": "Thanks for sharing! Congrats!", "votes": 1 }, { "id": 2821070, "postDate": "2024-05-17T22:13:56.847Z", "content": "

thanks for sharing!

", "rawMarkdown": "thanks for sharing!" } ], "comments": [ { "id": 1556663, "author_name": "Pranshu15", "author_url": "", "post_date": "2021-10-25T04:27:16.193000", "content": "

@rinnqd Congrats!
\nDid you check any GradCAM activations of your CNNs to see on what basis it is making the decisions? I am curious as even trained radiologists don't know what to look for to find MGMT promoter.

", "votes": 5, "replies": [ { "id": 1568458, "author_name": "Firas Baba", "author_url": "", "post_date": "2021-11-02T19:11:01.730000", "content": "

I didn't try to explain the model to be honest. I think that explaining the output from a 3D matrix will take forever which is not worth it.

", "votes": 0, "replies": [] } ] }, { "id": 1565902, "author_name": "Old Monk", "author_url": "", "post_date": "2021-10-31T06:47:35.200000", "content": "

Awesome approach and write up, Congratulations @rinnqd for your solo gold medal!

", "votes": 1, "replies": [] }, { "id": 1563198, "author_name": "Quadeer Shaikh", "author_url": "", "post_date": "2021-10-28T05:59:08.247000", "content": "

Very well explained. This was a well formulated approach. Congrats

", "votes": 1, "replies": [] }, { "id": 1561435, "author_name": "Kaveh Shahhosseini", "author_url": "", "post_date": "2021-10-27T14:28:34.613000", "content": "

Congrats @rinnqd

", "votes": 1, "replies": [] }, { "id": 1561149, "author_name": "Allan Bond", "author_url": "", "post_date": "2021-10-27T11:43:01.610000", "content": "

Congratulations on the competition! Thanks for not only sharing what worked for you but also what you tried that didn't work so well.

", "votes": 1, "replies": [] }, { "id": 1559443, "author_name": "Shivam Bansal", "author_url": "", "post_date": "2021-10-27T02:45:26.943000", "content": "

Nice Approach, Congratulations @rinnqd

", "votes": 1, "replies": [] }, { "id": 1558094, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-10-26T03:55:53.680000", "content": "

Congrats, Baba 🙌

", "votes": 1, "replies": [] }, { "id": 1557956, "author_name": "Michael Harnett", "author_url": "", "post_date": "2021-10-25T23:54:46.490000", "content": "

Congrats!! I enjoyed this read, it is SO frustrating when you're trying all these different methods to improve your score and yet they all make it worse I feel that.

", "votes": 1, "replies": [] }, { "id": 1556851, "author_name": "Johansen_Thomas", "author_url": "", "post_date": "2021-10-25T07:10:23.357000", "content": "

Congratulations.

", "votes": 1, "replies": [] }, { "id": 1556659, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-10-25T04:23:08.463000", "content": "

congratulations

", "votes": 1, "replies": [] }, { "id": 1556631, "author_name": "RavikantKumar", "author_url": "", "post_date": "2021-10-25T03:51:48.937000", "content": "

Great approach! Nice and simple.

", "votes": 1, "replies": [] }, { "id": 1556598, "author_name": "medahmed krichen", "author_url": "", "post_date": "2021-10-25T02:58:31.877000", "content": "

@rinnqd You are Awesome,we are really proud of you,
\nbut what is most intersting is the simplicity of the models ,i mean, since the start i was expecting from the first winners a terrifying ensemble of 300 models that no one hear of ,which didn't hapen i think that due to the poor weird small dataset (it's sad , espacily if it can literally save many lives) also i want to refer to the using of \"The best central image trick\" that was a cleaver and yet simple .
\nThis is surely one of the most interesting competitions i have ever wittnessed and your solution was even more interesting.
\nGood Job again !

", "votes": 1, "replies": [] }, { "id": 1563616, "author_name": "", "author_url": "", "post_date": "2021-10-28T13:08:05.107000", "content": "

Not only do you have machine learning skills but great writing skills as well! this is really well written!
\nThank you for the post! your way of thinking really insightful.

\n

If I was to compete I would for sure be one of those LB overfitters.

\n

Congrats!

", "votes": 2, "replies": [] }, { "id": 1559562, "author_name": "Shubham Varshney", "author_url": "", "post_date": "2021-10-27T05:29:41.773000", "content": "

Nice Approach, Congratulations @rinnqd

", "votes": 2, "replies": [] }, { "id": 1558913, "author_name": "atfujita", "author_url": "", "post_date": "2021-10-26T14:52:37.073000", "content": "

Congratulations!

", "votes": 2, "replies": [] }, { "id": 2894261, "author_name": "helme", "author_url": "", "post_date": "2024-06-28T11:25:24.417000", "content": "

\n

@rinnqd You're absolutely right about your assumption regarding public and private performance, as can be seen in the upper plot, where I scattered public vs. private.

", "votes": 0, "replies": [] }, { "id": 2859058, "author_name": "complexsimple", "author_url": "", "post_date": "2024-06-06T19:04:18.843000", "content": "

Could anyone clarify what EDA is, exactly?

", "votes": 0, "replies": [] }, { "id": 2792002, "author_name": "ghassen fatnassi", "author_url": "", "post_date": "2024-05-04T02:44:15.270000", "content": "

<3 <3 legend

", "votes": 0, "replies": [] }, { "id": 1977818, "author_name": "kanish chugh", "author_url": "", "post_date": "2022-10-08T09:36:56.277000", "content": "

Have you tried group normalization? as the MRI images generally benefit from using channel type normalization.

", "votes": 0, "replies": [] }, { "id": 1962753, "author_name": "Jaewook Kim", "author_url": "", "post_date": "2022-09-30T00:27:16.427000", "content": "

I think it's a good enlightenment for me. I join some competition for the first time, and before I knew it, I forgot about EDA and focused only on LB. I wondered what it was all about…
\nI think I realize once again that a good model is built on solid EDA.
\nThank you very much. @rinnqd 🙏

", "votes": 0, "replies": [] }, { "id": 1851274, "author_name": "Jihed 503", "author_url": "", "post_date": "2022-07-11T06:00:06.970000", "content": "

Very insightful !

", "votes": 0, "replies": [] }, { "id": 1581414, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-11-13T18:20:37.237000", "content": "

Have you tried to use voxel normalization like shown here?

", "votes": 0, "replies": [] }, { "id": 1557353, "author_name": "", "author_url": "", "post_date": "2021-10-25T16:01:37.700000", "content": "", "votes": 1, "replies": [] }, { "id": 1564015, "author_name": "Konstantin Titarchuk", "author_url": "", "post_date": "2021-10-28T19:20:21.837000", "content": "

Thanks for sharing this!!!:))

", "votes": 1, "replies": [] }, { "id": 1556805, "author_name": "Swikwislkdjc", "author_url": "", "post_date": "2021-10-25T06:41:28.193000", "content": "

Thanks for sharing! Congrats!

", "votes": 1, "replies": [] }, { "id": 2821070, "author_name": "ghassen fatnassi", "author_url": "", "post_date": "2024-05-17T22:13:56.847000", "content": "

thanks for sharing!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1556179": "First of all congratulations to all the winners! Thanks as well to Kaggle and RSNA for hosting this competition and giving us the chance to work on such an interesting problem.\n\nOn the contrary of what top solutions look like, my final solution was one of the very first baselines I started with. There was no model ensembling, no complex/big models, and no sophisticated training techniques.\n\n### **Team name secret:**\nAs many of you have seen, the team name was set to “I hate this competition” during the last month and I will explain below the reasons that made me hate spending time on this competition.\n- In every competition, I always start with a very simple model and then submit it. After this, I try to use different validation strategies in order to find the best strategy that really reflects the model performance. What made me upset about this competition is that the CV score was almost random in my early models and that no strategy worked to validate the performance of the model.\n- Bigger models were giving noisy CV scores (almost random) and I couldn’t fine-tune any of them. I only limited my models to resnet10-50 and b0-b3 during all my experiments (large resents, densents and efficientnets failed).\n- The batch norm layer in this Efficientnet implementation [here](https://github.com/lukemelas/EfficientNet-PyTorch) was giving non-sense output when I used 3 spatial inputs for the CNN. I found that other people suffered from this problem after searching about this issue and I felt disappointed since many of my experiments were just a total failure. I switched to **monai** library at that time.\n- Ensembling was not really improving the score. Once I change the folds, the scores completely change and the std of the scores was so high.\n- Seeing people getting 0.8+ on the public LB while my models were stuck at around 0.5-0.6 was annoying for me.\n- I was certain and 100% sure that most of the public LB scores were just random scores and that all of the scores will drop in the private LB.\n- I saw someone in the discussion forum who shared that he/she got 0.7+ in the public LB by just using random predictions.\nI was reading the discussions every day and I was really amazed by how many people were targeting 0.7 and 0.8+ scores. This seemed like an impossible mission for me. My utmost ambition was getting a model that scores 0.6+ on validation and public LB (which I failed to get even after the competition ended). \n\n### **What I concluded after a whole week doing EDA:**\n- Public LB does make sense if you don’t overfit your validation set.\n- Most of the public models fail to learn any useful patterns.\n- Many teams are just focusing on improving their Public scores and didn’t even take into consideration the shakeup that could happen at the end of the competition.\n- There will be no escape from the shakeup.\n- It will be very easy to get a top100 position in the private LB with a simple model because many many competitors only focus on the public score.\n- Teaming up (in order to ensemble different models) won’t help to improve any score.\n- I should not invest more than 2 weeks working on this competition because the risk is so high and I cannot guarantee to get any medal. \n- The chances to be in the top 1% or in the middle of the LB (500-700) are almost the same.\n\n### **Very slow validation strategy:**\nTraining the same model with the same everything (parameters, hardware, fixing all random seeds), with the same training data and the same validation set will give different aucroc scores. I remember that when I trained a EF-b0 more than 100 times (same everything) I got CV scores between 0.53 and 0.62 with a high std (can’t remember it). Training 5 folds 100 times, reduced drastically the std of the scores and I got scores (after averaging the 5 folds) between 0.52 and 0.56.\nGuess what? To judge any model/idea/approach I used to train 100 models (Every model is trained 20 times X 5folds). I ranked the experiments based on the scores average. Then, I selected the top 5 ideas+models and I ran 250 models per idea (Every model is trained 50 times X 5folds. The folds in the second stage are different from the first stage folds). After this, I re-ranked the ideas based on the average score for the 250 models (I do the oof of every idea/model and I average the 50 oofs).\nPlease note that some of what I called models in this section represent 4 trained models (on \"FLAIR\", \"T1w\", \"T1wCE\", \"T2w”). Whenever I want to try an idea I apply it to the CNN model using all the 4 types data and I train 4 different models and I average the 4 models. I have excluded the “T2w” data in most of the final models (I think that more than 50% of the models were not using the “T2w” data). Anyway, let’s skip this part because the ideas and models I was trying were totally random. In fact, most of the ideas I tried were just ideas I get when I close my eyes to sleep. I have to be honest, this competition ruined many of my nights and made me feel like a stupid loser.\n\n### **What was the top one idea after the 2 stage ranking?**\n\nI am not sure but I remember that I started with 8 simple ideas and then I tried some sophisticated and customized models (around 20).\nThe top 2 models after applying the 2 stage validation filtering were using the same model and same training techniques but the top 1 was using all of the 4 different structural multi-parametric MRI data (FLAIR, t1w, t1wce, t2w) while the second top model was just using the “T1wCE” data.\n\n### **The final model:**\n- 3D CNN\n- Resnet10\n- BCE loss\n- Adam optimizer\n- 15 epochs\n- LR: epoch 1->10; lr = 0.0001 | epoch 10 to 15 lr=0.00005\n- Image size: 256x256\n- Batch size: 8 (the bigger bs I use the worse CV I get, I was alternating between bs=4 and bs=8)\n- No mixed-precision is used.\n- Used a small trick to build the 3D images. Let’s call it “The best central image trick”.\n- One epoch takes around 1minute and 20 seconds using an RTX 3090.\n\n### **The best central image trick:**\nEach independent case has a different number of images for all the MRI scans. Using all the scans will confuse the model to learn the spatial dependence of the brain pixels.\nFor example, Let’s assume that case_1 has 80 T1wCE scans and case_2 has 500 T1wCE scans and that we will use 40 images as an input for the model.\nWhat people did, is that they take the central image (aka image number 40 (80//2) for case_1 and image number 250 (500//2) in case_2) and then they build the 3d images this way:\ncase_1: from image number 20 to image number 60\ncase_2: from image number 480 to image number 520.\nIn this example, we will end up with 2 3d images that don’t represent the same portion of the brain. Thus, the model will not only fail to learn the tumor pattern but will also start learning some spatial patterns that are not useful in our case.\nWhat I wanted to do and failed is to select a fixed starting and ending point for all the brains and train with the same information for all the cases. But, I couldn't find a way to successfully make it work.\nAfter many failures, I found that using the biggest image as a central image (the image that contains the largest brain cutaway view) will slightly improve my local CV (improvements were between 0.01 and 0.02). I think that this was the only 100% successful experiment I did in this competition. It is not exactly what I wanted, but it kinda worked.\n\n### **What did not work:**\n- 2d CNNs\n- 4 backbones 3D CNNs (each for one structural multi-parametric MRI folder)\n- Ensembling\n- Pretrained on brain images\n- Using the metadata from the DCIM images\n- Stacking the output from the different CNNs using based tree models.\n- Deep CNNs\n- Some tricks I did to normalize the voxels in some consistent ways. (It has been more than a month since I did that and I now think that what I did seems completely stupid lol)\n\n### **One funny fact:**\nI forgot to select my best submissions since I am on vacation and I forgot about the last day of the competition. The cool thing is that my best 2 models had the best public LB score and they were automatically selected as my final submissions :D\n\n### **One quick final thought:**\nI did not enjoy making submissions in this competition as much as I enjoyed the private LB results which explains the fact that the team name was changed from “I hate this competition” to “I love this competition”. The data was too small and the models were barely learning, I am sure that if everyone re-runs his best model again we will still have a small shakeup in the top teams. Finally, I feel super happy to have had my first first-place finish and my first solo gold and I apologize for not interacting with the community in the discussion forum. See you in the next competitions :D \n\n### **Code**\nTraining code is available [here](https://github.com/FirasBaba/rsna-resnet10)\nThe winner inference notebook can be found [here](https://www.kaggle.com/rinnqd/monai-simple-prediction-from-flair)", "1556663": "@rinnqd Congrats! \nDid you check any GradCAM activations of your CNNs to see on what basis it is making the decisions? I am curious as even trained radiologists don't know what to look for to find MGMT promoter.", "1565902": "Awesome approach and write up, Congratulations @rinnqd for your solo gold medal!", "1563198": "Very well explained. This was a well formulated approach. Congrats", "1561435": "Congrats @rinnqd ", "1561149": "Congratulations on the competition! Thanks for not only sharing what worked for you but also what you tried that didn't work so well.", "1559443": "Nice Approach, Congratulations @rinnqd ", "1558094": "Congrats, Baba 🙌", "1557956": "Congrats!! I enjoyed this read, it is SO frustrating when you're trying all these different methods to improve your score and yet they all make it worse I feel that.", "1556851": "Congratulations. ", "1556659": "congratulations", "1556631": "Great approach! Nice and simple.", "1556598": "@rinnqd You are Awesome,we are really proud of you,\nbut what is most intersting is the simplicity of the models ,i mean, since the start i was expecting from the first winners a terrifying ensemble of 300 models that no one hear of ,which didn't hapen i think that due to the poor weird small dataset (it's sad , espacily if it can literally save many lives) also i want to refer to the using of \"The best central image trick\" that was a cleaver and yet simple .\nThis is surely one of the most interesting competitions i have ever wittnessed and your solution was even more interesting.\nGood Job again !", "1563616": "Not only do you have machine learning skills but great writing skills as well! this is really well written! \nThank you for the post! your way of thinking really insightful. \n\nIf I was to compete I would for sure be one of those LB overfitters.\n\nCongrats! ", "1559562": "Nice Approach, Congratulations @rinnqd", "1558913": "Congratulations!", "2894261": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1813930%2F3d45c899871c66755ec84186a3dc1453%2Fdownload_challenge.png?generation=1719573789980168&alt=media)\n\n@rinnqd You're absolutely right about your assumption regarding public and private performance, as can be seen in the upper plot, where I scattered public vs. private.", "2859058": "Could anyone clarify what EDA is, exactly?", "2792002": "<3 <3 legend", "1977818": "Have you tried group normalization? as the MRI images generally benefit from using channel type normalization.", "1962753": "I think it's a good enlightenment for me. I join some competition for the first time, and before I knew it, I forgot about EDA and focused only on LB. I wondered what it was all about...\nI think I realize once again that a good model is built on solid EDA.\nThank you very much. @rinnqd 🙏", "1851274": "Very insightful !", "1581414": "Have you tried to use voxel normalization like shown [here](https://www.kaggle.com/ren4yu/normalized-voxels-align-planes-and-crop)?", "1557353": "", "1564015": "Thanks for sharing this!!!:))", "1556805": "Thanks for sharing! Congrats!", "2821070": "thanks for sharing!" } }