![\"corrs\"](\"https://i.ibb.co/GpJVrM3/Screenshot-from-2021-10-15-14-03-36.png\")
To start with, I want to congratulate my teammate @authman. Mad man finally did it. He just became competitions master after countless high silver medals.
\nI wanted to spend more time in this competition but unfortunately I was stuck at Optiver Realized Volatility Prediction and @authman was working on G2Net Gravitational Wave Detection. We started this competition when 2.5 weeks left so we couldn't try every idea we had but at least some of them worked and we landed in 12th place.
\nAfter starting the competition, I quickly read through discussions and notebooks, but the quality of shared content was really poor. I only learned dicom preprocessing from @davidbroberts' notebooks.
\nWe thought 144^3 spatial dimensions is good enough for both segmentation and classification so we decided to use MRIs and masks in that shape. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We converted ground-truth segmentation masks to non-mutually exclusive one-hot encoded masks with this code.
\ndef convert_labels(mask):\n\n \"\"\"\n Convert 3D spatial segmentation mask to 4D one-hot encoded segmentation mask\n\n Parameters\n ----------\n mask [np.ndarray of shape (depth, height, width)]: Array of 3D segmentation mask\n\n Returns\n -------\n mask [np.ndarray of shape (channel, depth, height, width)]: Array of 4D one-hot encoded segmentation mask\n \"\"\"\n\n one_hot_encoded_mask = np.stack([\n np.logical_or(mask == 1, mask == 2, mask == 3), # Whole tumor\n np.logical_or(mask == 2, mask == 3), # Tumor core\n (mask == 3), # Enhancing tumor\n ]).astype(np.uint8)\n\n return one_hot_encoded_mask\nFinally, we saved MRIs and masks as npy files for training faster. Preprocessing code for nii files can be found here.
\nSingle shuffled train/test split with 80/20 ratio is used for validation. Case directories were split so when every modality is included, it worked like group split without leak.
\nWe used random rotation on X, Y and Z axes between -90 and 90 radians with 25% chance because task 1 MRIs were resampled and registered. We had to break the perfect alignment because task 2 wasn't resampled and registered. We also standardized every MRI like this mri = (mri - mri.mean()) / mri.std() which probably wasn't a correct way to do it.
This was my first segmentation experience so I used a high-level library and we had no time to write our models from scratch. We used SegResNet model from monai for segmentation. We trained 5 single split models for FLAIR, T1w, T1wCE, T2w and all modalities included. All of them trained with dice loss and converged at 0.2-0.3 validation loss. We used a dynamic threshold for converting sigmoided output to labels. Cutoff point is dynamically found using (mean prediction of positives + mean prediction of negatives) / 2.
We used 144^3 spatial dimensions in task 2 MRIs as well. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We didn't use any voi lut or downcasted MRIs to 8 bit because it was lossy and redundant. We saved MRIs to npy files as 16 bit numpy arrays with default values for training faster. Preprocessing code for dicom files can be found here.
\nWe validated our segmentation models by visualizing predictions on task 2 MRIs. They were \"good enough\" but extracted features or using predicted masks as additional channels didn't help our models to score better. It was really sad for us to abandon segmentation idea because we dedicated our first week to it.
\nWe used 5 shuffled stratified folds for cross-validation. Folds are stratified on MGMT_value.
\nWe used random rotation on X, Y and Z axes between -15 and 15 radians and random 3d elastic deformations with 100% chance. We standardized every MRI like this mri = (mri - mri.mean()) / mri.std() here as well.
We used 3D DenseNet121 and DenseNet169 models with classification head and models are trained with bce with logits loss. We noticed that our models can reach 0.66-0.67 val loss consistently with cosine annealing scheduler if it was a lucky run. Loss was correlated with ROC AUC score only after reaching that point so we run our models countless times until we get lucky and validation loss reaches 0.66-0.67. When that happens, val ROC AUC score was always > 0.6. That's when I thought we had a solid chance to grab gold medal. We trained DenseNet121 and DenseNet169 models for every modality separately so we had 4 (modality) * 5 (folds) * 2 (DenseNet121 + DenseNet169) models at the end. Their scores can be seen below.
\n------------------------------\nEvaluating densenet121\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.625220\nFold 2 - ROC AUC Score: 0.579333\nFold 3 - ROC AUC Score: 0.678390\nFold 4 - ROC AUC Score: 0.612221\nFold 5 - ROC AUC Score: 0.605067\n------------------------------\nOOF ROC AUC Score: 0.624692\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595601\nFold 2 - ROC AUC Score: 0.656616\nFold 3 - ROC AUC Score: 0.563338\nFold 4 - ROC AUC Score: 0.521013\nFold 5 - ROC AUC Score: 0.591654\n------------------------------\nOOF ROC AUC Score: 0.576726\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.626393\nFold 2 - ROC AUC Score: 0.595433\nFold 3 - ROC AUC Score: 0.639046\nFold 4 - ROC AUC Score: 0.615201\nFold 5 - ROC AUC Score: 0.518629\n------------------------------\nOOF ROC AUC Score: 0.598063\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.664516\nFold 2 - ROC AUC Score: 0.566745\nFold 3 - ROC AUC Score: 0.621162\nFold 4 - ROC AUC Score: 0.664978\nFold 5 - ROC AUC Score: 0.666170\n------------------------------\nOOF ROC AUC Score: 0.63008\n------------------------------\n------------------------------\nEvaluating densenet169\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.621114\nFold 2 - ROC AUC Score: 0.584309\nFold 3 - ROC AUC Score: 0.619672\nFold 4 - ROC AUC Score: 0.619672\nFold 5 - ROC AUC Score: 0.625633\n------------------------------\nOOF ROC AUC Score: 0.608471\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595894\nFold 2 - ROC AUC Score: 0.556499\nFold 3 - ROC AUC Score: 0.558569\nFold 4 - ROC AUC Score: 0.581818\nFold 5 - ROC AUC Score: 0.620566\n------------------------------\nOOF ROC AUC Score: 0.579769\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.563343\nFold 2 - ROC AUC Score: 0.569379\nFold 3 - ROC AUC Score: 0.613711\nFold 4 - ROC AUC Score: 0.616990\nFold 5 - ROC AUC Score: 0.559762\n------------------------------\nOOF ROC AUC Score: 0.57746\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.673314\nFold 2 - ROC AUC Score: 0.645492\nFold 3 - ROC AUC Score: 0.616692\nFold 4 - ROC AUC Score: 0.561550\nFold 5 - ROC AUC Score: 0.585097\n------------------------------\nOOF ROC AUC Score: 0.603427\n------------------------------\nWe didn't use any post-processing to logits or sigmoided outputs. We only used average blending and we submit the predictions. We tried assigning different weights to different models but it wasn't better than average blending. I think it was because predictions' correlations were really low and all of them were contributing to final blend score regardless of their OOF scores.
\n
Our final average blending score was
\nBlend\n-----\nFold 1 - ROC AUC Score: 0.707038\nFold 2 - ROC AUC Score: 0.682377\nFold 3 - ROC AUC Score: 0.712668\nFold 4 - ROC AUC Score: 0.698957\nFold 5 - ROC AUC Score: 0.709687\n------------------------------\nOOF ROC AUC Score: 0.692443\n------------------------------\nHere are the links of everything I made for this competition.
\nKaggle Notebooks:
\n\nKaggle Datasets:
\n\nGitHub Repository
\n\nU-Net: Convolutional Networks for Biomedical Image Segmentation
\n3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
\nTrialing U-Net Training Modifications for Segmenting Gliomas Using Open Source Deep Learning Framework
\n3D MRI brain tumor segmentation using autoencoder regularization
To start with, I want to congratulate my teammate @authman. Mad man finally did it. He just became competitions master after countless high silver medals.
\nI wanted to spend more time in this competition but unfortunately I was stuck at Optiver Realized Volatility Prediction and @authman was working on G2Net Gravitational Wave Detection. We started this competition when 2.5 weeks left so we couldn't try every idea we had but at least some of them worked and we landed in 12th place.
\nAfter starting the competition, I quickly read through discussions and notebooks, but the quality of shared content was really poor. I only learned dicom preprocessing from @davidbroberts' notebooks.
\nWe thought 144^3 spatial dimensions is good enough for both segmentation and classification so we decided to use MRIs and masks in that shape. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We converted ground-truth segmentation masks to non-mutually exclusive one-hot encoded masks with this code.
\ndef convert_labels(mask):\n\n \"\"\"\n Convert 3D spatial segmentation mask to 4D one-hot encoded segmentation mask\n\n Parameters\n ----------\n mask [np.ndarray of shape (depth, height, width)]: Array of 3D segmentation mask\n\n Returns\n -------\n mask [np.ndarray of shape (channel, depth, height, width)]: Array of 4D one-hot encoded segmentation mask\n \"\"\"\n\n one_hot_encoded_mask = np.stack([\n np.logical_or(mask == 1, mask == 2, mask == 3), # Whole tumor\n np.logical_or(mask == 2, mask == 3), # Tumor core\n (mask == 3), # Enhancing tumor\n ]).astype(np.uint8)\n\n return one_hot_encoded_mask\nFinally, we saved MRIs and masks as npy files for training faster. Preprocessing code for nii files can be found here.
\nSingle shuffled train/test split with 80/20 ratio is used for validation. Case directories were split so when every modality is included, it worked like group split without leak.
\nWe used random rotation on X, Y and Z axes between -90 and 90 radians with 25% chance because task 1 MRIs were resampled and registered. We had to break the perfect alignment because task 2 wasn't resampled and registered. We also standardized every MRI like this mri = (mri - mri.mean()) / mri.std() which probably wasn't a correct way to do it.
This was my first segmentation experience so I used a high-level library and we had no time to write our models from scratch. We used SegResNet model from monai for segmentation. We trained 5 single split models for FLAIR, T1w, T1wCE, T2w and all modalities included. All of them trained with dice loss and converged at 0.2-0.3 validation loss. We used a dynamic threshold for converting sigmoided output to labels. Cutoff point is dynamically found using (mean prediction of positives + mean prediction of negatives) / 2.
We used 144^3 spatial dimensions in task 2 MRIs as well. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We didn't use any voi lut or downcasted MRIs to 8 bit because it was lossy and redundant. We saved MRIs to npy files as 16 bit numpy arrays with default values for training faster. Preprocessing code for dicom files can be found here.
\nWe validated our segmentation models by visualizing predictions on task 2 MRIs. They were \"good enough\" but extracted features or using predicted masks as additional channels didn't help our models to score better. It was really sad for us to abandon segmentation idea because we dedicated our first week to it.
\nWe used 5 shuffled stratified folds for cross-validation. Folds are stratified on MGMT_value.
\nWe used random rotation on X, Y and Z axes between -15 and 15 radians and random 3d elastic deformations with 100% chance. We standardized every MRI like this mri = (mri - mri.mean()) / mri.std() here as well.
We used 3D DenseNet121 and DenseNet169 models with classification head and models are trained with bce with logits loss. We noticed that our models can reach 0.66-0.67 val loss consistently with cosine annealing scheduler if it was a lucky run. Loss was correlated with ROC AUC score only after reaching that point so we run our models countless times until we get lucky and validation loss reaches 0.66-0.67. When that happens, val ROC AUC score was always > 0.6. That's when I thought we had a solid chance to grab gold medal. We trained DenseNet121 and DenseNet169 models for every modality separately so we had 4 (modality) * 5 (folds) * 2 (DenseNet121 + DenseNet169) models at the end. Their scores can be seen below.
\n------------------------------\nEvaluating densenet121\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.625220\nFold 2 - ROC AUC Score: 0.579333\nFold 3 - ROC AUC Score: 0.678390\nFold 4 - ROC AUC Score: 0.612221\nFold 5 - ROC AUC Score: 0.605067\n------------------------------\nOOF ROC AUC Score: 0.624692\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595601\nFold 2 - ROC AUC Score: 0.656616\nFold 3 - ROC AUC Score: 0.563338\nFold 4 - ROC AUC Score: 0.521013\nFold 5 - ROC AUC Score: 0.591654\n------------------------------\nOOF ROC AUC Score: 0.576726\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.626393\nFold 2 - ROC AUC Score: 0.595433\nFold 3 - ROC AUC Score: 0.639046\nFold 4 - ROC AUC Score: 0.615201\nFold 5 - ROC AUC Score: 0.518629\n------------------------------\nOOF ROC AUC Score: 0.598063\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.664516\nFold 2 - ROC AUC Score: 0.566745\nFold 3 - ROC AUC Score: 0.621162\nFold 4 - ROC AUC Score: 0.664978\nFold 5 - ROC AUC Score: 0.666170\n------------------------------\nOOF ROC AUC Score: 0.63008\n------------------------------\n------------------------------\nEvaluating densenet169\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.621114\nFold 2 - ROC AUC Score: 0.584309\nFold 3 - ROC AUC Score: 0.619672\nFold 4 - ROC AUC Score: 0.619672\nFold 5 - ROC AUC Score: 0.625633\n------------------------------\nOOF ROC AUC Score: 0.608471\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595894\nFold 2 - ROC AUC Score: 0.556499\nFold 3 - ROC AUC Score: 0.558569\nFold 4 - ROC AUC Score: 0.581818\nFold 5 - ROC AUC Score: 0.620566\n------------------------------\nOOF ROC AUC Score: 0.579769\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.563343\nFold 2 - ROC AUC Score: 0.569379\nFold 3 - ROC AUC Score: 0.613711\nFold 4 - ROC AUC Score: 0.616990\nFold 5 - ROC AUC Score: 0.559762\n------------------------------\nOOF ROC AUC Score: 0.57746\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.673314\nFold 2 - ROC AUC Score: 0.645492\nFold 3 - ROC AUC Score: 0.616692\nFold 4 - ROC AUC Score: 0.561550\nFold 5 - ROC AUC Score: 0.585097\n------------------------------\nOOF ROC AUC Score: 0.603427\n------------------------------\nWe didn't use any post-processing to logits or sigmoided outputs. We only used average blending and we submit the predictions. We tried assigning different weights to different models but it wasn't better than average blending. I think it was because predictions' correlations were really low and all of them were contributing to final blend score regardless of their OOF scores.
\n
Our final average blending score was
\nBlend\n-----\nFold 1 - ROC AUC Score: 0.707038\nFold 2 - ROC AUC Score: 0.682377\nFold 3 - ROC AUC Score: 0.712668\nFold 4 - ROC AUC Score: 0.698957\nFold 5 - ROC AUC Score: 0.709687\n------------------------------\nOOF ROC AUC Score: 0.692443\n------------------------------\nHere are the links of everything I made for this competition.
\nKaggle Notebooks:
\n\nKaggle Datasets:
\n\nGitHub Repository
\n\nU-Net: Convolutional Networks for Biomedical Image Segmentation
\n3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
\nTrialing U-Net Training Modifications for Segmenting Gliomas Using Open Source Deep Learning Framework
\n3D MRI brain tumor segmentation using autoencoder regularization
Nice approach, Congrats
", "rawMarkdown": "Nice approach, Congrats", "votes": 5 }, { "id": 1549882, "postDate": "2021-10-19T09:00:37.793Z", "content": "@gunesevitan thank you for the writeup!
\nCould you elaborate a little bit what kind of segmentation did you use and how many classes did the tumor segmentation contain? Did you extract classical histogram/texture features or only ML based ones from the segmentation?
Sorry for not being clear. Ground-truth segmentation mask labels were; 1 for the necrotic center and non-enhancing tumor, 2 for edema, and 4 for enhancing tumor. Our segmentation model's output contained 3 non-mutually exlusive classes derived from ground-truth segmentation masks. Our classes were:
\nFor extracted segmentation features, pyradomics was used. For each modality (4), and for each segmentation class (3), pyradomics generates ~120 features given a 3d volume and a mask for a grand total of 1440 extracted features. This process takes a while to run… but the RoI wasn't as great as desired.
", "rawMarkdown": "For extracted segmentation features, pyradomics was used. For each modality (4), and for each segmentation class (3), pyradomics generates [~120 features](https://pyradiomics.readthedocs.io/en/latest/features.html) given a 3d volume and a mask for a grand total of 1440 extracted features. This process takes a while to run... but the RoI wasn't as great as desired.", "votes": 4 }, { "id": 1550109, "postDate": "2021-10-19T12:59:50.997Z", "content": "I am really surprised that none of these pyradiomics extracted features showed higher correlations, i must admit that i would have expected to get something better from these features.
\nMaybe, just as a thought, additionally - i would have done a union of the segmentation classes (just to access all the tumor and sorroundings and extract at least the features from that united one… any chance to check that in form of a late sub? =)
", "rawMarkdown": "I am really surprised that none of these pyradiomics extracted features showed higher correlations, i must admit that i would have expected to get something better from these features.\n\nMaybe, just as a thought, additionally - i would have done a union of the segmentation classes (just to access all the tumor and sorroundings and extract at least the features from that united one... any chance to check that in form of a late sub? =)\n", "votes": 3 }, { "id": 1550161, "postDate": "2021-10-19T13:42:51.500Z", "content": "Under normal circumstances, absolutely. But I think this competition doesn't rate late subs—they didn't even rate non-selected final 2 subs. All other submissions forced scored -1 AUC. I really wanted to check the private lb ranking of one of our submissions which was a 80 model blend. 20 models per modality, 9hour private lb submission run time, that used the pretrained segmentation model frozen logits multiplied by the input before having another go through the encoder + classifier. I totally agree with you though, our intent from the get-go was to blend these statistical/shape/etc features with features directly learned from the volume.
\nOne of the papers I read mentioned that even location was a statistically significant feature, which felt extremely strange to me, but who knows. A larger study size would also help get rid of bad artifacts.
", "rawMarkdown": "Under normal circumstances, absolutely. But I think this competition doesn't rate late subs—they didn't even rate non-selected final 2 subs. All other submissions forced scored -1 AUC. I really wanted to check the private lb ranking of one of our submissions which was a [80 model blend](https://www.kaggle.com/authman/subtest). 20 models per modality, 9hour private lb submission run time, that used the pretrained segmentation model frozen logits multiplied by the input before having another go through the encoder + classifier. I totally agree with you though, our intent from the get-go was to blend these statistical/shape/etc features with features directly learned from the volume.\n\nOne of the papers I read mentioned that even location was a statistically significant feature, which felt extremely strange to me, but who knows. A larger study size would also help get rid of bad artifacts.", "votes": 4 }, { "id": 1550316, "postDate": "2021-10-19T15:47:48.680Z", "content": "I second that on the location of the tumor: even the newest world health organisation's grading takes the localisation (still!) into account and makes difference between midline and hemispheric gliomas (mainly because they have genotypic differences and thus different survival).
\nIt's a pitty that no late subs are allowed =( maybe the next years the whole dataset will become public… and/or we get more curated data.
", "rawMarkdown": "I second that on the location of the tumor: even the newest world health organisation's grading takes the localisation (still!) into account and makes difference between midline and hemispheric gliomas (mainly because they have genotypic differences and thus different survival).\n\nIt's a pitty that no late subs are allowed =( maybe the next years the whole dataset will become public... and/or we get more curated data.", "votes": 2 }, { "id": 1550349, "postDate": "2021-10-19T16:20:03.833Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1550454, "postDate": "2021-10-19T18:30:03.440Z", "content": "Thanks for the kind words and for the tip. Looks like we still have some playing around to do :-).
", "rawMarkdown": "Thanks for the kind words and for the tip. Looks like we still have some playing around to do :-)." }, { "id": 1550591, "postDate": "2021-10-19T19:54:12.610Z", "content": "@jonathanchan… if i would do further research, i would definitely try to analyse some of the following features:
\nThe tumours are usually not homogenous in terms of their composition, there may be areas of it that show lower agressivity (and thus underlying different genotype). To assess the genetic composition of the tumor, biopsies are performed (if the tumors are not resected in toto). These biopsies aim usually to the largest solid, enhancing part of the tumor (because that has the highest possibility to get a higher grade tissue) - given that this area is well reachable without damaging eloquent areas microsurgically. Considering this, one could pinpoint \"possible\" rather small volumes where the probes could have been taken from the tumor and extract features (radiomics) from these volumes only.
anatomic localisation (midline, hemispheric: frontal, temporal, central and so on) could make a categorical variable that may help - maybe as a variable in form of \"distance to sella\", or \"distance to frontal pole\"
tumoral hemorrhages distorts all the features so the presence of it like a binary variable could help (or simply ommit those cases)
analyse boundary of the zones actively contrast agent enhancing - peripheral oedema instead of the \"enhancing tumor\" + \"peripheral oedema\" try a \"transition zone\" class (should be at least relatively simple to compute), maybe the same for necrosis - enhancement
Just to name a few that pop into my mind!
", "rawMarkdown": "@jonathanchan... if i would do further research, i would definitely try to analyse some of the following features:\n\n1. The tumours are usually not homogenous in terms of their composition, there may be areas of it that show lower agressivity (and thus underlying different genotype). To assess the genetic composition of the tumor, biopsies are performed (if the tumors are not resected in toto). These biopsies aim usually to the largest solid, enhancing part of the tumor (because that has the highest possibility to get a higher grade tissue) - given that this area is well reachable without damaging eloquent areas microsurgically. Considering this, one could pinpoint \"possible\" rather small volumes where the probes could have been taken from the tumor and extract features (radiomics) from these volumes only.\n\n2. anatomic localisation (midline, hemispheric: frontal, temporal, central and so on) could make a categorical variable that may help - maybe as a variable in form of \"distance to sella\", or \"distance to frontal pole\"\n\n3. tumoral hemorrhages distorts all the features so the presence of it like a binary variable could help (or simply ommit those cases)\n\n4. analyse boundary of the zones actively contrast agent enhancing - peripheral oedema instead of the \"enhancing tumor\" + \"peripheral oedema\" try a \"transition zone\" class (should be at least relatively simple to compute), maybe the same for necrosis - enhancement\n\nJust to name a few that pop into my mind!", "votes": 4 }, { "id": 1550817, "postDate": "2021-10-20T03:48:44.090Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 1550240, "postDate": "2021-10-19T14:49:33.277Z", "content": "Congrats @gunesevitan and team, you did a good job with 3D models
", "rawMarkdown": "Congrats @gunesevitan and team, you did a good job with 3D models", "votes": 4 }, { "id": 1557154, "postDate": "2021-10-25T13:23:15.997Z", "content": "Thanks for sharing your approach! Congratulations🎉
", "rawMarkdown": "Thanks for sharing your approach! Congratulations🎉", "votes": 2 }, { "id": 1556937, "postDate": "2021-10-25T09:00:01.180Z", "content": "@gunesevitan Good work.
", "rawMarkdown": "@gunesevitan Good work. ", "votes": 2 }, { "id": 1554404, "postDate": "2021-10-23T04:14:53.103Z", "content": "congratulations! Have you been able to find the feature importance of the brain scans? The ROC of around 60 may not be a good fit for practical application.
", "rawMarkdown": "congratulations! Have you been able to find the feature importance of the brain scans? The ROC of around 60 may not be a good fit for practical application. ", "votes": 2, "replies": [ { "id": 1556134, "postDate": "2021-10-24T15:36:10.610Z", "content": "Thanks. We didn't use any handcrafted features in final solution because they didn't work.
", "rawMarkdown": "Thanks. We didn't use any handcrafted features in final solution because they didn't work.", "votes": 1 } ] }, { "id": 1551133, "postDate": "2021-10-20T10:35:33.497Z", "content": "congrats @authman I am really happy for you , very well deserved for both of you @gunesevitan
", "rawMarkdown": "congrats @authman I am really happy for you , very well deserved for both of you @gunesevitan ", "votes": 2 }, { "id": 1550955, "postDate": "2021-10-20T07:15:10.537Z", "content": "Congrats @gunesevita!! Great work… and lots to learn here! 👌🙏🙌
\nWould be interested to know how you were so confident of winning Gold as @authman noted in an earlier post:
\nhttps://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279672
Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 2 }, { "id": 1550066, "postDate": "2021-10-19T12:14:20.400Z", "content": "congratulations
", "rawMarkdown": "congratulations", "votes": 2 }, { "id": 1557366, "postDate": "2021-10-25T16:06:35.150Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1550325, "postDate": "2021-10-19T15:54:50.357Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 1556666, "author_name": "Pranshu15", "author_url": "", "post_date": "2021-10-25T04:36:36.400000", "content": "Nice approach, Congrats
", "votes": 5, "replies": [] }, { "id": 1549882, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T09:00:37.793000", "content": "@gunesevitan thank you for the writeup!
\nCould you elaborate a little bit what kind of segmentation did you use and how many classes did the tumor segmentation contain? Did you extract classical histogram/texture features or only ML based ones from the segmentation?
Sorry for not being clear. Ground-truth segmentation mask labels were; 1 for the necrotic center and non-enhancing tumor, 2 for edema, and 4 for enhancing tumor. Our segmentation model's output contained 3 non-mutually exlusive classes derived from ground-truth segmentation masks. Our classes were:
\nFor extracted segmentation features, pyradomics was used. For each modality (4), and for each segmentation class (3), pyradomics generates ~120 features given a 3d volume and a mask for a grand total of 1440 extracted features. This process takes a while to run… but the RoI wasn't as great as desired.
", "votes": 4, "replies": [] }, { "id": 1550109, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T12:59:50.997000", "content": "I am really surprised that none of these pyradiomics extracted features showed higher correlations, i must admit that i would have expected to get something better from these features.
\nMaybe, just as a thought, additionally - i would have done a union of the segmentation classes (just to access all the tumor and sorroundings and extract at least the features from that united one… any chance to check that in form of a late sub? =)
", "votes": 3, "replies": [] }, { "id": 1550161, "author_name": "عثمان", "author_url": "", "post_date": "2021-10-19T13:42:51.500000", "content": "Under normal circumstances, absolutely. But I think this competition doesn't rate late subs—they didn't even rate non-selected final 2 subs. All other submissions forced scored -1 AUC. I really wanted to check the private lb ranking of one of our submissions which was a 80 model blend. 20 models per modality, 9hour private lb submission run time, that used the pretrained segmentation model frozen logits multiplied by the input before having another go through the encoder + classifier. I totally agree with you though, our intent from the get-go was to blend these statistical/shape/etc features with features directly learned from the volume.
\nOne of the papers I read mentioned that even location was a statistically significant feature, which felt extremely strange to me, but who knows. A larger study size would also help get rid of bad artifacts.
", "votes": 4, "replies": [] }, { "id": 1550316, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T15:47:48.680000", "content": "I second that on the location of the tumor: even the newest world health organisation's grading takes the localisation (still!) into account and makes difference between midline and hemispheric gliomas (mainly because they have genotypic differences and thus different survival).
\nIt's a pitty that no late subs are allowed =( maybe the next years the whole dataset will become public… and/or we get more curated data.
", "votes": 2, "replies": [] }, { "id": 1550349, "author_name": "", "author_url": "", "post_date": "2021-10-19T16:20:03.833000", "content": "", "votes": 2, "replies": [] }, { "id": 1550454, "author_name": "عثمان", "author_url": "", "post_date": "2021-10-19T18:30:03.440000", "content": "Thanks for the kind words and for the tip. Looks like we still have some playing around to do :-).
", "votes": 0, "replies": [] }, { "id": 1550591, "author_name": "dr. Konya", "author_url": "", "post_date": "2021-10-19T19:54:12.610000", "content": "@jonathanchan… if i would do further research, i would definitely try to analyse some of the following features:
\nThe tumours are usually not homogenous in terms of their composition, there may be areas of it that show lower agressivity (and thus underlying different genotype). To assess the genetic composition of the tumor, biopsies are performed (if the tumors are not resected in toto). These biopsies aim usually to the largest solid, enhancing part of the tumor (because that has the highest possibility to get a higher grade tissue) - given that this area is well reachable without damaging eloquent areas microsurgically. Considering this, one could pinpoint \"possible\" rather small volumes where the probes could have been taken from the tumor and extract features (radiomics) from these volumes only.
anatomic localisation (midline, hemispheric: frontal, temporal, central and so on) could make a categorical variable that may help - maybe as a variable in form of \"distance to sella\", or \"distance to frontal pole\"
tumoral hemorrhages distorts all the features so the presence of it like a binary variable could help (or simply ommit those cases)
analyse boundary of the zones actively contrast agent enhancing - peripheral oedema instead of the \"enhancing tumor\" + \"peripheral oedema\" try a \"transition zone\" class (should be at least relatively simple to compute), maybe the same for necrosis - enhancement
Just to name a few that pop into my mind!
", "votes": 4, "replies": [] }, { "id": 1550817, "author_name": "", "author_url": "", "post_date": "2021-10-20T03:48:44.090000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1550240, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-10-19T14:49:33.277000", "content": "Congrats @gunesevitan and team, you did a good job with 3D models
", "votes": 4, "replies": [] }, { "id": 1557154, "author_name": "Aditya Sharma", "author_url": "", "post_date": "2021-10-25T13:23:15.997000", "content": "Thanks for sharing your approach! Congratulations🎉
", "votes": 2, "replies": [] }, { "id": 1556937, "author_name": "Muhammad Maaz", "author_url": "", "post_date": "2021-10-25T09:00:01.180000", "content": "@gunesevitan Good work.
", "votes": 2, "replies": [] }, { "id": 1554404, "author_name": "Dac-Thanh Van", "author_url": "", "post_date": "2021-10-23T04:14:53.103000", "content": "congratulations! Have you been able to find the feature importance of the brain scans? The ROC of around 60 may not be a good fit for practical application.
", "votes": 2, "replies": [ { "id": 1556134, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2021-10-24T15:36:10.610000", "content": "Thanks. We didn't use any handcrafted features in final solution because they didn't work.
", "votes": 1, "replies": [] } ] }, { "id": 1551133, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-10-20T10:35:33.497000", "content": "congrats @authman I am really happy for you , very well deserved for both of you @gunesevitan
", "votes": 2, "replies": [] }, { "id": 1550955, "author_name": "Kamal Das", "author_url": "", "post_date": "2021-10-20T07:15:10.537000", "content": "Congrats @gunesevita!! Great work… and lots to learn here! 👌🙏🙌
\nWould be interested to know how you were so confident of winning Gold as @authman noted in an earlier post:
\nhttps://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279672
Congratulations!
", "votes": 2, "replies": [] }, { "id": 1550066, "author_name": "MohammadHossein Givkashi", "author_url": "", "post_date": "2021-10-19T12:14:20.400000", "content": "congratulations
", "votes": 2, "replies": [] }, { "id": 1557366, "author_name": "", "author_url": "", "post_date": "2021-10-25T16:06:35.150000", "content": "", "votes": 2, "replies": [] }, { "id": 1550325, "author_name": "", "author_url": "", "post_date": "2021-10-19T15:54:50.357000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1549866": "To start with, I want to congratulate my teammate @authman. Mad man finally did it. He just became competitions master after countless high silver medals.\n\nI wanted to spend more time in this competition but unfortunately I was stuck at Optiver Realized Volatility Prediction and @authman was working on G2Net Gravitational Wave Detection. We started this competition when 2.5 weeks left so we couldn't try every idea we had but at least some of them worked and we landed in 12th place.\n\nAfter starting the competition, I quickly read through discussions and notebooks, but the quality of shared content was really poor. I only learned dicom preprocessing from @davidbroberts' notebooks.\n\n## Task 1 Data Preparation\nWe thought 144^3 spatial dimensions is good enough for both segmentation and classification so we decided to use MRIs and masks in that shape. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We converted ground-truth segmentation masks to non-mutually exclusive one-hot encoded masks with this code.\n\n```\ndef convert_labels(mask):\n\n \"\"\"\n Convert 3D spatial segmentation mask to 4D one-hot encoded segmentation mask\n\n Parameters\n ----------\n mask [np.ndarray of shape (depth, height, width)]: Array of 3D segmentation mask\n\n Returns\n -------\n mask [np.ndarray of shape (channel, depth, height, width)]: Array of 4D one-hot encoded segmentation mask\n \"\"\"\n\n one_hot_encoded_mask = np.stack([\n np.logical_or(mask == 1, mask == 2, mask == 3), # Whole tumor\n np.logical_or(mask == 2, mask == 3), # Tumor core\n (mask == 3), # Enhancing tumor\n ]).astype(np.uint8)\n\n return one_hot_encoded_mask\n```\n\nFinally, we saved MRIs and masks as npy files for training faster. Preprocessing code for nii files can be found [here](https://github.com/gunesevitan/rsna-miccai-brain-tumor-radiogenomic-classification/blob/master/src/nii_utils.py).\n\n## Task 1 Validation\nSingle shuffled train/test split with 80/20 ratio is used for validation. Case directories were split so when every modality is included, it worked like group split without leak.\n\n## Task 1 Preprocessing\nWe used random rotation on X, Y and Z axes between -90 and 90 radians with 25% chance because task 1 MRIs were resampled and registered. We had to break the perfect alignment because task 2 wasn't resampled and registered. We also standardized every MRI like this `mri = (mri - mri.mean()) / mri.std()` which probably wasn't a correct way to do it.\n\n## Task 1 Models\nThis was my first segmentation experience so I used a high-level library and we had no time to write our models from scratch. We used [SegResNet](https://docs.monai.io/en/latest/networks.html#segresnet) model from monai for segmentation. We trained 5 single split models for FLAIR, T1w, T1wCE, T2w and all modalities included. All of them trained with dice loss and converged at 0.2-0.3 validation loss. We used a dynamic threshold for converting sigmoided output to labels. Cutoff point is dynamically found using `(mean prediction of positives + mean prediction of negatives) / 2`.\n\n## Task 2 Data Preparation\nWe used 144^3 spatial dimensions in task 2 MRIs as well. We removed empty slices along X-Y, X-Z and Y-Z axes then resized them from the longest axis. We didn't use any voi lut or downcasted MRIs to 8 bit because it was lossy and redundant. We saved MRIs to npy files as 16 bit numpy arrays with default values for training faster. Preprocessing code for dicom files can be found [here](https://github.com/gunesevitan/rsna-miccai-brain-tumor-radiogenomic-classification/blob/master/src/dicom_utils.py).\n\n## Task 2 Segmentation and Feature Extraction\nWe validated our segmentation models by visualizing predictions on task 2 MRIs. They were \"good enough\" but extracted features or using predicted masks as additional channels didn't help our models to score better. It was really sad for us to abandon segmentation idea because we dedicated our first week to it. \n\n## Task 2 Validation\nWe used 5 shuffled stratified folds for cross-validation. Folds are stratified on MGMT_value.\n\n## Task 2 Preprocessing\nWe used random rotation on X, Y and Z axes between -15 and 15 radians and random 3d elastic deformations with 100% chance. We standardized every MRI like this `mri = (mri - mri.mean()) / mri.std()` here as well.\n\n## Task 2 Models\nWe used 3D DenseNet121 and DenseNet169 models with classification head and models are trained with bce with logits loss. We noticed that our models can reach 0.66-0.67 val loss consistently with cosine annealing scheduler if it was a lucky run. Loss was correlated with ROC AUC score only after reaching that point so we run our models countless times until we get lucky and validation loss reaches 0.66-0.67. When that happens, val ROC AUC score was always > 0.6. That's when I thought we had a solid chance to grab gold medal. We trained DenseNet121 and DenseNet169 models for every modality separately so we had 4 (modality) * 5 (folds) * 2 (DenseNet121 + DenseNet169) models at the end. Their scores can be seen below.\n\n```\n------------------------------\nEvaluating densenet121\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.625220\nFold 2 - ROC AUC Score: 0.579333\nFold 3 - ROC AUC Score: 0.678390\nFold 4 - ROC AUC Score: 0.612221\nFold 5 - ROC AUC Score: 0.605067\n------------------------------\nOOF ROC AUC Score: 0.624692\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595601\nFold 2 - ROC AUC Score: 0.656616\nFold 3 - ROC AUC Score: 0.563338\nFold 4 - ROC AUC Score: 0.521013\nFold 5 - ROC AUC Score: 0.591654\n------------------------------\nOOF ROC AUC Score: 0.576726\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.626393\nFold 2 - ROC AUC Score: 0.595433\nFold 3 - ROC AUC Score: 0.639046\nFold 4 - ROC AUC Score: 0.615201\nFold 5 - ROC AUC Score: 0.518629\n------------------------------\nOOF ROC AUC Score: 0.598063\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.664516\nFold 2 - ROC AUC Score: 0.566745\nFold 3 - ROC AUC Score: 0.621162\nFold 4 - ROC AUC Score: 0.664978\nFold 5 - ROC AUC Score: 0.666170\n------------------------------\nOOF ROC AUC Score: 0.63008\n------------------------------\n```\n\n```\n------------------------------\nEvaluating densenet169\n------------------------------\n\nFLAIR\n-----\nFold 1 - ROC AUC Score: 0.621114\nFold 2 - ROC AUC Score: 0.584309\nFold 3 - ROC AUC Score: 0.619672\nFold 4 - ROC AUC Score: 0.619672\nFold 5 - ROC AUC Score: 0.625633\n------------------------------\nOOF ROC AUC Score: 0.608471\n------------------------------\n\nT1w\n---\nFold 1 - ROC AUC Score: 0.595894\nFold 2 - ROC AUC Score: 0.556499\nFold 3 - ROC AUC Score: 0.558569\nFold 4 - ROC AUC Score: 0.581818\nFold 5 - ROC AUC Score: 0.620566\n------------------------------\nOOF ROC AUC Score: 0.579769\n------------------------------\n\nT1wCE\n-----\nFold 1 - ROC AUC Score: 0.563343\nFold 2 - ROC AUC Score: 0.569379\nFold 3 - ROC AUC Score: 0.613711\nFold 4 - ROC AUC Score: 0.616990\nFold 5 - ROC AUC Score: 0.559762\n------------------------------\nOOF ROC AUC Score: 0.57746\n------------------------------\n\nT2w\n---\nFold 1 - ROC AUC Score: 0.673314\nFold 2 - ROC AUC Score: 0.645492\nFold 3 - ROC AUC Score: 0.616692\nFold 4 - ROC AUC Score: 0.561550\nFold 5 - ROC AUC Score: 0.585097\n------------------------------\nOOF ROC AUC Score: 0.603427\n------------------------------\n```\n\n## Task 2 Post-processing, Blending and Submission\nWe didn't use any post-processing to logits or sigmoided outputs. We only used average blending and we submit the predictions. We tried assigning different weights to different models but it wasn't better than average blending. I think it was because predictions' correlations were really low and all of them were contributing to final blend score regardless of their OOF scores.\n\n\n\nOur final average blending score was\n\n```\nBlend\n-----\nFold 1 - ROC AUC Score: 0.707038\nFold 2 - ROC AUC Score: 0.682377\nFold 3 - ROC AUC Score: 0.712668\nFold 4 - ROC AUC Score: 0.698957\nFold 5 - ROC AUC Score: 0.709687\n------------------------------\nOOF ROC AUC Score: 0.692443\n------------------------------\n```\n\n## What didn't work\n* 2D segmentation by randomly selecting 3 slices from X-Y, X-Z or Y-Z axes\n* Using U-Net or U-Net variations with 144^3 spatial dimensions, it wasn't possible to fit enough data into memory\n* SegResNet with variational auto-encoder\n* Features extracted from segmentations\n* ResNet or EfficientNet classification models\n* Weighted blending\n* Only using folds with > 0.6 ROC AUC score\n* Only using FLAIR\n* Only using FLAIR and T2w\n\n## Links\nHere are the links of everything I made for this competition.\n\nKaggle Notebooks:\n* [RSNA-MICCAI BTRC - Inference and Blend](https://www.kaggle.com/gunesevitan/rsna-miccai-btrc-inference-and-blend)\n\nKaggle Datasets:\n* [https://www.kaggle.com/gunesevitan/rsnamiccai-btrc-dataset](https://www.kaggle.com/gunesevitan/rsnamiccai-btrc-dataset)\n\nGitHub Repository\n* [RSNA-MICCAI Brain Tumor Radiogenomic Classification](https://github.com/gunesevitan/rsna-miccai-brain-tumor-radiogenomic-classification)\n\n## References\n[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597)\n[3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://arxiv.org/abs/1606.06650)\n[Trialing U-Net Training Modifications for Segmenting Gliomas Using Open Source Deep Learning Framework](https://www.researchgate.net/publication/350381125_Trialing_U-Net_Training_Modifications_for_Segmenting_Gliomas_Using_Open_Source_Deep_Learning_Framework)\n[3D MRI brain tumor segmentation using autoencoder regularization](https://arxiv.org/abs/1606.06650)", "1556666": "Nice approach, Congrats", "1549882": "@gunesevitan thank you for the writeup!\nCould you elaborate a little bit what kind of segmentation did you use and how many classes did the tumor segmentation contain? Did you extract classical histogram/texture features or only ML based ones from the segmentation?\n", "1550240": "Congrats @gunesevitan and team, you did a good job with 3D models", "1557154": "Thanks for sharing your approach! Congratulations🎉", "1556937": "@gunesevitan Good work. ", "1554404": "congratulations! Have you been able to find the feature importance of the brain scans? The ROC of around 60 may not be a good fit for practical application. ", "1551133": "congrats @authman I am really happy for you , very well deserved for both of you @gunesevitan ", "1550955": "Congrats @gunesevita!! Great work... and lots to learn here! 👌🙏🙌\n\nWould be interested to know how you were so confident of winning Gold as @authman noted in an earlier post:\nhttps://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/279672\n\n\n", "1550278": "Congratulations!", "1550066": "congratulations", "1557366": "", "1550325": "" } }