![\"download\"](\"https://user-images.githubusercontent.com/17668390/130027730-bd4d7afe-9b14-4dc7-b0f5-56f6b61e5e1f.png\")
![\"1download\"](\"https://user-images.githubusercontent.com/17668390/130027739-eebd6fe1-650c-4e50-ab57-6df1f918cef4.png\")
Till now, we have applied a bunch of strategies, still getting the validation loss(BCE) in the range of 0.69-0.68. our oof files scores are always staying almost close to 0.5.
\nI don't know if our model is actually learning or not.
So to you guys, please tell me I am wrong here. As I am feeling very miserable as the models are failing to generalise.
\nI would be really happy if someone tells me that I am doing it wrong, and their models are actually training pretty good.
PS I am not telling anyone to share their pipeline, just want to know if you are facing the same problem, or actually, you are getting good results.
\nPPS: Not used any external data, just competition's data
\nThanks
Till now, we have applied a bunch of strategies, still getting the validation loss(BCE) in the range of 0.69-0.68. our oof files scores are always staying almost close to 0.5.
\nI don't know if our model is actually learning or not.
So to you guys, please tell me I am wrong here. As I am feeling very miserable as the models are failing to generalise.
\nI would be really happy if someone tells me that I am doing it wrong, and their models are actually training pretty good.
PS I am not telling anyone to share their pipeline, just want to know if you are facing the same problem, or actually, you are getting good results.
\nPPS: Not used any external data, just competition's data
\nThanks
I have dealt with the same problem. My 0.764 is much higher than the cross validation suggests it should be so I'm attributing it largely to statistical aberration of low sample size. I'm also running some post processing that weights the ranking between FLAIR and T2 channels that gives a slight boost.
\nI get nearly identical results whether I'm using a simple, shallow 2 layer CNN or EFF-B7. My intuition is leading me to this being more a data centric AI problem rather than a modeling one and the majority of my time spent has been with prepping the data. I've found only one thing that seems to have some effect.
\nI'm using Effnet-B0 3D CNN and have put significant work into orienting, interpolating to scale (1mm x 1mm x 1mm), and normalizing the images. The one thing that shows signs of life is normalizing in 3D voxel space using 3D CLAHE. Most public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.
Some nice points @tivfrvqhs5
\nIf you don't mind, can you tell me for how are you normalizing them? I mean using the imagenet standard mean,std or the actual mean,std of the data.
Multi-dimensional CLAHE is the normalization method. The procedure is outlined in this paper.
", "rawMarkdown": "Multi-dimensional CLAHE is the normalization method. The procedure is outlined in [this](https://ieeexplore.ieee.org/document/8895993) paper.", "votes": 11 }, { "id": 1480838, "postDate": "2021-08-19T07:38:50.940Z", "content": "The mclahe provides tf 1.x and numpy implementation. And for tf, there is a work progress in tf 2.x. In the meantime we can use tf_clahe package.
Instead of tf.clahe, here is another close alternative
# https://github.com/tensorflow/models\ndef equalize(image, mode='grayscale'):\n def scale_channel(im, c):\n \"\"\"Scale the data in the channel to implement equalize.\"\"\"\n im = tf.cast(im[..., c], tf.int32)\n # Compute the histogram of the image channel.\n histo = tf.histogram_fixed_width(im, [0, 255], nbins=256)\n\n # For the purposes of computing the step, filter out the nonzeros.\n nonzero = tf.where(tf.not_equal(histo, 0))\n nonzero_histo = tf.reshape(tf.gather(histo, nonzero), [-1])\n step = (tf.reduce_sum(nonzero_histo) - nonzero_histo[-1]) // 255\n\n def build_lut(histo, step):\n # Compute the cumulative sum, shifting by step // 2\n # and then normalization by step.\n lut = (tf.cumsum(histo) + (step // 2)) // step\n # Shift lut, prepending with 0.\n lut = tf.concat([[0], lut[:-1]], 0)\n # Clip the counts to be in range. This is done\n # in the C code for image.point.\n return tf.clip_by_value(lut, 0, 255)\n\n # If step is zero, return the original image. Otherwise, build\n # lut from the full histogram and step and then index from it.\n result = tf.cond(\n tf.equal(step, 0), lambda: im,\n lambda: tf.gather(build_lut(histo, step), im))\n return tf.cast(result, tf.uint8)\n\n if mode == 'grayscale':\n image = scale_channel(image, 0)\n return tf.cast(image, tf.float32)\n elif mode == 'rgb':\n s1 = scale_channel(image, 0)\n s2 = scale_channel(image, 1)\n s3 = scale_channel(image, 2)\n image = tf.stack([s1, s2, s3], -1)\n return tf.cast(image, tf.float32)\n@ipythonx in which part are you using mclahe,I am confused.
\nI mean do you use this when you are converting your dicoms to numpy arrays or first convert them to numpy arrays and then apply mclahe
@mrinath cc: tf_clahe developer @isaacsears
\nWe used the above tf 2.x implementation of clahe. But afaik, it's not yet supported for 3d volume. But we can use it in tf_image_augmentation pipeline. This augmentation function is not much efficient (we think) as for slice-wise operation and would be better for batch-wise transformation (known issue).
![\"download\"](\"https://user-images.githubusercontent.com/17668390/130039120-8b9518cc-fbd5-4190-91d0-593ea52874af.png\")
\n![\"download\"](\"https://user-images.githubusercontent.com/17668390/130039175-c93ad866-7829-4541-9f05-2f5250a71433.png\")
![\"download\"](\"https://user-images.githubusercontent.com/17668390/130039226-0d0d4768-8922-4a95-bb50-836e08671520.png\")
![\"download\"](\"https://user-images.githubusercontent.com/17668390/130039256-73e40f1a-0bed-4755-a286-987198711658.png\")
For pytorch, we can use the official mclahe-numpy implementation.
\n\nMost public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.
\n
Could you please elaborate on that?
", "rawMarkdown": "@tivfrvqhs5 \n\n> Most public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.\n\nCould you please elaborate on that? " }, { "id": 1481311, "postDate": "2021-08-19T12:27:54.053Z", "content": "Hi @tivfrvqhs5 is 764 your score is from kfold ensemble or single fold?
", "rawMarkdown": "Hi @tivfrvqhs5 is 764 your score is from kfold ensemble or single fold?" } ] }, { "id": 1503997, "postDate": "2021-09-06T01:18:24.210Z", "content": "As the Venn diagram in the link below shows, there is some overlap in the cases between Task 1 and Task 2 of BraTS 2021.
\n![\"Venn\"](\"https://dl.easyuploader.cloud/20210906093509_43495551.jpg\")
Specifically, out of the Task 2 training data hosted on Kaggle, 577 cases overlap with the Task 1 training data. Therefore, I conducted a segmentation-based MGMT classification experiment as described in Yogananda et al. 2021, giving only Task 2 labels to the 577 cases of Task 1 data that overlap with Task 2. The reason for the above experimental setup is that the MRI images of Task 1 are better than those of Task 2 because they are carefully co-registered.
\nThe model used was a light-weight pre-trained 3D CNN model, and all 5 folds were trained with using 4 modalities.
The experimental results were disappointing š
\nUnlike the excellent results in Yogananda's paper, my local AUC was about 0.56, which is slightly better than chance level, almost nothing was learned. At this point, I am not sure if our MRI data does not have enough information to classify MGMT status, or if features for MGMT classification are contained in something else that cannot be learned by the CNN model. However, since Yogananda et al. succeeded in classifying MGMT with 3D CNN, I think it is possible that the label given in the competition, which is supposed to be Ground Truth, might be wrong in some cases.
In addition, Baid et al., 2021 has the following definition of MGMT.
\n\n\nThe percent methylation above 10% was interpreted as positive. A sample below 10% methylation was interpreted as negative. A sample below 10% methylation was interpreted as negative. For the latter approach, a total of 17 MGMT promoter CpG sites were amplified by nested polymerase chain reaction (PCR) using a bisulte treated DNA template. Quantitative PCR was performed for each CpG site to determine its methylation status. A result of 2% or more methylated CpG sites in the MGMT promoter (out of 17 total sites) was considered a positive result.
\n
Taking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.
", "rawMarkdown": "As the Venn diagram in the link below shows, there is some overlap in the cases between Task 1 and Task 2 of BraTS 2021.\n\n\n\nSpecifically, out of the Task 2 training data hosted on Kaggle, 577 cases overlap with the Task 1 training data. Therefore, I conducted a segmentation-based MGMT classification experiment as described in [Yogananda et al. 2021](http://www.ajnr.org/content/early/2021/03/04/ajnr.A7029), giving only Task 2 labels to the 577 cases of Task 1 data that overlap with Task 2. The reason for the above experimental setup is that the MRI images of Task 1 are better than those of Task 2 because they are carefully co-registered.\nThe model used was a light-weight pre-trained 3D CNN model, and all 5 folds were trained with using 4 modalities.\n\nThe experimental results were disappointing š\nUnlike the excellent results in Yogananda's paper, my local AUC was about 0.56, which is slightly better than chance level, almost nothing was learned. At this point, I am not sure if our MRI data does not have enough information to classify MGMT status, or if features for MGMT classification are contained in something else that cannot be learned by the CNN model. However, since Yogananda et al. succeeded in classifying MGMT with 3D CNN, I think it is possible that the label given in the competition, which is supposed to be Ground Truth, might be wrong in some cases. \n\nIn addition, [Baid et al., 2021](https://arxiv.org/abs/2107.02314) has the following definition of MGMT.\n\n> **The percent methylation above 10% was interpreted as positive. A sample below 10% methylation was interpreted as negative.** A sample below 10% methylation was interpreted as negative. For the latter approach, a total of 17 MGMT promoter CpG sites were amplified by nested polymerase chain reaction (PCR) using a bisulte treated DNA template. Quantitative PCR was performed for each CpG site to determine its methylation status. **A result of 2% or more methylated CpG sites in the MGMT promoter (out of 17 total sites) was considered a positive result.**\n\nTaking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.", "votes": 12, "replies": [ { "id": 1504065, "postDate": "2021-09-06T04:02:04.647Z", "content": "Taking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.
\nYes, I support this, It would have been better for the models.
\n@ujjwalbaid and other competition organizers, is there still time left to do this?
As no one could reproduce their result or get useful information from visual features(even combining task1/2 data), the paper's claim feels suspicious for me.
", "rawMarkdown": "As no one could reproduce their result or get useful information from visual features(even combining task1/2 data), the paper's claim feels suspicious for me.", "votes": 3 }, { "id": 1505586, "postDate": "2021-09-07T11:48:51.143Z", "content": "@steaphan
\nMe too, it's impossible for this data.
The paper of Yogananda et al. 2021 (http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf) used the data from the TCIA database, and achieves a mean AUC of 0.93 [SD, 0.01] for predicting MGMT status.
\nThe paper has the following descriptions, so I trust it a lot just from reading the paper.
\nI don't know whether they downloaded data from here (I am not sure):
\nBut if yes, then the dataset they downloaded should be part of the BraTS2021 segmentation dataset, as stated below:
\n\"For BraTS'17, expert neuroradiologists have radiologically assessed the complete original TCIA glioma collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and categorized each scan as pre- or post-operative. Subsequently, all the pre-operative TCIA scans (135 GBM and 108 LGG) were annotated by experts for the various glioma sub-regions and included in this year's BraTS datasets. \" (https://www.med.upenn.edu/cbica/brats2021/).
\nSo I am a little shocked that you only get about an AUC of 0.56 by his method.
", "rawMarkdown": "The paper of Yogananda et al. 2021 (http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf) used the data from the TCIA database, and achieves **a mean AUC of 0.93 [SD, 0.01]** for predicting MGMT status. \n\nThe paper has the following descriptions, so I trust it a lot just from reading the paper.\n- \"We developed a fully-automated, highly accurate, deep learning network for determining the methylation status of the MGMT promoter that **outperforms previously reported algorithms**.\" \n- \"This result represents **an important milestone** toward using MR imaging to predict prognosis and treatment response\".\n\n\nI don't know whether they downloaded data from here (I am not sure):\n- BraTS-TCGA-GBM: \nhttps://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=24282666#24282666136361c88f054395a8f63c49b15f9ae8\n- BraTS-TCGA-LGG:\nhttps://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=24282668#24282668197861a846e445a795694ff2a50eb66c\n\n\nBut if yes, then the dataset they downloaded should be part of the BraTS2021 segmentation dataset, as stated below: \n\n\"For BraTS'17, expert neuroradiologists have radiologically assessed the complete original TCIA glioma collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and categorized each scan as pre- or post-operative. Subsequently, all the pre-operative TCIA scans (135 GBM and 108 LGG) were annotated by experts for the various glioma sub-regions and included in this year's BraTS datasets. \" (https://www.med.upenn.edu/cbica/brats2021/).\n\nSo I am a little shocked that you only get about **an AUC of 0.56 by his method**.\n\n\n", "votes": 4 }, { "id": 1508833, "postDate": "2021-09-10T15:50:54.277Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1509144, "postDate": "2021-09-10T22:35:17.313Z", "content": "I'm shocked, just as you are shocked.
\nI hope I have made some fundamental mistake, otherwise I think this competition will turn into a lottery competition where the final ranking will be almost entirely governed by randomness.
In fact, for simple brain tumor segmentation, I have been able to get good results even with a light model and no particular pursuit of accuracy.
\nThe three figures in the link below are 3D segmentation results obtained by 3D CNN using the task1 dataset. (Left: Ground Truth map, Right: Predicted map)
\nYou can see that the mapping is reasonably good (the validation result of case-wise BCE is about 0.05, which is not bad).
![\"case](\"https://dl.easyuploader.cloud/20210911070450_50495159.png\")
![\"case](\"https://dl.easyuploader.cloud/20210911070450_35375478.png\")
![\"case](\"https://dl.easyuploader.cloud/20210911070450_6e525873.png\")
However, when I followed the method of Yogananda's paper and used multi-class segmentation according to MGMT labels, I got a disappointing result of AUC ~ 0.56 š
\nI have a feeling that this number can be improved somewhat. However, I think it may not be as good as the value in their paper.
\nIn short, my 3D CNN model can segment brain tumors, but it does not have enough expressive power to classify MGMTs.
By the way, I think your current score is not as high as the AUC presented in Yogananda's paper, but I was curious why you support Yogananda's paper at this stage.
\nSo far, even though Yogananda's paper is known to many of the participants, I don't think there are any reports that have been able to significantly increase LB scores independent of randomness in ways other than Hand Labeling or Probing.
In any case, I hope that my impression is wrong and that this competition will be meaningful.
", "rawMarkdown": "I'm shocked, just as you are shocked.\nI hope I have made some fundamental mistake, otherwise I think this competition will turn into a lottery competition where the final ranking will be almost entirely governed by randomness.\n\nIn fact, for simple brain tumor segmentation, I have been able to get good results even with a light model and no particular pursuit of accuracy.\nThe three figures in the link below are 3D segmentation results obtained by 3D CNN using the task1 dataset. (Left: Ground Truth map, Right: Predicted map)\nYou can see that the mapping is reasonably good (the validation result of case-wise BCE is about 0.05, which is not bad).\n\n \n\n \n\n \n\nHowever, when I followed the method of Yogananda's paper and used multi-class segmentation according to MGMT labels, I got a disappointing result of AUC ~ 0.56 š\nI have a feeling that this number can be improved somewhat. However, I think it may not be as good as the value in their paper.\nIn short, my 3D CNN model can segment brain tumors, but it does not have enough expressive power to classify MGMTs.\n\nBy the way, I think your current score is not as high as the AUC presented in Yogananda's paper, but I was curious why you support Yogananda's paper at this stage.\nSo far, even though Yogananda's paper is known to many of the participants, I don't think there are any reports that have been able to significantly increase LB scores independent of randomness in ways other than Hand Labeling or Probing.\n\nIn any case, I hope that my impression is wrong and that this competition will be meaningful.", "votes": 1 }, { "id": 1509331, "postDate": "2021-09-11T06:38:23.510Z", "content": "I didn't mean I support Yogananda's method, I mean that the dataset he used is part of the BraTS2021 dataset and he got a mean AUC of 0.93. Now using more data in BraTS2021, you can only get an AUC ~ 0.56. Other competitors also do not get very good results now, so one reason may be the dataset, one reason may be the results he reported.
\nI didn't try Yogananda's method, I am using the traditional radiogenomics method (extract radiomic features + machine learning methods like SVM, RandomForest). But I can only get a mean CV AUC of ~0.6.
", "rawMarkdown": "I didn't mean I support Yogananda's method, I mean that the dataset he used is part of the BraTS2021 dataset and he got a mean AUC of 0.93. Now using more data in BraTS2021, you can only get an AUC ~ 0.56. Other competitors also do not get very good results now, so one reason may be the dataset, one reason may be the results he reported.\n\nI didn't try Yogananda's method, I am using the traditional radiogenomics method (extract radiomic features + machine learning methods like SVM, RandomForest). But I can only get a mean CV AUC of ~0.6.\n", "votes": 1 }, { "id": 1511960, "postDate": "2021-09-13T19:26:20.597Z", "content": "@maxwell110 - So you used 32x32x32 75% overlapping voxels with a good deal of augmentation? And you leveraged a pretrained segmentation backbone as your starting point?
\nThis is similar to the method used by Yogananda et al. 2021.
\nI am in the process of doing experimenting with this methodology and just reached the stage of capturing the 75% overlapping voxels with at least some tumor contained (6 channel segmentationā¦ first 3 channels are MGMT positive and last 3 are MGMT negative). I now will serialize these as tfrecords, create an input and augmentation pipeline, and leverage a VNet with a backbone pretrained on a similar task as a starting point.
\nThat being said, if you already did this, would you be able to share your code? It would save me a lot of effort if it's already been tried in entirety.
\nFor more detail on the procedure from the paper please see this excerpt:
\n\n", "rawMarkdown": "@maxwell110 - So you used 32x32x32 75% overlapping voxels with a good deal of augmentation? And you leveraged a pretrained segmentation backbone as your starting point?\n\nThis is similar to the method used by [Yogananda et al. 2021](http://www.ajnr.org/content/early/2021/03/04/ajnr.A7029).\n\n---\n\nI am in the process of doing experimenting with this methodology and just reached the stage of capturing the 75% overlapping voxels with at least some tumor contained (6 channel segmentation... first 3 channels are MGMT positive and last 3 are MGMT negative). I now will serialize these as tfrecords, create an input and augmentation pipeline, and leverage a VNet with a backbone pretrained on a similar task as a starting point.\n\n---\n\nThat being said, if you already did this, would you be able to share your code? It would save me a lot of effort if it's already been tried in entirety.\n\n---\n\nFor more detail on the procedure from the paper please see this excerpt:\n\n> Seventy-five percent overlapping 3D patches (size: 32 Ć 32 Ć 32 voxels) were extracted from the training and in-training validation dataset. The patch extraction was performed as a translation in the x-y-z-plane. During training, only patches with at least 1 tumor voxel were included; thus, the number of patches included per training cases varied depending on the size of the tumor. For testing however, the entire image was sampled, including background masked voxels (of value zero). No patch from the same subject was mixed with the training, in-training validation, or testing datasets to prevent the problem of data leakage. Data augmentation steps included horizontal and vertical flipping, random and translational rotation, the addition of salt and pepper noise, the addition of Gaussian noise, and projective transformation. Additional data augmentation steps included down-sampling images by 50% and 25% (reducing the voxel resolution to 2 and 4 mm3). The data augmentation provided a total of approximately 300,000 patches for training and 300,000 patches for in-training validation for each fold. The networks were implemented using the Tensorflow30 backend engine, the Keras Python package, and an Adaptive Moment Estimation optimizer (Adam). The initial learning rate was set to 10ā5 with a batch size of 15 and maximal epochs of 100 for each fold.\n>\n> MGMT-net outputs 2 segmentation volumes (V1 and V2), which are combined to generate the voxelwise prediction of methylated and unmethylated MGMT promoter tumor voxels, respectively. The 2 volumes are fused, and the largest connected component (the 3D-connected component algorithm in Matlab [MathWorks]) is obtained as the single tumor-segmentation map. Majority voting over the voxelwise classes of methylated or unmethylated type provided a single MGMT promoter classification for each subject. Tesla V100s, P100, P40, and K80 NVIDIA-GPUs were used to implement the networks. This MGMT promoter determination process is fully automated, and a tumor segmentation map is a natural output of the voxelwise classification approach." }, { "id": 1754770, "postDate": "2022-04-14T02:11:11.890Z", "content": "Seventy-five percent overlapping 3D patches (size: 32 Ć 32 Ć 32 voxels) were extracted from the training and in-training validation dataset. The patch extraction was performed as a translation in the x-y-z-plane. During training, only patches with at least 1 tumor voxel were included; thus, the number of patches included per training cases varied depending on the size of the tumor. For testing however, the entire image was sampled, including background masked voxels (of value zero). No patch from the same subject was mixed with the training, in-training validation, or testing datasets to prevent the problem of data leakage. Data augmentation steps included horizontal and vertical flipping, random and translational rotation, the addition of salt and pepper noise, the addition of Gaussian noise, and projective transformation. Additional data augmentation steps included down-sampling images by 50% and 25% (reducing the voxel resolution to 2 and 4 mm3). The data augmentation provided a total of approximately 300,000 patches for training and 300,000 patches for in-training validation for each fold. The networks were implemented using the Tensorflow30 backend engine, the Keras Python package, and an Adaptive Moment Estimation optimizer (Adam). The initial learning rate was set to 10ā5 with a batch size of 15 and maximal epochs of 100 for each fold.
\nMGMT-net outputs 2 segmentation volumes (V1 and V2), which are combined to generate the voxelwise prediction of methylated and unmethylated MGMT promoter tumor voxels, respectively. The 2 volumes are fused, and the largest connected component (the 3D-connected component algorithm in Matlab [MathWorks]) is obtained as the single tumor-segmentation map. Majority voting over the voxelwise classes of methylated or unmethylated type provided a single MGMT promoter classification for each subject. Tesla V100s, P100, P40, and K80 NVIDIA-GPUs were used to implement the networks. This MGMT promoter determination process is fully automated, and a tumor segmentation map is a natural output of the voxelwise classification approach.
\n
hey @maxwell110 can you share the implementations of the paper here? Yogananda et al. 2021.
", "rawMarkdown": "hey @maxwell110 can you share the implementations of the paper here? Yogananda et al. 2021." } ] }, { "id": 1479406, "postDate": "2021-08-18T13:02:53.787Z", "content": "I'm starting to believe that there isn't sufficient data to create anything clinically useful here. There's too much variation in the images for such a small sample size.
\nI think the top scoring models will be learning on something other than MGMT_value and won't actually perform in the real world.
\nIt makes me wonder if a competition like this might do more harm than good by legitimizing mediocre solutions based on a fraction of a score from a lucky run.
", "rawMarkdown": "I'm starting to believe that there isn't sufficient data to create anything clinically useful here. There's too much variation in the images for such a small sample size.\n\nI think the top scoring models will be learning on something other than MGMT_value and won't actually perform in the real world. \n\nIt makes me wonder if a competition like this might do more harm than good by legitimizing mediocre solutions based on a fraction of a score from a lucky run.", "votes": 12, "replies": [ { "id": 1481021, "postDate": "2021-08-19T09:18:49.150Z", "content": "\n\n\nI think the top-scoring models will be learning on something other than MGMT_value and won't actually perform in the real world.
\n
Big Issue!
", "rawMarkdown": "@davidbroberts \n\n> I think the top-scoring models will be learning on something other than MGMT_value and won't actually perform in the real world.\n\nBig Issue! ", "votes": 1 }, { "id": 1503292, "postDate": "2021-09-05T08:26:17.693Z", "content": "@davidbroberts Yeah was wondering the same, why did they put such low examples ?
", "rawMarkdown": "@davidbroberts Yeah was wondering the same, why did they put such low examples ?" } ] }, { "id": 1480093, "postDate": "2021-08-18T19:49:27.843Z", "content": "I think monitoring cross-entropy loss might be best there. I haven't been able to get under 0.66 validation (single fold). I definitely think this is going to be more of a data centric problem as David Austin describes. There are some ideas I want to try but really have to limit the resolution dimensions to get any experiments done. I've spent more time pre-processing and finding ways to manipulate 3D data than actual modeling.
", "rawMarkdown": "I think monitoring cross-entropy loss might be best there. I haven't been able to get under 0.66 validation (single fold). I definitely think this is going to be more of a data centric problem as David Austin describes. There are some ideas I want to try but really have to limit the resolution dimensions to get any experiments done. I've spent more time pre-processing and finding ways to manipulate 3D data than actual modeling.", "votes": 5 }, { "id": 1483081, "postDate": "2021-08-20T12:48:40.740Z", "content": "We havenāt yet, but one approach we will be trying is taking data from task 1 (along with labels) converting to a similar representation to Task 2 data and then training a segmentation model to act as an intermediary model/step in classification.
\nWe know volume/shape has some part in determining the MGMT status, so I think this intuitively makes sense.
\nIām hoping by the end of the competition we will hopefully see models scoring above 75% consistently w/out hand labels.
\nWhether this will, in fact, be clinically relevant, remains to be seen.
", "rawMarkdown": "We havenāt yet, but one approach we will be trying is taking data from task 1 (along with labels) converting to a similar representation to Task 2 data and then training a segmentation model to act as an intermediary model/step in classification.\n\nWe know volume/shape has some part in determining the MGMT status, so I think this intuitively makes sense. \n\nIām hoping by the end of the competition we will hopefully see models scoring above 75% consistently w/out hand labels. \n\nWhether this will, in fact, be clinically relevant, remains to be seen. ", "votes": 3, "replies": [ { "id": 1483179, "postDate": "2021-08-20T13:44:16.820Z", "content": "Could you please elaborate more what you meant by converting Task-1 data to a similar representation to Task 2. The task-1 data does not have MGMT status.
\nI am trying to understand how we can use task-1 data for task-2. If you could share your knowledge on that it will be really appreciated.
Both Task 1 and Task 2 have a BraTSID associated with each discrete piece of data. You can use this BraTSID as a key to combining the segmentation maps and MGMT status. At least that's the hope. I have yet to do the diligence and convert between one and the otherā¦
\nThat being said we will be working on it soon and we generally share everything publically as we go. So keep an eye out!
", "rawMarkdown": "Both Task 1 and Task 2 have a BraTSID associated with each discrete piece of data. You can use this BraTSID as a key to combining the segmentation maps and MGMT status. At least that's the hope. I have yet to do the diligence and convert between one and the other...\n\nThat being said we will be working on it soon and we generally share everything publically as we go. So keep an eye out!", "votes": 4 } ] }, { "id": 1503128, "postDate": "2021-09-05T03:10:01.803Z", "content": "I'm struggling to generate good models, and I look forward to the winning solutions. All of my models basically guess average(MGMT_value), and it is hard to trust CV.
", "rawMarkdown": "I'm struggling to generate good models, and I look forward to the winning solutions. All of my models basically guess average(MGMT_value), and it is hard to trust CV.", "votes": 4, "replies": [ { "id": 1503201, "postDate": "2021-09-05T06:11:55.977Z", "content": "Any reason why you are not believing in CV?
", "rawMarkdown": "Any reason why you are not believing in CV?" }, { "id": 1503210, "postDate": "2021-09-05T06:23:57.863Z", "content": "For example, some of my best folds score 0.67, even 0.65 one time. Now just study the range of the predictionsā¦I saw that they clustered around average(MGMT_value), and had a few crazy outlier predictions at 0.90+ā¦ if the model can get lucky and through noise/accident it gets a couple outliers correct, your CV will look super awesome. But it won't hold up on LB - so I was overfitting on the epochs with fantastic CV.
", "rawMarkdown": "For example, some of my best folds score 0.67, even 0.65 one time. Now just study the range of the predictions...I saw that they clustered around average(MGMT_value), and had a few crazy outlier predictions at 0.90+... if the model can get lucky and through noise/accident it gets a couple outliers correct, your CV will look super awesome. But it won't hold up on LB - so I was overfitting on the epochs with fantastic CV.", "votes": 2 } ] }, { "id": 1494529, "postDate": "2021-08-28T18:38:36.737Z", "content": "Using a 3d CNN on the FLAIR images I'm getting a validation score of BCE 0.6811 on the validation set, 80/20 train test split. In this model I randomly rotated the images 180 degrees, which seems to yield a 1% improvement. However, I'm not doing a k-fold CV so this improvement could potentially be noise. I'm using a little dropout of p=.9 in the top two layers to regularize a bit. I haven't sensitivity tested w/ and w/o dropout to see what happens. I'm using 200x200x70 resolution using 3 convolution layers with channels (in-out): 1-8, 8-16, 16-8. Best score is at 6th epoch, .001 LR.
", "rawMarkdown": "Using a 3d CNN on the FLAIR images I'm getting a validation score of BCE 0.6811 on the validation set, 80/20 train test split. In this model I randomly rotated the images 180 degrees, which seems to yield a 1% improvement. However, I'm not doing a k-fold CV so this improvement could potentially be noise. I'm using a little dropout of p=.9 in the top two layers to regularize a bit. I haven't sensitivity tested w/ and w/o dropout to see what happens. I'm using 200x200x70 resolution using 3 convolution layers with channels (in-out): 1-8, 8-16, 16-8. Best score is at 6th epoch, .001 LR.", "votes": 4, "replies": [ { "id": 1494563, "postDate": "2021-08-28T19:16:53.930Z", "content": "Are you using your own custom model? if yes, it is good. I could not finetune the big effnet3d's, I was also thinking of making a custom 3D NN myself.
", "rawMarkdown": "Are you using your own custom model? if yes, it is good. I could not finetune the big effnet3d's, I was also thinking of making a custom 3D NN myself." }, { "id": 1494591, "postDate": "2021-08-28T20:09:00.327Z", "content": "It's an adaptation of this R torch example, https://torch.mlverse.org/start/guess_the_correlation/ I'm new to neural networks so I basically just took this starter example, changed the 2d cnn functions to 3d and adjusted the channel sizes to something that would run. I'll reply with the cnn layers that I ended up with when I got home later today. Pardon the R! Haha. Are you finding you're able to train on a desktop computer graphics card? I can't even fit 2 observations into gpu memory with my nvidia 1070, so I'm just cpu training.
", "rawMarkdown": "It's an adaptation of this R torch example, https://torch.mlverse.org/start/guess_the_correlation/ I'm new to neural networks so I basically just took this starter example, changed the 2d cnn functions to 3d and adjusted the channel sizes to something that would run. I'll reply with the cnn layers that I ended up with when I got home later today. Pardon the R! Haha. Are you finding you're able to train on a desktop computer graphics card? I can't even fit 2 observations into gpu memory with my nvidia 1070, so I'm just cpu training." }, { "id": 1494616, "postDate": "2021-08-28T21:01:41.410Z", "content": "You can try running on kaggle's notebook,it worked when I tried
", "rawMarkdown": "You can try running on kaggle's notebook,it worked when I tried" }, { "id": 1495816, "postDate": "2021-08-29T20:12:01.257Z", "content": "This is the 3d cnn that I ended up with after adapting the starter tutorial linked above. So far it just uses the 1 channel, the FLAIR images, but it'd probably be easy to use all sets of images and change the first in_channels from 1 to 4:
\nnet <- nn_module(
\n\"MRI-3d-cnn\",
\ninitialize = function() {
\nself$conv1 <- nn_conv3d(in_channels = 1, out_channels = 8, kernel_size = c(3,3,3),stride=1)\nself$conv2 <- nn_conv3d(in_channels = 8, out_channels = 16, kernel_size = c(3,3,3),stride=1)\nself$conv3 <- nn_conv3d(in_channels = 16, out_channels = 8, kernel_size = c(3,3,3),stride=1)\n\nself$fc1 <- nn_linear(in_features = 161* 8*23, out_features = 16)\nself$fc2 <- nn_linear(in_features = 16, out_features = 1)\n},
\nforward = function(x) {
\nx %>% \n self$conv1() %>% \n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv2() %>%\n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv3() %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n torch_flatten(start_dim = 2) %>%\n self$fc1() %>%\n nnf_relu() %>%\n\n self$fc2()\n}
", "rawMarkdown": "This is the 3d cnn that I ended up with after adapting the starter tutorial linked above. So far it just uses the 1 channel, the FLAIR images, but it'd probably be easy to use all sets of images and change the first in_channels from 1 to 4:\n\nnet <- nn_module(\n \n \"MRI-3d-cnn\",\n \n initialize = function() {\n \n self$conv1 <- nn_conv3d(in_channels = 1, out_channels = 8, kernel_size = c(3,3,3),stride=1)\n self$conv2 <- nn_conv3d(in_channels = 8, out_channels = 16, kernel_size = c(3,3,3),stride=1)\n self$conv3 <- nn_conv3d(in_channels = 16, out_channels = 8, kernel_size = c(3,3,3),stride=1)\n \n self$fc1 <- nn_linear(in_features = 161* 8*23, out_features = 16)\n self$fc2 <- nn_linear(in_features = 16, out_features = 1)\n \n },\n \n forward = function(x) {\n \n x %>% \n self$conv1() %>% \n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv2() %>%\n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv3() %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n torch_flatten(start_dim = 2) %>%\n self$fc1() %>%\n nnf_relu() %>%\n\n self$fc2()\n }", "votes": 1 } ] }, { "id": 1487443, "postDate": "2021-08-23T16:23:59.370Z", "content": "Itās the same for me. My (single fold) validation loss is always around 0.65 - 0.69 and the validation roc score is hardly increasing. Unfortunately, I can only tell you what didnāt work for me:
\nAfter reading this and similar discussion, Iāll definitely switch to k-fold CV in order to assess my changes instead of a single train/validation split.
\nIām wondering if I should spend some time in playing with the orientations between the scans (sagittal vs. coronal vs. axial). Do you have any experiences? Is it worth it?
", "rawMarkdown": "Itās the same for me. My (single fold) validation loss is always around 0.65 - 0.69 and the validation roc score is hardly increasing. Unfortunately, I can only tell you what didnāt work for me:\n\n- It seems that the architecture is not that important (I tried various architectures such as 3D EfficientNet or 3D ResNet with ā4Dā images (4D because of the shape: 4xWIDTHxHEIGHTxDEPTH) as well as combining CNNs trained on each MRI type, respectively)\n- I put a lot of time in different slicing techniques, but again, nothing stood out in particular. For example, I took SLICING_NUMBER (e.g., 32) slices from the middle of an image or I only took each x-th (e.g., 2nd, 3rt, etc.) slice from an image. I even tried to average images\n- I tried different augmentation techniques for example 3D-MRI-specific augmentations such as RandomGhosting or RandomAnisotropy as well as basic 2D augmentations such as CLAHE etc. Again, none of the methods was superior to the others. \n\nAfter reading this and similar discussion, Iāll definitely switch to k-fold CV in order to assess my changes instead of a single train/validation split.\n\nIām wondering if I should spend some time in playing with the orientations between the scans (sagittal vs. coronal vs. axial). Do you have any experiences? Is it worth it? ", "votes": 4, "replies": [ { "id": 1487464, "postDate": "2021-08-23T16:36:45.313Z", "content": "I don't know if that would work, mainly now I am thinking of using some other MRI data.
\nThe thing is that I am worried if my model is finding useful features for MGMT. I am trying to use CAM to see what the model is learning,
\nSo I really don't know how to move ahead now.
I think we may have to handcraft some features.
\nIf you do the research there are some physiological identifiers that inform on MGMT Methylation Status. I believe things like tumour volume, etc. will be important.
\nSee here for a pretty cool paper. --> http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf
", "rawMarkdown": "I think we may have to handcraft some features.\n\nIf you do the research there are some physiological identifiers that inform on MGMT Methylation Status. I believe things like tumour volume, etc. will be important.\n\nSee here for a pretty cool paper. --> http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf", "votes": 7 }, { "id": 1487604, "postDate": "2021-08-23T18:00:58.147Z", "content": "Very interesting paper, thanks for sharing @dschettler8845
\nIf you know could you kindly clear me this thing.
\nIn this paper, they have classified between methylated and unmethylated MGMT promoter types,
\nand here we are doing, presence of MGMT promoter methylation classification.
\nAre both things the same?
@mrinath: I also had the idea of using external data to train a model from scratch followed by fine-tuning with the data from the challenge. I came across this git repo. You might find it useful too.
\n@dschettler8845: Interesting. I haven't though about such an approach. Thanks for the paper!
", "rawMarkdown": "@mrinath: I also had the idea of using external data to train a model from scratch followed by fine-tuning with the data from the challenge. I came across [this](https://github.com/Tencent/MedicalNet) git repo. You might find it useful too.\n\n@dschettler8845: Interesting. I haven't though about such an approach. Thanks for the paper!", "votes": 2 }, { "id": 1488827, "postDate": "2021-08-24T14:17:34.497Z", "content": "@mrinath - Yes. I believe they are the same thing. This is from the above paper:
\n\n\nMGMT is a DNA repair
\n
\n enzyme that protects normal and glioma cells from alkylating
\n chemotherapeutic agents. The methylation of the MGMT promoter is an example of epigenetic silencing, which results in a
\n loss of function of the MGMT enzyme and its protective effect
\n on glioma cells. The survival benefit incurred by MGMT promoter methylation in patients treated with temozolomide
\n (TMZ) was determined in 2005.1 Subsequent work by Stupp et
\n al2 has shown that in patients who received both radiation and
\n temozolomide, MGMT promoter methylation improved median survival compared with patients with unmethylated gliomas (21.7 versus 12.7 months).2 Long-term follow-up from that
\n initial study has further substantiated the survival benefit.2,3As
\n a result, determining MGMT promoter methylation status is
\n an important step in predicting survival and determining
\n treatment.
And this is from the competition description:
\n\n\nThe presence of a specific genetic sequence in the tumour known as MGMT promoter methylation has been shown to be a favourable prognostic factor and a strong predictor of responsiveness to chemotherapy.
\n
tl;dr
\nThe MGMT promoter is a thing everyone has. It protects tissues (BUT ALSO CANCER CELLS) from things (like chemo). When the MGMT promoter is methylated it will stop functioning properly (i.e. it stops protecting tissues from things like chemo). This might seem like a bad thing, but because it will ALSO protect the tumour cells from chemotherapeuticā¦ when it gets methylated (silenced), the chemotherapy can work as intended and treat the tumour! As the competition description also describes MGMT Promoter Methylation as what we are classifying, I believe it to be identifying the same thing.
\nI hope this helps!
", "rawMarkdown": "@mrinath - Yes. I believe they are the same thing. This is from the above paper:\n\n> MGMT is a DNA repair\nenzyme that protects normal and glioma cells from alkylating\nchemotherapeutic agents. The methylation of the MGMT promoter is an example of epigenetic silencing, which results in a\nloss of function of the MGMT enzyme and its protective effect\non glioma cells. The survival benefit incurred by MGMT promoter methylation in patients treated with temozolomide\n(TMZ) was determined in 2005.1 Subsequent work by Stupp et\nal2 has shown that in patients who received both radiation and\ntemozolomide, MGMT promoter methylation improved median survival compared with patients with unmethylated gliomas (21.7 versus 12.7 months).2 Long-term follow-up from that\ninitial study has further substantiated the survival benefit.2,3As\na result, determining MGMT promoter methylation status is\nan important step in predicting survival and determining\ntreatment.\n\nAnd this is from the competition description:\n\n> The presence of a specific genetic sequence in the tumour known as MGMT promoter methylation has been shown to be a favourable prognostic factor and a strong predictor of responsiveness to chemotherapy.\n\n---\n\n**tl;dr**\n\n**The MGMT promoter is a thing everyone has. It *protects* tissues (BUT ALSO CANCER CELLS) from things (like chemo). When the MGMT promoter is [*methylated*](https://en.wikipedia.org/wiki/Methylation#:~:text=Methylation%20is%20a%20form%20of,group%20replacing%20a%20hydrogen%20atom.&text=In%20biological%20systems%2C%20methylation%20is,protein%20function%2C%20and%20RNA%20processing.) it will stop functioning properly (i.e. it stops protecting tissues from things like chemo). This might seem like a bad thing, but because it will ALSO protect the tumour cells from chemotherapeutic... when it gets methylated (*silenced*), the chemotherapy can work as intended and treat the tumour! As the competition description also describes MGMT Promoter Methylation as what we are classifying, I believe it to be identifying the same thing.**\n\n---\n\nI hope this helps!\n\n ", "votes": 7 }, { "id": 1488853, "postDate": "2021-08-24T14:36:58.643Z", "content": "Thank you very much!!
", "rawMarkdown": "Thank you very much!!" } ] }, { "id": 1516068, "postDate": "2021-09-17T20:57:37.860Z", "content": "No matter what I try, I still have val auc of 0.5-0.6, the model simply doesn't learn anything.
", "rawMarkdown": "No matter what I try, I still have val auc of 0.5-0.6, the model simply doesn't learn anything.", "votes": 1 }, { "id": 1502187, "postDate": "2021-09-04T01:42:06.893Z", "content": "I've tried 3d CNN and 2d CNN + RNN. As same as you~ My val & test results only over 0.5~0.6. :(
", "rawMarkdown": "I've tried 3d CNN and 2d CNN + RNN. As same as you~ My val & test results only over 0.5~0.6. :(", "votes": 1 }, { "id": 1479010, "postDate": "2021-08-18T08:43:56.277Z", "content": "For one strategy, I picked up one slice including glioblastoma per MRI type & ID and used them for 4ch CNN training, but the val-auc was only 0.6.
\nEven if I used more slices, the result was not so different.
\nIf this competition is meaningful, we should detect some slices which contribute to the MGMT 0/1 classification.
\nJust using slices including tumor is not enough, I guess.
I think you are correct
\n@maxwell110 said that this is a type of \"weakly supervised learning\".
\nyou can check out his comment, in this discussion post-https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/252900
Is external data the key here?š¤š¤
", "rawMarkdown": "Is external data the key here?š¤š¤", "votes": 1 }, { "id": 1505074, "postDate": "2021-09-06T22:59:49.557Z", "content": "No success here either. I thought I had figured something out but then I computed the standard deviation and saw that it is too high (approx. 0.16). My mean auc CV is 0.61 but because the std is so high, the score does not mean anything. This is why I like regular CV more than OOF. By having two estimators (standard deviation and mean), there is more information.
", "rawMarkdown": "No success here either. I thought I had figured something out but then I computed the standard deviation and saw that it is too high (approx. 0.16). My mean auc CV is 0.61 but because the std is so high, the score does not mean anything. This is why I like regular CV more than OOF. By having two estimators (standard deviation and mean), there is more information.", "votes": 2 }, { "id": 1551959, "postDate": "2021-10-21T03:29:18.477Z", "content": "Thank you for this discussion, was feeling lost as well
", "rawMarkdown": "Thank you for this discussion, was feeling lost as well" }, { "id": 1518109, "postDate": "2021-09-20T12:08:07.720Z", "content": "Me too. I dont think my model learn something..
", "rawMarkdown": "Me too. I dont think my model learn something.." }, { "id": 1514195, "postDate": "2021-09-15T19:03:44.330Z", "content": "what is the general training accuracy and roc for everyone? i guess public test set is quite disjoint from training set
", "rawMarkdown": "what is the general training accuracy and roc for everyone? i guess public test set is quite disjoint from training set" }, { "id": 1500327, "postDate": "2021-09-02T09:37:43.797Z", "content": "Did you come to a conclusion on the question? I too wonder if this is possible at all, I tried to train a few models on the segmented tumors, but the model learns absolutely nothing.
", "rawMarkdown": "Did you come to a conclusion on the question? I too wonder if this is possible at all, I tried to train a few models on the segmented tumors, but the model learns absolutely nothing. ", "replies": [ { "id": 1500332, "postDate": "2021-09-02T09:43:10.640Z", "content": "There are some papers stating that they are actually successful in detecting MGMT status,
\nlike this one http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf
\nSo I guess it's actually possible, but I don't know why our models are rarely learning.
@mrinath My best understanding is that - the result from the paper uses tumor masks, whereas our dataset doesn't have that.
", "rawMarkdown": "@mrinath My best understanding is that - the result from the paper uses tumor masks, whereas our dataset doesn't have that. ", "votes": 1 }, { "id": 1502978, "postDate": "2021-09-04T19:46:24.037Z", "content": "Yes, you are correct, thats why I think many people are trying to work with task 1.
\nI can be wrong
I tried to use one of the top segmentator from 2019 BRATS, it is really good and I can clearly see that, for most of the time segment the correct place for tumor. Although the network learns NOTHING from the segmented tumor. I even include some additional parts (a large bounding box to be precise) around the segmented tumor in case it misses something, but still nothing.
\nI'm starting to think that either my model is crap, or the appearance feature in our data is just not helping (maybe too low resolution?); maybe the model only learn from the location, and I willingly give up the spatial information of the tumor when I rely solely on the segmented images. This can explain why the 3D CNN performs somewhat well: My hypothesis is that it tries to predict the MGMT base on where the tumor is located in the brain. That being said, if the hypothesis holds true, then I am very pessimistic about this competition, so I wholeheartedly hope that I made a fool out of myself by stating this hypothesis :D
", "rawMarkdown": "I tried to use one of the top segmentator from 2019 BRATS, it is really good and I can clearly see that, for most of the time segment the correct place for tumor. Although the network learns NOTHING from the segmented tumor. I even include some additional parts (a large bounding box to be precise) around the segmented tumor in case it misses something, but still nothing. \n\nI'm starting to think that either my model is crap, or the appearance feature in our data is just not helping (maybe too low resolution?); maybe the model only learn from the location, and I willingly give up the spatial information of the tumor when I rely solely on the segmented images. This can explain why the 3D CNN performs somewhat well: My hypothesis is that it tries to predict the MGMT base on where the tumor is located in the brain. That being said, if the hypothesis holds true, then I am very pessimistic about this competition, so I wholeheartedly hope that I made a fool out of myself by stating this hypothesis :D ", "votes": 4 } ] }, { "id": 1479127, "postDate": "2021-08-18T09:54:25.237Z", "content": "Same here. It would be nice to get some clues from the people on top of the lb.
", "rawMarkdown": "Same here. It would be nice to get some clues from the people on top of the lb.", "replies": [ { "id": 1479149, "postDate": "2021-08-18T10:07:48.370Z", "content": "Hope the top scores are not from hand labelling
", "rawMarkdown": "Hope the top scores are not from hand labelling", "votes": 1 }, { "id": 1479374, "postDate": "2021-08-18T12:51:14.883Z", "content": "Even if they are, they would score 0 in the private test set, right?
", "rawMarkdown": "Even if they are, they would score 0 in the private test set, right?", "votes": 2 } ] }, { "id": 1511601, "postDate": "2021-09-13T14:38:01.780Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1511865, "postDate": "2021-09-13T18:28:20.390Z", "content": "Thats Handlabelling, you can see the [HL] tag
", "rawMarkdown": "Thats Handlabelling, you can see the [HL] tag" } ] } ], "comments": [ { "id": 1479828, "author_name": "David Austin", "author_url": "", "post_date": "2021-08-18T16:55:53.333000", "content": "I have dealt with the same problem. My 0.764 is much higher than the cross validation suggests it should be so I'm attributing it largely to statistical aberration of low sample size. I'm also running some post processing that weights the ranking between FLAIR and T2 channels that gives a slight boost.
\nI get nearly identical results whether I'm using a simple, shallow 2 layer CNN or EFF-B7. My intuition is leading me to this being more a data centric AI problem rather than a modeling one and the majority of my time spent has been with prepping the data. I've found only one thing that seems to have some effect.
\nI'm using Effnet-B0 3D CNN and have put significant work into orienting, interpolating to scale (1mm x 1mm x 1mm), and normalizing the images. The one thing that shows signs of life is normalizing in 3D voxel space using 3D CLAHE. Most public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.
Some nice points @tivfrvqhs5
\nIf you don't mind, can you tell me for how are you normalizing them? I mean using the imagenet standard mean,std or the actual mean,std of the data.
Multi-dimensional CLAHE is the normalization method. The procedure is outlined in this paper.
", "votes": 11, "replies": [] }, { "id": 1480838, "author_name": "Innat", "author_url": "", "post_date": "2021-08-19T07:38:50.940000", "content": "The mclahe provides tf 1.x and numpy implementation. And for tf, there is a work progress in tf 2.x. In the meantime we can use tf_clahe package.
Instead of tf.clahe, here is another close alternative
# https://github.com/tensorflow/models\ndef equalize(image, mode='grayscale'):\n def scale_channel(im, c):\n \"\"\"Scale the data in the channel to implement equalize.\"\"\"\n im = tf.cast(im[..., c], tf.int32)\n # Compute the histogram of the image channel.\n histo = tf.histogram_fixed_width(im, [0, 255], nbins=256)\n\n # For the purposes of computing the step, filter out the nonzeros.\n nonzero = tf.where(tf.not_equal(histo, 0))\n nonzero_histo = tf.reshape(tf.gather(histo, nonzero), [-1])\n step = (tf.reduce_sum(nonzero_histo) - nonzero_histo[-1]) // 255\n\n def build_lut(histo, step):\n # Compute the cumulative sum, shifting by step // 2\n # and then normalization by step.\n lut = (tf.cumsum(histo) + (step // 2)) // step\n # Shift lut, prepending with 0.\n lut = tf.concat([[0], lut[:-1]], 0)\n # Clip the counts to be in range. This is done\n # in the C code for image.point.\n return tf.clip_by_value(lut, 0, 255)\n\n # If step is zero, return the original image. Otherwise, build\n # lut from the full histogram and step and then index from it.\n result = tf.cond(\n tf.equal(step, 0), lambda: im,\n lambda: tf.gather(build_lut(histo, step), im))\n return tf.cast(result, tf.uint8)\n\n if mode == 'grayscale':\n image = scale_channel(image, 0)\n return tf.cast(image, tf.float32)\n elif mode == 'rgb':\n s1 = scale_channel(image, 0)\n s2 = scale_channel(image, 1)\n s3 = scale_channel(image, 2)\n image = tf.stack([s1, s2, s3], -1)\n return tf.cast(image, tf.float32)\n@ipythonx in which part are you using mclahe,I am confused.
\nI mean do you use this when you are converting your dicoms to numpy arrays or first convert them to numpy arrays and then apply mclahe
@mrinath cc: tf_clahe developer @isaacsears
\nWe used the above tf 2.x implementation of clahe. But afaik, it's not yet supported for 3d volume. But we can use it in tf_image_augmentation pipeline. This augmentation function is not much efficient (we think) as for slice-wise operation and would be better for batch-wise transformation (known issue).
\n
For pytorch, we can use the official mclahe-numpy implementation.
\n\nMost public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.
\n
Could you please elaborate on that?
", "votes": 0, "replies": [] }, { "id": 1481311, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-19T12:27:54.053000", "content": "Hi @tivfrvqhs5 is 764 your score is from kfold ensemble or single fold?
", "votes": 0, "replies": [] } ] }, { "id": 1503997, "author_name": "Maxwell", "author_url": "", "post_date": "2021-09-06T01:18:24.210000", "content": "As the Venn diagram in the link below shows, there is some overlap in the cases between Task 1 and Task 2 of BraTS 2021.
\n
Specifically, out of the Task 2 training data hosted on Kaggle, 577 cases overlap with the Task 1 training data. Therefore, I conducted a segmentation-based MGMT classification experiment as described in Yogananda et al. 2021, giving only Task 2 labels to the 577 cases of Task 1 data that overlap with Task 2. The reason for the above experimental setup is that the MRI images of Task 1 are better than those of Task 2 because they are carefully co-registered.
\nThe model used was a light-weight pre-trained 3D CNN model, and all 5 folds were trained with using 4 modalities.
The experimental results were disappointing š
\nUnlike the excellent results in Yogananda's paper, my local AUC was about 0.56, which is slightly better than chance level, almost nothing was learned. At this point, I am not sure if our MRI data does not have enough information to classify MGMT status, or if features for MGMT classification are contained in something else that cannot be learned by the CNN model. However, since Yogananda et al. succeeded in classifying MGMT with 3D CNN, I think it is possible that the label given in the competition, which is supposed to be Ground Truth, might be wrong in some cases.
In addition, Baid et al., 2021 has the following definition of MGMT.
\n\n\nThe percent methylation above 10% was interpreted as positive. A sample below 10% methylation was interpreted as negative. A sample below 10% methylation was interpreted as negative. For the latter approach, a total of 17 MGMT promoter CpG sites were amplified by nested polymerase chain reaction (PCR) using a bisulte treated DNA template. Quantitative PCR was performed for each CpG site to determine its methylation status. A result of 2% or more methylated CpG sites in the MGMT promoter (out of 17 total sites) was considered a positive result.
\n
Taking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.
", "votes": 12, "replies": [ { "id": 1504065, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-09-06T04:02:04.647000", "content": "Taking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.
\nYes, I support this, It would have been better for the models.
\n@ujjwalbaid and other competition organizers, is there still time left to do this?
As no one could reproduce their result or get useful information from visual features(even combining task1/2 data), the paper's claim feels suspicious for me.
", "votes": 3, "replies": [] }, { "id": 1505586, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-09-07T11:48:51.143000", "content": "@steaphan
\nMe too, it's impossible for this data.
The paper of Yogananda et al. 2021 (http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf) used the data from the TCIA database, and achieves a mean AUC of 0.93 [SD, 0.01] for predicting MGMT status.
\nThe paper has the following descriptions, so I trust it a lot just from reading the paper.
\nI don't know whether they downloaded data from here (I am not sure):
\nBut if yes, then the dataset they downloaded should be part of the BraTS2021 segmentation dataset, as stated below:
\n\"For BraTS'17, expert neuroradiologists have radiologically assessed the complete original TCIA glioma collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and categorized each scan as pre- or post-operative. Subsequently, all the pre-operative TCIA scans (135 GBM and 108 LGG) were annotated by experts for the various glioma sub-regions and included in this year's BraTS datasets. \" (https://www.med.upenn.edu/cbica/brats2021/).
\nSo I am a little shocked that you only get about an AUC of 0.56 by his method.
", "votes": 4, "replies": [] }, { "id": 1508833, "author_name": "", "author_url": "", "post_date": "2021-09-10T15:50:54.277000", "content": "", "votes": 0, "replies": [] }, { "id": 1509144, "author_name": "Maxwell", "author_url": "", "post_date": "2021-09-10T22:35:17.313000", "content": "I'm shocked, just as you are shocked.
\nI hope I have made some fundamental mistake, otherwise I think this competition will turn into a lottery competition where the final ranking will be almost entirely governed by randomness.
In fact, for simple brain tumor segmentation, I have been able to get good results even with a light model and no particular pursuit of accuracy.
\nThe three figures in the link below are 3D segmentation results obtained by 3D CNN using the task1 dataset. (Left: Ground Truth map, Right: Predicted map)
\nYou can see that the mapping is reasonably good (the validation result of case-wise BCE is about 0.05, which is not bad).
However, when I followed the method of Yogananda's paper and used multi-class segmentation according to MGMT labels, I got a disappointing result of AUC ~ 0.56 š
\nI have a feeling that this number can be improved somewhat. However, I think it may not be as good as the value in their paper.
\nIn short, my 3D CNN model can segment brain tumors, but it does not have enough expressive power to classify MGMTs.
By the way, I think your current score is not as high as the AUC presented in Yogananda's paper, but I was curious why you support Yogananda's paper at this stage.
\nSo far, even though Yogananda's paper is known to many of the participants, I don't think there are any reports that have been able to significantly increase LB scores independent of randomness in ways other than Hand Labeling or Probing.
In any case, I hope that my impression is wrong and that this competition will be meaningful.
", "votes": 1, "replies": [] }, { "id": 1509331, "author_name": "ping", "author_url": "", "post_date": "2021-09-11T06:38:23.510000", "content": "I didn't mean I support Yogananda's method, I mean that the dataset he used is part of the BraTS2021 dataset and he got a mean AUC of 0.93. Now using more data in BraTS2021, you can only get an AUC ~ 0.56. Other competitors also do not get very good results now, so one reason may be the dataset, one reason may be the results he reported.
\nI didn't try Yogananda's method, I am using the traditional radiogenomics method (extract radiomic features + machine learning methods like SVM, RandomForest). But I can only get a mean CV AUC of ~0.6.
", "votes": 1, "replies": [] }, { "id": 1511960, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-09-13T19:26:20.597000", "content": "@maxwell110 - So you used 32x32x32 75% overlapping voxels with a good deal of augmentation? And you leveraged a pretrained segmentation backbone as your starting point?
\nThis is similar to the method used by Yogananda et al. 2021.
\nI am in the process of doing experimenting with this methodology and just reached the stage of capturing the 75% overlapping voxels with at least some tumor contained (6 channel segmentationā¦ first 3 channels are MGMT positive and last 3 are MGMT negative). I now will serialize these as tfrecords, create an input and augmentation pipeline, and leverage a VNet with a backbone pretrained on a similar task as a starting point.
\nThat being said, if you already did this, would you be able to share your code? It would save me a lot of effort if it's already been tried in entirety.
\nFor more detail on the procedure from the paper please see this excerpt:
\n\n", "votes": 0, "replies": [] }, { "id": 1754770, "author_name": "Bibhabasu Mohapatra", "author_url": "", "post_date": "2022-04-14T02:11:11.890000", "content": "Seventy-five percent overlapping 3D patches (size: 32 Ć 32 Ć 32 voxels) were extracted from the training and in-training validation dataset. The patch extraction was performed as a translation in the x-y-z-plane. During training, only patches with at least 1 tumor voxel were included; thus, the number of patches included per training cases varied depending on the size of the tumor. For testing however, the entire image was sampled, including background masked voxels (of value zero). No patch from the same subject was mixed with the training, in-training validation, or testing datasets to prevent the problem of data leakage. Data augmentation steps included horizontal and vertical flipping, random and translational rotation, the addition of salt and pepper noise, the addition of Gaussian noise, and projective transformation. Additional data augmentation steps included down-sampling images by 50% and 25% (reducing the voxel resolution to 2 and 4 mm3). The data augmentation provided a total of approximately 300,000 patches for training and 300,000 patches for in-training validation for each fold. The networks were implemented using the Tensorflow30 backend engine, the Keras Python package, and an Adaptive Moment Estimation optimizer (Adam). The initial learning rate was set to 10ā5 with a batch size of 15 and maximal epochs of 100 for each fold.
\nMGMT-net outputs 2 segmentation volumes (V1 and V2), which are combined to generate the voxelwise prediction of methylated and unmethylated MGMT promoter tumor voxels, respectively. The 2 volumes are fused, and the largest connected component (the 3D-connected component algorithm in Matlab [MathWorks]) is obtained as the single tumor-segmentation map. Majority voting over the voxelwise classes of methylated or unmethylated type provided a single MGMT promoter classification for each subject. Tesla V100s, P100, P40, and K80 NVIDIA-GPUs were used to implement the networks. This MGMT promoter determination process is fully automated, and a tumor segmentation map is a natural output of the voxelwise classification approach.
\n
hey @maxwell110 can you share the implementations of the paper here? Yogananda et al. 2021.
", "votes": 0, "replies": [] } ] }, { "id": 1479406, "author_name": "David Roberts", "author_url": "", "post_date": "2021-08-18T13:02:53.787000", "content": "I'm starting to believe that there isn't sufficient data to create anything clinically useful here. There's too much variation in the images for such a small sample size.
\nI think the top scoring models will be learning on something other than MGMT_value and won't actually perform in the real world.
\nIt makes me wonder if a competition like this might do more harm than good by legitimizing mediocre solutions based on a fraction of a score from a lucky run.
", "votes": 12, "replies": [ { "id": 1481021, "author_name": "Innat", "author_url": "", "post_date": "2021-08-19T09:18:49.150000", "content": "\n\n\nI think the top-scoring models will be learning on something other than MGMT_value and won't actually perform in the real world.
\n
Big Issue!
", "votes": 1, "replies": [] }, { "id": 1503292, "author_name": "Md Zarif Ul Alam", "author_url": "", "post_date": "2021-09-05T08:26:17.693000", "content": "@davidbroberts Yeah was wondering the same, why did they put such low examples ?
", "votes": 0, "replies": [] } ] }, { "id": 1480093, "author_name": "Tim Yee", "author_url": "", "post_date": "2021-08-18T19:49:27.843000", "content": "I think monitoring cross-entropy loss might be best there. I haven't been able to get under 0.66 validation (single fold). I definitely think this is going to be more of a data centric problem as David Austin describes. There are some ideas I want to try but really have to limit the resolution dimensions to get any experiments done. I've spent more time pre-processing and finding ways to manipulate 3D data than actual modeling.
", "votes": 5, "replies": [] }, { "id": 1483081, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-20T12:48:40.740000", "content": "We havenāt yet, but one approach we will be trying is taking data from task 1 (along with labels) converting to a similar representation to Task 2 data and then training a segmentation model to act as an intermediary model/step in classification.
\nWe know volume/shape has some part in determining the MGMT status, so I think this intuitively makes sense.
\nIām hoping by the end of the competition we will hopefully see models scoring above 75% consistently w/out hand labels.
\nWhether this will, in fact, be clinically relevant, remains to be seen.
", "votes": 3, "replies": [ { "id": 1483179, "author_name": "Wfarzana", "author_url": "", "post_date": "2021-08-20T13:44:16.820000", "content": "Could you please elaborate more what you meant by converting Task-1 data to a similar representation to Task 2. The task-1 data does not have MGMT status.
\nI am trying to understand how we can use task-1 data for task-2. If you could share your knowledge on that it will be really appreciated.
Both Task 1 and Task 2 have a BraTSID associated with each discrete piece of data. You can use this BraTSID as a key to combining the segmentation maps and MGMT status. At least that's the hope. I have yet to do the diligence and convert between one and the otherā¦
\nThat being said we will be working on it soon and we generally share everything publically as we go. So keep an eye out!
", "votes": 4, "replies": [] } ] }, { "id": 1503128, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2021-09-05T03:10:01.803000", "content": "I'm struggling to generate good models, and I look forward to the winning solutions. All of my models basically guess average(MGMT_value), and it is hard to trust CV.
", "votes": 4, "replies": [ { "id": 1503201, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-09-05T06:11:55.977000", "content": "Any reason why you are not believing in CV?
", "votes": 0, "replies": [] }, { "id": 1503210, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2021-09-05T06:23:57.863000", "content": "For example, some of my best folds score 0.67, even 0.65 one time. Now just study the range of the predictionsā¦I saw that they clustered around average(MGMT_value), and had a few crazy outlier predictions at 0.90+ā¦ if the model can get lucky and through noise/accident it gets a couple outliers correct, your CV will look super awesome. But it won't hold up on LB - so I was overfitting on the epochs with fantastic CV.
", "votes": 2, "replies": [] } ] }, { "id": 1494529, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2021-08-28T18:38:36.737000", "content": "Using a 3d CNN on the FLAIR images I'm getting a validation score of BCE 0.6811 on the validation set, 80/20 train test split. In this model I randomly rotated the images 180 degrees, which seems to yield a 1% improvement. However, I'm not doing a k-fold CV so this improvement could potentially be noise. I'm using a little dropout of p=.9 in the top two layers to regularize a bit. I haven't sensitivity tested w/ and w/o dropout to see what happens. I'm using 200x200x70 resolution using 3 convolution layers with channels (in-out): 1-8, 8-16, 16-8. Best score is at 6th epoch, .001 LR.
", "votes": 4, "replies": [ { "id": 1494563, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-28T19:16:53.930000", "content": "Are you using your own custom model? if yes, it is good. I could not finetune the big effnet3d's, I was also thinking of making a custom 3D NN myself.
", "votes": 0, "replies": [] }, { "id": 1494591, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2021-08-28T20:09:00.327000", "content": "It's an adaptation of this R torch example, https://torch.mlverse.org/start/guess_the_correlation/ I'm new to neural networks so I basically just took this starter example, changed the 2d cnn functions to 3d and adjusted the channel sizes to something that would run. I'll reply with the cnn layers that I ended up with when I got home later today. Pardon the R! Haha. Are you finding you're able to train on a desktop computer graphics card? I can't even fit 2 observations into gpu memory with my nvidia 1070, so I'm just cpu training.
", "votes": 0, "replies": [] }, { "id": 1494616, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-28T21:01:41.410000", "content": "You can try running on kaggle's notebook,it worked when I tried
", "votes": 0, "replies": [] }, { "id": 1495816, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2021-08-29T20:12:01.257000", "content": "This is the 3d cnn that I ended up with after adapting the starter tutorial linked above. So far it just uses the 1 channel, the FLAIR images, but it'd probably be easy to use all sets of images and change the first in_channels from 1 to 4:
\nnet <- nn_module(
\n\"MRI-3d-cnn\",
\ninitialize = function() {
\nself$conv1 <- nn_conv3d(in_channels = 1, out_channels = 8, kernel_size = c(3,3,3),stride=1)\nself$conv2 <- nn_conv3d(in_channels = 8, out_channels = 16, kernel_size = c(3,3,3),stride=1)\nself$conv3 <- nn_conv3d(in_channels = 16, out_channels = 8, kernel_size = c(3,3,3),stride=1)\n\nself$fc1 <- nn_linear(in_features = 161* 8*23, out_features = 16)\nself$fc2 <- nn_linear(in_features = 16, out_features = 1)\n},
\nforward = function(x) {
\nx %>% \n self$conv1() %>% \n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv2() %>%\n nnf_dropout3d(p=.9) %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n self$conv3() %>%\n nnf_relu() %>%\n nnf_avg_pool3d(2) %>%\n\n torch_flatten(start_dim = 2) %>%\n self$fc1() %>%\n nnf_relu() %>%\n\n self$fc2()\n}
", "votes": 1, "replies": [] } ] }, { "id": 1487443, "author_name": "Oberfink", "author_url": "", "post_date": "2021-08-23T16:23:59.370000", "content": "Itās the same for me. My (single fold) validation loss is always around 0.65 - 0.69 and the validation roc score is hardly increasing. Unfortunately, I can only tell you what didnāt work for me:
\nAfter reading this and similar discussion, Iāll definitely switch to k-fold CV in order to assess my changes instead of a single train/validation split.
\nIām wondering if I should spend some time in playing with the orientations between the scans (sagittal vs. coronal vs. axial). Do you have any experiences? Is it worth it?
", "votes": 4, "replies": [ { "id": 1487464, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-23T16:36:45.313000", "content": "I don't know if that would work, mainly now I am thinking of using some other MRI data.
\nThe thing is that I am worried if my model is finding useful features for MGMT. I am trying to use CAM to see what the model is learning,
\nSo I really don't know how to move ahead now.
I think we may have to handcraft some features.
\nIf you do the research there are some physiological identifiers that inform on MGMT Methylation Status. I believe things like tumour volume, etc. will be important.
\nSee here for a pretty cool paper. --> http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf
", "votes": 7, "replies": [] }, { "id": 1487604, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-23T18:00:58.147000", "content": "Very interesting paper, thanks for sharing @dschettler8845
\nIf you know could you kindly clear me this thing.
\nIn this paper, they have classified between methylated and unmethylated MGMT promoter types,
\nand here we are doing, presence of MGMT promoter methylation classification.
\nAre both things the same?
@mrinath: I also had the idea of using external data to train a model from scratch followed by fine-tuning with the data from the challenge. I came across this git repo. You might find it useful too.
\n@dschettler8845: Interesting. I haven't though about such an approach. Thanks for the paper!
", "votes": 2, "replies": [] }, { "id": 1488827, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-24T14:17:34.497000", "content": "@mrinath - Yes. I believe they are the same thing. This is from the above paper:
\n\n\nMGMT is a DNA repair
\n
\n enzyme that protects normal and glioma cells from alkylating
\n chemotherapeutic agents. The methylation of the MGMT promoter is an example of epigenetic silencing, which results in a
\n loss of function of the MGMT enzyme and its protective effect
\n on glioma cells. The survival benefit incurred by MGMT promoter methylation in patients treated with temozolomide
\n (TMZ) was determined in 2005.1 Subsequent work by Stupp et
\n al2 has shown that in patients who received both radiation and
\n temozolomide, MGMT promoter methylation improved median survival compared with patients with unmethylated gliomas (21.7 versus 12.7 months).2 Long-term follow-up from that
\n initial study has further substantiated the survival benefit.2,3As
\n a result, determining MGMT promoter methylation status is
\n an important step in predicting survival and determining
\n treatment.
And this is from the competition description:
\n\n\nThe presence of a specific genetic sequence in the tumour known as MGMT promoter methylation has been shown to be a favourable prognostic factor and a strong predictor of responsiveness to chemotherapy.
\n
tl;dr
\nThe MGMT promoter is a thing everyone has. It protects tissues (BUT ALSO CANCER CELLS) from things (like chemo). When the MGMT promoter is methylated it will stop functioning properly (i.e. it stops protecting tissues from things like chemo). This might seem like a bad thing, but because it will ALSO protect the tumour cells from chemotherapeuticā¦ when it gets methylated (silenced), the chemotherapy can work as intended and treat the tumour! As the competition description also describes MGMT Promoter Methylation as what we are classifying, I believe it to be identifying the same thing.
\nI hope this helps!
", "votes": 7, "replies": [] }, { "id": 1488853, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-24T14:36:58.643000", "content": "Thank you very much!!
", "votes": 0, "replies": [] } ] }, { "id": 1516068, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-09-17T20:57:37.860000", "content": "No matter what I try, I still have val auc of 0.5-0.6, the model simply doesn't learn anything.
", "votes": 1, "replies": [] }, { "id": 1502187, "author_name": "Ziquan Wei", "author_url": "", "post_date": "2021-09-04T01:42:06.893000", "content": "I've tried 3d CNN and 2d CNN + RNN. As same as you~ My val & test results only over 0.5~0.6. :(
", "votes": 1, "replies": [] }, { "id": 1479010, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-08-18T08:43:56.277000", "content": "For one strategy, I picked up one slice including glioblastoma per MRI type & ID and used them for 4ch CNN training, but the val-auc was only 0.6.
\nEven if I used more slices, the result was not so different.
\nIf this competition is meaningful, we should detect some slices which contribute to the MGMT 0/1 classification.
\nJust using slices including tumor is not enough, I guess.
I think you are correct
\n@maxwell110 said that this is a type of \"weakly supervised learning\".
\nyou can check out his comment, in this discussion post-https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/252900
Is external data the key here?š¤š¤
", "votes": 1, "replies": [] }, { "id": 1505074, "author_name": "noidea", "author_url": "", "post_date": "2021-09-06T22:59:49.557000", "content": "No success here either. I thought I had figured something out but then I computed the standard deviation and saw that it is too high (approx. 0.16). My mean auc CV is 0.61 but because the std is so high, the score does not mean anything. This is why I like regular CV more than OOF. By having two estimators (standard deviation and mean), there is more information.
", "votes": 2, "replies": [] }, { "id": 1551959, "author_name": "Tanmay Gupta", "author_url": "", "post_date": "2021-10-21T03:29:18.477000", "content": "Thank you for this discussion, was feeling lost as well
", "votes": 0, "replies": [] }, { "id": 1518109, "author_name": "wyyyywwy", "author_url": "", "post_date": "2021-09-20T12:08:07.720000", "content": "Me too. I dont think my model learn something..
", "votes": 0, "replies": [] }, { "id": 1514195, "author_name": "Shummi Ahmed", "author_url": "", "post_date": "2021-09-15T19:03:44.330000", "content": "what is the general training accuracy and roc for everyone? i guess public test set is quite disjoint from training set
", "votes": 0, "replies": [] }, { "id": 1500327, "author_name": "Woprime", "author_url": "", "post_date": "2021-09-02T09:37:43.797000", "content": "Did you come to a conclusion on the question? I too wonder if this is possible at all, I tried to train a few models on the segmented tumors, but the model learns absolutely nothing.
", "votes": 0, "replies": [ { "id": 1500332, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-09-02T09:43:10.640000", "content": "There are some papers stating that they are actually successful in detecting MGMT status,
\nlike this one http://www.ajnr.org/content/ajnr/early/2021/03/04/ajnr.A7029.full.pdf
\nSo I guess it's actually possible, but I don't know why our models are rarely learning.
@mrinath My best understanding is that - the result from the paper uses tumor masks, whereas our dataset doesn't have that.
", "votes": 1, "replies": [] }, { "id": 1502978, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-09-04T19:46:24.037000", "content": "Yes, you are correct, thats why I think many people are trying to work with task 1.
\nI can be wrong
I tried to use one of the top segmentator from 2019 BRATS, it is really good and I can clearly see that, for most of the time segment the correct place for tumor. Although the network learns NOTHING from the segmented tumor. I even include some additional parts (a large bounding box to be precise) around the segmented tumor in case it misses something, but still nothing.
\nI'm starting to think that either my model is crap, or the appearance feature in our data is just not helping (maybe too low resolution?); maybe the model only learn from the location, and I willingly give up the spatial information of the tumor when I rely solely on the segmented images. This can explain why the 3D CNN performs somewhat well: My hypothesis is that it tries to predict the MGMT base on where the tumor is located in the brain. That being said, if the hypothesis holds true, then I am very pessimistic about this competition, so I wholeheartedly hope that I made a fool out of myself by stating this hypothesis :D
", "votes": 4, "replies": [] } ] }, { "id": 1479127, "author_name": "novice03", "author_url": "", "post_date": "2021-08-18T09:54:25.237000", "content": "Same here. It would be nice to get some clues from the people on top of the lb.
", "votes": 0, "replies": [ { "id": 1479149, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-18T10:07:48.370000", "content": "Hope the top scores are not from hand labelling
", "votes": 1, "replies": [] }, { "id": 1479374, "author_name": "novice03", "author_url": "", "post_date": "2021-08-18T12:51:14.883000", "content": "Even if they are, they would score 0 in the private test set, right?
", "votes": 2, "replies": [] } ] }, { "id": 1511601, "author_name": "", "author_url": "", "post_date": "2021-09-13T14:38:01.780000", "content": "", "votes": 0, "replies": [ { "id": 1511865, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-09-13T18:28:20.390000", "content": "Thats Handlabelling, you can see the [HL] tag
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1478867": "Till now, we have applied a bunch of strategies, still getting the validation loss(BCE) in the range of 0.69-0.68. our oof files scores are always staying almost close to 0.5.\nI don't know if our model is actually learning or not.\n\nSo to you guys, please tell me I am wrong here. As I am feeling very miserable as the models are failing to generalise.\nI would be really happy if someone tells me that I am doing it wrong, and their models are actually training pretty good.\n\nPS I am not telling anyone to share their pipeline, just want to know if you are facing the same problem, or actually, you are getting good results.\n\nPPS: Not used any external data, just competition's data\nThanks", "1479828": "I have dealt with the same problem. My 0.764 is much higher than the cross validation suggests it should be so I'm attributing it largely to statistical aberration of low sample size. I'm also running some post processing that weights the ranking between FLAIR and T2 channels that gives a slight boost.\nI get nearly identical results whether I'm using a simple, shallow 2 layer CNN or EFF-B7. My intuition is leading me to this being more a data centric AI problem rather than a modeling one and the majority of my time spent has been with prepping the data. I've found only one thing that seems to have some effect.\nI'm using Effnet-B0 3D CNN and have put significant work into orienting, interpolating to scale (1mm x 1mm x 1mm), and normalizing the images. The one thing that shows signs of life is normalizing in 3D voxel space using 3D CLAHE. Most public notebooks are normalizing at the 2D plane level and this leads to significant discontinuities in 3D space.", "1503997": "As the Venn diagram in the link below shows, there is some overlap in the cases between Task 1 and Task 2 of BraTS 2021.\n\n\n\nSpecifically, out of the Task 2 training data hosted on Kaggle, 577 cases overlap with the Task 1 training data. Therefore, I conducted a segmentation-based MGMT classification experiment as described in [Yogananda et al. 2021](http://www.ajnr.org/content/early/2021/03/04/ajnr.A7029), giving only Task 2 labels to the 577 cases of Task 1 data that overlap with Task 2. The reason for the above experimental setup is that the MRI images of Task 1 are better than those of Task 2 because they are carefully co-registered.\nThe model used was a light-weight pre-trained 3D CNN model, and all 5 folds were trained with using 4 modalities.\n\nThe experimental results were disappointing š\nUnlike the excellent results in Yogananda's paper, my local AUC was about 0.56, which is slightly better than chance level, almost nothing was learned. At this point, I am not sure if our MRI data does not have enough information to classify MGMT status, or if features for MGMT classification are contained in something else that cannot be learned by the CNN model. However, since Yogananda et al. succeeded in classifying MGMT with 3D CNN, I think it is possible that the label given in the competition, which is supposed to be Ground Truth, might be wrong in some cases. \n\nIn addition, [Baid et al., 2021](https://arxiv.org/abs/2107.02314) has the following definition of MGMT.\n\n> **The percent methylation above 10% was interpreted as positive. A sample below 10% methylation was interpreted as negative.** A sample below 10% methylation was interpreted as negative. For the latter approach, a total of 17 MGMT promoter CpG sites were amplified by nested polymerase chain reaction (PCR) using a bisulte treated DNA template. Quantitative PCR was performed for each CpG site to determine its methylation status. **A result of 2% or more methylated CpG sites in the MGMT promoter (out of 17 total sites) was considered a positive result.**\n\nTaking this into account, the MGMT value should be expressed as a more ambiguous soft label, rather than as something that can be firmly distinguished between 0 and 1.", "1479406": "I'm starting to believe that there isn't sufficient data to create anything clinically useful here. There's too much variation in the images for such a small sample size.\n\nI think the top scoring models will be learning on something other than MGMT_value and won't actually perform in the real world. \n\nIt makes me wonder if a competition like this might do more harm than good by legitimizing mediocre solutions based on a fraction of a score from a lucky run.", "1480093": "I think monitoring cross-entropy loss might be best there. I haven't been able to get under 0.66 validation (single fold). I definitely think this is going to be more of a data centric problem as David Austin describes. There are some ideas I want to try but really have to limit the resolution dimensions to get any experiments done. I've spent more time pre-processing and finding ways to manipulate 3D data than actual modeling.", "1483081": "We havenāt yet, but one approach we will be trying is taking data from task 1 (along with labels) converting to a similar representation to Task 2 data and then training a segmentation model to act as an intermediary model/step in classification.\n\nWe know volume/shape has some part in determining the MGMT status, so I think this intuitively makes sense. \n\nIām hoping by the end of the competition we will hopefully see models scoring above 75% consistently w/out hand labels. \n\nWhether this will, in fact, be clinically relevant, remains to be seen. ", "1503128": "I'm struggling to generate good models, and I look forward to the winning solutions. All of my models basically guess average(MGMT_value), and it is hard to trust CV.", "1494529": "Using a 3d CNN on the FLAIR images I'm getting a validation score of BCE 0.6811 on the validation set, 80/20 train test split. In this model I randomly rotated the images 180 degrees, which seems to yield a 1% improvement. However, I'm not doing a k-fold CV so this improvement could potentially be noise. I'm using a little dropout of p=.9 in the top two layers to regularize a bit. I haven't sensitivity tested w/ and w/o dropout to see what happens. I'm using 200x200x70 resolution using 3 convolution layers with channels (in-out): 1-8, 8-16, 16-8. Best score is at 6th epoch, .001 LR.", "1487443": "Itās the same for me. My (single fold) validation loss is always around 0.65 - 0.69 and the validation roc score is hardly increasing. Unfortunately, I can only tell you what didnāt work for me:\n\n- It seems that the architecture is not that important (I tried various architectures such as 3D EfficientNet or 3D ResNet with ā4Dā images (4D because of the shape: 4xWIDTHxHEIGHTxDEPTH) as well as combining CNNs trained on each MRI type, respectively)\n- I put a lot of time in different slicing techniques, but again, nothing stood out in particular. For example, I took SLICING_NUMBER (e.g., 32) slices from the middle of an image or I only took each x-th (e.g., 2nd, 3rt, etc.) slice from an image. I even tried to average images\n- I tried different augmentation techniques for example 3D-MRI-specific augmentations such as RandomGhosting or RandomAnisotropy as well as basic 2D augmentations such as CLAHE etc. Again, none of the methods was superior to the others. \n\nAfter reading this and similar discussion, Iāll definitely switch to k-fold CV in order to assess my changes instead of a single train/validation split.\n\nIām wondering if I should spend some time in playing with the orientations between the scans (sagittal vs. coronal vs. axial). Do you have any experiences? Is it worth it? ", "1516068": "No matter what I try, I still have val auc of 0.5-0.6, the model simply doesn't learn anything.", "1502187": "I've tried 3d CNN and 2d CNN + RNN. As same as you~ My val & test results only over 0.5~0.6. :(", "1479010": "For one strategy, I picked up one slice including glioblastoma per MRI type & ID and used them for 4ch CNN training, but the val-auc was only 0.6. \nEven if I used more slices, the result was not so different.\nIf this competition is meaningful, we should detect some slices which contribute to the MGMT 0/1 classification.\nJust using slices including tumor is not enough, I guess.", "1478870": "Is external data the key here?š¤š¤", "1505074": "No success here either. I thought I had figured something out but then I computed the standard deviation and saw that it is too high (approx. 0.16). My mean auc CV is 0.61 but because the std is so high, the score does not mean anything. This is why I like regular CV more than OOF. By having two estimators (standard deviation and mean), there is more information.", "1551959": "Thank you for this discussion, was feeling lost as well", "1518109": "Me too. I dont think my model learn something..", "1514195": "what is the general training accuracy and roc for everyone? i guess public test set is quite disjoint from training set", "1500327": "Did you come to a conclusion on the question? I too wonder if this is possible at all, I tried to train a few models on the segmented tumors, but the model learns absolutely nothing. ", "1479127": "Same here. It would be nice to get some clues from the people on top of the lb.", "1511601": "" } }