![\"\"](\"https://raw.githubusercontent.com/SauravMaheshkar/SauravMaheshkar/main/assets/RSNA-MICCAI/Fluke-Training-Loss.svg\")
For an in-depth comparison of the various models such as b0 vs b1 vs b2 or b0 with different seeds, head over to the accompanying wandb report.
\nI've attached some proof here of some of my experiments using the same code as the \"Efficientnet3D with one MRI type\" kernel (with some minor changes for wandb logging) which proves that much of what we're doing is nothing but a random fluke.
The motivation for these experiments come from Chai Time Kaggle Talks with Anjum Sayed (Datasaurus) Video on the Weights and Biases Channel. Anjum mentioned that a good way to check if the models are learning anything is to just change the random seeds and see if it affects the performance.
\n
![\"\"](\"https://raw.githubusercontent.com/SauravMaheshkar/SauravMaheshkar/main/assets/RSNA-MICCAI/Fluke-Validation-Loss.svg\")
Naming Convention - arch-seed
| Name | \nTraining Loss | \nValidation Loss | \nEPOCHS | \nBATCH_SIZE | \n
|---|---|---|---|---|
| baseline-efficientnet3d-b0-42 | \n0.6814 | \n0.5914 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-12 | \n0.6913 | \n0.6420 | \n10 | \n4 | \n
| augment-efficientnet3d-b2-42 | \n0.7125 | \n5.6498 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-2021 | \n0.7293 | \n0.6019 | \n10 | \n4 | \n
| baseline-efficientnet3d-b2-12 | \n0.7431 | \n0.8652 | \n10 | \n4 | \n
| baseline-efficientnet3d-b1-12 | \n0.7461 | \n0.5110 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-21 | \n0.7643 | \n0.7170 | \n10 | \n4 | \n
For an in-depth comparison of the various models such as b0 vs b1 vs b2 or b0 with different seeds, head over to the accompanying wandb report.
\nI've attached some proof here of some of my experiments using the same code as the \"Efficientnet3D with one MRI type\" kernel (with some minor changes for wandb logging) which proves that much of what we're doing is nothing but a random fluke.
The motivation for these experiments come from Chai Time Kaggle Talks with Anjum Sayed (Datasaurus) Video on the Weights and Biases Channel. Anjum mentioned that a good way to check if the models are learning anything is to just change the random seeds and see if it affects the performance.
\n
Naming Convention - arch-seed
| Name | \nTraining Loss | \nValidation Loss | \nEPOCHS | \nBATCH_SIZE | \n
|---|---|---|---|---|
| baseline-efficientnet3d-b0-42 | \n0.6814 | \n0.5914 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-12 | \n0.6913 | \n0.6420 | \n10 | \n4 | \n
| augment-efficientnet3d-b2-42 | \n0.7125 | \n5.6498 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-2021 | \n0.7293 | \n0.6019 | \n10 | \n4 | \n
| baseline-efficientnet3d-b2-12 | \n0.7431 | \n0.8652 | \n10 | \n4 | \n
| baseline-efficientnet3d-b1-12 | \n0.7461 | \n0.5110 | \n10 | \n4 | \n
| baseline-efficientnet3d-b0-21 | \n0.7643 | \n0.7170 | \n10 | \n4 | \n
Very interesting point. Actually, the mentioned kernel does not learn at all - the output scores vary in [0.4,0.6] range, which on average give the graded 0.642. A well-trained model shall give values in a wider range, close to 0 and 1.
", "rawMarkdown": "Very interesting point. Actually, the mentioned kernel does not learn at all - the output scores vary in [0.4,0.6] range, which on average give the graded 0.642. A well-trained model shall give values in a wider range, close to 0 and 1.", "votes": null }, { "id": "1507797", "postDate": "09/09/2021 14:43:57", "content": "The mentioned kernel has several flaws - wrong sorting of the input files, missing sigmoid layer, and what is more important - problematic model. The used batch normalization layers are imported from the functional API, and they don't work well in the eval mode. Therefore, the obtained predictions are worthless and you can rely on neither loss nor auc. I recommend switching onto implementations provided by Monai - they use batch normalization layers from torch.nn package. The code is available here: monai
", "rawMarkdown": "The mentioned kernel has several flaws - wrong sorting of the input files, missing sigmoid layer, and what is more important - problematic model. The used batch normalization layers are imported from the functional API, and they don't work well in the eval mode. Therefore, the obtained predictions are worthless and you can rely on neither loss nor auc. I recommend switching onto implementations provided by Monai - they use batch normalization layers from torch.nn package. The code is available here: [monai](https://www.kaggle.com/mikecho/monai-v060-deep-learning-in-healthcare-imaging)", "votes": null }, { "id": "1507830", "postDate": "09/09/2021 15:20:21", "content": "Exactly, the missing activation layer was bugging me a lot too !!
\nPS: Already working on another pipeline using monai 😁
", "rawMarkdown": "Exactly, the missing activation layer was bugging me a lot too !!\n\nPS: Already working on another pipeline using monai 😁", "votes": null }, { "id": "1508119", "postDate": "09/09/2021 21:39:53", "content": "Where does it use functional batch norms? I'm not seeing it.
", "rawMarkdown": "Where does it use functional batch norms? I'm not seeing it.", "votes": null }, { "id": "1508257", "postDate": "09/10/2021 03:45:29", "content": "I think @mikecho, is referring to the codebase of the EfficientNet-PyTorch-3D library, which uses nn.BatchNorm3d in the module definitions. For example,
class MBConvBlock3D(nn.Module):\n ...\n self._bn0 = nn.BatchNorm3d(...)\nHave made the \"Reproducibility Kernel\" Public now.
", "rawMarkdown": "Have made the \"Reproducibility Kernel\" Public now.", "votes": null }, { "id": "1508457", "postDate": "09/10/2021 08:46:19", "content": "Sorry, actually @returnofsputnik is right. Only adaptive_avg_pool3d is from the functional API. I was analyzing that some time ago and I mixed up those two layers.
\nAnyway, I used the implementation of Monai and the problem was exactly the same. It works only when I change track_running_stats to False on each batchnorm layer, so maybe this is also a way to run the above implementation.
\nDoes anyone know why it is necessary to set this parameter? Without this, the network returns the same probabilities per batch in the eval mode. When the network is reloaded (weights stored and loaded from file), the problem disappears though.
@jhasanov it is not that these models don't learn at all, but rather that the loss and auc calculated in the validation epochs are worthless. As I mentioned above, there is a problem with the batchnorm layers in the used architecture that doesn't work well in the eval mode without setting track_running_stats to False on each of the batchnorm modules. When you do it, the network starts to produce a meaningful output in the eval mode during training. However, you need to remember to include sigmoid layer, and remove rounding the probabilities to 0/1 for auc calculation
", "rawMarkdown": "jhasanov it is not that these models don't learn at all, but rather that the loss and auc calculated in the validation epochs are worthless. As I mentioned above, there is a problem with the batchnorm layers in the used architecture that doesn't work well in the eval mode without setting track_running_stats to False on each of the batchnorm modules. When you do it, the network starts to produce a meaningful output in the eval mode during training. However, you need to remember to include sigmoid layer, and remove rounding the probabilities to 0/1 for auc calculation", "votes": null }, { "id": "1508545", "postDate": "09/10/2021 10:30:46", "content": "Nice sharing..
", "rawMarkdown": "Nice sharing..", "votes": null }, { "id": "1508584", "postDate": "09/10/2021 11:12:41", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing.", "votes": null }, { "id": "1509300", "postDate": "09/11/2021 05:48:59", "content": "Appreciate the feedback 😁
", "rawMarkdown": "Appreciate the feedback 😁", "votes": null }, { "id": "1509301", "postDate": "09/11/2021 05:49:04", "content": "Appreciate the feedback 😁
", "rawMarkdown": "Appreciate the feedback 😁", "votes": null }, { "id": "1509927", "postDate": "09/11/2021 19:37:15", "content": "HI @sauravmaheshkar, I have just published a notebook based on Monai and Densenet121 that fixes the errors of the kernel that you are mentioning. In the current version, there is a bit too much overfiting as the amount of data is quite small, and no augmentation is implemented at this point. I will be uploading new versions soon.
", "rawMarkdown": "HI @sauravmaheshkar, I have just published a [notebook](https://www.kaggle.com/mikecho/rsna-miccai-monai-ensemble?scriptVersionId=74508923) based on Monai and Densenet121 that fixes the errors of the kernel that you are mentioning. In the current version, there is a bit too much overfiting as the amount of data is quite small, and no augmentation is implemented at this point. I will be uploading new versions soon.", "votes": null }, { "id": "1510094", "postDate": "09/12/2021 05:20:03", "content": "Hey @mikecho, seems like you beat me to it 😅. Even I was working on a pipeline using monai. I think it's enough that a simple DenseNet121 achieves a better score than EfficientNet3D-b2. Hopefully, we'll see an improvement in learning from now on. I'll be uploading newer kernels as well.
", "rawMarkdown": "Hey @mikecho, seems like you beat me to it 😅. Even I was working on a pipeline using monai. I think it's enough that a simple DenseNet121 achieves a better score than EfficientNet3D-b2. Hopefully, we'll see an improvement in learning from now on. I'll be uploading newer kernels as well.", "votes": null }, { "id": "1511085", "postDate": "09/13/2021 05:22:32", "content": "Thanks for pointing out in detail.
", "rawMarkdown": "Thanks for pointing out in detail.", "votes": null }, { "id": "1513001", "postDate": "09/14/2021 18:13:37", "content": "Thanks @sauravmaheshkar for pointing this out and for these ensuing insightful discussions!
", "rawMarkdown": "Thanks @sauravmaheshkar for pointing this out and for these ensuing insightful discussions!", "votes": null }, { "id": "1513273", "postDate": "09/15/2021 03:09:56", "content": "Thanks for the feedback 😁
", "rawMarkdown": "Thanks for the feedback 😁", "votes": null }, { "id": "1513365", "postDate": "09/15/2021 05:42:14", "content": "Hello, which version is you refered?
", "rawMarkdown": "Hello, which version is you refered?", "votes": null }, { "id": "1513950", "postDate": "09/15/2021 14:45:24", "content": "Version Number of what ? The kernel in question, my reproducibility kernel ,… ?
", "rawMarkdown": "Version Number of what ? The kernel in question, my reproducibility kernel ,... ?", "votes": null }, { "id": "1515490", "postDate": "09/17/2021 06:53:30", "content": "thanks very good👍👍👍
", "rawMarkdown": "thanks very good👍👍👍", "votes": null }, { "id": "1515854", "postDate": "09/17/2021 15:02:45", "content": "Thanks for the feedback 😁
", "rawMarkdown": "Thanks for the feedback 😁", "votes": null } ], "comments": [ { "id": 1507450, "author_name": "jhasanov", "author_url": "", "post_date": "09/09/2021 08:03:36", "content": "Very interesting point. Actually, the mentioned kernel does not learn at all - the output scores vary in [0.4,0.6] range, which on average give the graded 0.642. A well-trained model shall give values in a wider range, close to 0 and 1.
", "votes": null, "replies": [ { "id": 1508486, "author_name": "mikecho", "author_url": "", "post_date": "09/10/2021 09:20:50", "content": "@jhasanov it is not that these models don't learn at all, but rather that the loss and auc calculated in the validation epochs are worthless. As I mentioned above, there is a problem with the batchnorm layers in the used architecture that doesn't work well in the eval mode without setting track_running_stats to False on each of the batchnorm modules. When you do it, the network starts to produce a meaningful output in the eval mode during training. However, you need to remember to include sigmoid layer, and remove rounding the probabilities to 0/1 for auc calculation
", "votes": null, "replies": [] }, { "id": 1511085, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/13/2021 05:22:32", "content": "Thanks for pointing out in detail.
", "votes": null, "replies": [] } ] }, { "id": 1507797, "author_name": "mikecho", "author_url": "", "post_date": "09/09/2021 14:43:57", "content": "The mentioned kernel has several flaws - wrong sorting of the input files, missing sigmoid layer, and what is more important - problematic model. The used batch normalization layers are imported from the functional API, and they don't work well in the eval mode. Therefore, the obtained predictions are worthless and you can rely on neither loss nor auc. I recommend switching onto implementations provided by Monai - they use batch normalization layers from torch.nn package. The code is available here: monai
", "votes": null, "replies": [ { "id": 1507830, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/09/2021 15:20:21", "content": "Exactly, the missing activation layer was bugging me a lot too !!
\nPS: Already working on another pipeline using monai 😁
", "votes": null, "replies": [] }, { "id": 1508119, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/09/2021 21:39:53", "content": "Where does it use functional batch norms? I'm not seeing it.
", "votes": null, "replies": [] }, { "id": 1508257, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/10/2021 03:45:29", "content": "I think @mikecho, is referring to the codebase of the EfficientNet-PyTorch-3D library, which uses nn.BatchNorm3d in the module definitions. For example,
class MBConvBlock3D(nn.Module):\n ...\n self._bn0 = nn.BatchNorm3d(...)\nSorry, actually @returnofsputnik is right. Only adaptive_avg_pool3d is from the functional API. I was analyzing that some time ago and I mixed up those two layers.
\nAnyway, I used the implementation of Monai and the problem was exactly the same. It works only when I change track_running_stats to False on each batchnorm layer, so maybe this is also a way to run the above implementation.
\nDoes anyone know why it is necessary to set this parameter? Without this, the network returns the same probabilities per batch in the eval mode. When the network is reloaded (weights stored and loaded from file), the problem disappears though.
Have made the \"Reproducibility Kernel\" Public now.
", "votes": null, "replies": [] }, { "id": 1508545, "author_name": "hawkeat", "author_url": "", "post_date": "09/10/2021 10:30:46", "content": "Nice sharing..
", "votes": null, "replies": [ { "id": 1509301, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/11/2021 05:49:04", "content": "Appreciate the feedback 😁
", "votes": null, "replies": [] } ] }, { "id": 1508584, "author_name": "dhruvkhatri", "author_url": "", "post_date": "09/10/2021 11:12:41", "content": "Thank you for sharing.
", "votes": null, "replies": [ { "id": 1509300, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/11/2021 05:48:59", "content": "Appreciate the feedback 😁
", "votes": null, "replies": [] } ] }, { "id": 1509927, "author_name": "mikecho", "author_url": "", "post_date": "09/11/2021 19:37:15", "content": "HI @sauravmaheshkar, I have just published a notebook based on Monai and Densenet121 that fixes the errors of the kernel that you are mentioning. In the current version, there is a bit too much overfiting as the amount of data is quite small, and no augmentation is implemented at this point. I will be uploading new versions soon.
", "votes": null, "replies": [ { "id": 1510094, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/12/2021 05:20:03", "content": "Hey @mikecho, seems like you beat me to it 😅. Even I was working on a pipeline using monai. I think it's enough that a simple DenseNet121 achieves a better score than EfficientNet3D-b2. Hopefully, we'll see an improvement in learning from now on. I'll be uploading newer kernels as well.
", "votes": null, "replies": [] } ] }, { "id": 1513001, "author_name": "conandoyle", "author_url": "", "post_date": "09/14/2021 18:13:37", "content": "Thanks @sauravmaheshkar for pointing this out and for these ensuing insightful discussions!
", "votes": null, "replies": [ { "id": 1513273, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/15/2021 03:09:56", "content": "Thanks for the feedback 😁
", "votes": null, "replies": [] } ] }, { "id": 1513365, "author_name": "zekun98", "author_url": "", "post_date": "09/15/2021 05:42:14", "content": "Hello, which version is you refered?
", "votes": null, "replies": [ { "id": 1513950, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/15/2021 14:45:24", "content": "Version Number of what ? The kernel in question, my reproducibility kernel ,… ?
", "votes": null, "replies": [] } ] }, { "id": 1515490, "author_name": "saliblue", "author_url": "", "post_date": "09/17/2021 06:53:30", "content": "thanks very good👍👍👍
", "votes": null, "replies": [ { "id": 1515854, "author_name": "sauravmaheshkar", "author_url": "", "post_date": "09/17/2021 15:02:45", "content": "Thanks for the feedback 😁
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1507421": "### [**Link to the Kernel to Reproduce Experiments**](https://www.kaggle.com/sauravmaheshkar/rsna-miccai-the-random-seed-fluke/)\n\nFor an in-depth comparison of the various models such as **b0 vs b1 vs b2** or **b0 with different seeds**, head over to the [**accompanying wandb report**](https://wandb.ai/sauravmaheshkar/RSNA-MICCAI/reports/The-Fluke--VmlldzoxMDA2MDQy). \n\nI've attached some proof here of some of my experiments using the same code as the \"[**Efficientnet3D with one MRI type**](https://www.kaggle.com/rluethy/efficientnet3d-with-one-mri-type)\" kernel (with some minor changes for **`wandb`** logging) which proves that much of what we're doing is nothing but a random fluke. \n\nThe motivation for these experiments come from [**Chai Time Kaggle Talks with Anjum Sayed (Datasaurus)**](https://youtu.be/udw-uSV66EQ) Video on the [**Weights and Biases Channel**](https://www.youtube.com/WeightsBiases). Anjum mentioned that a good way to check if the models are learning anything is to just change the random seeds and see if it affects the performance.\n\n## [**Weights and Biases Report ⭐️**](https://wandb.ai/sauravmaheshkar/RSNA-MICCAI/reports/The-Fluke--VmlldzoxMDA2MDQy) | [**Weights and Biases Project**](https://wandb.ai/sauravmaheshkar/RSNA-MICCAI)\n\n---\n\n# Models Don't Learn 🤷🏻\n\n\n\n\n\nNaming Convention - `arch-seed`\n\n|**Name** |**Training Loss**|**Validation Loss**|**EPOCHS**|**BATCH_SIZE**|\n|-------------------------------|-------------|---------------|:------:|:----------:|\n|baseline-efficientnet3d-b0-42 |0.6814|0.5914|10 |4 |64 |\n|baseline-efficientnet3d-b0-12 |0.6913|0.6420|10 |4 |64 |\n|augment-efficientnet3d-b2-42 |0.7125|5.6498|10 |4 |64 |\n|baseline-efficientnet3d-b0-2021|0.7293|0.6019|10 |4 |64 |\n|baseline-efficientnet3d-b2-12 |0.7431|0.8652|10 |4 |64 |\n|baseline-efficientnet3d-b1-12 |0.7461|0.5110|10 |4 |64 |\n|baseline-efficientnet3d-b0-21 |0.7643|0.7170|10 |4 |64 |", "1507450": "Very interesting point. Actually, the mentioned kernel does not learn at all - the output scores vary in [0.4,0.6] range, which on average give the graded 0.642. A well-trained model shall give values in a wider range, close to 0 and 1.", "1507797": "The mentioned kernel has several flaws - wrong sorting of the input files, missing sigmoid layer, and what is more important - problematic model. The used batch normalization layers are imported from the functional API, and they don't work well in the eval mode. Therefore, the obtained predictions are worthless and you can rely on neither loss nor auc. I recommend switching onto implementations provided by Monai - they use batch normalization layers from torch.nn package. The code is available here: [monai](https://www.kaggle.com/mikecho/monai-v060-deep-learning-in-healthcare-imaging)", "1507830": "Exactly, the missing activation layer was bugging me a lot too !!\n\nPS: Already working on another pipeline using monai 😁", "1508119": "Where does it use functional batch norms? I'm not seeing it.", "1508257": "I think @mikecho, is referring to the codebase of the `EfficientNet-PyTorch-3D` library, which uses `nn.BatchNorm3d` in the module definitions. For example, \n\n```\nclass MBConvBlock3D(nn.Module):\n ...\n self._bn0 = nn.BatchNorm3d(...)\n```", "1508258": "Have made the \"Reproducibility Kernel\" Public now.", "1508457": "Sorry, actually @returnofsputnik is right. Only adaptive_avg_pool3d is from the functional API. I was analyzing that some time ago and I mixed up those two layers.\nAnyway, I used the implementation of Monai and the problem was exactly the same. It works only when I change track_running_stats to False on each batchnorm layer, so maybe this is also a way to run the above implementation.\nDoes anyone know why it is necessary to set this parameter? Without this, the network returns the same probabilities per batch in the eval mode. When the network is reloaded (weights stored and loaded from file), the problem disappears though.", "1508486": "jhasanov it is not that these models don't learn at all, but rather that the loss and auc calculated in the validation epochs are worthless. As I mentioned above, there is a problem with the batchnorm layers in the used architecture that doesn't work well in the eval mode without setting track_running_stats to False on each of the batchnorm modules. When you do it, the network starts to produce a meaningful output in the eval mode during training. However, you need to remember to include sigmoid layer, and remove rounding the probabilities to 0/1 for auc calculation", "1508545": "Nice sharing..", "1508584": "Thank you for sharing.", "1509300": "Appreciate the feedback 😁", "1509301": "Appreciate the feedback 😁", "1509927": "HI @sauravmaheshkar, I have just published a [notebook](https://www.kaggle.com/mikecho/rsna-miccai-monai-ensemble?scriptVersionId=74508923) based on Monai and Densenet121 that fixes the errors of the kernel that you are mentioning. In the current version, there is a bit too much overfiting as the amount of data is quite small, and no augmentation is implemented at this point. I will be uploading new versions soon.", "1510094": "Hey @mikecho, seems like you beat me to it 😅. Even I was working on a pipeline using monai. I think it's enough that a simple DenseNet121 achieves a better score than EfficientNet3D-b2. Hopefully, we'll see an improvement in learning from now on. I'll be uploading newer kernels as well.", "1511085": "Thanks for pointing out in detail.", "1513001": "Thanks @sauravmaheshkar for pointing this out and for these ensuing insightful discussions!", "1513273": "Thanks for the feedback 😁", "1513365": "Hello, which version is you refered?", "1513950": "Version Number of what ? The kernel in question, my reproducibility kernel ,... ?", "1515490": "thanks very good👍👍👍", "1515854": "Thanks for the feedback 😁" }, "source": "meta" }