{ "id": 257189, "title": "Cross Validation/Validation Strategies (Tips & Tricks)", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/257189", "author_name": "Zabir Al Nazi Nabil", "post_date": "2021-08-02T22:54:10.187000", "votes": 10, "comment_count": 0, "views": 0, "content": "

It seems the public test dataset is really small to have a big impact on the final leaderboard score. So, CV is the main thing to focus on. Let's compile a list of CV strategies that are useful for this kind of task.

\n

Cross-validation definition

\n

https://neptune.ai/blog/cross-validation-in-machine-learning-how-to-do-it-right (types of CV)

\n

https://stackoverflow.com/questions/25889637/how-to-use-k-fold-cross-validation-in-a-neural-network (k-fold in NN)

\n

https://www.kaggle.com/satishgunjal/tutorial-k-fold-cross-validation (good code demonstration)

\n

Tutorials

\n

https://machinelearningmastery.com/difference-test-validation-datasets/ (val vs test [useful for beginners])

\n

https://towardsdatascience.com/cross-validation-c4fae714f1c5 (beginner)

\n

https://www.simplilearn.com/tutorials/machine-learning-tutorial/cross-validation (intro materials)

\n

https://wandb.ai/authors/Kaggle-ML-CV/reports/Kaggle-s-Intermediate-ML-Cross-Validation--VmlldzoxOTY3NjQ (intermediate)

\n

https://machinelearningmastery.com/data-preparation-without-data-leakage/ (data leakage)

\n

Video tutorials / presentation

\n

https://www.youtube.com/watch?v=Q0QmziFcfU0 (data leakage, in-depth)

\n

Tips & Tricks

\n

https://www.datapred.com/blog/advanced-cross-validation-tips (simple discussion)

\n

https://towardsdatascience.com/increase-the-accuracy-of-your-cnn-by-following-these-5-tips-i-learned-from-the-kaggle-community-27227ad39554 (general to any kaggle computer vision competition)

\n

https://www.kaggle.com/general/18793 (compilation of strategies)

\n

https://mlwave.com/kaggle-ensembling-guide/ (long discussion + explanation)

\n

Kaggle Discussions

\n

https://www.kaggle.com/questions-and-answers/47057

\n

To summarize:

\n
    \n
  1. Use 5-10 fold cross-validation (go high if you have the computational resources, but considering the amount of data available to you).
    2. \n
  2. Look for leakage. Don't use the same subjects in a single fold. (Especially, if you are using a 2-d model and training on individual slices, don't mix slices from the same subject/patient/study in multiple folds)
    3. \n
  3. Use multi-stage cross-validation if possible.
    4. \n
", "messages": [ { "id": 1416305, "postDate": "2021-08-02T22:54:10.187Z", "content": "

It seems the public test dataset is really small to have a big impact on the final leaderboard score. So, CV is the main thing to focus on. Let's compile a list of CV strategies that are useful for this kind of task.

\n

Cross-validation definition

\n

https://neptune.ai/blog/cross-validation-in-machine-learning-how-to-do-it-right (types of CV)

\n

https://stackoverflow.com/questions/25889637/how-to-use-k-fold-cross-validation-in-a-neural-network (k-fold in NN)

\n

https://www.kaggle.com/satishgunjal/tutorial-k-fold-cross-validation (good code demonstration)

\n

Tutorials

\n

https://machinelearningmastery.com/difference-test-validation-datasets/ (val vs test [useful for beginners])

\n

https://towardsdatascience.com/cross-validation-c4fae714f1c5 (beginner)

\n

https://www.simplilearn.com/tutorials/machine-learning-tutorial/cross-validation (intro materials)

\n

https://wandb.ai/authors/Kaggle-ML-CV/reports/Kaggle-s-Intermediate-ML-Cross-Validation--VmlldzoxOTY3NjQ (intermediate)

\n

https://machinelearningmastery.com/data-preparation-without-data-leakage/ (data leakage)

\n

Video tutorials / presentation

\n

https://www.youtube.com/watch?v=Q0QmziFcfU0 (data leakage, in-depth)

\n

Tips & Tricks

\n

https://www.datapred.com/blog/advanced-cross-validation-tips (simple discussion)

\n

https://towardsdatascience.com/increase-the-accuracy-of-your-cnn-by-following-these-5-tips-i-learned-from-the-kaggle-community-27227ad39554 (general to any kaggle computer vision competition)

\n

https://www.kaggle.com/general/18793 (compilation of strategies)

\n

https://mlwave.com/kaggle-ensembling-guide/ (long discussion + explanation)

\n

Kaggle Discussions

\n

https://www.kaggle.com/questions-and-answers/47057

\n

To summarize:

\n
    \n
  1. Use 5-10 fold cross-validation (go high if you have the computational resources, but considering the amount of data available to you).
    2. \n
  2. Look for leakage. Don't use the same subjects in a single fold. (Especially, if you are using a 2-d model and training on individual slices, don't mix slices from the same subject/patient/study in multiple folds)
    3. \n
  3. Use multi-stage cross-validation if possible.
    4. \n
", "rawMarkdown": "It seems the public test dataset is really small to have a big impact on the final leaderboard score. So, CV is the main thing to focus on. Let's compile a list of CV strategies that are useful for this kind of task.\n\n### **Cross-validation definition**\nhttps://neptune.ai/blog/cross-validation-in-machine-learning-how-to-do-it-right (types of CV)\n\nhttps://stackoverflow.com/questions/25889637/how-to-use-k-fold-cross-validation-in-a-neural-network (k-fold in NN)\n\nhttps://www.kaggle.com/satishgunjal/tutorial-k-fold-cross-validation (good code demonstration)\n\n### **Tutorials**\nhttps://machinelearningmastery.com/difference-test-validation-datasets/ (val vs test [useful for beginners])\n\nhttps://towardsdatascience.com/cross-validation-c4fae714f1c5 (beginner)\n\nhttps://www.simplilearn.com/tutorials/machine-learning-tutorial/cross-validation (intro materials)\n\nhttps://wandb.ai/authors/Kaggle-ML-CV/reports/Kaggle-s-Intermediate-ML-Cross-Validation--VmlldzoxOTY3NjQ (intermediate)\n\nhttps://machinelearningmastery.com/data-preparation-without-data-leakage/ (data leakage)\n\n### **Video tutorials / presentation**\nhttps://www.youtube.com/watch?v=Q0QmziFcfU0 (data leakage, in-depth)\n\n### **Tips & Tricks**\nhttps://www.datapred.com/blog/advanced-cross-validation-tips (simple discussion)\n\nhttps://towardsdatascience.com/increase-the-accuracy-of-your-cnn-by-following-these-5-tips-i-learned-from-the-kaggle-community-27227ad39554 (general to any kaggle computer vision competition)\n\nhttps://www.kaggle.com/general/18793 (compilation of strategies)\n\nhttps://mlwave.com/kaggle-ensembling-guide/ (long discussion + explanation)\n\n###**Kaggle Discussions**\n\nhttps://www.kaggle.com/questions-and-answers/47057\n\n\n##**To summarize**:\n1. Use 5-10 fold cross-validation (go high if you have the computational resources, but considering the amount of data available to you).\n2. Look for leakage. Don't use the same subjects in a single fold. (Especially, if you are using a 2-d model and training on individual slices, don't mix slices from the same subject/patient/study in multiple folds)\n3. Use multi-stage cross-validation if possible.\n", "votes": 10 } ], "comments": [], "raw_markdown_by_id": { "1416305": "It seems the public test dataset is really small to have a big impact on the final leaderboard score. So, CV is the main thing to focus on. Let's compile a list of CV strategies that are useful for this kind of task.\n\n### **Cross-validation definition**\nhttps://neptune.ai/blog/cross-validation-in-machine-learning-how-to-do-it-right (types of CV)\n\nhttps://stackoverflow.com/questions/25889637/how-to-use-k-fold-cross-validation-in-a-neural-network (k-fold in NN)\n\nhttps://www.kaggle.com/satishgunjal/tutorial-k-fold-cross-validation (good code demonstration)\n\n### **Tutorials**\nhttps://machinelearningmastery.com/difference-test-validation-datasets/ (val vs test [useful for beginners])\n\nhttps://towardsdatascience.com/cross-validation-c4fae714f1c5 (beginner)\n\nhttps://www.simplilearn.com/tutorials/machine-learning-tutorial/cross-validation (intro materials)\n\nhttps://wandb.ai/authors/Kaggle-ML-CV/reports/Kaggle-s-Intermediate-ML-Cross-Validation--VmlldzoxOTY3NjQ (intermediate)\n\nhttps://machinelearningmastery.com/data-preparation-without-data-leakage/ (data leakage)\n\n### **Video tutorials / presentation**\nhttps://www.youtube.com/watch?v=Q0QmziFcfU0 (data leakage, in-depth)\n\n### **Tips & Tricks**\nhttps://www.datapred.com/blog/advanced-cross-validation-tips (simple discussion)\n\nhttps://towardsdatascience.com/increase-the-accuracy-of-your-cnn-by-following-these-5-tips-i-learned-from-the-kaggle-community-27227ad39554 (general to any kaggle computer vision competition)\n\nhttps://www.kaggle.com/general/18793 (compilation of strategies)\n\nhttps://mlwave.com/kaggle-ensembling-guide/ (long discussion + explanation)\n\n###**Kaggle Discussions**\n\nhttps://www.kaggle.com/questions-and-answers/47057\n\n\n##**To summarize**:\n1. Use 5-10 fold cross-validation (go high if you have the computational resources, but considering the amount of data available to you).\n2. Look for leakage. Don't use the same subjects in a single fold. (Especially, if you are using a 2-d model and training on individual slices, don't mix slices from the same subject/patient/study in multiple folds)\n3. Use multi-stage cross-validation if possible.\n" } }