![\"cross-validation\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7917824%2F11f9f88d4fa3d032ce05734bc464ce66%2Fcv.png?generation=1695798735354050&alt=media\")
I heard from kaggle that it's important to trust your local CV score and not overfit to the public leaderboard. For this competition, I noticed that if you just do a normal k-fold CV, you'll get nulls in the validation split. To fix this, I learned I could do a stratified k-fold, stratifying on sm_name. However, this limits to just 3 folds max before getting the minimum categories per split error. Does anyone have a good take on this?
I heard from kaggle that it's important to trust your local CV score and not overfit to the public leaderboard. For this competition, I noticed that if you just do a normal k-fold CV, you'll get nulls in the validation split. To fix this, I learned I could do a stratified k-fold, stratifying on sm_name. However, this limits to just 3 folds max before getting the minimum categories per split error. Does anyone have a good take on this?
The competition data is split into train and test so that the test data contains 90 % of the myeloid and B cells. This setting should be simulated by the cross-validation strategy.
\nBecause there are six cell types of which two are used for the test data, I suggest to make four cross-validation folds, each one based on 90 % of one of the other four cell-types. The diagram shows training data (green), test data (red) and one of the four validation folds (blue). The few missing cell_type–sm_name combinations (black) can be ignored:
\n
The complete source code is in the SCP Quickstart notebook.
", "rawMarkdown": "The competition data is split into train and test so that the test data contains 90 % of the myeloid and B cells. This setting should be simulated by the cross-validation strategy. \n\nBecause there are six cell types of which two are used for the test data, I suggest to make four cross-validation folds, each one based on 90 % of one of the other four cell-types. The diagram shows training data (green), test data (red) and one of the four validation folds (blue). The few missing cell_type–sm_name combinations (black) can be ignored:\n\n\n\nThe complete source code is in the [SCP Quickstart](https://www.kaggle.com/code/ambrosm/scp-quickstart) notebook.", "votes": null }, { "id": "2458385", "postDate": "09/27/2023 14:53:40", "content": "Awesome, thanks for the info and code! I'll take a look at this strategy, sounds practical.
", "rawMarkdown": "Awesome, thanks for the info and code! I'll take a look at this strategy, sounds practical.", "votes": null } ], "comments": [ { "id": 2457831, "author_name": "ambrosm", "author_url": "", "post_date": "09/27/2023 07:23:29", "content": "The competition data is split into train and test so that the test data contains 90 % of the myeloid and B cells. This setting should be simulated by the cross-validation strategy.
\nBecause there are six cell types of which two are used for the test data, I suggest to make four cross-validation folds, each one based on 90 % of one of the other four cell-types. The diagram shows training data (green), test data (red) and one of the four validation folds (blue). The few missing cell_type–sm_name combinations (black) can be ignored:
\n
The complete source code is in the SCP Quickstart notebook.
", "votes": null, "replies": [ { "id": 2458385, "author_name": "yuqizheng", "author_url": "", "post_date": "09/27/2023 14:53:40", "content": "Awesome, thanks for the info and code! I'll take a look at this strategy, sounds practical.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2457571": "I heard from kaggle that it's important to trust your local CV score and not overfit to the public leaderboard. For this competition, I noticed that if you just do a normal k-fold CV, you'll get nulls in the validation split. To fix this, I learned I could do a stratified k-fold, stratifying on `sm_name`. However, this limits to just 3 folds max before getting the minimum categories per split error. Does anyone have a good take on this?", "2457831": "The competition data is split into train and test so that the test data contains 90 % of the myeloid and B cells. This setting should be simulated by the cross-validation strategy. \n\nBecause there are six cell types of which two are used for the test data, I suggest to make four cross-validation folds, each one based on 90 % of one of the other four cell-types. The diagram shows training data (green), test data (red) and one of the four validation folds (blue). The few missing cell_type–sm_name combinations (black) can be ignored:\n\n\n\nThe complete source code is in the [SCP Quickstart](https://www.kaggle.com/code/ambrosm/scp-quickstart) notebook.", "2458385": "Awesome, thanks for the info and code! I'll take a look at this strategy, sounds practical." }, "source": "meta" }