What CV you used ? How bad was CV-LB correspondence ?
\nWhat tricks/lessons to overcome poor CV - LB correspondence ?
\nHave you used the NK-cells as local validation fold - was it good for you ? If yes - how did you come to it ?
Our side:
\nPains: Started with AmbrosM and MT schemes - but observed a poor LB CV correspondence for both. Especially comparing different types of models say boosting and NN seemed hopeless. Lacked ideas and so worked with them in the following ways a) still sometimes local improvement was good on LB b) searched for cases which improve BOTH these CV (even that does not always work) and such search was not easy
Good news 1: At some late point observed that for many cases of PYBOOST model - the first fold on AmbrosM scheme - i.e. NK-cells have pretty good correspondence with LB. Started to think what was going on.
\n2.1 Clever people observed that NK cells close to LB immediately looking on EDA : https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/457793#2541075
\nbut we come to that point - only after much pain , is there any trick/lesson to reduce that pain in future ?
2.2 Actually we have also seen hint for that by the EDA with clustermap:
\nhttps://www.kaggle.com/code/alexandervc/op2-eda-baseline-s?scriptVersionId=147818286&cellId=21
\nbut did not believe that should pay attention on that…
Trick/lesson 1: Started to look on accumulated statistics of submits - just calculated correlation between LB scores and scores on each folds - observed: the NK-fold (AmbrosM-1) and first fold in MT scheme are much better - about 0.5 correlated with LB. While the other folds correlated badly.
Trick/lesson 2: tried several other metrics to correlate with the LB for our submits - observed the average row-wise correlation - is better correlated with LB, than competition metric mrrmse itself ! Strange enough.
Lesson: save your OOF predicts - then you will be able analyze with many metrics and folds - how to fit LB better ( after many submits are accumulated). May be even train a small model to predict LB from your local metrics (not actually tried that now).
do not overthink: Turned back to random folds. Observed that a) results on random CV quite similar to results on AmbrosM CV b) in some cases modified Random folds (with only test drugs) shows better corresponde to LB - for some NN correlation was about 0.9 !
Check on big uplifts Begin to compare some cases when we have model-1 and model-2 and param change gives strong uplift - checked metrics on those cases. Observed that for some NN the NK-fold is not perfect - uplift on LB - does not correspond to uplfit on CV, but works find with modified random folds.
\nStarted to work mainly with random folds.
What CV you used ? How bad was CV-LB correspondence ?
\nWhat tricks/lessons to overcome poor CV - LB correspondence ?
\nHave you used the NK-cells as local validation fold - was it good for you ? If yes - how did you come to it ?
Our side:
\nPains: Started with AmbrosM and MT schemes - but observed a poor LB CV correspondence for both. Especially comparing different types of models say boosting and NN seemed hopeless. Lacked ideas and so worked with them in the following ways a) still sometimes local improvement was good on LB b) searched for cases which improve BOTH these CV (even that does not always work) and such search was not easy
Good news 1: At some late point observed that for many cases of PYBOOST model - the first fold on AmbrosM scheme - i.e. NK-cells have pretty good correspondence with LB. Started to think what was going on.
\n2.1 Clever people observed that NK cells close to LB immediately looking on EDA : https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/457793#2541075
\nbut we come to that point - only after much pain , is there any trick/lesson to reduce that pain in future ?
2.2 Actually we have also seen hint for that by the EDA with clustermap:
\nhttps://www.kaggle.com/code/alexandervc/op2-eda-baseline-s?scriptVersionId=147818286&cellId=21
\nbut did not believe that should pay attention on that…
Trick/lesson 1: Started to look on accumulated statistics of submits - just calculated correlation between LB scores and scores on each folds - observed: the NK-fold (AmbrosM-1) and first fold in MT scheme are much better - about 0.5 correlated with LB. While the other folds correlated badly.
Trick/lesson 2: tried several other metrics to correlate with the LB for our submits - observed the average row-wise correlation - is better correlated with LB, than competition metric mrrmse itself ! Strange enough.
Lesson: save your OOF predicts - then you will be able analyze with many metrics and folds - how to fit LB better ( after many submits are accumulated). May be even train a small model to predict LB from your local metrics (not actually tried that now).
do not overthink: Turned back to random folds. Observed that a) results on random CV quite similar to results on AmbrosM CV b) in some cases modified Random folds (with only test drugs) shows better corresponde to LB - for some NN correlation was about 0.9 !
Check on big uplifts Begin to compare some cases when we have model-1 and model-2 and param change gives strong uplift - checked metrics on those cases. Observed that for some NN the NK-fold is not perfect - uplift on LB - does not correspond to uplfit on CV, but works find with modified random folds.
\nStarted to work mainly with random folds.
Thank you for sharing the details of your cross-validation journey! Finding a suitable local validation set was arguably the most challenging aspect of this competition and the lack of a solid validation set had me worried about overfitting to the public leaderboard all through the competition. I tried to be clever about it in the beginning but gave up pretty quickly and started relying mostly on random splits for my neural nets, whereas AmbrosM's CV scheme seemed to work better for random and boosted forests.
", "rawMarkdown": "Thank you for sharing the details of your cross-validation journey! Finding a suitable local validation set was arguably the most challenging aspect of this competition and the lack of a solid validation set had me worried about overfitting to the public leaderboard all through the competition. I tried to be clever about it in the beginning but gave up pretty quickly and started relying mostly on random splits for my neural nets, whereas AmbrosM's CV scheme seemed to work better for random and boosted forests.", "votes": null } ], "comments": [ { "id": 2545824, "author_name": "frenio", "author_url": "", "post_date": "12/01/2023 19:22:55", "content": "Thank you for sharing the details of your cross-validation journey! Finding a suitable local validation set was arguably the most challenging aspect of this competition and the lack of a solid validation set had me worried about overfitting to the public leaderboard all through the competition. I tried to be clever about it in the beginning but gave up pretty quickly and started relying mostly on random splits for my neural nets, whereas AmbrosM's CV scheme seemed to work better for random and boosted forests.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2545763": "What CV you used ? How bad was CV-LB correspondence ?\nWhat tricks/lessons to overcome poor CV - LB correspondence ? \nHave you used the NK-cells as local validation fold - was it good for you ? If yes - how did you come to it ? \n\nOur side: \n\n1. **Pains:** Started with AmbrosM and MT schemes - but observed a poor LB CV correspondence for both. Especially comparing different types of models say boosting and NN seemed hopeless. Lacked ideas and so worked with them in the following ways a) still sometimes local improvement was good on LB b) searched for cases which improve BOTH these CV (even that does not always work) and such search was not easy \n\n2. **Good news 1:** At some late point observed that for many cases of PYBOOST model - the first fold on AmbrosM scheme - i.e. NK-cells have pretty good correspondence with LB. Started to think what was going on.\n\n 2.1 Clever people observed that NK cells close to LB immediately looking on EDA : https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/457793#2541075\nbut we come to that point - only after much pain , is there any trick/lesson to reduce that pain in future ? \n\n 2.2 Actually we have also seen hint for that by the EDA with clustermap:\nhttps://www.kaggle.com/code/alexandervc/op2-eda-baseline-s?scriptVersionId=147818286&cellId=21\nbut did not believe that should pay attention on that... \n\n3. **Trick/lesson 1:** Started to look on accumulated statistics of submits - just calculated correlation between LB scores and scores on each folds - observed: the NK-fold (AmbrosM-1) and first fold in MT scheme are much better - about 0.5 correlated with LB. While the other folds correlated badly.\n\n4. **Trick/lesson 2:** tried several other metrics to correlate with the LB for our submits - observed the average row-wise correlation - is better correlated with LB, than competition metric mrrmse itself ! Strange enough.\n\n5. **Lesson:** save your OOF predicts - then you will be able analyze with many metrics and folds - how to fit LB better ( after many submits are accumulated). May be even train a small model to predict LB from your local metrics (not actually tried that now). \n\n6. **do not overthink:** Turned back to random folds. Observed that a) results on random CV quite similar to results on AmbrosM CV b) in some cases modified Random folds (with only test drugs) shows better corresponde to LB - for some NN correlation was about 0.9 ! \n\n7. **Check on big uplifts** Begin to compare some cases when we have model-1 and model-2 and param change gives strong uplift - checked metrics on those cases. Observed that for some NN the NK-fold is not perfect - uplift on LB - does not correspond to uplfit on CV, but works find with modified random folds.\nStarted to work mainly with random folds.", "2545824": "Thank you for sharing the details of your cross-validation journey! Finding a suitable local validation set was arguably the most challenging aspect of this competition and the lack of a solid validation set had me worried about overfitting to the public leaderboard all through the competition. I tried to be clever about it in the beginning but gave up pretty quickly and started relying mostly on random splits for my neural nets, whereas AmbrosM's CV scheme seemed to work better for random and boosted forests." }, "source": "meta" }