{ "id": 444494, "title": "CV , LB results thread. Welcome to use AmbrosM's or MT's CV schemes", "url": "/competitions/open-problems-single-cell-perturbations/discussion/444494", "author_name": "", "post_date": "2023-10-02T10:33:12.511967300Z", "votes": 11, "comment_count": 8, "views": 0, "content": "

Kind of traditional thread - welcome to share your CV LB results.
\nWelcome to use CV scheme proposed by AmbrosM - discussion, notebook or MT's scheme: discussion, notebook.
\nMT proposes to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.

\n

AmbrosM:
\nLB 0.629 CV 0.965
\nModel: Ridge5, one-hot compound (only), tsvd100
\nLink: https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=9

\n
\n

My:
\nLB 0.625 CV 1.023748 02/10/2023
\nModel: tsvd35, Ridge for each component with selected by CV alpha, Target Encoder both cell-type+compound, with smooth = 1e7
\nLink: V33 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning/notebook?scriptVersionId=144955131
\nPS: LB 0.624 by blending 50 random folds - version 32 same nb.

\n
\n

Variation of above:
\nLB 0.615 CV 0.999888 02/10/2023
\nModel: similar to above (Ridge, TE for both cell and compound), but TE-smoothing parameters optimized separately for each tsvd component - as it was done before alpha-Ridge
\nLink: V43 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144976393
\nPS: LB 0.614 by blending 50 random folds on submission - version 45 same nb

\n

LB 0.612 CV0.996639 03/10/2023
\nModel: same model, other alpha Ridge, smoothing for Target Encoder - more optimization by CV
\nLink: V53: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145059766
\nPS: Blending 50 gives same LB0.612, in contrast of getting improvements in previous cases (V54)

", "messages": [ { "id": "2464537", "postDate": "10/02/2023 10:33:12", "content": "

Kind of traditional thread - welcome to share your CV LB results.
\nWelcome to use CV scheme proposed by AmbrosM - discussion, notebook or MT's scheme: discussion, notebook.
\nMT proposes to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.

\n

AmbrosM:
\nLB 0.629 CV 0.965
\nModel: Ridge5, one-hot compound (only), tsvd100
\nLink: https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=9

\n
\n

My:
\nLB 0.625 CV 1.023748 02/10/2023
\nModel: tsvd35, Ridge for each component with selected by CV alpha, Target Encoder both cell-type+compound, with smooth = 1e7
\nLink: V33 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning/notebook?scriptVersionId=144955131
\nPS: LB 0.624 by blending 50 random folds - version 32 same nb.

\n
\n

Variation of above:
\nLB 0.615 CV 0.999888 02/10/2023
\nModel: similar to above (Ridge, TE for both cell and compound), but TE-smoothing parameters optimized separately for each tsvd component - as it was done before alpha-Ridge
\nLink: V43 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144976393
\nPS: LB 0.614 by blending 50 random folds on submission - version 45 same nb

\n

LB 0.612 CV0.996639 03/10/2023
\nModel: same model, other alpha Ridge, smoothing for Target Encoder - more optimization by CV
\nLink: V53: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145059766
\nPS: Blending 50 gives same LB0.612, in contrast of getting improvements in previous cases (V54)

", "rawMarkdown": "Kind of traditional thread - welcome to share your CV LB results.\nWelcome to use CV scheme proposed by AmbrosM - [discussion](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/443395#2457831 ), [notebook](https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=8) or MT's scheme: [discussion](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/444494#2466644), [notebook](https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy/notebook). \nMT proposes to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.\n\n\nAmbrosM: \nLB 0.629 CV 0.965\nModel: Ridge5, one-hot compound (only), tsvd100\nLink: https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=9\n\n----------\n\nMy:\nLB 0.625 CV 1.023748 02/10/2023\nModel: tsvd35, Ridge for each component with selected by CV alpha, Target Encoder both cell-type+compound, with smooth = 1e7\nLink: V33 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning/notebook?scriptVersionId=144955131\nPS: LB 0.624 by blending 50 random folds - version 32 same nb. \n\n----------------\nVariation of above:\nLB 0.615 CV 0.999888 02/10/2023\nModel: similar to above (Ridge, TE for both cell and compound), but TE-smoothing parameters optimized separately for each tsvd component - as it was done before alpha-Ridge\nLink: V43 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144976393\nPS: LB 0.614 by blending 50 random folds on submission - version 45 same nb\n\nLB 0.612 CV0.996639 03/10/2023\nModel: same model, other alpha Ridge, smoothing for Target Encoder - more optimization by CV \nLink: V53: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145059766\nPS: Blending 50 gives same LB0.612, in contrast of getting improvements in previous cases (V54)", "votes": null }, { "id": "2466134", "postDate": "10/03/2023 15:22:58", "content": "

Thanks for sharing!!

\n

I am still struggling with a reliable local CV.

\n

Interesting CV scheme by AmbrosM by the way. I model the output at the lowest (cell-compound-gene) level (yeh… a lot of noise). Feeling AmbrosM' train/val split may be too harsh on the train (putting almost an entire cell type into val), I am just using a random 5-fold on the entire train set. Apparently my local CV doesn't correlate well with LB That drives me crazy.

\n

I am thinking of normalizing the output, but have no good idea so far. Or AmbrosM's train/val split is ultimately the way to go to avoid overfitting to the val.

\n

Below are my results.

\n

Model: Simple 4 layers NN. Input: one hots for compound and cell type, tsvd embedding for gene. Ensemble over models trained on 5 different folds.

\n", "rawMarkdown": "Thanks for sharing!! \n\nI am still struggling with a reliable local CV. \n\nInteresting CV scheme by AmbrosM by the way. I model the output at the lowest (cell-compound-gene) level (yeh... a lot of noise). Feeling AmbrosM' train/val split may be too harsh on the train (putting almost an entire cell type into val), I am just using a random 5-fold on the entire train set. **Apparently my local CV doesn't correlate well with LB** That drives me crazy.\n\nI am thinking of normalizing the output, but have no good idea so far. Or AmbrosM's train/val split is ultimately the way to go to avoid overfitting to the val.\n\nBelow are my results. \n\nModel: Simple 4 layers NN. Input: one hots for compound and cell type, tsvd embedding for gene. Ensemble over models trained on 5 different folds.\n\n- Version 1, gene embed=TSVD32, early stopping after 3 epochs \n**LB 0.606**\n**CV All cells: 0.860**\nBreak down:\nB cells, 1.372\nMyeloid cells, 1.818\nNK cells, 0.897\nT cells CD4+, 0.964\nT cells CD8+, 0.696\nT regulatory cells, 0.707\n\n\n- Version 2, gene embed=TSVD35, early stopping after 3 epochs \n**LB 0.607**\n**CV All cells: 0.872**\nBreak down:\nB cells, 1.363\nMyeloid cells, 1.821\nNK cells, 0.903\nT cells CD4+, 0.980\nT cells CD8+, 0.718\nT regulatory cells, 0.715\n\n\n- Version 3, gene embed=TSVD35, early stopping after 5 epochs \n**LB 0.623**\n**CV All cells: 0.847**\nBreak down:\nB cells, 1.325\nMyeloid cells, 1.751\nNK cells, 0.881\nT cells CD4+, 0.959\nT cells CD8+, 0.689\nT regulatory cells, 0.694", "votes": null }, { "id": "2466644", "postDate": "10/04/2023 02:30:29", "content": "

I doubt that generalization is possible with this small scale data, but I would like to support the objectives of this competition. Anyway, unless we see some breakthroughs, it seems to me it is ultimately an interpolation problem as the simple average over compounds shows a good score.
\nTherefore, if we adopt my CV strategy as an example, the smaller regularization showed better score. Note that I confirmed that alpha=5 was a good choice in AmbrosM's scheme.

\n

I am trying to see if I can generalize the problem somehow, but it is too tough….

", "rawMarkdown": "I doubt that generalization is possible with this small scale data, but I would like to support the objectives of this competition. Anyway, unless we see some breakthroughs, it seems to me it is ultimately an interpolation problem as the simple average over compounds shows a good score.\nTherefore, if we adopt [my CV strategy](https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy/notebook) as an example, the smaller regularization showed better score. Note that I confirmed that alpha=5 was a good choice in AmbrosM's scheme.\n\nI am trying to see if I can generalize the problem somehow, but it is too tough....", "votes": null }, { "id": "2466987", "postDate": "10/04/2023 08:05:27", "content": "

Cool ! Thanks a lot for sharing !
\nSo you propose to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.
\nPS
\nWould be great to see LB score for submission with alpha = 5 - to check the LB - CV correspondence works for your scheme. AmbrosM submission with 0.629 have different tsvd = 100 , and yours tsvd = 30 - so cannot rely on him directly.

", "rawMarkdown": "Cool ! Thanks a lot for sharing ! \nSo you propose to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.\nPS\nWould be great to see LB score for submission with alpha = 5 - to check the LB - CV correspondence works for your scheme. AmbrosM submission with 0.629 have different tsvd = 100 , and yours tsvd = 30 - so cannot rely on him directly.", "votes": null }, { "id": "2467146", "postDate": "10/04/2023 10:32:08", "content": "

That makes sense. I found the CV-LB scores were 2.863-0.630 to alpha=5 and 2.670-0.615 to alpha=0.1.

", "rawMarkdown": "That makes sense. I found the CV-LB scores were 2.863-0.630 to alpha=5 and 2.670-0.615 to alpha=0.1.", "votes": null }, { "id": "2467403", "postDate": "10/04/2023 14:17:44", "content": "

Cool ! Thanks a lot for sharing !

", "rawMarkdown": "Cool ! Thanks a lot for sharing !", "votes": null }, { "id": "2468980", "postDate": "10/06/2023 03:37:39", "content": "

Thanks for sharing!
\nMy results are strange for now:

\n

OP2: BL submission - Version 4: ignores cell type, 3-fold CV, target-encoded features (20 PC)
\nCV 0.970 LB 0.621

\n

OP2: BL submission - Version 5: same + emsemble with means by compound and cell tupe (just as in public notebooks)
\nLB 0.608

\n

OP2: BL submission - Version 8: 7-fold CV, target-encoded features (both drug and cell type, 20 PC), PCA of targets, prediction of 10 PC (locally better than 5, 15, and 25) and inverse transform + ensemble as above
\nCV 1.783 LB 0.613

\n

So no correlation between CV vs LB when I try other features within the same model, and ignoring the cell type seems better. I didn't yet tried other CV schemes, will see)

\n

Upd: also tried smooth instead of adding random noise to target-encoded features, but locally it caused worse performance, so didn't check on LB.

", "rawMarkdown": "Thanks for sharing!\nMy results are strange for now:\n\n[OP2: BL submission - Version 4](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=144624870): ignores cell type, 3-fold CV, target-encoded features (20 PC)\nCV 0.970 LB 0.621\n\n[OP2: BL submission - Version 5](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=144626453): same + emsemble with means by compound and cell tupe (just as in public notebooks)\nLB 0.608\n\n[OP2: BL submission - Version 8](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=145411450): 7-fold CV, target-encoded features (both drug and cell type, 20 PC), PCA of targets, prediction of 10 PC (locally better than 5, 15, and 25) and inverse transform + ensemble as above\nCV 1.783 LB 0.613\n\nSo no correlation between CV vs LB when I try other features within the same model, and ignoring the cell type seems better. I didn't yet tried other CV schemes, will see)\n\nUpd: also tried smooth instead of adding random noise to target-encoded features, but locally it caused worse performance, so didn't check on LB.", "votes": null }, { "id": "2498328", "postDate": "10/25/2023 08:28:56", "content": "

Here is a class with similar interface as sklearn Kfold to support proposed validation schemes by AmbrosM, MT, etc:

\n

https://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes

\n

And some analysis.
\nMore analysis of the CV schemes can be found in slides: https://docs.google.com/presentation/d/1wiz0Wmt4D54pqMMsIOyJHuQYMZ3hTBZQQnjbLzwoGYY/edit?usp=sharing and sheet: https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing

\n

\"\"
\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes?scriptVersionId=147920488&cellId=22

", "rawMarkdown": "Here is a class with similar interface as sklearn Kfold to support proposed validation schemes by AmbrosM, MT, etc:\n\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes\n\nAnd some analysis.\nMore analysis of the CV schemes can be found in slides: https://docs.google.com/presentation/d/1wiz0Wmt4D54pqMMsIOyJHuQYMZ3hTBZQQnjbLzwoGYY/edit?usp=sharing and sheet: https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Fc5df653d72b357d22c4a27b3a8d9aebc%2FScreenshot%202023-10-25%20105924.png?generation=1698224422228209&alt=media)\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes?scriptVersionId=147920488&cellId=22", "votes": null }, { "id": "2508324", "postDate": "11/01/2023 15:54:05", "content": "

Here is some systemic comparison of the results for AmbrosM, MT and random CV schemes.
\nMainly AmbrosM and random are quite correlated and requires much stronger regularization.
\nFor one submitted example they give better correlation to LB, in contrast to model from the original notebooks where MT scheme was better.

\n

Still the correspondence CV LB is quite poor.

\n

https://www.kaggle.com/code/alexandervc/op2-gentle-param-tuner?scriptVersionId=148888605&cellId=1 :

\n

\"\"

", "rawMarkdown": "Here is some systemic comparison of the results for AmbrosM, MT and random CV schemes.\nMainly AmbrosM and random are quite correlated and requires much stronger regularization.\nFor one submitted example they give better correlation to LB, in contrast to model from the original notebooks where MT scheme was better.\n\nStill the correspondence CV LB is quite poor. \n\nhttps://www.kaggle.com/code/alexandervc/op2-gentle-param-tuner?scriptVersionId=148888605&cellId=1 :\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F5b68a5cbf7d402fd82310f2a64229c16%2FScreenshot%202023-11-01%20165025.png?generation=1698853873551523&alt=media)", "votes": null } ], "comments": [ { "id": 2466134, "author_name": "qihuaz", "author_url": "", "post_date": "10/03/2023 15:22:58", "content": "

Thanks for sharing!!

\n

I am still struggling with a reliable local CV.

\n

Interesting CV scheme by AmbrosM by the way. I model the output at the lowest (cell-compound-gene) level (yeh… a lot of noise). Feeling AmbrosM' train/val split may be too harsh on the train (putting almost an entire cell type into val), I am just using a random 5-fold on the entire train set. Apparently my local CV doesn't correlate well with LB That drives me crazy.

\n

I am thinking of normalizing the output, but have no good idea so far. Or AmbrosM's train/val split is ultimately the way to go to avoid overfitting to the val.

\n

Below are my results.

\n

Model: Simple 4 layers NN. Input: one hots for compound and cell type, tsvd embedding for gene. Ensemble over models trained on 5 different folds.

\n", "votes": null, "replies": [] }, { "id": 2466644, "author_name": "masato114", "author_url": "", "post_date": "10/04/2023 02:30:29", "content": "

I doubt that generalization is possible with this small scale data, but I would like to support the objectives of this competition. Anyway, unless we see some breakthroughs, it seems to me it is ultimately an interpolation problem as the simple average over compounds shows a good score.
\nTherefore, if we adopt my CV strategy as an example, the smaller regularization showed better score. Note that I confirmed that alpha=5 was a good choice in AmbrosM's scheme.

\n

I am trying to see if I can generalize the problem somehow, but it is too tough….

", "votes": null, "replies": [ { "id": 2466987, "author_name": "alexandervc", "author_url": "", "post_date": "10/04/2023 08:05:27", "content": "

Cool ! Thanks a lot for sharing !
\nSo you propose to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.
\nPS
\nWould be great to see LB score for submission with alpha = 5 - to check the LB - CV correspondence works for your scheme. AmbrosM submission with 0.629 have different tsvd = 100 , and yours tsvd = 30 - so cannot rely on him directly.

", "votes": null, "replies": [ { "id": 2467146, "author_name": "masato114", "author_url": "", "post_date": "10/04/2023 10:32:08", "content": "

That makes sense. I found the CV-LB scores were 2.863-0.630 to alpha=5 and 2.670-0.615 to alpha=0.1.

", "votes": null, "replies": [ { "id": 2467403, "author_name": "alexandervc", "author_url": "", "post_date": "10/04/2023 14:17:44", "content": "

Cool ! Thanks a lot for sharing !

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2468980, "author_name": "antoninadolgorukova", "author_url": "", "post_date": "10/06/2023 03:37:39", "content": "

Thanks for sharing!
\nMy results are strange for now:

\n

OP2: BL submission - Version 4: ignores cell type, 3-fold CV, target-encoded features (20 PC)
\nCV 0.970 LB 0.621

\n

OP2: BL submission - Version 5: same + emsemble with means by compound and cell tupe (just as in public notebooks)
\nLB 0.608

\n

OP2: BL submission - Version 8: 7-fold CV, target-encoded features (both drug and cell type, 20 PC), PCA of targets, prediction of 10 PC (locally better than 5, 15, and 25) and inverse transform + ensemble as above
\nCV 1.783 LB 0.613

\n

So no correlation between CV vs LB when I try other features within the same model, and ignoring the cell type seems better. I didn't yet tried other CV schemes, will see)

\n

Upd: also tried smooth instead of adding random noise to target-encoded features, but locally it caused worse performance, so didn't check on LB.

", "votes": null, "replies": [] }, { "id": 2498328, "author_name": "alexandervc", "author_url": "", "post_date": "10/25/2023 08:28:56", "content": "

Here is a class with similar interface as sklearn Kfold to support proposed validation schemes by AmbrosM, MT, etc:

\n

https://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes

\n

And some analysis.
\nMore analysis of the CV schemes can be found in slides: https://docs.google.com/presentation/d/1wiz0Wmt4D54pqMMsIOyJHuQYMZ3hTBZQQnjbLzwoGYY/edit?usp=sharing and sheet: https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing

\n

\"\"
\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes?scriptVersionId=147920488&cellId=22

", "votes": null, "replies": [] }, { "id": 2508324, "author_name": "alexandervc", "author_url": "", "post_date": "11/01/2023 15:54:05", "content": "

Here is some systemic comparison of the results for AmbrosM, MT and random CV schemes.
\nMainly AmbrosM and random are quite correlated and requires much stronger regularization.
\nFor one submitted example they give better correlation to LB, in contrast to model from the original notebooks where MT scheme was better.

\n

Still the correspondence CV LB is quite poor.

\n

https://www.kaggle.com/code/alexandervc/op2-gentle-param-tuner?scriptVersionId=148888605&cellId=1 :

\n

\"\"

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2464537": "Kind of traditional thread - welcome to share your CV LB results.\nWelcome to use CV scheme proposed by AmbrosM - [discussion](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/443395#2457831 ), [notebook](https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=8) or MT's scheme: [discussion](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/444494#2466644), [notebook](https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy/notebook). \nMT proposes to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.\n\n\nAmbrosM: \nLB 0.629 CV 0.965\nModel: Ridge5, one-hot compound (only), tsvd100\nLink: https://www.kaggle.com/code/ambrosm/scp-quickstart?scriptVersionId=144293041&cellId=9\n\n----------\n\nMy:\nLB 0.625 CV 1.023748 02/10/2023\nModel: tsvd35, Ridge for each component with selected by CV alpha, Target Encoder both cell-type+compound, with smooth = 1e7\nLink: V33 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning/notebook?scriptVersionId=144955131\nPS: LB 0.624 by blending 50 random folds - version 32 same nb. \n\n----------------\nVariation of above:\nLB 0.615 CV 0.999888 02/10/2023\nModel: similar to above (Ridge, TE for both cell and compound), but TE-smoothing parameters optimized separately for each tsvd component - as it was done before alpha-Ridge\nLink: V43 https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=144976393\nPS: LB 0.614 by blending 50 random folds on submission - version 45 same nb\n\nLB 0.612 CV0.996639 03/10/2023\nModel: same model, other alpha Ridge, smoothing for Target Encoder - more optimization by CV \nLink: V53: https://www.kaggle.com/code/alexandervc/op2-models-cv-tuning?scriptVersionId=145059766\nPS: Blending 50 gives same LB0.612, in contrast of getting improvements in previous cases (V54)", "2466134": "Thanks for sharing!! \n\nI am still struggling with a reliable local CV. \n\nInteresting CV scheme by AmbrosM by the way. I model the output at the lowest (cell-compound-gene) level (yeh... a lot of noise). Feeling AmbrosM' train/val split may be too harsh on the train (putting almost an entire cell type into val), I am just using a random 5-fold on the entire train set. **Apparently my local CV doesn't correlate well with LB** That drives me crazy.\n\nI am thinking of normalizing the output, but have no good idea so far. Or AmbrosM's train/val split is ultimately the way to go to avoid overfitting to the val.\n\nBelow are my results. \n\nModel: Simple 4 layers NN. Input: one hots for compound and cell type, tsvd embedding for gene. Ensemble over models trained on 5 different folds.\n\n- Version 1, gene embed=TSVD32, early stopping after 3 epochs \n**LB 0.606**\n**CV All cells: 0.860**\nBreak down:\nB cells, 1.372\nMyeloid cells, 1.818\nNK cells, 0.897\nT cells CD4+, 0.964\nT cells CD8+, 0.696\nT regulatory cells, 0.707\n\n\n- Version 2, gene embed=TSVD35, early stopping after 3 epochs \n**LB 0.607**\n**CV All cells: 0.872**\nBreak down:\nB cells, 1.363\nMyeloid cells, 1.821\nNK cells, 0.903\nT cells CD4+, 0.980\nT cells CD8+, 0.718\nT regulatory cells, 0.715\n\n\n- Version 3, gene embed=TSVD35, early stopping after 5 epochs \n**LB 0.623**\n**CV All cells: 0.847**\nBreak down:\nB cells, 1.325\nMyeloid cells, 1.751\nNK cells, 0.881\nT cells CD4+, 0.959\nT cells CD8+, 0.689\nT regulatory cells, 0.694", "2466644": "I doubt that generalization is possible with this small scale data, but I would like to support the objectives of this competition. Anyway, unless we see some breakthroughs, it seems to me it is ultimately an interpolation problem as the simple average over compounds shows a good score.\nTherefore, if we adopt [my CV strategy](https://www.kaggle.com/code/masato114/scp-quickstart-another-cv-strategy/notebook) as an example, the smaller regularization showed better score. Note that I confirmed that alpha=5 was a good choice in AmbrosM's scheme.\n\nI am trying to see if I can generalize the problem somehow, but it is too tough....", "2466987": "Cool ! Thanks a lot for sharing ! \nSo you propose to put in validation SAME CELL-TYPES as on LB, while AmbrosM proposes SAME COMPOUNDS.\nPS\nWould be great to see LB score for submission with alpha = 5 - to check the LB - CV correspondence works for your scheme. AmbrosM submission with 0.629 have different tsvd = 100 , and yours tsvd = 30 - so cannot rely on him directly.", "2467146": "That makes sense. I found the CV-LB scores were 2.863-0.630 to alpha=5 and 2.670-0.615 to alpha=0.1.", "2467403": "Cool ! Thanks a lot for sharing !", "2468980": "Thanks for sharing!\nMy results are strange for now:\n\n[OP2: BL submission - Version 4](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=144624870): ignores cell type, 3-fold CV, target-encoded features (20 PC)\nCV 0.970 LB 0.621\n\n[OP2: BL submission - Version 5](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=144626453): same + emsemble with means by compound and cell tupe (just as in public notebooks)\nLB 0.608\n\n[OP2: BL submission - Version 8](https://www.kaggle.com/code/antoninadolgorukova/op2-bl-submission?scriptVersionId=145411450): 7-fold CV, target-encoded features (both drug and cell type, 20 PC), PCA of targets, prediction of 10 PC (locally better than 5, 15, and 25) and inverse transform + ensemble as above\nCV 1.783 LB 0.613\n\nSo no correlation between CV vs LB when I try other features within the same model, and ignoring the cell type seems better. I didn't yet tried other CV schemes, will see)\n\nUpd: also tried smooth instead of adding random noise to target-encoded features, but locally it caused worse performance, so didn't check on LB.", "2498328": "Here is a class with similar interface as sklearn Kfold to support proposed validation schemes by AmbrosM, MT, etc:\n\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes\n\nAnd some analysis.\nMore analysis of the CV schemes can be found in slides: https://docs.google.com/presentation/d/1wiz0Wmt4D54pqMMsIOyJHuQYMZ3hTBZQQnjbLzwoGYY/edit?usp=sharing and sheet: https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Fc5df653d72b357d22c4a27b3a8d9aebc%2FScreenshot%202023-10-25%20105924.png?generation=1698224422228209&alt=media)\nhttps://www.kaggle.com/code/alexandervc/op2-class-for-custom-cv-schemes?scriptVersionId=147920488&cellId=22", "2508324": "Here is some systemic comparison of the results for AmbrosM, MT and random CV schemes.\nMainly AmbrosM and random are quite correlated and requires much stronger regularization.\nFor one submitted example they give better correlation to LB, in contrast to model from the original notebooks where MT scheme was better.\n\nStill the correspondence CV LB is quite poor. \n\nhttps://www.kaggle.com/code/alexandervc/op2-gentle-param-tuner?scriptVersionId=148888605&cellId=1 :\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F5b68a5cbf7d402fd82310f2a64229c16%2FScreenshot%202023-11-01%20165025.png?generation=1698853873551523&alt=media)" }, "source": "meta" }