![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F4e4593763bab047a0a77c89a8a64b982%2FScreenshot%202023-12-02%20133431.png?generation=1701520508010298&alt=media\")
Here is notebook https://www.kaggle.com/code/alexandervc/op2-public-vs-private-scores
\nto analyze public and private scores for submissions. (For the present and the last year competitions).
Correlations are very high Pearson : 0.98 (2023), 0.99 (2022); Spearman: 0.96 (2023), 0.89 (2022).
\nIt seems it is quite surprising, is not it ?
\nSeems for that competition we can \"trust LB, not CV\" )))
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Fc0ceaf26cc02e86bda29652ea55be000%2FScreenshot%202023-12-02%20133548.png?generation=1701520577228715&alt=media\")
Here is notebook https://www.kaggle.com/code/alexandervc/op2-public-vs-private-scores
\nto analyze public and private scores for submissions. (For the present and the last year competitions).
Correlations are very high Pearson : 0.98 (2023), 0.99 (2022); Spearman: 0.96 (2023), 0.89 (2022).
\nIt seems it is quite surprising, is not it ?
\nSeems for that competition we can \"trust LB, not CV\" )))
\n\n
Let me add another observation and then propose an explanation.
\nObservation: In the following table, which shows the cv scores of my four models, some rows are negatively correlated. For NK cells, ridge is best and Py-boost is worst; for CD4+ cells, ridge is worst and ExtraTrees is best. The negative correlation means that how well the models perform for NK cells has nothing to do with how well they perform for CD4+ cells.
\nBetween private and public leaderboard, I have a correlation of 0.98 (with 57 submissions) like you.
\n| \n | Py-boost | \nRidge recommender | \nKNN recommender | \nExtraTrees | \n
|---|---|---|---|---|
| NK cells | \n1.012 | \n0.940 | \n0.955 | \n0.992 | \n
| T cells CD4+ | \n0.904 | \n0.929 | \n0.928 | \n0.882 | \n
| T cells CD8+ | \n0.780 | \n0.776 | \n0.758 | \n0.758 | \n
| T regulatory cells | \n0.925 | \n0.883 | \n0.881 | \n0.879 | \n
Proposed explanation: We may assume that every cell type has a particular characteristic parameter which affects all compounds, but which our models fail to predict accurately when given only 17 training compounds. This characteristic could be a gene expression pattern, something related to the cell count, or an artefact produced by Limma (note that Limma is run independently for every cell type).
\nAs the model cannot predict this parameter, it guesses the parameter more or less successfully. If the model guesses the parameters for B and Myeloid cells well, private and public leaderboard both show good scores. If the model guesses the parameters badly for B and Myeloid cells, all predictions are biased and private and public leaderboard both have bad scores. But the quality of the guess for B and Myeloid cells has nothing to do with the quality of the guess for NK cells or T regulatory cells.
\nThe question remains: Can we describe this parameter?
", "rawMarkdown": "Let me add another observation and then propose an explanation.\n\n**Observation:** In the following table, which shows the cv scores of [my four models](https://www.kaggle.com/code/ambrosm/scp-26-py-boost-recommender-system-and-et), some rows are negatively correlated. For NK cells, ridge is best and Py-boost is worst; for CD4+ cells, ridge is worst and ExtraTrees is best. The negative correlation means that how well the models perform for NK cells has nothing to do with how well they perform for CD4+ cells. \n\nBetween private and public leaderboard, I have a correlation of 0.98 (with 57 submissions) like you.\n\n| | Py-boost | Ridge recommender | KNN recommender | ExtraTrees |\n| --- | --- | --- | --- | --- |\n| NK cells | 1.012 | 0.940 | 0.955 | 0.992 |\n| T cells CD4+ | 0.904 | 0.929 | 0.928 | 0.882 |\n| T cells CD8+ | 0.780 | 0.776 | 0.758 | 0.758 |\n| T regulatory cells | 0.925 | 0.883 | 0.881 | 0.879 |\n\n**Proposed explanation:** We may assume that every cell type has a particular characteristic parameter which affects all compounds, but which our models fail to predict accurately when given only 17 training compounds. This characteristic could be a gene expression pattern, something related to the cell count, or an artefact produced by Limma (note that Limma is run independently for every cell type).\n\nAs the model cannot predict this parameter, it guesses the parameter more or less successfully. If the model guesses the parameters for B and Myeloid cells well, private and public leaderboard both show good scores. If the model guesses the parameters badly for B and Myeloid cells, all predictions are biased and private and public leaderboard both have bad scores. But the quality of the guess for B and Myeloid cells has nothing to do with the quality of the guess for NK cells or T regulatory cells. \n\nThe question remains: Can we describe this parameter?", "votes": null }, { "id": "2546706", "postDate": "12/02/2023 18:17:48", "content": "Thanks a lot for sharing !
\nMy feeling is that NK fold is the most close to test cells.
\nHere is a table of LB and CV scores by your CV-scheme for several modifications of the pyboost.
\nThe improvement on LB is reflected by the improvement on the NK fold (green - values are greater than 1), and almost never on the other folds (purple - values are less than 1). (CD8 fold is the most different from LB/NK fold)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F0badcb40e203ddae759f7542c69786d1%2FScreenshot%202023-12-04%20184200.png?generation=1701711745114452&alt=media\")
https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing
\n(Sheet 2 - \"Pyboost&NKfold\").
PS
\nWould you be so kind to share the LB scores - of these models , please
| \n | Py-boost | \nRidge recommender | \nKNN recommender | \nExtraTrees | \n
|---|---|---|---|---|
| Private | \n0.748 | \n0.780 | \n0.775 | \n0.761 | \n
| Public | \n0.571 | \n0.592 | \n0.586 | \n0.573 | \n
Strangely enough - here T-cells CD4+ fold sometimes corresponds to LB:
\n(not the NK-fold)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2Ffa1017a129a9733f48c61f6a7e5cc0c0%2F2022-09-20%20(2).png?generation=1701711822927299&alt=media\")
https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing
\n(Sheet 2 - \"Pyboost&NKfold\").
Let me add another observation and then propose an explanation.
\nObservation: In the following table, which shows the cv scores of my four models, some rows are negatively correlated. For NK cells, ridge is best and Py-boost is worst; for CD4+ cells, ridge is worst and ExtraTrees is best. The negative correlation means that how well the models perform for NK cells has nothing to do with how well they perform for CD4+ cells.
\nBetween private and public leaderboard, I have a correlation of 0.98 (with 57 submissions) like you.
\n| \n | Py-boost | \nRidge recommender | \nKNN recommender | \nExtraTrees | \n
|---|---|---|---|---|
| NK cells | \n1.012 | \n0.940 | \n0.955 | \n0.992 | \n
| T cells CD4+ | \n0.904 | \n0.929 | \n0.928 | \n0.882 | \n
| T cells CD8+ | \n0.780 | \n0.776 | \n0.758 | \n0.758 | \n
| T regulatory cells | \n0.925 | \n0.883 | \n0.881 | \n0.879 | \n
Proposed explanation: We may assume that every cell type has a particular characteristic parameter which affects all compounds, but which our models fail to predict accurately when given only 17 training compounds. This characteristic could be a gene expression pattern, something related to the cell count, or an artefact produced by Limma (note that Limma is run independently for every cell type).
\nAs the model cannot predict this parameter, it guesses the parameter more or less successfully. If the model guesses the parameters for B and Myeloid cells well, private and public leaderboard both show good scores. If the model guesses the parameters badly for B and Myeloid cells, all predictions are biased and private and public leaderboard both have bad scores. But the quality of the guess for B and Myeloid cells has nothing to do with the quality of the guess for NK cells or T regulatory cells.
\nThe question remains: Can we describe this parameter?
", "votes": null, "replies": [ { "id": 2546706, "author_name": "alexandervc", "author_url": "", "post_date": "12/02/2023 18:17:48", "content": "Thanks a lot for sharing !
\nMy feeling is that NK fold is the most close to test cells.
\nHere is a table of LB and CV scores by your CV-scheme for several modifications of the pyboost.
\nThe improvement on LB is reflected by the improvement on the NK fold (green - values are greater than 1), and almost never on the other folds (purple - values are less than 1). (CD8 fold is the most different from LB/NK fold)
https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing
\n(Sheet 2 - \"Pyboost&NKfold\").
PS
\nWould you be so kind to share the LB scores - of these models , please
| \n | Py-boost | \nRidge recommender | \nKNN recommender | \nExtraTrees | \n
|---|---|---|---|---|
| Private | \n0.748 | \n0.780 | \n0.775 | \n0.761 | \n
| Public | \n0.571 | \n0.592 | \n0.586 | \n0.573 | \n
Strangely enough - here T-cells CD4+ fold sometimes corresponds to LB:
\n(not the NK-fold)
https://docs.google.com/spreadsheets/d/1APN63PMaWZygVjYimK9Ivt0RvifdAU5JRYkxiDn4szw/edit?usp=sharing
\n(Sheet 2 - \"Pyboost&NKfold\").