{ "id": 349591, "title": "CV vs LB Classical Discussion", "url": "/competitions/open-problems-multimodal/discussion/349591", "author_name": "Martin Kovacevic Buvinic", "post_date": "2022-09-01T22:45:24.715000", "votes": 32, "comment_count": 22, "views": 0, "content": "

Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). Multi just use 30% of the rows, meaning it actually has 16780 cells.

\n

Therefore the weight for cite will be 0.743 and multi 0.257

\n

In my case my out of folds CV for cite is 0.8882 and for multi is 0.6601. If I use the weights to sum my out of folds score I actually score 0.829 which is actually the same score of my public lb

\n

This is using 5KFold for cite and multi:
\nCV: 0.829
\nLB: 0.829

\n

This is using 3 GroupKFold for cite, 4 GroupKFold for multi using day column.
\nCV: 0.8273
\nLB: 0.829

\n

Please share your results here (If you want 😄)

", "messages": [ { "id": 1922997, "postDate": "2022-09-01T22:45:24.717Z", "content": "

Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). Multi just use 30% of the rows, meaning it actually has 16780 cells.

\n

Therefore the weight for cite will be 0.743 and multi 0.257

\n

In my case my out of folds CV for cite is 0.8882 and for multi is 0.6601. If I use the weights to sum my out of folds score I actually score 0.829 which is actually the same score of my public lb

\n

This is using 5KFold for cite and multi:
\nCV: 0.829
\nLB: 0.829

\n

This is using 3 GroupKFold for cite, 4 GroupKFold for multi using day column.
\nCV: 0.8273
\nLB: 0.829

\n

Please share your results here (If you want 😄)

", "rawMarkdown": "Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). Multi just use 30% of the rows, meaning it actually has 16780 cells.\n\nTherefore the weight for cite will be 0.743 and multi 0.257\n\nIn my case my out of folds CV for cite is 0.8882 and for multi is 0.6601. If I use the weights to sum my out of folds score I actually score 0.829 which is actually the same score of my public lb\n\nThis is using 5KFold for cite and multi:\nCV: 0.829\nLB: 0.829\n\nThis is using 3 GroupKFold for cite, 4 GroupKFold for multi using day column.\nCV: 0.8273\nLB: 0.829\n\nPlease share your results here (If you want 😄)", "votes": 31 }, { "id": 1932748, "postDate": "2022-09-09T21:59:55.870Z", "content": "

Just worked on cite for now .. will keep updating here

\n

Multinome 0.67~ Ensemble

\n\n

Cite : ensemble 0.905

\n", "rawMarkdown": "Just worked on cite for now .. *will keep updating here*\n\n**Multinome** 0.67~ Ensemble \n- Approach 1 NN: 0.6682\n\n**Cite** : ensemble 0.905\n- Approach 1 NN : 0.88992\n- Approach 2 NN: 0.8926066667\n- Approach 3 LGBM: 0.8844 \n", "votes": 3, "replies": [ { "id": 1988863, "postDate": "2022-10-15T15:33:25.273Z", "content": "

may I ask what validation strategy(fold split) are you using?

", "rawMarkdown": "may I ask what validation strategy(fold split) are you using?", "votes": 1 }, { "id": 1988994, "postDate": "2022-10-15T17:00:57.270Z", "content": "

Hi.. By donor for now but thinking of trying few more ideas so the networks can learn but more as scores bit stuck at one point 😄

", "rawMarkdown": "Hi.. By donor for now but thinking of trying few more ideas so the networks can learn but more as scores bit stuck at one point 😄", "votes": 2 } ] }, { "id": 1924699, "postDate": "2022-09-03T10:01:42.127Z", "content": "

CV: 0.905
\nLB: 0.848

\n

I think I am overfitting :(

", "rawMarkdown": "CV: 0.905\nLB: 0.848\n\nI think I am overfitting :(", "votes": 3, "replies": [ { "id": 1924864, "postDate": "2022-09-03T13:23:35.560Z", "content": "

Are you using day / patient as a feature ?

", "rawMarkdown": "Are you using day / patient as a feature ?" }, { "id": 1925219, "postDate": "2022-09-03T18:18:02.880Z", "content": "

No, I don't use the metadata for anything.

", "rawMarkdown": "No, I don't use the metadata for anything.", "votes": 2 } ] }, { "id": 1971305, "postDate": "2022-10-04T14:17:14.347Z", "content": "

Hi @chouchouchen , As AmbrosM explained me, after The data update of 2022-09-10 the weights changed a bit. They are now 0.7105 and 0.2895 . There is still a difference between the CV and LB though.

", "rawMarkdown": "Hi @chouchouchen , As AmbrosM explained me, after [The data update of 2022-09-10](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/350933) the weights changed a bit.[ They are now 0.7105 and 0.2895 ](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/353523). There is still a difference between the CV and LB though.", "votes": 1 }, { "id": 1926845, "postDate": "2022-09-05T07:18:58.880Z", "content": "

Greetings, Martin!

\n

If we look at this notebook (Submission part) we will see the text:
\nThe CITEseq test predictions have 48663 rows (i.e., cells) and 140 columns (i.e. proteins). 48663 * 140 = 6812820. The final submission will have 65,744,180 rows, of which the first 6,812,820 are for the CITEseq predictions and the remaining 58,931,360 for the Multiome predictions.

\n

And if we divide number of CITESEQ and MULTIOME rows in submission file by total submission length we will get 0.104 for CITESEQ and 0.896 for MULTIOME.

\n

Are You sure about 0.743 for SITE and 0.257 for MULTI?

", "rawMarkdown": "Greetings, Martin!\n\nIf we look [at this notebook (Submission part)](http://www.kaggle.com/code/ambrosm/msci-citeseq-quickstart#Submission) we will see the text:\nThe CITEseq test predictions have 48663 rows (i.e., cells) and 140 columns (i.e. proteins). 48663 * 140 = 6812820. **The final submission will have 65,744,180 rows, of which the first 6,812,820 are for the CITEseq predictions and the remaining 58,931,360 for the Multiome predictions.**\n\nAnd if we divide number of CITESEQ and MULTIOME rows in submission file by total submission length we will get 0.104 for CITESEQ and 0.896 for MULTIOME.\n\nAre You sure about 0.743 for SITE and 0.257 for MULTI?", "votes": 2, "replies": [ { "id": 1927280, "postDate": "2022-09-05T14:01:48.923Z", "content": "

The 6,812,820 rows of cite has more cell ids compared to the 58,931,360 rows for multi. Correlation score is computed per cell_id, the correct way to compute is to compare the amount of cell ids not rows.

", "rawMarkdown": "The 6,812,820 rows of cite has more cell ids compared to the 58,931,360 rows for multi. Correlation score is computed per cell_id, the correct way to compute is to compare the amount of cell ids not rows.", "votes": 5 }, { "id": 1927318, "postDate": "2022-09-05T14:46:57.067Z", "content": "

thank you!

", "rawMarkdown": "thank you!", "votes": 1 } ] }, { "id": 1989173, "postDate": "2022-10-15T18:48:36.357Z", "content": "

i am not sure Kfold is doing data leakage for me how exactly are you applying kfold?

", "rawMarkdown": "i am not sure Kfold is doing data leakage for me how exactly are you applying kfold?\n" }, { "id": 1972361, "postDate": "2022-10-05T05:48:53.670Z", "content": "

CITEseq:0.899(group by donor)
\nMultiome:0.672(Kfold) 0.670(group by donor)

\n

And I'm submitting one technique as a sample submission to check the score.
\nCITEseq:0.250
\nMultiome:-0.437
\nPublic:0.812

\n

if it's not my calculation error, I think the real weight of the techniques in the public is slightly off.

", "rawMarkdown": "CITEseq:0.899(group by donor)\nMultiome:0.672(Kfold) 0.670(group by donor)\n\nAnd I'm submitting one technique as a sample submission to check the score.\nCITEseq:0.250\nMultiome:-0.437\nPublic:0.812\n\nif it's not my calculation error, I think the real weight of the techniques in the public is slightly off.", "replies": [ { "id": 1973234, "postDate": "2022-10-05T14:36:54.910Z", "content": "

May I ask is ur cite an ensemble or SIngle Model MLP

", "rawMarkdown": "May I ask is ur cite an ensemble or SIngle Model MLP" }, { "id": 1974074, "postDate": "2022-10-06T04:13:25.583Z", "content": "

All these are for single models, but publicLB0.250/-0.437/0.812 are best even with ensembles.

", "rawMarkdown": "All these are for single models, but publicLB0.250/-0.437/0.812 are best even with ensembles." } ] }, { "id": 1930943, "postDate": "2022-09-08T11:29:49.487Z", "content": "

What's the updated CV/LB scores after the re-scoring?

\n

And wouldn't donorwise split make more sense to guage the correlation between Public Test Score and CV. Since the daywise split makes sense for the Private test, which we do not get.

\n

I'm planning to make a thorough investigation of different splits and CV correlations once I finalized my training and submission pipeline. So far not getting good enough validation scores to even consider submitting 😔

", "rawMarkdown": "What's the updated CV/LB scores after the re-scoring?\n\nAnd wouldn't donorwise split make more sense to guage the correlation between Public Test Score and CV. Since the daywise split makes sense for the Private test, which we do not get. \n\nI'm planning to make a thorough investigation of different splits and CV correlations once I finalized my training and submission pipeline. So far not getting good enough validation scores to even consider submitting 😔" }, { "id": 1949929, "postDate": "2022-09-22T02:06:48.813Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1930866, "postDate": "2022-09-08T09:24:04.307Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1930904, "postDate": "2022-09-08T10:04:00.420Z", "content": "

Previously, there were 65443 test rows (48663 for cite, 16780 for multi). Now, 7016 test rows from cite were removed from scoring (they are still present in the submission file but their score doesn't count). So now we have 41647 rows for cite and still 16780 rows for multi for a total of 58427 rows.
\nSo the new weights are 0.712 for cite and 0.288 for multi.
\nHope this helps :)

", "rawMarkdown": "Previously, there were 65443 test rows (48663 for cite, 16780 for multi). Now, 7016 test rows from cite were removed from scoring (they are still present in the submission file but their score doesn't count). So now we have 41647 rows for cite and still 16780 rows for multi for a total of 58427 rows.\nSo the new weights are 0.712 for cite and 0.288 for multi.\nHope this helps :)", "votes": 5 }, { "id": 1930939, "postDate": "2022-09-08T11:20:52.550Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1931572, "postDate": "2022-09-08T19:36:45.450Z", "content": "

Correct, but those new weights are just for public leaderboard, for private they are the same as they where before

", "rawMarkdown": "Correct, but those new weights are just for public leaderboard, for private they are the same as they where before", "votes": 1 }, { "id": 1931599, "postDate": "2022-09-08T20:47:19.910Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1931646, "postDate": "2022-09-08T22:19:05.930Z", "content": "

Well, the private test set is hidden so we don't know the weights. But, in general, it is almost the same distribution as the public test set. But that's not by any mean a rule.

", "rawMarkdown": "Well, the private test set is hidden so we don't know the weights. But, in general, it is almost the same distribution as the public test set. But that's not by any mean a rule.", "votes": 2 } ] } ], "comments": [ { "id": 1932748, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-09-09T21:59:55.870000", "content": "

Just worked on cite for now .. will keep updating here

\n

Multinome 0.67~ Ensemble

\n\n

Cite : ensemble 0.905

\n", "votes": 3, "replies": [ { "id": 1988863, "author_name": "Priyanshu Chaudhary", "author_url": "", "post_date": "2022-10-15T15:33:25.273000", "content": "

may I ask what validation strategy(fold split) are you using?

", "votes": 1, "replies": [] }, { "id": 1988994, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-10-15T17:00:57.270000", "content": "

Hi.. By donor for now but thinking of trying few more ideas so the networks can learn but more as scores bit stuck at one point 😄

", "votes": 2, "replies": [] } ] }, { "id": 1924699, "author_name": "Karim Fadi", "author_url": "", "post_date": "2022-09-03T10:01:42.127000", "content": "

CV: 0.905
\nLB: 0.848

\n

I think I am overfitting :(

", "votes": 3, "replies": [ { "id": 1924864, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-09-03T13:23:35.560000", "content": "

Are you using day / patient as a feature ?

", "votes": 0, "replies": [] }, { "id": 1925219, "author_name": "Karim Fadi", "author_url": "", "post_date": "2022-09-03T18:18:02.880000", "content": "

No, I don't use the metadata for anything.

", "votes": 2, "replies": [] } ] }, { "id": 1971305, "author_name": "Elias", "author_url": "", "post_date": "2022-10-04T14:17:14.347000", "content": "

Hi @chouchouchen , As AmbrosM explained me, after The data update of 2022-09-10 the weights changed a bit. They are now 0.7105 and 0.2895 . There is still a difference between the CV and LB though.

", "votes": 1, "replies": [] }, { "id": 1926845, "author_name": "Oleg Khudyakov", "author_url": "", "post_date": "2022-09-05T07:18:58.880000", "content": "

Greetings, Martin!

\n

If we look at this notebook (Submission part) we will see the text:
\nThe CITEseq test predictions have 48663 rows (i.e., cells) and 140 columns (i.e. proteins). 48663 * 140 = 6812820. The final submission will have 65,744,180 rows, of which the first 6,812,820 are for the CITEseq predictions and the remaining 58,931,360 for the Multiome predictions.

\n

And if we divide number of CITESEQ and MULTIOME rows in submission file by total submission length we will get 0.104 for CITESEQ and 0.896 for MULTIOME.

\n

Are You sure about 0.743 for SITE and 0.257 for MULTI?

", "votes": 2, "replies": [ { "id": 1927280, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2022-09-05T14:01:48.923000", "content": "

The 6,812,820 rows of cite has more cell ids compared to the 58,931,360 rows for multi. Correlation score is computed per cell_id, the correct way to compute is to compare the amount of cell ids not rows.

", "votes": 5, "replies": [] }, { "id": 1927318, "author_name": "Oleg Khudyakov", "author_url": "", "post_date": "2022-09-05T14:46:57.067000", "content": "

thank you!

", "votes": 1, "replies": [] } ] }, { "id": 1989173, "author_name": "Priyanshu Chaudhary", "author_url": "", "post_date": "2022-10-15T18:48:36.357000", "content": "

i am not sure Kfold is doing data leakage for me how exactly are you applying kfold?

", "votes": 0, "replies": [] }, { "id": 1972361, "author_name": "museas", "author_url": "", "post_date": "2022-10-05T05:48:53.670000", "content": "

CITEseq:0.899(group by donor)
\nMultiome:0.672(Kfold) 0.670(group by donor)

\n

And I'm submitting one technique as a sample submission to check the score.
\nCITEseq:0.250
\nMultiome:-0.437
\nPublic:0.812

\n

if it's not my calculation error, I think the real weight of the techniques in the public is slightly off.

", "votes": 0, "replies": [ { "id": 1973234, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-10-05T14:36:54.910000", "content": "

May I ask is ur cite an ensemble or SIngle Model MLP

", "votes": 0, "replies": [] }, { "id": 1974074, "author_name": "museas", "author_url": "", "post_date": "2022-10-06T04:13:25.583000", "content": "

All these are for single models, but publicLB0.250/-0.437/0.812 are best even with ensembles.

", "votes": 0, "replies": [] } ] }, { "id": 1930943, "author_name": "AbdurRafae", "author_url": "", "post_date": "2022-09-08T11:29:49.487000", "content": "

What's the updated CV/LB scores after the re-scoring?

\n

And wouldn't donorwise split make more sense to guage the correlation between Public Test Score and CV. Since the daywise split makes sense for the Private test, which we do not get.

\n

I'm planning to make a thorough investigation of different splits and CV correlations once I finalized my training and submission pipeline. So far not getting good enough validation scores to even consider submitting 😔

", "votes": 0, "replies": [] }, { "id": 1949929, "author_name": "", "author_url": "", "post_date": "2022-09-22T02:06:48.813000", "content": "", "votes": 0, "replies": [] }, { "id": 1930866, "author_name": "", "author_url": "", "post_date": "2022-09-08T09:24:04.307000", "content": "", "votes": 0, "replies": [ { "id": 1930904, "author_name": "Karim Fadi", "author_url": "", "post_date": "2022-09-08T10:04:00.420000", "content": "

Previously, there were 65443 test rows (48663 for cite, 16780 for multi). Now, 7016 test rows from cite were removed from scoring (they are still present in the submission file but their score doesn't count). So now we have 41647 rows for cite and still 16780 rows for multi for a total of 58427 rows.
\nSo the new weights are 0.712 for cite and 0.288 for multi.
\nHope this helps :)

", "votes": 5, "replies": [] }, { "id": 1930939, "author_name": "", "author_url": "", "post_date": "2022-09-08T11:20:52.550000", "content": "", "votes": 1, "replies": [] }, { "id": 1931572, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2022-09-08T19:36:45.450000", "content": "

Correct, but those new weights are just for public leaderboard, for private they are the same as they where before

", "votes": 1, "replies": [] }, { "id": 1931599, "author_name": "", "author_url": "", "post_date": "2022-09-08T20:47:19.910000", "content": "", "votes": 0, "replies": [] }, { "id": 1931646, "author_name": "Karim Fadi", "author_url": "", "post_date": "2022-09-08T22:19:05.930000", "content": "

Well, the private test set is hidden so we don't know the weights. But, in general, it is almost the same distribution as the public test set. But that's not by any mean a rule.

", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1922997": "Cite has 48663 test rows, and multi has 55935 rows (rows = amount of cells). Multi just use 30% of the rows, meaning it actually has 16780 cells.\n\nTherefore the weight for cite will be 0.743 and multi 0.257\n\nIn my case my out of folds CV for cite is 0.8882 and for multi is 0.6601. If I use the weights to sum my out of folds score I actually score 0.829 which is actually the same score of my public lb\n\nThis is using 5KFold for cite and multi:\nCV: 0.829\nLB: 0.829\n\nThis is using 3 GroupKFold for cite, 4 GroupKFold for multi using day column.\nCV: 0.8273\nLB: 0.829\n\nPlease share your results here (If you want 😄)", "1932748": "Just worked on cite for now .. *will keep updating here*\n\n**Multinome** 0.67~ Ensemble \n- Approach 1 NN: 0.6682\n\n**Cite** : ensemble 0.905\n- Approach 1 NN : 0.88992\n- Approach 2 NN: 0.8926066667\n- Approach 3 LGBM: 0.8844 \n", "1924699": "CV: 0.905\nLB: 0.848\n\nI think I am overfitting :(", "1971305": "Hi @chouchouchen , As AmbrosM explained me, after [The data update of 2022-09-10](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/350933) the weights changed a bit.[ They are now 0.7105 and 0.2895 ](https://www.kaggle.com/competitions/open-problems-multimodal/discussion/353523). There is still a difference between the CV and LB though.", "1926845": "Greetings, Martin!\n\nIf we look [at this notebook (Submission part)](http://www.kaggle.com/code/ambrosm/msci-citeseq-quickstart#Submission) we will see the text:\nThe CITEseq test predictions have 48663 rows (i.e., cells) and 140 columns (i.e. proteins). 48663 * 140 = 6812820. **The final submission will have 65,744,180 rows, of which the first 6,812,820 are for the CITEseq predictions and the remaining 58,931,360 for the Multiome predictions.**\n\nAnd if we divide number of CITESEQ and MULTIOME rows in submission file by total submission length we will get 0.104 for CITESEQ and 0.896 for MULTIOME.\n\nAre You sure about 0.743 for SITE and 0.257 for MULTI?", "1989173": "i am not sure Kfold is doing data leakage for me how exactly are you applying kfold?\n", "1972361": "CITEseq:0.899(group by donor)\nMultiome:0.672(Kfold) 0.670(group by donor)\n\nAnd I'm submitting one technique as a sample submission to check the score.\nCITEseq:0.250\nMultiome:-0.437\nPublic:0.812\n\nif it's not my calculation error, I think the real weight of the techniques in the public is slightly off.", "1930943": "What's the updated CV/LB scores after the re-scoring?\n\nAnd wouldn't donorwise split make more sense to guage the correlation between Public Test Score and CV. Since the daywise split makes sense for the Private test, which we do not get. \n\nI'm planning to make a thorough investigation of different splits and CV correlations once I finalized my training and submission pipeline. So far not getting good enough validation scores to even consider submitting 😔", "1949929": "", "1930866": "" } }