{ "id": 353993, "title": "CV !~ PB : find a correct CV ? or ignore PB ? ", "url": "/competitions/open-problems-multimodal/discussion/353993", "author_name": "Chouchou", "post_date": "2022-09-20T16:51:17.237000", "votes": 4, "comment_count": 22, "views": 0, "content": "

Hello,

\n

Here are my CV scores:
\nCITESEQ 0.89
\nMulti 0.65

\n

However, I can only get 0.806 in PB

\n

According to this discussion CV vs LB Classical Discussion, the weight for cite will be 0.743 and multi 0.257.

\n

I expect to get 0.89x0.743+0.65x0.257 = 0.8283

\n

What is the problem ? Do you know how to align the CV and PB ?
\nOr should I trust my CV because the private dataset is larger?

\n

Thanks

", "messages": [ { "id": 1951079, "postDate": "2022-09-22T18:38:12.890Z", "content": "

Hi @chouchouchen
\nHow do you cross-validate? KFold? GroupKFold(donor)? GroupKFold(day)? The difference can be as high as 0.009.

\n

Other possible explanations are:

\n\n

Combining your CITEseq with a public Multiome or vice-versa might give an idea of where the problem lies.

\n

But if I look at one of my results (CITEseq 0.892, Multiome 0.667, LB 0.811), I see a similar gap. Perhaps we have to accept that the test data is more difficult to predict than the validation data.

\n

An offset between cv and lb scores is nothing unusual in Kaggle competitions. As long as cv improvements correlate with lb improvements, everything is fine.

", "rawMarkdown": "Hi @chouchouchen \nHow do you cross-validate? KFold? GroupKFold(donor)? GroupKFold(day)? The difference can be as high as 0.009.\n\nOther possible explanations are:\n- Different preprocessing for train and test\n- Data leak between train and validation\n- Bug in creating the submission file (all these ugly manipulations with evaluation_ids.csv)\n\nCombining your CITEseq with a public Multiome or vice-versa might give an idea of where the problem lies.\n\nBut if I look at one of my results (CITEseq 0.892, Multiome 0.667, LB 0.811), I see a similar gap. Perhaps we have to accept that the test data is more difficult to predict than the validation data.\n\nAn offset between cv and lb scores is nothing unusual in Kaggle competitions. As long as cv improvements correlate with lb improvements, everything is fine.", "votes": 7, "replies": [ { "id": 1951787, "postDate": "2022-09-23T08:44:15.737Z", "content": "

Thanks a lot AmbrsM.

\n

I cross-validate using KFold.

\n

I tested donor and day features and didn't see any difference in terms of score. So I stopped considering them as important information, which is a wrong idea. I could have used them in my cross-validate strategy, as you said GroupKFold(donor) or GroupKFold(day). It could maybe mitigate the gap between CV and PB scores. I will come back to try.

", "rawMarkdown": "Thanks a lot AmbrsM.\n\nI cross-validate using KFold. \n\nI tested donor and day features and didn't see any difference in terms of score. So I stopped considering them as important information, which is a wrong idea. I could have used them in my cross-validate strategy, as you said GroupKFold(donor) or GroupKFold(day). It could maybe mitigate the gap between CV and PB scores. I will come back to try.", "votes": 1 } ] }, { "id": 1947813, "postDate": "2022-09-20T16:51:17.237Z", "content": "

Hello,

\n

Here are my CV scores:
\nCITESEQ 0.89
\nMulti 0.65

\n

However, I can only get 0.806 in PB

\n

According to this discussion CV vs LB Classical Discussion, the weight for cite will be 0.743 and multi 0.257.

\n

I expect to get 0.89x0.743+0.65x0.257 = 0.8283

\n

What is the problem ? Do you know how to align the CV and PB ?
\nOr should I trust my CV because the private dataset is larger?

\n

Thanks

", "rawMarkdown": "Hello, \n\nHere are my CV scores:\nCITESEQ 0.89 \nMulti 0.65\n\nHowever, I can only get 0.806 in PB\n\nAccording to this discussion [CV vs LB Classical Discussion] (https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349591), the weight for cite will be 0.743 and multi 0.257.\n\nI expect to get 0.89x0.743+0.65x0.257 = 0.8283\n\nWhat is the problem ? Do you know how to align the CV and PB ?\nOr should I trust my CV because the private dataset is larger?\n\nThanks ", "votes": 3 }, { "id": 1979937, "postDate": "2022-10-09T20:27:42.673Z", "content": "

One may consider the following proposal on CV:
\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860

", "rawMarkdown": "One may consider the following proposal on CV:\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860", "votes": 1, "replies": [ { "id": 1980490, "postDate": "2022-10-10T07:42:17.923Z", "content": "

that's interesting

", "rawMarkdown": "that's interesting" } ] }, { "id": 1953227, "postDate": "2022-09-24T10:13:53.340Z", "content": "

From the Data page -
\n\"Your task is to predict the labels corresponding to the inputs in the test set. To facilitate submission scoring, we only require predictions on a subset of the Multiome data. This subset was created by sampling 30% of the Multiome rows, and for each row, 15% of the columns. The sample of columns varies from row-to-row. All of the CITEseq labels are scored.\"

\n

So for Multiome, even if CV shows improvement if the scored rows are not like the set that improved it may not make any difference to LB score.

\n

Like others have mentioned , GroupKFold(donor) does help at least for public LB. But since private LB is an unseen day not sure how that will turn out. For final submissions selections not sure how much public LB will matter.

", "rawMarkdown": "From the Data page - \n\"Your task is to predict the labels corresponding to the inputs in the test set. To facilitate submission scoring, we only require predictions on a subset of the Multiome data. This subset was created by sampling 30% of the Multiome rows, and for each row, 15% of the columns. The sample of columns varies from row-to-row. All of the CITEseq labels are scored.\"\n\nSo for Multiome, even if CV shows improvement if the scored rows are not like the set that improved it may not make any difference to LB score.\n\nLike others have mentioned , GroupKFold(donor) does help at least for public LB. But since private LB is an unseen day not sure how that will turn out. For final submissions selections not sure how much public LB will matter. \n", "votes": 1 }, { "id": 1951706, "postDate": "2022-09-23T07:39:35.530Z", "content": "

Hi @chouchouchen , it is very often the case : the CV score is better than the PB score, even with smart cross validation. The only concern you should have is when your CV score increases and the PB score decreases…. 😉
\nMine : (CITEseq 0.89337, Multiome 0.671, LB 0.812)

", "rawMarkdown": "Hi @chouchouchen , it is very often the case : the CV score is better than the PB score, even with smart cross validation. The only concern you should have is when your CV score increases and the PB score decreases.... 😉\nMine : (CITEseq 0.89337, Multiome 0.671, LB 0.812)", "votes": 2, "replies": [ { "id": 1951789, "postDate": "2022-09-23T08:47:02.063Z", "content": "

Yeah. Thanks for your advise. I should stop being stuck and keep looking for improvement in CV.

", "rawMarkdown": "Yeah. Thanks for your advise. I should stop being stuck and keep looking for improvement in CV. " }, { "id": 1960032, "postDate": "2022-09-28T12:55:02.160Z", "content": "

my cv score increases but my PB result decreases😂,Idon't know why, and this happend manys times

", "rawMarkdown": "my cv score increases but my PB result decreases😂,Idon't know why, and this happend manys times" }, { "id": 1960193, "postDate": "2022-09-28T14:18:59.120Z", "content": "

And I want to ask a question when training: Is it better to have the same training set dimension as the target dimension?,And I tried more features for targets and train datasets(like 256 and 512 or more..) ,it's not as good as your results of Multiome part with 128 features and targets.https://www.kaggle.com/code/pourchot/multiome-with-keras-ensemble

", "rawMarkdown": "And I want to ask a question when training: Is it better to have the same training set dimension as the target dimension?,And I tried more features for targets and train datasets(like 256 and 512 or more..) ,it's not as good as your results of Multiome part with 128 features and targets.[https://www.kaggle.com/code/pourchot/multiome-with-keras-ensemble](url)" }, { "id": 1962659, "postDate": "2022-09-29T20:29:26.423Z", "content": "

Hi @kunmingxie , I’m afraid there is no rules, such dimensions are so unusual…

", "rawMarkdown": "Hi @kunmingxie , I’m afraid there is no rules, such dimensions are so unusual…" }, { "id": 1968672, "postDate": "2022-10-03T07:04:27.240Z", "content": "
\n

my cv score increases but my PB result decreases

\n
\n

it seems to be model overfitting on train data or something like that

", "rawMarkdown": "> my cv score increases but my PB result decreases\n\nit seems to be model overfitting on train data or something like that" } ] }, { "id": 1951408, "postDate": "2022-09-23T03:52:29.127Z", "content": "

You may check AmbrosM's notebooks, the GroupKFold(donor) really helps to lower the over-fitting in the cv. Adding this decreases the gap between my cv and lb (it lowers the cv and increases the lb, this makes). It may also help you get a more reasonable cv score.

", "rawMarkdown": "You may check AmbrosM's notebooks, the GroupKFold(donor) really helps to lower the over-fitting in the cv. Adding this decreases the gap between my cv and lb (it lowers the cv and increases the lb, this makes). It may also help you get a more reasonable cv score. ", "votes": 2, "replies": [ { "id": 1951790, "postDate": "2022-09-23T08:47:20.353Z", "content": "

Thank you. Will try this.

", "rawMarkdown": "Thank you. Will try this. " }, { "id": 1956432, "postDate": "2022-09-26T13:04:10.127Z", "content": "

I tried the GroupKFold, and it lowered both the cv and the lb. It doesn't seem to work on my model.

", "rawMarkdown": "I tried the GroupKFold, and it lowered both the cv and the lb. It doesn't seem to work on my model.", "votes": 1 }, { "id": 1956455, "postDate": "2022-09-26T13:13:41.257Z", "content": "

I think it's normal that it lowers the CV, because having the same donor both in train and validation should be easier to predict. However, it's interesting that it leads to better LB score too..

\n

Did you average the predictions over the 3 donors/folds, when using GroupKFold?

", "rawMarkdown": "I think it's normal that it lowers the CV, because having the same donor both in train and validation should be easier to predict. However, it's interesting that it leads to better LB score too.. \n\nDid you average the predictions over the 3 donors/folds, when using GroupKFold?", "votes": 1 }, { "id": 1977494, "postDate": "2022-10-08T04:26:58.400Z", "content": "

Yeah, I average them. It is quiet strange, since many comments said this lower the PB, but PB increases in my side … I didn't figure out why.

", "rawMarkdown": "Yeah, I average them. It is quiet strange, since many comments said this lower the PB, but PB increases in my side ... I didn't figure out why." } ] }, { "id": 1949932, "postDate": "2022-09-22T02:08:38.350Z", "content": "

Scroll down a bit more in the discussion you linked to get the updated weighting after the test set data fix

\n

\"So the new weights are 0.712 for cite and 0.288 for multi.\"

", "rawMarkdown": "Scroll down a bit more in the discussion you linked to get the updated weighting after the test set data fix\n\n\"So the new weights are 0.712 for cite and 0.288 for multi.\"", "votes": -1, "replies": [ { "id": 1950350, "postDate": "2022-09-22T09:00:11.643Z", "content": "

I use these weights too. But i can get 0.82088
\n0.890.712+0.650.288 = 0.82088

\n

There is gap with my PB score 0.806

", "rawMarkdown": "I use these weights too. But i can get 0.82088\n0.89*0.712+0.65*0.288 = 0.82088\n\nThere is gap with my PB score 0.806" } ] }, { "id": 1961823, "postDate": "2022-09-29T11:27:26.410Z", "content": "

So I see your PB score is better than 0.806,congratuation!May I ask you what do you do to improve score? Are you use GroupKFold?

", "rawMarkdown": "So I see your PB score is better than 0.806,congratuation!May I ask you what do you do to improve score? Are you use GroupKFold?", "replies": [ { "id": 1963354, "postDate": "2022-09-30T08:27:25.440Z", "content": "

GroupKFold doesn't help too much form my side.

", "rawMarkdown": "GroupKFold doesn't help too much form my side. ", "votes": 1 } ] }, { "id": 1979068, "postDate": "2022-10-09T06:18:28.850Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1978942, "postDate": "2022-10-09T04:37:14.463Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1951079, "author_name": "AmbrosM", "author_url": "", "post_date": "2022-09-22T18:38:12.890000", "content": "

Hi @chouchouchen
\nHow do you cross-validate? KFold? GroupKFold(donor)? GroupKFold(day)? The difference can be as high as 0.009.

\n

Other possible explanations are:

\n\n

Combining your CITEseq with a public Multiome or vice-versa might give an idea of where the problem lies.

\n

But if I look at one of my results (CITEseq 0.892, Multiome 0.667, LB 0.811), I see a similar gap. Perhaps we have to accept that the test data is more difficult to predict than the validation data.

\n

An offset between cv and lb scores is nothing unusual in Kaggle competitions. As long as cv improvements correlate with lb improvements, everything is fine.

", "votes": 7, "replies": [ { "id": 1951787, "author_name": "Chouchou", "author_url": "", "post_date": "2022-09-23T08:44:15.737000", "content": "

Thanks a lot AmbrsM.

\n

I cross-validate using KFold.

\n

I tested donor and day features and didn't see any difference in terms of score. So I stopped considering them as important information, which is a wrong idea. I could have used them in my cross-validate strategy, as you said GroupKFold(donor) or GroupKFold(day). It could maybe mitigate the gap between CV and PB scores. I will come back to try.

", "votes": 1, "replies": [] } ] }, { "id": 1979937, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2022-10-09T20:27:42.673000", "content": "

One may consider the following proposal on CV:
\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860

", "votes": 1, "replies": [ { "id": 1980490, "author_name": "Chouchou", "author_url": "", "post_date": "2022-10-10T07:42:17.923000", "content": "

that's interesting

", "votes": 0, "replies": [] } ] }, { "id": 1953227, "author_name": "something4kag", "author_url": "", "post_date": "2022-09-24T10:13:53.340000", "content": "

From the Data page -
\n\"Your task is to predict the labels corresponding to the inputs in the test set. To facilitate submission scoring, we only require predictions on a subset of the Multiome data. This subset was created by sampling 30% of the Multiome rows, and for each row, 15% of the columns. The sample of columns varies from row-to-row. All of the CITEseq labels are scored.\"

\n

So for Multiome, even if CV shows improvement if the scored rows are not like the set that improved it may not make any difference to LB score.

\n

Like others have mentioned , GroupKFold(donor) does help at least for public LB. But since private LB is an unseen day not sure how that will turn out. For final submissions selections not sure how much public LB will matter.

", "votes": 1, "replies": [] }, { "id": 1951706, "author_name": "Laurent Pourchot", "author_url": "", "post_date": "2022-09-23T07:39:35.530000", "content": "

Hi @chouchouchen , it is very often the case : the CV score is better than the PB score, even with smart cross validation. The only concern you should have is when your CV score increases and the PB score decreases…. 😉
\nMine : (CITEseq 0.89337, Multiome 0.671, LB 0.812)

", "votes": 2, "replies": [ { "id": 1951789, "author_name": "Chouchou", "author_url": "", "post_date": "2022-09-23T08:47:02.063000", "content": "

Yeah. Thanks for your advise. I should stop being stuck and keep looking for improvement in CV.

", "votes": 0, "replies": [] }, { "id": 1960032, "author_name": "Kunming Xie", "author_url": "", "post_date": "2022-09-28T12:55:02.160000", "content": "

my cv score increases but my PB result decreases😂,Idon't know why, and this happend manys times

", "votes": 0, "replies": [] }, { "id": 1960193, "author_name": "Kunming Xie", "author_url": "", "post_date": "2022-09-28T14:18:59.120000", "content": "

And I want to ask a question when training: Is it better to have the same training set dimension as the target dimension?,And I tried more features for targets and train datasets(like 256 and 512 or more..) ,it's not as good as your results of Multiome part with 128 features and targets.https://www.kaggle.com/code/pourchot/multiome-with-keras-ensemble

", "votes": 0, "replies": [] }, { "id": 1962659, "author_name": "Laurent Pourchot", "author_url": "", "post_date": "2022-09-29T20:29:26.423000", "content": "

Hi @kunmingxie , I’m afraid there is no rules, such dimensions are so unusual…

", "votes": 0, "replies": [] }, { "id": 1968672, "author_name": "Oleg Khudyakov", "author_url": "", "post_date": "2022-10-03T07:04:27.240000", "content": "
\n

my cv score increases but my PB result decreases

\n
\n

it seems to be model overfitting on train data or something like that

", "votes": 0, "replies": [] } ] }, { "id": 1951408, "author_name": "no-magic", "author_url": "", "post_date": "2022-09-23T03:52:29.127000", "content": "

You may check AmbrosM's notebooks, the GroupKFold(donor) really helps to lower the over-fitting in the cv. Adding this decreases the gap between my cv and lb (it lowers the cv and increases the lb, this makes). It may also help you get a more reasonable cv score.

", "votes": 2, "replies": [ { "id": 1951790, "author_name": "Chouchou", "author_url": "", "post_date": "2022-09-23T08:47:20.353000", "content": "

Thank you. Will try this.

", "votes": 0, "replies": [] }, { "id": 1956432, "author_name": "A Beginner", "author_url": "", "post_date": "2022-09-26T13:04:10.127000", "content": "

I tried the GroupKFold, and it lowered both the cv and the lb. It doesn't seem to work on my model.

", "votes": 1, "replies": [] }, { "id": 1956455, "author_name": "pinouche", "author_url": "", "post_date": "2022-09-26T13:13:41.257000", "content": "

I think it's normal that it lowers the CV, because having the same donor both in train and validation should be easier to predict. However, it's interesting that it leads to better LB score too..

\n

Did you average the predictions over the 3 donors/folds, when using GroupKFold?

", "votes": 1, "replies": [] }, { "id": 1977494, "author_name": "no-magic", "author_url": "", "post_date": "2022-10-08T04:26:58.400000", "content": "

Yeah, I average them. It is quiet strange, since many comments said this lower the PB, but PB increases in my side … I didn't figure out why.

", "votes": 0, "replies": [] } ] }, { "id": 1949932, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2022-09-22T02:08:38.350000", "content": "

Scroll down a bit more in the discussion you linked to get the updated weighting after the test set data fix

\n

\"So the new weights are 0.712 for cite and 0.288 for multi.\"

", "votes": -1, "replies": [ { "id": 1950350, "author_name": "Chouchou", "author_url": "", "post_date": "2022-09-22T09:00:11.643000", "content": "

I use these weights too. But i can get 0.82088
\n0.890.712+0.650.288 = 0.82088

\n

There is gap with my PB score 0.806

", "votes": 0, "replies": [] } ] }, { "id": 1961823, "author_name": "ZhangYuan2000", "author_url": "", "post_date": "2022-09-29T11:27:26.410000", "content": "

So I see your PB score is better than 0.806,congratuation!May I ask you what do you do to improve score? Are you use GroupKFold?

", "votes": 0, "replies": [ { "id": 1963354, "author_name": "Chouchou", "author_url": "", "post_date": "2022-09-30T08:27:25.440000", "content": "

GroupKFold doesn't help too much form my side.

", "votes": 1, "replies": [] } ] }, { "id": 1979068, "author_name": "", "author_url": "", "post_date": "2022-10-09T06:18:28.850000", "content": "", "votes": 0, "replies": [] }, { "id": 1978942, "author_name": "", "author_url": "", "post_date": "2022-10-09T04:37:14.463000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1951079": "Hi @chouchouchen \nHow do you cross-validate? KFold? GroupKFold(donor)? GroupKFold(day)? The difference can be as high as 0.009.\n\nOther possible explanations are:\n- Different preprocessing for train and test\n- Data leak between train and validation\n- Bug in creating the submission file (all these ugly manipulations with evaluation_ids.csv)\n\nCombining your CITEseq with a public Multiome or vice-versa might give an idea of where the problem lies.\n\nBut if I look at one of my results (CITEseq 0.892, Multiome 0.667, LB 0.811), I see a similar gap. Perhaps we have to accept that the test data is more difficult to predict than the validation data.\n\nAn offset between cv and lb scores is nothing unusual in Kaggle competitions. As long as cv improvements correlate with lb improvements, everything is fine.", "1947813": "Hello, \n\nHere are my CV scores:\nCITESEQ 0.89 \nMulti 0.65\n\nHowever, I can only get 0.806 in PB\n\nAccording to this discussion [CV vs LB Classical Discussion] (https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349591), the weight for cite will be 0.743 and multi 0.257.\n\nI expect to get 0.89x0.743+0.65x0.257 = 0.8283\n\nWhat is the problem ? Do you know how to align the CV and PB ?\nOr should I trust my CV because the private dataset is larger?\n\nThanks ", "1979937": "One may consider the following proposal on CV:\nhttps://www.kaggle.com/competitions/open-problems-multimodal/discussion/358860", "1953227": "From the Data page - \n\"Your task is to predict the labels corresponding to the inputs in the test set. To facilitate submission scoring, we only require predictions on a subset of the Multiome data. This subset was created by sampling 30% of the Multiome rows, and for each row, 15% of the columns. The sample of columns varies from row-to-row. All of the CITEseq labels are scored.\"\n\nSo for Multiome, even if CV shows improvement if the scored rows are not like the set that improved it may not make any difference to LB score.\n\nLike others have mentioned , GroupKFold(donor) does help at least for public LB. But since private LB is an unseen day not sure how that will turn out. For final submissions selections not sure how much public LB will matter. \n", "1951706": "Hi @chouchouchen , it is very often the case : the CV score is better than the PB score, even with smart cross validation. The only concern you should have is when your CV score increases and the PB score decreases.... 😉\nMine : (CITEseq 0.89337, Multiome 0.671, LB 0.812)", "1951408": "You may check AmbrosM's notebooks, the GroupKFold(donor) really helps to lower the over-fitting in the cv. Adding this decreases the gap between my cv and lb (it lowers the cv and increases the lb, this makes). It may also help you get a more reasonable cv score. ", "1949932": "Scroll down a bit more in the discussion you linked to get the updated weighting after the test set data fix\n\n\"So the new weights are 0.712 for cite and 0.288 for multi.\"", "1961823": "So I see your PB score is better than 0.806,congratuation!May I ask you what do you do to improve score? Are you use GroupKFold?", "1979068": "", "1978942": "" } }