For my experiment, I have settled with 13M dataset with about 10M train_set and 2M test_set.
\nI have got a matrix AUC score by test_set 0.7683 .
\nHowever, my submissions score (public score) is just 0.690. Who have met such situation before?
I guess some of you might have had such puzzle.
\nSolved or not, could you please share your analysis or experience in the comment
", "messages": [ { "id": "1064563", "postDate": "10/30/2020 08:54:43", "content": "For my experiment, I have settled with 13M dataset with about 10M train_set and 2M test_set.
\nI have got a matrix AUC score by test_set 0.7683 .
\nHowever, my submissions score (public score) is just 0.690. Who have met such situation before?
I guess some of you might have had such puzzle.
\nSolved or not, could you please share your analysis or experience in the comment
", "rawMarkdown": "For my experiment, I have settled with 13M dataset with about 10M train_set and 2M test_set.\n\nI have got a matrix AUC score by test_set **0.7683** . \nHowever, my submissions score (public score) is just **0.690**. Who have met such situation before? \n\n**I guess some of you might have had such puzzle.**\n\nSolved or not, could you please share your analysis or experience in the comment", "votes": null }, { "id": "1064585", "postDate": "10/30/2020 09:36:30", "content": "Almost everyone who shared their CV-LB scores here have reported very stable correlation and a small absolute difference. In some cases LB is a bit higher than CV.
\nLocal 0.768 vs LB 0.690 implies something is definitely off. I would agree with @aquatic's comment that it is most likely due to some difference in the feature preparation for local test vs private test. Maybe use the public test iterator to check / verify the feature values but beyond that it is up to you to investigate your pipeline in detail.
", "rawMarkdown": "Almost everyone who shared their CV-LB scores [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192919) have reported very stable correlation and a small absolute difference. In some cases LB is a bit higher than CV.\n\nLocal **0.768** vs LB **0.690** implies something is definitely off. I would agree with @aquatic's [comment](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192919#1063896) that it is most likely due to some difference in the feature preparation for local test vs private test. Maybe use the public test iterator to check / verify the feature values but beyond that it is up to you to investigate your pipeline in detail.", "votes": null }, { "id": "1069066", "postDate": "11/04/2020 03:07:27", "content": "Pretty good, thanks. I have just found some outliers null value in test cases which might cause a low performance in public testing.
", "rawMarkdown": "Pretty good, thanks. I have just found some outliers null value in test cases which might cause a low performance in public testing.", "votes": null }, { "id": "1069070", "postDate": "11/04/2020 03:16:59", "content": "Btw, sir, I want to ask you an additional question that could I divide one dataset into several pieces and predict them in different model then combining the result, that is, in some columns there are nulls in part of features, and I could not find an apropos substitution. Thus, I want to ignore such features while predicting the relevant columns. Is this idea legal？
", "rawMarkdown": "Btw, sir, I want to ask you an additional question that could I divide one dataset into several pieces and predict them in different model then combining the result, that is, in some columns there are nulls in part of features, and I could not find an apropos substitution. Thus, I want to ignore such features while predicting the relevant columns. Is this idea legal？", "votes": null }, { "id": "1069080", "postDate": "11/04/2020 03:33:12", "content": "You can use any approach for predicting as per the test API workflow. It's usually a good idea to test any approach on a validation dataset from train data before going to the test data and let the validation decide whether a particular idea is good or not.
", "rawMarkdown": "You can use any approach for predicting as per the test API workflow. It's usually a good idea to test any approach on a validation dataset from train data before going to the test data and let the validation decide whether a particular idea is good or not.", "votes": null }, { "id": "1069161", "postDate": "11/04/2020 06:52:44", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null } ], "comments": [ { "id": 1064585, "author_name": "rohanrao", "author_url": "", "post_date": "10/30/2020 09:36:30", "content": "Almost everyone who shared their CV-LB scores here have reported very stable correlation and a small absolute difference. In some cases LB is a bit higher than CV.
\nLocal 0.768 vs LB 0.690 implies something is definitely off. I would agree with @aquatic's comment that it is most likely due to some difference in the feature preparation for local test vs private test. Maybe use the public test iterator to check / verify the feature values but beyond that it is up to you to investigate your pipeline in detail.
", "votes": null, "replies": [ { "id": 1069066, "author_name": "puyuzhou", "author_url": "", "post_date": "11/04/2020 03:07:27", "content": "Pretty good, thanks. I have just found some outliers null value in test cases which might cause a low performance in public testing.
", "votes": null, "replies": [] }, { "id": 1069070, "author_name": "puyuzhou", "author_url": "", "post_date": "11/04/2020 03:16:59", "content": "Btw, sir, I want to ask you an additional question that could I divide one dataset into several pieces and predict them in different model then combining the result, that is, in some columns there are nulls in part of features, and I could not find an apropos substitution. Thus, I want to ignore such features while predicting the relevant columns. Is this idea legal？
", "votes": null, "replies": [] }, { "id": 1069080, "author_name": "rohanrao", "author_url": "", "post_date": "11/04/2020 03:33:12", "content": "You can use any approach for predicting as per the test API workflow. It's usually a good idea to test any approach on a validation dataset from train data before going to the test data and let the validation decide whether a particular idea is good or not.
", "votes": null, "replies": [] }, { "id": 1069161, "author_name": "puyuzhou", "author_url": "", "post_date": "11/04/2020 06:52:44", "content": "Thank you!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1064563": "For my experiment, I have settled with 13M dataset with about 10M train_set and 2M test_set.\n\nI have got a matrix AUC score by test_set **0.7683** . \nHowever, my submissions score (public score) is just **0.690**. Who have met such situation before? \n\n**I guess some of you might have had such puzzle.**\n\nSolved or not, could you please share your analysis or experience in the comment", "1064585": "Almost everyone who shared their CV-LB scores [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192919) have reported very stable correlation and a small absolute difference. In some cases LB is a bit higher than CV.\n\nLocal **0.768** vs LB **0.690** implies something is definitely off. I would agree with @aquatic's [comment](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/192919#1063896) that it is most likely due to some difference in the feature preparation for local test vs private test. Maybe use the public test iterator to check / verify the feature values but beyond that it is up to you to investigate your pipeline in detail.", "1069066": "Pretty good, thanks. I have just found some outliers null value in test cases which might cause a low performance in public testing.", "1069070": "Btw, sir, I want to ask you an additional question that could I divide one dataset into several pieces and predict them in different model then combining the result, that is, in some columns there are nulls in part of features, and I could not find an apropos substitution. Thus, I want to ignore such features while predicting the relevant columns. Is this idea legal？", "1069080": "You can use any approach for predicting as per the test API workflow. It's usually a good idea to test any approach on a validation dataset from train data before going to the test data and let the validation decide whether a particular idea is good or not.", "1069161": "Thank you!" }, "source": "meta" }