In one week I will know if I'm wrong or right. In this competition with a highly noisy dataset, my strategy was to find issues to have a high mean of my score (by CV or on the public LB) and a low variance of the score when I change the random state value in my notebook.
\nThe best public notebook has sometimes very high scores, but a high variance of its score.
\nAnd you, was was your strategy ?
", "messages": [ { "id": "3069686", "postDate": "12/11/2024 18:54:29", "content": "In one week I will know if I'm wrong or right. In this competition with a highly noisy dataset, my strategy was to find issues to have a high mean of my score (by CV or on the public LB) and a low variance of the score when I change the random state value in my notebook.
\nThe best public notebook has sometimes very high scores, but a high variance of its score.
\nAnd you, was was your strategy ?
", "rawMarkdown": "In one week I will know if I'm wrong or right. In this competition with a highly noisy dataset, my strategy was to find issues to have a high mean of my score (by CV or on the public LB) and a low variance of the score when I change the random state value in my notebook.\n\nThe best public notebook has sometimes very high scores, but a high variance of its score.\n\nAnd you, was was your strategy ?", "votes": null }, { "id": "3069740", "postDate": "12/11/2024 20:11:04", "content": "Let's wait for a week, I think we are better off discussing this based on the results of our work @adaubas
\nI agree that high scoring public notebooks have lots of issues and are unlikely to sustain.
", "rawMarkdown": "Let's wait for a week, I think we are better off discussing this based on the results of our work @adaubas \n\nI agree that high scoring public notebooks have lots of issues and are unlikely to sustain.", "votes": null }, { "id": "3069909", "postDate": "12/12/2024 03:16:51", "content": "just waiting to see that
", "rawMarkdown": "just waiting to see that", "votes": null }, { "id": "3070030", "postDate": "12/12/2024 06:48:17", "content": "Not sure what my strategy will be, but there are three reasonable strategies. Two submissions with high variance, one with high variance and one with low variance and the last one would be two low variance submissions. In all cases it's important to choose two submissions that aren't too highly correlated.
\nWhich strategy is the right one depends on one's attitude towards risk. Afterall, only the better submission of the two counts towards the leaderboard.
Thank you @lennarthaupts,
\nI don't trust the first strategy Two submissions with high variance even if those two submssions aren't too highly correlated. It's a strategy where chance plays too big a role.
There are clear data leaks. Atleast these data leaks need to be addressed. the train score and test score are very different as well. some amount of regularization is needed to be done. A more rigorous CV and calculation of thresholds based on k folds. Check on any data leaks in CV.
", "rawMarkdown": "There are clear data leaks. Atleast these data leaks need to be addressed. the train score and test score are very different as well. some amount of regularization is needed to be done. A more rigorous CV and calculation of thresholds based on k folds. Check on any data leaks in CV.", "votes": null }, { "id": "3072879", "postDate": "12/15/2024 17:57:52", "content": "I agree, it's also too much of a gamble for my liking. 😄
", "rawMarkdown": "I agree, it's also too much of a gamble for my liking. 😄", "votes": null } ], "comments": [ { "id": 3069740, "author_name": "ravi20076", "author_url": "", "post_date": "12/11/2024 20:11:04", "content": "Let's wait for a week, I think we are better off discussing this based on the results of our work @adaubas
\nI agree that high scoring public notebooks have lots of issues and are unlikely to sustain.
", "votes": null, "replies": [] }, { "id": 3069909, "author_name": "trcnveli", "author_url": "", "post_date": "12/12/2024 03:16:51", "content": "just waiting to see that
", "votes": null, "replies": [] }, { "id": 3070030, "author_name": "gunesevitan", "author_url": "", "post_date": "12/12/2024 06:48:17", "content": "Not sure what my strategy will be, but there are three reasonable strategies. Two submissions with high variance, one with high variance and one with low variance and the last one would be two low variance submissions. In all cases it's important to choose two submissions that aren't too highly correlated.
\nWhich strategy is the right one depends on one's attitude towards risk. Afterall, only the better submission of the two counts towards the leaderboard.
Thank you @lennarthaupts,
\nI don't trust the first strategy Two submissions with high variance even if those two submssions aren't too highly correlated. It's a strategy where chance plays too big a role.
I agree, it's also too much of a gamble for my liking. 😄
", "votes": null, "replies": [] } ] } ] }, { "id": 3072072, "author_name": "dristi0705", "author_url": "", "post_date": "12/14/2024 16:44:29", "content": "There are clear data leaks. Atleast these data leaks need to be addressed. the train score and test score are very different as well. some amount of regularization is needed to be done. A more rigorous CV and calculation of thresholds based on k folds. Check on any data leaks in CV.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3069686": "In one week I will know if I'm wrong or right. In this competition with a highly noisy dataset, my strategy was to find issues to have a high mean of my score (by CV or on the public LB) and a low variance of the score when I change the random state value in my notebook.\n\nThe best public notebook has sometimes very high scores, but a high variance of its score.\n\nAnd you, was was your strategy ?", "3069740": "Let's wait for a week, I think we are better off discussing this based on the results of our work @adaubas \n\nI agree that high scoring public notebooks have lots of issues and are unlikely to sustain.", "3069909": "just waiting to see that", "3070030": "* Find irregularities in the dataset and clean as much as possible\n* Minimize proxy metrics like rmse while maximize qwk\n* Minimize fold score variance\n* Apply postprocessing that helps to deal with qwk instability", "3071674": "Not sure what my strategy will be, but there are three reasonable strategies. Two submissions with high variance, one with high variance and one with low variance and the last one would be two low variance submissions. In all cases it's important to choose two submissions that aren't too highly correlated.\nWhich strategy is the right one depends on one's attitude towards risk. Afterall, only the better submission of the two counts towards the leaderboard.", "3071749": "Thank you @lennarthaupts,\nI don't trust the first strategy `Two submissions with high variance` even if those two submssions aren't too highly correlated. It's a strategy where chance plays too big a role.", "3072072": "There are clear data leaks. Atleast these data leaks need to be addressed. the train score and test score are very different as well. some amount of regularization is needed to be done. A more rigorous CV and calculation of thresholds based on k folds. Check on any data leaks in CV.", "3072879": "I agree, it's also too much of a gamble for my liking. 😄" }, "source": "meta" }