Hey, I am a beginner, During the competition, I checked the code of the high-scoring public notebooks and found data leakage and instability. For example, in the high-scoring notebook with LB: 0.494, the LB score of Model_1 is actually very low (0.368), while Model_2 and Model_3 have higher LB scores (0.4x). If I were to choose one submission from these three models, I might consider Model_2 and Model_3. But how do I choose one of them? The LB result of Model_2 is higher than that of Model_3, and Model_2's CV is also better than Model_3's. Now, the results are out, Model_3 performed better than Model_2 on the private leaderboard.
\nMy question is how can I distinguish the quality of these two models? Do I need to evaluate them using additional metrics?
\nNotes:
\nHey, I am a beginner, During the competition, I checked the code of the high-scoring public notebooks and found data leakage and instability. For example, in the high-scoring notebook with LB: 0.494, the LB score of Model_1 is actually very low (0.368), while Model_2 and Model_3 have higher LB scores (0.4x). If I were to choose one submission from these three models, I might consider Model_2 and Model_3. But how do I choose one of them? The LB result of Model_2 is higher than that of Model_3, and Model_2's CV is also better than Model_3's. Now, the results are out, Model_3 performed better than Model_2 on the private leaderboard.
\nMy question is how can I distinguish the quality of these two models? Do I need to evaluate them using additional metrics?
\nNotes:
\nIn this competition, it is very hard to choose the best notebook and in fact probably best to not over engineer the solution and go with the simplest method.
\nIn most cases you'd go with the best local CV that correlates with LB, in this case, I don't believe anyone had a decent correlation between the two.
", "rawMarkdown": "In this competition, it is very hard to choose the best notebook and in fact probably best to not over engineer the solution and go with the simplest method.\n\nIn most cases you'd go with the best local CV that correlates with LB, in this case, I don't believe anyone had a decent correlation between the two.", "votes": null }, { "id": "3076437", "postDate": "12/20/2024 00:57:24", "content": "Got it, thank you.😃
", "rawMarkdown": "Got it, thank you.😃", "votes": null } ], "comments": [ { "id": 3076430, "author_name": "bsmelbs", "author_url": "", "post_date": "12/20/2024 00:54:06", "content": "In this competition, it is very hard to choose the best notebook and in fact probably best to not over engineer the solution and go with the simplest method.
\nIn most cases you'd go with the best local CV that correlates with LB, in this case, I don't believe anyone had a decent correlation between the two.
", "votes": null, "replies": [ { "id": 3076437, "author_name": "passionfruit216", "author_url": "", "post_date": "12/20/2024 00:57:24", "content": "Got it, thank you.😃
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3076420": "Hey, I am a beginner, During the competition, I checked the code of the high-scoring public notebooks and found data leakage and instability. For example, in the high-scoring notebook with LB: 0.494, the LB score of Model_1 is actually very low (0.368), while Model_2 and Model_3 have higher LB scores (0.4x). If I were to choose one submission from these three models, I might consider Model_2 and Model_3. But how do I choose one of them? The LB result of Model_2 is higher than that of Model_3, and Model_2's CV is also better than Model_3's. Now, the results are out, Model_3 performed better than Model_2 on the private leaderboard. \n\n\n**My question is how can I distinguish the quality of these two models? Do I need to evaluate them using additional metrics?**\n\n\nNotes:\n- high-scoring notebook with LB : 0.494:[https://www.kaggle.com/code/kuosys/cmi-reproducible-results-fixseed-lgb-cpu-lb-494](url)\n- Model_1 : [https://www.kaggle.com/code/passionfruit216/ensemble?scriptVersionId=209929185](url)\n- Model_2 : [https://www.kaggle.com/code/passionfruit216/model-2?scriptVersionId=212417018](url)\n- Model_3 : [https://www.kaggle.com/code/passionfruit216/model3?scriptVersionId=211292173](url)", "3076430": "In this competition, it is very hard to choose the best notebook and in fact probably best to not over engineer the solution and go with the simplest method.\n\nIn most cases you'd go with the best local CV that correlates with LB, in this case, I don't believe anyone had a decent correlation between the two.", "3076437": "Got it, thank you.😃" }, "source": "meta" }