I spent quite a bit of time on this competition, and had a really awesome CV. Given everyone was saying that CV and LB were well correlated I did not bother submit and only worked on test prediction last few days of the comp. When I submitted I got a score lower than best public notebooks!
\nI spent few days and found my mistake. I implemented the competition metric wrongly, i.e. I compute the score by session then average. For each session I computed recall@20 then average over submissions, instead of computing all the true positives, then divide by all positives.
\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.
\nI am not sure I would have got a much better result if I had not made this blunder, but at least I know why my killer CV did not translate into a good LB score.
\nLesson learned.
", "messages": [ { "id": "2128975", "postDate": "02/04/2023 08:21:18", "content": "I spent quite a bit of time on this competition, and had a really awesome CV. Given everyone was saying that CV and LB were well correlated I did not bother submit and only worked on test prediction last few days of the comp. When I submitted I got a score lower than best public notebooks!
\nI spent few days and found my mistake. I implemented the competition metric wrongly, i.e. I compute the score by session then average. For each session I computed recall@20 then average over submissions, instead of computing all the true positives, then divide by all positives.
\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.
\nI am not sure I would have got a much better result if I had not made this blunder, but at least I know why my killer CV did not translate into a good LB score.
\nLesson learned.
", "rawMarkdown": "I spent quite a bit of time on this competition, and had a really awesome CV. Given everyone was saying that CV and LB were well correlated I did not bother submit and only worked on test prediction last few days of the comp. When I submitted I got a score lower than best public notebooks!\n\nI spent few days and found my mistake. I implemented the competition metric wrongly, i.e. I compute the score by session then average. For each session I computed recall@20 then average over submissions, instead of computing all the true positives, then divide by all positives.\n\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.\n\nI am not sure I would have got a much better result if I had not made this blunder, but at least I know why my killer CV did not translate into a good LB score.\n\nLesson learned.", "votes": null }, { "id": "2128982", "postDate": "02/04/2023 08:33:08", "content": "What would your score have been if it weren't for this bug?
", "rawMarkdown": "What would your score have been if it weren't for this bug?", "votes": null }, { "id": "2129023", "postDate": "02/04/2023 09:04:38", "content": "I had made the mistake of implementing competition metric wrong at least 2 times. But my wrong CV was directional, so every improvement in cv meant improvement in LB. Only in the last month, I implemented the correct one. For me, this competition was solving bugs till the end. Also I don't think you can directly optimize the competition metric ?
", "rawMarkdown": "I had made the mistake of implementing competition metric wrong at least 2 times. But my wrong CV was directional, so every improvement in cv meant improvement in LB. Only in the last month, I implemented the correct one. For me, this competition was solving bugs till the end. Also I don't think you can directly optimize the competition metric ?", "votes": null }, { "id": "2129083", "postDate": "02/04/2023 09:59:13", "content": "I did the same mistake at the beginning, but managed to find it early. Another one was that I split data by chunks without shuffling, and CV scores for the last chunks were much higher than for the first ones (because sessions with lower ID occur earlier I suppose).
", "rawMarkdown": "I did the same mistake at the beginning, but managed to find it early. Another one was that I split data by chunks without shuffling, and CV scores for the last chunks were much higher than for the first ones (because sessions with lower ID occur earlier I suppose).", "votes": null }, { "id": "2129213", "postDate": "02/04/2023 12:16:50", "content": "I don't know. I tuned everything for the wrong metric.
", "rawMarkdown": "I don't know. I tuned everything for the wrong metric.", "votes": null }, { "id": "2129217", "postDate": "02/04/2023 12:18:33", "content": "xgboost ndcg@20- metric was very well correlated with my wrong metric.
\nI don't know if there is a good proxy for the actual computation metric indeed.
", "rawMarkdown": "xgboost ndcg@20- metric was very well correlated with my wrong metric.\n\nI don't know if there is a good proxy for the actual computation metric indeed.", "votes": null }, { "id": "2129218", "postDate": "02/04/2023 12:18:49", "content": "Yes, shufling everything is a must have.
", "rawMarkdown": "Yes, shufling everything is a must have.", "votes": null }, { "id": "2129232", "postDate": "02/04/2023 12:49:06", "content": "I did the same because I didn't read public notebooks. Thanks to my teammate @nlztrk detected that after I waste couple weeks trying to replicate my teams' results.
", "rawMarkdown": "I did the same because I didn't read public notebooks. Thanks to my teammate @nlztrk detected that after I waste couple weeks trying to replicate my teams' results.", "votes": null }, { "id": "2129233", "postDate": "02/04/2023 12:49:21", "content": "MAP@20 was a good proxy metric imo. I didn't use ndcg because order of predictions is also important for that metric.
", "rawMarkdown": "MAP@20 was a good proxy metric imo. I didn't use ndcg because order of predictions is also important for that metric.", "votes": null }, { "id": "2129283", "postDate": "02/04/2023 13:45:11", "content": "I had somewhat similar experience but the other way around - one misplaced line of code causing my CV score to bounce up and down and making me rack my brain on why my ranker did not work. Fortunately for me however I still had some time to turn things around.
\nGood luck in your next competition!
same I used map@20 for early stopping
", "rawMarkdown": "same I used map@20 for early stopping", "votes": null }, { "id": "2129503", "postDate": "02/04/2023 17:06:28", "content": "\n\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.
\n
I guess even if you have some strategy for the solution you are going to build you should anyway check the discussion section here on kaggle. There were other people who misunderstood the metric just the same way you misunderstood it, including me, and I remember there was a clarification answer from the organizers about how exactly the metric is being calculated.
", "rawMarkdown": ">I spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.\n\nI guess even if you have some strategy for the solution you are going to build you should anyway check the discussion section here on kaggle. There were other people who misunderstood the metric just the same way you misunderstood it, including me, and I remember there was a clarification answer from the organizers about how exactly the metric is being calculated.", "votes": null }, { "id": "2129616", "postDate": "02/04/2023 18:50:39", "content": "Sorry for you @cpmpml, but I made the same mistake at the start of the competition and spent more than one week to figure it out.
", "rawMarkdown": "Sorry for you @cpmpml, but I made the same mistake at the start of the competition and spent more than one week to figure it out.", "votes": null }, { "id": "2130038", "postDate": "02/05/2023 05:06:33", "content": "In my experiment, the early stop rounds is same with ndgc/map@20
", "rawMarkdown": "In my experiment, the early stop rounds is same with ndgc/map@20", "votes": null }, { "id": "2130400", "postDate": "02/05/2023 12:37:15", "content": "If I read correctly I would not have made my mistake. Issue was not that I did no read.
", "rawMarkdown": "If I read correctly I would not have made my mistake. Issue was not that I did no read.", "votes": null }, { "id": "2130401", "postDate": "02/05/2023 12:39:05", "content": "I read the public notebooks and foudn the ode to be awfully slow. I coded an efficient way to compute the metric (dataframe merge on GPU) but I introduced the bug at the same time!
", "rawMarkdown": "I read the public notebooks and foudn the ode to be awfully slow. I coded an efficient way to compute the metric (dataframe merge on GPU) but I introduced the bug at the same time!", "votes": null }, { "id": "2130674", "postDate": "02/05/2023 16:07:47", "content": "It is very nice of you to share your error solution with us. Thank you for our experience
", "rawMarkdown": "It is very nice of you to share your error solution with us. Thank you for our experience", "votes": null }, { "id": "2130807", "postDate": "02/05/2023 17:39:31", "content": "The desciption was not clear enough as many people misunderstood it, not just you.
", "rawMarkdown": "The desciption was not clear enough as many people misunderstood it, not just you.", "votes": null }, { "id": "2131135", "postDate": "02/05/2023 22:23:53", "content": "I read the piece where the description did not have the cap at 20. I did not see a discussion of the same error as me. The description is clear, it is me who did not read it correctly.
", "rawMarkdown": "I read the piece where the description did not have the cap at 20. I did not see a discussion of the same error as me. The description is clear, it is me who did not read it correctly.", "votes": null }, { "id": "2131488", "postDate": "02/06/2023 06:48:34", "content": "One trick I use is to create a leaderboard simulation and submit against that. The key of course is to ensure that it implements the leaderboard exactly, which requires a bit of probing sometimes to ensure the logic is exact.
\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.
", "rawMarkdown": "One trick I use is to create a leaderboard simulation and submit against that. The key of course is to ensure that it implements the leaderboard exactly, which requires a bit of probing sometimes to ensure the logic is exact.\n\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.", "votes": null }, { "id": "2131634", "postDate": "02/06/2023 08:49:34", "content": "\n\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.
\n
I wanted to put this in my post but it would have sounded as if Kaggle was wrong and caused my issue. My issue is that I did not read the formula correctly.
\nThis said, I fully agree with you. There have been competitions where it took weeks if not months to get a correct metric implementation.
", "rawMarkdown": "> I think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.\n\nI wanted to put this in my post but it would have sounded as if Kaggle was wrong and caused my issue. My issue is that I did not read the formula correctly. \n\nThis said, I fully agree with you. There have been competitions where it took weeks if not months to get a correct metric implementation.", "votes": null }, { "id": "2131636", "postDate": "02/06/2023 08:51:00", "content": "That's bad too, but at leats you felt urged to submit. That's the issue with too good a CV: I did not submit till that last day.
", "rawMarkdown": "That's bad too, but at leats you felt urged to submit. That's the issue with too good a CV: I did not submit till that last day.", "votes": null }, { "id": "2140624", "postDate": "02/11/2023 23:53:49", "content": "Because of this post, I also submitted, and realized I had a tiny error in my submission, which for Godaddy competition resulted in quite significant LB error.
", "rawMarkdown": "Because of this post, I also submitted, and realized I had a tiny error in my submission, which for Godaddy competition resulted in quite significant LB error.", "votes": null }, { "id": "2141027", "postDate": "02/12/2023 11:48:06", "content": "Thanks for sharing that this post was somewhat useful. I will also submit ASAP in godaddy I just entered.
", "rawMarkdown": "Thanks for sharing that this post was somewhat useful. I will also submit ASAP in godaddy I just entered.", "votes": null }, { "id": "2141787", "postDate": "02/13/2023 05:05:26", "content": "Thank you for sharing your experience and lessons learned from this competition.
\nIt's a great reminder for all of us to be meticulous when it comes to reading the evaluation page and understanding the competition metric. Your mistake is a valuable lesson for everyone to double-check their work before submitting. Keep up the good work and don't be discouraged, there's always something to learn from each competition. Best of luck on your next project!
", "rawMarkdown": "Thank you for sharing your experience and lessons learned from this competition. \n\nIt's a great reminder for all of us to be meticulous when it comes to reading the evaluation page and understanding the competition metric. Your mistake is a valuable lesson for everyone to double-check their work before submitting. Keep up the good work and don't be discouraged, there's always something to learn from each competition. Best of luck on your next project!", "votes": null }, { "id": "2141933", "postDate": "02/13/2023 07:51:01", "content": "Thank you !
", "rawMarkdown": "Thank you !", "votes": null } ], "comments": [ { "id": 2128982, "author_name": "group16", "author_url": "", "post_date": "02/04/2023 08:33:08", "content": "What would your score have been if it weren't for this bug?
", "votes": null, "replies": [ { "id": 2129213, "author_name": "cpmpml", "author_url": "", "post_date": "02/04/2023 12:16:50", "content": "I don't know. I tuned everything for the wrong metric.
", "votes": null, "replies": [] } ] }, { "id": 2129023, "author_name": "nikhilmishradev", "author_url": "", "post_date": "02/04/2023 09:04:38", "content": "I had made the mistake of implementing competition metric wrong at least 2 times. But my wrong CV was directional, so every improvement in cv meant improvement in LB. Only in the last month, I implemented the correct one. For me, this competition was solving bugs till the end. Also I don't think you can directly optimize the competition metric ?
", "votes": null, "replies": [ { "id": 2129217, "author_name": "cpmpml", "author_url": "", "post_date": "02/04/2023 12:18:33", "content": "xgboost ndcg@20- metric was very well correlated with my wrong metric.
\nI don't know if there is a good proxy for the actual computation metric indeed.
", "votes": null, "replies": [ { "id": 2129233, "author_name": "gunesevitan", "author_url": "", "post_date": "02/04/2023 12:49:21", "content": "MAP@20 was a good proxy metric imo. I didn't use ndcg because order of predictions is also important for that metric.
", "votes": null, "replies": [ { "id": 2129341, "author_name": "nikhilmishradev", "author_url": "", "post_date": "02/04/2023 14:34:16", "content": "same I used map@20 for early stopping
", "votes": null, "replies": [ { "id": 2130038, "author_name": "gongbi", "author_url": "", "post_date": "02/05/2023 05:06:33", "content": "In my experiment, the early stop rounds is same with ndgc/map@20
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2129083, "author_name": "greenwolf", "author_url": "", "post_date": "02/04/2023 09:59:13", "content": "I did the same mistake at the beginning, but managed to find it early. Another one was that I split data by chunks without shuffling, and CV scores for the last chunks were much higher than for the first ones (because sessions with lower ID occur earlier I suppose).
", "votes": null, "replies": [ { "id": 2129218, "author_name": "cpmpml", "author_url": "", "post_date": "02/04/2023 12:18:49", "content": "Yes, shufling everything is a must have.
", "votes": null, "replies": [] } ] }, { "id": 2129232, "author_name": "gunesevitan", "author_url": "", "post_date": "02/04/2023 12:49:06", "content": "I did the same because I didn't read public notebooks. Thanks to my teammate @nlztrk detected that after I waste couple weeks trying to replicate my teams' results.
", "votes": null, "replies": [ { "id": 2130401, "author_name": "cpmpml", "author_url": "", "post_date": "02/05/2023 12:39:05", "content": "I read the public notebooks and foudn the ode to be awfully slow. I coded an efficient way to compute the metric (dataframe merge on GPU) but I introduced the bug at the same time!
", "votes": null, "replies": [] } ] }, { "id": 2129283, "author_name": "hoangnguyen719", "author_url": "", "post_date": "02/04/2023 13:45:11", "content": "I had somewhat similar experience but the other way around - one misplaced line of code causing my CV score to bounce up and down and making me rack my brain on why my ranker did not work. Fortunately for me however I still had some time to turn things around.
\nGood luck in your next competition!
That's bad too, but at leats you felt urged to submit. That's the issue with too good a CV: I did not submit till that last day.
", "votes": null, "replies": [] } ] }, { "id": 2129503, "author_name": "artemfedorov", "author_url": "", "post_date": "02/04/2023 17:06:28", "content": "\n\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.
\n
I guess even if you have some strategy for the solution you are going to build you should anyway check the discussion section here on kaggle. There were other people who misunderstood the metric just the same way you misunderstood it, including me, and I remember there was a clarification answer from the organizers about how exactly the metric is being calculated.
", "votes": null, "replies": [ { "id": 2130400, "author_name": "cpmpml", "author_url": "", "post_date": "02/05/2023 12:37:15", "content": "If I read correctly I would not have made my mistake. Issue was not that I did no read.
", "votes": null, "replies": [ { "id": 2130807, "author_name": "artemfedorov", "author_url": "", "post_date": "02/05/2023 17:39:31", "content": "The desciption was not clear enough as many people misunderstood it, not just you.
", "votes": null, "replies": [ { "id": 2131135, "author_name": "cpmpml", "author_url": "", "post_date": "02/05/2023 22:23:53", "content": "I read the piece where the description did not have the cap at 20. I did not see a discussion of the same error as me. The description is clear, it is me who did not read it correctly.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2129616, "author_name": "titericz", "author_url": "", "post_date": "02/04/2023 18:50:39", "content": "Sorry for you @cpmpml, but I made the same mistake at the start of the competition and spent more than one week to figure it out.
", "votes": null, "replies": [] }, { "id": 2130674, "author_name": "ibrahimkaratas", "author_url": "", "post_date": "02/05/2023 16:07:47", "content": "It is very nice of you to share your error solution with us. Thank you for our experience
", "votes": null, "replies": [] }, { "id": 2131488, "author_name": "kaggleqrdl", "author_url": "", "post_date": "02/06/2023 06:48:34", "content": "One trick I use is to create a leaderboard simulation and submit against that. The key of course is to ensure that it implements the leaderboard exactly, which requires a bit of probing sometimes to ensure the logic is exact.
\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.
", "votes": null, "replies": [ { "id": 2131634, "author_name": "cpmpml", "author_url": "", "post_date": "02/06/2023 08:49:34", "content": "\n\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.
\n
I wanted to put this in my post but it would have sounded as if Kaggle was wrong and caused my issue. My issue is that I did not read the formula correctly.
\nThis said, I fully agree with you. There have been competitions where it took weeks if not months to get a correct metric implementation.
", "votes": null, "replies": [] } ] }, { "id": 2140624, "author_name": "returnofsputnik", "author_url": "", "post_date": "02/11/2023 23:53:49", "content": "Because of this post, I also submitted, and realized I had a tiny error in my submission, which for Godaddy competition resulted in quite significant LB error.
", "votes": null, "replies": [ { "id": 2141027, "author_name": "cpmpml", "author_url": "", "post_date": "02/12/2023 11:48:06", "content": "Thanks for sharing that this post was somewhat useful. I will also submit ASAP in godaddy I just entered.
", "votes": null, "replies": [] } ] }, { "id": 2141787, "author_name": "giranntu", "author_url": "", "post_date": "02/13/2023 05:05:26", "content": "Thank you for sharing your experience and lessons learned from this competition.
\nIt's a great reminder for all of us to be meticulous when it comes to reading the evaluation page and understanding the competition metric. Your mistake is a valuable lesson for everyone to double-check their work before submitting. Keep up the good work and don't be discouraged, there's always something to learn from each competition. Best of luck on your next project!
", "votes": null, "replies": [ { "id": 2141933, "author_name": "cpmpml", "author_url": "", "post_date": "02/13/2023 07:51:01", "content": "Thank you !
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2128975": "I spent quite a bit of time on this competition, and had a really awesome CV. Given everyone was saying that CV and LB were well correlated I did not bother submit and only worked on test prediction last few days of the comp. When I submitted I got a score lower than best public notebooks!\n\nI spent few days and found my mistake. I implemented the competition metric wrongly, i.e. I compute the score by session then average. For each session I computed recall@20 then average over submissions, instead of computing all the true positives, then divide by all positives.\n\nI spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.\n\nI am not sure I would have got a much better result if I had not made this blunder, but at least I know why my killer CV did not translate into a good LB score.\n\nLesson learned.", "2128982": "What would your score have been if it weren't for this bug?", "2129023": "I had made the mistake of implementing competition metric wrong at least 2 times. But my wrong CV was directional, so every improvement in cv meant improvement in LB. Only in the last month, I implemented the correct one. For me, this competition was solving bugs till the end. Also I don't think you can directly optimize the competition metric ?", "2129083": "I did the same mistake at the beginning, but managed to find it early. Another one was that I split data by chunks without shuffling, and CV scores for the last chunks were much higher than for the first ones (because sessions with lower ID occur earlier I suppose).", "2129213": "I don't know. I tuned everything for the wrong metric.", "2129217": "xgboost ndcg@20- metric was very well correlated with my wrong metric.\n\nI don't know if there is a good proxy for the actual computation metric indeed.", "2129218": "Yes, shufling everything is a must have.", "2129232": "I did the same because I didn't read public notebooks. Thanks to my teammate @nlztrk detected that after I waste couple weeks trying to replicate my teams' results.", "2129233": "MAP@20 was a good proxy metric imo. I didn't use ndcg because order of predictions is also important for that metric.", "2129283": "I had somewhat similar experience but the other way around - one misplaced line of code causing my CV score to bounce up and down and making me rack my brain on why my ranker did not work. Fortunately for me however I still had some time to turn things around.\nGood luck in your next competition!", "2129341": "same I used map@20 for early stopping", "2129503": ">I spent 2 months optimizing the wrong metric. I should have been more careful when reading the evaluation page.\n\nI guess even if you have some strategy for the solution you are going to build you should anyway check the discussion section here on kaggle. There were other people who misunderstood the metric just the same way you misunderstood it, including me, and I remember there was a clarification answer from the organizers about how exactly the metric is being calculated.", "2129616": "Sorry for you @cpmpml, but I made the same mistake at the start of the competition and spent more than one week to figure it out.", "2130038": "In my experiment, the early stop rounds is same with ndgc/map@20", "2130400": "If I read correctly I would not have made my mistake. Issue was not that I did no read.", "2130401": "I read the public notebooks and foudn the ode to be awfully slow. I coded an efficient way to compute the metric (dataframe merge on GPU) but I introduced the bug at the same time!", "2130674": "It is very nice of you to share your error solution with us. Thank you for our experience", "2130807": "The desciption was not clear enough as many people misunderstood it, not just you.", "2131135": "I read the piece where the description did not have the cap at 20. I did not see a discussion of the same error as me. The description is clear, it is me who did not read it correctly.", "2131488": "One trick I use is to create a leaderboard simulation and submit against that. The key of course is to ensure that it implements the leaderboard exactly, which requires a bit of probing sometimes to ensure the logic is exact.\n\nI think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.", "2131634": "> I think it would be great if Kaggle provided the evaluation code in every comp so this would be less of a mystery.\n\nI wanted to put this in my post but it would have sounded as if Kaggle was wrong and caused my issue. My issue is that I did not read the formula correctly. \n\nThis said, I fully agree with you. There have been competitions where it took weeks if not months to get a correct metric implementation.", "2131636": "That's bad too, but at leats you felt urged to submit. That's the issue with too good a CV: I did not submit till that last day.", "2140624": "Because of this post, I also submitted, and realized I had a tiny error in my submission, which for Godaddy competition resulted in quite significant LB error.", "2141027": "Thanks for sharing that this post was somewhat useful. I will also submit ASAP in godaddy I just entered.", "2141787": "Thank you for sharing your experience and lessons learned from this competition. \n\nIt's a great reminder for all of us to be meticulous when it comes to reading the evaluation page and understanding the competition metric. Your mistake is a valuable lesson for everyone to double-check their work before submitting. Keep up the good work and don't be discouraged, there's always something to learn from each competition. Best of luck on your next project!", "2141933": "Thank you !" }, "source": "meta" }