![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F99029%2Fc9ecfda58141eddee282d98f151b020a%2FSad%20Pablo%20Escobar%20Meme.png?generation=1708499662930998&alt=media\")
It has been a week since the metric issue was revealed and the host said it would take a few days for a fix, however, it doesn’t seem any fixes are in place yet. Yes, we can work on our models locally by improving AUC and other things, but, In my opinion, this situation negatively affects the competition as makes the public LB kind of meaningless and doesn’t encourage new teams to join.
\nIt is very unfortunate, because for this competition the host engagement and other organizational aspects are well above any of the Kaggle standards. So may be we will have some updates on the status of the fix and a probable ETA of the corrected metric?
", "messages": [ { "id": "2660140", "postDate": "02/20/2024 11:25:00", "content": "It has been a week since the metric issue was revealed and the host said it would take a few days for a fix, however, it doesn’t seem any fixes are in place yet. Yes, we can work on our models locally by improving AUC and other things, but, In my opinion, this situation negatively affects the competition as makes the public LB kind of meaningless and doesn’t encourage new teams to join.
\nIt is very unfortunate, because for this competition the host engagement and other organizational aspects are well above any of the Kaggle standards. So may be we will have some updates on the status of the fix and a probable ETA of the corrected metric?
", "rawMarkdown": "It has been a week since the metric issue was revealed and the host said it would take a few days for a fix, however, it doesn’t seem any fixes are in place yet. Yes, we can work on our models locally by improving AUC and other things, but, In my opinion, this situation negatively affects the competition as makes the public LB kind of meaningless and doesn’t encourage new teams to join. \n\nIt is very unfortunate, because for this competition the host engagement and other organizational aspects are well above any of the Kaggle standards. So may be we will have some updates on the status of the fix and a probable ETA of the corrected metric?", "votes": null }, { "id": "2660149", "postDate": "02/20/2024 11:34:56", "content": "Hi Oleksiy,
\nI will provide later today new update about the fix and it's implementation. Thank you for your patience.
Could we test it before the update ? it would be really annoying if the new one is also hackable.
\ni have suggested a metric and tested it in detail, so far it seems OK and handles HC stability concerns. I also shared my test code you can use it to test your metric ideas.
Dobry vecer Tomas @tomasjeline2, I guess you are still facing issue with the metric as you wrote about a possible new update today ?
", "rawMarkdown": "Dobry vecer Tomas @tomasjeline2, I guess you are still facing issue with the metric as you wrote about a possible new update today ?", "votes": null }, { "id": "2660997", "postDate": "02/20/2024 23:59:09", "content": "As pointed out, please make sure the new version is not \"hackable\". Never underestimate the will of people to hack it. Just look at the current LB, even with the official announcement that the metric will be changed (hence LB reset), there is a lot of interest in hacking to the max, I guess, with the hope some of that hacking pays off with the new metric.
\nIf it can be hacked, it will be hacked.
", "rawMarkdown": "As pointed out, please make sure the new version is not \"hackable\". Never underestimate the will of people to hack it. Just look at the current LB, even with the official announcement that the metric will be changed (hence LB reset), there is a lot of interest in hacking to the max, I guess, with the hope some of that hacking pays off with the new metric.\n\nIf it can be hacked, it will be hacked.", "votes": null }, { "id": "2661354", "postDate": "02/21/2024 07:15:27", "content": "
it has been 9 days :(
", "rawMarkdown": "\n\nit has been 9 days :(", "votes": null }, { "id": "2661403", "postDate": "02/21/2024 08:09:19", "content": "Dear @carloshuertas
\nPeople have been \"playing the system\" for over 70 years. Maybe simply go with a strictly proper scoring rule which, in principle, represents an honest assessment of a classifiers probabilistic performance?
\nAll the best,
\ncarl
The link doesn't work, anyway I know which one you do mean. It has other issues, I will respond to you in the original thread.
", "rawMarkdown": "The link doesn't work, anyway I know which one you do mean. It has other issues, I will respond to you in the original thread.", "votes": null }, { "id": "2661822", "postDate": "02/21/2024 14:19:58", "content": "@carlmcbrideellis thanks for the second paper by GNEITING.
", "rawMarkdown": "carlmcbrideellis thanks for the second paper by GNEITING.", "votes": null }, { "id": "2661837", "postDate": "02/21/2024 14:34:31", "content": "![](\"https://i.imgur.com/T57Ffq1.png\")
What we are doing at the moment:
\n![](\"https://i.imgur.com/RM6rPoK.png\")
Keep the faith and mind the bug 😉
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6191256%2F8b8a5e30f3ced3ed8354c3f630f678f7%2Fbreaking-bad.jpg?generation=1708529533340453&alt=media\")
Hi Oleksiy,
\nI will provide later today new update about the fix and it's implementation. Thank you for your patience.
Could we test it before the update ? it would be really annoying if the new one is also hackable.
\ni have suggested a metric and tested it in detail, so far it seems OK and handles HC stability concerns. I also shared my test code you can use it to test your metric ideas.
The link doesn't work, anyway I know which one you do mean. It has other issues, I will respond to you in the original thread.
", "votes": null, "replies": [] } ] }, { "id": 2660874, "author_name": "pourchot", "author_url": "", "post_date": "02/20/2024 21:23:08", "content": "Dobry vecer Tomas @tomasjeline2, I guess you are still facing issue with the metric as you wrote about a possible new update today ?
", "votes": null, "replies": [] } ] }, { "id": 2660997, "author_name": "carloshuertas", "author_url": "", "post_date": "02/20/2024 23:59:09", "content": "As pointed out, please make sure the new version is not \"hackable\". Never underestimate the will of people to hack it. Just look at the current LB, even with the official announcement that the metric will be changed (hence LB reset), there is a lot of interest in hacking to the max, I guess, with the hope some of that hacking pays off with the new metric.
\nIf it can be hacked, it will be hacked.
", "votes": null, "replies": [ { "id": 2661403, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "02/21/2024 08:09:19", "content": "Dear @carloshuertas
\nPeople have been \"playing the system\" for over 70 years. Maybe simply go with a strictly proper scoring rule which, in principle, represents an honest assessment of a classifiers probabilistic performance?
\nAll the best,
\ncarl
@carlmcbrideellis thanks for the second paper by GNEITING.
", "votes": null, "replies": [] } ] } ] }, { "id": 2661354, "author_name": "davutpolat", "author_url": "", "post_date": "02/21/2024 07:15:27", "content": "
it has been 9 days :(
", "votes": null, "replies": [ { "id": 2661837, "author_name": "jetakow", "author_url": "", "post_date": "02/21/2024 14:34:31", "content": "
What we are doing at the moment:
\n
Keep the faith and mind the bug 😉
\n