Hello.
\nCurrently, there is a lot of information being shared about metric hacking in discussions, and it seems like many participants are using it to increase their scores.
In the end, is metric hacking against a rule of this competition? If it is, where is the line drawn between what is considered metric hacking and what is not?
\nSeeing many participants using metric hacking makes me feel like it's hard to win any medal without using metric hacking.
", "messages": [ { "id": 2812363, "postDate": "2024-05-14T07:45:55.977Z", "content": "I believe that manually reviewing submissions for hacking isn't practical. The metric remains vulnerable to exploitation, and the potential benefits from hacking are much greater than those from genuine model development. Many participants are likely to challenge the organizers' measures, as the decision between potentially having a top 100 submission and risking disqualification versus having a non-competitive, legitimate solution is obvious. A successful Kaggle competition cannot rely on a flawed metric, regardless of how many temporary fixes are applied.
", "rawMarkdown": "I believe that manually reviewing submissions for hacking isn't practical. The metric remains vulnerable to exploitation, and the potential benefits from hacking are much greater than those from genuine model development. Many participants are likely to challenge the organizers' measures, as the decision between potentially having a top 100 submission and risking disqualification versus having a non-competitive, legitimate solution is obvious. A successful Kaggle competition cannot rely on a flawed metric, regardless of how many temporary fixes are applied.", "votes": 4, "replies": [ { "id": 2812403, "postDate": "2024-05-14T08:05:35.373Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2812404, "postDate": "2024-05-14T08:05:51.217Z", "content": "I genuinely can't comprehend how the metric remains unchanged, despite it being obvious to anyone with Kaggle experience what the result will be. An unhackable metric was even proposed. For me, this competition is done.
", "rawMarkdown": "I genuinely can't comprehend how the metric remains unchanged, despite it being obvious to anyone with Kaggle experience what the result will be. An unhackable metric was even proposed. For me, this competition is done.", "votes": 4 } ] }, { "id": 2812243, "postDate": "2024-05-14T06:53:48.160Z", "content": "Hello.
\nCurrently, there is a lot of information being shared about metric hacking in discussions, and it seems like many participants are using it to increase their scores.
In the end, is metric hacking against a rule of this competition? If it is, where is the line drawn between what is considered metric hacking and what is not?
\nSeeing many participants using metric hacking makes me feel like it's hard to win any medal without using metric hacking.
", "rawMarkdown": "Hello. \nCurrently, there is a lot of information being shared about metric hacking in discussions, and it seems like many participants are using it to increase their scores. \n\nIn the end, is metric hacking against a rule of this competition? If it is, where is the line drawn between what is considered metric hacking and what is not? \n\nSeeing many participants using metric hacking makes me feel like it's hard to win any medal without using metric hacking.", "votes": 1 }, { "id": 2812350, "postDate": "2024-05-14T07:36:41.527Z", "content": "The hosts said it is no longer against the rules
", "rawMarkdown": "The hosts said it is no longer against the rules", "votes": 2, "replies": [ { "id": 2812449, "postDate": "2024-05-14T08:27:20.377Z", "content": "Thank you for your reply! I would like to see where the host said that. Could you provide a link?
", "rawMarkdown": "Thank you for your reply! I would like to see where the host said that. Could you provide a link?", "replies": [ { "id": 2812489, "postDate": "2024-05-14T08:52:02.720Z", "content": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497337
", "rawMarkdown": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497337" }, { "id": 2812502, "postDate": "2024-05-14T08:57:11.690Z", "content": "I appreciate your help. :)
", "rawMarkdown": "I appreciate your help. :)" } ] } ] } ], "comments": [ { "id": 2812363, "author_name": "ArcherPan", "author_url": "", "post_date": "2024-05-14T07:45:55.977000", "content": "I believe that manually reviewing submissions for hacking isn't practical. The metric remains vulnerable to exploitation, and the potential benefits from hacking are much greater than those from genuine model development. Many participants are likely to challenge the organizers' measures, as the decision between potentially having a top 100 submission and risking disqualification versus having a non-competitive, legitimate solution is obvious. A successful Kaggle competition cannot rely on a flawed metric, regardless of how many temporary fixes are applied.
", "votes": 4, "replies": [ { "id": 2812403, "author_name": "", "author_url": "", "post_date": "2024-05-14T08:05:35.373000", "content": "", "votes": 0, "replies": [] }, { "id": 2812404, "author_name": "Niko", "author_url": "", "post_date": "2024-05-14T08:05:51.217000", "content": "I genuinely can't comprehend how the metric remains unchanged, despite it being obvious to anyone with Kaggle experience what the result will be. An unhackable metric was even proposed. For me, this competition is done.
", "votes": 4, "replies": [] } ] }, { "id": 2812350, "author_name": "JM", "author_url": "", "post_date": "2024-05-14T07:36:41.527000", "content": "The hosts said it is no longer against the rules
", "votes": 2, "replies": [ { "id": 2812449, "author_name": "Kouhei Miyazaki", "author_url": "", "post_date": "2024-05-14T08:27:20.377000", "content": "Thank you for your reply! I would like to see where the host said that. Could you provide a link?
", "votes": 0, "replies": [ { "id": 2812489, "author_name": "JM", "author_url": "", "post_date": "2024-05-14T08:52:02.720000", "content": "https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/497337
", "votes": 0, "replies": [] }, { "id": 2812502, "author_name": "Kouhei Miyazaki", "author_url": "", "post_date": "2024-05-14T08:57:11.690000", "content": "I appreciate your help. :)
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2812363": "I believe that manually reviewing submissions for hacking isn't practical. The metric remains vulnerable to exploitation, and the potential benefits from hacking are much greater than those from genuine model development. Many participants are likely to challenge the organizers' measures, as the decision between potentially having a top 100 submission and risking disqualification versus having a non-competitive, legitimate solution is obvious. A successful Kaggle competition cannot rely on a flawed metric, regardless of how many temporary fixes are applied.", "2812243": "Hello. \nCurrently, there is a lot of information being shared about metric hacking in discussions, and it seems like many participants are using it to increase their scores. \n\nIn the end, is metric hacking against a rule of this competition? If it is, where is the line drawn between what is considered metric hacking and what is not? \n\nSeeing many participants using metric hacking makes me feel like it's hard to win any medal without using metric hacking.", "2812350": "The hosts said it is no longer against the rules" } }