{
  "id": 505727,
  "title": "What is the most efficient way to disqualify metric hacking notebooks?",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/505727",
  "author_name": "",
  "post_date": "2024-05-18T19:46:27.288567900Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>In case metric hacking notebooks will be disqualified, it might not be an easy task!</p>\n<p>I would suggest doing the following:</p>\n<ol>\n<li><p>Start with notebooks that had high scores on the public leaderboard, eg., take top 500</p></li>\n<li><p>Check if they are part of the best submissions on the test set.</p></li>\n<li><p>Check if these notebooks have some scripts or code snippets in common and if they used metric hacking </p></li>\n<li><p>Delete the submissions from the rankings.</p></li>\n<li><p>Repeat </p></li>\n</ol>\n<ul>\n<li>How would you proceed?</li>\n</ul>",
  "messages": [
    {
      "id": "2822834",
      "postDate": "05/18/2024 19:46:27",
      "content": "<p>In case metric hacking notebooks will be disqualified, it might not be an easy task!</p>\n<p>I would suggest doing the following:</p>\n<ol>\n<li><p>Start with notebooks that had high scores on the public leaderboard, eg., take top 500</p></li>\n<li><p>Check if they are part of the best submissions on the test set.</p></li>\n<li><p>Check if these notebooks have some scripts or code snippets in common and if they used metric hacking </p></li>\n<li><p>Delete the submissions from the rankings.</p></li>\n<li><p>Repeat </p></li>\n</ol>\n<ul>\n<li>How would you proceed?</li>\n</ul>",
      "rawMarkdown": "In case metric hacking notebooks will be disqualified, it might not be an easy task!\n\nI would suggest doing the following:\n\n1. Start with notebooks that had high scores on the public leaderboard, eg., take top 500\n\n2. Check if they are part of the best submissions on the test set.\n\n3. Check if these notebooks have some scripts or code snippets in common and if they used metric hacking \n\n4. Delete the submissions from the rankings.\n\n5. Repeat \n\n- How would you proceed?",
      "votes": null
    },
    {
      "id": "2822886",
      "postDate": "05/18/2024 20:28:37",
      "content": "<p>Metric hacking is allowed in this competition, so this is perhaps not realizable <a href=\"https://www.kaggle.com/jeannkouagou\" target=\"_blank\">@jeannkouagou</a> </p>",
      "rawMarkdown": "Metric hacking is allowed in this competition, so this is perhaps not realizable @jeannkouagou",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2822886,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "05/18/2024 20:28:37",
      "content": "<p>Metric hacking is allowed in this competition, so this is perhaps not realizable <a href=\"https://www.kaggle.com/jeannkouagou\" target=\"_blank\">@jeannkouagou</a> </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2822834": "In case metric hacking notebooks will be disqualified, it might not be an easy task!\n\nI would suggest doing the following:\n\n1. Start with notebooks that had high scores on the public leaderboard, eg., take top 500\n\n2. Check if they are part of the best submissions on the test set.\n\n3. Check if these notebooks have some scripts or code snippets in common and if they used metric hacking \n\n4. Delete the submissions from the rankings.\n\n5. Repeat \n\n- How would you proceed?",
    "2822886": "Metric hacking is allowed in this competition, so this is perhaps not realizable @jeannkouagou"
  },
  "source": "meta"
}