{
  "id": 497003,
  "title": "The public leaderboard gets less and less reliable over time",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/497003",
  "author_name": "",
  "post_date": "2024-04-23T08:35:47.579697200Z",
  "votes": 14,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I expect that high score (about 0.63) and hacked notebook will be published soon. I am afraid that this hacking trick may generalize well on the private dataset, then many robust solutions will fall drastically (even about 200 places). Will the host check these solutions? </p>",
  "messages": [
    {
      "id": "2769244",
      "postDate": "04/23/2024 08:35:47",
      "content": "<p>I expect that high score (about 0.63) and hacked notebook will be published soon. I am afraid that this hacking trick may generalize well on the private dataset, then many robust solutions will fall drastically (even about 200 places). Will the host check these solutions? </p>",
      "rawMarkdown": "I expect that high score (about 0.63) and hacked notebook will be published soon. I am afraid that this hacking trick may generalize well on the private dataset, then many robust solutions will fall drastically (even about 200 places). Will the host check these solutions?",
      "votes": null
    },
    {
      "id": "2769256",
      "postDate": "04/23/2024 08:45:21",
      "content": "<p>To be honest, I was very surprised to see a score of 0.639 and I was looking forward to seeing what improvements were made to the first place to achieve such a huge improvement.</p>",
      "rawMarkdown": "To be honest, I was very surprised to see a score of 0.639 and I was looking forward to seeing what improvements were made to the first place to achieve such a huge improvement.",
      "votes": null
    },
    {
      "id": "2769273",
      "postDate": "04/23/2024 08:55:38",
      "content": "<p>Well, organizers had a chance to change the problematic metric, despite many valid propositions from community, they decided to change the data instead. Will they now need to check manually every top scoring submission?.. I hope not, otherwise this competition will turn into disaster that is even worse than the legendary ICR. </p>",
      "rawMarkdown": "Well, organizers had a chance to change the problematic metric, despite many valid propositions from community, they decided to change the data instead. Will they now need to check manually every top scoring submission?.. I hope not, otherwise this competition will turn into disaster that is even worse than the legendary ICR.",
      "votes": null
    },
    {
      "id": "2769358",
      "postDate": "04/23/2024 09:52:29",
      "content": "<p>So they will manually check if a participants used future information? It is impossible  tbh…</p>",
      "rawMarkdown": "So they will manually check if a participants used future information? It is impossible  tbh...",
      "votes": null
    },
    {
      "id": "2769377",
      "postDate": "04/23/2024 10:07:37",
      "content": "<p>This is already the second time, is it necessary to start the competition again?</p>",
      "rawMarkdown": "This is already the second time, is it necessary to start the competition again?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2769256,
      "author_name": "wenjhuang",
      "author_url": "",
      "post_date": "04/23/2024 08:45:21",
      "content": "<p>To be honest, I was very surprised to see a score of 0.639 and I was looking forward to seeing what improvements were made to the first place to achieve such a huge improvement.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2769273,
      "author_name": "kononenko",
      "author_url": "",
      "post_date": "04/23/2024 08:55:38",
      "content": "<p>Well, organizers had a chance to change the problematic metric, despite many valid propositions from community, they decided to change the data instead. Will they now need to check manually every top scoring submission?.. I hope not, otherwise this competition will turn into disaster that is even worse than the legendary ICR. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2769358,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "04/23/2024 09:52:29",
      "content": "<p>So they will manually check if a participants used future information? It is impossible  tbh…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2769377,
      "author_name": "yunsuxiaozi",
      "author_url": "",
      "post_date": "04/23/2024 10:07:37",
      "content": "<p>This is already the second time, is it necessary to start the competition again?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2769244": "I expect that high score (about 0.63) and hacked notebook will be published soon. I am afraid that this hacking trick may generalize well on the private dataset, then many robust solutions will fall drastically (even about 200 places). Will the host check these solutions?",
    "2769256": "To be honest, I was very surprised to see a score of 0.639 and I was looking forward to seeing what improvements were made to the first place to achieve such a huge improvement.",
    "2769273": "Well, organizers had a chance to change the problematic metric, despite many valid propositions from community, they decided to change the data instead. Will they now need to check manually every top scoring submission?.. I hope not, otherwise this competition will turn into disaster that is even worse than the legendary ICR.",
    "2769358": "So they will manually check if a participants used future information? It is impossible  tbh...",
    "2769377": "This is already the second time, is it necessary to start the competition again?"
  },
  "source": "meta"
}