{
  "id": 585298,
  "title": "Do you need more metrics?",
  "url": "/competitions/aeroclub-recsys-2025/discussion/585298",
  "author_name": "",
  "post_date": "2025-06-19T11:08:53.274367Z",
  "votes": 6,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Some of you may be wondering — why HitRate@3?</p>\n<p>The answer is simple: we chose this metric because it closely reflects the business objective behind the problem  solving. Also, Kaggle requires us to define only 1 official metric — it's a technical limitation of the platform.</p>\n<p>That said, we fully understand that HitRate@3 is far from ideal when it comes to evaluating ranking models, especially when item list lengths vary.</p>\n<p>Question to participants:</p>\n<p>Would you find it useful to have an additional evaluation metric (MRR) to assess your submissions on an alternative leaderboard?</p>\n<p>To improve the participant experience, I'm considering adding a shadow leaderboard that would evaluate your submissions using a different metric — most likely MRR (Mean Reciprocal Rank).</p>\n<p>⚠️ Just to be clear: this additional metric will not be used to determine the competition winners. It’s simply an extra signal to help you better validate your models on the same test data.</p>\n<p>Let me know if you'd find this helpful…</p>",
  "messages": [
    {
      "id": "3227851",
      "postDate": "06/19/2025 11:08:53",
      "content": "<p>Some of you may be wondering — why HitRate@3?</p>\n<p>The answer is simple: we chose this metric because it closely reflects the business objective behind the problem  solving. Also, Kaggle requires us to define only 1 official metric — it's a technical limitation of the platform.</p>\n<p>That said, we fully understand that HitRate@3 is far from ideal when it comes to evaluating ranking models, especially when item list lengths vary.</p>\n<p>Question to participants:</p>\n<p>Would you find it useful to have an additional evaluation metric (MRR) to assess your submissions on an alternative leaderboard?</p>\n<p>To improve the participant experience, I'm considering adding a shadow leaderboard that would evaluate your submissions using a different metric — most likely MRR (Mean Reciprocal Rank).</p>\n<p>⚠️ Just to be clear: this additional metric will not be used to determine the competition winners. It’s simply an extra signal to help you better validate your models on the same test data.</p>\n<p>Let me know if you'd find this helpful…</p>",
      "rawMarkdown": "Some of you may be wondering — why HitRate@3?\n\nThe answer is simple: we chose this metric because it closely reflects the business objective behind the problem  solving. Also, Kaggle requires us to define only 1 official metric — it's a technical limitation of the platform.\n\nThat said, we fully understand that HitRate@3 is far from ideal when it comes to evaluating ranking models, especially when item list lengths vary.\n\nQuestion to participants:\n\nWould you find it useful to have an additional evaluation metric (MRR) to assess your submissions on an alternative leaderboard?\n\nTo improve the participant experience, I'm considering adding a shadow leaderboard that would evaluate your submissions using a different metric — most likely MRR (Mean Reciprocal Rank).\n\n⚠️ Just to be clear: this additional metric will not be used to determine the competition winners. It’s simply an extra signal to help you better validate your models on the same test data.\n\nLet me know if you'd find this helpful...",
      "votes": null
    },
    {
      "id": "3229649",
      "postDate": "06/21/2025 21:20:02",
      "content": "<p>Congratulations! You're a master now</p>",
      "rawMarkdown": "Congratulations! You're a master now",
      "votes": null
    },
    {
      "id": "3229657",
      "postDate": "06/21/2025 21:59:08",
      "content": "<p>oops… 😂</p>",
      "rawMarkdown": "oops... 😂",
      "votes": null
    },
    {
      "id": "3246137",
      "postDate": "07/10/2025 12:10:10",
      "content": "<p>So will the shadow leaderboard be created?</p>",
      "rawMarkdown": "So will the shadow leaderboard be created?",
      "votes": null
    },
    {
      "id": "3246185",
      "postDate": "07/10/2025 14:15:10",
      "content": "<p>Thanks for highlighting back the topic <a href=\"https://www.kaggle.com/mango789\" target=\"_blank\">@mango789</a>. After <a href=\"https://www.kaggle.com/discussions/competition-hosting/585297\" target=\"_blank\">consultation</a> with Kaggle team and observing current leaderboard for 20+ days we decided to stay in one metric classic Kaggle setup. <strong>We won't overcomplicate the competition setup with additional metric and data.</strong> It seems that current setup, data split and metric are balanced enough. </p>",
      "rawMarkdown": "Thanks for highlighting back the topic @mango789. After [consultation](https://www.kaggle.com/discussions/competition-hosting/585297) with Kaggle team and observing current leaderboard for 20+ days we decided to stay in one metric classic Kaggle setup. **We won't overcomplicate the competition setup with additional metric and data.** It seems that current setup, data split and metric are balanced enough.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3229649,
      "author_name": "antonoof",
      "author_url": "",
      "post_date": "06/21/2025 21:20:02",
      "content": "<p>Congratulations! You're a master now</p>",
      "votes": null,
      "replies": [
        {
          "id": 3229657,
          "author_name": "samvelkoch",
          "author_url": "",
          "post_date": "06/21/2025 21:59:08",
          "content": "<p>oops… 😂</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3246137,
      "author_name": "mango789",
      "author_url": "",
      "post_date": "07/10/2025 12:10:10",
      "content": "<p>So will the shadow leaderboard be created?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3246185,
          "author_name": "samvelkoch",
          "author_url": "",
          "post_date": "07/10/2025 14:15:10",
          "content": "<p>Thanks for highlighting back the topic <a href=\"https://www.kaggle.com/mango789\" target=\"_blank\">@mango789</a>. After <a href=\"https://www.kaggle.com/discussions/competition-hosting/585297\" target=\"_blank\">consultation</a> with Kaggle team and observing current leaderboard for 20+ days we decided to stay in one metric classic Kaggle setup. <strong>We won't overcomplicate the competition setup with additional metric and data.</strong> It seems that current setup, data split and metric are balanced enough. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3227851": "Some of you may be wondering — why HitRate@3?\n\nThe answer is simple: we chose this metric because it closely reflects the business objective behind the problem  solving. Also, Kaggle requires us to define only 1 official metric — it's a technical limitation of the platform.\n\nThat said, we fully understand that HitRate@3 is far from ideal when it comes to evaluating ranking models, especially when item list lengths vary.\n\nQuestion to participants:\n\nWould you find it useful to have an additional evaluation metric (MRR) to assess your submissions on an alternative leaderboard?\n\nTo improve the participant experience, I'm considering adding a shadow leaderboard that would evaluate your submissions using a different metric — most likely MRR (Mean Reciprocal Rank).\n\n⚠️ Just to be clear: this additional metric will not be used to determine the competition winners. It’s simply an extra signal to help you better validate your models on the same test data.\n\nLet me know if you'd find this helpful...",
    "3229649": "Congratulations! You're a master now",
    "3229657": "oops... 😂",
    "3246137": "So will the shadow leaderboard be created?",
    "3246185": "Thanks for highlighting back the topic @mango789. After [consultation](https://www.kaggle.com/discussions/competition-hosting/585297) with Kaggle team and observing current leaderboard for 20+ days we decided to stay in one metric classic Kaggle setup. **We won't overcomplicate the competition setup with additional metric and data.** It seems that current setup, data split and metric are balanced enough."
  },
  "source": "meta"
}