{
  "id": 579455,
  "title": "Clarification on 90-minute CPU runtime limit for private test set",
  "url": "/competitions/birdclef-2025/discussion/579455",
  "author_name": "",
  "post_date": "2025-05-17T16:57:39.043396900Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I have a question regarding the 90-minute CPU runtime limit for submissions.</p>\n<p>My current inference notebook is already quite close to the 90-minute limit when running on the public test set (which I understand is approximately 34% of the total test data).</p>\n<p>I'd like to clarify if this 90-minute CPU runtime limit also applies to the final scoring on the private test set after the competition ends. If so, this would mean our notebooks need to be efficient enough to process roughly three times the amount of data (the full test set) within the same 90-minute window.</p>\n<p>Understanding this is crucial for planning our final solutions, especially concerning model complexity, ensembling strategies, and any pre/post-processing steps.</p>\n<p>Could you please confirm how the runtime limit will be applied to the private leaderboard scoring?</p>\n<p>Thanks in advance for any clarification!</p>",
  "messages": [
    {
      "id": "3204000",
      "postDate": "05/17/2025 16:57:39",
      "content": "<p>Hi everyone,</p>\n<p>I have a question regarding the 90-minute CPU runtime limit for submissions.</p>\n<p>My current inference notebook is already quite close to the 90-minute limit when running on the public test set (which I understand is approximately 34% of the total test data).</p>\n<p>I'd like to clarify if this 90-minute CPU runtime limit also applies to the final scoring on the private test set after the competition ends. If so, this would mean our notebooks need to be efficient enough to process roughly three times the amount of data (the full test set) within the same 90-minute window.</p>\n<p>Understanding this is crucial for planning our final solutions, especially concerning model complexity, ensembling strategies, and any pre/post-processing steps.</p>\n<p>Could you please confirm how the runtime limit will be applied to the private leaderboard scoring?</p>\n<p>Thanks in advance for any clarification!</p>",
      "rawMarkdown": "Hi everyone,\n\nI have a question regarding the 90-minute CPU runtime limit for submissions.\n\nMy current inference notebook is already quite close to the 90-minute limit when running on the public test set (which I understand is approximately 34% of the total test data).\n\nI'd like to clarify if this 90-minute CPU runtime limit also applies to the final scoring on the private test set after the competition ends. If so, this would mean our notebooks need to be efficient enough to process roughly three times the amount of data (the full test set) within the same 90-minute window.\n\nUnderstanding this is crucial for planning our final solutions, especially concerning model complexity, ensembling strategies, and any pre/post-processing steps.\n\nCould you please confirm how the runtime limit will be applied to the private leaderboard scoring?\n\nThanks in advance for any clarification!",
      "votes": null
    },
    {
      "id": "3204021",
      "postDate": "05/17/2025 17:30:20",
      "content": "<p>When you submit a kernel, your inference code runs over all the test datasets (public and private parts). This means your submission has already received its private score, but it will only be shown after the competition deadline. Until then, you can only see its public score. So, you don't need to worry if your submission will be run against a larger dataset later. The only thing that matters is that your submission runs without error and you can see its public score on the leaderboard.</p>",
      "rawMarkdown": "When you submit a kernel, your inference code runs over all the test datasets (public and private parts). This means your submission has already received its private score, but it will only be shown after the competition deadline. Until then, you can only see its public score. So, you don't need to worry if your submission will be run against a larger dataset later. The only thing that matters is that your submission runs without error and you can see its public score on the leaderboard.",
      "votes": null
    },
    {
      "id": "3204034",
      "postDate": "05/17/2025 17:54:28",
      "content": "<p>Thanks a lot for the clarification! That's very helpful to know. It's great that we can push the runtime to its limit, as this will allow for more extensive ensembling of models/checkpoints.</p>",
      "rawMarkdown": "Thanks a lot for the clarification! That's very helpful to know. It's great that we can push the runtime to its limit, as this will allow for more extensive ensembling of models/checkpoints.",
      "votes": null
    },
    {
      "id": "3204850",
      "postDate": "05/19/2025 01:12:41",
      "content": "<p>If your submission is scored, then it has already been scored for private lb.</p>",
      "rawMarkdown": "If your submission is scored, then it has already been scored for private lb.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3204021,
      "author_name": "habedi",
      "author_url": "",
      "post_date": "05/17/2025 17:30:20",
      "content": "<p>When you submit a kernel, your inference code runs over all the test datasets (public and private parts). This means your submission has already received its private score, but it will only be shown after the competition deadline. Until then, you can only see its public score. So, you don't need to worry if your submission will be run against a larger dataset later. The only thing that matters is that your submission runs without error and you can see its public score on the leaderboard.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3204034,
          "author_name": "carloscalvinp",
          "author_url": "",
          "post_date": "05/17/2025 17:54:28",
          "content": "<p>Thanks a lot for the clarification! That's very helpful to know. It's great that we can push the runtime to its limit, as this will allow for more extensive ensembling of models/checkpoints.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3204850,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "05/19/2025 01:12:41",
      "content": "<p>If your submission is scored, then it has already been scored for private lb.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3204000": "Hi everyone,\n\nI have a question regarding the 90-minute CPU runtime limit for submissions.\n\nMy current inference notebook is already quite close to the 90-minute limit when running on the public test set (which I understand is approximately 34% of the total test data).\n\nI'd like to clarify if this 90-minute CPU runtime limit also applies to the final scoring on the private test set after the competition ends. If so, this would mean our notebooks need to be efficient enough to process roughly three times the amount of data (the full test set) within the same 90-minute window.\n\nUnderstanding this is crucial for planning our final solutions, especially concerning model complexity, ensembling strategies, and any pre/post-processing steps.\n\nCould you please confirm how the runtime limit will be applied to the private leaderboard scoring?\n\nThanks in advance for any clarification!",
    "3204021": "When you submit a kernel, your inference code runs over all the test datasets (public and private parts). This means your submission has already received its private score, but it will only be shown after the competition deadline. Until then, you can only see its public score. So, you don't need to worry if your submission will be run against a larger dataset later. The only thing that matters is that your submission runs without error and you can see its public score on the leaderboard.",
    "3204034": "Thanks a lot for the clarification! That's very helpful to know. It's great that we can push the runtime to its limit, as this will allow for more extensive ensembling of models/checkpoints.",
    "3204850": "If your submission is scored, then it has already been scored for private lb."
  },
  "source": "meta"
}