{
  "id": 401428,
  "title": "How will time limits be enforced on private LB?",
  "url": "/competitions/asl-signs/discussion/401428",
  "author_name": "",
  "post_date": "2023-04-13T06:57:50.094622Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Since it's likely the case that for most models the computation time per sample depends on the number of frames, and that furthermore most top submissions likely get close to the time limit, how can we be sure that a submission which passes the time limit on public LB will also pass the time limit on private LB? Are the distributions of frames for the public and private validation datasets very close to each other, or will the limits on private LB be more lax?</p>",
  "messages": [
    {
      "id": "2220159",
      "postDate": "04/13/2023 06:57:50",
      "content": "<p>Since it's likely the case that for most models the computation time per sample depends on the number of frames, and that furthermore most top submissions likely get close to the time limit, how can we be sure that a submission which passes the time limit on public LB will also pass the time limit on private LB? Are the distributions of frames for the public and private validation datasets very close to each other, or will the limits on private LB be more lax?</p>",
      "rawMarkdown": "Since it's likely the case that for most models the computation time per sample depends on the number of frames, and that furthermore most top submissions likely get close to the time limit, how can we be sure that a submission which passes the time limit on public LB will also pass the time limit on private LB? Are the distributions of frames for the public and private validation datasets very close to each other, or will the limits on private LB be more lax?",
      "votes": null
    },
    {
      "id": "2220181",
      "postDate": "04/13/2023 07:18:16",
      "content": "<p>From previous competition I usually saw that submissions are scored on the whole test dataset (public and private).<br>\nThey show you only the public score but not the private one. Once the competition ends the private score of your submissions is revealed.I hope I didn't miss any discussion topic that said otherwise for this competition.</p>\n<p>I hope to hear from the organizers anyways because knowing how scoring is handled is critical for participants.</p>",
      "rawMarkdown": "From previous competition I usually saw that submissions are scored on the whole test dataset (public and private).\nThey show you only the public score but not the private one. Once the competition ends the private score of your submissions is revealed.I hope I didn't miss any discussion topic that said otherwise for this competition.\n\nI hope to hear from the organizers anyways because knowing how scoring is handled is critical for participants.",
      "votes": null
    },
    {
      "id": "2220883",
      "postDate": "04/13/2023 18:40:51",
      "content": "<p>I can confirm that this is how we have done it in previous competitions we have hosted.</p>\n<p>The whole test set is used for evaluation, but scores are only given for the public and private parts separately.</p>",
      "rawMarkdown": "I can confirm that this is how we have done it in previous competitions we have hosted.\n\nThe whole test set is used for evaluation, but scores are only given for the public and private parts separately.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2220181,
      "author_name": "pietromaldini1",
      "author_url": "",
      "post_date": "04/13/2023 07:18:16",
      "content": "<p>From previous competition I usually saw that submissions are scored on the whole test dataset (public and private).<br>\nThey show you only the public score but not the private one. Once the competition ends the private score of your submissions is revealed.I hope I didn't miss any discussion topic that said otherwise for this competition.</p>\n<p>I hope to hear from the organizers anyways because knowing how scoring is handled is critical for participants.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2220883,
          "author_name": "mdecoster",
          "author_url": "",
          "post_date": "04/13/2023 18:40:51",
          "content": "<p>I can confirm that this is how we have done it in previous competitions we have hosted.</p>\n<p>The whole test set is used for evaluation, but scores are only given for the public and private parts separately.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2220159": "Since it's likely the case that for most models the computation time per sample depends on the number of frames, and that furthermore most top submissions likely get close to the time limit, how can we be sure that a submission which passes the time limit on public LB will also pass the time limit on private LB? Are the distributions of frames for the public and private validation datasets very close to each other, or will the limits on private LB be more lax?",
    "2220181": "From previous competition I usually saw that submissions are scored on the whole test dataset (public and private).\nThey show you only the public score but not the private one. Once the competition ends the private score of your submissions is revealed.I hope I didn't miss any discussion topic that said otherwise for this competition.\n\nI hope to hear from the organizers anyways because knowing how scoring is handled is critical for participants.",
    "2220883": "I can confirm that this is how we have done it in previous competitions we have hosted.\n\nThe whole test set is used for evaluation, but scores are only given for the public and private parts separately."
  },
  "source": "meta"
}