{
  "id": 415322,
  "title": "How long does it take for you to submit the answer? Why did I submit it for a long time and finally got 0 points?",
  "url": "/competitions/asl-fingerspelling/discussion/415322",
  "author_name": "",
  "post_date": "2023-06-06T02:44:43.653505Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>How long does it take to score after submitting answers</p>",
  "messages": [
    {
      "id": "2289215",
      "postDate": "06/06/2023 02:44:43",
      "content": "<p>How long does it take to score after submitting answers</p>",
      "rawMarkdown": "How long does it take to score after submitting answers",
      "votes": null
    },
    {
      "id": "2289415",
      "postDate": "06/06/2023 06:27:39",
      "content": "<p>Hi Chaton,</p>\n<p>That really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).</p>\n<p>As for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference</a> , for example, gets a zero score.</p>",
      "rawMarkdown": "Hi Chaton,\n\nThat really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).\n\nAs for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook [https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference](url) , for example, gets a zero score.",
      "votes": null
    },
    {
      "id": "2290969",
      "postDate": "06/07/2023 08:25:46",
      "content": "<p>Hi Chaton,</p>\n<p>Someone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: <br>\n111111111<br>\non correct phrase:<br>\nlap<br>\nthen the length is 3 and the distance is 9, then the score would be:<br>\n(3-9)/3 = -3<br>\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.</p>\n<p>It can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.</p>",
      "rawMarkdown": "Hi Chaton,\n\nSomeone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: \n111111111\non correct phrase:\nlap\nthen the length is 3 and the distance is 9, then the score would be:\n(3-9)/3 = -3\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.\n\nIt can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.",
      "votes": null
    },
    {
      "id": "2291237",
      "postDate": "06/07/2023 12:30:45",
      "content": "<p>This is also my interpretation</p>",
      "rawMarkdown": "This is also my interpretation",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2289415,
      "author_name": "wonderingalice",
      "author_url": "",
      "post_date": "06/06/2023 06:27:39",
      "content": "<p>Hi Chaton,</p>\n<p>That really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).</p>\n<p>As for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference</a> , for example, gets a zero score.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2290969,
      "author_name": "thefinniest",
      "author_url": "",
      "post_date": "06/07/2023 08:25:46",
      "content": "<p>Hi Chaton,</p>\n<p>Someone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: <br>\n111111111<br>\non correct phrase:<br>\nlap<br>\nthen the length is 3 and the distance is 9, then the score would be:<br>\n(3-9)/3 = -3<br>\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.</p>\n<p>It can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2291237,
          "author_name": "mdecoster",
          "author_url": "",
          "post_date": "06/07/2023 12:30:45",
          "content": "<p>This is also my interpretation</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2289215": "How long does it take to score after submitting answers",
    "2289415": "Hi Chaton,\n\nThat really depends on the size of your model. After you submit your model, it is evaluated on all samples in the test set. So the scoring time is  proportional to the average runtime per sample of your TfLite model  (on the hardware that is being used for scoring, which we don't know).\n\nAs for the zero score: scores are truncated or rounded (don't know which) at three significant digits, so if that results in a score that is lower than 0.001, you get a zero score. The random untrained model in my notebook [https://www.kaggle.com/code/wonderingalice/working-sample-submission-and-inference](url) , for example, gets a zero score.",
    "2290969": "Hi Chaton,\n\nSomeone else may correct me but I think that the metric for the score in this competition (N - D)/N where N is length of correct phrase and D is the levenshtein distance, allows for negative scores. For example if you predict: \n111111111\non correct phrase:\nlap\nthen the length is 3 and the distance is 9, then the score would be:\n(3-9)/3 = -3\nand the leaderboard seems to have 0 as the lowest score rather than allowing negatives.\n\nIt can happen that your model predicts very long phrases where there may be only shorter correct phrases. This can pull down your score towards or below 0.",
    "2291237": "This is also my interpretation"
  },
  "source": "meta"
}