{
  "id": 500274,
  "title": "Inconsistent scoring times?",
  "url": "/competitions/birdclef-2024/discussion/500274",
  "author_name": "",
  "post_date": "2024-05-05T03:29:23.027932100Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Just sent through a number of submissions today and I had 2 that worked. Then I tried 2 at the same time (unsure if that matters) and my first failed. My second is still running. They are all the same script with a different training of the same model. So it should be nearly identical timing to submit? Is anyone able to explain what is going on to me a bit?</p>",
  "messages": [
    {
      "id": "2793868",
      "postDate": "05/05/2024 03:29:23",
      "content": "<p>Just sent through a number of submissions today and I had 2 that worked. Then I tried 2 at the same time (unsure if that matters) and my first failed. My second is still running. They are all the same script with a different training of the same model. So it should be nearly identical timing to submit? Is anyone able to explain what is going on to me a bit?</p>",
      "rawMarkdown": "Just sent through a number of submissions today and I had 2 that worked. Then I tried 2 at the same time (unsure if that matters) and my first failed. My second is still running. They are all the same script with a different training of the same model. So it should be nearly identical timing to submit? Is anyone able to explain what is going on to me a bit?",
      "votes": null
    },
    {
      "id": "2794472",
      "postDate": "05/05/2024 10:08:39",
      "content": "<p>I've experienced the model to be 60 times slower after training. There's a discussion <a href=\"https://www.kaggle.com/competitions/birdclef-2024/discussion/499530\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2024/discussion/499530</a>.</p>\n<p>The reason is the existence of small but non-zero weights, that are processed slowly. I don't know, if this is your case, at least your model is able to process the data within 2 hours. For me, rounding the weights helped a lot, and the inference time remains stable.</p>",
      "rawMarkdown": "I've experienced the model to be 60 times slower after training. There's a discussion https://www.kaggle.com/competitions/birdclef-2024/discussion/499530.\n\nThe reason is the existence of small but non-zero weights, that are processed slowly. I don't know, if this is your case, at least your model is able to process the data within 2 hours. For me, rounding the weights helped a lot, and the inference time remains stable.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2794472,
      "author_name": "kdmitrie",
      "author_url": "",
      "post_date": "05/05/2024 10:08:39",
      "content": "<p>I've experienced the model to be 60 times slower after training. There's a discussion <a href=\"https://www.kaggle.com/competitions/birdclef-2024/discussion/499530\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2024/discussion/499530</a>.</p>\n<p>The reason is the existence of small but non-zero weights, that are processed slowly. I don't know, if this is your case, at least your model is able to process the data within 2 hours. For me, rounding the weights helped a lot, and the inference time remains stable.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2793868": "Just sent through a number of submissions today and I had 2 that worked. Then I tried 2 at the same time (unsure if that matters) and my first failed. My second is still running. They are all the same script with a different training of the same model. So it should be nearly identical timing to submit? Is anyone able to explain what is going on to me a bit?",
    "2794472": "I've experienced the model to be 60 times slower after training. There's a discussion https://www.kaggle.com/competitions/birdclef-2024/discussion/499530.\n\nThe reason is the existence of small but non-zero weights, that are processed slowly. I don't know, if this is your case, at least your model is able to process the data within 2 hours. For me, rounding the weights helped a lot, and the inference time remains stable."
  },
  "source": "meta"
}