{
  "id": 398660,
  "title": "The same network structure with different weights lead to a huge difference in inference time.",
  "url": "/competitions/birdclef-2023/discussion/398660",
  "author_name": "",
  "post_date": "2023-03-31T05:17:07.521976400Z",
  "votes": 2,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I used the same model, such as tf_efficientnetv2_b2, with one model pre-trained on previous data and the other one not. This means that both models have the same structure, but different weights. The first model received a timeout error, while the second model completed the inference within 40 minutes.</p>\n<p>I'm puzzled about this. Do weights really have such a big impact on inference time? Or do you have any methods to speed up inference?</p>",
  "messages": [
    {
      "id": "2203755",
      "postDate": "03/31/2023 05:17:07",
      "content": "<p>I used the same model, such as tf_efficientnetv2_b2, with one model pre-trained on previous data and the other one not. This means that both models have the same structure, but different weights. The first model received a timeout error, while the second model completed the inference within 40 minutes.</p>\n<p>I'm puzzled about this. Do weights really have such a big impact on inference time? Or do you have any methods to speed up inference?</p>",
      "rawMarkdown": "I used the same model, such as tf_efficientnetv2_b2, with one model pre-trained on previous data and the other one not. This means that both models have the same structure, but different weights. The first model received a timeout error, while the second model completed the inference within 40 minutes.\n\nI'm puzzled about this. Do weights really have such a big impact on inference time? Or do you have any methods to speed up inference?",
      "votes": null
    },
    {
      "id": "2203834",
      "postDate": "03/31/2023 06:53:14",
      "content": "<p>You can refer to this discussion:<br>\n<a href=\"https://www.kaggle.com/competitions/birdclef-2023/discussion/396546\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2023/discussion/396546</a></p>",
      "rawMarkdown": "You can refer to this discussion:\nhttps://www.kaggle.com/competitions/birdclef-2023/discussion/396546",
      "votes": null
    },
    {
      "id": "2204267",
      "postDate": "03/31/2023 13:03:14",
      "content": "<p>Hey, seems like the problem regarding timm models is still unsolved? Would be great to hear back from you! <br>\nBest,<br>\nJan</p>",
      "rawMarkdown": "Hey, seems like the problem regarding timm models is still unsolved? Would be great to hear back from you! \nBest,\nJan",
      "votes": null
    },
    {
      "id": "2204384",
      "postDate": "03/31/2023 14:19:46",
      "content": "<p>Thank you for your reply! However, it seems that the issue still hasn't been resolved in this discussion.</p>",
      "rawMarkdown": "Thank you for your reply! However, it seems that the issue still hasn't been resolved in this discussion.",
      "votes": null
    },
    {
      "id": "2205439",
      "postDate": "04/01/2023 14:52:17",
      "content": "<p>i don't think it's regarding timm, i also used torchvision.models and it didn't work, i think it's pytorch or lightning problem</p>",
      "rawMarkdown": "i don't think it's regarding timm, i also used torchvision.models and it didn't work, i think it's pytorch or lightning problem",
      "votes": null
    },
    {
      "id": "2205473",
      "postDate": "04/01/2023 15:44:17",
      "content": "<p>I didn't use the lightning module and rewrote the inference part of the code, but the problem still exists. It doesn't seem to be a problem with the lightning module</p>",
      "rawMarkdown": "I didn't use the lightning module and rewrote the inference part of the code, but the problem still exists. It doesn't seem to be a problem with the lightning module",
      "votes": null
    },
    {
      "id": "2205872",
      "postDate": "04/02/2023 03:04:19",
      "content": "<p>are you using tensorflow or pytorch?</p>",
      "rawMarkdown": "are you using tensorflow or pytorch?",
      "votes": null
    },
    {
      "id": "2205949",
      "postDate": "04/02/2023 05:56:46",
      "content": "<p>I'm using pytorch.</p>",
      "rawMarkdown": "I'm using pytorch.",
      "votes": null
    },
    {
      "id": "2235564",
      "postDate": "04/26/2023 06:23:14",
      "content": "<p>I was having this issue too, see <a href=\"https://www.kaggle.com/competitions/birdclef-2023/discussion/401584\" target=\"_blank\">this discussion</a></p>\n<p>I haven't had this problem since I used  <code>torch.set_flush_denormal(True)</code> in the training notebook, so I am hoping that is the solution.  The explanation is that the training weights have a lot of values that are very close to zero but not quite zero, which is computationally expensive.</p>",
      "rawMarkdown": "I was having this issue too, see [this discussion](https://www.kaggle.com/competitions/birdclef-2023/discussion/401584)\n\nI haven't had this problem since I used  `torch.set_flush_denormal(True)` in the training notebook, so I am hoping that is the solution.  The explanation is that the training weights have a lot of values that are very close to zero but not quite zero, which is computationally expensive.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2203834,
      "author_name": "aryankhatana",
      "author_url": "",
      "post_date": "03/31/2023 06:53:14",
      "content": "<p>You can refer to this discussion:<br>\n<a href=\"https://www.kaggle.com/competitions/birdclef-2023/discussion/396546\" target=\"_blank\">https://www.kaggle.com/competitions/birdclef-2023/discussion/396546</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 2204267,
          "author_name": "janbrederecke",
          "author_url": "",
          "post_date": "03/31/2023 13:03:14",
          "content": "<p>Hey, seems like the problem regarding timm models is still unsolved? Would be great to hear back from you! <br>\nBest,<br>\nJan</p>",
          "votes": null,
          "replies": [
            {
              "id": 2205439,
              "author_name": "himanshunayal",
              "author_url": "",
              "post_date": "04/01/2023 14:52:17",
              "content": "<p>i don't think it's regarding timm, i also used torchvision.models and it didn't work, i think it's pytorch or lightning problem</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2205473,
                  "author_name": "xiaohuhayou",
                  "author_url": "",
                  "post_date": "04/01/2023 15:44:17",
                  "content": "<p>I didn't use the lightning module and rewrote the inference part of the code, but the problem still exists. It doesn't seem to be a problem with the lightning module</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2205872,
                      "author_name": "himanshunayal",
                      "author_url": "",
                      "post_date": "04/02/2023 03:04:19",
                      "content": "<p>are you using tensorflow or pytorch?</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2205949,
                          "author_name": "xiaohuhayou",
                          "author_url": "",
                          "post_date": "04/02/2023 05:56:46",
                          "content": "<p>I'm using pytorch.</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "id": 2204384,
          "author_name": "xiaohuhayou",
          "author_url": "",
          "post_date": "03/31/2023 14:19:46",
          "content": "<p>Thank you for your reply! However, it seems that the issue still hasn't been resolved in this discussion.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2235564,
      "author_name": "ollypowell",
      "author_url": "",
      "post_date": "04/26/2023 06:23:14",
      "content": "<p>I was having this issue too, see <a href=\"https://www.kaggle.com/competitions/birdclef-2023/discussion/401584\" target=\"_blank\">this discussion</a></p>\n<p>I haven't had this problem since I used  <code>torch.set_flush_denormal(True)</code> in the training notebook, so I am hoping that is the solution.  The explanation is that the training weights have a lot of values that are very close to zero but not quite zero, which is computationally expensive.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2203755": "I used the same model, such as tf_efficientnetv2_b2, with one model pre-trained on previous data and the other one not. This means that both models have the same structure, but different weights. The first model received a timeout error, while the second model completed the inference within 40 minutes.\n\nI'm puzzled about this. Do weights really have such a big impact on inference time? Or do you have any methods to speed up inference?",
    "2203834": "You can refer to this discussion:\nhttps://www.kaggle.com/competitions/birdclef-2023/discussion/396546",
    "2204267": "Hey, seems like the problem regarding timm models is still unsolved? Would be great to hear back from you! \nBest,\nJan",
    "2204384": "Thank you for your reply! However, it seems that the issue still hasn't been resolved in this discussion.",
    "2205439": "i don't think it's regarding timm, i also used torchvision.models and it didn't work, i think it's pytorch or lightning problem",
    "2205473": "I didn't use the lightning module and rewrote the inference part of the code, but the problem still exists. It doesn't seem to be a problem with the lightning module",
    "2205872": "are you using tensorflow or pytorch?",
    "2205949": "I'm using pytorch.",
    "2235564": "I was having this issue too, see [this discussion](https://www.kaggle.com/competitions/birdclef-2023/discussion/401584)\n\nI haven't had this problem since I used  `torch.set_flush_denormal(True)` in the training notebook, so I am hoping that is the solution.  The explanation is that the training weights have a lot of values that are very close to zero but not quite zero, which is computationally expensive."
  },
  "source": "meta"
}