{
  "id": 510076,
  "title": "Optimize inference with openvino and multi threading for pytorch models",
  "url": "/competitions/birdclef-2024/discussion/510076",
  "author_name": "",
  "post_date": "2024-06-04T20:50:24.112444800Z",
  "votes": 6,
  "comment_count": 6,
  "views": 0,
  "content": "<p>So this took me some time to figure out. Sharing for help. </p>\n<p>I have converted pyt models to onnx and then to openvino. This way, the inference times drop sharply. I could fit upto  8 models this way (<a href=\"https://www.kaggle.com/code/lmhongkhnh/birdclef2024-multithread-with-openvino-inference\" target=\"_blank\">Reference notebook</a>). Check out the notebook below. </p>\n<p><a href=\"https://www.kaggle.com/code/arindamroy23/inference-optimized-with-openvino-and-multithread\" target=\"_blank\">Optimize inference with openvino and multi threading for pytorch models</a></p>\n<p>** This is not a high scoring notebook. Hopefully is not against kaggle rules. It is just for sharing inference method</p>",
  "messages": [
    {
      "id": "2855529",
      "postDate": "06/04/2024 20:50:24",
      "content": "<p>So this took me some time to figure out. Sharing for help. </p>\n<p>I have converted pyt models to onnx and then to openvino. This way, the inference times drop sharply. I could fit upto  8 models this way (<a href=\"https://www.kaggle.com/code/lmhongkhnh/birdclef2024-multithread-with-openvino-inference\" target=\"_blank\">Reference notebook</a>). Check out the notebook below. </p>\n<p><a href=\"https://www.kaggle.com/code/arindamroy23/inference-optimized-with-openvino-and-multithread\" target=\"_blank\">Optimize inference with openvino and multi threading for pytorch models</a></p>\n<p>** This is not a high scoring notebook. Hopefully is not against kaggle rules. It is just for sharing inference method</p>",
      "rawMarkdown": "So this took me some time to figure out. Sharing for help. \n\nI have converted pyt models to onnx and then to openvino. This way, the inference times drop sharply. I could fit upto  8 models this way ([Reference notebook](https://www.kaggle.com/code/lmhongkhnh/birdclef2024-multithread-with-openvino-inference)). Check out the notebook below. \n\n[Optimize inference with openvino and multi threading for pytorch models](https://www.kaggle.com/code/arindamroy23/inference-optimized-with-openvino-and-multithread)\n\n** This is not a high scoring notebook. Hopefully is not against kaggle rules. It is just for sharing inference method",
      "votes": null
    },
    {
      "id": "2855710",
      "postDate": "06/05/2024 01:27:05",
      "content": "<p>Thank you for sharing! This is very helpful to me. Have you confirmed whether it is possible to use two eca_nfnet_l0 for inference?</p>",
      "rawMarkdown": "Thank you for sharing! This is very helpful to me. Have you confirmed whether it is possible to use two eca_nfnet_l0 for inference?",
      "votes": null
    },
    {
      "id": "2856115",
      "postDate": "06/05/2024 06:58:10",
      "content": "<p>nfnet has a layer that is not supported by openvino. But in those cases, I run the model with onnx. </p>",
      "rawMarkdown": "nfnet has a layer that is not supported by openvino. But in those cases, I run the model with onnx.",
      "votes": null
    },
    {
      "id": "2856362",
      "postDate": "06/05/2024 08:32:39",
      "content": "<pre><code> node  onnx_model.graph.node:\n     node.op_type == :\n         attribute  node.attribute:\n             attribute.name == :\n                 attribute.i == :\n                    node.output.remove(node.output[])\n                    node.output.remove(node.output[])\n                attribute.i = \n</code></pre>\n<p>You can try to modify BN layer in the model in this way, and after I do so, I can use OpenVINO for inference of nfnet.</p>",
      "rawMarkdown": "```python\nfor node in onnx_model.graph.node:\n    if node.op_type == \"BatchNormalization\":\n        for attribute in node.attribute:\n            if attribute.name == 'training_mode':\n                if attribute.i == 1:\n                    node.output.remove(node.output[1])\n                    node.output.remove(node.output[1])\n                attribute.i = 0\n```\nYou can try to modify BN layer in the model in this way, and after I do so, I can use OpenVINO for inference of nfnet.",
      "votes": null
    },
    {
      "id": "2856511",
      "postDate": "06/05/2024 10:31:12",
      "content": "<p>How much testing time does it take using nfnet with openvino?</p>",
      "rawMarkdown": "How much testing time does it take using nfnet with openvino?",
      "votes": null
    },
    {
      "id": "2856799",
      "postDate": "06/05/2024 13:57:58",
      "content": "<p>Testing the inference of 10 audio files took 70 seconds.</p>",
      "rawMarkdown": "Testing the inference of 10 audio files took 70 seconds.",
      "votes": null
    },
    {
      "id": "2857309",
      "postDate": "06/05/2024 19:44:06",
      "content": "<p>Intriguing, thank you</p>",
      "rawMarkdown": "Intriguing, thank you",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2855710,
      "author_name": "shawntung",
      "author_url": "",
      "post_date": "06/05/2024 01:27:05",
      "content": "<p>Thank you for sharing! This is very helpful to me. Have you confirmed whether it is possible to use two eca_nfnet_l0 for inference?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2856115,
          "author_name": "arindamroy23",
          "author_url": "",
          "post_date": "06/05/2024 06:58:10",
          "content": "<p>nfnet has a layer that is not supported by openvino. But in those cases, I run the model with onnx. </p>",
          "votes": null,
          "replies": [
            {
              "id": 2856362,
              "author_name": "shawntung",
              "author_url": "",
              "post_date": "06/05/2024 08:32:39",
              "content": "<pre><code> node  onnx_model.graph.node:\n     node.op_type == :\n         attribute  node.attribute:\n             attribute.name == :\n                 attribute.i == :\n                    node.output.remove(node.output[])\n                    node.output.remove(node.output[])\n                attribute.i = \n</code></pre>\n<p>You can try to modify BN layer in the model in this way, and after I do so, I can use OpenVINO for inference of nfnet.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2856511,
                  "author_name": "tanxxx",
                  "author_url": "",
                  "post_date": "06/05/2024 10:31:12",
                  "content": "<p>How much testing time does it take using nfnet with openvino?</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2856799,
                      "author_name": "shawntung",
                      "author_url": "",
                      "post_date": "06/05/2024 13:57:58",
                      "content": "<p>Testing the inference of 10 audio files took 70 seconds.</p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2857309,
      "author_name": "tztang",
      "author_url": "",
      "post_date": "06/05/2024 19:44:06",
      "content": "<p>Intriguing, thank you</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2855529": "So this took me some time to figure out. Sharing for help. \n\nI have converted pyt models to onnx and then to openvino. This way, the inference times drop sharply. I could fit upto  8 models this way ([Reference notebook](https://www.kaggle.com/code/lmhongkhnh/birdclef2024-multithread-with-openvino-inference)). Check out the notebook below. \n\n[Optimize inference with openvino and multi threading for pytorch models](https://www.kaggle.com/code/arindamroy23/inference-optimized-with-openvino-and-multithread)\n\n** This is not a high scoring notebook. Hopefully is not against kaggle rules. It is just for sharing inference method",
    "2855710": "Thank you for sharing! This is very helpful to me. Have you confirmed whether it is possible to use two eca_nfnet_l0 for inference?",
    "2856115": "nfnet has a layer that is not supported by openvino. But in those cases, I run the model with onnx.",
    "2856362": "```python\nfor node in onnx_model.graph.node:\n    if node.op_type == \"BatchNormalization\":\n        for attribute in node.attribute:\n            if attribute.name == 'training_mode':\n                if attribute.i == 1:\n                    node.output.remove(node.output[1])\n                    node.output.remove(node.output[1])\n                attribute.i = 0\n```\nYou can try to modify BN layer in the model in this way, and after I do so, I can use OpenVINO for inference of nfnet.",
    "2856511": "How much testing time does it take using nfnet with openvino?",
    "2856799": "Testing the inference of 10 audio files took 70 seconds.",
    "2857309": "Intriguing, thank you"
  },
  "source": "meta"
}