{
  "id": 401587,
  "title": "Faster inference using concurrent ThreadPoolExecutor.",
  "url": "/competitions/birdclef-2023/discussion/401587",
  "author_name": "",
  "post_date": "2023-04-14T01:23:58.872289Z",
  "votes": 22,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I have shared the code <a href=\"https://www.kaggle.com/code/leonshangguan/faster-eb0-sed-model-inference\" target=\"_blank\">here</a>. By using concurrent ThreadPoolExecutor, it saves approximately 25% in inference time</p>",
  "messages": [
    {
      "id": "2221083",
      "postDate": "04/14/2023 01:23:58",
      "content": "<p>I have shared the code <a href=\"https://www.kaggle.com/code/leonshangguan/faster-eb0-sed-model-inference\" target=\"_blank\">here</a>. By using concurrent ThreadPoolExecutor, it saves approximately 25% in inference time</p>",
      "rawMarkdown": "I have shared the code [here](https://www.kaggle.com/code/leonshangguan/faster-eb0-sed-model-inference). By using concurrent ThreadPoolExecutor, it saves approximately 25% in inference time",
      "votes": null
    },
    {
      "id": "2221206",
      "postDate": "04/14/2023 04:47:52",
      "content": "<p>Thanks for sharing your effort! The same with you, I'm struggling inference time, so it can be very helpful for my work.</p>\n<p>And then I published the test dataset for simulating inference time. It  contains 200 test_sound_scape files (copied 1 file with different names). We can estimate inference time.<br>\nI hope our efforts will be rewarded.</p>\n<p><a href=\"https://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test\" target=\"_blank\">https://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test</a></p>",
      "rawMarkdown": "Thanks for sharing your effort! The same with you, I'm struggling inference time, so it can be very helpful for my work.\n\nAnd then I published the test dataset for simulating inference time. It  contains 200 test_sound_scape files (copied 1 file with different names). We can estimate inference time.\nI hope our efforts will be rewarded.\n\nhttps://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test",
      "votes": null
    },
    {
      "id": "2221319",
      "postDate": "04/14/2023 06:25:23",
      "content": "<p>Thanks for sharing! Not sure why I didn't try multi-threading before. I have been too focused on model optimisation and so far have not had much success other than just converting TF model to onnx which slashed inference time by 50%.</p>",
      "rawMarkdown": "Thanks for sharing! Not sure why I didn't try multi-threading before. I have been too focused on model optimisation and so far have not had much success other than just converting TF model to onnx which slashed inference time by 50%.",
      "votes": null
    },
    {
      "id": "2221564",
      "postDate": "04/14/2023 11:22:11",
      "content": "<p>Good job! To my naive understanding, ThreadPoolExecutor allows you to set the batch size = 4, because there are 4 cpu cores. Is this correct?</p>",
      "rawMarkdown": "Good job! To my naive understanding, ThreadPoolExecutor allows you to set the batch size = 4, because there are 4 cpu cores. Is this correct?",
      "votes": null
    },
    {
      "id": "2221599",
      "postDate": "04/14/2023 11:56:24",
      "content": "<p>it allows you to run 4 inferences at the same time. Depending on how you parallelize - it could be 4 test files or 4 models</p>",
      "rawMarkdown": "it allows you to run 4 inferences at the same time. Depending on how you parallelize - it could be 4 test files or 4 models",
      "votes": null
    },
    {
      "id": "2221639",
      "postDate": "04/14/2023 12:28:21",
      "content": "<p>Thanks for your kind clarification. The thing is my own CNN mel classifier can only process one test file at a time. When I set test batch size = 4, \"Notebook Threw Exception\" occurs. I am still trying to figure out the reason.</p>",
      "rawMarkdown": "Thanks for your kind clarification. The thing is my own CNN mel classifier can only process one test file at a time. When I set test batch size = 4, \"Notebook Threw Exception\" occurs. I am still trying to figure out the reason.",
      "votes": null
    },
    {
      "id": "2225598",
      "postDate": "04/18/2023 09:30:01",
      "content": "<p>what does onnx format do to slash inference time ?  Is there quantization being done internally ? </p>",
      "rawMarkdown": "what does onnx format do to slash inference time ?  Is there quantization being done internally ?",
      "votes": null
    },
    {
      "id": "2228844",
      "postDate": "04/20/2023 21:40:12",
      "content": "<p>Thanks for sharing! </p>\n<p>Did you also try to use process pool? It might be much faster because of the way python is implementing multithreading..</p>\n<p>The Devastator.</p>",
      "rawMarkdown": "Thanks for sharing! \n\nDid you also try to use process pool? It might be much faster because of the way python is implementing multithreading..\n\nThe Devastator.",
      "votes": null
    },
    {
      "id": "2229998",
      "postDate": "04/21/2023 22:46:18",
      "content": "<p>no I didn't. not familiar to process pool.</p>",
      "rawMarkdown": "no I didn't. not familiar to process pool.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2221206,
      "author_name": "atsunorifujita",
      "author_url": "",
      "post_date": "04/14/2023 04:47:52",
      "content": "<p>Thanks for sharing your effort! The same with you, I'm struggling inference time, so it can be very helpful for my work.</p>\n<p>And then I published the test dataset for simulating inference time. It  contains 200 test_sound_scape files (copied 1 file with different names). We can estimate inference time.<br>\nI hope our efforts will be rewarded.</p>\n<p><a href=\"https://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test\" target=\"_blank\">https://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2221319,
      "author_name": "nymfree",
      "author_url": "",
      "post_date": "04/14/2023 06:25:23",
      "content": "<p>Thanks for sharing! Not sure why I didn't try multi-threading before. I have been too focused on model optimisation and so far have not had much success other than just converting TF model to onnx which slashed inference time by 50%.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2225598,
          "author_name": "nyleve",
          "author_url": "",
          "post_date": "04/18/2023 09:30:01",
          "content": "<p>what does onnx format do to slash inference time ?  Is there quantization being done internally ? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2221564,
      "author_name": "aphysict",
      "author_url": "",
      "post_date": "04/14/2023 11:22:11",
      "content": "<p>Good job! To my naive understanding, ThreadPoolExecutor allows you to set the batch size = 4, because there are 4 cpu cores. Is this correct?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2221599,
          "author_name": "nymfree",
          "author_url": "",
          "post_date": "04/14/2023 11:56:24",
          "content": "<p>it allows you to run 4 inferences at the same time. Depending on how you parallelize - it could be 4 test files or 4 models</p>",
          "votes": null,
          "replies": [
            {
              "id": 2221639,
              "author_name": "aphysict",
              "author_url": "",
              "post_date": "04/14/2023 12:28:21",
              "content": "<p>Thanks for your kind clarification. The thing is my own CNN mel classifier can only process one test file at a time. When I set test batch size = 4, \"Notebook Threw Exception\" occurs. I am still trying to figure out the reason.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2228844,
      "author_name": "thedevastator",
      "author_url": "",
      "post_date": "04/20/2023 21:40:12",
      "content": "<p>Thanks for sharing! </p>\n<p>Did you also try to use process pool? It might be much faster because of the way python is implementing multithreading..</p>\n<p>The Devastator.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2229998,
          "author_name": "leonshangguan",
          "author_url": "",
          "post_date": "04/21/2023 22:46:18",
          "content": "<p>no I didn't. not familiar to process pool.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2221083": "I have shared the code [here](https://www.kaggle.com/code/leonshangguan/faster-eb0-sed-model-inference). By using concurrent ThreadPoolExecutor, it saves approximately 25% in inference time",
    "2221206": "Thanks for sharing your effort! The same with you, I'm struggling inference time, so it can be very helpful for my work.\n\nAnd then I published the test dataset for simulating inference time. It  contains 200 test_sound_scape files (copied 1 file with different names). We can estimate inference time.\nI hope our efforts will be rewarded.\n\nhttps://www.kaggle.com/datasets/atsunorifujita/birdclef-2023-test",
    "2221319": "Thanks for sharing! Not sure why I didn't try multi-threading before. I have been too focused on model optimisation and so far have not had much success other than just converting TF model to onnx which slashed inference time by 50%.",
    "2221564": "Good job! To my naive understanding, ThreadPoolExecutor allows you to set the batch size = 4, because there are 4 cpu cores. Is this correct?",
    "2221599": "it allows you to run 4 inferences at the same time. Depending on how you parallelize - it could be 4 test files or 4 models",
    "2221639": "Thanks for your kind clarification. The thing is my own CNN mel classifier can only process one test file at a time. When I set test batch size = 4, \"Notebook Threw Exception\" occurs. I am still trying to figure out the reason.",
    "2225598": "what does onnx format do to slash inference time ?  Is there quantization being done internally ?",
    "2228844": "Thanks for sharing! \n\nDid you also try to use process pool? It might be much faster because of the way python is implementing multithreading..\n\nThe Devastator.",
    "2229998": "no I didn't. not familiar to process pool."
  },
  "source": "meta"
}