I was following birdclef 2023 solutions, and quite a few of them used torchaudio's MelSpectrogram as part of feature extractor on GPU (locally). For kaggle inference, when I try to convert this model (MelSpectrogram->timm->fc), I get weird OpenVino error.
\nIf anyone has helpful hints/solution to solve this it would be great. Currently, I've ditched openvino and inferencing using torch.jit (which i suspect is slower).
", "messages": [ { "id": "2784706", "postDate": "04/30/2024 12:53:38", "content": "I was following birdclef 2023 solutions, and quite a few of them used torchaudio's MelSpectrogram as part of feature extractor on GPU (locally). For kaggle inference, when I try to convert this model (MelSpectrogram->timm->fc), I get weird OpenVino error.
\nIf anyone has helpful hints/solution to solve this it would be great. Currently, I've ditched openvino and inferencing using torch.jit (which i suspect is slower).
", "rawMarkdown": "I was following birdclef 2023 solutions, and quite a few of them used torchaudio's MelSpectrogram as part of feature extractor on GPU (locally). For kaggle inference, when I try to convert this model (MelSpectrogram->timm->fc), I get weird OpenVino error. \n\nIf anyone has helpful hints/solution to solve this it would be great. Currently, I've ditched openvino and inferencing using torch.jit (which i suspect is slower).", "votes": null }, { "id": "2784749", "postDate": "04/30/2024 13:25:42", "content": "I don't know about the openvino issue - but you can pull the melspectrogram pipeline outside your model and do it separate from the openvino.
", "rawMarkdown": "I don't know about the openvino issue - but you can pull the melspectrogram pipeline outside your model and do it separate from the openvino.", "votes": null }, { "id": "2784754", "postDate": "04/30/2024 13:29:11", "content": "Yep, I know that. Just wanted additional flexibility in my train pipeline, hence MelSpec inside nnet.
", "rawMarkdown": "Yep, I know that. Just wanted additional flexibility in my train pipeline, hence MelSpec inside nnet.", "votes": null }, { "id": "2784875", "postDate": "04/30/2024 14:25:41", "content": "This is how I did it and used the output as an input to my inference notebook.
\nTMP_DIR = \n! -p {TMP_DIR}\n!pip wheel openvino --wheel-dir={TMP_DIR} --no-cache-dir\n!pip install wheels/openvino-2024.0.0-14509-cp310-cp310-manylinux2014_x86_64.whl --no-index --find-links wheels/\n!pip install openvino-dev[onnx]\nrand_input = torch.randn(input_shape_of_your_model)\nmodel = ... \n\ninput_names = ... \noutput_names = ... \ntorch.onnx.export(model, rand_input, , input_names=input_names, output_names=output_names)\n!mo --input_model model.onnx --compress_to_fp16 \nDoes your model contain a torchaudio layer?
\nimport torchaudio as T\n\nself.spect_layer= nn(\n T(),\n T(top_db = ),\n)\nLike @pheadrus, when I try onnx conversion without removing the torchaudio layer, there are errors due to operations not implemented in onnx.
", "rawMarkdown": "Does your model contain a torchaudio layer? \n```\nimport torchaudio.transforms as T\n\nself.spect_layer= nn.Sequential(\n T.MelSpectrogram(),\n T.AmplitudeToDB(top_db = 80),\n)\n```\nLike @pheadrus, when I try onnx conversion without removing the torchaudio layer, there are errors due to operations not implemented in onnx.", "votes": null }, { "id": "2785286", "postDate": "04/30/2024 17:48:03", "content": "Yes I am using torchaudio, I have the following ones:
\ntorchaudio.transforms.MelSpectrogram\ntorchaudio.transforms.TimeMasking\ntorchaudio.transforms.FrequencyMasking\nYes, because of the STFT implementation in pytorch.
\nYou can write STFT by yourself with Conv1D proposed by 12th place last year.
\nI tried, which makes inference super slow… maybe I did not use it correctly
@honglihang is it still a dead end converting eca_nfnet model for openvino ? I've read that in your past solution write up (I'm stuck with weird error on my side)
", "rawMarkdown": "honglihang is it still a dead end converting eca_nfnet model for openvino ? I've read that in your past solution write up (I'm stuck with weird error on my side)", "votes": null }, { "id": "2812627", "postDate": "05/14/2024 10:33:22", "content": "I remember last year's 9th place manually modified the eca_nfnet and made it work. He or She left a link in last year's 9th place solution discussion.
", "rawMarkdown": "I remember last year's 9th place manually modified the eca_nfnet and made it work. He or She left a link in last year's 9th place solution discussion.", "votes": null }, { "id": "2824018", "postDate": "05/19/2024 14:44:27", "content": "You can use https://github.com/adobe-research/convmelspec
", "rawMarkdown": "You can use https://github.com/adobe-research/convmelspec", "votes": null } ], "comments": [ { "id": 2784749, "author_name": "llleeeoooh", "author_url": "", "post_date": "04/30/2024 13:25:42", "content": "I don't know about the openvino issue - but you can pull the melspectrogram pipeline outside your model and do it separate from the openvino.
", "votes": null, "replies": [ { "id": 2784754, "author_name": "pheadrus", "author_url": "", "post_date": "04/30/2024 13:29:11", "content": "Yep, I know that. Just wanted additional flexibility in my train pipeline, hence MelSpec inside nnet.
", "votes": null, "replies": [] } ] }, { "id": 2784875, "author_name": "snnclsr", "author_url": "", "post_date": "04/30/2024 14:25:41", "content": "This is how I did it and used the output as an input to my inference notebook.
\nTMP_DIR = \n! -p {TMP_DIR}\n!pip wheel openvino --wheel-dir={TMP_DIR} --no-cache-dir\n!pip install wheels/openvino-2024.0.0-14509-cp310-cp310-manylinux2014_x86_64.whl --no-index --find-links wheels/\n!pip install openvino-dev[onnx]\nrand_input = torch.randn(input_shape_of_your_model)\nmodel = ... \n\ninput_names = ... \noutput_names = ... \ntorch.onnx.export(model, rand_input, , input_names=input_names, output_names=output_names)\n!mo --input_model model.onnx --compress_to_fp16 \nDoes your model contain a torchaudio layer?
\nimport torchaudio as T\n\nself.spect_layer= nn(\n T(),\n T(top_db = ),\n)\nLike @pheadrus, when I try onnx conversion without removing the torchaudio layer, there are errors due to operations not implemented in onnx.
", "votes": null, "replies": [ { "id": 2785286, "author_name": "snnclsr", "author_url": "", "post_date": "04/30/2024 17:48:03", "content": "Yes I am using torchaudio, I have the following ones:
\ntorchaudio.transforms.MelSpectrogram\ntorchaudio.transforms.TimeMasking\ntorchaudio.transforms.FrequencyMasking\nYes, because of the STFT implementation in pytorch.
\nYou can write STFT by yourself with Conv1D proposed by 12th place last year.
\nI tried, which makes inference super slow… maybe I did not use it correctly
@honglihang is it still a dead end converting eca_nfnet model for openvino ? I've read that in your past solution write up (I'm stuck with weird error on my side)
", "votes": null, "replies": [ { "id": 2812627, "author_name": "honglihang", "author_url": "", "post_date": "05/14/2024 10:33:22", "content": "I remember last year's 9th place manually modified the eca_nfnet and made it work. He or She left a link in last year's 9th place solution discussion.
", "votes": null, "replies": [] } ] } ] }, { "id": 2824018, "author_name": "atamazian", "author_url": "", "post_date": "05/19/2024 14:44:27", "content": "You can use https://github.com/adobe-research/convmelspec
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2784706": "I was following birdclef 2023 solutions, and quite a few of them used torchaudio's MelSpectrogram as part of feature extractor on GPU (locally). For kaggle inference, when I try to convert this model (MelSpectrogram->timm->fc), I get weird OpenVino error. \n\nIf anyone has helpful hints/solution to solve this it would be great. Currently, I've ditched openvino and inferencing using torch.jit (which i suspect is slower).", "2784749": "I don't know about the openvino issue - but you can pull the melspectrogram pipeline outside your model and do it separate from the openvino.", "2784754": "Yep, I know that. Just wanted additional flexibility in my train pipeline, hence MelSpec inside nnet.", "2784875": "This is how I did it and used the output as an input to my inference notebook.\n\n- First install the openvino related libraries\n```bash\nTMP_DIR = \"wheels\"\n!mkdir -p {TMP_DIR}\n!pip wheel openvino --wheel-dir={TMP_DIR} --no-cache-dir\n!pip install wheels/openvino-2024.0.0-14509-cp310-cp310-manylinux2014_x86_64.whl --no-index --find-links wheels/\n!pip install openvino-dev[onnx]\n```\n- Convert your model to onnx.\n```python\nrand_input = torch.randn(input_shape_of_your_model)\nmodel = ... # your model\n\ninput_names = ... # the names of the params in your forward function\noutput_names = ... # the output variable name in your forward function\ntorch.onnx.export(model, rand_input, \"model.onnx\", input_names=input_names, output_names=output_names)\n```\n- Convert your model to openvino\n```bash\n!mo --input_model model.onnx --compress_to_fp16 # fp16 is optional\n```", "2784933": "Does your model contain a torchaudio layer? \n```\nimport torchaudio.transforms as T\n\nself.spect_layer= nn.Sequential(\n T.MelSpectrogram(),\n T.AmplitudeToDB(top_db = 80),\n)\n```\nLike @pheadrus, when I try onnx conversion without removing the torchaudio layer, there are errors due to operations not implemented in onnx.", "2785286": "Yes I am using torchaudio, I have the following ones:\n\n```python\ntorchaudio.transforms.MelSpectrogram\ntorchaudio.transforms.TimeMasking\ntorchaudio.transforms.FrequencyMasking\n```", "2809309": "Yes, because of the STFT implementation in pytorch.\nYou can write STFT by yourself with Conv1D proposed by 12th place last year.\nI tried, which makes inference super slow... maybe I did not use it correctly", "2812594": "honglihang is it still a dead end converting eca_nfnet model for openvino ? I've read that in your past solution write up (I'm stuck with weird error on my side)", "2812627": "I remember last year's 9th place manually modified the eca_nfnet and made it work. He or She left a link in last year's 9th place solution discussion.", "2824018": "You can use https://github.com/adobe-research/convmelspec" }, "source": "meta" }