![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8913321%2Fb528bee0b88989ea700181891b81ed23%2Finference.png?generation=1714381758161200&alt=media\")
UPDATE: FIXED (we had cat submission.csv) at the end.
We have been stuck on submission timeouts recently which we cannot explain.
\nWe have already optimized our inference speed quite a bit, and when we test in our kaggle notebook on the unlabelled soundscapes, we observe inference speeds of around 1.5s / 4 minute file. This should take ~27.5 minutes in total for 1100 files. When we submit this, we obtain a notebook timeout error and we see that our notebook is still running ~2h+…
We are using Torch data-loader num_workers=8 with prefetch_factor=2 and ONNX for inference speedup, with an efficientNet_b0 model as a baseline. We don't observe any memory leaks when testing with the unlabelled_soundscapes or other weird behaviour that could lead to our submission timeouts..
What we have tried:
\nWhat can cause this:
\nAny help is appreciated since we could not get any notebook to score yet!
", "messages": [ { "id": "2782324", "postDate": "04/29/2024 09:02:58", "content": "UPDATE: FIXED (we had cat submission.csv) at the end.
We have been stuck on submission timeouts recently which we cannot explain.
\nWe have already optimized our inference speed quite a bit, and when we test in our kaggle notebook on the unlabelled soundscapes, we observe inference speeds of around 1.5s / 4 minute file. This should take ~27.5 minutes in total for 1100 files. When we submit this, we obtain a notebook timeout error and we see that our notebook is still running ~2h+…
We are using Torch data-loader num_workers=8 with prefetch_factor=2 and ONNX for inference speedup, with an efficientNet_b0 model as a baseline. We don't observe any memory leaks when testing with the unlabelled_soundscapes or other weird behaviour that could lead to our submission timeouts..
What we have tried:
\nWhat can cause this:
\nAny help is appreciated since we could not get any notebook to score yet!
", "rawMarkdown": "**UPDATE: FIXED (we had `cat submission.csv`) at the end.**\n\n\nWe have been stuck on submission timeouts recently which we cannot explain.\n\nWe have already optimized our inference speed quite a bit, and when we test in our kaggle notebook on the `unlabelled soundscapes`, we observe inference speeds of around `1.5s / 4 minute file`. This should take ~27.5 minutes in total for 1100 files. When we submit this, we obtain a notebook timeout error and we see that our notebook is still running ~2h+...\n\nWe are using Torch data-loader `num_workers=8` with `prefetch_factor=2` and ONNX for inference speedup, with an efficientNet_b0 model as a baseline. We don't observe any memory leaks when testing with the unlabelled_soundscapes or other weird behaviour that could lead to our submission timeouts..\n\nWhat we have tried:\n\n- Predicting randomly with torch random instead of giving it to the model. (This might mean something is going on with the data loading on the submission notebook, that is different from when we run the Kaggle notebook)\n- Ensured Kaggle notebook is on CPU (Accelerator = None)\n\nWhat can cause this:\n\n- Considerably slower hardware on submission runtime compared to notebook runtime.\n- Different environment on submission notebook? Maybe different package versions / python version that can cause this?\n\nAny help is appreciated since we could not get any notebook to score yet!", "votes": null }, { "id": "2782332", "postDate": "04/29/2024 09:09:19", "content": "Here you can see our Kaggle inference times on the unlabelled_soundscapes:
\n\ntest in our local kaggle notebook
\n
local or on kaggle? You should test on Kaggle.
", "rawMarkdown": "> test in our local kaggle notebook\n\nlocal or on kaggle? You should test on Kaggle.", "votes": null }, { "id": "2782425", "postDate": "04/29/2024 09:52:54", "content": "That was worded a bit confusingly indeed, I meant on Kaggle (removed the local keyword). Testing was done on Kaggle with a screenshot of our inference times above. Locally we even see faster inference times.
", "rawMarkdown": "That was worded a bit confusingly indeed, I meant on Kaggle (removed the local keyword). Testing was done on Kaggle with a screenshot of our inference times above. Locally we even see faster inference times.", "votes": null }, { "id": "2782479", "postDate": "04/29/2024 10:24:23", "content": "OK, you ran your code with 1100 unlabelled soundscapes on Kaggle, and it took 27 minutes to run?
", "rawMarkdown": "OK, you ran your code with 1100 unlabelled soundscapes on Kaggle, and it took 27 minutes to run?", "votes": null }, { "id": "2782641", "postDate": "04/29/2024 12:00:53", "content": "We figured it out. We had !cat submission.csv at the end of our notebook haha, which was fine for 10 samples but not for 1100. That caused the timeout for submission :)
Here you can see our Kaggle inference times on the unlabelled_soundscapes:
\n\ntest in our local kaggle notebook
\n
local or on kaggle? You should test on Kaggle.
", "votes": null, "replies": [ { "id": 2782425, "author_name": "hugodeheer", "author_url": "", "post_date": "04/29/2024 09:52:54", "content": "That was worded a bit confusingly indeed, I meant on Kaggle (removed the local keyword). Testing was done on Kaggle with a screenshot of our inference times above. Locally we even see faster inference times.
", "votes": null, "replies": [ { "id": 2782479, "author_name": "cpmpml", "author_url": "", "post_date": "04/29/2024 10:24:23", "content": "OK, you ran your code with 1100 unlabelled soundscapes on Kaggle, and it took 27 minutes to run?
", "votes": null, "replies": [ { "id": 2782641, "author_name": "hugodeheer", "author_url": "", "post_date": "04/29/2024 12:00:53", "content": "We figured it out. We had !cat submission.csv at the end of our notebook haha, which was fine for 10 samples but not for 1100. That caused the timeout for submission :)