![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F36054%2Feb3b16d1d1655c3fa3701077051cfae1%2Fsubmissions.jpg?generation=1596464010455873&alt=media\")
The competition rule states:\n> This is a Code Competition\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit to Competition\" button to be active after a commit, the following conditions must be met:\nCPU Notebook <= 9 hours run-time\nGPU Notebook <= 2 hours run-time
\n\n[Hypothesis]:\nIf your notebook times out, it will still say the submission \"succeeded\", the remaining rows would default to \"nocall.\" You might actually have a good model, it might just be too slow.
\n\n@snovik1975 pointed this out in this thread: https://www.kaggle.com/c/birdsong-recognition/discussion/163528
\n\n[Observation 1]: \nMy efficientnet model kept scoring model kept scoring ~0.557 on public LB despite scoring ~ 0.67 micro F1 on stratified k-fold holdout set. After I made some tweaks speeding up prediction loop, the same model jumped to my current LB score of ~0.568.
\n\n[Observation 2]: \nThe correlation coefficient between my 0.568 efficientnet and the 0.568 ResNest public notebook is ~0.76, this \"should\" be diverse enough that averaging the two models would improve score, which is the case on local validation. Surprise surprise (not really at this point), the ensemble scores 0.559 public LB. I think what is happening is that the ensemble is now too slow, the notebook times out and the remaining rows default to \"nocall.\"
\n\nIf you're model is stuck at ~0.55x, despite improved local CV score, check and make sure it's actually able to infer labels on all 150 files within the allocated 2 GPU hours or 9 CPU hours. You might actually have a good model, the inference routing might just be too slow.
\n\nGood luck all!
", "messages": [ { "id": "956380", "postDate": "08/03/2020 13:02:15", "content": "The competition rule states:\n> This is a Code Competition\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit to Competition\" button to be active after a commit, the following conditions must be met:\nCPU Notebook <= 9 hours run-time\nGPU Notebook <= 2 hours run-time
\n\n[Hypothesis]:\nIf your notebook times out, it will still say the submission \"succeeded\", the remaining rows would default to \"nocall.\" You might actually have a good model, it might just be too slow.
\n\n@snovik1975 pointed this out in this thread: https://www.kaggle.com/c/birdsong-recognition/discussion/163528
\n\n[Observation 1]: \nMy efficientnet model kept scoring model kept scoring ~0.557 on public LB despite scoring ~ 0.67 micro F1 on stratified k-fold holdout set. After I made some tweaks speeding up prediction loop, the same model jumped to my current LB score of ~0.568.
\n\n[Observation 2]: \nThe correlation coefficient between my 0.568 efficientnet and the 0.568 ResNest public notebook is ~0.76, this \"should\" be diverse enough that averaging the two models would improve score, which is the case on local validation. Surprise surprise (not really at this point), the ensemble scores 0.559 public LB. I think what is happening is that the ensemble is now too slow, the notebook times out and the remaining rows default to \"nocall.\"
\n\nIf you're model is stuck at ~0.55x, despite improved local CV score, check and make sure it's actually able to infer labels on all 150 files within the allocated 2 GPU hours or 9 CPU hours. You might actually have a good model, the inference routing might just be too slow.
\n\nGood luck all!
", "rawMarkdown": "The competition rule states:\n> This is a Code Competition\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit to Competition\" button to be active after a commit, the following conditions must be met:\nCPU Notebook <= 9 hours run-time\nGPU Notebook <= 2 hours run-time\n\n[Hypothesis]:\nIf your notebook times out, it will still say the submission \"succeeded\", the remaining rows would default to \"nocall.\" You might actually have a good model, it might just be too slow.\n\n @snovik1975 pointed this out in this thread: https://www.kaggle.com/c/birdsong-recognition/discussion/163528\n\n[Observation 1]: \nMy efficientnet model kept scoring model kept scoring ~0.557 on public LB despite scoring ~ 0.67 micro F1 on stratified k-fold holdout set. After I made some tweaks speeding up prediction loop, the same model jumped to my current LB score of ~0.568.\n\n[Observation 2]: \nThe correlation coefficient between my 0.568 efficientnet and the 0.568 ResNest public notebook is ~0.76, this \"should\" be diverse enough that averaging the two models would improve score, which is the case on local validation. Surprise surprise (not really at this point), the ensemble scores 0.559 public LB. I think what is happening is that the ensemble is now too slow, the notebook times out and the remaining rows default to \"nocall.\"\n\nIf you're model is stuck at ~0.55x, despite improved local CV score, check and make sure it's actually able to infer labels on all 150 files within the allocated 2 GPU hours or 9 CPU hours. You might actually have a good model, the inference routing might just be too slow.\n\nGood luck all!", "votes": null }, { "id": "956445", "postDate": "08/03/2020 14:04:08", "content": "No. If the model does not have enough time, it will be labeled Notebook Timeout.\nAnd you can always see how many minutes the result will appear.
", "rawMarkdown": "No. If the model does not have enough time, it will be labeled Notebook Timeout.\nAnd you can always see how many minutes the result will appear.", "votes": null }, { "id": "956457", "postDate": "08/03/2020 14:20:00", "content": "Actually mine all says \"succeeded,\" even the ones that clearly failed.
\n\nHere's a fresh example, there was an error that caused the versions 1 and 3 to fail, it still says \"succeeded,\" once I fixed the error on version 4, then it score 0.568:\n
Note how many minutes later the rating appears, I have it for 32 minutes\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3577701%2F363a68ef194976db9a7576a6e46c0bdd%2F2020-08-03.png?generation=1596464754086486&alt=media\")
How do you check how long the notebook ran to create the submission?
", "rawMarkdown": "How do you check how long the notebook ran to create the submission?", "votes": null }, { "id": "956498", "postDate": "08/03/2020 14:46:03", "content": "Just watch how long after sending the solution the result will appear
", "rawMarkdown": "Just watch how long after sending the solution the result will appear", "votes": null }, { "id": "956517", "postDate": "08/03/2020 15:15:53", "content": "Alright will time my next submission. Although, I'm fairly sure I've had submissions take more 2 hours. :)
\n\nIt would be nice to receive information how long a it took to submit a notebook.
", "rawMarkdown": "Alright will time my next submission. Although, I'm fairly sure I've had submissions take more 2 hours. :)\n\nIt would be nice to receive information how long a it took to submit a notebook.", "votes": null }, { "id": "956716", "postDate": "08/03/2020 18:00:15", "content": "Kernel will report timeout and not succeeded. My 2 first attempts exceeded the 2 hours due to librosa.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2F0a1a15e6a9846415633a66a58aa48eac%2Ftimeout.png?generation=1596477592282086&alt=media\")
Good to know! Thanks for sharing.
", "rawMarkdown": "Good to know! Thanks for sharing.", "votes": null }, { "id": "957830", "postDate": "08/04/2020 15:26:22", "content": "Thanks for sharing Mendrika. Can you tell us what kind of error you had (and fixed) on you notebook?
", "rawMarkdown": "Thanks for sharing Mendrika. Can you tell us what kind of error you had (and fixed) on you notebook?", "votes": null }, { "id": "992020", "postDate": "08/30/2020 20:15:05", "content": "\n\nMy 2 first attempts exceeded the 2 hours due to librosa.
\n
Why is librosa the cause of the timeout?
", "rawMarkdown": "> My 2 first attempts exceeded the 2 hours due to librosa.\n\nWhy is librosa the cause of the timeout?", "votes": null }, { "id": "992057", "postDate": "08/30/2020 21:36:06", "content": "I'm not 100% sure but it could come from an issue in resolving the final underlying library for MP3 support. There is a warning before fallback to another library that works. It seems to take time on Kaggle platform.
\nhttps://stackoverflow.com/questions/59854527/librosa-load-takes-too-long-to-loadsample-mp3-files
\nI'm using torchaudio and it's around 2x to 3x faster to load the same MP3.
No. If the model does not have enough time, it will be labeled Notebook Timeout.\nAnd you can always see how many minutes the result will appear.
", "votes": null, "replies": [ { "id": 956457, "author_name": "ramarlina", "author_url": "", "post_date": "08/03/2020 14:20:00", "content": "Actually mine all says \"succeeded,\" even the ones that clearly failed.
\n\nHere's a fresh example, there was an error that caused the versions 1 and 3 to fail, it still says \"succeeded,\" once I fixed the error on version 4, then it score 0.568:\n
Note how many minutes later the rating appears, I have it for 32 minutes\n
How do you check how long the notebook ran to create the submission?
", "votes": null, "replies": [] }, { "id": 956498, "author_name": "vlomme", "author_url": "", "post_date": "08/03/2020 14:46:03", "content": "Just watch how long after sending the solution the result will appear
", "votes": null, "replies": [] }, { "id": 956517, "author_name": "ramarlina", "author_url": "", "post_date": "08/03/2020 15:15:53", "content": "Alright will time my next submission. Although, I'm fairly sure I've had submissions take more 2 hours. :)
\n\nIt would be nice to receive information how long a it took to submit a notebook.
", "votes": null, "replies": [] }, { "id": 957830, "author_name": "adriel", "author_url": "", "post_date": "08/04/2020 15:26:22", "content": "Thanks for sharing Mendrika. Can you tell us what kind of error you had (and fixed) on you notebook?
", "votes": null, "replies": [] } ] }, { "id": 956716, "author_name": "mpware", "author_url": "", "post_date": "08/03/2020 18:00:15", "content": "Kernel will report timeout and not succeeded. My 2 first attempts exceeded the 2 hours due to librosa.\n
Good to know! Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 992020, "author_name": "cpmpml", "author_url": "", "post_date": "08/30/2020 20:15:05", "content": "\n\nMy 2 first attempts exceeded the 2 hours due to librosa.
\n
Why is librosa the cause of the timeout?
", "votes": null, "replies": [] }, { "id": 992057, "author_name": "mpware", "author_url": "", "post_date": "08/30/2020 21:36:06", "content": "I'm not 100% sure but it could come from an issue in resolving the final underlying library for MP3 support. There is a warning before fallback to another library that works. It seems to take time on Kaggle platform.
\nhttps://stackoverflow.com/questions/59854527/librosa-load-takes-too-long-to-loadsample-mp3-files
\nI'm using torchaudio and it's around 2x to 3x faster to load the same MP3.