In this competition there is restriction for testing time that limits this time with one hour. That makes impossible to use many models for blending. In order to decide how many model we can use I suggest to do it in experimental way:
\nHere I will post current estimation for (S/T)*. Lets start with one of my submissions:
\nIn this competition there is restriction for testing time that limits this time with one hour. That makes impossible to use many models for blending. In order to decide how many model we can use I suggest to do it in experimental way:
\nHere I will post current estimation for (S/T)*. Lets start with one of my submissions:
\nI think most of the time for evaluation is for audio clip loading. I've tried blending with two or three models but the time for submission increased only slightly which means prediction is made in a few minutes for the whole dataset. If we share the input feature between models, the submission time won't increase so much.
", "rawMarkdown": "I think most of the time for evaluation is for audio clip loading. I've tried blending with two or three models but the time for submission increased only slightly which means prediction is made in a few minutes for the whole dataset. If we share the input feature between models, the submission time won't increase so much.", "votes": null }, { "id": "985745", "postDate": "08/26/2020 01:51:37", "content": "@koza4ukdmitrij What about profiling each component of your code to see what part needs optimization to cut processing time?
", "rawMarkdown": "koza4ukdmitrij What about profiling each component of your code to see what part needs optimization to cut processing time?", "votes": null }, { "id": "985840", "postDate": "08/26/2020 03:44:29", "content": "Which method of reading audio files is the fastest in your opinion? I met somewhere on the forum that some methods are faster.
", "rawMarkdown": "Which method of reading audio files is the fastest in your opinion? I met somewhere on the forum that some methods are faster.", "votes": null }, { "id": "985900", "postDate": "08/26/2020 05:03:10", "content": "I don't know, how I can do that during submussion. I can add profiling in the inference notebook, but I can see the output for a hidden test set.
", "rawMarkdown": "I don't know, how I can do that during submussion. I can add profiling in the inference notebook, but I can see the output for a hidden test set.", "votes": null }, { "id": "985906", "postDate": "08/26/2020 05:08:56", "content": "Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.
", "rawMarkdown": "Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.", "votes": null }, { "id": "985993", "postDate": "08/26/2020 06:24:35", "content": "In some cases, using CPU Notebook is one way. It takes a lot of time to submit, but you can use it for 9 hours.
\nThis is my beginner's opinion…
Agree, I have to take into account only GPU evaluation, cause GPU is available during final submussion.
", "rawMarkdown": "Agree, I have to take into account only GPU evaluation, cause GPU is available during final submussion.", "votes": null }, { "id": "986425", "postDate": "08/26/2020 13:31:11", "content": "@koza4ukdmitrij not sure if you can print any type of metric while the notebook/code is running on testing.
", "rawMarkdown": "koza4ukdmitrij not sure if you can print any type of metric while the notebook/code is running on testing.", "votes": null }, { "id": "986685", "postDate": "08/26/2020 17:29:10", "content": "I confirm that time for evaluation is mainly for audio clips loading. Pay attention to not load same clip multiple times if you want to ensemble multiple models. I've just tried to ensemble 4 CV5 models (so 20 inferences) to get a baseline and it runs in around 70-80min with GPU.
", "rawMarkdown": "I confirm that time for evaluation is mainly for audio clips loading. Pay attention to not load same clip multiple times if you want to ensemble multiple models. I've just tried to ensemble 4 CV5 models (so 20 inferences) to get a baseline and it runs in around 70-80min with GPU.", "votes": null }, { "id": "987009", "postDate": "08/26/2020 22:59:19", "content": "I'm not sure since I always use librosa.load. I'm not sure how many portions of time are spent on loading, but the total inference time for one model takes around 20 - 22min.
", "rawMarkdown": "I'm not sure since I always use librosa.load. I'm not sure how many portions of time are spent on loading, but the total inference time for one model takes around 20 - 22min.", "votes": null }, { "id": "987014", "postDate": "08/26/2020 23:03:05", "content": "\n\nThank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.
\n
Yep, I think so. One thing I forgot to write above is T0 includes the time to convert the audio clips to audio features. If you are to use different audio features for different models, or to do TTA with different audio features, then it takes more time.
", "rawMarkdown": "> Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.\n\nYep, I think so. One thing I forgot to write above is T0 includes the time to convert the audio clips to audio features. If you are to use different audio features for different models, or to do TTA with different audio features, then it takes more time.", "votes": null } ], "comments": [ { "id": 985635, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/25/2020 22:14:41", "content": "I think most of the time for evaluation is for audio clip loading. I've tried blending with two or three models but the time for submission increased only slightly which means prediction is made in a few minutes for the whole dataset. If we share the input feature between models, the submission time won't increase so much.
", "votes": null, "replies": [ { "id": 985840, "author_name": "sapr3s", "author_url": "", "post_date": "08/26/2020 03:44:29", "content": "Which method of reading audio files is the fastest in your opinion? I met somewhere on the forum that some methods are faster.
", "votes": null, "replies": [ { "id": 987009, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/26/2020 22:59:19", "content": "I'm not sure since I always use librosa.load. I'm not sure how many portions of time are spent on loading, but the total inference time for one model takes around 20 - 22min.
", "votes": null, "replies": [] } ] }, { "id": 985906, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "08/26/2020 05:08:56", "content": "Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.
", "votes": null, "replies": [ { "id": 987014, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/26/2020 23:03:05", "content": "\n\nThank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.
\n
Yep, I think so. One thing I forgot to write above is T0 includes the time to convert the audio clips to audio features. If you are to use different audio features for different models, or to do TTA with different audio features, then it takes more time.
", "votes": null, "replies": [] } ] }, { "id": 986685, "author_name": "mpware", "author_url": "", "post_date": "08/26/2020 17:29:10", "content": "I confirm that time for evaluation is mainly for audio clips loading. Pay attention to not load same clip multiple times if you want to ensemble multiple models. I've just tried to ensemble 4 CV5 models (so 20 inferences) to get a baseline and it runs in around 70-80min with GPU.
", "votes": null, "replies": [] } ] }, { "id": 985745, "author_name": "cv13j0", "author_url": "", "post_date": "08/26/2020 01:51:37", "content": "@koza4ukdmitrij What about profiling each component of your code to see what part needs optimization to cut processing time?
", "votes": null, "replies": [ { "id": 985900, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "08/26/2020 05:03:10", "content": "I don't know, how I can do that during submussion. I can add profiling in the inference notebook, but I can see the output for a hidden test set.
", "votes": null, "replies": [] }, { "id": 986425, "author_name": "cv13j0", "author_url": "", "post_date": "08/26/2020 13:31:11", "content": "@koza4ukdmitrij not sure if you can print any type of metric while the notebook/code is running on testing.
", "votes": null, "replies": [] } ] }, { "id": 985993, "author_name": "teyosan1229", "author_url": "", "post_date": "08/26/2020 06:24:35", "content": "In some cases, using CPU Notebook is one way. It takes a lot of time to submit, but you can use it for 9 hours.
\nThis is my beginner's opinion…
Agree, I have to take into account only GPU evaluation, cause GPU is available during final submussion.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "985594": "In this competition there is restriction for testing time that limits this time with one hour. That makes impossible to use many models for blending. In order to decide how many model we can use I suggest to do it in experimental way:\n- For each submission we know time for submission (T min) and our model's size (S Mb).\n- S/T can be a estimation for size for 1 minute of evaluation.\n- Gathering different S/T that you share with society we can get better estimation (S/T)*. Each separate estimation S/T can be unstable cause of model's variance and submission time estimation error.\n- (S/T)* can help you to decide, how many of your model you can use.\n\n**Here** I will post current estimation for (S/T)*. Lets start with one of my submissions:\n - S=105.61 MB\n - T=24 min (so long!)\n - (S/T)*=(105.61 / 24)=4.4 MB/min", "985635": "I think most of the time for evaluation is for audio clip loading. I've tried blending with two or three models but the time for submission increased only slightly which means prediction is made in a few minutes for the whole dataset. If we share the input feature between models, the submission time won't increase so much.", "985745": "koza4ukdmitrij What about profiling each component of your code to see what part needs optimization to cut processing time?", "985840": "Which method of reading audio files is the fastest in your opinion? I met somewhere on the forum that some methods are faster.", "985900": "I don't know, how I can do that during submussion. I can add profiling in the inference notebook, but I can see the output for a hidden test set.", "985906": "Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.", "985993": "In some cases, using CPU Notebook is one way. It takes a lot of time to submit, but you can use it for 9 hours.\nThis is my beginner's opinion...", "986170": "Agree, I have to take into account only GPU evaluation, cause GPU is available during final submussion.", "986425": "koza4ukdmitrij not sure if you can print any type of metric while the notebook/code is running on testing.", "986685": "I confirm that time for evaluation is mainly for audio clips loading. Pay attention to not load same clip multiple times if you want to ensemble multiple models. I've just tried to ensemble 4 CV5 models (so 20 inferences) to get a baseline and it runs in around 70-80min with GPU.", "987009": "I'm not sure since I always use librosa.load. I'm not sure how many portions of time are spent on loading, but the total inference time for one model takes around 20 - 22min.", "987014": "> Thank you for response! So, I can submit solution without any model, find time T0 and just substruct the T0 from a future time T and (T-T0) will be an estimation for model time only.\n\nYep, I think so. One thing I forgot to write above is T0 includes the time to convert the audio clips to audio features. If you are to use different audio features for different models, or to do TTA with different audio features, then it takes more time." }, "source": "meta" }