{ "id": 209516, "title": "Submission Time Longer Than Save Time", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/209516", "author_name": "", "post_date": "2021-01-07T19:16:59.281036300Z", "votes": 1, "comment_count": 12, "views": 0, "content": "

Hello,

\n

It's been 7 hours since I submitted my notebook. I'm still getting the \"Notebook Running\" icon from the submission. My model took 5 hours and 29 minutes to save for the first version (where I realized I accidentally had internet enabled, and subsequently resaved last night), and 5 hours and 27 minutes for the second version. Now that I go to submit it, it takes over 1.5 hours longer (right now). I'll post later with the time it finishes at, if it doesn't time out at 9 hours, which is seeming ever more likely.

\n

Is anyone else having this problem? Also, if so, any pointers on how to fix it, and what to expect?

", "messages": [ { "id": "1143159", "postDate": "01/07/2021 19:16:59", "content": "

Hello,

\n

It's been 7 hours since I submitted my notebook. I'm still getting the \"Notebook Running\" icon from the submission. My model took 5 hours and 29 minutes to save for the first version (where I realized I accidentally had internet enabled, and subsequently resaved last night), and 5 hours and 27 minutes for the second version. Now that I go to submit it, it takes over 1.5 hours longer (right now). I'll post later with the time it finishes at, if it doesn't time out at 9 hours, which is seeming ever more likely.

\n

Is anyone else having this problem? Also, if so, any pointers on how to fix it, and what to expect?

", "rawMarkdown": "Hello,\n\nIt's been 7 hours since I submitted my notebook. I'm still getting the \"Notebook Running\" icon from the submission. My model took 5 hours and 29 minutes to save for the first version (where I realized I accidentally had internet enabled, and subsequently resaved last night), and 5 hours and 27 minutes for the second version. Now that I go to submit it, it takes over 1.5 hours longer (right now). I'll post later with the time it finishes at, if it doesn't time out at 9 hours, which is seeming ever more likely.\n\nIs anyone else having this problem? Also, if so, any pointers on how to fix it, and what to expect?", "votes": null }, { "id": "1143209", "postDate": "01/07/2021 19:41:44", "content": "

Maybe there aren't enough servers.

", "rawMarkdown": "Maybe there aren't enough servers.", "votes": null }, { "id": "1143271", "postDate": "01/07/2021 20:20:16", "content": "

Perhaps… One hour later it's still running. Getting worried it might time out.

", "rawMarkdown": "Perhaps... One hour later it's still running. Getting worried it might time out.", "votes": null }, { "id": "1143339", "postDate": "01/07/2021 21:09:34", "content": "

The hidden dataset is larger than the public test dataset. So it should take longer

", "rawMarkdown": "The hidden dataset is larger than the public test dataset. So it should take longer", "votes": null }, { "id": "1143347", "postDate": "01/07/2021 21:15:25", "content": "

Ohhhhh, that's the reason. For some reason my model takes way longer on the test sets than the training sets. Thanks!

", "rawMarkdown": "Ohhhhh, that's the reason. For some reason my model takes way longer on the test sets than the training sets. Thanks!", "votes": null }, { "id": "1144002", "postDate": "01/08/2021 07:13:17", "content": "

'cause the notebook save on 3k samples and for scoring it will run on 14k samples. so it will be way longer.

", "rawMarkdown": "'cause the notebook save on 3k samples and for scoring it will run on 14k samples. so it will be way longer.", "votes": null }, { "id": "1146198", "postDate": "01/09/2021 15:42:03", "content": "

Is this only inference? Otherwise, I would recommend splitting training and inference in separate notebooks.

", "rawMarkdown": "Is this only inference? Otherwise, I would recommend splitting training and inference in separate notebooks.", "votes": null }, { "id": "1153377", "postDate": "01/14/2021 20:30:21", "content": "

Am I correct to think that the second notebook for inference should be performed without internet ? Coz I've been getting an error saying no internet and tpu allowed.

", "rawMarkdown": "Am I correct to think that the second notebook for inference should be performed without internet ? Coz I've been getting an error saying no internet and tpu allowed.", "votes": null }, { "id": "1153383", "postDate": "01/14/2021 20:54:11", "content": "

Yes, needs to be without internet and no TPU (GPU is fine)

", "rawMarkdown": "Yes, needs to be without internet and no TPU (GPU is fine)", "votes": null }, { "id": "1153386", "postDate": "01/14/2021 20:56:57", "content": "

No internet because otherwise you could just leak the private dataset, I guess…

", "rawMarkdown": "No internet because otherwise you could just leak the private dataset, I guess...", "votes": null }, { "id": "1153467", "postDate": "01/14/2021 23:18:39", "content": "

I get it now, thanks a lot! Now I understand why people were creating to notebooks, one specifically for inference !

", "rawMarkdown": "I get it now, thanks a lot! Now I understand why people were creating to notebooks, one specifically for inference !", "votes": null }, { "id": "1216002", "postDate": "02/24/2021 04:41:31", "content": "

Hey @moewie94, @marcelbischoff, @kudzayiking

\n

My notebook's inference time on the public test set(3k images) is around 10 minutes.
\nBut when I submit it for submission, the notebook runs on 3k + 14 k images and takes more than 7 hours to submit.
\nThe notebook runs on GPU during both times.

\n

Expected time should be (10/3) *17 = roughly 56 minutes(or 1 hour) ?

\n

Any thoughts on why it must be taking so long on submission?

", "rawMarkdown": "Hey @moewie94, @marcelbischoff, @kudzayiking \n\nMy notebook's inference time on the public test set(3k images) is around 10 minutes.\nBut when I submit it for submission, the notebook runs on 3k + 14 k images and takes more than 7 hours to submit. \nThe notebook runs on GPU during both times.\n\nExpected time should be (10/3) *17 = roughly 56 minutes(or 1 hour) ?\n\nAny thoughts on why it must be taking so long on submission?", "votes": null }, { "id": "1216336", "postDate": "02/24/2021 08:07:15", "content": "

Hi @amritpal333 , i am sure most people's submit time is around 7 , 8 hours ,i am sure the queue the notebooks the run inference in a particular order .

", "rawMarkdown": "Hi @amritpal333 , i am sure most people's submit time is around 7 , 8 hours ,i am sure the queue the notebooks the run inference in a particular order .", "votes": null } ], "comments": [ { "id": 1143209, "author_name": "hiramcho", "author_url": "", "post_date": "01/07/2021 19:41:44", "content": "

Maybe there aren't enough servers.

", "votes": null, "replies": [ { "id": 1143271, "author_name": "kyleberdy", "author_url": "", "post_date": "01/07/2021 20:20:16", "content": "

Perhaps… One hour later it's still running. Getting worried it might time out.

", "votes": null, "replies": [] } ] }, { "id": 1143339, "author_name": "richardepstein", "author_url": "", "post_date": "01/07/2021 21:09:34", "content": "

The hidden dataset is larger than the public test dataset. So it should take longer

", "votes": null, "replies": [ { "id": 1143347, "author_name": "kyleberdy", "author_url": "", "post_date": "01/07/2021 21:15:25", "content": "

Ohhhhh, that's the reason. For some reason my model takes way longer on the test sets than the training sets. Thanks!

", "votes": null, "replies": [] } ] }, { "id": 1144002, "author_name": "moewie94", "author_url": "", "post_date": "01/08/2021 07:13:17", "content": "

'cause the notebook save on 3k samples and for scoring it will run on 14k samples. so it will be way longer.

", "votes": null, "replies": [] }, { "id": 1146198, "author_name": "marcelbischoff", "author_url": "", "post_date": "01/09/2021 15:42:03", "content": "

Is this only inference? Otherwise, I would recommend splitting training and inference in separate notebooks.

", "votes": null, "replies": [ { "id": 1153377, "author_name": "kudzayiking", "author_url": "", "post_date": "01/14/2021 20:30:21", "content": "

Am I correct to think that the second notebook for inference should be performed without internet ? Coz I've been getting an error saying no internet and tpu allowed.

", "votes": null, "replies": [] }, { "id": 1153383, "author_name": "marcelbischoff", "author_url": "", "post_date": "01/14/2021 20:54:11", "content": "

Yes, needs to be without internet and no TPU (GPU is fine)

", "votes": null, "replies": [] }, { "id": 1153386, "author_name": "marcelbischoff", "author_url": "", "post_date": "01/14/2021 20:56:57", "content": "

No internet because otherwise you could just leak the private dataset, I guess…

", "votes": null, "replies": [] }, { "id": 1153467, "author_name": "kudzayiking", "author_url": "", "post_date": "01/14/2021 23:18:39", "content": "

I get it now, thanks a lot! Now I understand why people were creating to notebooks, one specifically for inference !

", "votes": null, "replies": [] } ] }, { "id": 1216002, "author_name": "amritpal333", "author_url": "", "post_date": "02/24/2021 04:41:31", "content": "

Hey @moewie94, @marcelbischoff, @kudzayiking

\n

My notebook's inference time on the public test set(3k images) is around 10 minutes.
\nBut when I submit it for submission, the notebook runs on 3k + 14 k images and takes more than 7 hours to submit.
\nThe notebook runs on GPU during both times.

\n

Expected time should be (10/3) *17 = roughly 56 minutes(or 1 hour) ?

\n

Any thoughts on why it must be taking so long on submission?

", "votes": null, "replies": [ { "id": 1216336, "author_name": "kudzayiking", "author_url": "", "post_date": "02/24/2021 08:07:15", "content": "

Hi @amritpal333 , i am sure most people's submit time is around 7 , 8 hours ,i am sure the queue the notebooks the run inference in a particular order .

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1143159": "Hello,\n\nIt's been 7 hours since I submitted my notebook. I'm still getting the \"Notebook Running\" icon from the submission. My model took 5 hours and 29 minutes to save for the first version (where I realized I accidentally had internet enabled, and subsequently resaved last night), and 5 hours and 27 minutes for the second version. Now that I go to submit it, it takes over 1.5 hours longer (right now). I'll post later with the time it finishes at, if it doesn't time out at 9 hours, which is seeming ever more likely.\n\nIs anyone else having this problem? Also, if so, any pointers on how to fix it, and what to expect?", "1143209": "Maybe there aren't enough servers.", "1143271": "Perhaps... One hour later it's still running. Getting worried it might time out.", "1143339": "The hidden dataset is larger than the public test dataset. So it should take longer", "1143347": "Ohhhhh, that's the reason. For some reason my model takes way longer on the test sets than the training sets. Thanks!", "1144002": "'cause the notebook save on 3k samples and for scoring it will run on 14k samples. so it will be way longer.", "1146198": "Is this only inference? Otherwise, I would recommend splitting training and inference in separate notebooks.", "1153377": "Am I correct to think that the second notebook for inference should be performed without internet ? Coz I've been getting an error saying no internet and tpu allowed.", "1153383": "Yes, needs to be without internet and no TPU (GPU is fine)", "1153386": "No internet because otherwise you could just leak the private dataset, I guess...", "1153467": "I get it now, thanks a lot! Now I understand why people were creating to notebooks, one specifically for inference !", "1216002": "Hey @moewie94, @marcelbischoff, @kudzayiking \n\nMy notebook's inference time on the public test set(3k images) is around 10 minutes.\nBut when I submit it for submission, the notebook runs on 3k + 14 k images and takes more than 7 hours to submit. \nThe notebook runs on GPU during both times.\n\nExpected time should be (10/3) *17 = roughly 56 minutes(or 1 hour) ?\n\nAny thoughts on why it must be taking so long on submission?", "1216336": "Hi @amritpal333 , i am sure most people's submit time is around 7 , 8 hours ,i am sure the queue the notebooks the run inference in a particular order ." }, "source": "meta" }