{ "id": 447823, "title": "Submitted notebook runs for 10 hours but success. That's weird!", "url": "/competitions/bengaliai-speech/discussion/447823", "author_name": "jason", "post_date": "2023-10-17T11:27:04.952000", "votes": 3, "comment_count": 11, "views": 0, "content": "

I submitted a notebook today, it ran for 10 hours, and it worked😂

", "messages": [ { "id": 2485683, "postDate": "2023-10-17T11:27:04.953Z", "content": "

I submitted a notebook today, it ran for 10 hours, and it worked😂

", "rawMarkdown": "I submitted a notebook today, it ran for 10 hours, and it worked😂", "votes": 3 }, { "id": 2485714, "postDate": "2023-10-17T12:00:43.483Z", "content": "

Its not weird, your notebook does not start running immediately after you submit it, it's probably in a queue, and wait until its turn comes up to start executing. Same thing is happening to my notebooks too.

\n

I don't think it's a good thing in the last days of the competition. Kaggle will not accept solutions which complete running after the competition deadline, even if started 10 hours ago, and took only around 3 hours to run.

\n

So basically submitting anything after the first 12 hours of the last day is risky.

\n

And what if someone gets lucky and their submission gets run only in 3 hours (which is the actual time their notebook takes to run). In that case aren't they getting an advantage of more submission time.

\n

IMO, this is quite a big issue, and should be fixed, or submissions made at least a fixed number of hours, before the competition deadline should be accepted and run on priority.

\n

Tagging @ashleychow and kaggle team

", "rawMarkdown": "Its not weird, your notebook does not start running immediately after you submit it, it's probably in a queue, and wait until its turn comes up to start executing. Same thing is happening to my notebooks too.\n\nI don't think it's a good thing in the last days of the competition. Kaggle will not accept solutions which complete running after the competition deadline, even if started 10 hours ago, and took only around 3 hours to run.\n\nSo basically submitting anything after the first 12 hours of the last day is risky.\n\nAnd what if someone gets lucky and their submission gets run only in 3 hours (which is the actual time their notebook takes to run). In that case aren't they getting an advantage of more submission time.\n\nIMO, this is quite a big issue, and should be fixed, or submissions made at least a fixed number of hours, before the competition deadline should be accepted and run on priority.\n\nTagging @ashleychow and kaggle team", "votes": 1, "replies": [ { "id": 2485733, "postDate": "2023-10-17T12:20:06.427Z", "content": "

Oh,I see,thank you for your explanation😀

", "rawMarkdown": "Oh,I see,thank you for your explanation😀", "votes": 1 }, { "id": 2485848, "postDate": "2023-10-17T13:53:50.903Z", "content": "

I don´t think your explanation is right. The time to load large files before starting the sessions is added on top. If 100 gigabytes have to be loaded initially, this can take an hour. However, this time is not taken into account for the code runtime. Therefore, the code can run successfully even with a runtime of 10 hours.

\n

But your conclusion is right anyway, you should always plan some extra time making the last submissions.

", "rawMarkdown": "I don´t think your explanation is right. The time to load large files before starting the sessions is added on top. If 100 gigabytes have to be loaded initially, this can take an hour. However, this time is not taken into account for the code runtime. Therefore, the code can run successfully even with a runtime of 10 hours.\n\nBut your conclusion is right anyway, you should always plan some extra time making the last submissions.", "replies": [ { "id": 2485865, "postDate": "2023-10-17T14:03:15.390Z", "content": "

@benbla My same notebook ran in 3 hours, 6 hours and then 9 hours approximately. The only 2 possible explanations I can think of is queuing, or using different versions of gpus for the same submission. queuing is more likely

", "rawMarkdown": "@benbla My same notebook ran in 3 hours, 6 hours and then 9 hours approximately. The only 2 possible explanations I can think of is queuing, or using different versions of gpus for the same submission. queuing is more likely" }, { "id": 2485871, "postDate": "2023-10-17T14:07:07.877Z", "content": "

with a same model? I have always more or less the same submission time.

", "rawMarkdown": "with a same model? I have always more or less the same submission time." }, { "id": 2485873, "postDate": "2023-10-17T14:07:37.157Z", "content": "

yes, with the same model. I am surprised then, can't think of some other reason.
\nI also hope kaggle introduces a new feature, which tells about the actual run time of the submission, without having to monitor it manually

", "rawMarkdown": "yes, with the same model. I am surprised then, can't think of some other reason.\nI also hope kaggle introduces a new feature, which tells about the actual run time of the submission, without having to monitor it manually" }, { "id": 2485884, "postDate": "2023-10-17T14:12:25.137Z", "content": "

you mean same checkpoint or same model but with different checkpoint?

", "rawMarkdown": "you mean same checkpoint or same model but with different checkpoint?" }, { "id": 2485892, "postDate": "2023-10-17T14:16:40.690Z", "content": "

checkpoints were different. But I guess it should not affect inference speed in my model

", "rawMarkdown": "checkpoints were different. But I guess it should not affect inference speed in my model" }, { "id": 2485898, "postDate": "2023-10-17T14:19:20.500Z", "content": "

well i am assuming you are using an autoregressive model, in this case your model can hallucinate and generate texts with different length for the same audio which can change your runtime.

", "rawMarkdown": "well i am assuming you are using an autoregressive model, in this case your model can hallucinate and generate texts with different length for the same audio which can change your runtime.", "votes": 3 }, { "id": 2485906, "postDate": "2023-10-17T14:22:14.290Z", "content": "

yes I agree, this could be an issue. Thanks for pointing it out.

", "rawMarkdown": "yes I agree, this could be an issue. Thanks for pointing it out." } ] }, { "id": 2486701, "postDate": "2023-10-18T05:49:09.753Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 2485714, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-17T12:00:43.483000", "content": "

Its not weird, your notebook does not start running immediately after you submit it, it's probably in a queue, and wait until its turn comes up to start executing. Same thing is happening to my notebooks too.

\n

I don't think it's a good thing in the last days of the competition. Kaggle will not accept solutions which complete running after the competition deadline, even if started 10 hours ago, and took only around 3 hours to run.

\n

So basically submitting anything after the first 12 hours of the last day is risky.

\n

And what if someone gets lucky and their submission gets run only in 3 hours (which is the actual time their notebook takes to run). In that case aren't they getting an advantage of more submission time.

\n

IMO, this is quite a big issue, and should be fixed, or submissions made at least a fixed number of hours, before the competition deadline should be accepted and run on priority.

\n

Tagging @ashleychow and kaggle team

", "votes": 1, "replies": [ { "id": 2485733, "author_name": "jason", "author_url": "", "post_date": "2023-10-17T12:20:06.427000", "content": "

Oh,I see,thank you for your explanation😀

", "votes": 1, "replies": [] }, { "id": 2485848, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2023-10-17T13:53:50.903000", "content": "

I don´t think your explanation is right. The time to load large files before starting the sessions is added on top. If 100 gigabytes have to be loaded initially, this can take an hour. However, this time is not taken into account for the code runtime. Therefore, the code can run successfully even with a runtime of 10 hours.

\n

But your conclusion is right anyway, you should always plan some extra time making the last submissions.

", "votes": 0, "replies": [ { "id": 2485865, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-17T14:03:15.390000", "content": "

@benbla My same notebook ran in 3 hours, 6 hours and then 9 hours approximately. The only 2 possible explanations I can think of is queuing, or using different versions of gpus for the same submission. queuing is more likely

", "votes": 0, "replies": [] }, { "id": 2485871, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-17T14:07:07.877000", "content": "

with a same model? I have always more or less the same submission time.

", "votes": 0, "replies": [] }, { "id": 2485873, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-17T14:07:37.157000", "content": "

yes, with the same model. I am surprised then, can't think of some other reason.
\nI also hope kaggle introduces a new feature, which tells about the actual run time of the submission, without having to monitor it manually

", "votes": 0, "replies": [] }, { "id": 2485884, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-17T14:12:25.137000", "content": "

you mean same checkpoint or same model but with different checkpoint?

", "votes": 0, "replies": [] }, { "id": 2485892, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-17T14:16:40.690000", "content": "

checkpoints were different. But I guess it should not affect inference speed in my model

", "votes": 0, "replies": [] }, { "id": 2485898, "author_name": "tugstugi", "author_url": "", "post_date": "2023-10-17T14:19:20.500000", "content": "

well i am assuming you are using an autoregressive model, in this case your model can hallucinate and generate texts with different length for the same audio which can change your runtime.

", "votes": 3, "replies": [] }, { "id": 2485906, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-10-17T14:22:14.290000", "content": "

yes I agree, this could be an issue. Thanks for pointing it out.

", "votes": 0, "replies": [] } ] }, { "id": 2486701, "author_name": "", "author_url": "", "post_date": "2023-10-18T05:49:09.753000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2485683": "I submitted a notebook today, it ran for 10 hours, and it worked😂", "2485714": "Its not weird, your notebook does not start running immediately after you submit it, it's probably in a queue, and wait until its turn comes up to start executing. Same thing is happening to my notebooks too.\n\nI don't think it's a good thing in the last days of the competition. Kaggle will not accept solutions which complete running after the competition deadline, even if started 10 hours ago, and took only around 3 hours to run.\n\nSo basically submitting anything after the first 12 hours of the last day is risky.\n\nAnd what if someone gets lucky and their submission gets run only in 3 hours (which is the actual time their notebook takes to run). In that case aren't they getting an advantage of more submission time.\n\nIMO, this is quite a big issue, and should be fixed, or submissions made at least a fixed number of hours, before the competition deadline should be accepted and run on priority.\n\nTagging @ashleychow and kaggle team" } }