@herbison,
\nMy morning submission \"was scoring\" for more than 7 hours instead of usual 4.5 hours. Now my saved notebook has been in a queue for more than 1.5 hour.
\nIt means that even the submissions' waiting time is more than 1.5 hour.
\nCould you, please, push forward at least the submissions submitted to this competition so that the in-time submitted submissions could finish before the deadline?
@addisonhoward I am sure that with so many interesting HW heavy competitions now the situation will repeat. But at least on the last day of a competition there should be some priorities set for the actual submissions waiting in queue.
", "messages": [ { "id": "2741792", "postDate": "04/08/2024 15:45:52", "content": "@herbison,
\nMy morning submission \"was scoring\" for more than 7 hours instead of usual 4.5 hours. Now my saved notebook has been in a queue for more than 1.5 hour.
\nIt means that even the submissions' waiting time is more than 1.5 hour.
\nCould you, please, push forward at least the submissions submitted to this competition so that the in-time submitted submissions could finish before the deadline?
@addisonhoward I am sure that with so many interesting HW heavy competitions now the situation will repeat. But at least on the last day of a competition there should be some priorities set for the actual submissions waiting in queue.
", "rawMarkdown": "herbison,\nMy morning submission \"was scoring\" for more than 7 hours instead of usual 4.5 hours. Now my saved notebook has been in a queue for more than 1.5 hour. \nIt means that even the submissions' waiting time is more than 1.5 hour.\nCould you, please, push forward at least the submissions submitted to this competition so that the in-time submitted submissions could finish before the deadline?\n\n@addisonhoward I am sure that with so many interesting HW heavy competitions now the situation will repeat. But at least on the last day of a competition there should be some priorities set for the actual submissions waiting in queue.", "votes": null }, { "id": "2741805", "postDate": "04/08/2024 15:50:44", "content": "I also observe similar situation. I hope it is not due to DDoS attack on Kaggle servers.
\n While trying to run the notebook with GPU P100, I am in a queue waiting for resources.
I think a lot of people are in a similar situation, maybe if we upvote this enough it'll happen 😅. Personally, my models only take 20 minutes to run once I get a gpu, but I too have been stuck in a queue for awhile, and both of my final models that I plan on selecting have yet to be submitted 😐.
", "rawMarkdown": "I think a lot of people are in a similar situation, maybe if we upvote this enough it'll happen 😅. Personally, my models only take 20 minutes to run *once I get a gpu*, but I too have been stuck in a queue for awhile, and both of my final models that I plan on selecting have yet to be submitted 😐.", "votes": null }, { "id": "2741879", "postDate": "04/08/2024 16:24:46", "content": "Something really happened. After nearly 2 hours my notebook ran and after I submitted my last submission I could see the sign \"Traffic ahead\".😐
", "rawMarkdown": "*Something* really happened. After nearly 2 hours my notebook ran and after I submitted my last submission I could see the sign \"Traffic ahead\".😐", "votes": null }, { "id": "2741995", "postDate": "04/08/2024 17:29:11", "content": "For me it did not happen, what I did was interactive gpus + submit through interactive gpu. Submission takes same time as before with no queue time at all
", "rawMarkdown": "For me it did not happen, what I did was interactive gpus + submit through interactive gpu. Submission takes same time as before with no queue time at all", "votes": null }, { "id": "2742039", "postDate": "04/08/2024 17:50:43", "content": "Thanks for the information. I know from past discussions with Kaggle team (then it was about TPU availability) that interactive sessions are always prioritized.
\nBut giving the same priority to consequent submissions doesn't seem to me very fair.
\nBTW interactive session is not a good solution for me because of time reasons.
@blankaf I added capacity to the fleet (and I'm working on adding more!) to catch up to the demand boost.
\nIt looks like the GPU queue has almost completely cleared now too, so things should be looking much better now.
Thanks for your patience!
", "rawMarkdown": "blankaf I added capacity to the fleet (and I'm working on adding more!) to catch up to the demand boost.\nIt looks like the GPU queue has *almost* completely cleared now too, so things should be looking much better now.\n\nThanks for your patience!", "votes": null }, { "id": "2742118", "postDate": "04/08/2024 18:39:59", "content": "Thank you @herbison.
\nI expect even higher demand for GPU time in 8 days, when LLM Prompt Recovery will be ending.
\nBTW it would be nice from Google, the parent company, to give some more powerful resources when Kagglers are mostly for free testing their new product!😀
Thank you for the heads-up!
", "rawMarkdown": "Thank you for the heads-up!", "votes": null } ], "comments": [ { "id": 2741805, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "04/08/2024 15:50:44", "content": "I also observe similar situation. I hope it is not due to DDoS attack on Kaggle servers.
\n While trying to run the notebook with GPU P100, I am in a queue waiting for resources.
I think a lot of people are in a similar situation, maybe if we upvote this enough it'll happen 😅. Personally, my models only take 20 minutes to run once I get a gpu, but I too have been stuck in a queue for awhile, and both of my final models that I plan on selecting have yet to be submitted 😐.
", "votes": null, "replies": [ { "id": 2741879, "author_name": "blankaf", "author_url": "", "post_date": "04/08/2024 16:24:46", "content": "Something really happened. After nearly 2 hours my notebook ran and after I submitted my last submission I could see the sign \"Traffic ahead\".😐
", "votes": null, "replies": [] } ] }, { "id": 2741995, "author_name": "nikhilmishradev", "author_url": "", "post_date": "04/08/2024 17:29:11", "content": "For me it did not happen, what I did was interactive gpus + submit through interactive gpu. Submission takes same time as before with no queue time at all
", "votes": null, "replies": [ { "id": 2742039, "author_name": "blankaf", "author_url": "", "post_date": "04/08/2024 17:50:43", "content": "Thanks for the information. I know from past discussions with Kaggle team (then it was about TPU availability) that interactive sessions are always prioritized.
\nBut giving the same priority to consequent submissions doesn't seem to me very fair.
\nBTW interactive session is not a good solution for me because of time reasons.
@blankaf I added capacity to the fleet (and I'm working on adding more!) to catch up to the demand boost.
\nIt looks like the GPU queue has almost completely cleared now too, so things should be looking much better now.
Thanks for your patience!
", "votes": null, "replies": [ { "id": 2742118, "author_name": "blankaf", "author_url": "", "post_date": "04/08/2024 18:39:59", "content": "Thank you @herbison.
\nI expect even higher demand for GPU time in 8 days, when LLM Prompt Recovery will be ending.
\nBTW it would be nice from Google, the parent company, to give some more powerful resources when Kagglers are mostly for free testing their new product!😀
Thank you for the heads-up!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2741792": "herbison,\nMy morning submission \"was scoring\" for more than 7 hours instead of usual 4.5 hours. Now my saved notebook has been in a queue for more than 1.5 hour. \nIt means that even the submissions' waiting time is more than 1.5 hour.\nCould you, please, push forward at least the submissions submitted to this competition so that the in-time submitted submissions could finish before the deadline?\n\n@addisonhoward I am sure that with so many interesting HW heavy competitions now the situation will repeat. But at least on the last day of a competition there should be some priorities set for the actual submissions waiting in queue.", "2741805": "I also observe similar situation. I hope it is not due to DDoS attack on Kaggle servers. \n While trying to run the notebook with GPU P100, I am in a queue waiting for resources.", "2741815": "I think a lot of people are in a similar situation, maybe if we upvote this enough it'll happen 😅. Personally, my models only take 20 minutes to run *once I get a gpu*, but I too have been stuck in a queue for awhile, and both of my final models that I plan on selecting have yet to be submitted 😐.", "2741879": "*Something* really happened. After nearly 2 hours my notebook ran and after I submitted my last submission I could see the sign \"Traffic ahead\".😐", "2741995": "For me it did not happen, what I did was interactive gpus + submit through interactive gpu. Submission takes same time as before with no queue time at all", "2742039": "Thanks for the information. I know from past discussions with Kaggle team (then it was about TPU availability) that interactive sessions are always prioritized.\nBut giving the same priority to consequent submissions doesn't seem to me very fair.\nBTW interactive session is not a good solution for me because of time reasons.", "2742080": "blankaf I added capacity to the fleet (and I'm working on adding more!) to catch up to the demand boost.\nIt looks like the GPU queue has *almost* completely cleared now too, so things should be looking much better now.\n\nThanks for your patience!", "2742118": "Thank you @herbison. \nI expect even higher demand for GPU time in 8 days, when LLM Prompt Recovery will be ending.\nBTW it would be nice from Google, the parent company, to give some more powerful resources when Kagglers are mostly for free testing their new product!😀", "2742198": "Thank you for the heads-up!" }, "source": "meta" }