Since yesterday kaggle gpu are working like K80. Two submissions timed out which they would have succeeded . Training time for epoch has drastically shoot up in kaggle from 15 to 18 minutes per epoch for same amount of data.
\nLooks like final day submissions are in Gods Hand. If god blesses then sub will succeed or else time out.
\nWhy we have this severe crunch during last days.
\nWhat is meaning of last day then when technically its extremely difficult to train or submit because of resource misery .
\n", "messages": [ { "id": 1459461, "postDate": "2021-08-08T11:40:33.837Z", "content": "Since yesterday kaggle gpu are working like K80. Two submissions timed out which they would have succeeded . Training time for epoch has drastically shoot up in kaggle from 15 to 18 minutes per epoch for same amount of data.
\nLooks like final day submissions are in Gods Hand. If god blesses then sub will succeed or else time out.
\nWhy we have this severe crunch during last days.
\nWhat is meaning of last day then when technically its extremely difficult to train or submit because of resource misery .
\n", "rawMarkdown": "Since yesterday kaggle gpu are working like K80. Two submissions timed out which they would have succeeded . Training time for epoch has drastically shoot up in kaggle from 15 to 18 minutes per epoch for same amount of data.\n\nLooks like final day submissions are in Gods Hand. If god blesses then sub will succeed or else time out. \n\nWhy we have this severe crunch during last days. \n\nWhat is meaning of last day then when technically its extremely difficult to train or submit because of resource misery . \n\n@juliaelliott ", "votes": 23 }, { "id": 1459554, "postDate": "2021-08-08T12:49:08.130Z", "content": "I dont believe you will get K80, did you confirm that or just a feeling?
\nI rather believe you get slower CPU, which has happened in the past.
@jaideepvalani Can you please share a kernel with me which has experienced this slowdown? It will help me see the difference between the two runs. Kaggle does not have K80s anymore, only P100s and we generally only use identical/equivalent hardware on every run.
", "rawMarkdown": "@jaideepvalani Can you please share a kernel with me which has experienced this slowdown? It will help me see the difference between the two runs. Kaggle does not have K80s anymore, only P100s and we generally only use identical/equivalent hardware on every run.", "votes": 1, "replies": [ { "id": 1463486, "postDate": "2021-08-10T08:01:58.453Z", "content": "sorry it was just simile that I put seeing slow down in the performances.
\nBut yes overall ever since docker image was upgraded there was drop in the performance especially in Yolo Model training. Under same condition when I trained it in Google Colab it worked perfectly fine as expected.
\nYou can simply fork this kernel and check run time per epoch. with given option it is suppose to be quick
\nhttps://www.kaggle.com/ankurgupta29/siim-c-19-yolo-v5-train
I'm also experiencing this. Submission took >9 hours which was taking just a few hours to complete.
", "rawMarkdown": "I'm also experiencing this. Submission took >9 hours which was taking just a few hours to complete. ", "votes": 1 }, { "id": 1459631, "postDate": "2021-08-08T13:33:56.837Z", "content": "I just replaced my weights with a newer version of weights and run my submission kernel the same as yesterday. But it has thrown, \"Notebook Exceeded Allowed Compute\" today.
", "rawMarkdown": "I just replaced my weights with a newer version of weights and run my submission kernel the same as yesterday. But it has thrown, \"Notebook Exceeded Allowed Compute\" today.", "votes": 2 }, { "id": 1460814, "postDate": "2021-08-09T04:52:37.297Z", "content": "Same issue, the notebook which ran in 6-7 hours now taking > 9 hours.
\nToday is the last day and we can't make final submissions, I think the kaggle team should look into this.
I dont believe you will get K80, did you confirm that or just a feeling?
\nI rather believe you get slower CPU, which has happened in the past.
@jaideepvalani Can you please share a kernel with me which has experienced this slowdown? It will help me see the difference between the two runs. Kaggle does not have K80s anymore, only P100s and we generally only use identical/equivalent hardware on every run.
", "votes": 1, "replies": [ { "id": 1463486, "author_name": "Jaideep", "author_url": "", "post_date": "2021-08-10T08:01:58.453000", "content": "sorry it was just simile that I put seeing slow down in the performances.
\nBut yes overall ever since docker image was upgraded there was drop in the performance especially in Yolo Model training. Under same condition when I trained it in Google Colab it worked perfectly fine as expected.
\nYou can simply fork this kernel and check run time per epoch. with given option it is suppose to be quick
\nhttps://www.kaggle.com/ankurgupta29/siim-c-19-yolo-v5-train
I'm also experiencing this. Submission took >9 hours which was taking just a few hours to complete.
", "votes": 1, "replies": [] }, { "id": 1459631, "author_name": "Mohammad Zunaed", "author_url": "", "post_date": "2021-08-08T13:33:56.837000", "content": "I just replaced my weights with a newer version of weights and run my submission kernel the same as yesterday. But it has thrown, \"Notebook Exceeded Allowed Compute\" today.
", "votes": 2, "replies": [] }, { "id": 1460814, "author_name": "Muhammad Ahmed", "author_url": "", "post_date": "2021-08-09T04:52:37.297000", "content": "Same issue, the notebook which ran in 6-7 hours now taking > 9 hours.
\nToday is the last day and we can't make final submissions, I think the kaggle team should look into this.