![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F712087%2Fd23690a4fca5c256fc23fcfa9696bfa0%2Fqutoa_error.png?generation=1578949205758575&alt=media\")
Does anyone face errors with Notebooks? I did some minimal modifications to a Notebook than ran successfully but now I get \"Submission CSV not found\". Was the memory of Notebooks reduced to e.g. 4GB? It's very disappointing because I can't debug this further and can't make any further submissions ...
\n\nOn top of this issue I also spend my whole GPU quota in just a few hours. I think the counter is not working correctly. @philculliton Could you please take a look if Notebooks are working as expected?
\n\nI also saw some similar issues:
\n\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F712087%2F0a5460e0af6df38e6f169ce6d0dfa337%2Ferror.png?generation=1578949153271232&alt=media\")
Does anyone face errors with Notebooks? I did some minimal modifications to a Notebook than ran successfully but now I get \"Submission CSV not found\". Was the memory of Notebooks reduced to e.g. 4GB? It's very disappointing because I can't debug this further and can't make any further submissions ...
\n\nOn top of this issue I also spend my whole GPU quota in just a few hours. I think the counter is not working correctly. @philculliton Could you please take a look if Notebooks are working as expected?
\n\nI also saw some similar issues:
\n\n
The \"Submission CSV Not Found\" error is thrown in the event a submission.csv was not outputted. This could mean that your notebook is timing out before generating a submission.csv file. Based on your GPU quota running out, this is the likely scenario. The fast quota burn issue is under investigation; we will update if any findings determine an issue.
Thanks for investigating!
\n\n\n\n\nThis could mean that your notebook is timing out before generating a submission.csv file.
\n
I don't think that this is the cause, unless the time limit was changed. The notebook was just shared with you.
\n\n@seesee Unfortunately, the error would mean that the timeout would be happening on the re-run. I know that it seems strange that the failed version has a shorter runtime on commit than a successful version, but in the larger re-run, especially if your code is not making the most efficient use of resources, then this is possible.
", "rawMarkdown": "seesee Unfortunately, the error would mean that the timeout would be happening on the re-run. I know that it seems strange that the failed version has a shorter runtime on commit than a successful version, but in the larger re-run, especially if your code is not making the most efficient use of resources, then this is possible.", "votes": null }, { "id": "718059", "postDate": "01/13/2020 23:56:12", "content": "I see, thanks. I guess I am just hitting the 10800s limit. Version 6 was just a lucky outlier and one should target <1000s for the public set.
\n\nEdit: The GPU Quota issue is still interesting. Two failed commits and some interactive sessions should not create 29hours. I'd expect 2 * 3h + 4h = 10h.
", "rawMarkdown": "I see, thanks. I guess I am just hitting the 10800s limit. Version 6 was just a lucky outlier and one should target <1000s for the public set.\n\nEdit: The GPU Quota issue is still interesting. Two failed commits and some interactive sessions should not create 29hours. I'd expect 2 * 3h + 4h = 10h.", "votes": null }, { "id": "718230", "postDate": "01/14/2020 07:38:54", "content": "@seesee my notebooks run somewhere around 1080-1124 s on public test set but never failed on private test.
", "rawMarkdown": "seesee my notebooks run somewhere around 1080-1124 s on public test set but never failed on private test.", "votes": null }, { "id": "718553", "postDate": "01/14/2020 14:21:52", "content": "Hi @seesee - You had a few sessions that experienced some internal failures, which our code was not accounting correctly for. This will be patched going forward. Thanks for your report!
", "rawMarkdown": "Hi @seesee - You had a few sessions that experienced some internal failures, which our code was not accounting correctly for. This will be patched going forward. Thanks for your report!", "votes": null }, { "id": "718578", "postDate": "01/14/2020 14:52:51", "content": "Thanks for investigating this very fast. As the competition is ending soon, I'd be happy if I could make one more submission attempt. Is it possible to re-increase my quota by ~4h?
\n\nEdit: Thanks everyone for helping me fix these issues!
", "rawMarkdown": "Thanks for investigating this very fast. As the competition is ending soon, I'd be happy if I could make one more submission attempt. Is it possible to re-increase my quota by ~4h?\n\nEdit: Thanks everyone for helping me fix these issues!", "votes": null }, { "id": "720606", "postDate": "01/16/2020 15:11:36", "content": "Hi @seesee Were you able to resolve the \"Submission CSV not found\" issue?
\n\nI'm seeing the same \"Submission CSV not found\" error today, using the same pipeline as previous version, submitted successfully 6 days ago. The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.
\n\nI'm wondering whether Kaggle has changed backend hardware during the past 6 days, as all the reported \"Submission CSV not found\" issues seem to happen within this time period.\n@jplotts Do you mind looking into this for me as well? I can share my notebook with you. Thanks!
", "rawMarkdown": "Hi @seesee Were you able to resolve the \"Submission CSV not found\" issue?\n\nI'm seeing the same \"Submission CSV not found\" error today, using the same pipeline as previous version, submitted successfully 6 days ago. The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.\n\nI'm wondering whether Kaggle has changed backend hardware during the past 6 days, as all the reported \"Submission CSV not found\" issues seem to happen within this time period.\n@jplotts Do you mind looking into this for me as well? I can share my notebook with you. Thanks!", "votes": null }, { "id": "720616", "postDate": "01/16/2020 15:27:12", "content": "Hi @boliu0, yes I was able to resolve the errors.
\n\n> The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.
\n\nYour are right, I didn't know that. I actually spend quite some time making my Notebooks run faster as suggested by @juliaelliott. However, this was not causing the error. It was a bug in my code which caused an IndexError at the decoding stage. I found it by running the inference pipeline on the whole training set. I don't recall the sample that caused the error. Good luck!
Edit: I can also confirm that the quota is increasing as expected now.
", "rawMarkdown": "Hi @boliu0, yes I was able to resolve the errors.\n\n> The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.\n\nYour are right, I didn't know that. I actually spend quite some time making my Notebooks run faster as suggested by @juliaelliott. However, this was not causing the error. It was a bug in my code which caused an `IndexError` at the decoding stage. I found it by running the inference pipeline on the whole training set. I don't recall the sample that caused the error. Good luck!\n\nEdit: I can also confirm that the quota is increasing as expected now.", "votes": null }, { "id": "720650", "postDate": "01/16/2020 16:10:08", "content": "Thanks. So you're saying it's not due to OOM, but it's a bug which occurs only in some sample in private set and training set, but not occur in the 346 public set, that's why you don't have error during commit?
", "rawMarkdown": "Thanks. So you're saying it's not due to OOM, but it's a bug which occurs only in some sample in private set and training set, but not occur in the 346 public set, that's why you don't have error during commit?", "votes": null }, { "id": "720820", "postDate": "01/16/2020 19:27:03", "content": "Yes, I think that's what happened in my case.
", "rawMarkdown": "Yes, I think that's what happened in my case.", "votes": null } ], "comments": [ { "id": 718017, "author_name": "juliaelliott", "author_url": "", "post_date": "01/13/2020 22:14:29", "content": "The \"Submission CSV Not Found\" error is thrown in the event a submission.csv was not outputted. This could mean that your notebook is timing out before generating a submission.csv file. Based on your GPU quota running out, this is the likely scenario. The fast quota burn issue is under investigation; we will update if any findings determine an issue.
Thanks for investigating!
\n\n\n\n\nThis could mean that your notebook is timing out before generating a submission.csv file.
\n
I don't think that this is the cause, unless the time limit was changed. The notebook was just shared with you.
\n\n@seesee Unfortunately, the error would mean that the timeout would be happening on the re-run. I know that it seems strange that the failed version has a shorter runtime on commit than a successful version, but in the larger re-run, especially if your code is not making the most efficient use of resources, then this is possible.
", "votes": null, "replies": [] }, { "id": 718059, "author_name": "seesee", "author_url": "", "post_date": "01/13/2020 23:56:12", "content": "I see, thanks. I guess I am just hitting the 10800s limit. Version 6 was just a lucky outlier and one should target <1000s for the public set.
\n\nEdit: The GPU Quota issue is still interesting. Two failed commits and some interactive sessions should not create 29hours. I'd expect 2 * 3h + 4h = 10h.
", "votes": null, "replies": [] }, { "id": 718230, "author_name": "axel81", "author_url": "", "post_date": "01/14/2020 07:38:54", "content": "@seesee my notebooks run somewhere around 1080-1124 s on public test set but never failed on private test.
", "votes": null, "replies": [] } ] }, { "id": 718553, "author_name": "jplotts", "author_url": "", "post_date": "01/14/2020 14:21:52", "content": "Hi @seesee - You had a few sessions that experienced some internal failures, which our code was not accounting correctly for. This will be patched going forward. Thanks for your report!
", "votes": null, "replies": [ { "id": 718578, "author_name": "seesee", "author_url": "", "post_date": "01/14/2020 14:52:51", "content": "Thanks for investigating this very fast. As the competition is ending soon, I'd be happy if I could make one more submission attempt. Is it possible to re-increase my quota by ~4h?
\n\nEdit: Thanks everyone for helping me fix these issues!
", "votes": null, "replies": [] }, { "id": 720606, "author_name": "boliu0", "author_url": "", "post_date": "01/16/2020 15:11:36", "content": "Hi @seesee Were you able to resolve the \"Submission CSV not found\" issue?
\n\nI'm seeing the same \"Submission CSV not found\" error today, using the same pipeline as previous version, submitted successfully 6 days ago. The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.
\n\nI'm wondering whether Kaggle has changed backend hardware during the past 6 days, as all the reported \"Submission CSV not found\" issues seem to happen within this time period.\n@jplotts Do you mind looking into this for me as well? I can share my notebook with you. Thanks!
", "votes": null, "replies": [] }, { "id": 720616, "author_name": "seesee", "author_url": "", "post_date": "01/16/2020 15:27:12", "content": "Hi @boliu0, yes I was able to resolve the errors.
\n\n> The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.
\n\nYour are right, I didn't know that. I actually spend quite some time making my Notebooks run faster as suggested by @juliaelliott. However, this was not causing the error. It was a bug in my code which caused an IndexError at the decoding stage. I found it by running the inference pipeline on the whole training set. I don't recall the sample that caused the error. Good luck!
Edit: I can also confirm that the quota is increasing as expected now.
", "votes": null, "replies": [] }, { "id": 720650, "author_name": "boliu0", "author_url": "", "post_date": "01/16/2020 16:10:08", "content": "Thanks. So you're saying it's not due to OOM, but it's a bug which occurs only in some sample in private set and training set, but not occur in the 346 public set, that's why you don't have error during commit?
", "votes": null, "replies": [] }, { "id": 720820, "author_name": "seesee", "author_url": "", "post_date": "01/16/2020 19:27:03", "content": "Yes, I think that's what happened in my case.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "717986": "Does anyone face errors with Notebooks? I did some minimal modifications to a Notebook than ran successfully but now I get \"Submission CSV not found\". Was the memory of Notebooks reduced to e.g. 4GB? It's very disappointing because I can't debug this further and can't make any further submissions ...\n\nOn top of this issue I also spend my whole GPU quota in just a few hours. I think the counter is not working correctly. @philculliton Could you please take a look if Notebooks are working as expected?\n\nI also saw some similar issues:\n\n* https://www.kaggle.com/c/bengaliai-cv19/discussion/125639\n* https://www.kaggle.com/c/bengaliai-cv19/discussion/125724\n\n\n\n\n\n\n", "718017": "The \"Submission CSV Not Found\" error is thrown in the event a `submission.csv` was not outputted. This could mean that your notebook is timing out before generating a `submission.csv` file. Based on your GPU quota running out, this is the likely scenario. The fast quota burn issue is under investigation; we will update if any findings determine an issue.", "718044": "Thanks for investigating!\n\n> This could mean that your notebook is timing out before generating a submission.csv file.\n\nI don't think that this is the cause, unless the time limit was changed. The notebook was just shared with you.\n\n* Version 6 (successful) takes 1176.6s\n* Version 13 (failed) takes 1138.8s", "718055": "seesee Unfortunately, the error would mean that the timeout would be happening on the re-run. I know that it seems strange that the failed version has a shorter runtime on commit than a successful version, but in the larger re-run, especially if your code is not making the most efficient use of resources, then this is possible.", "718059": "I see, thanks. I guess I am just hitting the 10800s limit. Version 6 was just a lucky outlier and one should target <1000s for the public set.\n\nEdit: The GPU Quota issue is still interesting. Two failed commits and some interactive sessions should not create 29hours. I'd expect 2 * 3h + 4h = 10h.", "718230": "seesee my notebooks run somewhere around 1080-1124 s on public test set but never failed on private test.", "718553": "Hi @seesee - You had a few sessions that experienced some internal failures, which our code was not accounting correctly for. This will be patched going forward. Thanks for your report!", "718578": "Thanks for investigating this very fast. As the competition is ending soon, I'd be happy if I could make one more submission attempt. Is it possible to re-increase my quota by ~4h?\n\nEdit: Thanks everyone for helping me fix these issues!", "720606": "Hi @seesee Were you able to resolve the \"Submission CSV not found\" issue?\n\nI'm seeing the same \"Submission CSV not found\" error today, using the same pipeline as previous version, submitted successfully 6 days ago. The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.\n\nI'm wondering whether Kaggle has changed backend hardware during the past 6 days, as all the reported \"Submission CSV not found\" issues seem to happen within this time period.\n@jplotts Do you mind looking into this for me as well? I can share my notebook with you. Thanks!", "720616": "Hi @boliu0, yes I was able to resolve the errors.\n\n> The two version have same commit runtime, so I don't think it's private set time out issue. For private set time out, we get \"Notebook Timeout\" errors instead.\n\nYour are right, I didn't know that. I actually spend quite some time making my Notebooks run faster as suggested by @juliaelliott. However, this was not causing the error. It was a bug in my code which caused an `IndexError` at the decoding stage. I found it by running the inference pipeline on the whole training set. I don't recall the sample that caused the error. Good luck!\n\nEdit: I can also confirm that the quota is increasing as expected now.", "720650": "Thanks. So you're saying it's not due to OOM, but it's a bug which occurs only in some sample in private set and training set, but not occur in the 346 public set, that's why you don't have error during commit?", "720820": "Yes, I think that's what happened in my case." }, "source": "meta" }