Those who may have actually attempted to debug a code competition where the code executes properly on the (sample) test data provided but fails on the actual test data may be aware of how frustrating the process can be….esp if one has very limited time at disposal to work on the comp. There are only 4-5 generic error messages thrown and there is no guarantee that the messages are correct (for e.g. you may get notebook-exception error for a memory-exceeded error). Anyway, here is what I have discovered so far:
\nWhat I am trying to do. Create all spects at one shot and store them. Then load models and start predicting
What is happening: Code working fine with (sole)sample test OGG. Also working fine if I replicate the sample test OGG 5500 times (number of actual test records). But fails during evaluation. First used to throw notebook exception and then after some debugging & changes, it starts to throw sub not found error off late
Root cause: No issues with model or other things. The code fails at generating the spects. Initially I suspected num_workers > 1 causes these things (in past discussions, parallel processes have been suspect), but even single process fails. Finally commenting the line where I save the Mel spects eliminates the error
Why is this happening: This is where I need some help. I can see that each spect is roughly 300KB. So 5500 files would take a max of 2 GB of disk space. I think we have 19.6 GB disk space available. CPU RAM is 16 GB. During local execution(when I take the solo test.OGG and add 5500 duplicates and create spects for all of them), I see that at max, couple of GB are used during the spect extraction process. So it isn’t a RAM problem either (more so because the program works fine when I comment the save-to-disk line of code). By all accounts this is a local disk issue. But test records are all of 60 sec duration. There are 5500 of them. I am sure these won’t exceed couple of GB of disk space. And moreover, the submission works fine when I generate spects at runtime for couple of 100 records only. It failed for 2000 test recs. Only other issue I see could be some failure due to disk IO operations when there is too much load and maybe Kaggle is not able to handle it
Any suggestions or directions would be helpful…
\nRESOLUTION: It all starts when Kaggle used to have a 500 file output limit for notebooks. Users noticed and complained and Kaggle being young and nimble in those days acknowledged the problem and fixed it (nowadays they dont acknowledge also but that is a separate story).
\nBut while the core problem was fixed, some related effects still remained. For e.g. though one could create > 500 files now, the file display on the \"Data\" tab shows only 500 of them, leading to the somewhat erroneous conclusion that only 500 files were being created. A unix command shows that other files are there but are just not displayed. This is an irritant no doubt, but the users could quickly figure this out themselves.
\nAnother not-so-minor irritant happens when one tries to create a dataset out of the notebook output as is done here: https://www.kaggle.com/product-feedback/181143. The 500 file limit makes its presence here. But at least this is debuggable.
\nLastly comes the problem with code-competitions. This time the 500 file limit is not something easily debuggable (well unless you are kind of aware of the past history of such a limit). Here we need to submit the code and the code is run with test data populated at run-time. The online editor as well as the \"save and run all\" version works perfectly fine and the 500 file limit does not apply to them. But looks like when the code is run in the background for competition evaluation the hydra raises its head again. Since debugging is not allowed to prevent data-probing, we end up getting wierd, unrelated errors. We dont know what exactly happens but once 500 files are exceeded, but there is no doubt that this results in a failed submission eventually. I tried creating a sample sub BEFORE the creation of 500+ spects. I still get a submission fail error. So it is not as if additional files are just getting auto-deleted.
\nAs of now there is no workaround, we just have to avoid generating >500 files and tweak our pipeline accordingly. This is also the first time this problem is documented. I spent some time to do so as debugging a code competitions is one of the worst experiences one can ever have.
\nI would like to Thank @michaln for helping debug this error
", "messages": [ { "id": 1772358, "postDate": "2022-04-30T06:58:20.323Z", "content": "Those who may have actually attempted to debug a code competition where the code executes properly on the (sample) test data provided but fails on the actual test data may be aware of how frustrating the process can be….esp if one has very limited time at disposal to work on the comp. There are only 4-5 generic error messages thrown and there is no guarantee that the messages are correct (for e.g. you may get notebook-exception error for a memory-exceeded error). Anyway, here is what I have discovered so far:
\nWhat I am trying to do. Create all spects at one shot and store them. Then load models and start predicting
What is happening: Code working fine with (sole)sample test OGG. Also working fine if I replicate the sample test OGG 5500 times (number of actual test records). But fails during evaluation. First used to throw notebook exception and then after some debugging & changes, it starts to throw sub not found error off late
Root cause: No issues with model or other things. The code fails at generating the spects. Initially I suspected num_workers > 1 causes these things (in past discussions, parallel processes have been suspect), but even single process fails. Finally commenting the line where I save the Mel spects eliminates the error
Why is this happening: This is where I need some help. I can see that each spect is roughly 300KB. So 5500 files would take a max of 2 GB of disk space. I think we have 19.6 GB disk space available. CPU RAM is 16 GB. During local execution(when I take the solo test.OGG and add 5500 duplicates and create spects for all of them), I see that at max, couple of GB are used during the spect extraction process. So it isn’t a RAM problem either (more so because the program works fine when I comment the save-to-disk line of code). By all accounts this is a local disk issue. But test records are all of 60 sec duration. There are 5500 of them. I am sure these won’t exceed couple of GB of disk space. And moreover, the submission works fine when I generate spects at runtime for couple of 100 records only. It failed for 2000 test recs. Only other issue I see could be some failure due to disk IO operations when there is too much load and maybe Kaggle is not able to handle it
Any suggestions or directions would be helpful…
\nRESOLUTION: It all starts when Kaggle used to have a 500 file output limit for notebooks. Users noticed and complained and Kaggle being young and nimble in those days acknowledged the problem and fixed it (nowadays they dont acknowledge also but that is a separate story).
\nBut while the core problem was fixed, some related effects still remained. For e.g. though one could create > 500 files now, the file display on the \"Data\" tab shows only 500 of them, leading to the somewhat erroneous conclusion that only 500 files were being created. A unix command shows that other files are there but are just not displayed. This is an irritant no doubt, but the users could quickly figure this out themselves.
\nAnother not-so-minor irritant happens when one tries to create a dataset out of the notebook output as is done here: https://www.kaggle.com/product-feedback/181143. The 500 file limit makes its presence here. But at least this is debuggable.
\nLastly comes the problem with code-competitions. This time the 500 file limit is not something easily debuggable (well unless you are kind of aware of the past history of such a limit). Here we need to submit the code and the code is run with test data populated at run-time. The online editor as well as the \"save and run all\" version works perfectly fine and the 500 file limit does not apply to them. But looks like when the code is run in the background for competition evaluation the hydra raises its head again. Since debugging is not allowed to prevent data-probing, we end up getting wierd, unrelated errors. We dont know what exactly happens but once 500 files are exceeded, but there is no doubt that this results in a failed submission eventually. I tried creating a sample sub BEFORE the creation of 500+ spects. I still get a submission fail error. So it is not as if additional files are just getting auto-deleted.
\nAs of now there is no workaround, we just have to avoid generating >500 files and tweak our pipeline accordingly. This is also the first time this problem is documented. I spent some time to do so as debugging a code competitions is one of the worst experiences one can ever have.
\nI would like to Thank @michaln for helping debug this error
", "rawMarkdown": "Those who may have actually attempted to debug a code competition where the code executes properly on the (sample) test data provided but fails on the actual test data may be aware of how frustrating the process can be….esp if one has very limited time at disposal to work on the comp. There are only 4-5 generic error messages thrown and there is no guarantee that the messages are correct (for e.g. you may get notebook-exception error for a memory-exceeded error). Anyway, here is what I have discovered so far:\n\n- What I am trying to do. Create all spects at one shot and store them. Then load models and start predicting\n\n- What is happening: Code working fine with (sole)sample test OGG. Also working fine if I replicate the sample test OGG 5500 times (number of actual test records). But fails during evaluation. First used to throw notebook exception and then after some debugging & changes, it starts to throw sub not found error off late\n\n- Root cause: No issues with model or other things. The code fails at generating the spects. Initially I suspected num_workers > 1 causes these things (in past discussions, parallel processes have been suspect), but even single process fails. Finally commenting the line where I save the Mel spects eliminates the error\n\n- Why is this happening: This is where I need some help. I can see that each spect is roughly 300KB. So 5500 files would take a max of 2 GB of disk space. I think we have 19.6 GB disk space available. CPU RAM is 16 GB. During local execution(when I take the solo test.OGG and add 5500 duplicates and create spects for all of them), I see that at max, couple of GB are used during the spect extraction process. So it isn’t a RAM problem either (more so because the program works fine when I comment the save-to-disk line of code). By all accounts this is a local disk issue. But test records are all of 60 sec duration. There are 5500 of them. I am sure these won’t exceed couple of GB of disk space. And moreover, the submission works fine when I generate spects at runtime for couple of 100 records only. It failed for 2000 test recs. Only other issue I see could be some failure due to disk IO operations when there is too much load and maybe Kaggle is not able to handle it\n\nAny suggestions or directions would be helpful...\n\nRESOLUTION: It all starts when Kaggle used to have a 500 file output limit for notebooks. Users noticed and complained and Kaggle being young and nimble in those days acknowledged the problem and fixed it (nowadays they dont acknowledge also but that is a separate story). \n\nBut while the core problem was fixed, some related effects still remained. For e.g. though one could create > 500 files now, the file display on the \"Data\" tab shows only 500 of them, leading to the somewhat erroneous conclusion that only 500 files were being created. A unix command shows that other files are there but are just not displayed. This is an irritant no doubt, but the users could quickly figure this out themselves. \n\nAnother not-so-minor irritant happens when one tries to create a dataset out of the notebook output as is done here: https://www.kaggle.com/product-feedback/181143. The 500 file limit makes its presence here. But at least this is debuggable. \n\nLastly comes the problem with code-competitions. This time the 500 file limit is not something easily debuggable (well unless you are kind of aware of the past history of such a limit). Here we need to submit the code and the code is run with test data populated at run-time. The online editor as well as the \"save and run all\" version works perfectly fine and the 500 file limit does not apply to them. But looks like when the code is run in the background for competition evaluation the hydra raises its head again. Since debugging is not allowed to prevent data-probing, we end up getting wierd, unrelated errors. We dont know what exactly happens but once 500 files are exceeded, but there is no doubt that this results in a failed submission eventually. I tried creating a sample sub BEFORE the creation of 500+ spects. I still get a submission fail error. So it is not as if additional files are just getting auto-deleted. \n\nAs of now there is no workaround, we just have to avoid generating >500 files and tweak our pipeline accordingly. This is also the first time this problem is documented. I spent some time to do so as debugging a code competitions is one of the worst experiences one can ever have. \n\nI would like to Thank @michaln for helping debug this error", "votes": 7 }, { "id": 1774648, "postDate": "2022-05-02T11:09:08.987Z", "content": "I experienced the same problem in this competition and consumed over 30 submissions to submit custom models successfully. In my pipeline, predictions are saved for each audio file and then read to generate the submission file at the end. Currently my pipeline does not save them per audio file.
\nIn fact, I believe Kaggle Notebook can generate more than 500 files. I guess when reading and scoring submission.csv, if the number of files named before \"submission.csv\" in a dictionary order exceeds the allowed amount (probably around 100), the submission system will not be able to find submission.csv
\nFor example, this notebook stores spectrograms in each file, but the submission succeeded because the spectrograms are stored in the directory \"test_images\".
", "rawMarkdown": "I experienced the same problem in this competition and consumed over 30 submissions to submit custom models successfully. In my pipeline, predictions are saved for each audio file and then read to generate the submission file at the end. Currently my pipeline does not save them per audio file.\n\nIn fact, I believe Kaggle Notebook can generate more than 500 files. I guess when reading and scoring submission.csv, if the number of files named before \"submission.csv\" in a dictionary order exceeds the allowed amount (probably around 100), the submission system will not be able to find submission.csv\n\nFor example, [this notebook](https://www.kaggle.com/code/jirkaborovec/birdclef-lightning-flash-inference) stores spectrograms in each file, but the submission succeeded because the spectrograms are stored in the directory \"test_images\".", "votes": 2, "replies": [ { "id": 1774672, "postDate": "2022-05-02T11:26:16.947Z", "content": "Thanks. This is interesting. I did have an audio_images directory where the spects are stored and the submission file is created in the root. I verified that if I limit my num of spects to around 490 it works fine and something above 510 it fails (no other change in code). I will have a look at the kernel you shared.
", "rawMarkdown": "Thanks. This is interesting. I did have an audio_images directory where the spects are stored and the submission file is created in the root. I verified that if I limit my num of spects to around 490 it works fine and something above 510 it fails (no other change in code). I will have a look at the kernel you shared." } ] }, { "id": 1772855, "postDate": "2022-04-30T16:02:36.183Z", "content": "Notebooks used to fail when you output too many files but that seems to change. Now if you output lots of files it only keeps 500 of them and throws away the rest:
\nhttps://www.kaggle.com/michaln/output-many-files-test
So i guess when you save your spects and submission file the submission file is not kept and is deleted with most of the spects and your submission fails.
\nTry to delete the generated spects at the end of the notebook.
@michaln Brilliant, I believe you may be right (will verify it tomorrow). Strangely this limit is nowhere documented by Kaggle. An announcement last year on increasing the size of Kaggle outputs makes no such reference - https://www.kaggle.com/product-feedback/195163. Indeed the interactive editor and even the \"save all and commit\" version has no such limitations on num of output files. I can generate and save as many of them. It is the code competition run that seems to have this limitation and this compounds the debugging.
\nA tiny hint that there is some file count limit that manifests itself occasionally comes from here - https://www.kaggle.com/product-feedback/181143. The author creates 2000+ files but when he tries to create a dataset with this output, he runs into the 500 file limit.
\nStrange nobody has reported this issue so far. But I guess code competitions are a recent phenomenon (2 years). Most of the inference could be online in a continuous pipeline flow (generate image for batch, predict, repeat). I will change my logic too accordingly. But @sohier, @addisonhoward, this information needs to be added to the documentation under code competitions guide & or debugging pages.
", "rawMarkdown": "@michaln Brilliant, I believe you may be right (will verify it tomorrow). Strangely this limit is nowhere documented by Kaggle. An announcement last year on increasing the size of Kaggle outputs makes no such reference - https://www.kaggle.com/product-feedback/195163. Indeed the interactive editor and even the \"save all and commit\" version has no such limitations on num of output files. I can generate and save as many of them. It is the code competition run that seems to have this limitation and this compounds the debugging.\n\nA tiny hint that there is some file count limit that manifests itself occasionally comes from here - https://www.kaggle.com/product-feedback/181143. The author creates 2000+ files but when he tries to create a dataset with this output, he runs into the 500 file limit. \n\nStrange nobody has reported this issue so far. But I guess code competitions are a recent phenomenon (2 years). Most of the inference could be online in a continuous pipeline flow (generate image for batch, predict, repeat). I will change my logic too accordingly. But @sohier, @addisonhoward, this information needs to be added to the documentation under code competitions guide & or debugging pages.", "votes": 1 }, { "id": 1772953, "postDate": "2022-04-30T17:57:34.860Z", "content": "I couldn't find the information anywhere either so I was a little bit surprised. I remember that if you create too many nested directories the notebook fails in the end with \"nice\" error message: \"Output path … contains too many nested subdirectories (max 6)0\" which is also not mentioned anywhere (not sure it is still happening though) and I think it used to fail for too many files too.
\nRemoving the files from output without an error and without mentioning it anywhere is pretty unusual and makes debugging during code competitions even more tricky. But I think people either don't generate so many files or if they do they zip/delete them (at least that's what I do) so it's usually not an issue.
\nAnyway I hope it will solve your submission problem. Good luck in the competition.
", "rawMarkdown": "I couldn't find the information anywhere either so I was a little bit surprised. I remember that if you create too many nested directories the notebook fails in the end with \"nice\" error message: \"Output path ... contains too many nested subdirectories (max 6)0\" which is also not mentioned anywhere (not sure it is still happening though) and I think it used to fail for too many files too. \n\nRemoving the files from output without an error and without mentioning it anywhere is pretty unusual and makes debugging during code competitions even more tricky. But I think people either don't generate so many files or if they do they zip/delete them (at least that's what I do) so it's usually not an issue.\n\nAnyway I hope it will solve your submission problem. Good luck in the competition.", "votes": 1 }, { "id": 1774469, "postDate": "2022-05-02T07:35:47.920Z", "content": "Thanks @michaln Yes the problem is with the 500 file limit which seems to apply only during code evaluation stage. I have updated the details in the main topic above, with the findings as this piece of information is not documented anywhere.
", "rawMarkdown": "Thanks @michaln Yes the problem is with the 500 file limit which seems to apply only during code evaluation stage. I have updated the details in the main topic above, with the findings as this piece of information is not documented anywhere. ", "votes": 1 } ] }, { "id": 1772364, "postDate": "2022-04-30T07:12:07.153Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1774648, "author_name": "ynktk", "author_url": "", "post_date": "2022-05-02T11:09:08.987000", "content": "I experienced the same problem in this competition and consumed over 30 submissions to submit custom models successfully. In my pipeline, predictions are saved for each audio file and then read to generate the submission file at the end. Currently my pipeline does not save them per audio file.
\nIn fact, I believe Kaggle Notebook can generate more than 500 files. I guess when reading and scoring submission.csv, if the number of files named before \"submission.csv\" in a dictionary order exceeds the allowed amount (probably around 100), the submission system will not be able to find submission.csv
\nFor example, this notebook stores spectrograms in each file, but the submission succeeded because the spectrograms are stored in the directory \"test_images\".
", "votes": 2, "replies": [ { "id": 1774672, "author_name": "Allohvk", "author_url": "", "post_date": "2022-05-02T11:26:16.947000", "content": "Thanks. This is interesting. I did have an audio_images directory where the spects are stored and the submission file is created in the root. I verified that if I limit my num of spects to around 490 it works fine and something above 510 it fails (no other change in code). I will have a look at the kernel you shared.
", "votes": 0, "replies": [] } ] }, { "id": 1772855, "author_name": "Michal", "author_url": "", "post_date": "2022-04-30T16:02:36.183000", "content": "Notebooks used to fail when you output too many files but that seems to change. Now if you output lots of files it only keeps 500 of them and throws away the rest:
\nhttps://www.kaggle.com/michaln/output-many-files-test
So i guess when you save your spects and submission file the submission file is not kept and is deleted with most of the spects and your submission fails.
\nTry to delete the generated spects at the end of the notebook.
@michaln Brilliant, I believe you may be right (will verify it tomorrow). Strangely this limit is nowhere documented by Kaggle. An announcement last year on increasing the size of Kaggle outputs makes no such reference - https://www.kaggle.com/product-feedback/195163. Indeed the interactive editor and even the \"save all and commit\" version has no such limitations on num of output files. I can generate and save as many of them. It is the code competition run that seems to have this limitation and this compounds the debugging.
\nA tiny hint that there is some file count limit that manifests itself occasionally comes from here - https://www.kaggle.com/product-feedback/181143. The author creates 2000+ files but when he tries to create a dataset with this output, he runs into the 500 file limit.
\nStrange nobody has reported this issue so far. But I guess code competitions are a recent phenomenon (2 years). Most of the inference could be online in a continuous pipeline flow (generate image for batch, predict, repeat). I will change my logic too accordingly. But @sohier, @addisonhoward, this information needs to be added to the documentation under code competitions guide & or debugging pages.
", "votes": 1, "replies": [] }, { "id": 1772953, "author_name": "Michal", "author_url": "", "post_date": "2022-04-30T17:57:34.860000", "content": "I couldn't find the information anywhere either so I was a little bit surprised. I remember that if you create too many nested directories the notebook fails in the end with \"nice\" error message: \"Output path … contains too many nested subdirectories (max 6)0\" which is also not mentioned anywhere (not sure it is still happening though) and I think it used to fail for too many files too.
\nRemoving the files from output without an error and without mentioning it anywhere is pretty unusual and makes debugging during code competitions even more tricky. But I think people either don't generate so many files or if they do they zip/delete them (at least that's what I do) so it's usually not an issue.
\nAnyway I hope it will solve your submission problem. Good luck in the competition.
", "votes": 1, "replies": [] }, { "id": 1774469, "author_name": "Allohvk", "author_url": "", "post_date": "2022-05-02T07:35:47.920000", "content": "Thanks @michaln Yes the problem is with the 500 file limit which seems to apply only during code evaluation stage. I have updated the details in the main topic above, with the findings as this piece of information is not documented anywhere.
", "votes": 1, "replies": [] } ] }, { "id": 1772364, "author_name": "", "author_url": "", "post_date": "2022-04-30T07:12:07.153000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1772358": "Those who may have actually attempted to debug a code competition where the code executes properly on the (sample) test data provided but fails on the actual test data may be aware of how frustrating the process can be….esp if one has very limited time at disposal to work on the comp. There are only 4-5 generic error messages thrown and there is no guarantee that the messages are correct (for e.g. you may get notebook-exception error for a memory-exceeded error). Anyway, here is what I have discovered so far:\n\n- What I am trying to do. Create all spects at one shot and store them. Then load models and start predicting\n\n- What is happening: Code working fine with (sole)sample test OGG. Also working fine if I replicate the sample test OGG 5500 times (number of actual test records). But fails during evaluation. First used to throw notebook exception and then after some debugging & changes, it starts to throw sub not found error off late\n\n- Root cause: No issues with model or other things. The code fails at generating the spects. Initially I suspected num_workers > 1 causes these things (in past discussions, parallel processes have been suspect), but even single process fails. Finally commenting the line where I save the Mel spects eliminates the error\n\n- Why is this happening: This is where I need some help. I can see that each spect is roughly 300KB. So 5500 files would take a max of 2 GB of disk space. I think we have 19.6 GB disk space available. CPU RAM is 16 GB. During local execution(when I take the solo test.OGG and add 5500 duplicates and create spects for all of them), I see that at max, couple of GB are used during the spect extraction process. So it isn’t a RAM problem either (more so because the program works fine when I comment the save-to-disk line of code). By all accounts this is a local disk issue. But test records are all of 60 sec duration. There are 5500 of them. I am sure these won’t exceed couple of GB of disk space. And moreover, the submission works fine when I generate spects at runtime for couple of 100 records only. It failed for 2000 test recs. Only other issue I see could be some failure due to disk IO operations when there is too much load and maybe Kaggle is not able to handle it\n\nAny suggestions or directions would be helpful...\n\nRESOLUTION: It all starts when Kaggle used to have a 500 file output limit for notebooks. Users noticed and complained and Kaggle being young and nimble in those days acknowledged the problem and fixed it (nowadays they dont acknowledge also but that is a separate story). \n\nBut while the core problem was fixed, some related effects still remained. For e.g. though one could create > 500 files now, the file display on the \"Data\" tab shows only 500 of them, leading to the somewhat erroneous conclusion that only 500 files were being created. A unix command shows that other files are there but are just not displayed. This is an irritant no doubt, but the users could quickly figure this out themselves. \n\nAnother not-so-minor irritant happens when one tries to create a dataset out of the notebook output as is done here: https://www.kaggle.com/product-feedback/181143. The 500 file limit makes its presence here. But at least this is debuggable. \n\nLastly comes the problem with code-competitions. This time the 500 file limit is not something easily debuggable (well unless you are kind of aware of the past history of such a limit). Here we need to submit the code and the code is run with test data populated at run-time. The online editor as well as the \"save and run all\" version works perfectly fine and the 500 file limit does not apply to them. But looks like when the code is run in the background for competition evaluation the hydra raises its head again. Since debugging is not allowed to prevent data-probing, we end up getting wierd, unrelated errors. We dont know what exactly happens but once 500 files are exceeded, but there is no doubt that this results in a failed submission eventually. I tried creating a sample sub BEFORE the creation of 500+ spects. I still get a submission fail error. So it is not as if additional files are just getting auto-deleted. \n\nAs of now there is no workaround, we just have to avoid generating >500 files and tweak our pipeline accordingly. This is also the first time this problem is documented. I spent some time to do so as debugging a code competitions is one of the worst experiences one can ever have. \n\nI would like to Thank @michaln for helping debug this error", "1774648": "I experienced the same problem in this competition and consumed over 30 submissions to submit custom models successfully. In my pipeline, predictions are saved for each audio file and then read to generate the submission file at the end. Currently my pipeline does not save them per audio file.\n\nIn fact, I believe Kaggle Notebook can generate more than 500 files. I guess when reading and scoring submission.csv, if the number of files named before \"submission.csv\" in a dictionary order exceeds the allowed amount (probably around 100), the submission system will not be able to find submission.csv\n\nFor example, [this notebook](https://www.kaggle.com/code/jirkaborovec/birdclef-lightning-flash-inference) stores spectrograms in each file, but the submission succeeded because the spectrograms are stored in the directory \"test_images\".", "1772855": "Notebooks used to fail when you output too many files but that seems to change. Now if you output lots of files it only keeps 500 of them and throws away the rest:\nhttps://www.kaggle.com/michaln/output-many-files-test\n\nSo i guess when you save your spects and submission file the submission file is not kept and is deleted with most of the spects and your submission fails.\nTry to delete the generated spects at the end of the notebook.", "1772364": "" } }