Hi everyone,
\nI'm trying to deduce why my submission on the private leaderboard scored 0 and trying to create a sample submission script that generates random masks (so that I don't wait hours for the actual model to work) and creates submission.csv from those masks
\nWould you please have a look at my notebook and give me a hint on why my submission.csv gives a \"Submission Scoring Error\"?
\n[https://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62675235] (see version 18)
\nThanks!
Hi everyone,
\nI'm trying to deduce why my submission on the private leaderboard scored 0 and trying to create a sample submission script that generates random masks (so that I don't wait hours for the actual model to work) and creates submission.csv from those masks
\nWould you please have a look at my notebook and give me a hint on why my submission.csv gives a \"Submission Scoring Error\"?
\n[https://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62675235] (see version 18)
\nThanks!
@philculliton
\nI think I finally pinpointed the issue.
\nIt seems that the tifffile library is the culprit. It works fine on the public test files but fails on the first private test file. I'm not talking about reading the file into the memory. I'm talking about just accessing the file with the library.
\nThis function gets the tiff file shape. It does not read the file into the memory and thus it's not an OOM error.
def get_shape(tiffile):\n tif=tiff.TiffFile(tiffile)\n shape=tif.series[0].shape\n print('get shape', shape)\n return shape\nIf I add this function call to my script - the submit fails (\"Submission scoring error\"), even though the script finishes its job without giving any errors. The submission.csv also looks fine.
\nIf I comment this line out - the script passes and submission works as expected.
\nI was experiencing these unexplainable \"Submission scoring error\"s during the competition. So I decided to amend the sample_submission.csv instead of recreating it from the scratch. Thus my inference script passes the public test without problems but for the public test it submitted empty lines. Thus the visible public score was fine, but the private test was 0.
\nI think it's unfair to punish those of us who used tifffile library with 0 scores on public only because there was some peculiarly formatted file that causes tifffile library to fail, while working perfectly normal on the public dataset tiffs.
\nThere appears to be some sort of an exception handling bug on the kaggle side. It seems that when the notebook cell fails with an exception it just continues to the next cell without halting the script. This results in an incomplete submission.csv and eventually 0 score on the private test.
\nYou can try my notebook and see it yourself (line 6 contains the cell which you can comment out):
\nhttps://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62678919
Also try to work around with my zero-scored submission.
\nUse single network instead ensemble and comment all post-processing things give me successful private score.
\nNow try to extend it piece by piece.
NOTE: I'm using Rasterio instead of tifffile.
\nInitial consumption: hidden OOM issue on private data submission.
\nSeems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.
\nRegarding your notebook: did you try to visualize what you generate?
", "rawMarkdown": "Also try to work around with my zero-scored submission.\n\nUse single network instead ensemble and comment all post-processing things give me successful private score.\nNow try to extend it piece by piece.\n\nNOTE: I'm using Rasterio instead of tifffile.\n\nInitial consumption: hidden OOM issue on private data submission.\n\nSeems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.\n\nRegarding your notebook: did you try to visualize what you generate?", "votes": null }, { "id": "1304299", "postDate": "05/12/2021 14:42:01", "content": "See my answer below. It's some sort of bug in the kaggle handling of errors. It seems that when the notebook cell fails with an error it just continues to the next cell without halting the script. This results in an incomplete submission.csv and eventually 0 score on the private test.
", "rawMarkdown": "See my answer below. It's some sort of bug in the kaggle handling of errors. It seems that when the notebook cell fails with an error it just continues to the next cell without halting the script. This results in an incomplete submission.csv and eventually 0 score on the private test.", "votes": null }, { "id": "1304311", "postDate": "05/12/2021 14:49:08", "content": "yes, look like this.
\nThe same behavior as during commit.
Before it was better when it takes 9 hours to submit and then report an error…
\nLike kaggle updates, make bugs worse!
\nAnd the saddest that this update take place during active competition with OOM problems from the beginning.
the behavior of continuing to next cell when prior one fails is something Jupyter notebook does in this case, not the kaggle team. I've seen such incidents, although they were harmless in my case. We need to figure out what type of errors will cause the notebook to stop further processing.
", "rawMarkdown": "the behavior of continuing to next cell when prior one fails is something Jupyter notebook does in this case, not the kaggle team. I've seen such incidents, although they were harmless in my case. We need to figure out what type of errors will cause the notebook to stop further processing.", "votes": null }, { "id": "1304533", "postDate": "05/12/2021 17:19:33", "content": "Before it works in different way.
\nIf there was OOM issue it runs 9 hours and show error in the end, now it become even successful for public and 0 scored for private.
How debug this?
\nBut main thing that it changes during competition…
", "rawMarkdown": "Before it works in different way. \nIf there was OOM issue it runs 9 hours and show error in the end, now it become even successful for public and 0 scored for private.\n\nHow debug this?\n\nBut main thing that it changes during competition...", "votes": null }, { "id": "1304560", "postDate": "05/12/2021 17:42:02", "content": "@andrasferenczi What do you mean Jupyter Notebook does it? Jupyter notebook stops at an exception.
", "rawMarkdown": "andrasferenczi What do you mean Jupyter Notebook does it? Jupyter notebook stops at an exception.", "votes": null }, { "id": "1304591", "postDate": "05/12/2021 18:09:11", "content": "@sakvaua ,if I were to speculate, I would think the jupyter notebook is run from a script, such as papermill or runipy, or perhaps the convert the notebook to a .py file using nbconvert. I don't know how the latter would behave, but in the first 2 cases the notebook runs \"as is\". So any errors thrown/swallowed will be treated accordingly
", "rawMarkdown": "sakvaua ,if I were to speculate, I would think the jupyter notebook is run from a script, such as papermill or runipy, or perhaps the convert the notebook to a .py file using nbconvert. I don't know how the latter would behave, but in the first 2 cases the notebook runs \"as is\". So any errors thrown/swallowed will be treated accordingly", "votes": null }, { "id": "1305204", "postDate": "05/13/2021 06:37:13", "content": "@sakvaua what andras says is true. If you have your loop for the test images in one cell and the subsequent cell has the submission create from dictionary or whatever then convert to csv, if the loop cell fails it will still go on to do the next cell. If the loop is using the sample submission ids or the submission create is filling predictions in it only, you can end up with a sample submission of just ids and blank preds or NaNs. see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/data?scriptVersionId=56895272 (v8 of this)
\nOOM is a bit trickier and depends on when memory allocation error occurs. Have plenty of examples for TPU where the notebook will fail totally and get \"Your notebook tried to allocate more memory than is available.\" That won't produce a version of notebook with cells at all. If that happened in a submission I would presume it would show as some error under my submissions.
\nBut other examples where the training cell will get \"resource exhausted. attempting to allocate xxx\" and in those cases the notebook runs and creates a version with cells but of course is a failure. If that cell fails then subsequently a submission file is created in future cells then you can still get a file created but blank or NaNs.
\nYou can change from notebook to script by using File - > Editor Type -> Script and maybe doing that for submissions is better. It should fail totally if anything fails as there are no cells.
", "rawMarkdown": "sakvaua what andras says is true. If you have your loop for the test images in one cell and the subsequent cell has the submission create from dictionary or whatever then convert to csv, if the loop cell fails it will still go on to do the next cell. If the loop is using the sample submission ids or the submission create is filling predictions in it only, you can end up with a sample submission of just ids and blank preds or NaNs. see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/data?scriptVersionId=56895272 (v8 of this) \n\nOOM is a bit trickier and depends on when memory allocation error occurs. Have plenty of examples for TPU where the notebook will fail totally and get \"Your notebook tried to allocate more memory than is available.\" That won't produce a version of notebook with cells at all. If that happened in a submission I would presume it would show as some error under my submissions. \n\nBut other examples where the training cell will get \"resource exhausted. attempting to allocate xxx\" and in those cases the notebook runs and creates a version with cells but of course is a failure. If that cell fails then subsequently a submission file is created in future cells then you can still get a file created but blank or NaNs.\n\nYou can change from notebook to script by using File - > Editor Type -> Script and maybe doing that for submissions is better. It should fail totally if anything fails as there are no cells.", "votes": null }, { "id": "1305378", "postDate": "05/13/2021 08:29:52", "content": "Well, having the benefit of the visible private leaderboard I can do a lot of things to prevent 0 scores from happening. This is a type of knowledge I wish I had during the competition.
\nI see no valid reason to continue running notebook cells after an unhandled exception happened. The script should abort and not proceed with the creation of the csv file using incomplete data.
I do think changing from notebook to script by using File - > Editor Type -> Script and using that for submissions once you have a working version should help . It should fail totally if anything fails as there are no cells. Then under your submissions it should have Notebook Threw Exception Error -
\nThis is an example I just ran to test as a script instead of a notebook. Both Public and Private have error. Previously Public as a notebook would have a score and Status Succeeded
Submission and Description Status Private Score Public Score Use for Final Score
\nHuBMAP sub s2v3 script Notebook Threw Exception Error Error
\n(version 5/5)
\n37 minutes ago by something4kag
@philculliton
\nI think I finally pinpointed the issue.
\nIt seems that the tifffile library is the culprit. It works fine on the public test files but fails on the first private test file. I'm not talking about reading the file into the memory. I'm talking about just accessing the file with the library.
\nThis function gets the tiff file shape. It does not read the file into the memory and thus it's not an OOM error.
def get_shape(tiffile):\n tif=tiff.TiffFile(tiffile)\n shape=tif.series[0].shape\n print('get shape', shape)\n return shape\nIf I add this function call to my script - the submit fails (\"Submission scoring error\"), even though the script finishes its job without giving any errors. The submission.csv also looks fine.
\nIf I comment this line out - the script passes and submission works as expected.
\nI was experiencing these unexplainable \"Submission scoring error\"s during the competition. So I decided to amend the sample_submission.csv instead of recreating it from the scratch. Thus my inference script passes the public test without problems but for the public test it submitted empty lines. Thus the visible public score was fine, but the private test was 0.
\nI think it's unfair to punish those of us who used tifffile library with 0 scores on public only because there was some peculiarly formatted file that causes tifffile library to fail, while working perfectly normal on the public dataset tiffs.
\nThere appears to be some sort of an exception handling bug on the kaggle side. It seems that when the notebook cell fails with an exception it just continues to the next cell without halting the script. This results in an incomplete submission.csv and eventually 0 score on the private test.
\nYou can try my notebook and see it yourself (line 6 contains the cell which you can comment out):
\nhttps://www.kaggle.com/sakvaua/testtifffilesubmit?scriptVersionId=62678919
the behavior of continuing to next cell when prior one fails is something Jupyter notebook does in this case, not the kaggle team. I've seen such incidents, although they were harmless in my case. We need to figure out what type of errors will cause the notebook to stop further processing.
", "votes": null, "replies": [] }, { "id": 1304533, "author_name": "hypocrites", "author_url": "", "post_date": "05/12/2021 17:19:33", "content": "Before it works in different way.
\nIf there was OOM issue it runs 9 hours and show error in the end, now it become even successful for public and 0 scored for private.
How debug this?
\nBut main thing that it changes during competition…
", "votes": null, "replies": [] }, { "id": 1304560, "author_name": "sakvaua", "author_url": "", "post_date": "05/12/2021 17:42:02", "content": "@andrasferenczi What do you mean Jupyter Notebook does it? Jupyter notebook stops at an exception.
", "votes": null, "replies": [] }, { "id": 1304591, "author_name": "andrasferenczi", "author_url": "", "post_date": "05/12/2021 18:09:11", "content": "@sakvaua ,if I were to speculate, I would think the jupyter notebook is run from a script, such as papermill or runipy, or perhaps the convert the notebook to a .py file using nbconvert. I don't know how the latter would behave, but in the first 2 cases the notebook runs \"as is\". So any errors thrown/swallowed will be treated accordingly
", "votes": null, "replies": [] }, { "id": 1305204, "author_name": "something4kag", "author_url": "", "post_date": "05/13/2021 06:37:13", "content": "@sakvaua what andras says is true. If you have your loop for the test images in one cell and the subsequent cell has the submission create from dictionary or whatever then convert to csv, if the loop cell fails it will still go on to do the next cell. If the loop is using the sample submission ids or the submission create is filling predictions in it only, you can end up with a sample submission of just ids and blank preds or NaNs. see https://www.kaggle.com/matjes/hubmap-efficient-sampling-deepflash2-sub/data?scriptVersionId=56895272 (v8 of this)
\nOOM is a bit trickier and depends on when memory allocation error occurs. Have plenty of examples for TPU where the notebook will fail totally and get \"Your notebook tried to allocate more memory than is available.\" That won't produce a version of notebook with cells at all. If that happened in a submission I would presume it would show as some error under my submissions.
\nBut other examples where the training cell will get \"resource exhausted. attempting to allocate xxx\" and in those cases the notebook runs and creates a version with cells but of course is a failure. If that cell fails then subsequently a submission file is created in future cells then you can still get a file created but blank or NaNs.
\nYou can change from notebook to script by using File - > Editor Type -> Script and maybe doing that for submissions is better. It should fail totally if anything fails as there are no cells.
", "votes": null, "replies": [] }, { "id": 1305378, "author_name": "sakvaua", "author_url": "", "post_date": "05/13/2021 08:29:52", "content": "Well, having the benefit of the visible private leaderboard I can do a lot of things to prevent 0 scores from happening. This is a type of knowledge I wish I had during the competition.
\nI see no valid reason to continue running notebook cells after an unhandled exception happened. The script should abort and not proceed with the creation of the csv file using incomplete data.
I do think changing from notebook to script by using File - > Editor Type -> Script and using that for submissions once you have a working version should help . It should fail totally if anything fails as there are no cells. Then under your submissions it should have Notebook Threw Exception Error -
\nThis is an example I just ran to test as a script instead of a notebook. Both Public and Private have error. Previously Public as a notebook would have a score and Status Succeeded
Submission and Description Status Private Score Public Score Use for Final Score
\nHuBMAP sub s2v3 script Notebook Threw Exception Error Error
\n(version 5/5)
\n37 minutes ago by something4kag
Also try to work around with my zero-scored submission.
\nUse single network instead ensemble and comment all post-processing things give me successful private score.
\nNow try to extend it piece by piece.
NOTE: I'm using Rasterio instead of tifffile.
\nInitial consumption: hidden OOM issue on private data submission.
\nSeems there are a lot updates on Kaggle during last time and handling of this issue was bad and probably become worse.
\nRegarding your notebook: did you try to visualize what you generate?
", "votes": null, "replies": [ { "id": 1304299, "author_name": "sakvaua", "author_url": "", "post_date": "05/12/2021 14:42:01", "content": "See my answer below. It's some sort of bug in the kaggle handling of errors. It seems that when the notebook cell fails with an error it just continues to the next cell without halting the script. This results in an incomplete submission.csv and eventually 0 score on the private test.
", "votes": null, "replies": [] }, { "id": 1304311, "author_name": "hypocrites", "author_url": "", "post_date": "05/12/2021 14:49:08", "content": "yes, look like this.
\nThe same behavior as during commit.
Before it was better when it takes 9 hours to submit and then report an error…
\nLike kaggle updates, make bugs worse!
\nAnd the saddest that this update take place during active competition with OOM problems from the beginning.