{
  "id": 181512,
  "title": "\"Submission Scoring Error\"",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/181512",
  "author_name": "",
  "post_date": "2020-09-09T06:52:38.201967200Z",
  "votes": 1,
  "comment_count": 10,
  "views": 0,
  "content": "<p>I get \"Submission Scoring Error\" running on the hidden test set. Everything works fine on the training set.</p>\n<p>I'm a bit stumped. Have had the \"missing submission file\" error, but this is new to me.  </p>",
  "messages": [
    {
      "id": "1003670",
      "postDate": "09/09/2020 06:52:38",
      "content": "<p>I get \"Submission Scoring Error\" running on the hidden test set. Everything works fine on the training set.</p>\n<p>I'm a bit stumped. Have had the \"missing submission file\" error, but this is new to me.  </p>",
      "rawMarkdown": "I get \"Submission Scoring Error\" running on the hidden test set. Everything works fine on the training set.\n\n\nI'm a bit stumped. Have had the \"missing submission file\" error, but this is new to me.",
      "votes": null
    },
    {
      "id": "1003767",
      "postDate": "09/09/2020 08:26:58",
      "content": "<p>Hello, I have the same problem and no clue…</p>",
      "rawMarkdown": "Hello, I have the same problem and no clue...",
      "votes": null
    },
    {
      "id": "1003825",
      "postDate": "09/09/2020 09:32:09",
      "content": "<p>for my test <br>\nsubmission file must contain all weeks between  -12 to 133 for each patient 's FVC pred values and confidence<br>\nif you miss any one it will be error</p>\n<p>hope useful for you  </p>",
      "rawMarkdown": "for my test \nsubmission file must contain all weeks between  -12 to 133 for each patient 's FVC pred values and confidence\nif you miss any one it will be error\n\nhope useful for you",
      "votes": null
    },
    {
      "id": "1003971",
      "postDate": "09/09/2020 11:31:51",
      "content": "<p><a href=\"https://www.kaggle.com/ajenningsfrankston\" target=\"_blank\">@ajenningsfrankston</a> </p>\n<p>I'm having trouble with submission scoring errors as well. Sometimes I even have trouble with notebooks that had a successful submission but only small code changes like for example a switch of objective function… which should change nothing related to the submission. :-/ I searched a bit and found <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/148873\" target=\"_blank\">this discussion</a> useful. I copied the overview of errors.</p>\n<p>Personally I find it really bad that there is no further traceback or better description of \"what was exactly going wrong\". </p>\n<blockquote>\n  <p>Resolving Submission Errors: Below is a quick rundown of the some errors that users may receive on submission. Note that we are not in a position to share more granular details of any precise exceptions/errors, as doing so would make the competition vulnerable to error-probing by savvy participants.</p>\n  <p>Notebook Timeout: Your submission notebook exceeded the allowed runtime cap for the competition. Review the competition's Code Requirements for details, and note that the privately held rerun test set could be orders of magnitude larger than the publicly shared validation set.</p>\n  <p>Submission Scoring Error: Your notebook generated a submission.csv file with incorrect format. Some examples causing this are having the wrong number of rows or columns, empty values, an incorrect datatype for a value, or invalid submission values from what is expected for the privately rerun test set.</p>\n  <p>Notebook Threw Exception: While rerunning your code on the public test set (privately held), your notebook hit an error.</p>\n  <p>Submission CSV Not Found: Your Notebook did not output a submission.csv file. Some examples causing this are if your Notebook times out, or if it runs into an error. There is a chance this will be the error returned instead of \"Notebook Timeout\" depending on error sequencing.</p>\n  <p>No Submit to Competition button on my notebook's output: This indicates that you have enabled something that is prohibited for the submissions on that competition. Review the competition's Code Requirements for details<br>\n  Notebook Exceeded Allowed Compute: This indicates you have violated a Code Requirement constraint during the rerun.</p>\n  <p>Kaggle Error: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.</p>\n</blockquote>",
      "rawMarkdown": "ajenningsfrankston \n\nI'm having trouble with submission scoring errors as well. Sometimes I even have trouble with notebooks that had a successful submission but only small code changes like for example a switch of objective function... which should change nothing related to the submission. :-/ I searched a bit and found [this discussion](https://www.kaggle.com/c/global-wheat-detection/discussion/148873) useful. I copied the overview of errors.\n\nPersonally I find it really bad that there is no further traceback or better description of \"what was exactly going wrong\". \n\n> Resolving Submission Errors: Below is a quick rundown of the some errors that users may receive on submission. Note that we are not in a position to share more granular details of any precise exceptions/errors, as doing so would make the competition vulnerable to error-probing by savvy participants.\n\n>Notebook Timeout: Your submission notebook exceeded the allowed runtime cap for the competition. Review the competition's Code Requirements for details, and note that the privately held rerun test set could be orders of magnitude larger than the publicly shared validation set.\n\n>Submission Scoring Error: Your notebook generated a submission.csv file with incorrect format. Some examples causing this are having the wrong number of rows or columns, empty values, an incorrect datatype for a value, or invalid submission values from what is expected for the privately rerun test set.\n\n>Notebook Threw Exception: While rerunning your code on the public test set (privately held), your notebook hit an error.\n\n>Submission CSV Not Found: Your Notebook did not output a submission.csv file. Some examples causing this are if your Notebook times out, or if it runs into an error. There is a chance this will be the error returned instead of \"Notebook Timeout\" depending on error sequencing.\n\n>No Submit to Competition button on my notebook's output: This indicates that you have enabled something that is prohibited for the submissions on that competition. Review the competition's Code Requirements for details\nNotebook Exceeded Allowed Compute: This indicates you have violated a Code Requirement constraint during the rerun.\n\n>Kaggle Error: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.",
      "votes": null
    },
    {
      "id": "1004265",
      "postDate": "09/09/2020 15:12:04",
      "content": "<p>I found out one problem : I must have some NaN in my FVC values. <br>\nDoing the code below before writing the submission file enables the submission to pass : </p>\n<pre><code>submission['FVC'] = submission['FVC'].fillna(2000)\nsubmission['Confidence'] = submission['Confidence'].fillna(200)\n</code></pre>\n<p>Now, why do I have NaN in my FVC values is another question… I never had NaN either in my test or validations set…</p>",
      "rawMarkdown": "I found out one problem : I must have some NaN in my FVC values. \nDoing the code below before writing the submission file enables the submission to pass : \n```\nsubmission['FVC'] = submission['FVC'].fillna(2000)\nsubmission['Confidence'] = submission['Confidence'].fillna(200)\n\n```\nNow, why do I have NaN in my FVC values is another question... I never had NaN either in my test or validations set...",
      "votes": null
    },
    {
      "id": "1004494",
      "postDate": "09/09/2020 18:46:04",
      "content": "<p>You might be presuming something about the DICOM data:</p>\n<p>file names are not always 1.dcm, 2.dcm, etc, contiguously. Even in the training data, there are missing numbers in the sequence.</p>\n<p>Some files don't have all the Dicom tags that you might expect. Always check for the tag before using it, or use a \"try:\" construct.</p>\n<p>Depending on your model, memory usage with 200 test patients is greater than the 5 sample test patients.</p>",
      "rawMarkdown": "You might be presuming something about the DICOM data:\n\nfile names are not always 1.dcm, 2.dcm, etc, contiguously. Even in the training data, there are missing numbers in the sequence.\n\nSome files don't have all the Dicom tags that you might expect. Always check for the tag before using it, or use a \"try:\" construct.\n\nDepending on your model, memory usage with 200 test patients is greater than the 5 sample test patients.",
      "votes": null
    },
    {
      "id": "1004656",
      "postDate": "09/09/2020 22:43:51",
      "content": "<p>Thanks. I do use a \"try except\" to handle the image files. But I have not handled the exception because I assumed that the test data was all well behaved. Yes, bad idea. </p>",
      "rawMarkdown": "Thanks. I do use a \"try except\" to handle the image files. But I have not handled the exception because I assumed that the test data was all well behaved. Yes, bad idea.",
      "votes": null
    },
    {
      "id": "1005821",
      "postDate": "09/10/2020 19:36:26",
      "content": "<p>Same problem here…</p>",
      "rawMarkdown": "Same problem here...",
      "votes": null
    },
    {
      "id": "1006286",
      "postDate": "09/11/2020 06:37:30",
      "content": "<p>Thanks for the helpful comments. I now do an integrity check on the submission: looking for all Patient_Week and numbers. If any problems are found then I don't write the \"submission.csv\" file. In this way I can isolate problems.  </p>\n<p>In my code I am combining predictions from data and the images. I found that when the image file fails, I was leaving NaN in the predictions. To test this, I changed the sample test set to add in images that I know fail. I am hoping that handling all these exceptions will allow a clean submission. </p>",
      "rawMarkdown": "Thanks for the helpful comments. I now do an integrity check on the submission: looking for all Patient_Week and numbers. If any problems are found then I don't write the \"submission.csv\" file. In this way I can isolate problems.  \n\nIn my code I am combining predictions from data and the images. I found that when the image file fails, I was leaving NaN in the predictions. To test this, I changed the sample test set to add in images that I know fail. I am hoping that handling all these exceptions will allow a clean submission.",
      "votes": null
    },
    {
      "id": "1006943",
      "postDate": "09/11/2020 16:59:18",
      "content": "<p>I finnally solved, i was building paths on my custom test patients instead of building those paths on global test patients. Take care of path dirs, how do you manage test patients etc, i hope it can helps you!</p>",
      "rawMarkdown": "I finnally solved, i was building paths on my custom test patients instead of building those paths on global test patients. Take care of path dirs, how do you manage test patients etc, i hope it can helps you!",
      "votes": null
    },
    {
      "id": "1007242",
      "postDate": "09/12/2020 01:17:33",
      "content": "<p>Fixed this. Thanks. Moral of the story is never make assumptions about the test data. </p>",
      "rawMarkdown": "Fixed this. Thanks. Moral of the story is never make assumptions about the test data.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1003767,
      "author_name": "filipix",
      "author_url": "",
      "post_date": "09/09/2020 08:26:58",
      "content": "<p>Hello, I have the same problem and no clue…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1003825,
      "author_name": "tantai",
      "author_url": "",
      "post_date": "09/09/2020 09:32:09",
      "content": "<p>for my test <br>\nsubmission file must contain all weeks between  -12 to 133 for each patient 's FVC pred values and confidence<br>\nif you miss any one it will be error</p>\n<p>hope useful for you  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1003971,
      "author_name": "allunia",
      "author_url": "",
      "post_date": "09/09/2020 11:31:51",
      "content": "<p><a href=\"https://www.kaggle.com/ajenningsfrankston\" target=\"_blank\">@ajenningsfrankston</a> </p>\n<p>I'm having trouble with submission scoring errors as well. Sometimes I even have trouble with notebooks that had a successful submission but only small code changes like for example a switch of objective function… which should change nothing related to the submission. :-/ I searched a bit and found <a href=\"https://www.kaggle.com/c/global-wheat-detection/discussion/148873\" target=\"_blank\">this discussion</a> useful. I copied the overview of errors.</p>\n<p>Personally I find it really bad that there is no further traceback or better description of \"what was exactly going wrong\". </p>\n<blockquote>\n  <p>Resolving Submission Errors: Below is a quick rundown of the some errors that users may receive on submission. Note that we are not in a position to share more granular details of any precise exceptions/errors, as doing so would make the competition vulnerable to error-probing by savvy participants.</p>\n  <p>Notebook Timeout: Your submission notebook exceeded the allowed runtime cap for the competition. Review the competition's Code Requirements for details, and note that the privately held rerun test set could be orders of magnitude larger than the publicly shared validation set.</p>\n  <p>Submission Scoring Error: Your notebook generated a submission.csv file with incorrect format. Some examples causing this are having the wrong number of rows or columns, empty values, an incorrect datatype for a value, or invalid submission values from what is expected for the privately rerun test set.</p>\n  <p>Notebook Threw Exception: While rerunning your code on the public test set (privately held), your notebook hit an error.</p>\n  <p>Submission CSV Not Found: Your Notebook did not output a submission.csv file. Some examples causing this are if your Notebook times out, or if it runs into an error. There is a chance this will be the error returned instead of \"Notebook Timeout\" depending on error sequencing.</p>\n  <p>No Submit to Competition button on my notebook's output: This indicates that you have enabled something that is prohibited for the submissions on that competition. Review the competition's Code Requirements for details<br>\n  Notebook Exceeded Allowed Compute: This indicates you have violated a Code Requirement constraint during the rerun.</p>\n  <p>Kaggle Error: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.</p>\n</blockquote>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1004265,
      "author_name": "filipix",
      "author_url": "",
      "post_date": "09/09/2020 15:12:04",
      "content": "<p>I found out one problem : I must have some NaN in my FVC values. <br>\nDoing the code below before writing the submission file enables the submission to pass : </p>\n<pre><code>submission['FVC'] = submission['FVC'].fillna(2000)\nsubmission['Confidence'] = submission['Confidence'].fillna(200)\n</code></pre>\n<p>Now, why do I have NaN in my FVC values is another question… I never had NaN either in my test or validations set…</p>",
      "votes": null,
      "replies": [
        {
          "id": 1004494,
          "author_name": "richardepstein",
          "author_url": "",
          "post_date": "09/09/2020 18:46:04",
          "content": "<p>You might be presuming something about the DICOM data:</p>\n<p>file names are not always 1.dcm, 2.dcm, etc, contiguously. Even in the training data, there are missing numbers in the sequence.</p>\n<p>Some files don't have all the Dicom tags that you might expect. Always check for the tag before using it, or use a \"try:\" construct.</p>\n<p>Depending on your model, memory usage with 200 test patients is greater than the 5 sample test patients.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1004656,
          "author_name": "ajenningsfrankston",
          "author_url": "",
          "post_date": "09/09/2020 22:43:51",
          "content": "<p>Thanks. I do use a \"try except\" to handle the image files. But I have not handled the exception because I assumed that the test data was all well behaved. Yes, bad idea. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1005821,
      "author_name": "enric1296",
      "author_url": "",
      "post_date": "09/10/2020 19:36:26",
      "content": "<p>Same problem here…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1006286,
      "author_name": "ajenningsfrankston",
      "author_url": "",
      "post_date": "09/11/2020 06:37:30",
      "content": "<p>Thanks for the helpful comments. I now do an integrity check on the submission: looking for all Patient_Week and numbers. If any problems are found then I don't write the \"submission.csv\" file. In this way I can isolate problems.  </p>\n<p>In my code I am combining predictions from data and the images. I found that when the image file fails, I was leaving NaN in the predictions. To test this, I changed the sample test set to add in images that I know fail. I am hoping that handling all these exceptions will allow a clean submission. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1006943,
      "author_name": "enric1296",
      "author_url": "",
      "post_date": "09/11/2020 16:59:18",
      "content": "<p>I finnally solved, i was building paths on my custom test patients instead of building those paths on global test patients. Take care of path dirs, how do you manage test patients etc, i hope it can helps you!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1007242,
      "author_name": "ajenningsfrankston",
      "author_url": "",
      "post_date": "09/12/2020 01:17:33",
      "content": "<p>Fixed this. Thanks. Moral of the story is never make assumptions about the test data. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1003670": "I get \"Submission Scoring Error\" running on the hidden test set. Everything works fine on the training set.\n\n\nI'm a bit stumped. Have had the \"missing submission file\" error, but this is new to me.",
    "1003767": "Hello, I have the same problem and no clue...",
    "1003825": "for my test \nsubmission file must contain all weeks between  -12 to 133 for each patient 's FVC pred values and confidence\nif you miss any one it will be error\n\nhope useful for you",
    "1003971": "ajenningsfrankston \n\nI'm having trouble with submission scoring errors as well. Sometimes I even have trouble with notebooks that had a successful submission but only small code changes like for example a switch of objective function... which should change nothing related to the submission. :-/ I searched a bit and found [this discussion](https://www.kaggle.com/c/global-wheat-detection/discussion/148873) useful. I copied the overview of errors.\n\nPersonally I find it really bad that there is no further traceback or better description of \"what was exactly going wrong\". \n\n> Resolving Submission Errors: Below is a quick rundown of the some errors that users may receive on submission. Note that we are not in a position to share more granular details of any precise exceptions/errors, as doing so would make the competition vulnerable to error-probing by savvy participants.\n\n>Notebook Timeout: Your submission notebook exceeded the allowed runtime cap for the competition. Review the competition's Code Requirements for details, and note that the privately held rerun test set could be orders of magnitude larger than the publicly shared validation set.\n\n>Submission Scoring Error: Your notebook generated a submission.csv file with incorrect format. Some examples causing this are having the wrong number of rows or columns, empty values, an incorrect datatype for a value, or invalid submission values from what is expected for the privately rerun test set.\n\n>Notebook Threw Exception: While rerunning your code on the public test set (privately held), your notebook hit an error.\n\n>Submission CSV Not Found: Your Notebook did not output a submission.csv file. Some examples causing this are if your Notebook times out, or if it runs into an error. There is a chance this will be the error returned instead of \"Notebook Timeout\" depending on error sequencing.\n\n>No Submit to Competition button on my notebook's output: This indicates that you have enabled something that is prohibited for the submissions on that competition. Review the competition's Code Requirements for details\nNotebook Exceeded Allowed Compute: This indicates you have violated a Code Requirement constraint during the rerun.\n\n>Kaggle Error: A rare system error. Please try resubmitting to resolve the error and contact Kaggle Support if it persists.",
    "1004265": "I found out one problem : I must have some NaN in my FVC values. \nDoing the code below before writing the submission file enables the submission to pass : \n```\nsubmission['FVC'] = submission['FVC'].fillna(2000)\nsubmission['Confidence'] = submission['Confidence'].fillna(200)\n\n```\nNow, why do I have NaN in my FVC values is another question... I never had NaN either in my test or validations set...",
    "1004494": "You might be presuming something about the DICOM data:\n\nfile names are not always 1.dcm, 2.dcm, etc, contiguously. Even in the training data, there are missing numbers in the sequence.\n\nSome files don't have all the Dicom tags that you might expect. Always check for the tag before using it, or use a \"try:\" construct.\n\nDepending on your model, memory usage with 200 test patients is greater than the 5 sample test patients.",
    "1004656": "Thanks. I do use a \"try except\" to handle the image files. But I have not handled the exception because I assumed that the test data was all well behaved. Yes, bad idea.",
    "1005821": "Same problem here...",
    "1006286": "Thanks for the helpful comments. I now do an integrity check on the submission: looking for all Patient_Week and numbers. If any problems are found then I don't write the \"submission.csv\" file. In this way I can isolate problems.  \n\nIn my code I am combining predictions from data and the images. I found that when the image file fails, I was leaving NaN in the predictions. To test this, I changed the sample test set to add in images that I know fail. I am hoping that handling all these exceptions will allow a clean submission.",
    "1006943": "I finnally solved, i was building paths on my custom test patients instead of building those paths on global test patients. Take care of path dirs, how do you manage test patients etc, i hope it can helps you!",
    "1007242": "Fixed this. Thanks. Moral of the story is never make assumptions about the test data."
  },
  "source": "meta"
}