{
  "id": 214394,
  "title": "What does \"Submission Scoring Error\" mean?",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/214394",
  "author_name": "",
  "post_date": "2021-01-26T14:02:09.077081400Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I actually managed to make a Submission yesterday!    It ran, as expected,for about 2 hours, but the result was the terse message \"Submission Scoring Error\".   What does this mean?    Was \"submission.csv\" missing?   Incorrect format?   Incorrect order of image_id's?   TIA,  -- Mark</p>",
  "messages": [
    {
      "id": "1170882",
      "postDate": "01/26/2021 14:02:09",
      "content": "<p>I actually managed to make a Submission yesterday!    It ran, as expected,for about 2 hours, but the result was the terse message \"Submission Scoring Error\".   What does this mean?    Was \"submission.csv\" missing?   Incorrect format?   Incorrect order of image_id's?   TIA,  -- Mark</p>",
      "rawMarkdown": "I actually managed to make a Submission yesterday!    It ran, as expected,for about 2 hours, but the result was the terse message \"Submission Scoring Error\".   What does this mean?    Was \"submission.csv\" missing?   Incorrect format?   Incorrect order of image_id's?   TIA,  -- Mark",
      "votes": null
    },
    {
      "id": "1170977",
      "postDate": "01/26/2021 15:07:47",
      "content": "<p>As far as I understand, it can be any of the ones you mentioned. </p>\n<p>If its any help, I had problems when:</p>\n<ol>\n<li>Did not spell correctly the names of the submission columns  (\"id\", \"predicted\")</li>\n<li>The submission had id missings (I was naively scoring only the 5 ids that matched the public test set, but forgot that when the submission is scored, extra images are added to the test folder 😅)</li>\n<li>The submission had all the id columns filled with the proper ids but some RLE predictions were empty strings (it seems minimum of 1 pair of numbers for each id are expected).</li>\n<li>The order of the rows did not seem to matter.</li>\n</ol>\n<p>Happy coding!</p>",
      "rawMarkdown": "As far as I understand, it can be any of the ones you mentioned. \n\nIf its any help, I had problems when:\n1. Did not spell correctly the names of the submission columns  (\"id\", \"predicted\")\n2. The submission had id missings (I was naively scoring only the 5 ids that matched the public test set, but forgot that when the submission is scored, extra images are added to the test folder 😅)\n3. The submission had all the id columns filled with the proper ids but some RLE predictions were empty strings (it seems minimum of 1 pair of numbers for each id are expected).\n4. The order of the rows did not seem to matter.\n\nHappy coding!",
      "votes": null
    },
    {
      "id": "1171369",
      "postDate": "01/26/2021 19:39:42",
      "content": "<p>My Submission, which gets \"Submission Scoring Error\", writes a lot of debugging information, both to <code>stdout</code> <strong>and</strong> <code>stderr</code>.   Any problems with that?</p>",
      "rawMarkdown": "My Submission, which gets \"Submission Scoring Error\", writes a lot of debugging information, both to ```stdout``` **and** ```stderr```.   Any problems with that?",
      "votes": null
    },
    {
      "id": "1171454",
      "postDate": "01/26/2021 20:41:28",
      "content": "<p>I got item#2 with an indirect root cause: GPU OOM. For instance GPU operations surrounded by <code>try/except</code> will make private images prediction fail and then generate submission.csv without required id. </p>",
      "rawMarkdown": "I got item#2 with an indirect root cause: GPU OOM. For instance GPU operations surrounded by `try/except` will make private images prediction fail and then generate submission.csv without required id.",
      "votes": null
    },
    {
      "id": "1171468",
      "postDate": "01/26/2021 20:54:25",
      "content": "<p>Don't think so, as long as it generates a submission.csv file, I think it's all that matters. <br>\nWhen a new version of the notebook to be submitted is committed, does the generated csv have the proper name,column names, ids (without the .tiff) and RLE encoded segments? <br>\nAre the ids hardcoded, or calculated from the names of the files in the test folder?</p>\n<p>Maybe you could generate a csv file that scores 0% at the end of the notebook (by generating a submission with the file ids and \"0 1\" as predictions) to make sure that the code runs completely, so if it scores 0 % it would mean that the notebook ran completely. If it gets a Submission Scoring Error again (assuming the dummy submission had the proper column names, id names….) it would mean something in the code breaks before the file is generated. Making submissions moving this snippet across the notebook might help you identify wich part of the code is the one that breaks while testing the private dataset.</p>",
      "rawMarkdown": "Don't think so, as long as it generates a submission.csv file, I think it's all that matters. \nWhen a new version of the notebook to be submitted is committed, does the generated csv have the proper name,column names, ids (without the .tiff) and RLE encoded segments? \nAre the ids hardcoded, or calculated from the names of the files in the test folder?\n\nMaybe you could generate a csv file that scores 0% at the end of the notebook (by generating a submission with the file ids and \"0 1\" as predictions) to make sure that the code runs completely, so if it scores 0 % it would mean that the notebook ran completely. If it gets a Submission Scoring Error again (assuming the dummy submission had the proper column names, id names....) it would mean something in the code breaks before the file is generated. Making submissions moving this snippet across the notebook might help you identify wich part of the code is the one that breaks while testing the private dataset.",
      "votes": null
    },
    {
      "id": "1171566",
      "postDate": "01/27/2021 00:32:36",
      "content": "<p>maybe，your generated submission file is incorrect.you can try it.</p>\n<p>sample_submission_df = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv',index_col='id')<br>\nsample_submission_df.loc[idxs] = encs<br>\nsample_submission_df.to_csv('./submission.csv')</p>",
      "rawMarkdown": "maybe，your generated submission file is incorrect.you can try it.\n\n\nsample_submission_df = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv',index_col='id')\nsample_submission_df.loc[idxs] = encs\nsample_submission_df.to_csv('./submission.csv')",
      "votes": null
    },
    {
      "id": "1172848",
      "postDate": "01/27/2021 15:14:01",
      "content": "<p>When I run my notebook on the test/ cases using [Save Version] it produces a <code>submission.csv</code> that I am able to read back in using <code>read_csv</code>, translate from RLE to binary image, and display it.   The results look pretty reasonable, collections of glom markers that follow the general shape of the test images.   So… it seems to be something specific to the Submission environment (which, of course, includes the hidden test cases).</p>",
      "rawMarkdown": "When I run my notebook on the test/ cases using [Save Version] it produces a ```submission.csv``` that I am able to read back in using ```read_csv```, translate from RLE to binary image, and display it.   The results look pretty reasonable, collections of glom markers that follow the general shape of the test images.   So... it seems to be something specific to the Submission environment (which, of course, includes the hidden test cases).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1170977,
      "author_name": "sirarslaan",
      "author_url": "",
      "post_date": "01/26/2021 15:07:47",
      "content": "<p>As far as I understand, it can be any of the ones you mentioned. </p>\n<p>If its any help, I had problems when:</p>\n<ol>\n<li>Did not spell correctly the names of the submission columns  (\"id\", \"predicted\")</li>\n<li>The submission had id missings (I was naively scoring only the 5 ids that matched the public test set, but forgot that when the submission is scored, extra images are added to the test folder 😅)</li>\n<li>The submission had all the id columns filled with the proper ids but some RLE predictions were empty strings (it seems minimum of 1 pair of numbers for each id are expected).</li>\n<li>The order of the rows did not seem to matter.</li>\n</ol>\n<p>Happy coding!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1171454,
          "author_name": "mpware",
          "author_url": "",
          "post_date": "01/26/2021 20:41:28",
          "content": "<p>I got item#2 with an indirect root cause: GPU OOM. For instance GPU operations surrounded by <code>try/except</code> will make private images prediction fail and then generate submission.csv without required id. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1171369,
      "author_name": "markalavin",
      "author_url": "",
      "post_date": "01/26/2021 19:39:42",
      "content": "<p>My Submission, which gets \"Submission Scoring Error\", writes a lot of debugging information, both to <code>stdout</code> <strong>and</strong> <code>stderr</code>.   Any problems with that?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1171468,
          "author_name": "sirarslaan",
          "author_url": "",
          "post_date": "01/26/2021 20:54:25",
          "content": "<p>Don't think so, as long as it generates a submission.csv file, I think it's all that matters. <br>\nWhen a new version of the notebook to be submitted is committed, does the generated csv have the proper name,column names, ids (without the .tiff) and RLE encoded segments? <br>\nAre the ids hardcoded, or calculated from the names of the files in the test folder?</p>\n<p>Maybe you could generate a csv file that scores 0% at the end of the notebook (by generating a submission with the file ids and \"0 1\" as predictions) to make sure that the code runs completely, so if it scores 0 % it would mean that the notebook ran completely. If it gets a Submission Scoring Error again (assuming the dummy submission had the proper column names, id names….) it would mean something in the code breaks before the file is generated. Making submissions moving this snippet across the notebook might help you identify wich part of the code is the one that breaks while testing the private dataset.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1171566,
      "author_name": "zzemily",
      "author_url": "",
      "post_date": "01/27/2021 00:32:36",
      "content": "<p>maybe，your generated submission file is incorrect.you can try it.</p>\n<p>sample_submission_df = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv',index_col='id')<br>\nsample_submission_df.loc[idxs] = encs<br>\nsample_submission_df.to_csv('./submission.csv')</p>",
      "votes": null,
      "replies": [
        {
          "id": 1172848,
          "author_name": "markalavin",
          "author_url": "",
          "post_date": "01/27/2021 15:14:01",
          "content": "<p>When I run my notebook on the test/ cases using [Save Version] it produces a <code>submission.csv</code> that I am able to read back in using <code>read_csv</code>, translate from RLE to binary image, and display it.   The results look pretty reasonable, collections of glom markers that follow the general shape of the test images.   So… it seems to be something specific to the Submission environment (which, of course, includes the hidden test cases).</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1170882": "I actually managed to make a Submission yesterday!    It ran, as expected,for about 2 hours, but the result was the terse message \"Submission Scoring Error\".   What does this mean?    Was \"submission.csv\" missing?   Incorrect format?   Incorrect order of image_id's?   TIA,  -- Mark",
    "1170977": "As far as I understand, it can be any of the ones you mentioned. \n\nIf its any help, I had problems when:\n1. Did not spell correctly the names of the submission columns  (\"id\", \"predicted\")\n2. The submission had id missings (I was naively scoring only the 5 ids that matched the public test set, but forgot that when the submission is scored, extra images are added to the test folder 😅)\n3. The submission had all the id columns filled with the proper ids but some RLE predictions were empty strings (it seems minimum of 1 pair of numbers for each id are expected).\n4. The order of the rows did not seem to matter.\n\nHappy coding!",
    "1171369": "My Submission, which gets \"Submission Scoring Error\", writes a lot of debugging information, both to ```stdout``` **and** ```stderr```.   Any problems with that?",
    "1171454": "I got item#2 with an indirect root cause: GPU OOM. For instance GPU operations surrounded by `try/except` will make private images prediction fail and then generate submission.csv without required id.",
    "1171468": "Don't think so, as long as it generates a submission.csv file, I think it's all that matters. \nWhen a new version of the notebook to be submitted is committed, does the generated csv have the proper name,column names, ids (without the .tiff) and RLE encoded segments? \nAre the ids hardcoded, or calculated from the names of the files in the test folder?\n\nMaybe you could generate a csv file that scores 0% at the end of the notebook (by generating a submission with the file ids and \"0 1\" as predictions) to make sure that the code runs completely, so if it scores 0 % it would mean that the notebook ran completely. If it gets a Submission Scoring Error again (assuming the dummy submission had the proper column names, id names....) it would mean something in the code breaks before the file is generated. Making submissions moving this snippet across the notebook might help you identify wich part of the code is the one that breaks while testing the private dataset.",
    "1171566": "maybe，your generated submission file is incorrect.you can try it.\n\n\nsample_submission_df = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv',index_col='id')\nsample_submission_df.loc[idxs] = encs\nsample_submission_df.to_csv('./submission.csv')",
    "1172848": "When I run my notebook on the test/ cases using [Save Version] it produces a ```submission.csv``` that I am able to read back in using ```read_csv```, translate from RLE to binary image, and display it.   The results look pretty reasonable, collections of glom markers that follow the general shape of the test images.   So... it seems to be something specific to the Submission environment (which, of course, includes the hidden test cases)."
  },
  "source": "meta"
}