![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F349155%2Fee72d0ebb18f9c49ac98e12bd70fdc50%2Fsubmission.png?generation=1699731798183696&alt=media\")
…it's been four days since the competition started and only two people managed to submit something else than sample_submission. No matter what I try, I keep getting the dreaded 'Submission Scoring Error,' and there's no way to debug it.
\nReplacing data in the sample_submission doesn't work.
\nCreating a new submission.csv from scratch with predictions sorted by slice doesn't work.
\nMy submission.csv looks pretty unremarkable.
Do the images in the hidden test dataset have the same format, the same number of channels ? Are they more or less the same size? Are the file names/folders formatted in the same way? Maybe there are some *.tiff files instead of .tif?
\nI'm out of ideas.
", "messages": [ { "id": 2521544, "postDate": "2023-11-11T19:53:42.643Z", "content": "…it's been four days since the competition started and only two people managed to submit something else than sample_submission. No matter what I try, I keep getting the dreaded 'Submission Scoring Error,' and there's no way to debug it.
\nReplacing data in the sample_submission doesn't work.
\nCreating a new submission.csv from scratch with predictions sorted by slice doesn't work.
\nMy submission.csv looks pretty unremarkable.
Do the images in the hidden test dataset have the same format, the same number of channels ? Are they more or less the same size? Are the file names/folders formatted in the same way? Maybe there are some *.tiff files instead of .tif?
\nI'm out of ideas.
", "rawMarkdown": "...it's been four days since the competition started and only two people managed to submit something else than sample_submission. No matter what I try, I keep getting the dreaded 'Submission Scoring Error,' and there's no way to debug it.\nReplacing data in the sample_submission doesn't work.\nCreating a new submission.csv from scratch with predictions sorted by slice doesn't work.\nMy submission.csv looks pretty unremarkable.\n\n\n\nDo the images in the hidden test dataset have the same format, the same number of channels ? Are they more or less the same size? Are the file names/folders formatted in the same way? Maybe there are some *.tiff files instead of .tif?\n\nI'm out of ideas.", "votes": 8 }, { "id": 2522090, "postDate": "2023-11-12T11:30:48.940Z", "content": "Same problem here. This is what I have tried so far: https://www.kaggle.com/code/clemchris/test-submission/notebook. Will run more tests tomorrow
", "rawMarkdown": "Same problem here. This is what I have tried so far: https://www.kaggle.com/code/clemchris/test-submission/notebook. Will run more tests tomorrow", "votes": 1, "replies": [ { "id": 2522119, "postDate": "2023-11-12T11:57:54.100Z", "content": "Excellent experiments clem-chris! Will try to use sample_submission.csv as a source of file names next, probably.
", "rawMarkdown": "Excellent experiments clem-chris! Will try to use sample_submission.csv as a source of file names next, probably. ", "votes": 1, "replies": [ { "id": 2523059, "postDate": "2023-11-13T08:12:09.473Z", "content": "I added one more test and can confirm that the \"1 0\" mask is the problem
", "rawMarkdown": "I added one more test and can confirm that the \"1 0\" mask is the problem", "votes": 1 } ] } ] }, { "id": 2521559, "postDate": "2023-11-11T20:02:54.053Z", "content": "Facing the same issue. I was getting the submission scoring error whatever I’ve tried :(
", "rawMarkdown": "Facing the same issue. I was getting the submission scoring error whatever I’ve tried :( ", "votes": 1 }, { "id": 2572177, "postDate": "2023-12-24T01:20:53.307Z", "content": "New to this competition，anyone solved it？😤
", "rawMarkdown": "New to this competition，anyone solved it？😤" }, { "id": 2523080, "postDate": "2023-11-13T08:26:06.543Z", "content": "Ho-ho-hooo… Apparently, a bug in the submission validation code considers an empty prediction mask incorrect prediction and gives \" 'Submission Scoring Error'. The current workaround of submitting 1 1 is sub-optimal as it introduces a spurious pixel which is penalized by the metric.
", "rawMarkdown": "Ho-ho-hooo... Apparently, a bug in the submission validation code considers an empty prediction mask incorrect prediction and gives \" 'Submission Scoring Error'. The current workaround of submitting 1 1 is sub-optimal as it introduces a spurious pixel which is penalized by the metric.", "replies": [ { "id": 2523145, "postDate": "2023-11-13T09:56:57.120Z", "content": "I was curious about that too, so I experimented a bit.
\nI used train_rle.csv to calculate metrics for the first 300 slices of kidney_1_dense.
\nWhen the empty_mask rle was changed from \"1 0\" to \"1 1\", the score dropped just slightly.
\n1.0 → 0.9997
I really hope that the host will soon respond to this issue, because I am sure that I will be penalized. However, changing to \"1 1\" should not shift the score that much, so if you want to know the rough LB, I think this measure is currently sufficient.
", "rawMarkdown": "I was curious about that too, so I experimented a bit.\nI used train_rle.csv to calculate metrics for the first 300 slices of kidney_1_dense.\nWhen the empty_mask rle was changed from \"1 0\" to \"1 1\", the score dropped just slightly.\n**1.0 → 0.9997**\n\nI really hope that the host will soon respond to this issue, because I am sure that I will be penalized. However, changing to \"1 1\" should not shift the score that much, so if you want to know the rough LB, I think this measure is currently sufficient.", "votes": 5 } ] }, { "id": 2521809, "postDate": "2023-11-12T05:19:12.377Z", "rawMarkdown": "", "votes": -2, "isDeleted": true, "replies": [ { "id": 2521897, "postDate": "2023-11-12T06:57:00.230Z", "content": "Oh, wow. That's an unorthodox view for a competition grandmaster. To drop the problem just because of some technical difficulties? How else would you develop your problem-solving skills, then? You won't learn much just by reading.
", "rawMarkdown": "Oh, wow. That's an unorthodox view for a competition grandmaster. To drop the problem just because of some technical difficulties? How else would you develop your problem-solving skills, then? You won't learn much just by reading.", "votes": 4 } ] } ], "comments": [ { "id": 2522090, "author_name": "clem-chris", "author_url": "", "post_date": "2023-11-12T11:30:48.940000", "content": "Same problem here. This is what I have tried so far: https://www.kaggle.com/code/clemchris/test-submission/notebook. Will run more tests tomorrow
", "votes": 1, "replies": [ { "id": 2522119, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-11-12T11:57:54.100000", "content": "Excellent experiments clem-chris! Will try to use sample_submission.csv as a source of file names next, probably.
", "votes": 1, "replies": [ { "id": 2523059, "author_name": "clem-chris", "author_url": "", "post_date": "2023-11-13T08:12:09.473000", "content": "I added one more test and can confirm that the \"1 0\" mask is the problem
", "votes": 1, "replies": [] } ] } ] }, { "id": 2521559, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2023-11-11T20:02:54.053000", "content": "Facing the same issue. I was getting the submission scoring error whatever I’ve tried :(
", "votes": 1, "replies": [] }, { "id": 2572177, "author_name": "Seeing Times", "author_url": "", "post_date": "2023-12-24T01:20:53.307000", "content": "New to this competition，anyone solved it？😤
", "votes": 0, "replies": [] }, { "id": 2523080, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-11-13T08:26:06.543000", "content": "Ho-ho-hooo… Apparently, a bug in the submission validation code considers an empty prediction mask incorrect prediction and gives \" 'Submission Scoring Error'. The current workaround of submitting 1 1 is sub-optimal as it introduces a spurious pixel which is penalized by the metric.
", "votes": 0, "replies": [ { "id": 2523145, "author_name": "YumeNeko", "author_url": "", "post_date": "2023-11-13T09:56:57.120000", "content": "I was curious about that too, so I experimented a bit.
\nI used train_rle.csv to calculate metrics for the first 300 slices of kidney_1_dense.
\nWhen the empty_mask rle was changed from \"1 0\" to \"1 1\", the score dropped just slightly.
\n1.0 → 0.9997
I really hope that the host will soon respond to this issue, because I am sure that I will be penalized. However, changing to \"1 1\" should not shift the score that much, so if you want to know the rough LB, I think this measure is currently sufficient.
", "votes": 5, "replies": [] } ] }, { "id": 2521809, "author_name": "", "author_url": "", "post_date": "2023-11-12T05:19:12.377000", "content": "", "votes": -2, "replies": [ { "id": 2521897, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-11-12T06:57:00.230000", "content": "Oh, wow. That's an unorthodox view for a competition grandmaster. To drop the problem just because of some technical difficulties? How else would you develop your problem-solving skills, then? You won't learn much just by reading.
", "votes": 4, "replies": [] } ] } ], "raw_markdown_by_id": { "2521544": "...it's been four days since the competition started and only two people managed to submit something else than sample_submission. No matter what I try, I keep getting the dreaded 'Submission Scoring Error,' and there's no way to debug it.\nReplacing data in the sample_submission doesn't work.\nCreating a new submission.csv from scratch with predictions sorted by slice doesn't work.\nMy submission.csv looks pretty unremarkable.\n\n\n\nDo the images in the hidden test dataset have the same format, the same number of channels ? Are they more or less the same size? Are the file names/folders formatted in the same way? Maybe there are some *.tiff files instead of .tif?\n\nI'm out of ideas.", "2522090": "Same problem here. This is what I have tried so far: https://www.kaggle.com/code/clemchris/test-submission/notebook. Will run more tests tomorrow", "2521559": "Facing the same issue. I was getting the submission scoring error whatever I’ve tried :( ", "2572177": "New to this competition，anyone solved it？😤", "2523080": "Ho-ho-hooo... Apparently, a bug in the submission validation code considers an empty prediction mask incorrect prediction and gives \" 'Submission Scoring Error'. The current workaround of submitting 1 1 is sub-optimal as it introduces a spurious pixel which is penalized by the metric.", "2521809": "" } }