{ "id": 532156, "title": "Submission Scoring Error", "url": "/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/532156", "author_name": "", "post_date": "2024-09-04T19:00:42.807840900Z", "votes": null, "comment_count": 14, "views": 0, "content": "

We get Submission Scoring Error when submitting this notebook

\n

We managed to do a dummy submission with 1/3 predicted probability for each class without any problem here

\n

We already wasted couple submission and we are still struggling to find the cause of the problem.

\n

Thank you for your help

", "messages": [ { "id": "2979390", "postDate": "09/04/2024 19:00:42", "content": "

We get Submission Scoring Error when submitting this notebook

\n

We managed to do a dummy submission with 1/3 predicted probability for each class without any problem here

\n

We already wasted couple submission and we are still struggling to find the cause of the problem.

\n

Thank you for your help

", "rawMarkdown": "We get Submission Scoring Error when submitting [this notebook](https://www.kaggle.com/code/fmgsf12/unet-1channel-submission-revised) \n\nWe managed to do a dummy submission with 1/3 predicted probability for each class without any problem [here](https://www.kaggle.com/fmgsf12/dummysubmission) \n\nWe already wasted couple submission and we are still struggling to find the cause of the problem.\n\nThank you for your help", "votes": null }, { "id": "2979875", "postDate": "09/05/2024 08:28:27", "content": "

for label in labels:\n assert not (np.isnan(submission_df[label])).all()

\n

Are you sure your model can't output nans even for hidden test?

", "rawMarkdown": "`for label in labels:\n assert not (np.isnan(submission_df[label])).all()`\n\nAre you sure your model can't output nans even for hidden test?", "votes": null }, { "id": "2979916", "postDate": "09/05/2024 09:05:31", "content": "

but if this was the case wouldn't it have raised an assertion error and changed the kaggle error message to \"Notebook Threw an Exception\"?

", "rawMarkdown": "but if this was the case wouldn't it have raised an assertion error and changed the kaggle error message to \"Notebook Threw an Exception\"?", "votes": null }, { "id": "2979918", "postDate": "09/05/2024 09:11:07", "content": "

Actually I'm not 100% sure. But I think no.

\n

So you are not receiving:

\n

\"Notebook Threw Exception
\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\"

", "rawMarkdown": "Actually I'm not 100% sure. But I think no.\n\nSo you are not receiving:\n\n\"Notebook Threw Exception\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\"", "votes": null }, { "id": "2979921", "postDate": "09/05/2024 09:13:51", "content": "

I'm actually receiving \"Submission Scoring Error\", sorry for the confusion

", "rawMarkdown": "I'm actually receiving \"Submission Scoring Error\", sorry for the confusion", "votes": null }, { "id": "2979974", "postDate": "09/05/2024 10:40:14", "content": "

If the exact same code succed with just imputing 1/3 everywhere. I think it only can be the output itself, may be some rows doesnt sum 1.

", "rawMarkdown": "If the exact same code succed with just imputing 1/3 everywhere. I think it only can be the output itself, may be some rows doesnt sum 1.", "votes": null }, { "id": "2980031", "postDate": "09/05/2024 11:48:01", "content": "

thank you for the hint, I'll try to submit the dummy predictions with the same code and see what happens

", "rawMarkdown": "thank you for the hint, I'll try to submit the dummy predictions with the same code and see what happens", "votes": null }, { "id": "2980053", "postDate": "09/05/2024 12:29:15", "content": "

The submission succeded without any problem

\n

I guess maybe I will try to make sure that all the rows sum to 1

\n

Thank you again for your suggestion

", "rawMarkdown": "The submission succeded without any problem\n\nI guess maybe I will try to make sure that all the rows sum to 1\n\nThank you again for your suggestion", "votes": null }, { "id": "2980736", "postDate": "09/06/2024 06:05:28", "content": "

my code generally works after i do something like df = df.fillna(0.33)

", "rawMarkdown": "my code generally works after i do something like `df = df.fillna(0.33)`", "votes": null }, { "id": "2980889", "postDate": "09/06/2024 09:11:55", "content": "

I tried normalising the row but it still given the some error

", "rawMarkdown": "I tried normalising the row but it still given the some error", "votes": null }, { "id": "2980891", "postDate": "09/06/2024 09:15:46", "content": "

Thank you for the reply. The thing os that the code is still passing the assertion, so there shouldn’t be any nan in the dataframe

", "rawMarkdown": "Thank you for the reply. The thing os that the code is still passing the assertion, so there shouldn’t be any nan in the dataframe", "votes": null }, { "id": "2980946", "postDate": "09/06/2024 10:25:46", "content": "

So just to clarify. I f you run your code and at the very end right before submission.to_csv() you overwrite submission[['Normal/Mild','Moderate','Severe']] = 1./3 it succed. But without overwrite the predictions it gets score error. That's correct?

", "rawMarkdown": "So just to clarify. I f you run your code and at the very end right before submission.to_csv() you overwrite submission[['Normal/Mild','Moderate','Severe']] = 1./3 it succed. But without overwrite the predictions it gets score error. That's correct?", "votes": null }, { "id": "2981028", "postDate": "09/06/2024 12:37:59", "content": "

Yes that is exactly the case

", "rawMarkdown": "Yes that is exactly the case", "votes": null }, { "id": "2981035", "postDate": "09/06/2024 12:46:45", "content": "

So In my opinion there is no doubt. There is something wrong with the values. If I was you I would remove the final assertions and instead add:

\n

submission = submission.fillna(1./3) #Rohit Chaudhari

\n

v = submission[['normal_mild','moderate','severe']].values

\n

v = v/v.sum(1).reshape(-1,1)

\n

submission[['normal_mild','moderate','severe']] = v

\n

Should run.

", "rawMarkdown": "So In my opinion there is no doubt. There is something wrong with the values. If I was you I would remove the final assertions and instead add: \n\nsubmission = submission.fillna(1./3) #Rohit Chaudhari\n\nv = submission[['normal_mild','moderate','severe']].values\n\nv = v/v.sum(1).reshape(-1,1)\n\nsubmission[['normal_mild','moderate','severe']] = v\n\nShould run.", "votes": null }, { "id": "2992908", "postDate": "09/19/2024 08:30:01", "content": "

Hi @fmgsf12 ,

\n

if you are still facing issue try this dataset to get better visibility over what is happening at inference time link

", "rawMarkdown": "Hi @fmgsf12 ,\n\nif you are still facing issue try this dataset to get better visibility over what is happening at inference time [link](https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/523247)", "votes": null } ], "comments": [ { "id": 2979875, "author_name": "sacuscreed", "author_url": "", "post_date": "09/05/2024 08:28:27", "content": "

for label in labels:\n assert not (np.isnan(submission_df[label])).all()

\n

Are you sure your model can't output nans even for hidden test?

", "votes": null, "replies": [ { "id": 2979916, "author_name": "fmgsf12", "author_url": "", "post_date": "09/05/2024 09:05:31", "content": "

but if this was the case wouldn't it have raised an assertion error and changed the kaggle error message to \"Notebook Threw an Exception\"?

", "votes": null, "replies": [ { "id": 2979918, "author_name": "sacuscreed", "author_url": "", "post_date": "09/05/2024 09:11:07", "content": "

Actually I'm not 100% sure. But I think no.

\n

So you are not receiving:

\n

\"Notebook Threw Exception
\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\"

", "votes": null, "replies": [ { "id": 2979921, "author_name": "fmgsf12", "author_url": "", "post_date": "09/05/2024 09:13:51", "content": "

I'm actually receiving \"Submission Scoring Error\", sorry for the confusion

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2979974, "author_name": "sacuscreed", "author_url": "", "post_date": "09/05/2024 10:40:14", "content": "

If the exact same code succed with just imputing 1/3 everywhere. I think it only can be the output itself, may be some rows doesnt sum 1.

", "votes": null, "replies": [ { "id": 2980031, "author_name": "fmgsf12", "author_url": "", "post_date": "09/05/2024 11:48:01", "content": "

thank you for the hint, I'll try to submit the dummy predictions with the same code and see what happens

", "votes": null, "replies": [] }, { "id": 2980053, "author_name": "fmgsf12", "author_url": "", "post_date": "09/05/2024 12:29:15", "content": "

The submission succeded without any problem

\n

I guess maybe I will try to make sure that all the rows sum to 1

\n

Thank you again for your suggestion

", "votes": null, "replies": [] }, { "id": 2980889, "author_name": "fmgsf12", "author_url": "", "post_date": "09/06/2024 09:11:55", "content": "

I tried normalising the row but it still given the some error

", "votes": null, "replies": [] } ] }, { "id": 2980736, "author_name": "rohitchaudhari25", "author_url": "", "post_date": "09/06/2024 06:05:28", "content": "

my code generally works after i do something like df = df.fillna(0.33)

", "votes": null, "replies": [ { "id": 2980891, "author_name": "fmgsf12", "author_url": "", "post_date": "09/06/2024 09:15:46", "content": "

Thank you for the reply. The thing os that the code is still passing the assertion, so there shouldn’t be any nan in the dataframe

", "votes": null, "replies": [] } ] }, { "id": 2980946, "author_name": "sacuscreed", "author_url": "", "post_date": "09/06/2024 10:25:46", "content": "

So just to clarify. I f you run your code and at the very end right before submission.to_csv() you overwrite submission[['Normal/Mild','Moderate','Severe']] = 1./3 it succed. But without overwrite the predictions it gets score error. That's correct?

", "votes": null, "replies": [ { "id": 2981028, "author_name": "fmgsf12", "author_url": "", "post_date": "09/06/2024 12:37:59", "content": "

Yes that is exactly the case

", "votes": null, "replies": [ { "id": 2981035, "author_name": "sacuscreed", "author_url": "", "post_date": "09/06/2024 12:46:45", "content": "

So In my opinion there is no doubt. There is something wrong with the values. If I was you I would remove the final assertions and instead add:

\n

submission = submission.fillna(1./3) #Rohit Chaudhari

\n

v = submission[['normal_mild','moderate','severe']].values

\n

v = v/v.sum(1).reshape(-1,1)

\n

submission[['normal_mild','moderate','severe']] = v

\n

Should run.

", "votes": null, "replies": [] } ] } ] }, { "id": 2992908, "author_name": "rohitchaudhari25", "author_url": "", "post_date": "09/19/2024 08:30:01", "content": "

Hi @fmgsf12 ,

\n

if you are still facing issue try this dataset to get better visibility over what is happening at inference time link

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2979390": "We get Submission Scoring Error when submitting [this notebook](https://www.kaggle.com/code/fmgsf12/unet-1channel-submission-revised) \n\nWe managed to do a dummy submission with 1/3 predicted probability for each class without any problem [here](https://www.kaggle.com/fmgsf12/dummysubmission) \n\nWe already wasted couple submission and we are still struggling to find the cause of the problem.\n\nThank you for your help", "2979875": "`for label in labels:\n assert not (np.isnan(submission_df[label])).all()`\n\nAre you sure your model can't output nans even for hidden test?", "2979916": "but if this was the case wouldn't it have raised an assertion error and changed the kaggle error message to \"Notebook Threw an Exception\"?", "2979918": "Actually I'm not 100% sure. But I think no.\n\nSo you are not receiving:\n\n\"Notebook Threw Exception\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\"", "2979921": "I'm actually receiving \"Submission Scoring Error\", sorry for the confusion", "2979974": "If the exact same code succed with just imputing 1/3 everywhere. I think it only can be the output itself, may be some rows doesnt sum 1.", "2980031": "thank you for the hint, I'll try to submit the dummy predictions with the same code and see what happens", "2980053": "The submission succeded without any problem\n\nI guess maybe I will try to make sure that all the rows sum to 1\n\nThank you again for your suggestion", "2980736": "my code generally works after i do something like `df = df.fillna(0.33)`", "2980889": "I tried normalising the row but it still given the some error", "2980891": "Thank you for the reply. The thing os that the code is still passing the assertion, so there shouldn’t be any nan in the dataframe", "2980946": "So just to clarify. I f you run your code and at the very end right before submission.to_csv() you overwrite submission[['Normal/Mild','Moderate','Severe']] = 1./3 it succed. But without overwrite the predictions it gets score error. That's correct?", "2981028": "Yes that is exactly the case", "2981035": "So In my opinion there is no doubt. There is something wrong with the values. If I was you I would remove the final assertions and instead add: \n\nsubmission = submission.fillna(1./3) #Rohit Chaudhari\n\nv = submission[['normal_mild','moderate','severe']].values\n\nv = v/v.sum(1).reshape(-1,1)\n\nsubmission[['normal_mild','moderate','severe']] = v\n\nShould run.", "2992908": "Hi @fmgsf12 ,\n\nif you are still facing issue try this dataset to get better visibility over what is happening at inference time [link](https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/523247)" }, "source": "meta" }