{ "id": 485836, "title": "Submissions are not working?", "url": "/competitions/home-credit-credit-risk-model-stability/discussion/485836", "author_name": "", "post_date": "2024-03-22T10:51:58.188627700Z", "votes": 4, "comment_count": 7, "views": 0, "content": "

I try to submit my work, and notebook is working fine, but it inevitably fails with Submission Scoring Error, despite submission file meets all the requirements (10 rows, case_id int, score float between 0 and 1). More than that, I tried to submit sample submission, output by starter notebook, which was scored fine previously, and it also fails. I see people face the same problem

\n

https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483635
\n(first comment here) https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483262

\n

but their concerns remain unanswered. @jetakow Can you please give any comments on that issue?

", "messages": [ { "id": "2710544", "postDate": "03/22/2024 10:51:58", "content": "

I try to submit my work, and notebook is working fine, but it inevitably fails with Submission Scoring Error, despite submission file meets all the requirements (10 rows, case_id int, score float between 0 and 1). More than that, I tried to submit sample submission, output by starter notebook, which was scored fine previously, and it also fails. I see people face the same problem

\n

https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483635
\n(first comment here) https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483262

\n

but their concerns remain unanswered. @jetakow Can you please give any comments on that issue?

", "rawMarkdown": "I try to submit my work, and notebook is working fine, but it inevitably fails with Submission Scoring Error, despite submission file meets all the requirements (10 rows, case_id int, score float between 0 and 1). More than that, I tried to submit sample submission, output by starter notebook, which was scored fine previously, and it also fails. I see people face the same problem\n\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483635\n(first comment here) https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483262\n\nbut their concerns remain unanswered. @jetakow Can you please give any comments on that issue?", "votes": null }, { "id": "2711486", "postDate": "03/22/2024 21:55:05", "content": "

The test files and sample_submission.csv that you see are dummy files. When you submit your notebook, these files are replaced with the actual test data, and your notebook is run against them. The actual test files are larger (the host mentioned that the whole test set is about 90% of the size of the training set, and the data used to evaluate the public leaderboard score constitutes about 30% of the entire test set). You don't have access to them because they are intended to imitate future, unseen data.

\n

This means that even if your code works fine on the dummy test files, it may encounter issues during the actual evaluation. Such issues could stem from format discrepancies, duplicated case IDs, among other reasons.

\n

To debug these problems, I'd suggest creating a test environment where you train your model on, say, 10% of the train case IDs and run your submission script on the other 90% of train case IDs. This approach will help you identify any errors without exhausting your submission quota.

", "rawMarkdown": "The test files and sample_submission.csv that you see are dummy files. When you submit your notebook, these files are replaced with the actual test data, and your notebook is run against them. The actual test files are larger (the host mentioned that the whole test set is about 90% of the size of the training set, and the data used to evaluate the public leaderboard score constitutes about 30% of the entire test set). You don't have access to them because they are intended to imitate future, unseen data. \n\nThis means that even if your code works fine on the dummy test files, it may encounter issues during the actual evaluation. Such issues could stem from format discrepancies, duplicated case IDs, among other reasons. \n\nTo debug these problems, I'd suggest creating a test environment where you train your model on, say, 10% of the train case IDs and run your submission script on the other 90% of train case IDs. This approach will help you identify any errors without exhausting your submission quota.", "votes": null }, { "id": "2712094", "postDate": "03/23/2024 09:46:08", "content": "

Same problem here, no solution yet.

\n\n

I already tried to:

\n\n

The sample of the submission file i created looks good.
\nIts called submission.csv and has these top 3 lines

\n

case_id,score
\n57633,0.0312
\n57634,0.1542
\n57630,0.054

", "rawMarkdown": "Same problem here, no solution yet. \n\n- My submission code runs green. \n- Then the Scoring Fails.\n- It already worked in the past\n\nI already tried to:\n- substitute null in score with 0\n- dedup on case_id\n- rounded score to 4 positions\n- use outer joins in case a case_id is not included in base file, but checked during eval\n\nThe sample of the submission file i created looks good.\nIts called submission.csv and has these top 3 lines\n\ncase_id,score\n57633,0.0312\n57634,0.1542\n57630,0.054", "votes": null }, { "id": "2712100", "postDate": "03/23/2024 09:50:27", "content": "

The problem the author describes is after all the code ran.
\nIts a problem exclusively related to the submission process.

\n\n

I already tried some format cleaning for potential errors in the submission file, see my other comment.

", "rawMarkdown": "The problem the author describes is after all the code ran.\nIts a problem exclusively related to the submission process.\n- Either the submission file has an error (format, name, location, values, etc.)\n- Or the submission code executed in the blackbox has a problem\n\nI already tried some format cleaning for potential errors in the submission file, see my other comment.", "votes": null }, { "id": "2712488", "postDate": "03/23/2024 15:48:26", "content": "

I am also facing similar issue. I tried belows, but the issue was not resolved.

\n\n

I'm guessing something is wrong with the system.

", "rawMarkdown": "I am also facing similar issue. I tried belows, but the issue was not resolved.\n- Even after reverting the version to the successfully submitted notebook and running it, the test data becomes 10 rows.\n- The same thing happens when you fork and run other public notebooks.\n\nI'm guessing something is wrong with the system.", "votes": null }, { "id": "2712524", "postDate": "03/23/2024 16:13:24", "content": "

Actually, that information resolved the issue for me. I was using prepared datasets, so evaluation of hidden test set was broken since it basically didn't retrieve the new data. Thanks!

", "rawMarkdown": "Actually, that information resolved the issue for me. I was using prepared datasets, so evaluation of hidden test set was broken since it basically didn't retrieve the new data. Thanks!", "votes": null }, { "id": "2712539", "postDate": "03/23/2024 16:27:39", "content": "

Ok, but the original problem is still there isn't it?
\nJust want to make sure that people who have the same problem, do not get confused by this post.

\n

It's great information and valuable advice, but will quite likely not help you solve the problem of this post.

", "rawMarkdown": "Ok, but the original problem is still there isn't it?\nJust want to make sure that people who have the same problem, do not get confused by this post.\n\nIt's great information and valuable advice, but will quite likely not help you solve the problem of this post.", "votes": null }, { "id": "2726368", "postDate": "04/01/2024 06:38:41", "content": "

Finally got it. Thanks @eivolkova
\nIt’s my first competition and I did not get that the notebook is automatically executed against the dummies first during submission

", "rawMarkdown": "Finally got it. Thanks @eivolkova \nIt’s my first competition and I did not get that the notebook is automatically executed against the dummies first during submission", "votes": null } ], "comments": [ { "id": 2711486, "author_name": "eivolkova", "author_url": "", "post_date": "03/22/2024 21:55:05", "content": "

The test files and sample_submission.csv that you see are dummy files. When you submit your notebook, these files are replaced with the actual test data, and your notebook is run against them. The actual test files are larger (the host mentioned that the whole test set is about 90% of the size of the training set, and the data used to evaluate the public leaderboard score constitutes about 30% of the entire test set). You don't have access to them because they are intended to imitate future, unseen data.

\n

This means that even if your code works fine on the dummy test files, it may encounter issues during the actual evaluation. Such issues could stem from format discrepancies, duplicated case IDs, among other reasons.

\n

To debug these problems, I'd suggest creating a test environment where you train your model on, say, 10% of the train case IDs and run your submission script on the other 90% of train case IDs. This approach will help you identify any errors without exhausting your submission quota.

", "votes": null, "replies": [ { "id": 2712100, "author_name": "aahhammer", "author_url": "", "post_date": "03/23/2024 09:50:27", "content": "

The problem the author describes is after all the code ran.
\nIts a problem exclusively related to the submission process.

\n\n

I already tried some format cleaning for potential errors in the submission file, see my other comment.

", "votes": null, "replies": [] }, { "id": 2712524, "author_name": "nerkan", "author_url": "", "post_date": "03/23/2024 16:13:24", "content": "

Actually, that information resolved the issue for me. I was using prepared datasets, so evaluation of hidden test set was broken since it basically didn't retrieve the new data. Thanks!

", "votes": null, "replies": [ { "id": 2712539, "author_name": "aahhammer", "author_url": "", "post_date": "03/23/2024 16:27:39", "content": "

Ok, but the original problem is still there isn't it?
\nJust want to make sure that people who have the same problem, do not get confused by this post.

\n

It's great information and valuable advice, but will quite likely not help you solve the problem of this post.

", "votes": null, "replies": [] } ] }, { "id": 2726368, "author_name": "aahhammer", "author_url": "", "post_date": "04/01/2024 06:38:41", "content": "

Finally got it. Thanks @eivolkova
\nIt’s my first competition and I did not get that the notebook is automatically executed against the dummies first during submission

", "votes": null, "replies": [] } ] }, { "id": 2712094, "author_name": "aahhammer", "author_url": "", "post_date": "03/23/2024 09:46:08", "content": "

Same problem here, no solution yet.

\n\n

I already tried to:

\n\n

The sample of the submission file i created looks good.
\nIts called submission.csv and has these top 3 lines

\n

case_id,score
\n57633,0.0312
\n57634,0.1542
\n57630,0.054

", "votes": null, "replies": [] }, { "id": 2712488, "author_name": "shisa07", "author_url": "", "post_date": "03/23/2024 15:48:26", "content": "

I am also facing similar issue. I tried belows, but the issue was not resolved.

\n\n

I'm guessing something is wrong with the system.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2710544": "I try to submit my work, and notebook is working fine, but it inevitably fails with Submission Scoring Error, despite submission file meets all the requirements (10 rows, case_id int, score float between 0 and 1). More than that, I tried to submit sample submission, output by starter notebook, which was scored fine previously, and it also fails. I see people face the same problem\n\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483635\n(first comment here) https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/483262\n\nbut their concerns remain unanswered. @jetakow Can you please give any comments on that issue?", "2711486": "The test files and sample_submission.csv that you see are dummy files. When you submit your notebook, these files are replaced with the actual test data, and your notebook is run against them. The actual test files are larger (the host mentioned that the whole test set is about 90% of the size of the training set, and the data used to evaluate the public leaderboard score constitutes about 30% of the entire test set). You don't have access to them because they are intended to imitate future, unseen data. \n\nThis means that even if your code works fine on the dummy test files, it may encounter issues during the actual evaluation. Such issues could stem from format discrepancies, duplicated case IDs, among other reasons. \n\nTo debug these problems, I'd suggest creating a test environment where you train your model on, say, 10% of the train case IDs and run your submission script on the other 90% of train case IDs. This approach will help you identify any errors without exhausting your submission quota.", "2712094": "Same problem here, no solution yet. \n\n- My submission code runs green. \n- Then the Scoring Fails.\n- It already worked in the past\n\nI already tried to:\n- substitute null in score with 0\n- dedup on case_id\n- rounded score to 4 positions\n- use outer joins in case a case_id is not included in base file, but checked during eval\n\nThe sample of the submission file i created looks good.\nIts called submission.csv and has these top 3 lines\n\ncase_id,score\n57633,0.0312\n57634,0.1542\n57630,0.054", "2712100": "The problem the author describes is after all the code ran.\nIts a problem exclusively related to the submission process.\n- Either the submission file has an error (format, name, location, values, etc.)\n- Or the submission code executed in the blackbox has a problem\n\nI already tried some format cleaning for potential errors in the submission file, see my other comment.", "2712488": "I am also facing similar issue. I tried belows, but the issue was not resolved.\n- Even after reverting the version to the successfully submitted notebook and running it, the test data becomes 10 rows.\n- The same thing happens when you fork and run other public notebooks.\n\nI'm guessing something is wrong with the system.", "2712524": "Actually, that information resolved the issue for me. I was using prepared datasets, so evaluation of hidden test set was broken since it basically didn't retrieve the new data. Thanks!", "2712539": "Ok, but the original problem is still there isn't it?\nJust want to make sure that people who have the same problem, do not get confused by this post.\n\nIt's great information and valuable advice, but will quite likely not help you solve the problem of this post.", "2726368": "Finally got it. Thanks @eivolkova \nIt’s my first competition and I did not get that the notebook is automatically executed against the dummies first during submission" }, "source": "meta" }