My notebook ran perfectly fine on the Kaggle kernel (8GB one) and produced submission.csv as it was supposed to, with no errors. However, when I submitted the file to the system, the notebook showed that it ran successfully but with the error message 'Submission Scoring Error', as seen in the picture below.
\nVery appreciated for any help 🙏
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6595796%2F7999ca2d16dc5809286b6a804fea3055%2Ffirefox_zb5JRRklfd.png?generation=1681242522520047&alt=media\")
\nI'm using Polars and LightGBM stack. I ran multiple tests to ensure that it's not a memory problem. Tried to submit both version that use memory_reduce_usage function but not working anyway (Most of the time, the kernel runs for less than 8 minutes and uses no more than 6GB out of 8GB.)
My submission code Tried this Not working
\nlimits = {:(,), :(,), :(,)}\n\n test, sample_submission iter_test:\n sample_submission[] = [(label.split()[][:]) label sample_submission[]]\n grp = test.level_group.values[]\n df = get_feature(test, grp)\n a,b = limits[grp]\n\n t (a,b):\n clf = optimized_model_save[]\n best_threshold = thresholds_save[][]\n p = clf.predict_proba(df.astype())[,] \n mask = sample_submission.question == t \n sample_submission.loc[mask, ] = (p > best_threshold).astype() \n\n env.predict(sample_submission[[, ]])\nWhich produce this submission file, right format.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6595796%2Fd084a0dd8220311f2f9e4fe6222460bc%2Ffirefox_owwHQMuvXQ.png?generation=1681242802228263&alt=media\")
The Final log
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6595796%2Fad4a5895be4fc30547476c5803a8fb7b%2Ffirefox_XM68Co4fKP.png?generation=1681242600010228&alt=media\")
Inside submitted notebook after the run
\n(No error at all 😨)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6595796%2F15fc9d5095198da083f2ad5daf22409e%2Ffirefox_m9PsvtjaFY.png?generation=1681243168477030&alt=media\")
My notebook ran perfectly fine on the Kaggle kernel (8GB one) and produced submission.csv as it was supposed to, with no errors. However, when I submitted the file to the system, the notebook showed that it ran successfully but with the error message 'Submission Scoring Error', as seen in the picture below.
\nVery appreciated for any help 🙏
\n
\nI'm using Polars and LightGBM stack. I ran multiple tests to ensure that it's not a memory problem. Tried to submit both version that use memory_reduce_usage function but not working anyway (Most of the time, the kernel runs for less than 8 minutes and uses no more than 6GB out of 8GB.)
My submission code Tried this Not working
\nlimits = {:(,), :(,), :(,)}\n\n test, sample_submission iter_test:\n sample_submission[] = [(label.split()[][:]) label sample_submission[]]\n grp = test.level_group.values[]\n df = get_feature(test, grp)\n a,b = limits[grp]\n\n t (a,b):\n clf = optimized_model_save[]\n best_threshold = thresholds_save[][]\n p = clf.predict_proba(df.astype())[,] \n mask = sample_submission.question == t \n sample_submission.loc[mask, ] = (p > best_threshold).astype() \n\n env.predict(sample_submission[[, ]])\nWhich produce this submission file, right format.
\n
The Final log
\n
Inside submitted notebook after the run
\n(No error at all 😨)
\n
I am facing similar issue.
\nTo debug this, I added exception catching block in my inference code where I am generating features.
\n (test, sample_submission) iter_test:\n :\n test_features = get_feature(test)\n :\n \n\n \nThis resulted in successful submission. However, the score was really poor and way off from the score I got on a holdout dataset I generated from the training set. So this leads me to think that while the submission is going through successfully (i.e. it has the right format, number of rows etc), the predicted values are somehow getting messed up. So frustrating how Kaggle provides no information here to help debug.
", "rawMarkdown": "I am facing similar issue. \n\nTo debug this, I added exception catching block in my inference code where I am generating features. \n\n```python\nfor (test, sample_submission) in iter_test:\n try:\n test_features = get_feature(test)\n except:\n break\n\n ## rest of the inference code\n```\n\nThis resulted in successful submission. However, the score was really poor and way off from the score I got on a holdout dataset I generated from the training set. So this leads me to think that while the submission is going through successfully (i.e. it has the right format, number of rows etc), the predicted values are somehow getting messed up. So frustrating how Kaggle provides no information here to help debug. ", "votes": 2 }, { "id": 2272617, "postDate": "2023-05-24T15:56:23.990Z", "content": "Hi, Thaweewat R
\nDid you solved the problem?
\nI have the same problem of \"Submission Scoring Error\" which is not related to any features extraction process.
\nIf you solved it, Can you explain how?
did you found a solution?
", "rawMarkdown": "did you found a solution?" } ] }, { "id": 2219440, "postDate": "2023-04-12T15:13:20.773Z", "content": "How do you handle missing features during inference?
\nI had this problem: I was doing a group_by and each group was turned into a new feature, but during inference some of these groups were missing (from the test set), which caused an unhelpful \"Scoring Error\"
I only submitted the simple model using elapsed time without any complicated group_by features, just for testing purposes. Therefore, I don't think this is the same problem as yours.
", "rawMarkdown": "I only submitted the simple model using elapsed time without any complicated group_by features, just for testing purposes. Therefore, I don't think this is the same problem as yours." }, { "id": 2231446, "postDate": "2023-04-23T10:12:13.473Z", "content": "So, what you do is that you use imputer objects from sklearn to handle missing features from inference. you fit them into your training data and use them to transform your test dataset. this may in some cases lead to data leakage, but also ensures that the test data remains to the same distribution as the training data. wherever, if the missingness is very low you can also drop the rows or that entire column if the missingness is too high (imputation will impose bias in this case). These are for you to handle during pre processing.
", "rawMarkdown": "So, what you do is that you use imputer objects from sklearn to handle missing features from inference. you fit them into your training data and use them to transform your test dataset. this may in some cases lead to data leakage, but also ensures that the test data remains to the same distribution as the training data. wherever, if the missingness is very low you can also drop the rows or that entire column if the missingness is too high (imputation will impose bias in this case). These are for you to handle during pre processing. ", "replies": [ { "id": 2231565, "postDate": "2023-04-23T12:44:32.897Z", "content": "I just ran the baseline notebook that actually passed at first by other people, though. I believe it's related to the API issue that the competition supervisor posted about a few days ago. I plan to find the time to test this new API update and see whether it's working or not.
", "rawMarkdown": "I just ran the baseline notebook that actually passed at first by other people, though. I believe it's related to the API issue that the competition supervisor posted about a few days ago. I plan to find the time to test this new API update and see whether it's working or not." } ] } ] }, { "id": 2222234, "postDate": "2023-04-15T04:35:52.290Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 2223345, "author_name": "Kushan Thakkar", "author_url": "", "post_date": "2023-04-16T06:42:31.140000", "content": "I am facing similar issue.
\nTo debug this, I added exception catching block in my inference code where I am generating features.
\n (test, sample_submission) iter_test:\n :\n test_features = get_feature(test)\n :\n \n\n \nThis resulted in successful submission. However, the score was really poor and way off from the score I got on a holdout dataset I generated from the training set. So this leads me to think that while the submission is going through successfully (i.e. it has the right format, number of rows etc), the predicted values are somehow getting messed up. So frustrating how Kaggle provides no information here to help debug.
", "votes": 2, "replies": [] }, { "id": 2272617, "author_name": "Space Time", "author_url": "", "post_date": "2023-05-24T15:56:23.990000", "content": "Hi, Thaweewat R
\nDid you solved the problem?
\nI have the same problem of \"Submission Scoring Error\" which is not related to any features extraction process.
\nIf you solved it, Can you explain how?
did you found a solution?
", "votes": 0, "replies": [] } ] }, { "id": 2219440, "author_name": "zack", "author_url": "", "post_date": "2023-04-12T15:13:20.773000", "content": "How do you handle missing features during inference?
\nI had this problem: I was doing a group_by and each group was turned into a new feature, but during inference some of these groups were missing (from the test set), which caused an unhelpful \"Scoring Error\"
I only submitted the simple model using elapsed time without any complicated group_by features, just for testing purposes. Therefore, I don't think this is the same problem as yours.
", "votes": 0, "replies": [] }, { "id": 2231446, "author_name": "Built By T1rth4nkar", "author_url": "", "post_date": "2023-04-23T10:12:13.473000", "content": "So, what you do is that you use imputer objects from sklearn to handle missing features from inference. you fit them into your training data and use them to transform your test dataset. this may in some cases lead to data leakage, but also ensures that the test data remains to the same distribution as the training data. wherever, if the missingness is very low you can also drop the rows or that entire column if the missingness is too high (imputation will impose bias in this case). These are for you to handle during pre processing.
", "votes": 0, "replies": [ { "id": 2231565, "author_name": "Thaweewat R", "author_url": "", "post_date": "2023-04-23T12:44:32.897000", "content": "I just ran the baseline notebook that actually passed at first by other people, though. I believe it's related to the API issue that the competition supervisor posted about a few days ago. I plan to find the time to test this new API update and see whether it's working or not.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2222234, "author_name": "", "author_url": "", "post_date": "2023-04-15T04:35:52.290000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2218560": "My notebook ran perfectly fine on the Kaggle kernel (8GB one) and produced submission.csv as it was supposed to, with no errors. However, when I submitted the file to the system, the notebook showed that it ran successfully but with the error message 'Submission Scoring Error', as seen in the picture below.\n\nVery appreciated for any help 🙏\n\n\nI'm using Polars and LightGBM stack. I ran multiple tests to ensure that it's not a memory problem. Tried to submit both version that use memory_reduce_usage function but not working anyway (Most of the time, the kernel runs for less than 8 minutes and uses no more than 6GB out of 8GB.)\n\n**My submission code** Tried [this](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396751) Not working\n\n```python\nlimits = {'0-4':(1,4), '5-12':(4,14), '13-22':(14,19)}\n\nfor test, sample_submission in iter_test:\n sample_submission['question'] = [int(label.split('_')[1][1:]) for label in sample_submission['session_id']]\n grp = test.level_group.values[0]\n df = get_feature(test, grp)\n a,b = limits[grp]\n \n for t in range(a,b):\n clf = optimized_model_save[f'model_{t}']\n best_threshold = thresholds_save[f'model_{t}'][0]\n p = clf.predict_proba(df.astype('float32'))[0,1] \n mask = sample_submission.question == t \n sample_submission.loc[mask, 'correct'] = (p > best_threshold).astype('int') \n\n env.predict(sample_submission[['session_id', 'correct']])\n```\n\n**Which produce this submission file, right format.**\n\n\n**The Final log**\n\n\n**Inside submitted notebook after the run**\n(No error at all 😨)\n\n\n", "2223345": "I am facing similar issue. \n\nTo debug this, I added exception catching block in my inference code where I am generating features. \n\n```python\nfor (test, sample_submission) in iter_test:\n try:\n test_features = get_feature(test)\n except:\n break\n\n ## rest of the inference code\n```\n\nThis resulted in successful submission. However, the score was really poor and way off from the score I got on a holdout dataset I generated from the training set. So this leads me to think that while the submission is going through successfully (i.e. it has the right format, number of rows etc), the predicted values are somehow getting messed up. So frustrating how Kaggle provides no information here to help debug. ", "2272617": "Hi, Thaweewat R\nDid you solved the problem?\nI have the same problem of \"Submission Scoring Error\" which is not related to any features extraction process.\nIf you solved it, Can you explain how?", "2219440": "How do you handle missing features during inference?\nI had this problem: I was doing a group_by and each group was turned into a new feature, but during inference some of these groups were missing (from the test set), which caused an unhelpful \"Scoring Error\"", "2222234": "" } }