I am going to list here some reasons why the submission fails. Please feel free to comment and add your ideas and share your experiences.
\nI confess I still have this error and I wonder why. I shall share here the various tests I did.
\nTO BE CONTINUED
", "messages": [ { "id": "2644058", "postDate": "02/09/2024 09:08:39", "content": "I am going to list here some reasons why the submission fails. Please feel free to comment and add your ideas and share your experiences.
\nI confess I still have this error and I wonder why. I shall share here the various tests I did.
\nTO BE CONTINUED
", "rawMarkdown": "I am going to list here some reasons why the submission fails. Please feel free to comment and add your ideas and share your experiences.\n\n1. Nan, Inf or -Inf in the submission file.\n2. Bad format of submission (bad column names or bad number of rows).\n3. Different column type in the hidden test set. (??)\n4. I am going to try a strategy mentioned by @ravi20076 in the comments, namely predicting by batches. Not only that, but after every batch I insert the result in the submission file and make a to_csv. Thus I hope there will be at least a valid submission, in case of timeout.\n\nI confess I still have this error and I wonder why. I shall share here the various tests I did.\n\nTO BE CONTINUED", "votes": null }, { "id": "2645506", "postDate": "02/10/2024 09:44:20", "content": "I also get notebook threw exception and i don't know why.I am so confused.
", "rawMarkdown": "I also get notebook threw exception and i don't know why.I am so confused.", "votes": null }, { "id": "2646421", "postDate": "02/10/2024 21:54:53", "content": "After lots of failed submissions before a successful one I concluded that Kaggle obfuscates some memory errors with the SUBMISSION scoring error, a fact that may mislead you in error checking.
\nMy take: Scrutinize all MERGE operations
@catadanna @georgem @sichaoshenhao I think it is high time Kaggle addressed this problem for everyone's benefit.
\nMore competitions are moving to code submissions and the sub-optimal practice of obfuscating all errors as notebook threw exception is very irritating. A lot of us face this too much in our daily submissions and waste time, energy, compute and effort and are unable to submit due to an unknown error.
The least the platform could do is as below-
\nmodel.predict_proba(Xtest). Rather loop the predictions in very small batches, say 5000 rows per batch and concatenate into a single array.i lost 3 days worth of submissions so far, but managed to widdle it down to the block of code thats causing it
\nand it's very likely actually an \"out of memory error\" even though it says \"Notebook Threw Exception\"
\ni'll know better tomorrow
\n*update it was out of memory error but said notebook threw exception
", "rawMarkdown": "i lost 3 days worth of submissions so far, but managed to widdle it down to the block of code thats causing it\n\nand it's very likely actually an \"out of memory error\" even though it says \"Notebook Threw Exception\"\n\ni'll know better tomorrow\n\n*update it was out of memory error but said notebook threw exception", "votes": null }, { "id": "2648455", "postDate": "02/12/2024 09:08:16", "content": "We could help you solve some errors in the discussion, or you could ask for help. You have to understand that making 3) to be possible it would open a gateway for new leakage of information from the test sample. Only this way we have fully under control how quickly one can learn about test set.
", "rawMarkdown": "We could help you solve some errors in the discussion, or you could ask for help. You have to understand that making 3) to be possible it would open a gateway for new leakage of information from the test sample. Only this way we have fully under control how quickly one can learn about test set.", "votes": null }, { "id": "2648752", "postDate": "02/12/2024 11:49:49", "content": "There is a separate error message for memory overrun, right? Due to too much feature engineering, I exceeded the memory limit and cleaned up garbage in the code, separating model training and inference to solve this problem. The error turned into throwing an exception.
", "rawMarkdown": "There is a separate error message for memory overrun, right? Due to too much feature engineering, I exceeded the memory limit and cleaned up garbage in the code, separating model training and inference to solve this problem. The error turned into throwing an exception.", "votes": null }, { "id": "2648755", "postDate": "02/12/2024 11:57:44", "content": "Thank you for raising these questions. May I ask if there will be any issues with model. predict_proba() in this competition?My code threw an exception when submit. https://www.kaggle.com/code/yunsuxiaozi/home-credit-baseline
", "rawMarkdown": "Thank you for raising these questions. May I ask if there will be any issues with model. predict_proba() in this competition?My code threw an exception when submit. https://www.kaggle.com/code/yunsuxiaozi/home-credit-baseline", "votes": null }, { "id": "2648757", "postDate": "02/12/2024 11:58:29", "content": "yeah i think it should say \"Notebook Exceeded Allowed Compute\" according to this https://www.kaggle.com/code-competition-debugging
\nbut for me i know it was a memory error but said \"Notebook Threw Exception\"
I don't see the exception, my best guess is that you should handle new categories that might be in test sample.
", "rawMarkdown": "I don't see the exception, my best guess is that you should handle new categories that might be in test sample.", "votes": null }, { "id": "2648875", "postDate": "02/12/2024 13:29:22", "content": "Perhaps it was because the test data was too large during the model prediction and needed to be passed in in batches. I have made the necessary changes now.https://www.kaggle.com/code/yunsuxiaozi/solution-of-threw-exception
", "rawMarkdown": "Perhaps it was because the test data was too large during the model prediction and needed to be passed in in batches. I have made the necessary changes now.https://www.kaggle.com/code/yunsuxiaozi/solution-of-threw-exception", "votes": null }, { "id": "2653744", "postDate": "02/15/2024 16:23:22", "content": "Hi Daniel,
\nIm getting error (Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset), from the logs i can see the entire script runs successfully but the scoring fails, pls help ive lost 5 submissions today
It was a type error for me
\nstatic_cb_0 has different types for certai columns respect to the data we have (i think uint in our data vs int during submission)
Appreciate the added information on how deal with it. At times these situations may be overlooked
", "rawMarkdown": "Appreciate the added information on how deal with it. At times these situations may be overlooked", "votes": null }, { "id": "2657125", "postDate": "02/18/2024 10:25:56", "content": "Hello, I have noticed that training data and test data may have different dtypes in the same file and same column. I guess many errors are caused by this. Can't the official modify the original data to solve this problem?
", "rawMarkdown": "Hello, I have noticed that training data and test data may have different dtypes in the same file and same column. I guess many errors are caused by this. Can't the official modify the original data to solve this problem?", "votes": null }, { "id": "2657240", "postDate": "02/18/2024 11:49:37", "content": "@yunsuxiaozi thanks for suggestion
", "rawMarkdown": "yunsuxiaozi thanks for suggestion", "votes": null }, { "id": "2659364", "postDate": "02/19/2024 19:42:04", "content": "Lost all my submissions to this error a few days back, still not sure what's causing it. Perhaps a little more granular feedback from Kaggle would've been useful. https://www.kaggle.com/code/adityashukla1/home-credit-risk-xgboost is my Notebook, things I've tried:
\nConfusing to figure out what's causing the error, if any pointers, please let me know, thanks!
", "rawMarkdown": "Lost all my submissions to this error a few days back, still not sure what's causing it. Perhaps a little more granular feedback from Kaggle would've been useful. [https://www.kaggle.com/code/adityashukla1/home-credit-risk-xgboost](url) is my Notebook, things I've tried:\n\n1. Saving the files and the model\n2. test_df.replace([np.inf, -np.inf], np.nan, inplace=True)\n3. y_pred = np.clip(np.nan_to_num(model.predict_proba(test_df)[:, 1], nan=0.3), 0, 1)\n4. casting case_id and score to int and float\n5. df_sub[\"score\"].isnull().any()\n\nConfusing to figure out what's causing the error, if any pointers, please let me know, thanks!", "votes": null }, { "id": "2767239", "postDate": "04/22/2024 07:39:42", "content": "I ran into this problem as well. For me, it wasn't memory or Nan in the submission file that solved it, but I was using a category encoder and found that there were unseen category values in the test set.
\nTelling the category encoder how to deal with missing and unseen values resolved the issue.
", "rawMarkdown": "I ran into this problem as well. For me, it wasn't memory or Nan in the submission file that solved it, but I was using a category encoder and found that there were unseen category values in the test set. \n\nTelling the category encoder how to deal with missing and unseen values resolved the issue.", "votes": null }, { "id": "2768460", "postDate": "04/22/2024 21:09:21", "content": "I have the notebook to submit the solution https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit?scriptVersionId=173370707. I tried many attempts including predict in batch size in the test set but still have the \"Notebook Threw Exception\" when submit. Could you please help me fix any error in my code so that I can submit. I would very appreciate
", "rawMarkdown": "I have the notebook to submit the solution https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit?scriptVersionId=173370707. I tried many attempts including predict in batch size in the test set but still have the \"Notebook Threw Exception\" when submit. Could you please help me fix any error in my code so that I can submit. I would very appreciate", "votes": null }, { "id": "2771344", "postDate": "04/24/2024 07:47:55", "content": "@quangkhinguynhng
\nTry to insert a try except logic to output empty submission on fail. Something like
:\n \nexcept e:\n (e)\n ()\nIn this way your able to find piece of error in your code
", "rawMarkdown": "quangkhinguynhng \nTry to insert a try except logic to output empty submission on fail. Something like\n\n```\ntry:\n #submission pipeline\nexcept Exception as e:\n print(e)\n return_empty_score()\n```\n\nIn this way your able to find piece of error in your code", "votes": null }, { "id": "2773523", "postDate": "04/24/2024 19:11:49", "content": "Hi I have tried but I cannot see the error in my code when submit, I still has Notebook Threw Exception. Do you have any way that can help me. I have tried many attempts but cannot pass this error. Please help me, I am so depressed
", "rawMarkdown": "Hi I have tried but I cannot see the error in my code when submit, I still has Notebook Threw Exception. Do you have any way that can help me. I have tried many attempts but cannot pass this error. Please help me, I am so depressed", "votes": null }, { "id": "2773554", "postDate": "04/24/2024 19:28:32", "content": "insert try except inside esch part of the script to find the place of the error
", "rawMarkdown": "insert try except inside esch part of the script to find the place of the error", "votes": null }, { "id": "2773592", "postDate": "04/24/2024 19:45:46", "content": "Hi, I have done like what you said https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit/notebook
\nBut still the same, I have public my notebook, could you help me fix the error. Thank you very much
", "rawMarkdown": "Hi, I have done like what you said https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit/notebook\n\nBut still the same, I have public my notebook, could you help me fix the error. Thank you very much", "votes": null }, { "id": "2803269", "postDate": "05/09/2024 12:07:50", "content": "That probably explains my issue too. Thank you for this.
", "rawMarkdown": "That probably explains my issue too. Thank you for this.", "votes": null } ], "comments": [ { "id": 2645506, "author_name": "sichaoshenhao", "author_url": "", "post_date": "02/10/2024 09:44:20", "content": "I also get notebook threw exception and i don't know why.I am so confused.
", "votes": null, "replies": [] }, { "id": 2646421, "author_name": "georgem", "author_url": "", "post_date": "02/10/2024 21:54:53", "content": "After lots of failed submissions before a successful one I concluded that Kaggle obfuscates some memory errors with the SUBMISSION scoring error, a fact that may mislead you in error checking.
\nMy take: Scrutinize all MERGE operations
@catadanna @georgem @sichaoshenhao I think it is high time Kaggle addressed this problem for everyone's benefit.
\nMore competitions are moving to code submissions and the sub-optimal practice of obfuscating all errors as notebook threw exception is very irritating. A lot of us face this too much in our daily submissions and waste time, energy, compute and effort and are unable to submit due to an unknown error.
The least the platform could do is as below-
\nmodel.predict_proba(Xtest). Rather loop the predictions in very small batches, say 5000 rows per batch and concatenate into a single array.We could help you solve some errors in the discussion, or you could ask for help. You have to understand that making 3) to be possible it would open a gateway for new leakage of information from the test sample. Only this way we have fully under control how quickly one can learn about test set.
", "votes": null, "replies": [ { "id": 2648755, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "02/12/2024 11:57:44", "content": "Thank you for raising these questions. May I ask if there will be any issues with model. predict_proba() in this competition?My code threw an exception when submit. https://www.kaggle.com/code/yunsuxiaozi/home-credit-baseline
", "votes": null, "replies": [ { "id": 2648815, "author_name": "jetakow", "author_url": "", "post_date": "02/12/2024 12:55:41", "content": "I don't see the exception, my best guess is that you should handle new categories that might be in test sample.
", "votes": null, "replies": [ { "id": 2648875, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "02/12/2024 13:29:22", "content": "Perhaps it was because the test data was too large during the model prediction and needed to be passed in in batches. I have made the necessary changes now.https://www.kaggle.com/code/yunsuxiaozi/solution-of-threw-exception
", "votes": null, "replies": [] }, { "id": 2657125, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "02/18/2024 10:25:56", "content": "Hello, I have noticed that training data and test data may have different dtypes in the same file and same column. I guess many errors are caused by this. Can't the official modify the original data to solve this problem?
", "votes": null, "replies": [ { "id": 2657240, "author_name": "jetakow", "author_url": "", "post_date": "02/18/2024 11:49:37", "content": "@yunsuxiaozi thanks for suggestion
", "votes": null, "replies": [] } ] } ] }, { "id": 2768460, "author_name": "quangkhinguynhng", "author_url": "", "post_date": "04/22/2024 21:09:21", "content": "I have the notebook to submit the solution https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit?scriptVersionId=173370707. I tried many attempts including predict in batch size in the test set but still have the \"Notebook Threw Exception\" when submit. Could you please help me fix any error in my code so that I can submit. I would very appreciate
", "votes": null, "replies": [ { "id": 2771344, "author_name": "stenford23", "author_url": "", "post_date": "04/24/2024 07:47:55", "content": "@quangkhinguynhng
\nTry to insert a try except logic to output empty submission on fail. Something like
:\n \nexcept e:\n (e)\n ()\nIn this way your able to find piece of error in your code
", "votes": null, "replies": [ { "id": 2773523, "author_name": "quangkhinguynhng", "author_url": "", "post_date": "04/24/2024 19:11:49", "content": "Hi I have tried but I cannot see the error in my code when submit, I still has Notebook Threw Exception. Do you have any way that can help me. I have tried many attempts but cannot pass this error. Please help me, I am so depressed
", "votes": null, "replies": [ { "id": 2773554, "author_name": "stenford23", "author_url": "", "post_date": "04/24/2024 19:28:32", "content": "insert try except inside esch part of the script to find the place of the error
", "votes": null, "replies": [ { "id": 2773592, "author_name": "quangkhinguynhng", "author_url": "", "post_date": "04/24/2024 19:45:46", "content": "Hi, I have done like what you said https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit/notebook
\nBut still the same, I have public my notebook, could you help me fix the error. Thank you very much
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2653744, "author_name": "ishan1989", "author_url": "", "post_date": "02/15/2024 16:23:22", "content": "Hi Daniel,
\nIm getting error (Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset), from the logs i can see the entire script runs successfully but the scoring fails, pls help ive lost 5 submissions today
Appreciate the added information on how deal with it. At times these situations may be overlooked
", "votes": null, "replies": [] } ] }, { "id": 2647894, "author_name": "at7459", "author_url": "", "post_date": "02/11/2024 20:52:21", "content": "i lost 3 days worth of submissions so far, but managed to widdle it down to the block of code thats causing it
\nand it's very likely actually an \"out of memory error\" even though it says \"Notebook Threw Exception\"
\ni'll know better tomorrow
\n*update it was out of memory error but said notebook threw exception
", "votes": null, "replies": [ { "id": 2648752, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "02/12/2024 11:49:49", "content": "There is a separate error message for memory overrun, right? Due to too much feature engineering, I exceeded the memory limit and cleaned up garbage in the code, separating model training and inference to solve this problem. The error turned into throwing an exception.
", "votes": null, "replies": [ { "id": 2648757, "author_name": "at7459", "author_url": "", "post_date": "02/12/2024 11:58:29", "content": "yeah i think it should say \"Notebook Exceeded Allowed Compute\" according to this https://www.kaggle.com/code-competition-debugging
\nbut for me i know it was a memory error but said \"Notebook Threw Exception\"
It was a type error for me
\nstatic_cb_0 has different types for certai columns respect to the data we have (i think uint in our data vs int during submission)
Lost all my submissions to this error a few days back, still not sure what's causing it. Perhaps a little more granular feedback from Kaggle would've been useful. https://www.kaggle.com/code/adityashukla1/home-credit-risk-xgboost is my Notebook, things I've tried:
\nConfusing to figure out what's causing the error, if any pointers, please let me know, thanks!
", "votes": null, "replies": [] }, { "id": 2767239, "author_name": "anthony35813", "author_url": "", "post_date": "04/22/2024 07:39:42", "content": "I ran into this problem as well. For me, it wasn't memory or Nan in the submission file that solved it, but I was using a category encoder and found that there were unseen category values in the test set.
\nTelling the category encoder how to deal with missing and unseen values resolved the issue.
", "votes": null, "replies": [ { "id": 2803269, "author_name": "majurranvimalan", "author_url": "", "post_date": "05/09/2024 12:07:50", "content": "That probably explains my issue too. Thank you for this.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2644058": "I am going to list here some reasons why the submission fails. Please feel free to comment and add your ideas and share your experiences.\n\n1. Nan, Inf or -Inf in the submission file.\n2. Bad format of submission (bad column names or bad number of rows).\n3. Different column type in the hidden test set. (??)\n4. I am going to try a strategy mentioned by @ravi20076 in the comments, namely predicting by batches. Not only that, but after every batch I insert the result in the submission file and make a to_csv. Thus I hope there will be at least a valid submission, in case of timeout.\n\nI confess I still have this error and I wonder why. I shall share here the various tests I did.\n\nTO BE CONTINUED", "2645506": "I also get notebook threw exception and i don't know why.I am so confused.", "2646421": "After lots of failed submissions before a successful one I concluded that Kaggle obfuscates some memory errors with the SUBMISSION scoring error, a fact that may mislead you in error checking.\nMy take: Scrutinize all MERGE operations", "2647541": "catadanna @georgem @sichaoshenhao I think it is high time Kaggle addressed this problem for everyone's benefit.\nMore competitions are moving to code submissions and the sub-optimal practice of obfuscating all errors as **notebook threw exception** is very irritating. A lot of us face this too much in our daily submissions and waste time, energy, compute and effort and are unable to submit due to an unknown error. \n\nThe least the platform could do is as below-\n1. Segregate the error by type and respond appropriately\n2. Pinpoint the location of the error in the notebook\n3. **Do not count the error as a submission. I lost 3 submissions today due to this problem**\n\n### Some additional error reasons from my end\n1. Try and predict in batches and don't use `model.predict_proba(Xtest)`. Rather loop the predictions in very small batches, say 5000 rows per batch and concatenate into a single array.\n2. Scrutinize merge operations and ensure that no duplicates/ Cartesians are being created\n3. Include a post-processing function to ensure that the predicted outputs are not nulls/ infinity values \n4. If one is editing a public notebook, please make sure that the utility script driving the code is not updated. I lost a submission yesterday due to this reason. It is always better to design one's own pipeline rather than editing public work.", "2647894": "i lost 3 days worth of submissions so far, but managed to widdle it down to the block of code thats causing it\n\nand it's very likely actually an \"out of memory error\" even though it says \"Notebook Threw Exception\"\n\ni'll know better tomorrow\n\n*update it was out of memory error but said notebook threw exception", "2648455": "We could help you solve some errors in the discussion, or you could ask for help. You have to understand that making 3) to be possible it would open a gateway for new leakage of information from the test sample. Only this way we have fully under control how quickly one can learn about test set.", "2648752": "There is a separate error message for memory overrun, right? Due to too much feature engineering, I exceeded the memory limit and cleaned up garbage in the code, separating model training and inference to solve this problem. The error turned into throwing an exception.", "2648755": "Thank you for raising these questions. May I ask if there will be any issues with model. predict_proba() in this competition?My code threw an exception when submit. https://www.kaggle.com/code/yunsuxiaozi/home-credit-baseline", "2648757": "yeah i think it should say \"Notebook Exceeded Allowed Compute\" according to this https://www.kaggle.com/code-competition-debugging\nbut for me i know it was a memory error but said \"Notebook Threw Exception\"", "2648815": "I don't see the exception, my best guess is that you should handle new categories that might be in test sample.", "2648875": "Perhaps it was because the test data was too large during the model prediction and needed to be passed in in batches. I have made the necessary changes now.https://www.kaggle.com/code/yunsuxiaozi/solution-of-threw-exception", "2653744": "Hi Daniel,\nIm getting error (Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset), from the logs i can see the entire script runs successfully but the scoring fails, pls help ive lost 5 submissions today", "2654553": "It was a type error for me\nstatic_cb_0 has different types for certai columns respect to the data we have (i think uint in our data vs int during submission)", "2654588": "Appreciate the added information on how deal with it. At times these situations may be overlooked", "2657125": "Hello, I have noticed that training data and test data may have different dtypes in the same file and same column. I guess many errors are caused by this. Can't the official modify the original data to solve this problem?", "2657240": "yunsuxiaozi thanks for suggestion", "2659364": "Lost all my submissions to this error a few days back, still not sure what's causing it. Perhaps a little more granular feedback from Kaggle would've been useful. [https://www.kaggle.com/code/adityashukla1/home-credit-risk-xgboost](url) is my Notebook, things I've tried:\n\n1. Saving the files and the model\n2. test_df.replace([np.inf, -np.inf], np.nan, inplace=True)\n3. y_pred = np.clip(np.nan_to_num(model.predict_proba(test_df)[:, 1], nan=0.3), 0, 1)\n4. casting case_id and score to int and float\n5. df_sub[\"score\"].isnull().any()\n\nConfusing to figure out what's causing the error, if any pointers, please let me know, thanks!", "2767239": "I ran into this problem as well. For me, it wasn't memory or Nan in the submission file that solved it, but I was using a category encoder and found that there were unseen category values in the test set. \n\nTelling the category encoder how to deal with missing and unseen values resolved the issue.", "2768460": "I have the notebook to submit the solution https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit?scriptVersionId=173370707. I tried many attempts including predict in batch size in the test set but still have the \"Notebook Threw Exception\" when submit. Could you please help me fix any error in my code so that I can submit. I would very appreciate", "2771344": "quangkhinguynhng \nTry to insert a try except logic to output empty submission on fail. Something like\n\n```\ntry:\n #submission pipeline\nexcept Exception as e:\n print(e)\n return_empty_score()\n```\n\nIn this way your able to find piece of error in your code", "2773523": "Hi I have tried but I cannot see the error in my code when submit, I still has Notebook Threw Exception. Do you have any way that can help me. I have tried many attempts but cannot pass this error. Please help me, I am so depressed", "2773554": "insert try except inside esch part of the script to find the place of the error", "2773592": "Hi, I have done like what you said https://www.kaggle.com/code/quangkhinguynhng/homecredit-lgm-submit/notebook\n\nBut still the same, I have public my notebook, could you help me fix the error. Thank you very much", "2803269": "That probably explains my issue too. Thank you for this." }, "source": "meta" }