Most of the points causing Submission Scoring Error are discussed here , However I thought of reiterating over those points and remind participants of this topic . As it goes your submission or inference notebook should get run within 9 hours of run time . As pointed out by @sohier here , this runtime is obfuscated with Kernel rerun on test set time + Some Random Additional Time which never push the entire kernel rerun time beyond 9 hrs .
\nI was recently adding some features which were improving my Local Validation score significantly , but I was getting lots of Submission Scoring Error , Now I thought of debugging this issue step by step rather than solely relying on my submissions.
Earlier I adopted this framework of flow.
\n1) Add Features .
\n2) Build Model
\n3) Validate Cv Performance
\n4) Build Inference Kernel !
However As far as this competition is concerned this is the most unreliable and lot of Submission Scoring Error Prone Framework !
Even though I knew about Time Series Emulator Api built by tito , I never used it ! I recently looked at it here , in that we have a code segment as follows
\npbar = tqdm(total=2500000)\nprevious_test_df = None\nfor (current_test, current_prediction_df) in iter_test:\n if previous_test_df is not None:\n answers = eval(current_test[\"prior_group_answers_correct\"].iloc[0])\n responses = eval(current_test[\"prior_group_responses\"].iloc[0])\n previous_test_df['answered_correctly'] = answers\n previous_test_df['user_answer'] = responses\n # Write all your update dict code here \n # your feature extraction and model training code here\n previous_test_df = current_test.copy()\n current_test = current_test[current_test.content_type_id == 0]\n # your prediction code here\n # Write all your Feature Extratcion using Dict Code Here , that is \n # Code which simply fetches user value from dict without updating\n current_test['answered_correctly'] = model.predict(current_test[FEATS])\n set_predict(current_test.loc[:,['row_id', 'answered_correctly']])\n pbar.update(len(current_test))\n[Note : Use Sample Size of 2.5 Million rows in that script only , otherwise you won't get a correct estimate of run time of your script ]
\nI noticed that my Feature generation code for the loop and prediction was showing more than 9 hours of estimated time . Moreover I got some error also related to LightGbm ! Also Note that none of this issues were visible while making Commit using env.predict method provided my RIIID Module , Hence I didn't noticed this. Now it became clear to me why Submission Error was coming in the first place. I will resolve this issues and make sure to make submission .
TLDR: Commiting your Inference Script just on env.predict by RIIID Module does not give you an indication whether your Inference Kernel will succeed on Private Test Set Re Run , Hence Don't rely on it .
\nHence for this competition a Workflow will be .
\n1) Add Features .
\n2) Train Model and Validate its performance.
\n3)[Very Important Step] Use Test Emulator Provided here , to ensure that entire step for Feature Generation and Prediction , lies within 9 hours of Window on 2.5 Million Rows .
\n4) Once you are sure that your Script works fine , then only go for submission.
\n5) If your script is taking too much time to generate features consider dropping less important features .
\n6) Only Submit your inference kernel if your model along with Feature Generation is taking less than 9 hours on emulator .
This competition is more about writing efficient code as much as it is about modelling ! If you face an error be patient debug it step by step rather than simply using brute force.
\nI shared it so that anyone who is stuck on this issue can overcome it quickly and feel less frustruation 😀 , that comes on seeing Submission Scoring Error.
Most of the points causing Submission Scoring Error are discussed here , However I thought of reiterating over those points and remind participants of this topic . As it goes your submission or inference notebook should get run within 9 hours of run time . As pointed out by @sohier here , this runtime is obfuscated with Kernel rerun on test set time + Some Random Additional Time which never push the entire kernel rerun time beyond 9 hrs .
\nI was recently adding some features which were improving my Local Validation score significantly , but I was getting lots of Submission Scoring Error , Now I thought of debugging this issue step by step rather than solely relying on my submissions.
Earlier I adopted this framework of flow.
\n1) Add Features .
\n2) Build Model
\n3) Validate Cv Performance
\n4) Build Inference Kernel !
However As far as this competition is concerned this is the most unreliable and lot of Submission Scoring Error Prone Framework !
Even though I knew about Time Series Emulator Api built by tito , I never used it ! I recently looked at it here , in that we have a code segment as follows
\npbar = tqdm(total=2500000)\nprevious_test_df = None\nfor (current_test, current_prediction_df) in iter_test:\n if previous_test_df is not None:\n answers = eval(current_test[\"prior_group_answers_correct\"].iloc[0])\n responses = eval(current_test[\"prior_group_responses\"].iloc[0])\n previous_test_df['answered_correctly'] = answers\n previous_test_df['user_answer'] = responses\n # Write all your update dict code here \n # your feature extraction and model training code here\n previous_test_df = current_test.copy()\n current_test = current_test[current_test.content_type_id == 0]\n # your prediction code here\n # Write all your Feature Extratcion using Dict Code Here , that is \n # Code which simply fetches user value from dict without updating\n current_test['answered_correctly'] = model.predict(current_test[FEATS])\n set_predict(current_test.loc[:,['row_id', 'answered_correctly']])\n pbar.update(len(current_test))\n[Note : Use Sample Size of 2.5 Million rows in that script only , otherwise you won't get a correct estimate of run time of your script ]
\nI noticed that my Feature generation code for the loop and prediction was showing more than 9 hours of estimated time . Moreover I got some error also related to LightGbm ! Also Note that none of this issues were visible while making Commit using env.predict method provided my RIIID Module , Hence I didn't noticed this. Now it became clear to me why Submission Error was coming in the first place. I will resolve this issues and make sure to make submission .
TLDR: Commiting your Inference Script just on env.predict by RIIID Module does not give you an indication whether your Inference Kernel will succeed on Private Test Set Re Run , Hence Don't rely on it .
\nHence for this competition a Workflow will be .
\n1) Add Features .
\n2) Train Model and Validate its performance.
\n3)[Very Important Step] Use Test Emulator Provided here , to ensure that entire step for Feature Generation and Prediction , lies within 9 hours of Window on 2.5 Million Rows .
\n4) Once you are sure that your Script works fine , then only go for submission.
\n5) If your script is taking too much time to generate features consider dropping less important features .
\n6) Only Submit your inference kernel if your model along with Feature Generation is taking less than 9 hours on emulator .
This competition is more about writing efficient code as much as it is about modelling ! If you face an error be patient debug it step by step rather than simply using brute force.
\nI shared it so that anyone who is stuck on this issue can overcome it quickly and feel less frustruation 😀 , that comes on seeing Submission Scoring Error.