The Official API demo Notebook says, in bold text:
\n\n\nThe lecture rows in
\ntest_dfshould not be submitted.
So my first attempt was to generate a prediction for each question row in the dataset, then try to assign that to the answered_correctly column in sample_prediction_df. But the submissions kept failing, until I tried just directly returning sample_prediction_df.
However, when I started testing my submissions on @its7171 's excellent iter_test Emulator notebook (Thank you for making that!!!), it started failing precisely because I was filtering out lecture rows, and thus the row length was not matching the submission df, which always seems to have the same number of rows as the test input in that dataset.
\nMy best guess is that must be right, since lots of people have used that notebook to validate their submissions? But on the other hand, how can the official example guide explicitly say the opposite of the correct thing?
\nSo I changed all my logic around to match that, and now the emulator is running fine. But the submission is still failing very early. (and now I'm out of submissions for the day… if the submisssion process is so trial-and-error, I'd at least expect more than 5 attempts per day??)
\nNow I've \"fixed\" that, but the submissions are still failing. The only submission that has succeeded was the dummy one where I just predicted 0.5 for everything, to make sure the problem was inside the loop.
\nSo, can anyone clarify for me - what exactly are we supposed to do with lecture rows on submission?
", "messages": [ { "id": "1135519", "postDate": "01/02/2021 10:08:01", "content": "The Official API demo Notebook says, in bold text:
\n\n\nThe lecture rows in
\ntest_dfshould not be submitted.
So my first attempt was to generate a prediction for each question row in the dataset, then try to assign that to the answered_correctly column in sample_prediction_df. But the submissions kept failing, until I tried just directly returning sample_prediction_df.
However, when I started testing my submissions on @its7171 's excellent iter_test Emulator notebook (Thank you for making that!!!), it started failing precisely because I was filtering out lecture rows, and thus the row length was not matching the submission df, which always seems to have the same number of rows as the test input in that dataset.
\nMy best guess is that must be right, since lots of people have used that notebook to validate their submissions? But on the other hand, how can the official example guide explicitly say the opposite of the correct thing?
\nSo I changed all my logic around to match that, and now the emulator is running fine. But the submission is still failing very early. (and now I'm out of submissions for the day… if the submisssion process is so trial-and-error, I'd at least expect more than 5 attempts per day??)
\nNow I've \"fixed\" that, but the submissions are still failing. The only submission that has succeeded was the dummy one where I just predicted 0.5 for everything, to make sure the problem was inside the loop.
\nSo, can anyone clarify for me - what exactly are we supposed to do with lecture rows on submission?
", "rawMarkdown": "The [Official API demo Notebook](https://www.kaggle.com/sohier/competition-api-detailed-introduction) says, in bold text:\n> **The lecture rows in `test_df` should not be submitted.**\n\nSo my first attempt was to generate a prediction for each **question** row in the dataset, then try to assign that to the `answered_correctly` column in `sample_prediction_df`. But the submissions kept failing, until I tried just directly returning `sample_prediction_df`.\n\nHowever, when I started testing my submissions on @its7171 's excellent [iter_test Emulator notebook](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator) (Thank you for making that!!!), it started failing precisely because I was filtering out lecture rows, and thus the row length was not matching the submission df, which always seems to have **the same number of rows as the test input** in that dataset.\n\nMy best guess is that **must** be right, since lots of people have used that notebook to validate their submissions? But on the other hand, how can the official example guide explicitly say the opposite of the correct thing?\n\nSo I changed all my logic around to match that, and now the emulator is running fine. But the submission is still failing very early. (and now I'm out of submissions for the day... if the submisssion process is so trial-and-error, I'd at least expect more than 5 attempts per day??)\n\nNow I've \"fixed\" that, but the submissions are still failing. The only submission that has succeeded was the dummy one where I just predicted 0.5 for everything, to make sure the problem was **inside** the loop.\n\nSo, can anyone clarify for me - **what exactly are we supposed to do with lecture rows on submission**?", "votes": null }, { "id": "1135523", "postDate": "01/02/2021 10:12:42", "content": "When you make preds, you didn't need them but when you update your stats, you need them just for once because the previous answers do have -1 in it as well. So once you have captured the target for all the rows into a column using \"eval\", then you filter to get non-lec rows and you can use that filtered df to update your stats etc depending on your requirements.
\nfor (current_test_df, current_prediction_df) in iter_test:\n prev_target_df = eval(current_test_df[\"prior_group_answers_correct\"].iloc[0]) # Extracting previous batch's targets\n\n if prev_test_df is not None:\n update_stats(prev_test_df, prev_target_df)\n\n prev_test_df = current_test_df.copy(deep = True) \n current_test_df = update_test_df(current_test_df)\n\n current_test_df[target_col] = model.predict(current_test_df[feat_col])\n env.predict(current_test_df.loc[current_test_df['content_type_id'] == 0, ['row_id', target_col]])\nOhhh wow, thank you!
", "rawMarkdown": "Ohhh wow, thank you!", "votes": null }, { "id": "1135544", "postDate": "01/02/2021 10:34:42", "content": "Drop the lectures from a dataframe which you are submitting to env.predict() - this function only expects predictions for questions, no lectures should go here.
\nHowever, keep in mind that \"prior_group_answers_correct\" field in the first row in each (but first) input dataframe within the main loop will contain answers for both questions and lectures from the previous batch. For lectures the \"answers\" are equal to \"-1\", which should be interpreted as NA.
", "rawMarkdown": "**Drop the lectures** from a dataframe which you are submitting to env.predict() - this function only expects predictions for questions, no lectures should go here.\n\nHowever, keep in mind that \"prior_group_answers_correct\" field in the first row in each (but first) input dataframe within the main loop will contain answers for **both** questions and lectures from the previous batch. For lectures the \"answers\" are equal to \"-1\", which should be interpreted as NA.", "votes": null }, { "id": "1135581", "postDate": "01/02/2021 11:01:53", "content": "Thanks!
\nIndeed, I'm pretty sure that was my problem: I was trying to apply my predictions in both places (the prior group data and the thing I was submitting to env.predict), but they are different shapes when there are lectures involved. So whichever way I \"fixed\" it, one of them would be wrong.
\nOf course, there's not really a good reason for me to try to compare my predictions to prior_group_answers_correct - I'm not updating any values based on how well I'm doing so far. It was just so I can get an idea of my AUC in the (tiny) interactive mode dataset. So that was an entirely silly sequence of failures.
", "rawMarkdown": "Thanks!\n\nIndeed, I'm pretty sure that was my problem: I was trying to apply my predictions in both places (the prior group data and the thing I was submitting to env.predict), but they are different shapes when there are lectures involved. So whichever way I \"fixed\" it, one of them would be wrong.\n\nOf course, there's not really a good reason for me to try to compare my predictions to prior_group_answers_correct - I'm not updating any values based on how well I'm doing so far. It was just so I can get an idea of my AUC in the (tiny) interactive mode dataset. So that was an entirely silly sequence of failures.", "votes": null } ], "comments": [ { "id": 1135523, "author_name": "adityaecdrid", "author_url": "", "post_date": "01/02/2021 10:12:42", "content": "When you make preds, you didn't need them but when you update your stats, you need them just for once because the previous answers do have -1 in it as well. So once you have captured the target for all the rows into a column using \"eval\", then you filter to get non-lec rows and you can use that filtered df to update your stats etc depending on your requirements.
\nfor (current_test_df, current_prediction_df) in iter_test:\n prev_target_df = eval(current_test_df[\"prior_group_answers_correct\"].iloc[0]) # Extracting previous batch's targets\n\n if prev_test_df is not None:\n update_stats(prev_test_df, prev_target_df)\n\n prev_test_df = current_test_df.copy(deep = True) \n current_test_df = update_test_df(current_test_df)\n\n current_test_df[target_col] = model.predict(current_test_df[feat_col])\n env.predict(current_test_df.loc[current_test_df['content_type_id'] == 0, ['row_id', target_col]])\nOhhh wow, thank you!
", "votes": null, "replies": [] } ] }, { "id": 1135544, "author_name": "vlpavlov", "author_url": "", "post_date": "01/02/2021 10:34:42", "content": "Drop the lectures from a dataframe which you are submitting to env.predict() - this function only expects predictions for questions, no lectures should go here.
\nHowever, keep in mind that \"prior_group_answers_correct\" field in the first row in each (but first) input dataframe within the main loop will contain answers for both questions and lectures from the previous batch. For lectures the \"answers\" are equal to \"-1\", which should be interpreted as NA.
", "votes": null, "replies": [ { "id": 1135581, "author_name": "yanamal", "author_url": "", "post_date": "01/02/2021 11:01:53", "content": "Thanks!
\nIndeed, I'm pretty sure that was my problem: I was trying to apply my predictions in both places (the prior group data and the thing I was submitting to env.predict), but they are different shapes when there are lectures involved. So whichever way I \"fixed\" it, one of them would be wrong.
\nOf course, there's not really a good reason for me to try to compare my predictions to prior_group_answers_correct - I'm not updating any values based on how well I'm doing so far. It was just so I can get an idea of my AUC in the (tiny) interactive mode dataset. So that was an entirely silly sequence of failures.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1135519": "The [Official API demo Notebook](https://www.kaggle.com/sohier/competition-api-detailed-introduction) says, in bold text:\n> **The lecture rows in `test_df` should not be submitted.**\n\nSo my first attempt was to generate a prediction for each **question** row in the dataset, then try to assign that to the `answered_correctly` column in `sample_prediction_df`. But the submissions kept failing, until I tried just directly returning `sample_prediction_df`.\n\nHowever, when I started testing my submissions on @its7171 's excellent [iter_test Emulator notebook](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator) (Thank you for making that!!!), it started failing precisely because I was filtering out lecture rows, and thus the row length was not matching the submission df, which always seems to have **the same number of rows as the test input** in that dataset.\n\nMy best guess is that **must** be right, since lots of people have used that notebook to validate their submissions? But on the other hand, how can the official example guide explicitly say the opposite of the correct thing?\n\nSo I changed all my logic around to match that, and now the emulator is running fine. But the submission is still failing very early. (and now I'm out of submissions for the day... if the submisssion process is so trial-and-error, I'd at least expect more than 5 attempts per day??)\n\nNow I've \"fixed\" that, but the submissions are still failing. The only submission that has succeeded was the dummy one where I just predicted 0.5 for everything, to make sure the problem was **inside** the loop.\n\nSo, can anyone clarify for me - **what exactly are we supposed to do with lecture rows on submission**?", "1135523": "When you make preds, you didn't need them but when you update your stats, you need them just for once because the previous answers do have -1 in it as well. So once you have captured the target for all the rows into a column using \"eval\", then you filter to get non-lec rows and you can use that filtered df to update your stats etc depending on your requirements.\n\n\n```\nfor (current_test_df, current_prediction_df) in iter_test:\n prev_target_df = eval(current_test_df[\"prior_group_answers_correct\"].iloc[0]) # Extracting previous batch's targets\n \n if prev_test_df is not None:\n update_stats(prev_test_df, prev_target_df)\n \n prev_test_df = current_test_df.copy(deep = True) \n current_test_df = update_test_df(current_test_df)\n \n current_test_df[target_col] = model.predict(current_test_df[feat_col])\n env.predict(current_test_df.loc[current_test_df['content_type_id'] == 0, ['row_id', target_col]])\n```", "1135530": "Ohhh wow, thank you!", "1135544": "**Drop the lectures** from a dataframe which you are submitting to env.predict() - this function only expects predictions for questions, no lectures should go here.\n\nHowever, keep in mind that \"prior_group_answers_correct\" field in the first row in each (but first) input dataframe within the main loop will contain answers for **both** questions and lectures from the previous batch. For lectures the \"answers\" are equal to \"-1\", which should be interpreted as NA.", "1135581": "Thanks!\n\nIndeed, I'm pretty sure that was my problem: I was trying to apply my predictions in both places (the prior group data and the thing I was submitting to env.predict), but they are different shapes when there are lectures involved. So whichever way I \"fixed\" it, one of them would be wrong.\n\nOf course, there's not really a good reason for me to try to compare my predictions to prior_group_answers_correct - I'm not updating any values based on how well I'm doing so far. It was just so I can get an idea of my AUC in the (tiny) interactive mode dataset. So that was an entirely silly sequence of failures." }, "source": "meta" }