After several verifications, I have come to the conclusion that both data provided by the new API (not just the latter data, as I mentioned here) are shuffled. It was difficult to detect, however, because it had not occurred for the three sample data at the time of the commit of the notebook.
\nThus, if your feature engineering process depends on the original order of the data, you need to sort the session data by the index column beforehand, or else it will not be processed as intended, and will make your LB score worse. This problem does not occur with the old API.
\n@philculliton I think this fact would be very confusing to participants since training data and sample test data always seem to be sorted by index (the first dataframe from API) or question number (the second dataframe from API). Would you consider addressing this issue?
", "messages": [ { "id": 2271582, "postDate": "2023-05-24T03:16:46.197Z", "content": "After several verifications, I have come to the conclusion that both data provided by the new API (not just the latter data, as I mentioned here) are shuffled. It was difficult to detect, however, because it had not occurred for the three sample data at the time of the commit of the notebook.
\nThus, if your feature engineering process depends on the original order of the data, you need to sort the session data by the index column beforehand, or else it will not be processed as intended, and will make your LB score worse. This problem does not occur with the old API.
\n@philculliton I think this fact would be very confusing to participants since training data and sample test data always seem to be sorted by index (the first dataframe from API) or question number (the second dataframe from API). Would you consider addressing this issue?
", "rawMarkdown": "After several verifications, I have come to the conclusion that **both data provided by the new API (not just the latter data, as I mentioned [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898#2266542)) are shuffled**. It was difficult to detect, however, because it had not occurred for the three sample data at the time of the commit of the notebook.\n\nThus, **if your feature engineering process depends on the original order of the data, you need to sort the session data by the index column beforehand, or else it will not be processed as intended, and will make your LB score worse**. This problem does not occur with the old API.\n\n@philculliton I think this fact would be very confusing to participants since training data and sample test data always seem to be sorted by index (the first dataframe from API) or question number (the second dataframe from API). Would you consider addressing this issue?", "votes": 62 }, { "id": 2271603, "postDate": "2023-05-24T03:49:59.900Z", "content": "That means, this is enough right?
\nimport jo_wilder_310\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nfor (test, sample_submission) in iter_test:\n test = test.sort_values(by = 'index')\n ...\nYes. In addition, you have to be mindful of the correspondence with the question number when substituting predictions from the model to the sample_submission DataFrame (not sorted by question number).
", "rawMarkdown": "Yes. In addition, you have to be mindful of the correspondence with the question number when substituting predictions from the model to the sample_submission DataFrame (not sorted by question number).", "votes": 3, "replies": [ { "id": 2271736, "postDate": "2023-05-24T05:56:29.633Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2272507, "postDate": "2023-05-24T14:45:28.020Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2272534, "postDate": "2023-05-24T14:57:57.527Z", "content": "sorting the sample_submission improves the score but is still not at par with the old API
\nfor (test, sample_submission) in (iter_test):\n test = test.sort_values(by = 'index')\n sample_submission['questions'] = sample_submission['session_id'].str.split('_').apply(lambda x: x[1])\n sample_submission = sample_submission.sort_values(by = 'questions')\n sample_submission = sample_submission[['session_id', 'correct']]\n@navinkumarmnk It seems to me that there is something wrong with your feature engineering process (perhaps in the feature_engineer function) rather than being caused by the API.
\n@pjmathematician In my case, with the proper sorting process, I was able to get decent results with the new API. However, there may be other issues that I am not aware of.
", "rawMarkdown": "@navinkumarmnk It seems to me that there is something wrong with your feature engineering process (perhaps in the feature_engineer function) rather than being caused by the API.\n\n@pjmathematician In my case, with the proper sorting process, I was able to get decent results with the new API. However, there may be other issues that I am not aware of." }, { "id": 2274417, "postDate": "2023-05-26T01:51:43.913Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2274861, "postDate": "2023-05-26T09:54:04.153Z", "content": "I have addressed the sort issue for index and question, but still the LB score is still destroyed.
\nI have created a new thread so if anyone is experiencing the same problem, we can share information.
Thanks for sharing, seems like now this is a problem for ALL of us 😔, replying just to make this post resurface.
", "rawMarkdown": "Thanks for sharing, seems like now this is a problem for ALL of us 😔, replying just to make this post resurface.", "votes": 1 }, { "id": 2279380, "postDate": "2023-05-29T10:20:46.670Z", "content": "Thanks a lot! Its help me to solve my problem.
", "rawMarkdown": "Thanks a lot! Its help me to solve my problem.", "votes": 1 }, { "id": 2274299, "postDate": "2023-05-25T19:23:23.513Z", "content": "Thanks a lot! Saved me a bunch of time, wasn't sure what was the issue
", "rawMarkdown": "Thanks a lot! Saved me a bunch of time, wasn't sure what was the issue", "votes": 1 }, { "id": 2272004, "postDate": "2023-05-24T08:34:49.563Z", "content": "Thanks for the detailed info on the problem. Just to confirm, @rsakata is there any other problem with the old API ?
", "rawMarkdown": "Thanks for the detailed info on the problem. Just to confirm, @rsakata is there any other problem with the old API ? ", "votes": 1, "replies": [ { "id": 2272603, "postDate": "2023-05-24T15:50:01.510Z", "content": "Maybe, there is not a problem with the old API.
", "rawMarkdown": "Maybe, there is not a problem with the old API.", "votes": 1 } ] }, { "id": 2277412, "postDate": "2023-05-27T17:42:20.873Z", "content": "Always impressed by the curiosity needed to find such things, thx for your contribution
", "rawMarkdown": "Always impressed by the curiosity needed to find such things, thx for your contribution", "votes": 2 }, { "id": 2284238, "postDate": "2023-06-01T20:37:39.210Z", "content": "available run local :
\njo_wilder_310.py
\n kaggle datasets download -d liudacheldieva/competition-game-jo-wilder-py
Example of run in:
\n https://www.kaggle.com/code/liudacheldieva/40ipynb-sucsess
\nDescription:
\n local run:
\nimport jo_wilder_310 as jo_wilder
\niter_test =jo_wilder.make_env_iter_test()
\nenv = iter_test
kaggle run:
\nimport jo_wilder_310 as jo_wilder
\nenv = jo_wilder.make_env()
\niter_test = env.iter_test()
Isnt feature engineering automatic with algorithms like py caret
", "rawMarkdown": "Isnt feature engineering automatic with algorithms like py caret" }, { "id": 2272870, "postDate": "2023-05-24T20:08:07.600Z", "content": "Thanks for sharing @rsakata 🤘
", "rawMarkdown": "Thanks for sharing @rsakata 🤘", "votes": 3 } ], "comments": [ { "id": 2271603, "author_name": "pjmathematician", "author_url": "", "post_date": "2023-05-24T03:49:59.900000", "content": "That means, this is enough right?
\nimport jo_wilder_310\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nfor (test, sample_submission) in iter_test:\n test = test.sort_values(by = 'index')\n ...\nYes. In addition, you have to be mindful of the correspondence with the question number when substituting predictions from the model to the sample_submission DataFrame (not sorted by question number).
", "votes": 3, "replies": [ { "id": 2271736, "author_name": "", "author_url": "", "post_date": "2023-05-24T05:56:29.633000", "content": "", "votes": 0, "replies": [] }, { "id": 2272507, "author_name": "", "author_url": "", "post_date": "2023-05-24T14:45:28.020000", "content": "", "votes": 0, "replies": [] }, { "id": 2272534, "author_name": "pjmathematician", "author_url": "", "post_date": "2023-05-24T14:57:57.527000", "content": "sorting the sample_submission improves the score but is still not at par with the old API
\nfor (test, sample_submission) in (iter_test):\n test = test.sort_values(by = 'index')\n sample_submission['questions'] = sample_submission['session_id'].str.split('_').apply(lambda x: x[1])\n sample_submission = sample_submission.sort_values(by = 'questions')\n sample_submission = sample_submission[['session_id', 'correct']]\n@navinkumarmnk It seems to me that there is something wrong with your feature engineering process (perhaps in the feature_engineer function) rather than being caused by the API.
\n@pjmathematician In my case, with the proper sorting process, I was able to get decent results with the new API. However, there may be other issues that I am not aware of.
", "votes": 0, "replies": [] }, { "id": 2274417, "author_name": "", "author_url": "", "post_date": "2023-05-26T01:51:43.913000", "content": "", "votes": 0, "replies": [] }, { "id": 2274861, "author_name": "tonic", "author_url": "", "post_date": "2023-05-26T09:54:04.153000", "content": "I have addressed the sort issue for index and question, but still the LB score is still destroyed.
\nI have created a new thread so if anyone is experiencing the same problem, we can share information.
Thanks for sharing, seems like now this is a problem for ALL of us 😔, replying just to make this post resurface.
", "votes": 1, "replies": [] }, { "id": 2279380, "author_name": "cruelkanade", "author_url": "", "post_date": "2023-05-29T10:20:46.670000", "content": "Thanks a lot! Its help me to solve my problem.
", "votes": 1, "replies": [] }, { "id": 2274299, "author_name": "Informhunter", "author_url": "", "post_date": "2023-05-25T19:23:23.513000", "content": "Thanks a lot! Saved me a bunch of time, wasn't sure what was the issue
", "votes": 1, "replies": [] }, { "id": 2272004, "author_name": "NikhilMishra", "author_url": "", "post_date": "2023-05-24T08:34:49.563000", "content": "Thanks for the detailed info on the problem. Just to confirm, @rsakata is there any other problem with the old API ?
", "votes": 1, "replies": [ { "id": 2272603, "author_name": "Jack (Japan)", "author_url": "", "post_date": "2023-05-24T15:50:01.510000", "content": "Maybe, there is not a problem with the old API.
", "votes": 1, "replies": [] } ] }, { "id": 2277412, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-05-27T17:42:20.873000", "content": "Always impressed by the curiosity needed to find such things, thx for your contribution
", "votes": 2, "replies": [] }, { "id": 2284238, "author_name": "", "author_url": "", "post_date": "2023-06-01T20:37:39.210000", "content": "available run local :
\njo_wilder_310.py
\n kaggle datasets download -d liudacheldieva/competition-game-jo-wilder-py
Example of run in:
\n https://www.kaggle.com/code/liudacheldieva/40ipynb-sucsess
\nDescription:
\n local run:
\nimport jo_wilder_310 as jo_wilder
\niter_test =jo_wilder.make_env_iter_test()
\nenv = iter_test
kaggle run:
\nimport jo_wilder_310 as jo_wilder
\nenv = jo_wilder.make_env()
\niter_test = env.iter_test()
Isnt feature engineering automatic with algorithms like py caret
", "votes": 0, "replies": [] }, { "id": 2272870, "author_name": "Gaurav Srivastava", "author_url": "", "post_date": "2023-05-24T20:08:07.600000", "content": "Thanks for sharing @rsakata 🤘
", "votes": 3, "replies": [] } ], "raw_markdown_by_id": { "2271582": "After several verifications, I have come to the conclusion that **both data provided by the new API (not just the latter data, as I mentioned [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898#2266542)) are shuffled**. It was difficult to detect, however, because it had not occurred for the three sample data at the time of the commit of the notebook.\n\nThus, **if your feature engineering process depends on the original order of the data, you need to sort the session data by the index column beforehand, or else it will not be processed as intended, and will make your LB score worse**. This problem does not occur with the old API.\n\n@philculliton I think this fact would be very confusing to participants since training data and sample test data always seem to be sorted by index (the first dataframe from API) or question number (the second dataframe from API). Would you consider addressing this issue?", "2271603": "That means, this is enough right?\n```\nimport jo_wilder_310\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nfor (test, sample_submission) in iter_test:\n test = test.sort_values(by = 'index')\n ...\n```", "2282967": "Thanks for sharing, seems like now this is a problem for ALL of us 😔, replying just to make this post resurface.", "2279380": "Thanks a lot! Its help me to solve my problem.", "2274299": "Thanks a lot! Saved me a bunch of time, wasn't sure what was the issue", "2272004": "Thanks for the detailed info on the problem. Just to confirm, @rsakata is there any other problem with the old API ? ", "2277412": "Always impressed by the curiosity needed to find such things, thx for your contribution", "2284238": "available run local :\n**jo_wilder_310.py** \n kaggle datasets download -d liudacheldieva/competition-game-jo-wilder-py \n\nExample of run in:\n https://www.kaggle.com/code/liudacheldieva/40ipynb-sucsess\nDescription:\n **local run:**\nimport jo_wilder_310 as jo_wilder\niter_test =jo_wilder.make_env_iter_test()\nenv = iter_test\n\n **kaggle run:**\nimport jo_wilder_310 as jo_wilder\nenv = jo_wilder.make_env()\niter_test = env.iter_test()", "2274527": "Isnt feature engineering automatic with algorithms like py caret", "2272870": "Thanks for sharing @rsakata 🤘" } }