Thank you very much to the team members( @wimwim @kingychiu @tangtunyu). It was my first time working in a team. I learned a lot, so I would like to participate in teams more actively in the future.
\nThe overall solution is explained here: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420235, so I would like to provide additional explanation about the part I worked on.
\nMy source codes: https://github.com/krk-san/3rd-place-solution-Predict-Student-Performance-from-Game-Play
\nIndividual model for each question (18 binary classifiers)
Single Best Model (5folds XGBoost)
\nCV: 0.702, Public LB: 0.700, Private LB: 0.701
These are the features removed because no correlation between cv and public scores.
\nWhen feature selection, I did it by out of fold (per fold×question). Otherwise, it would cause leakage, unfairly boost the CV score, and affect negatively ensemble weights, etc.
\nAdvantages:
\nDisadvantages:
\nIn my case, even though the CV score did not change with feature selection, the public score became worse, so I switched to a policy of no feature selection. However, when I saw the top solutions creating thousands of features and implementing feature selection, I now wish I had created a larger number of features and done feature selection.
\nI created almost perfectly reproduces train.csv from raw log data. Since the format slightly differs between old and recent data, encoding was troublesome, but the following data was obtained in the end.
\nAdditional dataset:
\nI used the above incomplete sessions, and while the CV score increased by about 0.002, there was no positive impact on the private score. As in the excellent solution of 7th place: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420119, adding the maximum level of each session might have provided good results for me.
", "messages": [ { "id": "2323587", "postDate": "06/30/2023 04:04:58", "content": "Thank you very much to the team members( @wimwim @kingychiu @tangtunyu). It was my first time working in a team. I learned a lot, so I would like to participate in teams more actively in the future.
\nThe overall solution is explained here: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420235, so I would like to provide additional explanation about the part I worked on.
\nMy source codes: https://github.com/krk-san/3rd-place-solution-Predict-Student-Performance-from-Game-Play
\nIndividual model for each question (18 binary classifiers)
Single Best Model (5folds XGBoost)
\nCV: 0.702, Public LB: 0.700, Private LB: 0.701
These are the features removed because no correlation between cv and public scores.
\nWhen feature selection, I did it by out of fold (per fold×question). Otherwise, it would cause leakage, unfairly boost the CV score, and affect negatively ensemble weights, etc.
\nAdvantages:
\nDisadvantages:
\nIn my case, even though the CV score did not change with feature selection, the public score became worse, so I switched to a policy of no feature selection. However, when I saw the top solutions creating thousands of features and implementing feature selection, I now wish I had created a larger number of features and done feature selection.
\nI created almost perfectly reproduces train.csv from raw log data. Since the format slightly differs between old and recent data, encoding was troublesome, but the following data was obtained in the end.
\nAdditional dataset:
\nI used the above incomplete sessions, and while the CV score increased by about 0.002, there was no positive impact on the private score. As in the excellent solution of 7th place: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420119, adding the maximum level of each session might have provided good results for me.
", "rawMarkdown": "Thank you very much to the team members( @wimwim @kingychiu @tangtunyu). It was my first time working in a team. I learned a lot, so I would like to participate in teams more actively in the future.\n\nThe overall solution is explained here: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420235, so I would like to provide additional explanation about the part I worked on.\n\nMy source codes: https://github.com/krk-san/3rd-place-solution-Predict-Student-Performance-from-Game-Play\n\n## The main topics are as follows\n\n- Model summary\n- Created features\n- Feature selection by out-of-fold to avoid leakage\n- Generate additional dataset from raw log data\n\n## Model summary\n\n- Individual model for each question (18 binary classifiers)\n- Single Best Model (5folds XGBoost)\n \n CV: 0.702, Public LB: 0.700, Private LB: 0.701\n \n\n## Created features\n\n### Features in our final submissions\n\n- Elapsed time diff between the previous level group end and the current level group start\n - This works because it almost represents the time the user takes to answer questions.\n- Elapsed time and index count between flag events\n - What I call flag events are events that must be passed during game progression. Although some events can occur in any order, most of them have a specific order.\n - These events can be extracted from users who have answered all 18 questions correctly. Some of them are users who take the shortest action as if they already knew the answer and finish the game with a small number of records. In addition, the next action to be taken is appended to the notebook when the flag event is passed, so I actually checked it while playing the game.\n- Prediction probabilities for previous questions\n - All prediction values from the questions before the target question\n - For example, to predict q3, then use q1 and q2 prediction probabilities\n - Sum of the most recent M (M=1,2,…) prediction probabilities\n- Aggregate features\n - elapsed time diff\n - screen_coor_x diff, screen_coor_y diff\n - index per elapsed time\n - Some boolean\n - elapsed time diff < 0\n - elapsed time diff > th\n - index counts and elapsed time for each category as a percentage of the total\n - hover_duration, page, …\n- Datetime\n- Screen aspect (based on max value and min value of screen_coor_x and screen_coor_y)\n\n### Features not in our final submissions\n\nThese are the features removed because no correlation between cv and public scores.\n\n- Some aggregate features\n - elapsed time diff forward (shift: -1)\n - elapsed time max - elapsed time current\n - room_fqid in the current record == room_fqid in the previous record\n- Some prediction probabilities features\n - Sum of percentages relative to the average correct rate\n - Sum of probabilities or percentages binned\n- Meta features\n - Aggregate features for oof prediction probabilities (not previous question)\n - The 2019 Data Science Bowl's 3rd place solution will be helpful.\n - https://www.kaggle.com/c/data-science-bowl-2019/discussion/127388\n- Script type (normal, nohumor, dry, nosnark)\n - This can be predicted from the unique text that appears\n- Sub fqids/room_fqids broken down comma by comma\n\n## Feature selection by out-of-fold to avoid leakage\n\nWhen feature selection, I did it by out of fold (per fold×question). Otherwise, it would cause leakage, unfairly boost the CV score, and affect negatively ensemble weights, etc.\n\nAdvantages:\n\n- Able to evaluate fairly the CV score of a model\n- When ensembles, prevent calculating mistakenly higher weights for feature-selected models\n\nDisadvantages:\n\n- More difficult to manage features\n- Increased computational cost due to the total number of features required\n - For example, when using the top 300 features, the total number of features is as follows:\n - NOT OOF: 300\n - OOF: (fold1 300) | (fold2 300) | ・・・|(fold K 300) ≥ 300\n - Of course, in the most cases, this difference is small.\n\nIn my case, even though the CV score did not change with feature selection, the public score became worse, so I switched to a policy of no feature selection. However, when I saw the top solutions creating thousands of features and implementing feature selection, I now wish I had created a larger number of features and done feature selection.\n\n## Additional dataset\n\nI created almost perfectly reproduces train.csv from raw log data. Since the format slightly differs between old and recent data, encoding was troublesome, but the following data was obtained in the end.\n\nAdditional dataset:\n\n- Complete sessions: 10,000+\n- Incomplete sessions (retire players): 200,000+\n\nI used the above incomplete sessions, and while the CV score increased by about 0.002, there was no positive impact on the private score. As in the excellent solution of 7th place: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420119, adding the maximum level of each session might have provided good results for me.", "votes": null }, { "id": "2323621", "postDate": "06/30/2023 04:53:59", "content": "@yyykrk congrats with 3 place and gold medal! Very interesting solution!
", "rawMarkdown": "yyykrk congrats with 3 place and gold medal! Very interesting solution!", "votes": null }, { "id": "2323648", "postDate": "06/30/2023 05:18:08", "content": "Your experimentation speed brings a lot of value to the team as well!
", "rawMarkdown": "Your experimentation speed brings a lot of value to the team as well!", "votes": null }, { "id": "2323690", "postDate": "06/30/2023 05:55:47", "content": "Nice! Congrats on the win!
", "rawMarkdown": "Nice! Congrats on the win!", "votes": null }, { "id": "2323846", "postDate": "06/30/2023 07:38:15", "content": "Congratulations on coming 3rd @yyykrk 🎉🎉🎉
", "rawMarkdown": "Congratulations on coming 3rd @yyykrk 🎉🎉🎉", "votes": null }, { "id": "2324090", "postDate": "06/30/2023 11:28:44", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "2324099", "postDate": "06/30/2023 11:33:52", "content": "I am very grateful to you because without your approach and models we would not have won the Gold Medal!
", "rawMarkdown": "I am very grateful to you because without your approach and models we would not have won the Gold Medal!", "votes": null }, { "id": "2343922", "postDate": "07/14/2023 05:17:48", "content": "Congratulations. Thanks for sharing the solution. Its a very useful learning resource.
\nDid you try any advanced programming techniques?
Thanks. I haven't tried anything too special.
", "rawMarkdown": "Thanks. I haven't tried anything too special.", "votes": null } ], "comments": [ { "id": 2323621, "author_name": "serangu", "author_url": "", "post_date": "06/30/2023 04:53:59", "content": "@yyykrk congrats with 3 place and gold medal! Very interesting solution!
", "votes": null, "replies": [ { "id": 2324090, "author_name": "yyykrk", "author_url": "", "post_date": "06/30/2023 11:28:44", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 2323648, "author_name": "kingychiu", "author_url": "", "post_date": "06/30/2023 05:18:08", "content": "Your experimentation speed brings a lot of value to the team as well!
", "votes": null, "replies": [ { "id": 2324099, "author_name": "yyykrk", "author_url": "", "post_date": "06/30/2023 11:33:52", "content": "I am very grateful to you because without your approach and models we would not have won the Gold Medal!
", "votes": null, "replies": [] } ] }, { "id": 2323690, "author_name": "orereisrael", "author_url": "", "post_date": "06/30/2023 05:55:47", "content": "Nice! Congrats on the win!
", "votes": null, "replies": [] }, { "id": 2323846, "author_name": "swapnilchowdhury", "author_url": "", "post_date": "06/30/2023 07:38:15", "content": "Congratulations on coming 3rd @yyykrk 🎉🎉🎉
", "votes": null, "replies": [] }, { "id": 2343922, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "07/14/2023 05:17:48", "content": "Congratulations. Thanks for sharing the solution. Its a very useful learning resource.
\nDid you try any advanced programming techniques?
Thanks. I haven't tried anything too special.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2323587": "Thank you very much to the team members( @wimwim @kingychiu @tangtunyu). It was my first time working in a team. I learned a lot, so I would like to participate in teams more actively in the future.\n\nThe overall solution is explained here: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420235, so I would like to provide additional explanation about the part I worked on.\n\nMy source codes: https://github.com/krk-san/3rd-place-solution-Predict-Student-Performance-from-Game-Play\n\n## The main topics are as follows\n\n- Model summary\n- Created features\n- Feature selection by out-of-fold to avoid leakage\n- Generate additional dataset from raw log data\n\n## Model summary\n\n- Individual model for each question (18 binary classifiers)\n- Single Best Model (5folds XGBoost)\n \n CV: 0.702, Public LB: 0.700, Private LB: 0.701\n \n\n## Created features\n\n### Features in our final submissions\n\n- Elapsed time diff between the previous level group end and the current level group start\n - This works because it almost represents the time the user takes to answer questions.\n- Elapsed time and index count between flag events\n - What I call flag events are events that must be passed during game progression. Although some events can occur in any order, most of them have a specific order.\n - These events can be extracted from users who have answered all 18 questions correctly. Some of them are users who take the shortest action as if they already knew the answer and finish the game with a small number of records. In addition, the next action to be taken is appended to the notebook when the flag event is passed, so I actually checked it while playing the game.\n- Prediction probabilities for previous questions\n - All prediction values from the questions before the target question\n - For example, to predict q3, then use q1 and q2 prediction probabilities\n - Sum of the most recent M (M=1,2,…) prediction probabilities\n- Aggregate features\n - elapsed time diff\n - screen_coor_x diff, screen_coor_y diff\n - index per elapsed time\n - Some boolean\n - elapsed time diff < 0\n - elapsed time diff > th\n - index counts and elapsed time for each category as a percentage of the total\n - hover_duration, page, …\n- Datetime\n- Screen aspect (based on max value and min value of screen_coor_x and screen_coor_y)\n\n### Features not in our final submissions\n\nThese are the features removed because no correlation between cv and public scores.\n\n- Some aggregate features\n - elapsed time diff forward (shift: -1)\n - elapsed time max - elapsed time current\n - room_fqid in the current record == room_fqid in the previous record\n- Some prediction probabilities features\n - Sum of percentages relative to the average correct rate\n - Sum of probabilities or percentages binned\n- Meta features\n - Aggregate features for oof prediction probabilities (not previous question)\n - The 2019 Data Science Bowl's 3rd place solution will be helpful.\n - https://www.kaggle.com/c/data-science-bowl-2019/discussion/127388\n- Script type (normal, nohumor, dry, nosnark)\n - This can be predicted from the unique text that appears\n- Sub fqids/room_fqids broken down comma by comma\n\n## Feature selection by out-of-fold to avoid leakage\n\nWhen feature selection, I did it by out of fold (per fold×question). Otherwise, it would cause leakage, unfairly boost the CV score, and affect negatively ensemble weights, etc.\n\nAdvantages:\n\n- Able to evaluate fairly the CV score of a model\n- When ensembles, prevent calculating mistakenly higher weights for feature-selected models\n\nDisadvantages:\n\n- More difficult to manage features\n- Increased computational cost due to the total number of features required\n - For example, when using the top 300 features, the total number of features is as follows:\n - NOT OOF: 300\n - OOF: (fold1 300) | (fold2 300) | ・・・|(fold K 300) ≥ 300\n - Of course, in the most cases, this difference is small.\n\nIn my case, even though the CV score did not change with feature selection, the public score became worse, so I switched to a policy of no feature selection. However, when I saw the top solutions creating thousands of features and implementing feature selection, I now wish I had created a larger number of features and done feature selection.\n\n## Additional dataset\n\nI created almost perfectly reproduces train.csv from raw log data. Since the format slightly differs between old and recent data, encoding was troublesome, but the following data was obtained in the end.\n\nAdditional dataset:\n\n- Complete sessions: 10,000+\n- Incomplete sessions (retire players): 200,000+\n\nI used the above incomplete sessions, and while the CV score increased by about 0.002, there was no positive impact on the private score. As in the excellent solution of 7th place: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420119, adding the maximum level of each session might have provided good results for me.", "2323621": "yyykrk congrats with 3 place and gold medal! Very interesting solution!", "2323648": "Your experimentation speed brings a lot of value to the team as well!", "2323690": "Nice! Congrats on the win!", "2323846": "Congratulations on coming 3rd @yyykrk 🎉🎉🎉", "2324090": "Thank you!", "2324099": "I am very grateful to you because without your approach and models we would not have won the Gold Medal!", "2343922": "Congratulations. Thanks for sharing the solution. Its a very useful learning resource. \nDid you try any advanced programming techniques?", "2344197": "Thanks. I haven't tried anything too special." }, "source": "meta" }