![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5005351%2F196a1b67b9b8bdf56ba47c91097e8d0d%2FPicture1.png?generation=1688196490904650&alt=media\")
The competition was really exciting and it gave us a chance to practice feature engineering. I'm very thankful for the support and help from my team @shinomoriaoshi @hoangnguyen719 and @martasprg. They were always there for me and together we made a big difference.
\nI would like to thank the hosts, and special thanks to @cpmpml and @pdnartreb for identifying the issue of data leak, which made the competition right back on track.
\nSpecial thanks to @cdeotte for his great starter notebooks and insights that helped me in the early phase of the competition.
\nHere's an overview of what each of us worked on:
\n· My main focus was on improving the XGBoost model and handling feature engineering.
\n· Minh Tri Phan worked on a Transformer model with a CV (cross-validation) score of 0.699 and a public leaderboard (LB) score of 0.7.
\n· Hoang processed the external data.
\n· Martin worked on selecting the most relevant features.
In our final submissions, we ensembled the XGBoost and Transformer models, which helped us achieve the gold position. Our ensemble submission had a public LB score of 0.705 and a private LB score of approximately 0.7025. Additionally, we had two other submissions with single XGBoost models, where one had a public LB score of 0.705 and a private LB score of 0.700.
\nCode: The code is a bit uncleaned, apologies for that. For any queries, contact me on LinkedIn
\nFE code: https://www.kaggle.com/code/chaudharypriyanshu/mb-fb5-train-xgb-25-11-external-data/notebook
\nInference code: https://www.kaggle.com/code/chaudharypriyanshu/inference-xgb-25-11-17/notebook
\nTraining code: https://www.kaggle.com/code/chaudharypriyanshu/mb-fb5-train-xgb-25-9-training/notebook
I created a 5-fold XGBoost model for each question (a total of 90 models). I used Kaggle kernels only to train XGBoost since it took only 45 mins on Kaggle’s P100 GPU to train all 90 models.
\nThe single XGBoost model achieved a Public leaderboard (LB) score of 0.705 and took 45-50 mins for inference, but it didn't perform as well on the private LB. When we included Hoang's external data, the model's score improved to 0.704 on the private LB. However, we decided not to use it because the public LB score was unusually low at 0.702.
Session length: simply accounts for the total length of the session per level group.
Instance features: I created Object click-based features (first object click, room coordinates of that click, I called them Instance features)that were most important and gave an improvement of 0.0007, when I added them with standard features. I created a total of 36 features since there were 12 instances where object clicks were present.
Magic bingo features: Inspired from the public notebooks. I created more such features for all 3 level groups and it improved the CV by 0.0003.
Standard features:
\na) Count features: I created count features based on Fqid, text_Fqid, room_fqid, level, and event_comb. These features capture the frequency of specific events or combinations.
b) Binning of indexes: I performed binning on indexes with bin sizes of approximately 30 or 50 in sorted order. Raw indexes worked better on the private LB, while binned features yielded better results on the public LB.
\nc) First and Sum features: I generated first and sum of elapsed_time_diff for all categorical columns. I found that min, max, and std did not work well in my case.
\nd) Aggregations based on hover duration.
Top Level Group Features: Used top 15-25 features (according to feature importance), Duration and instance features across different level groups.
Meta features: Using past questions predictions to predict the current question. i.e. for questiont I used all predictions for questions (1 to t-1). Using them gave an improvement of around 0.001.
We made improvements to retain the original order of the sequence during inference.
We found that there are approx. 250 sessions with abnormal indexing(interestingly all of them are from the 5th and 6th December 2020)
Created a function to preserve the original sequence for 99.5% of sequences, with only a small portion (0.5%) having misplaced events not more than 4-5 positions of the actual index.
Reindexed these abnormal sessions which improved or scored on LB slightly.
The below tables list our experiments with the best results.
\n| External Data Used | \nCV | \nPublic LB | \nPrivate LB | \nfinal Sub | \n
|---|---|---|---|---|
| No | \n0.6996 | \n0.701 | \n0.698 | \nNo | \n
| No | \n0.7001 | \n0.702 | \n0.700 | \nNo | \n
| No | \n0.6996 | \n0.701 | \n0.698 | \nNo | \n
| Yes((Public ED) | \n0.7015 | \n0.705 | \n0.700 | \nYes | \n
| Yes(Hoang's ED ) | \n0.7019 | \n0.702 | \n0.704 | \nNo | \n
| Yes (Hoang's ED) | \n0.7022 | \n0.703 | \n0.703 | \nNo | \n
The model is shown in the following figure:
\n
Particularly, it consists of 2 parts,
\n(i) Training a neural network, then extracting the embedding.
\n(ii) Concatenating the embedding from the neural network to a set of aggregated features, then training a gradient boosting model (XGBoost, CatBoost, LightGBM).
I was inspired by the RIIID competition and @letranduckinh’s solution, in which he customized the multi-head attention mechanism to adopt the time gap between 2 actions. In my opinion, if we have to relate the problem to an NLP problem, RIIID competition is like a token classification task (e.g., NER), meanwhile, this competition is like a document classification task. Therefore, I decided to use a transformer and some other recurrent network types.
\nI used the encoder-only structure as I didn’t see any motivation to have the decoder.
\nHowever, the transformer encoder alone didn’t work so well, so I decided to add some more (3) GRU layers in front of the encoder. The detailed architecture (Pytorch code) is given here (https://github.com/minhtriphan/Kaggle-competition---Predicting-Student-Performance/blob/main/Transformer/model.py).
NUM_COLS = [, , , , , , ]\nTXT_COLS = [, , , , , , ]\nOverall, the NN gets 0.695/0.700 in CV and public LB (before the API crisis, after that I never check how the NN works in the public LB anymore as it was combined always with XGBoost)
\nHowever, the NN in my case was not super satisfactory. I then decided to extract the embedding from the trained NN, concatenate them into a set of aggregated features, then use XGBoost to train the model. This helped me to get a huge boost in both CV and LB.
\nOverall, the scores of this approach are shown below,
\n| External Data Used | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| No | \n0.6993 | \n0.702 | \n0.697 | \n
| Yes | \n0.6989 | \n0.701 | \n0.699 | \n
Unfortunately, as I didn’t observe any gain in CV and public LB with external data, I decided not to choose that model to add to our model pool.
\nLinks:
\nTraining code: https://github.com/minhtriphan/Kaggle-competition---Predicting-Student-Performance----part-of--8th-solution.git
Inference code:
\nWithout external data: https://www.kaggle.com/code/shinomoriaoshi/psp-v7b-infer
\nWith external data: https://www.kaggle.com/code/shinomoriaoshi/psp-v9a-infer
Hoang has described his work in a separate thread that describes the preprocessing of external data, experimental results and why to trust CV over LB.
\nLink to Hoang's Part: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420315
The competition was really exciting and it gave us a chance to practice feature engineering. I'm very thankful for the support and help from my team @shinomoriaoshi @hoangnguyen719 and @martasprg. They were always there for me and together we made a big difference.
\nI would like to thank the hosts, and special thanks to @cpmpml and @pdnartreb for identifying the issue of data leak, which made the competition right back on track.
\nSpecial thanks to @cdeotte for his great starter notebooks and insights that helped me in the early phase of the competition.
\nHere's an overview of what each of us worked on:
\n· My main focus was on improving the XGBoost model and handling feature engineering.
\n· Minh Tri Phan worked on a Transformer model with a CV (cross-validation) score of 0.699 and a public leaderboard (LB) score of 0.7.
\n· Hoang processed the external data.
\n· Martin worked on selecting the most relevant features.
In our final submissions, we ensembled the XGBoost and Transformer models, which helped us achieve the gold position. Our ensemble submission had a public LB score of 0.705 and a private LB score of approximately 0.7025. Additionally, we had two other submissions with single XGBoost models, where one had a public LB score of 0.705 and a private LB score of 0.700.
\nCode: The code is a bit uncleaned, apologies for that. For any queries, contact me on LinkedIn
\nFE code: https://www.kaggle.com/code/chaudharypriyanshu/mb-fb5-train-xgb-25-11-external-data/notebook
\nInference code: https://www.kaggle.com/code/chaudharypriyanshu/inference-xgb-25-11-17/notebook
\nTraining code: https://www.kaggle.com/code/chaudharypriyanshu/mb-fb5-train-xgb-25-9-training/notebook
I created a 5-fold XGBoost model for each question (a total of 90 models). I used Kaggle kernels only to train XGBoost since it took only 45 mins on Kaggle’s P100 GPU to train all 90 models.
\nThe single XGBoost model achieved a Public leaderboard (LB) score of 0.705 and took 45-50 mins for inference, but it didn't perform as well on the private LB. When we included Hoang's external data, the model's score improved to 0.704 on the private LB. However, we decided not to use it because the public LB score was unusually low at 0.702.
Session length: simply accounts for the total length of the session per level group.
Instance features: I created Object click-based features (first object click, room coordinates of that click, I called them Instance features)that were most important and gave an improvement of 0.0007, when I added them with standard features. I created a total of 36 features since there were 12 instances where object clicks were present.
Magic bingo features: Inspired from the public notebooks. I created more such features for all 3 level groups and it improved the CV by 0.0003.
Standard features:
\na) Count features: I created count features based on Fqid, text_Fqid, room_fqid, level, and event_comb. These features capture the frequency of specific events or combinations.
b) Binning of indexes: I performed binning on indexes with bin sizes of approximately 30 or 50 in sorted order. Raw indexes worked better on the private LB, while binned features yielded better results on the public LB.
\nc) First and Sum features: I generated first and sum of elapsed_time_diff for all categorical columns. I found that min, max, and std did not work well in my case.
\nd) Aggregations based on hover duration.
Top Level Group Features: Used top 15-25 features (according to feature importance), Duration and instance features across different level groups.
Meta features: Using past questions predictions to predict the current question. i.e. for questiont I used all predictions for questions (1 to t-1). Using them gave an improvement of around 0.001.
We made improvements to retain the original order of the sequence during inference.
We found that there are approx. 250 sessions with abnormal indexing(interestingly all of them are from the 5th and 6th December 2020)
Created a function to preserve the original sequence for 99.5% of sequences, with only a small portion (0.5%) having misplaced events not more than 4-5 positions of the actual index.
Reindexed these abnormal sessions which improved or scored on LB slightly.
The below tables list our experiments with the best results.
\n| External Data Used | \nCV | \nPublic LB | \nPrivate LB | \nfinal Sub | \n
|---|---|---|---|---|
| No | \n0.6996 | \n0.701 | \n0.698 | \nNo | \n
| No | \n0.7001 | \n0.702 | \n0.700 | \nNo | \n
| No | \n0.6996 | \n0.701 | \n0.698 | \nNo | \n
| Yes((Public ED) | \n0.7015 | \n0.705 | \n0.700 | \nYes | \n
| Yes(Hoang's ED ) | \n0.7019 | \n0.702 | \n0.704 | \nNo | \n
| Yes (Hoang's ED) | \n0.7022 | \n0.703 | \n0.703 | \nNo | \n
The model is shown in the following figure:
\n
Particularly, it consists of 2 parts,
\n(i) Training a neural network, then extracting the embedding.
\n(ii) Concatenating the embedding from the neural network to a set of aggregated features, then training a gradient boosting model (XGBoost, CatBoost, LightGBM).
I was inspired by the RIIID competition and @letranduckinh’s solution, in which he customized the multi-head attention mechanism to adopt the time gap between 2 actions. In my opinion, if we have to relate the problem to an NLP problem, RIIID competition is like a token classification task (e.g., NER), meanwhile, this competition is like a document classification task. Therefore, I decided to use a transformer and some other recurrent network types.
\nI used the encoder-only structure as I didn’t see any motivation to have the decoder.
\nHowever, the transformer encoder alone didn’t work so well, so I decided to add some more (3) GRU layers in front of the encoder. The detailed architecture (Pytorch code) is given here (https://github.com/minhtriphan/Kaggle-competition---Predicting-Student-Performance/blob/main/Transformer/model.py).
NUM_COLS = [, , , , , , ]\nTXT_COLS = [, , , , , , ]\nOverall, the NN gets 0.695/0.700 in CV and public LB (before the API crisis, after that I never check how the NN works in the public LB anymore as it was combined always with XGBoost)
\nHowever, the NN in my case was not super satisfactory. I then decided to extract the embedding from the trained NN, concatenate them into a set of aggregated features, then use XGBoost to train the model. This helped me to get a huge boost in both CV and LB.
\nOverall, the scores of this approach are shown below,
\n| External Data Used | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| No | \n0.6993 | \n0.702 | \n0.697 | \n
| Yes | \n0.6989 | \n0.701 | \n0.699 | \n
Unfortunately, as I didn’t observe any gain in CV and public LB with external data, I decided not to choose that model to add to our model pool.
\nLinks:
\nTraining code: https://github.com/minhtriphan/Kaggle-competition---Predicting-Student-Performance----part-of--8th-solution.git
Inference code:
\nWithout external data: https://www.kaggle.com/code/shinomoriaoshi/psp-v7b-infer
\nWith external data: https://www.kaggle.com/code/shinomoriaoshi/psp-v9a-infer
Hoang has described his work in a separate thread that describes the preprocessing of external data, experimental results and why to trust CV over LB.
\nLink to Hoang's Part: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420315
Thanks a lot for your work on this guys! So grateful I had a chance to collaborate and learn from you all.
", "rawMarkdown": "Thanks a lot for your work on this guys! So grateful I had a chance to collaborate and learn from you all.", "votes": null }, { "id": "2327796", "postDate": "07/03/2023 07:20:22", "content": "Congratulations!
\nThank you for sharing your solution so quickly.
I am currently reviewing and looking back at the Discussions of the gold medalists in this competition.
\nI am not sure I understand the Magic bingo features that improved the CV by 0.0003.
\nI know you are busy and don't have much time, but I would appreciate it if you could tell me about the Magic bingo features.
Hey Ino,
\nIn the dataset, there are fqids, that have the word bingo in them. They signify that a user was asked to find something on the screen and press it, when the user presses the correct item bingo fqid pops up. Therefore these features indicate the time and number of clicks it took for the player to get the correct item so that bingo-based fqid appeared.
\nI'd advise going through one session and observing the fqid for level group 5-12, you will see some fqids that have word bingo in them.
Thank you for your quick and detailed response.
\nI understood that the word \"bingo\" was in the fqid, but did not know it would pop up when I pressed the correct item.
\nAs you said, I understand that features based on \"bingo\" words can help improve scores.
\nThank you so much for your help.
Congrats; nice correlation, Meta features on priors are the common theme , think that was the killer other than other smart features. And off course not being tempted by the lb 😀
\nBut nice nn approach as well by your team
", "rawMarkdown": "Congrats; nice correlation, Meta features on priors are the common theme , think that was the killer other than other smart features. And off course not being tempted by the lb 😀\n\nBut nice nn approach as well by your team", "votes": null } ], "comments": [ { "id": 2325433, "author_name": "hoangnguyen719", "author_url": "", "post_date": "07/01/2023 10:40:55", "content": "Thanks a lot for your work on this guys! So grateful I had a chance to collaborate and learn from you all.
", "votes": null, "replies": [] }, { "id": 2327796, "author_name": "inoway", "author_url": "", "post_date": "07/03/2023 07:20:22", "content": "Congratulations!
\nThank you for sharing your solution so quickly.
I am currently reviewing and looking back at the Discussions of the gold medalists in this competition.
\nI am not sure I understand the Magic bingo features that improved the CV by 0.0003.
\nI know you are busy and don't have much time, but I would appreciate it if you could tell me about the Magic bingo features.
Hey Ino,
\nIn the dataset, there are fqids, that have the word bingo in them. They signify that a user was asked to find something on the screen and press it, when the user presses the correct item bingo fqid pops up. Therefore these features indicate the time and number of clicks it took for the player to get the correct item so that bingo-based fqid appeared.
\nI'd advise going through one session and observing the fqid for level group 5-12, you will see some fqids that have word bingo in them.
Thank you for your quick and detailed response.
\nI understood that the word \"bingo\" was in the fqid, but did not know it would pop up when I pressed the correct item.
\nAs you said, I understand that features based on \"bingo\" words can help improve scores.
\nThank you so much for your help.
Congrats; nice correlation, Meta features on priors are the common theme , think that was the killer other than other smart features. And off course not being tempted by the lb 😀
\nBut nice nn approach as well by your team
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2325249": "The competition was really exciting and it gave us a chance to practice feature engineering. I'm very thankful for the support and help from my team @shinomoriaoshi @hoangnguyen719 and @martasprg. They were always there for me and together we made a big difference.\n\nI would like to thank the hosts, and special thanks to @cpmpml and @pdnartreb for identifying the issue of data leak, which made the competition right back on track.\n\nSpecial thanks to @cdeotte for his great starter notebooks and insights that helped me in the early phase of the competition.\n\n##Overview\nHere's an overview of what each of us worked on:\n· My main focus was on improving the XGBoost model and handling feature engineering.\n· Minh Tri Phan worked on a Transformer model with a CV (cross-validation) score of 0.699 and a public leaderboard (LB) score of 0.7.\n· Hoang processed the external data.\n· Martin worked on selecting the most relevant features.\n\nIn our final submissions, we ensembled the XGBoost and Transformer models, which helped us achieve the gold position. Our ensemble submission had a public LB score of **0.705** and a private LB score of approximately **0.7025**. Additionally, we had two other submissions with single XGBoost models, where one had a public LB score of **0.705** and a private LB score of **0.700**.\n\n##My Part\n\nCode: The code is a bit uncleaned, apologies for that. For any queries, contact me on [LinkedIn](