\n
![\"3.png\"](\"https://i.postimg.cc/Kv64VYSL/3.png\")
At first, thank you Kaggle and the hosts for hosting this interesting competition. Also, thanks the Kaggle community for all the sharings, discussions and kindness. I would like to share my solutions and the learning journey below (though the final ranking isn't that good 😓).
\nBecause the Macro-F1 score is derived on flattened prediction and groundtruth sequences, I think it would be better to somewhat retain the distribution of correctness when doing local CV. However, I find that it doesn't show superiority over the plain GroupKFold on session_id. Finally, I choose stratified KFold (k=5) based on #correct answers with 5 different seeds for stability of evaluation.
As shown in the figure below, the modeling process on the highest level consists of two stages, base model fitting and stacker training. The subsequent subsections describe these phases respectively.
\n
In stage1, the main model architecture is just the same as what I shared before. Nonetheless, more categorical features are taken into consideration, including event_comb (i.e., event_name + name), room_fqid, text_fqid, fqid, level and page (the corresponding embedding dimensions are shown in the figure). After the categorical embeddings are obtained, I use the difference of elapsed_time to tweak the scale of each event record (one column of the raw feature matrix), which can be thought of as the importance of that snapshot. Then a three-layer temporal convolution architecture is applied to extract the temporal pattern of the event sequence. After skip connection and readout, out-of-fold predictions (probas) are kept as the meta features for the stage2.
I still remember on 20th, Jun I thought that it seemed impossible to break 0.697 on local CV with this DL model. Over the past three months, I worked hard to explore and try various of model architecture, but all of them fail to surpass the one as discussed above. Therefore, I decided to try some ensemble methods to see if performance could improve. After some trials, restacking based on the meta features from the level-1 model with some FE features made me nearly touch the score of 0.7 on local side. With further blending of three stackers, I got to pass 0.7 by a small margin. The number of features (meta + FE) used in each level_group is shown as follows:
level_group \"0-4\": 3 + 22level_group \"5-12\": 13 + 26level_group \"13-22\": 18 + 30Following is the top performance report of my experiments.
\n| Model | \nCV | \nLB | \nPB | \n
|---|---|---|---|
| Single DL Model | \n0.6964 | \n0.697 | \n0.697 | \n
| Restack w/ Single XGB | \n0.6988 | \n0.699 | \n\n |
| Restack w/ Single CatBoost | \n0.6996 | \n0.698 | \n0.698 | \n
| Restack w/ CatBoost*1 + XGB*1 + LGBM*1 | \n0.6999 | \n0.701 | \n0.695 | \n
| Restack w/ CatBoost*1 + XGB*1 + LGBM*1 | \n0.7000 | \n0.699 | \n0.696 | \n
Though the performance indeed improves when restacking plays in, I think that I over-tune my model during the last week of the competition, which leads to overfit and fail to generalize to private test set.
\nhover_duration and the track of clicks (i.e., distance and angle between consecutive clicks).Although the final rank isn't good, I learn a lot from the Kaggle community and enjoy sharing some of my findings during this competitive journey. I'll keep progressing and try to clarify what I've lost (FE is crucial…). Thanks a lot for your patience!
", "messages": [ { "id": "2322825", "postDate": "06/29/2023 14:13:09", "content": "At first, thank you Kaggle and the hosts for hosting this interesting competition. Also, thanks the Kaggle community for all the sharings, discussions and kindness. I would like to share my solutions and the learning journey below (though the final ranking isn't that good 😓).
\nBecause the Macro-F1 score is derived on flattened prediction and groundtruth sequences, I think it would be better to somewhat retain the distribution of correctness when doing local CV. However, I find that it doesn't show superiority over the plain GroupKFold on session_id. Finally, I choose stratified KFold (k=5) based on #correct answers with 5 different seeds for stability of evaluation.
As shown in the figure below, the modeling process on the highest level consists of two stages, base model fitting and stacker training. The subsequent subsections describe these phases respectively.
\n
In stage1, the main model architecture is just the same as what I shared before. Nonetheless, more categorical features are taken into consideration, including event_comb (i.e., event_name + name), room_fqid, text_fqid, fqid, level and page (the corresponding embedding dimensions are shown in the figure). After the categorical embeddings are obtained, I use the difference of elapsed_time to tweak the scale of each event record (one column of the raw feature matrix), which can be thought of as the importance of that snapshot. Then a three-layer temporal convolution architecture is applied to extract the temporal pattern of the event sequence. After skip connection and readout, out-of-fold predictions (probas) are kept as the meta features for the stage2.
I still remember on 20th, Jun I thought that it seemed impossible to break 0.697 on local CV with this DL model. Over the past three months, I worked hard to explore and try various of model architecture, but all of them fail to surpass the one as discussed above. Therefore, I decided to try some ensemble methods to see if performance could improve. After some trials, restacking based on the meta features from the level-1 model with some FE features made me nearly touch the score of 0.7 on local side. With further blending of three stackers, I got to pass 0.7 by a small margin. The number of features (meta + FE) used in each level_group is shown as follows:
level_group \"0-4\": 3 + 22level_group \"5-12\": 13 + 26level_group \"13-22\": 18 + 30Following is the top performance report of my experiments.
\n| Model | \nCV | \nLB | \nPB | \n
|---|---|---|---|
| Single DL Model | \n0.6964 | \n0.697 | \n0.697 | \n
| Restack w/ Single XGB | \n0.6988 | \n0.699 | \n\n |
| Restack w/ Single CatBoost | \n0.6996 | \n0.698 | \n0.698 | \n
| Restack w/ CatBoost*1 + XGB*1 + LGBM*1 | \n0.6999 | \n0.701 | \n0.695 | \n
| Restack w/ CatBoost*1 + XGB*1 + LGBM*1 | \n0.7000 | \n0.699 | \n0.696 | \n
Though the performance indeed improves when restacking plays in, I think that I over-tune my model during the last week of the competition, which leads to overfit and fail to generalize to private test set.
\nhover_duration and the track of clicks (i.e., distance and angle between consecutive clicks).Although the final rank isn't good, I learn a lot from the Kaggle community and enjoy sharing some of my findings during this competitive journey. I'll keep progressing and try to clarify what I've lost (FE is crucial…). Thanks a lot for your patience!
", "rawMarkdown": "At first, thank you Kaggle and the hosts for hosting this interesting competition. Also, thanks the Kaggle community for all the sharings, discussions and kindness. I would like to share my solutions and the learning journey below (though the final ranking isn't that good 😓).\n\n# Overview\n* Add data from [fielddaylab](https://fielddaylab.wisc.edu/opengamedata/)\n* Use straified KFold based on #correct answers of each session\n* Implement two-level modeling\n * Level1: The base model is mainly based on my first sharing of the DL model architecture [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565).\n * Level2: The stackers (*e.g.,* xgb, lgbm and catboost) are trained with the oof probas (meta features) of the base model and simple FE features.\n\n# CV Scheme\nBecause the Macro-F1 score is derived on flattened prediction and groundtruth sequences, I think it would be better to somewhat retain the distribution of correctness when doing local CV. However, I find that it doesn't show superiority over the plain `GroupKFold` on `session_id`. Finally, I choose stratified KFold (k=5) based on #correct answers with 5 different seeds for stability of evaluation.\n\n# Modeling Process\nAs shown in the figure below, the modeling process on the highest level consists of two stages, *base model fitting* and *stacker training*. The subsequent subsections describe these phases respectively.\n[](https://postimg.cc/1fK9Vsz5)\n## Stage1 - Base Model Fitting\nIn stage1, the main model architecture is just the same as what I shared before. Nonetheless, more categorical features are taken into consideration, including `event_comb` (*i.e.,* `event_name` + `name`), `room_fqid`, `text_fqid`, `fqid`, `level` and `page` (the corresponding embedding dimensions are shown in the figure). After the categorical embeddings are obtained, I use the difference of `elapsed_time` to tweak the scale of each event record (one column of the raw feature matrix), which can be thought of as the importance of that snapshot. Then a three-layer temporal convolution architecture is applied to extract the temporal pattern of the event sequence. After skip connection and readout, out-of-fold predictions (probas) are kept as the meta features for the stage2.\n\n## Stage2 - Stacker Training\nI still remember on 20th, Jun I thought that it seemed impossible to break 0.697 on local CV with this DL model. Over the past three months, I worked hard to explore and try various of model architecture, but all of them fail to surpass the one as discussed above. Therefore, I decided to try some ensemble methods to see if performance could improve. After some trials, restacking based on the meta features from the level-1 model with some FE features made me nearly touch the score of 0.7 on local side. With further blending of three stackers, I got to pass 0.7 by a small margin. The number of features (meta + FE) used in each `level_group` is shown as follows:\n* `level_group` \"0-4\": 3 + 22\n* `level_group` \"5-12\": 13 + 26\n* `level_group` \"13-22\": 18 + 30\n\n# Performance Study\nFollowing is the top performance report of my experiments. \n\n| Model | CV | LB | PB |\n| -------------------------- | ------ | ----- | ----- |\n| Single DL Model | 0.6964 | 0.697 | 0.697 |\n| Restack w/ Single XGB | 0.6988 | 0.699 | 0.698 |\n| Restack w/ Single CatBoost | 0.6996 | 0.698 | 0.698 |\n| Restack w/ CatBoost\\*1 + XGB\\*1 + LGBM\\*1 | 0.6999 | **0.701** | 0.695 |\n| Restack w/ CatBoost\\*1 + XGB\\*1 + LGBM\\*1 | **0.7000** | 0.699 | 0.696 |\n\nThough the performance indeed improves when restacking plays in, I think that I over-tune my model during the last week of the competition, which leads to overfit and fail to generalize to private test set.\n\n# What Doesn't Work (on My Side)\n1. Retrain base models with all sessions (still find it hard to tune and select the right checkpoints...).\n2. Use question embedding to do question-wise attention on either raw event sequence or the temporal pattern extracted by temporal convolution architecture.\n3. Add more numeric features at level-1 input data, such as `hover_duration` and the track of clicks (*i.e.,* distance and angle between consecutive clicks).\n4. Try more temporal pattern extractors (*e.g.,* Conv3D, [SCINet](https://arxiv.org/abs/2106.09305)).\n\n# Conclusion\nAlthough the final rank isn't good, I learn a lot from the Kaggle community and enjoy sharing some of my findings during this competitive journey. I'll keep progressing and try to clarify what I've lost (FE is crucial...). Thanks a lot for your patience!", "votes": null }, { "id": "2325363", "postDate": "07/01/2023 09:21:45", "content": "Hi @abaojiang, the DL Notebook you posted is very well written and I learned a lot reading it and experimenting with it, but I couldn't get a competitive score out of it. Thanks for sharing.
", "rawMarkdown": "Hi @abaojiang, the DL Notebook you posted is very well written and I learned a lot reading it and experimenting with it, but I couldn't get a competitive score out of it. Thanks for sharing.", "votes": null }, { "id": "2325551", "postDate": "07/01/2023 12:54:01", "content": "Hi @gehallak,
\nI also learned a lot from the notebook you published, especially the analysis related to abnormal event counts and session lengths.
\nWith insufficient experiments, I failed to improve the performance with this model architecture in the end. There's still so much to learn. Thanks a lot for your comments and all the sharings 😊
", "rawMarkdown": "Hi @gehallak,\n\nI also learned a lot from the notebook you published, especially the analysis related to abnormal event counts and session lengths.\n\nWith insufficient experiments, I failed to improve the performance with this model architecture in the end. There's still so much to learn. Thanks a lot for your comments and all the sharings 😊", "votes": null } ], "comments": [ { "id": 2325363, "author_name": "gehallak", "author_url": "", "post_date": "07/01/2023 09:21:45", "content": "Hi @abaojiang, the DL Notebook you posted is very well written and I learned a lot reading it and experimenting with it, but I couldn't get a competitive score out of it. Thanks for sharing.
", "votes": null, "replies": [ { "id": 2325551, "author_name": "abaojiang", "author_url": "", "post_date": "07/01/2023 12:54:01", "content": "Hi @gehallak,
\nI also learned a lot from the notebook you published, especially the analysis related to abnormal event counts and session lengths.
\nWith insufficient experiments, I failed to improve the performance with this model architecture in the end. There's still so much to learn. Thanks a lot for your comments and all the sharings 😊
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2322825": "At first, thank you Kaggle and the hosts for hosting this interesting competition. Also, thanks the Kaggle community for all the sharings, discussions and kindness. I would like to share my solutions and the learning journey below (though the final ranking isn't that good 😓).\n\n# Overview\n* Add data from [fielddaylab](https://fielddaylab.wisc.edu/opengamedata/)\n* Use straified KFold based on #correct answers of each session\n* Implement two-level modeling\n * Level1: The base model is mainly based on my first sharing of the DL model architecture [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/398565).\n * Level2: The stackers (*e.g.,* xgb, lgbm and catboost) are trained with the oof probas (meta features) of the base model and simple FE features.\n\n# CV Scheme\nBecause the Macro-F1 score is derived on flattened prediction and groundtruth sequences, I think it would be better to somewhat retain the distribution of correctness when doing local CV. However, I find that it doesn't show superiority over the plain `GroupKFold` on `session_id`. Finally, I choose stratified KFold (k=5) based on #correct answers with 5 different seeds for stability of evaluation.\n\n# Modeling Process\nAs shown in the figure below, the modeling process on the highest level consists of two stages, *base model fitting* and *stacker training*. The subsequent subsections describe these phases respectively.\n[](https://postimg.cc/1fK9Vsz5)\n## Stage1 - Base Model Fitting\nIn stage1, the main model architecture is just the same as what I shared before. Nonetheless, more categorical features are taken into consideration, including `event_comb` (*i.e.,* `event_name` + `name`), `room_fqid`, `text_fqid`, `fqid`, `level` and `page` (the corresponding embedding dimensions are shown in the figure). After the categorical embeddings are obtained, I use the difference of `elapsed_time` to tweak the scale of each event record (one column of the raw feature matrix), which can be thought of as the importance of that snapshot. Then a three-layer temporal convolution architecture is applied to extract the temporal pattern of the event sequence. After skip connection and readout, out-of-fold predictions (probas) are kept as the meta features for the stage2.\n\n## Stage2 - Stacker Training\nI still remember on 20th, Jun I thought that it seemed impossible to break 0.697 on local CV with this DL model. Over the past three months, I worked hard to explore and try various of model architecture, but all of them fail to surpass the one as discussed above. Therefore, I decided to try some ensemble methods to see if performance could improve. After some trials, restacking based on the meta features from the level-1 model with some FE features made me nearly touch the score of 0.7 on local side. With further blending of three stackers, I got to pass 0.7 by a small margin. The number of features (meta + FE) used in each `level_group` is shown as follows:\n* `level_group` \"0-4\": 3 + 22\n* `level_group` \"5-12\": 13 + 26\n* `level_group` \"13-22\": 18 + 30\n\n# Performance Study\nFollowing is the top performance report of my experiments. \n\n| Model | CV | LB | PB |\n| -------------------------- | ------ | ----- | ----- |\n| Single DL Model | 0.6964 | 0.697 | 0.697 |\n| Restack w/ Single XGB | 0.6988 | 0.699 | 0.698 |\n| Restack w/ Single CatBoost | 0.6996 | 0.698 | 0.698 |\n| Restack w/ CatBoost\\*1 + XGB\\*1 + LGBM\\*1 | 0.6999 | **0.701** | 0.695 |\n| Restack w/ CatBoost\\*1 + XGB\\*1 + LGBM\\*1 | **0.7000** | 0.699 | 0.696 |\n\nThough the performance indeed improves when restacking plays in, I think that I over-tune my model during the last week of the competition, which leads to overfit and fail to generalize to private test set.\n\n# What Doesn't Work (on My Side)\n1. Retrain base models with all sessions (still find it hard to tune and select the right checkpoints...).\n2. Use question embedding to do question-wise attention on either raw event sequence or the temporal pattern extracted by temporal convolution architecture.\n3. Add more numeric features at level-1 input data, such as `hover_duration` and the track of clicks (*i.e.,* distance and angle between consecutive clicks).\n4. Try more temporal pattern extractors (*e.g.,* Conv3D, [SCINet](https://arxiv.org/abs/2106.09305)).\n\n# Conclusion\nAlthough the final rank isn't good, I learn a lot from the Kaggle community and enjoy sharing some of my findings during this competitive journey. I'll keep progressing and try to clarify what I've lost (FE is crucial...). Thanks a lot for your patience!", "2325363": "Hi @abaojiang, the DL Notebook you posted is very well written and I learned a lot reading it and experimenting with it, but I couldn't get a competitive score out of it. Thanks for sharing.", "2325551": "Hi @gehallak,\n\nI also learned a lot from the notebook you published, especially the analysis related to abnormal event counts and session lengths.\n\nWith insufficient experiments, I failed to improve the performance with this model architecture in the end. There's still so much to learn. Thanks a lot for your comments and all the sharings 😊" }, "source": "meta" }