Many kagglers have built models for each question, but will this solution really optimize the evaluation metrics for this competition?
\nAs this Discussion points out, improving the accuracy of the model on individual questions does not necessarily improve the overall macro F1.
\nSo, the idea comes to mind, \"Can't all questions be handled by a single model?\"
\nHowever, despite various experiments, this idea has not yet worked.
\nFirst, I input the question number as a categorical variable into the GBDT, but it did not work. (CV=0.672 using Chris' baseline features).
\nNext, inspired by the 1st place solution in a similar past competition Riiid, I built a transformer with question number embedding as a query, which also did not work. (CV=0.592)
\nThus my experiments have not been successful, but I still believe that this framework can be a powerful weapon if it can be successfully trained.
\nIf anyone is working on a similar interesting topic, please feel free to discuss it with me!
Many kagglers have built models for each question, but will this solution really optimize the evaluation metrics for this competition?
\nAs this Discussion points out, improving the accuracy of the model on individual questions does not necessarily improve the overall macro F1.
\nSo, the idea comes to mind, \"Can't all questions be handled by a single model?\"
\nHowever, despite various experiments, this idea has not yet worked.
\nFirst, I input the question number as a categorical variable into the GBDT, but it did not work. (CV=0.672 using Chris' baseline features).
\nNext, inspired by the 1st place solution in a similar past competition Riiid, I built a transformer with question number embedding as a query, which also did not work. (CV=0.592)
\nThus my experiments have not been successful, but I still believe that this framework can be a powerful weapon if it can be successfully trained.
\nIf anyone is working on a similar interesting topic, please feel free to discuss it with me!
I have tried the same idea.
\nSpecifically, I tried two patterns: one in which level is added to the features as a categorical variable, and the other in which level and level_group are added to the features as categorical variables.
\nLike you, I did not see any improvement in CV.
\nAlso, I am not certain if the experiment was correct with the CV and LB correlated but with a large discrepancy in scores.
I did not experiment with Transformer or it would have been helpful. Thank you very much.
", "rawMarkdown": "I have tried the same idea.\n\nSpecifically, I tried two patterns: one in which level is added to the features as a categorical variable, and the other in which level and level_group are added to the features as categorical variables.\n\nLike you, I did not see any improvement in CV.\nAlso, I am not certain if the experiment was correct with the CV and LB correlated but with a large discrepancy in scores.\n\nI did not experiment with Transformer or it would have been helpful. Thank you very much.", "votes": null }, { "id": "2268760", "postDate": "05/22/2023 01:38:07", "content": "Previously, I mentioned that a single model may not be able to handle information for multiple purposes, and speculated that you may have used a tree model. However, Transformers have the capability to contain different information for different questions and can understand the relationship between questions and events at different times in sequence.
", "rawMarkdown": "Previously, I mentioned that a single model may not be able to handle information for multiple purposes, and speculated that you may have used a tree model. However, Transformers have the capability to contain different information for different questions and can understand the relationship between questions and events at different times in sequence.", "votes": null } ], "comments": [ { "id": 2268017, "author_name": "konumaru", "author_url": "", "post_date": "05/21/2023 11:48:35", "content": "I have tried the same idea.
\nSpecifically, I tried two patterns: one in which level is added to the features as a categorical variable, and the other in which level and level_group are added to the features as categorical variables.
\nLike you, I did not see any improvement in CV.
\nAlso, I am not certain if the experiment was correct with the CV and LB correlated but with a large discrepancy in scores.
I did not experiment with Transformer or it would have been helpful. Thank you very much.
", "votes": null, "replies": [] }, { "id": 2268760, "author_name": "mengvision", "author_url": "", "post_date": "05/22/2023 01:38:07", "content": "Previously, I mentioned that a single model may not be able to handle information for multiple purposes, and speculated that you may have used a tree model. However, Transformers have the capability to contain different information for different questions and can understand the relationship between questions and events at different times in sequence.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2267950": "Many kagglers have built models for each question, but will this solution really optimize the evaluation metrics for this competition?\nAs [this Discussion](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409497) points out, improving the accuracy of the model on individual questions does not necessarily improve the overall macro F1.\nSo, the idea comes to mind, \"Can't all questions be handled by a single model?\"\nHowever, despite various experiments, this idea has not yet worked.\nFirst, I input the question number as a categorical variable into the GBDT, but it did not work. (CV=0.672 using [Chris' baseline](https://www.kaggle.com/code/cdeotte/xgboost-baseline-0-680) features).\nNext, inspired by the [1st place solution](https://arxiv.org/ftp/arxiv/papers/2102/2102.05038.pdf) in a similar past competition Riiid, I built a transformer with question number embedding as a query, which also did not work. (CV=0.592)\nThus my experiments have not been successful, but I still believe that this framework can be a powerful weapon if it can be successfully trained.\nIf anyone is working on a similar interesting topic, please feel free to discuss it with me!", "2268017": "I have tried the same idea.\n\nSpecifically, I tried two patterns: one in which level is added to the features as a categorical variable, and the other in which level and level_group are added to the features as categorical variables.\n\nLike you, I did not see any improvement in CV.\nAlso, I am not certain if the experiment was correct with the CV and LB correlated but with a large discrepancy in scores.\n\nI did not experiment with Transformer or it would have been helpful. Thank you very much.", "2268760": "Previously, I mentioned that a single model may not be able to handle information for multiple purposes, and speculated that you may have used a tree model. However, Transformers have the capability to contain different information for different questions and can understand the relationship between questions and events at different times in sequence." }, "source": "meta" }