Hi there!βπ
\nI think that it is a possibly a good idea to use predicted labels for seventeen other questions to to predict the last one π€ . For each question of course π.
\nWe can use a labels file in such case to train a meta model and after we score test data with the main model we can use this meta model to correct and improve the final result.
\nDid somebody tried this? Could you share your insights?π
\nThank you!ππ€
Hi there!βπ
\nI think that it is a possibly a good idea to use predicted labels for seventeen other questions to to predict the last one π€ . For each question of course π.
\nWe can use a labels file in such case to train a meta model and after we score test data with the main model we can use this meta model to correct and improve the final result.
\nDid somebody tried this? Could you share your insights?π
\nThank you!ππ€
FYI, https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/392131
", "rawMarkdown": "FYI, https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/392131", "votes": null }, { "id": "2262297", "postDate": "05/16/2023 20:09:27", "content": "@shinomoriaoshi thank you! π I also have an idea to compute individual threshold for each question to convert probabilities to 0's and 1's. Is there a valid idea in your opinion? May be discussion tread for this topic also already exists?π€
", "rawMarkdown": "shinomoriaoshi thank you! π I also have an idea to compute individual threshold for each question to convert probabilities to 0's and 1's. Is there a valid idea in your opinion? May be discussion tread for this topic also already exists?π€", "votes": null }, { "id": "2262471", "postDate": "05/17/2023 01:02:41", "content": "I am not sure the topic has been discussed somewhere in the forum, but nice idea. In my opinion, we have less data to validate with this idea, making the validation a bit overfitting and less reliable. There are some Kagglers reporting that scores of individual questions improve but the overall score decreases.
", "rawMarkdown": "I am not sure the topic has been discussed somewhere in the forum, but nice idea. In my opinion, we have less data to validate with this idea, making the validation a bit overfitting and less reliable. There are some Kagglers reporting that scores of individual questions improve but the overall score decreases.", "votes": null }, { "id": "2264582", "postDate": "05/18/2023 14:51:26", "content": "Hi @ivanisaev,
\nWhen optimization is done by finding the best threshold for each question (i.e., optimize locally), it's not guaranteed that the global optimal is found. And the key is that the final evaluation is done by deriving Macro-F1 on flattened predicting results and the groundtruth. That is, our goal is to balance out the overall precision and recall, not question-wise. Some forums have discussed this topic, which are shown as follows:
\n\nHope this helps, thanks!
", "rawMarkdown": "Hi @ivanisaev,\n\nWhen optimization is done by finding the best threshold **for each question** (*i.e.,* optimize locally), it's not guaranteed that the global optimal is found. And the key is that the final evaluation is done by deriving Macro-F1 on **flattened** predicting results and the groundtruth. That is, our goal is to balance out the **overall** precision and recall, not question-wise. Some forums have discussed this topic, which are shown as follows:\n\n1. [What happen with F1-score for all question?](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/391062)\n2. [Different Thresholds for Different Question Models](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/389217)\n\nHope this helps, thanks!", "votes": null }, { "id": "2264852", "postDate": "05/18/2023 19:22:08", "content": "@abaojiang thank you for useful thoughts and links! It helps!π
", "rawMarkdown": "abaojiang thank you for useful thoughts and links! It helps!π", "votes": null } ], "comments": [ { "id": 2260961, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "05/16/2023 02:48:11", "content": "FYI, https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/392131
", "votes": null, "replies": [ { "id": 2262297, "author_name": "ivanisaev", "author_url": "", "post_date": "05/16/2023 20:09:27", "content": "@shinomoriaoshi thank you! π I also have an idea to compute individual threshold for each question to convert probabilities to 0's and 1's. Is there a valid idea in your opinion? May be discussion tread for this topic also already exists?π€
", "votes": null, "replies": [ { "id": 2262471, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "05/17/2023 01:02:41", "content": "I am not sure the topic has been discussed somewhere in the forum, but nice idea. In my opinion, we have less data to validate with this idea, making the validation a bit overfitting and less reliable. There are some Kagglers reporting that scores of individual questions improve but the overall score decreases.
", "votes": null, "replies": [] }, { "id": 2264582, "author_name": "abaojiang", "author_url": "", "post_date": "05/18/2023 14:51:26", "content": "Hi @ivanisaev,
\nWhen optimization is done by finding the best threshold for each question (i.e., optimize locally), it's not guaranteed that the global optimal is found. And the key is that the final evaluation is done by deriving Macro-F1 on flattened predicting results and the groundtruth. That is, our goal is to balance out the overall precision and recall, not question-wise. Some forums have discussed this topic, which are shown as follows:
\n\nHope this helps, thanks!
", "votes": null, "replies": [ { "id": 2264852, "author_name": "ivanisaev", "author_url": "", "post_date": "05/18/2023 19:22:08", "content": "@abaojiang thank you for useful thoughts and links! It helps!π
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2260828": "Hi there!βπ\nI think that it is a possibly a good idea to use predicted labels for seventeen other questions to to predict the last one π€ . For each question of course π. \nWe can use a labels file in such case to train a meta model and after we score test data with the main model we can use this meta model to correct and improve the final result. \nDid somebody tried this? Could you share your insights?π\nThank you!ππ€", "2260961": "FYI, https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/392131", "2262297": "shinomoriaoshi thank you! π I also have an idea to compute individual threshold for each question to convert probabilities to 0's and 1's. Is there a valid idea in your opinion? May be discussion tread for this topic also already exists?π€", "2262471": "I am not sure the topic has been discussed somewhere in the forum, but nice idea. In my opinion, we have less data to validate with this idea, making the validation a bit overfitting and less reliable. There are some Kagglers reporting that scores of individual questions improve but the overall score decreases.", "2264582": "Hi @ivanisaev,\n\nWhen optimization is done by finding the best threshold **for each question** (*i.e.,* optimize locally), it's not guaranteed that the global optimal is found. And the key is that the final evaluation is done by deriving Macro-F1 on **flattened** predicting results and the groundtruth. That is, our goal is to balance out the **overall** precision and recall, not question-wise. Some forums have discussed this topic, which are shown as follows:\n\n1. [What happen with F1-score for all question?](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/391062)\n2. [Different Thresholds for Different Question Models](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/389217)\n\nHope this helps, thanks!", "2264852": "abaojiang thank you for useful thoughts and links! It helps!π" }, "source": "meta" }