I tired the full convolution neural network baseline which gives good results on 1,2,3,9,12,17 and 18 questions.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F579998%2Fb3ba0f932318d9dd043ab81e1e764829%2Fjw%20results.jpg?generation=1682352637741314&alt=media\")
\nAbove are result on the training on full train dataset on GPU during 8 hours. You can find more information in this notebook https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn
\nModel in this notebook is just a simple model and shows that in a few questions obtain a good results is not too difficult. I also tried transformer model that gives significantly better scores.
Question: 1 F1: 0.843 Question: 2 F1: 0.989
\nQuestion: 3 F1: 0.966 Question: 4 F1: 0.890
\nQuestion: 5 F1: 0.720 Question: 6 F1: 0.874
\nQuestion: 7 F1: 0.848 Question: 8 F1: 0.760
\nQuestion: 9 F1: 0.850 Question: 10 F1: 0.615
\nQuestion: 11 F1: 0.784 Question: 12 F1: 0.926
\nQuestion: 13 F1: 0.375 Question: 14, F1: 0.84
\nQuestion: 15, F1: 0.63 Question: 16 F1: 0.847
\nQuestion: 17 F1: 0.815 Question: 18 F1: 0.975
\nAverage F1 is about 0.8.
\nIf you find this information interesting I will be glad for your upvote, this motivates a lot to share the results! Thank you!
", "messages": [ { "id": "2232794", "postDate": "04/24/2023 16:12:53", "content": "I tired the full convolution neural network baseline which gives good results on 1,2,3,9,12,17 and 18 questions.
\n
\nAbove are result on the training on full train dataset on GPU during 8 hours. You can find more information in this notebook https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn
\nModel in this notebook is just a simple model and shows that in a few questions obtain a good results is not too difficult. I also tried transformer model that gives significantly better scores.
Question: 1 F1: 0.843 Question: 2 F1: 0.989
\nQuestion: 3 F1: 0.966 Question: 4 F1: 0.890
\nQuestion: 5 F1: 0.720 Question: 6 F1: 0.874
\nQuestion: 7 F1: 0.848 Question: 8 F1: 0.760
\nQuestion: 9 F1: 0.850 Question: 10 F1: 0.615
\nQuestion: 11 F1: 0.784 Question: 12 F1: 0.926
\nQuestion: 13 F1: 0.375 Question: 14, F1: 0.84
\nQuestion: 15, F1: 0.63 Question: 16 F1: 0.847
\nQuestion: 17 F1: 0.815 Question: 18 F1: 0.975
\nAverage F1 is about 0.8.
\nIf you find this information interesting I will be glad for your upvote, this motivates a lot to share the results! Thank you!
", "rawMarkdown": "I tired the full convolution neural network baseline which gives good results on 1,2,3,9,12,17 and 18 questions. \n\nAbove are result on the training on full train dataset on GPU during 8 hours. You can find more information in this notebook https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn\nModel in this notebook is just a simple model and shows that in a few questions obtain a good results is not too difficult. I also tried transformer model that gives significantly better scores.\n\nQuestion: 1 F1: 0.843 Question: 2 F1: 0.989\n\nQuestion: 3 F1: 0.966 Question: 4 F1: 0.890\n\nQuestion: 5 F1: 0.720 Question: 6 F1: 0.874\n\nQuestion: 7 F1: 0.848 Question: 8 F1: 0.760\n\nQuestion: 9 F1: 0.850 Question: 10 F1: 0.615\n\nQuestion: 11 F1: 0.784 Question: 12 F1: 0.926\n\nQuestion: 13 F1: 0.375 Question: 14, F1: 0.84\n\nQuestion: 15, F1: 0.63 Question: 16 F1: 0.847\n\nQuestion: 17 F1: 0.815 Question: 18 F1: 0.975\n\nAverage F1 is about 0.8.\n\nIf you find this information interesting I will be glad for your upvote, this motivates a lot to share the results! Thank you!", "votes": null }, { "id": "2232864", "postDate": "04/24/2023 17:15:05", "content": "Unfortunately, I think the F1 scores in your results are from binary averaging, meanwhile, the competition metric is with macro averaging. It is unfortunate as the only official source of this information is a very tiny tag at the bottom of this page, this also took me around 4 weeks to realize, as I thought I got an issue with cross-validation. I think it should be written down clearly. Great work, anyway!
", "rawMarkdown": "Unfortunately, I think the F1 scores in your results are from binary averaging, meanwhile, the competition metric is with macro averaging. It is unfortunate as the only official source of this information is a very tiny tag at the bottom of [this](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/evaluation) page, this also took me around 4 weeks to realize, as I thought I got an issue with cross-validation. I think it should be written down clearly. Great work, anyway!", "votes": null }, { "id": "2235978", "postDate": "04/26/2023 13:06:41", "content": "@shinomoriaoshi Thank you for your comment! It is very useful. Now rerun the training for transformer model with macro averaging F1. Will share the results!
", "rawMarkdown": "shinomoriaoshi Thank you for your comment! It is very useful. Now rerun the training for transformer model with macro averaging F1. Will share the results!", "votes": null }, { "id": "2236375", "postDate": "04/26/2023 19:55:36", "content": "Looking forward to your notebook. When I first saw the data of this competition, Transformer is the first thing I thought of, but the best I can do with a transformer currently is 0.6940 CV (haven't submitted it yet). I really doubt the power of Transformer in this competition, or maybe mine is not good enough.
", "rawMarkdown": "Looking forward to your notebook. When I first saw the data of this competition, Transformer is the first thing I thought of, but the best I can do with a transformer currently is 0.6940 CV (haven't submitted it yet). I really doubt the power of Transformer in this competition, or maybe mine is not good enough.", "votes": null }, { "id": "2236565", "postDate": "04/27/2023 01:22:09", "content": "0.6940 CV is also good. This is better than mine BERT-like model. I correctied metric to F1 macro average and scores I posted above decreased to near 0.6
", "rawMarkdown": "0.6940 CV is also good. This is better than mine BERT-like model. I correctied metric to F1 macro average and scores I posted above decreased to near 0.6", "votes": null } ], "comments": [ { "id": 2232864, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "04/24/2023 17:15:05", "content": "Unfortunately, I think the F1 scores in your results are from binary averaging, meanwhile, the competition metric is with macro averaging. It is unfortunate as the only official source of this information is a very tiny tag at the bottom of this page, this also took me around 4 weeks to realize, as I thought I got an issue with cross-validation. I think it should be written down clearly. Great work, anyway!
", "votes": null, "replies": [ { "id": 2235978, "author_name": "ivanisaev", "author_url": "", "post_date": "04/26/2023 13:06:41", "content": "@shinomoriaoshi Thank you for your comment! It is very useful. Now rerun the training for transformer model with macro averaging F1. Will share the results!
", "votes": null, "replies": [ { "id": 2236375, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "04/26/2023 19:55:36", "content": "Looking forward to your notebook. When I first saw the data of this competition, Transformer is the first thing I thought of, but the best I can do with a transformer currently is 0.6940 CV (haven't submitted it yet). I really doubt the power of Transformer in this competition, or maybe mine is not good enough.
", "votes": null, "replies": [ { "id": 2236565, "author_name": "ivanisaev", "author_url": "", "post_date": "04/27/2023 01:22:09", "content": "0.6940 CV is also good. This is better than mine BERT-like model. I correctied metric to F1 macro average and scores I posted above decreased to near 0.6
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2232794": "I tired the full convolution neural network baseline which gives good results on 1,2,3,9,12,17 and 18 questions. \n\nAbove are result on the training on full train dataset on GPU during 8 hours. You can find more information in this notebook https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn\nModel in this notebook is just a simple model and shows that in a few questions obtain a good results is not too difficult. I also tried transformer model that gives significantly better scores.\n\nQuestion: 1 F1: 0.843 Question: 2 F1: 0.989\n\nQuestion: 3 F1: 0.966 Question: 4 F1: 0.890\n\nQuestion: 5 F1: 0.720 Question: 6 F1: 0.874\n\nQuestion: 7 F1: 0.848 Question: 8 F1: 0.760\n\nQuestion: 9 F1: 0.850 Question: 10 F1: 0.615\n\nQuestion: 11 F1: 0.784 Question: 12 F1: 0.926\n\nQuestion: 13 F1: 0.375 Question: 14, F1: 0.84\n\nQuestion: 15, F1: 0.63 Question: 16 F1: 0.847\n\nQuestion: 17 F1: 0.815 Question: 18 F1: 0.975\n\nAverage F1 is about 0.8.\n\nIf you find this information interesting I will be glad for your upvote, this motivates a lot to share the results! Thank you!", "2232864": "Unfortunately, I think the F1 scores in your results are from binary averaging, meanwhile, the competition metric is with macro averaging. It is unfortunate as the only official source of this information is a very tiny tag at the bottom of [this](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/evaluation) page, this also took me around 4 weeks to realize, as I thought I got an issue with cross-validation. I think it should be written down clearly. Great work, anyway!", "2235978": "shinomoriaoshi Thank you for your comment! It is very useful. Now rerun the training for transformer model with macro averaging F1. Will share the results!", "2236375": "Looking forward to your notebook. When I first saw the data of this competition, Transformer is the first thing I thought of, but the best I can do with a transformer currently is 0.6940 CV (haven't submitted it yet). I really doubt the power of Transformer in this competition, or maybe mine is not good enough.", "2236565": "0.6940 CV is also good. This is better than mine BERT-like model. I correctied metric to F1 macro average and scores I posted above decreased to near 0.6" }, "source": "meta" }