{
  "id": 407501,
  "title": "🚀 Results of applying BERT-like approach",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/407501",
  "author_name": "Ivan Isaev",
  "post_date": "2023-05-06T18:19:39.191000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<p><strong>Hi Kagglers!👋🙂</strong><br>\n<strong>🔬Intro</strong> <br>\nAs I wrote <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400858\" target=\"_blank\">there</a> and <a href=\"https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn\" target=\"_blank\">there</a> i made a try to reconstruct model from <a href=\"https://arxiv.org/pdf/2204.13607.pdf\" target=\"_blank\">Process-BERT: A Framework for Representation Learning on Educational Process Data</a> precisely from authors <a href=\"https://github.com/alexscarlatos/clickstream-assessments/\" target=\"_blank\">GitHub repo</a> and firstly got promising results 🚀 with F1 near 0.8. But later with help of <a href=\"https://www.kaggle.com/thedevastator\" target=\"_blank\">@thedevastator</a> <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a>  I understood that I calculated F1-metric incorrectly. After correct calculation the results decreased to about 0.6.<br>\nNevertheless this was a quite huge work (few weeks) and a lot of useful experience 🙂.</p>\n<p><strong>🗒️ About notebook</strong><br>\n📌 I promised to share my code of this experiment and it is <a href=\"https://www.kaggle.com/code/ivanisaev/jo-wilder-bert-like-lstm-model/notebook\" target=\"_blank\">in this notebook</a>.<br>\n📌 Because of large amount of train data processing takes a lot of time (near the 24 hours with appropriate model training). I refactored code as clean as I could. But as I think it is still quite difficult to read. This is in particular because I modified the existing code.<br>\n📌 I want to share it. Maybe it will be useful for someone who also tries transformers in this competition or if somebody is also trying to reconstruct model from this publication.<br>\n📌 I didn't run this notebook on Kaggle due to long processing time. But I am planning also share later the notebook with training results will cells outputs if I will find it (couldn't to find it now). If you find this notebook useful -- I will be glad to answer your questions.<br>\nCheers!👋 </p>",
  "messages": [
    {
      "id": 2248328,
      "postDate": "2023-05-06T18:19:39.190Z",
      "content": "<p><strong>Hi Kagglers!👋🙂</strong><br>\n<strong>🔬Intro</strong> <br>\nAs I wrote <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400858\" target=\"_blank\">there</a> and <a href=\"https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn\" target=\"_blank\">there</a> i made a try to reconstruct model from <a href=\"https://arxiv.org/pdf/2204.13607.pdf\" target=\"_blank\">Process-BERT: A Framework for Representation Learning on Educational Process Data</a> precisely from authors <a href=\"https://github.com/alexscarlatos/clickstream-assessments/\" target=\"_blank\">GitHub repo</a> and firstly got promising results 🚀 with F1 near 0.8. But later with help of <a href=\"https://www.kaggle.com/thedevastator\" target=\"_blank\">@thedevastator</a> <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a>  I understood that I calculated F1-metric incorrectly. After correct calculation the results decreased to about 0.6.<br>\nNevertheless this was a quite huge work (few weeks) and a lot of useful experience 🙂.</p>\n<p><strong>🗒️ About notebook</strong><br>\n📌 I promised to share my code of this experiment and it is <a href=\"https://www.kaggle.com/code/ivanisaev/jo-wilder-bert-like-lstm-model/notebook\" target=\"_blank\">in this notebook</a>.<br>\n📌 Because of large amount of train data processing takes a lot of time (near the 24 hours with appropriate model training). I refactored code as clean as I could. But as I think it is still quite difficult to read. This is in particular because I modified the existing code.<br>\n📌 I want to share it. Maybe it will be useful for someone who also tries transformers in this competition or if somebody is also trying to reconstruct model from this publication.<br>\n📌 I didn't run this notebook on Kaggle due to long processing time. But I am planning also share later the notebook with training results will cells outputs if I will find it (couldn't to find it now). If you find this notebook useful -- I will be glad to answer your questions.<br>\nCheers!👋 </p>",
      "rawMarkdown": "**Hi Kagglers!👋🙂**\n**🔬Intro** \nAs I wrote [there](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400858) and [there](https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn) i made a try to reconstruct model from [Process-BERT: A Framework for Representation Learning on Educational Process Data](https://arxiv.org/pdf/2204.13607.pdf) precisely from authors [GitHub repo](https://github.com/alexscarlatos/clickstream-assessments/) and firstly got promising results 🚀 with F1 near 0.8. But later with help of @thedevastator @shinomoriaoshi  I understood that I calculated F1-metric incorrectly. After correct calculation the results decreased to about 0.6.\nNevertheless this was a quite huge work (few weeks) and a lot of useful experience 🙂.\n\n**🗒️ About notebook**\n📌 I promised to share my code of this experiment and it is [in this notebook](https://www.kaggle.com/code/ivanisaev/jo-wilder-bert-like-lstm-model/notebook).\n📌 Because of large amount of train data processing takes a lot of time (near the 24 hours with appropriate model training). I refactored code as clean as I could. But as I think it is still quite difficult to read. This is in particular because I modified the existing code.\n📌 I want to share it. Maybe it will be useful for someone who also tries transformers in this competition or if somebody is also trying to reconstruct model from this publication.\n📌 I didn't run this notebook on Kaggle due to long processing time. But I am planning also share later the notebook with training results will cells outputs if I will find it (couldn't to find it now). If you find this notebook useful -- I will be glad to answer your questions.\nCheers!👋 ",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2248328": "**Hi Kagglers!👋🙂**\n**🔬Intro** \nAs I wrote [there](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400858) and [there](https://www.kaggle.com/code/ivanisaev/0-99-f1-on-a-few-questions-with-simple-nn) i made a try to reconstruct model from [Process-BERT: A Framework for Representation Learning on Educational Process Data](https://arxiv.org/pdf/2204.13607.pdf) precisely from authors [GitHub repo](https://github.com/alexscarlatos/clickstream-assessments/) and firstly got promising results 🚀 with F1 near 0.8. But later with help of @thedevastator @shinomoriaoshi  I understood that I calculated F1-metric incorrectly. After correct calculation the results decreased to about 0.6.\nNevertheless this was a quite huge work (few weeks) and a lot of useful experience 🙂.\n\n**🗒️ About notebook**\n📌 I promised to share my code of this experiment and it is [in this notebook](https://www.kaggle.com/code/ivanisaev/jo-wilder-bert-like-lstm-model/notebook).\n📌 Because of large amount of train data processing takes a lot of time (near the 24 hours with appropriate model training). I refactored code as clean as I could. But as I think it is still quite difficult to read. This is in particular because I modified the existing code.\n📌 I want to share it. Maybe it will be useful for someone who also tries transformers in this competition or if somebody is also trying to reconstruct model from this publication.\n📌 I didn't run this notebook on Kaggle due to long processing time. But I am planning also share later the notebook with training results will cells outputs if I will find it (couldn't to find it now). If you find this notebook useful -- I will be glad to answer your questions.\nCheers!👋 "
  }
}