{
  "id": 209584,
  "title": "[70th] My transformer solution (with code)",
  "url": "/competitions/riiid-test-answer-prediction/writeups/yih-dar-shieh-70th-my-transformer-solution-with-co",
  "author_name": "",
  "post_date": "2021-01-08T00:26:57.750Z",
  "votes": 27,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Congratulations to all winners!</p>\n<p>Hope you all enjoyed and learned something in this great competition (ok, submission errors were painful …)</p>\n<p>I want to share briefly my solution here, although it is not the one I got the private LB scores (I have difficultly to debug my final version).</p>\n<p>My solution is transformer model, and here is the summary:</p>\n<ul>\n<li><p>Training / Validation is done with TensorFlow TPU<br>\n    * This is quite fast, 1 epoch takes 5 - 6 minutes (1M sequences of length  128)<br>\n    * Check <a href=\"https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students\" target=\"_blank\">TPU - Track knowledge states of 1M+ students in the wild</a>.<br>\n    * Use Google Colab TPU for training </p></li>\n<li><p>Use elapsed time / lag time : LB: 0.781 -&gt; 0.795 ~ 0.797</p></li>\n<li><p>I also included the loss for predicting the user actual response, not only the response correctness. This makes the training can run longer without overfitting.  LB: 0.797 -&gt; 0.800</p></li>\n<li><p>I included the count of (in previous time) each part and each correct answer (of the questions), and their correctness ratio. On CV, 0.807 / 0.808 -&gt; 0.809 / 0.810.</p></li>\n<li><p>Finally, I included the count of the correction and the correction ratio of each question seen by each user. On CV  0.809 / 0.810 -&gt; 0.813.</p></li>\n</ul>\n<p>Here is the link to my notebook:</p>\n<p><a href=\"https://www.kaggle.com/yihdarshieh/r3id-transformer/\" target=\"_blank\">R3ID - Transformer</a></p>\n<p>ps. Since I tried to fix the inference bug , the code is somehow messy …</p>",
  "messages": [
    {
      "id": "1143545",
      "postDate": "01/08/2021 00:19:57",
      "content": "<p>Congratulations to all winners!</p>\n<p>Hope you all enjoyed and learned something in this great competition (ok, submission errors were painful …)</p>\n<p>I want to share briefly my solution here, although it is not the one I got the private LB scores (I have difficultly to debug my final version).</p>\n<p>My solution is transformer model, and here is the summary:</p>\n<ul>\n<li><p>Training / Validation is done with TensorFlow TPU<br>\n    * This is quite fast, 1 epoch takes 5 - 6 minutes (1M sequences of length  128)<br>\n    * Check <a href=\"https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students\" target=\"_blank\">TPU - Track knowledge states of 1M+ students in the wild</a>.<br>\n    * Use Google Colab TPU for training </p></li>\n<li><p>Use elapsed time / lag time : LB: 0.781 -&gt; 0.795 ~ 0.797</p></li>\n<li><p>I also included the loss for predicting the user actual response, not only the response correctness. This makes the training can run longer without overfitting.  LB: 0.797 -&gt; 0.800</p></li>\n<li><p>I included the count of (in previous time) each part and each correct answer (of the questions), and their correctness ratio. On CV, 0.807 / 0.808 -&gt; 0.809 / 0.810.</p></li>\n<li><p>Finally, I included the count of the correction and the correction ratio of each question seen by each user. On CV  0.809 / 0.810 -&gt; 0.813.</p></li>\n</ul>\n<p>Here is the link to my notebook:</p>\n<p><a href=\"https://www.kaggle.com/yihdarshieh/r3id-transformer/\" target=\"_blank\">R3ID - Transformer</a></p>\n<p>ps. Since I tried to fix the inference bug , the code is somehow messy …</p>",
      "rawMarkdown": "Congratulations to all winners!\n\nHope you all enjoyed and learned something in this great competition (ok, submission errors were painful ...)\n\nI want to share briefly my solution here, although it is not the one I got the private LB scores (I have difficultly to debug my final version).\n\nMy solution is transformer model, and here is the summary:\n\n   - Training / Validation is done with TensorFlow TPU\n        * This is quite fast, 1 epoch takes 5 - 6 minutes (1M sequences of length  128)\n        * Check [TPU - Track knowledge states of 1M+ students in the wild](https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students).\n        * Use Google Colab TPU for training \n\n   - Use elapsed time / lag time : LB: 0.781 -> 0.795 ~ 0.797\n   - I also included the loss for predicting the user actual response, not only the response correctness. This makes the training can run longer without overfitting.  LB: 0.797 -> 0.800\n   - I included the count of (in previous time) each part and each correct answer (of the questions), and their correctness ratio. On CV, 0.807 / 0.808 -> 0.809 / 0.810.\n   - Finally, I included the count of the correction and the correction ratio of each question seen by each user. On CV  0.809 / 0.810 -> 0.813.\n\n\n\nHere is the link to my notebook:\n\n[R3ID - Transformer](https://www.kaggle.com/yihdarshieh/r3id-transformer/)\n\nps. Since I tried to fix the inference bug , the code is somehow messy ...",
      "votes": null
    },
    {
      "id": "1143559",
      "postDate": "01/08/2021 00:26:34",
      "content": "<p>Thanks for making it open source and really cool code ❤️! Congratulations 🎉</p>",
      "rawMarkdown": "Thanks for making it open source and really cool code ❤️! Congratulations 🎉",
      "votes": null
    },
    {
      "id": "1143578",
      "postDate": "01/08/2021 00:39:17",
      "content": "<p>Thanks for sharing your idea and codes with TPU.</p>",
      "rawMarkdown": "Thanks for sharing your idea and codes with TPU.",
      "votes": null
    },
    {
      "id": "1143688",
      "postDate": "01/08/2021 02:15:33",
      "content": "<p>Thanks for sharing you codes!!!  I really need it to figure out where did I wrong about my saint+ model.  Congratulations!</p>",
      "rawMarkdown": "Thanks for sharing you codes!!!  I really need it to figure out where did I wrong about my saint+ model.  Congratulations!",
      "votes": null
    },
    {
      "id": "1146232",
      "postDate": "01/09/2021 16:08:12",
      "content": "<p>thanks for sharing! using tpu is novel!</p>",
      "rawMarkdown": "thanks for sharing! using tpu is novel!",
      "votes": null
    },
    {
      "id": "1146345",
      "postDate": "01/09/2021 17:56:47",
      "content": "<p>Thank you mate!</p>",
      "rawMarkdown": "Thank you mate!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1143559,
      "author_name": "adityaecdrid",
      "author_url": "",
      "post_date": "01/08/2021 00:26:34",
      "content": "<p>Thanks for making it open source and really cool code ❤️! Congratulations 🎉</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1143578,
      "author_name": "sishihara",
      "author_url": "",
      "post_date": "01/08/2021 00:39:17",
      "content": "<p>Thanks for sharing your idea and codes with TPU.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1143688,
      "author_name": "maxchang0724",
      "author_url": "",
      "post_date": "01/08/2021 02:15:33",
      "content": "<p>Thanks for sharing you codes!!!  I really need it to figure out where did I wrong about my saint+ model.  Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1146232,
      "author_name": "zjjszj2",
      "author_url": "",
      "post_date": "01/09/2021 16:08:12",
      "content": "<p>thanks for sharing! using tpu is novel!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1146345,
      "author_name": "claverru",
      "author_url": "",
      "post_date": "01/09/2021 17:56:47",
      "content": "<p>Thank you mate!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1143545": "Congratulations to all winners!\n\nHope you all enjoyed and learned something in this great competition (ok, submission errors were painful ...)\n\nI want to share briefly my solution here, although it is not the one I got the private LB scores (I have difficultly to debug my final version).\n\nMy solution is transformer model, and here is the summary:\n\n   - Training / Validation is done with TensorFlow TPU\n        * This is quite fast, 1 epoch takes 5 - 6 minutes (1M sequences of length  128)\n        * Check [TPU - Track knowledge states of 1M+ students in the wild](https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students).\n        * Use Google Colab TPU for training \n\n   - Use elapsed time / lag time : LB: 0.781 -> 0.795 ~ 0.797\n   - I also included the loss for predicting the user actual response, not only the response correctness. This makes the training can run longer without overfitting.  LB: 0.797 -> 0.800\n   - I included the count of (in previous time) each part and each correct answer (of the questions), and their correctness ratio. On CV, 0.807 / 0.808 -> 0.809 / 0.810.\n   - Finally, I included the count of the correction and the correction ratio of each question seen by each user. On CV  0.809 / 0.810 -> 0.813.\n\n\n\nHere is the link to my notebook:\n\n[R3ID - Transformer](https://www.kaggle.com/yihdarshieh/r3id-transformer/)\n\nps. Since I tried to fix the inference bug , the code is somehow messy ...",
    "1143559": "Thanks for making it open source and really cool code ❤️! Congratulations 🎉",
    "1143578": "Thanks for sharing your idea and codes with TPU.",
    "1143688": "Thanks for sharing you codes!!!  I really need it to figure out where did I wrong about my saint+ model.  Congratulations!",
    "1146232": "thanks for sharing! using tpu is novel!",
    "1146345": "Thank you mate!"
  },
  "source": "meta"
}