{
  "id": 210276,
  "title": "39th Solution",
  "url": "/competitions/riiid-test-answer-prediction/writeups/abdessalem-boukil-39th-solution",
  "author_name": "",
  "post_date": "2021-01-10T09:14:11.147Z",
  "votes": 12,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I took me a while to post my solution since I was cleaning all the messy code I had. My solution was a SAINT-like model, however a bit simpler in architecture. I was surprised how stripping the model out of features that intuitively make sense can improve the score.</p>\n<p>It was d_model: 128 , att_heads: 2 , ff_model: 512 , dropout: 0.1 , 4 layers , trained on whole data for 100 + epochs. It breaks the 0.800 ROC bar around the 60th epoch.</p>\n<p>Since many people failed to successfully implement the SAINT model, I created a notebook with detailed explanation, and clear code that takes raw data, created a dataloader, train the SAINT model, then perform inference using Tito's iterator. The model in the notebook is 64 d_model, it score 0.789  in validation, scaling it would take it 0.800+.</p>\n<p>Notebook: <a href=\"https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution\" target=\"_blank\">https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution</a> </p>\n<p>Thanks for the awesome competition!</p>",
  "messages": [
    {
      "id": "1147068",
      "postDate": "01/10/2021 09:11:55",
      "content": "<p>I took me a while to post my solution since I was cleaning all the messy code I had. My solution was a SAINT-like model, however a bit simpler in architecture. I was surprised how stripping the model out of features that intuitively make sense can improve the score.</p>\n<p>It was d_model: 128 , att_heads: 2 , ff_model: 512 , dropout: 0.1 , 4 layers , trained on whole data for 100 + epochs. It breaks the 0.800 ROC bar around the 60th epoch.</p>\n<p>Since many people failed to successfully implement the SAINT model, I created a notebook with detailed explanation, and clear code that takes raw data, created a dataloader, train the SAINT model, then perform inference using Tito's iterator. The model in the notebook is 64 d_model, it score 0.789  in validation, scaling it would take it 0.800+.</p>\n<p>Notebook: <a href=\"https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution\" target=\"_blank\">https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution</a> </p>\n<p>Thanks for the awesome competition!</p>",
      "rawMarkdown": "I took me a while to post my solution since I was cleaning all the messy code I had. My solution was a SAINT-like model, however a bit simpler in architecture. I was surprised how stripping the model out of features that intuitively make sense can improve the score.\n\nIt was d_model: 128 , att_heads: 2 , ff_model: 512 , dropout: 0.1 , 4 layers , trained on whole data for 100 + epochs. It breaks the 0.800 ROC bar around the 60th epoch.\n\nSince many people failed to successfully implement the SAINT model, I created a notebook with detailed explanation, and clear code that takes raw data, created a dataloader, train the SAINT model, then perform inference using Tito's iterator. The model in the notebook is 64 d_model, it score 0.789  in validation, scaling it would take it 0.800+.\n\nNotebook: https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution \n\nThanks for the awesome competition!",
      "votes": null
    },
    {
      "id": "1147284",
      "postDate": "01/10/2021 12:08:04",
      "content": "<p>Congrats and thanks for sharing</p>",
      "rawMarkdown": "Congrats and thanks for sharing",
      "votes": null
    },
    {
      "id": "1147886",
      "postDate": "01/10/2021 19:33:28",
      "content": "<p>Congrats on your result and thank you for sharing a well explained notebook.</p>",
      "rawMarkdown": "Congrats on your result and thank you for sharing a well explained notebook.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1147284,
      "author_name": "darren1515",
      "author_url": "",
      "post_date": "01/10/2021 12:08:04",
      "content": "<p>Congrats and thanks for sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1147886,
      "author_name": "sishihara",
      "author_url": "",
      "post_date": "01/10/2021 19:33:28",
      "content": "<p>Congrats on your result and thank you for sharing a well explained notebook.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1147068": "I took me a while to post my solution since I was cleaning all the messy code I had. My solution was a SAINT-like model, however a bit simpler in architecture. I was surprised how stripping the model out of features that intuitively make sense can improve the score.\n\nIt was d_model: 128 , att_heads: 2 , ff_model: 512 , dropout: 0.1 , 4 layers , trained on whole data for 100 + epochs. It breaks the 0.800 ROC bar around the 60th epoch.\n\nSince many people failed to successfully implement the SAINT model, I created a notebook with detailed explanation, and clear code that takes raw data, created a dataloader, train the SAINT model, then perform inference using Tito's iterator. The model in the notebook is 64 d_model, it score 0.789  in validation, scaling it would take it 0.800+.\n\nNotebook: https://www.kaggle.com/abdessalemboukil/saint-training-inference-guide-39th-solution \n\nThanks for the awesome competition!",
    "1147284": "Congrats and thanks for sharing",
    "1147886": "Congrats on your result and thank you for sharing a well explained notebook."
  },
  "source": "meta"
}