{ "id": 209624, "title": "37th place solution (Transformer part)", "url": "/competitions/riiid-test-answer-prediction/discussion/209624", "author_name": "Dean", "post_date": "2021-01-08T03:53:14.441000", "votes": 25, "comment_count": 0, "views": 0, "content": "

First of all, I would like to thank oganizer and Kaggle for hosting the competition. In addition, thanks for @vk00st inviting us to merge as a team, and other teammates @unkoff @arvissu @syxuming, we have great team work.

\n

I also want to thank @manikanthr5 @wangsg @leadbest great SAKT starter notebook. I can’t implement the Transformer model by myself without these.

\n

I summarize the detail of our SAINT+ like model as follows.

\n

Input features

\n
    \n
  1. content_id
    2. \n
  2. response (answered_correctly)
    3. \n
  3. part
    4. \n
  4. prior_question_elapsed_time
    5. \n
  5. prior_question_had_explanation
    6. \n
  6. lag_time1 - convert lag time to seconds. if lag_time1 >= 300 than 300.
    7. \n
  7. lag_time2 - convert lag time to minutes. if lag_time2 >= 1440 than 300 (one day).
    8. \n
  8. lag_time3 - convert lag time to days. if lag_time3 >= 365 than 365 (one year).
    9. \n
\n

Lag time split to different time format boosting score around 0.003.

\n

Distinguish zero padding

\n\n
q_ = q_+1\npri_exp_ = pri_exp_+1\nres_ = qa_+1\n
\n

Transformer

\n
Encoder Input
\n\n
Decoder Input
\n\n
Parameter of Transformer
\n\n

Blending

\n\n

Result

\n\n

Sample code

\n", "messages": [ { "id": 1143765, "postDate": "2021-01-08T03:53:14.443Z", "content": "

First of all, I would like to thank oganizer and Kaggle for hosting the competition. In addition, thanks for @vk00st inviting us to merge as a team, and other teammates @unkoff @arvissu @syxuming, we have great team work.

\n

I also want to thank @manikanthr5 @wangsg @leadbest great SAKT starter notebook. I can’t implement the Transformer model by myself without these.

\n

I summarize the detail of our SAINT+ like model as follows.

\n

Input features

\n
    \n
  1. content_id
    2. \n
  2. response (answered_correctly)
    3. \n
  3. part
    4. \n
  4. prior_question_elapsed_time
    5. \n
  5. prior_question_had_explanation
    6. \n
  6. lag_time1 - convert lag time to seconds. if lag_time1 >= 300 than 300.
    7. \n
  7. lag_time2 - convert lag time to minutes. if lag_time2 >= 1440 than 300 (one day).
    8. \n
  8. lag_time3 - convert lag time to days. if lag_time3 >= 365 than 365 (one year).
    9. \n
\n

Lag time split to different time format boosting score around 0.003.

\n

Distinguish zero padding

\n\n
q_ = q_+1\npri_exp_ = pri_exp_+1\nres_ = qa_+1\n
\n

Transformer

\n
Encoder Input
\n\n
Decoder Input
\n\n
Parameter of Transformer
\n\n

Blending

\n\n

Result

\n\n

Sample code

\n", "rawMarkdown": "First of all, I would like to thank oganizer and Kaggle for hosting the competition. In addition, thanks for @vk00st inviting us to merge as a team, and other teammates @unkoff @arvissu @syxuming, we have great team work.\n\nI also want to thank @manikanthr5 @wangsg @leadbest great SAKT starter notebook. I can’t implement the Transformer model by myself without these.\n\nI summarize the detail of our SAINT+ like model as follows.\n\n\n## Input features\n\n1. content_id\n2. response (answered_correctly)\n3. part\n4. prior_question_elapsed_time\n5. prior_question_had_explanation\n6. lag_time1 - convert lag time to seconds. if lag_time1 >= 300 than 300.\n7. lag_time2 - convert lag time to minutes. if lag_time2 >= 1440 than 300 (one day).\n8. lag_time3 - convert lag time to days. if lag_time3 >= 365 than 365 (one year).\n\nLag time split to different time format boosting score around 0.003.\n\n## Distinguish zero padding\n* In order to distinguish zero padding and zero of **content_id**, I add one to **content_id** and **prior_question_had_explanation** and **response**. It will help model distinguish zero padding and features. It boost score around 0.003.\n\n```\nq_ = q_+1\npri_exp_ = pri_exp_+1\nres_ = qa_+1\n```\n### Transformer\n\n##### Encoder Input\n* question embedding\n* part embedding\n* position embedding\n* prior question had explanation embedding\n\n##### Decoder Input\n\n* position embedding\n* reponse embedding\n* prior elapsed time embedding\n* lag_time1 categorical embedding\n* lag_time2 categorical embedding\n* lag_time3 categorical embedding\n* Note that I tried categorical and continuous embedding in prior elapsed time and lag time. The performance of categorical embedding is better than continuous embedding.\n\n##### Parameter of Transformer\n* max sequence: 100\n* d model: 256 \n* number of layer of encoder: 2\n* number of layer of decoder: 2\n* batch size: 256\n* dropout: 0.1\n* learning rate: 5e-4 with AdamW\n\n### Blending\n\n* LGBM + Catboost + three Transformer\n* LGBM + two Transformer\n\n### Result\n\n* Finally, our team get LGBM public LB 0.794 and Transformer public LB 0.799.\n* Get public LB 0.804 with blending.\n\n### Sample code\n* [Training](https://www.kaggle.com/m10515009/saint-is-all-you-need-training-private-0-801)\n* [Inference](https://www.kaggle.com/m10515009/saint-is-all-you-need-inference-private-0-801)", "votes": 24 } ], "comments": [], "raw_markdown_by_id": { "1143765": "First of all, I would like to thank oganizer and Kaggle for hosting the competition. In addition, thanks for @vk00st inviting us to merge as a team, and other teammates @unkoff @arvissu @syxuming, we have great team work.\n\nI also want to thank @manikanthr5 @wangsg @leadbest great SAKT starter notebook. I can’t implement the Transformer model by myself without these.\n\nI summarize the detail of our SAINT+ like model as follows.\n\n\n## Input features\n\n1. content_id\n2. response (answered_correctly)\n3. part\n4. prior_question_elapsed_time\n5. prior_question_had_explanation\n6. lag_time1 - convert lag time to seconds. if lag_time1 >= 300 than 300.\n7. lag_time2 - convert lag time to minutes. if lag_time2 >= 1440 than 300 (one day).\n8. lag_time3 - convert lag time to days. if lag_time3 >= 365 than 365 (one year).\n\nLag time split to different time format boosting score around 0.003.\n\n## Distinguish zero padding\n* In order to distinguish zero padding and zero of **content_id**, I add one to **content_id** and **prior_question_had_explanation** and **response**. It will help model distinguish zero padding and features. It boost score around 0.003.\n\n```\nq_ = q_+1\npri_exp_ = pri_exp_+1\nres_ = qa_+1\n```\n### Transformer\n\n##### Encoder Input\n* question embedding\n* part embedding\n* position embedding\n* prior question had explanation embedding\n\n##### Decoder Input\n\n* position embedding\n* reponse embedding\n* prior elapsed time embedding\n* lag_time1 categorical embedding\n* lag_time2 categorical embedding\n* lag_time3 categorical embedding\n* Note that I tried categorical and continuous embedding in prior elapsed time and lag time. The performance of categorical embedding is better than continuous embedding.\n\n##### Parameter of Transformer\n* max sequence: 100\n* d model: 256 \n* number of layer of encoder: 2\n* number of layer of decoder: 2\n* batch size: 256\n* dropout: 0.1\n* learning rate: 5e-4 with AdamW\n\n### Blending\n\n* LGBM + Catboost + three Transformer\n* LGBM + two Transformer\n\n### Result\n\n* Finally, our team get LGBM public LB 0.794 and Transformer public LB 0.799.\n* Get public LB 0.804 with blending.\n\n### Sample code\n* [Training](https://www.kaggle.com/m10515009/saint-is-all-you-need-training-private-0-801)\n* [Inference](https://www.kaggle.com/m10515009/saint-is-all-you-need-inference-private-0-801)" } }