{
  "id": 209683,
  "title": "12th place solution NN part with code",
  "url": "/competitions/riiid-test-answer-prediction/discussion/209683",
  "author_name": "sakami",
  "post_date": "2021-01-08T08:16:00.888000",
  "votes": 58,
  "comment_count": 11,
  "views": 0,
  "content": "<p>First of all, I'd like to thank the organizers and kaggle team for hosting this great competition.<br>\nThis is our NN part solution. You can check our overall solution <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209635\" target=\"_blank\">here</a>.</p>\n<p>Our code is available <a href=\"https://github.com/sakami0000/kaggle_riiid\" target=\"_blank\">here</a>.</p>\n<ul>\n<li>Common</li>\n<li>SAINT-based model</li>\n<li>AKT-based model</li>\n</ul>\n<h2>Common</h2>\n<h4>Positional embedding</h4>\n<p>We used positional embedding in the same format as <a href=\"https://arxiv.org/abs/2006.15595\" target=\"_blank\">TUPE</a>.<br>\nIn this model, positional embedding is added after the dot product of Query and Key.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F62ccb03b74dd796e44b2c784f94736b5%2F2020-09-23%2023.32.18.png?generation=1610090691882141&amp;alt=media\" alt=\"\"></p>\n<p>In addition to that, we made the position ids of simultaneous samples same.</p>\n<pre><code>position_ids = (1 - (timestamp &lt;= timestamp.roll(1, dims=1)).long()).cumsum(dim=1)\n</code></pre>\n<h4>Attention score scaling</h4>\n<p>We scaled attention scores using lag time.</p>\n<p>Original attention scores are calculated by</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F690d548a2ab3f725dbabdd3c6c468884%2Fattention_scores_original.png?generation=1610091144516890&amp;alt=media\"></p>\n<p>We changed this to:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbabd99f1ed77fba4ceb1b1b6afdac776%2Fattention_scores_scaled.png?generation=1610091246220417&amp;alt=media\"></p>\n<p>where <code>function(t_ij)</code> is a monotonically increasing function for lag_time between sample i and j.<br>\nWe used following function:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F9cef341e24e2fbad1a40dab59ad8a6fa%2Fscaling_function.png?generation=1610093803120629&amp;alt=media\"></p>\n<p>where alpha is a hyper-parameter, which we set to 2.5.<br>\nThis improved our score about 0.01.</p>\n<h4>Sampling strategy</h4>\n<p>As is used in many public notebooks, our training samples are obtained by setting sliding windows.</p>\n<pre><code>for user_seq in tqdm(train_user_seqs.values(), desc='prepare train seqs'):\n    for start_idx in range(0, len(user_seq['content_id']), stride_size):\n        end_idx = start_idx + window_size\n\n        content_id = user_seq['content_id'][start_idx:end_idx]\n        part = user_seq['part'][start_idx:end_idx]\n        timestamp = user_seq['timestamp'][start_idx:end_idx]\n        elapsed_time = user_seq['elapsed_time'][start_idx:end_idx]\n        target = user_seq['answered_correctly'][start_idx:end_idx]\n</code></pre>\n<h2>SAINT-based model</h2>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2002.07033\" target=\"_blank\">https://arxiv.org/abs/2002.07033</a></li>\n<li>without lectures</li>\n<li>hidden_size: 256, encoder_layers: 8, decoder_layers: 8</li>\n<li>window_size: 100, 60</li>\n<li>encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li></ul></li>\n<li>decoder inputs:<ul>\n<li>response_ids</li>\n<li>lag_time</li>\n<li>elapsed_time</li></ul></li>\n</ul>\n<h2>AKT-based model</h2>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2007.12324\" target=\"_blank\">https://arxiv.org/abs/2007.12324</a></li>\n<li>without lectures</li>\n<li>add some self-attention layers at the beginning of the decoder</li>\n<li>mask simultaneous samples</li>\n<li>hidden_size: 256, encoder_layers: 8, decoder_layers: 8 (self-attention) + 4</li>\n<li>window_size: 100</li>\n<li>content encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li></ul></li>\n<li>knowledge encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li>\n<li>response_ids</li>\n<li>lag_time</li>\n<li>elapsed_time</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": 1144078,
      "postDate": "2021-01-08T08:16:00.890Z",
      "content": "<p>First of all, I'd like to thank the organizers and kaggle team for hosting this great competition.<br>\nThis is our NN part solution. You can check our overall solution <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209635\" target=\"_blank\">here</a>.</p>\n<p>Our code is available <a href=\"https://github.com/sakami0000/kaggle_riiid\" target=\"_blank\">here</a>.</p>\n<ul>\n<li>Common</li>\n<li>SAINT-based model</li>\n<li>AKT-based model</li>\n</ul>\n<h2>Common</h2>\n<h4>Positional embedding</h4>\n<p>We used positional embedding in the same format as <a href=\"https://arxiv.org/abs/2006.15595\" target=\"_blank\">TUPE</a>.<br>\nIn this model, positional embedding is added after the dot product of Query and Key.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F62ccb03b74dd796e44b2c784f94736b5%2F2020-09-23%2023.32.18.png?generation=1610090691882141&amp;alt=media\" alt=\"\"></p>\n<p>In addition to that, we made the position ids of simultaneous samples same.</p>\n<pre><code>position_ids = (1 - (timestamp &lt;= timestamp.roll(1, dims=1)).long()).cumsum(dim=1)\n</code></pre>\n<h4>Attention score scaling</h4>\n<p>We scaled attention scores using lag time.</p>\n<p>Original attention scores are calculated by</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F690d548a2ab3f725dbabdd3c6c468884%2Fattention_scores_original.png?generation=1610091144516890&amp;alt=media\"></p>\n<p>We changed this to:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbabd99f1ed77fba4ceb1b1b6afdac776%2Fattention_scores_scaled.png?generation=1610091246220417&amp;alt=media\"></p>\n<p>where <code>function(t_ij)</code> is a monotonically increasing function for lag_time between sample i and j.<br>\nWe used following function:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F9cef341e24e2fbad1a40dab59ad8a6fa%2Fscaling_function.png?generation=1610093803120629&amp;alt=media\"></p>\n<p>where alpha is a hyper-parameter, which we set to 2.5.<br>\nThis improved our score about 0.01.</p>\n<h4>Sampling strategy</h4>\n<p>As is used in many public notebooks, our training samples are obtained by setting sliding windows.</p>\n<pre><code>for user_seq in tqdm(train_user_seqs.values(), desc='prepare train seqs'):\n    for start_idx in range(0, len(user_seq['content_id']), stride_size):\n        end_idx = start_idx + window_size\n\n        content_id = user_seq['content_id'][start_idx:end_idx]\n        part = user_seq['part'][start_idx:end_idx]\n        timestamp = user_seq['timestamp'][start_idx:end_idx]\n        elapsed_time = user_seq['elapsed_time'][start_idx:end_idx]\n        target = user_seq['answered_correctly'][start_idx:end_idx]\n</code></pre>\n<h2>SAINT-based model</h2>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2002.07033\" target=\"_blank\">https://arxiv.org/abs/2002.07033</a></li>\n<li>without lectures</li>\n<li>hidden_size: 256, encoder_layers: 8, decoder_layers: 8</li>\n<li>window_size: 100, 60</li>\n<li>encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li></ul></li>\n<li>decoder inputs:<ul>\n<li>response_ids</li>\n<li>lag_time</li>\n<li>elapsed_time</li></ul></li>\n</ul>\n<h2>AKT-based model</h2>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2007.12324\" target=\"_blank\">https://arxiv.org/abs/2007.12324</a></li>\n<li>without lectures</li>\n<li>add some self-attention layers at the beginning of the decoder</li>\n<li>mask simultaneous samples</li>\n<li>hidden_size: 256, encoder_layers: 8, decoder_layers: 8 (self-attention) + 4</li>\n<li>window_size: 100</li>\n<li>content encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li></ul></li>\n<li>knowledge encoder inputs:<ul>\n<li>content_ids</li>\n<li>part</li>\n<li>response_ids</li>\n<li>lag_time</li>\n<li>elapsed_time</li></ul></li>\n</ul>",
      "rawMarkdown": "First of all, I'd like to thank the organizers and kaggle team for hosting this great competition.\nThis is our NN part solution. You can check our overall solution [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209635).\n\nOur code is available [here](https://github.com/sakami0000/kaggle_riiid).\n\n- Common\n- SAINT-based model\n- AKT-based model\n\n## Common\n\n#### Positional embedding\n\nWe used positional embedding in the same format as [TUPE](https://arxiv.org/abs/2006.15595).\nIn this model, positional embedding is added after the dot product of Query and Key.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F62ccb03b74dd796e44b2c784f94736b5%2F2020-09-23%2023.32.18.png?generation=1610090691882141&alt=media)\n\nIn addition to that, we made the position ids of simultaneous samples same.\n\n```\nposition_ids = (1 - (timestamp <= timestamp.roll(1, dims=1)).long()).cumsum(dim=1)\n```\n\n#### Attention score scaling\n\nWe scaled attention scores using lag time.\n\nOriginal attention scores are calculated by\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F690d548a2ab3f725dbabdd3c6c468884%2Fattention_scores_original.png?generation=1610091144516890&alt=media\" width=180>\n\nWe changed this to:\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbabd99f1ed77fba4ceb1b1b6afdac776%2Fattention_scores_scaled.png?generation=1610091246220417&alt=media\" width=280>\n\nwhere `function(t_ij)` is a monotonically increasing function for lag_time between sample i and j.\nWe used following function:\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F9cef341e24e2fbad1a40dab59ad8a6fa%2Fscaling_function.png?generation=1610093803120629&alt=media\" width=400>\n\nwhere alpha is a hyper-parameter, which we set to 2.5.\nThis improved our score about 0.01.\n\n#### Sampling strategy\n\nAs is used in many public notebooks, our training samples are obtained by setting sliding windows.\n\n```\nfor user_seq in tqdm(train_user_seqs.values(), desc='prepare train seqs'):\n    for start_idx in range(0, len(user_seq['content_id']), stride_size):\n        end_idx = start_idx + window_size\n\n        content_id = user_seq['content_id'][start_idx:end_idx]\n        part = user_seq['part'][start_idx:end_idx]\n        timestamp = user_seq['timestamp'][start_idx:end_idx]\n        elapsed_time = user_seq['elapsed_time'][start_idx:end_idx]\n        target = user_seq['answered_correctly'][start_idx:end_idx]\n```\n\n## SAINT-based model\n\n- https://arxiv.org/abs/2002.07033\n- without lectures\n- hidden_size: 256, encoder_layers: 8, decoder_layers: 8\n- window_size: 100, 60\n- encoder inputs:\n  - content_ids\n  - part\n- decoder inputs:\n  - response_ids\n  - lag_time\n  - elapsed_time\n\n## AKT-based model\n\n- https://arxiv.org/abs/2007.12324\n- without lectures\n- add some self-attention layers at the beginning of the decoder\n- mask simultaneous samples\n- hidden_size: 256, encoder_layers: 8, decoder_layers: 8 (self-attention) + 4\n- window_size: 100\n- content encoder inputs:\n  - content_ids\n  - part\n- knowledge encoder inputs:\n  - content_ids\n  - part\n  - response_ids\n  - lag_time\n  - elapsed_time",
      "votes": 58
    },
    {
      "id": 1146132,
      "postDate": "2021-01-09T14:59:30.903Z",
      "content": "<p>Congratulations !</p>",
      "rawMarkdown": "Congratulations !",
      "votes": 1
    },
    {
      "id": 1145104,
      "postDate": "2021-01-08T22:19:18.023Z",
      "content": "<p>Congratulations to new grandmasters pocket and owruby!</p>",
      "rawMarkdown": "Congratulations to new grandmasters pocket and owruby!",
      "votes": 1
    },
    {
      "id": 1145035,
      "postDate": "2021-01-08T20:26:28.820Z",
      "content": "<p><a href=\"https://www.kaggle.com/sakami\" target=\"_blank\">@sakami</a> Congrats and thank you for sharing. Can you share score of your best single model ?</p>",
      "rawMarkdown": "@sakami Congrats and thank you for sharing. Can you share score of your best single model ?",
      "votes": 1,
      "replies": [
        {
          "id": 1146071,
          "postDate": "2021-01-09T14:19:37.667Z",
          "content": "<p>Thank you!<br>\nSAINT: public 0.807, private 0.810<br>\nAKT: public 0.808, private 0.810</p>",
          "rawMarkdown": "Thank you!\nSAINT: public 0.807, private 0.810\nAKT: public 0.808, private 0.810",
          "votes": 1
        }
      ]
    },
    {
      "id": 1144982,
      "postDate": "2021-01-08T19:46:03.870Z",
      "content": "<p>Congrats and thank you for sharing! I always get inspired by your NN skills.</p>",
      "rawMarkdown": "Congrats and thank you for sharing! I always get inspired by your NN skills.",
      "votes": 1,
      "replies": [
        {
          "id": 1145018,
          "postDate": "2021-01-08T20:11:28.303Z",
          "content": "<p>Thank you! :)</p>",
          "rawMarkdown": "Thank you! :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 1159010,
      "postDate": "2021-01-19T01:30:12.103Z",
      "content": "<p>Our AKT Single scored public 0.813 / private 0.815<br>\nby changing window_size to 512. </p>",
      "rawMarkdown": "Our AKT Single scored public 0.813 / private 0.815\nby changing window_size to 512. ",
      "votes": 2
    },
    {
      "id": 1144375,
      "postDate": "2021-01-08T12:30:15.787Z",
      "content": "<p>Thanks for the clear &amp; detailed explanation, it is really enjoyable to read.</p>",
      "rawMarkdown": "Thanks for the clear & detailed explanation, it is really enjoyable to read.",
      "votes": 2
    },
    {
      "id": 1144303,
      "postDate": "2021-01-08T11:19:19.973Z",
      "content": "<p>Congrats on gold medal <a href=\"https://www.kaggle.com/sakami\" target=\"_blank\">@sakami</a> and team and thanks for sharing code and solution </p>",
      "rawMarkdown": "Congrats on gold medal @sakami and team and thanks for sharing code and solution ",
      "votes": 2
    },
    {
      "id": 1146898,
      "postDate": "2021-01-10T06:33:29.950Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 1146909,
          "postDate": "2021-01-10T06:47:05.307Z",
          "content": "<p>Yes, we used masking.<br>\nIn this competition, attention must be masked with an upper triangle matrix so that all samples can't refer to future samples. (<a href=\"https://github.com/sakami0000/kaggle_riiid/blob/main/src/models/saint.py#L509-L512\" target=\"_blank\">https://github.com/sakami0000/kaggle_riiid/blob/main/src/models/saint.py#L509-L512</a>)</p>",
          "rawMarkdown": "Yes, we used masking.\nIn this competition, attention must be masked with an upper triangle matrix so that all samples can't refer to future samples. (https://github.com/sakami0000/kaggle_riiid/blob/main/src/models/saint.py#L509-L512)"
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1146132,
      "author_name": "Alex",
      "author_url": "",
      "post_date": "2021-01-09T14:59:30.903000",
      "content": "<p>Congratulations !</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1145104,
      "author_name": "CoreyJamesLevinson",
      "author_url": "",
      "post_date": "2021-01-08T22:19:18.023000",
      "content": "<p>Congratulations to new grandmasters pocket and owruby!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1145035,
      "author_name": "Abdur Rehman",
      "author_url": "",
      "post_date": "2021-01-08T20:26:28.820000",
      "content": "<p><a href=\"https://www.kaggle.com/sakami\" target=\"_blank\">@sakami</a> Congrats and thank you for sharing. Can you share score of your best single model ?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1146071,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2021-01-09T14:19:37.667000",
          "content": "<p>Thank you!<br>\nSAINT: public 0.807, private 0.810<br>\nAKT: public 0.808, private 0.810</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1144982,
      "author_name": "u++",
      "author_url": "",
      "post_date": "2021-01-08T19:46:03.870000",
      "content": "<p>Congrats and thank you for sharing! I always get inspired by your NN skills.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1145018,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2021-01-08T20:11:28.303000",
          "content": "<p>Thank you! :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1159010,
      "author_name": "sakami",
      "author_url": "",
      "post_date": "2021-01-19T01:30:12.103000",
      "content": "<p>Our AKT Single scored public 0.813 / private 0.815<br>\nby changing window_size to 512. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1144375,
      "author_name": "Theo Viel",
      "author_url": "",
      "post_date": "2021-01-08T12:30:15.787000",
      "content": "<p>Thanks for the clear &amp; detailed explanation, it is really enjoyable to read.</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1144303,
      "author_name": "KhanhVD",
      "author_url": "",
      "post_date": "2021-01-08T11:19:19.973000",
      "content": "<p>Congrats on gold medal <a href=\"https://www.kaggle.com/sakami\" target=\"_blank\">@sakami</a> and team and thanks for sharing code and solution </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 1146898,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-01-10T06:33:29.950000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 1146909,
          "author_name": "sakami",
          "author_url": "",
          "post_date": "2021-01-10T06:47:05.307000",
          "content": "<p>Yes, we used masking.<br>\nIn this competition, attention must be masked with an upper triangle matrix so that all samples can't refer to future samples. (<a href=\"https://github.com/sakami0000/kaggle_riiid/blob/main/src/models/saint.py#L509-L512\" target=\"_blank\">https://github.com/sakami0000/kaggle_riiid/blob/main/src/models/saint.py#L509-L512</a>)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1144078": "First of all, I'd like to thank the organizers and kaggle team for hosting this great competition.\nThis is our NN part solution. You can check our overall solution [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209635).\n\nOur code is available [here](https://github.com/sakami0000/kaggle_riiid).\n\n- Common\n- SAINT-based model\n- AKT-based model\n\n## Common\n\n#### Positional embedding\n\nWe used positional embedding in the same format as [TUPE](https://arxiv.org/abs/2006.15595).\nIn this model, positional embedding is added after the dot product of Query and Key.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F62ccb03b74dd796e44b2c784f94736b5%2F2020-09-23%2023.32.18.png?generation=1610090691882141&alt=media)\n\nIn addition to that, we made the position ids of simultaneous samples same.\n\n```\nposition_ids = (1 - (timestamp <= timestamp.roll(1, dims=1)).long()).cumsum(dim=1)\n```\n\n#### Attention score scaling\n\nWe scaled attention scores using lag time.\n\nOriginal attention scores are calculated by\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F690d548a2ab3f725dbabdd3c6c468884%2Fattention_scores_original.png?generation=1610091144516890&alt=media\" width=180>\n\nWe changed this to:\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2Fbabd99f1ed77fba4ceb1b1b6afdac776%2Fattention_scores_scaled.png?generation=1610091246220417&alt=media\" width=280>\n\nwhere `function(t_ij)` is a monotonically increasing function for lag_time between sample i and j.\nWe used following function:\n\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2133818%2F9cef341e24e2fbad1a40dab59ad8a6fa%2Fscaling_function.png?generation=1610093803120629&alt=media\" width=400>\n\nwhere alpha is a hyper-parameter, which we set to 2.5.\nThis improved our score about 0.01.\n\n#### Sampling strategy\n\nAs is used in many public notebooks, our training samples are obtained by setting sliding windows.\n\n```\nfor user_seq in tqdm(train_user_seqs.values(), desc='prepare train seqs'):\n    for start_idx in range(0, len(user_seq['content_id']), stride_size):\n        end_idx = start_idx + window_size\n\n        content_id = user_seq['content_id'][start_idx:end_idx]\n        part = user_seq['part'][start_idx:end_idx]\n        timestamp = user_seq['timestamp'][start_idx:end_idx]\n        elapsed_time = user_seq['elapsed_time'][start_idx:end_idx]\n        target = user_seq['answered_correctly'][start_idx:end_idx]\n```\n\n## SAINT-based model\n\n- https://arxiv.org/abs/2002.07033\n- without lectures\n- hidden_size: 256, encoder_layers: 8, decoder_layers: 8\n- window_size: 100, 60\n- encoder inputs:\n  - content_ids\n  - part\n- decoder inputs:\n  - response_ids\n  - lag_time\n  - elapsed_time\n\n## AKT-based model\n\n- https://arxiv.org/abs/2007.12324\n- without lectures\n- add some self-attention layers at the beginning of the decoder\n- mask simultaneous samples\n- hidden_size: 256, encoder_layers: 8, decoder_layers: 8 (self-attention) + 4\n- window_size: 100\n- content encoder inputs:\n  - content_ids\n  - part\n- knowledge encoder inputs:\n  - content_ids\n  - part\n  - response_ids\n  - lag_time\n  - elapsed_time",
    "1146132": "Congratulations !",
    "1145104": "Congratulations to new grandmasters pocket and owruby!",
    "1145035": "@sakami Congrats and thank you for sharing. Can you share score of your best single model ?",
    "1144982": "Congrats and thank you for sharing! I always get inspired by your NN skills.",
    "1159010": "Our AKT Single scored public 0.813 / private 0.815\nby changing window_size to 512. ",
    "1144375": "Thanks for the clear & detailed explanation, it is really enjoyable to read.",
    "1144303": "Congrats on gold medal @sakami and team and thanks for sharing code and solution ",
    "1146898": ""
  }
}