{
  "id": 204867,
  "title": "Debugging the PyTorch transformer model",
  "url": "/competitions/riiid-test-answer-prediction/discussion/204867",
  "author_name": "",
  "post_date": "2020-12-17T08:11:33.249323600Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p><a href=\"https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged\" target=\"_blank\">https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged</a></p>\n<p>I tried to debug one of the earlier transformer model in this competition, however, I found that the inconsistency of the training vs putting the model in the <code>iter_env()</code>.</p>\n<p>I fixed several major issues in the original kernel: <a href=\"https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers\" target=\"_blank\">https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers</a></p>\n<ol>\n<li>The sorting by <code>user_id</code> should not be applied otherwise messing up the <code>iter_env()</code>'s order.</li>\n<li>Apparently the max length of a user interaction is hard-coded in the feature generation code and now it is fixed.</li>\n<li>An inference part is added, with every iteration the user's interaction updated into the dictionary. However, I am not sure what is wrong with it as it gives somewhat unwanted result.</li>\n</ol>",
  "messages": [
    {
      "id": "1116510",
      "postDate": "12/17/2020 08:11:33",
      "content": "<p><a href=\"https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged\" target=\"_blank\">https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged</a></p>\n<p>I tried to debug one of the earlier transformer model in this competition, however, I found that the inconsistency of the training vs putting the model in the <code>iter_env()</code>.</p>\n<p>I fixed several major issues in the original kernel: <a href=\"https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers\" target=\"_blank\">https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers</a></p>\n<ol>\n<li>The sorting by <code>user_id</code> should not be applied otherwise messing up the <code>iter_env()</code>'s order.</li>\n<li>Apparently the max length of a user interaction is hard-coded in the feature generation code and now it is fixed.</li>\n<li>An inference part is added, with every iteration the user's interaction updated into the dictionary. However, I am not sure what is wrong with it as it gives somewhat unwanted result.</li>\n</ol>",
      "rawMarkdown": "https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged\n\nI tried to debug one of the earlier transformer model in this competition, however, I found that the inconsistency of the training vs putting the model in the `iter_env()`.\n\nI fixed several major issues in the original kernel: https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers\n\n1. The sorting by `user_id` should not be applied otherwise messing up the `iter_env()`'s order.\n2. Apparently the max length of a user interaction is hard-coded in the feature generation code and now it is fixed.\n3. An inference part is added, with every iteration the user's interaction updated into the dictionary. However, I am not sure what is wrong with it as it gives somewhat unwanted result.",
      "votes": null
    },
    {
      "id": "1117081",
      "postDate": "12/17/2020 17:56:15",
      "content": "<p>I believe he mentions it is a WIP and has many bugs to be resolved:<br>\n\"It's a WIP and is based on my understanding. So for sure, there might be bugs in it\"</p>",
      "rawMarkdown": "I believe he mentions it is a WIP and has many bugs to be resolved:\n\"It's a WIP and is based on my understanding. So for sure, there might be bugs in it\"",
      "votes": null
    },
    {
      "id": "1117294",
      "postDate": "12/17/2020 22:20:47",
      "content": "<p>I know. I am not stating <a href=\"https://www.kaggle.com/adityaecdrid\" target=\"_blank\">@adityaecdrid</a> 's coding having bugs in a negative or criticizing way, I actually learned a lot from his code about the EDA during debugging.</p>",
      "rawMarkdown": "I know. I am not stating @adityaecdrid 's coding having bugs in a negative or criticizing way, I actually learned a lot from his code about the EDA during debugging.",
      "votes": null
    },
    {
      "id": "1117418",
      "postDate": "12/18/2020 03:06:04",
      "content": "<p>oh ok..BTW I did go thru' your code as well and upvoted. It is very informative. Thanks!!</p>",
      "rawMarkdown": "oh ok..BTW I did go thru' your code as well and upvoted. It is very informative. Thanks!!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1117081,
      "author_name": "allohvk",
      "author_url": "",
      "post_date": "12/17/2020 17:56:15",
      "content": "<p>I believe he mentions it is a WIP and has many bugs to be resolved:<br>\n\"It's a WIP and is based on my understanding. So for sure, there might be bugs in it\"</p>",
      "votes": null,
      "replies": [
        {
          "id": 1117294,
          "author_name": "scaomath",
          "author_url": "",
          "post_date": "12/17/2020 22:20:47",
          "content": "<p>I know. I am not stating <a href=\"https://www.kaggle.com/adityaecdrid\" target=\"_blank\">@adityaecdrid</a> 's coding having bugs in a negative or criticizing way, I actually learned a lot from his code about the EDA during debugging.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1117418,
          "author_name": "allohvk",
          "author_url": "",
          "post_date": "12/18/2020 03:06:04",
          "content": "<p>oh ok..BTW I did go thru' your code as well and upvoted. It is very informative. Thanks!!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1116510": "https://www.kaggle.com/scaomath/riiid-demystifying-transformers-in-pt-debugged\n\nI tried to debug one of the earlier transformer model in this competition, however, I found that the inconsistency of the training vs putting the model in the `iter_env()`.\n\nI fixed several major issues in the original kernel: https://www.kaggle.com/adityaecdrid/pytorch-demystifying-transformers\n\n1. The sorting by `user_id` should not be applied otherwise messing up the `iter_env()`'s order.\n2. Apparently the max length of a user interaction is hard-coded in the feature generation code and now it is fixed.\n3. An inference part is added, with every iteration the user's interaction updated into the dictionary. However, I am not sure what is wrong with it as it gives somewhat unwanted result.",
    "1117081": "I believe he mentions it is a WIP and has many bugs to be resolved:\n\"It's a WIP and is based on my understanding. So for sure, there might be bugs in it\"",
    "1117294": "I know. I am not stating @adityaecdrid 's coding having bugs in a negative or criticizing way, I actually learned a lot from his code about the EDA during debugging.",
    "1117418": "oh ok..BTW I did go thru' your code as well and upvoted. It is very informative. Thanks!!"
  },
  "source": "meta"
}