{
  "id": 209513,
  "title": "Updating User Last N Sequence for Transformer Model in O(1) Time Complexity",
  "url": "/competitions/riiid-test-answer-prediction/discussion/209513",
  "author_name": "william.wu",
  "post_date": "2021-01-07T19:12:01.967000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I saw many public notebooks are using <code>np.append</code> to update the user's last N sequence for SAKT/RAINT, the complexity of this operation is O(N), where N is the current sequence length, Instead of using <code>np.append</code>, we can use <code>deque</code>. Another benefit of <code>deque</code> is that it has a <code>maxlen</code> parameter, by setting this parameter to the length of the <code>MAX_SEQ</code> of our model, we don't need to trim it manually. I share my implementation here, hope it's useful.</p>\n<pre><code>class _UserFeats(object):\n    __slots__ = [\n        \"last_ques\", # last questions\n        \"last_ets\",  # last elapsed_times\n        \"last_lts\",  # last lag times\n        \"last_ans\",  # last answers\n    ]\n\n    def __init__(self, max_len, last_seq=None):\n        self.last_ques = deque(maxlen=max_len)\n        self.last_ets = deque(maxlen=max_len)\n        self.last_lts = deque(maxlen=max_len)\n        self.last_ans = deque(maxlen=max_len)\n\n        if last_seq is not None:\n            self.init(last_seq)\n\n    def init(self, last_seq):\n        (\n            last_ques,\n            last_ets,\n            last_lts,\n            last_ans,\n        ) = last_seq\n        self.last_ques.extend(last_ques)\n        self.last_ets.extend(last_ets)\n        self.last_lts.extend(last_lts)\n        self.last_ans.extend(last_ans)\n\n    def add_ques(self, ques):\n        self.last_ques.append(ques)\n\n    def add_et(self, et):\n        self.last_ets.append(et)\n\n    def add_lt(self, lt):\n        self.last_lts.append(lt)\n\n    def add_ans(self, ans):\n        self.last_ans.append(ans)\n\n    def __repr__(self):\n        return \"_UserFeats: last_ques={}, last_ets={}, last_lts={}, last_ans={}\".format(\n            self.last_ques,\n            self.last_ets,\n            self.last_lts,\n            self.last_ans,\n        )\n</code></pre>\n<p>You can store users' features in <code>defaultdict(lambda: _UserFeats)</code> format, and you can loop the <code>test_df</code> to generate the batch inputs for the Transformer model as following:</p>\n<pre><code>user_feats = defaultdict(lambda x: _UserFeats())\n\n# Inint with last N seq of each user in train.\n\n\nques_df = test_df[test_df[\"content_type_id\"] == 0]\nn_ques_rows = ques_df.shape[0]\nbatch_ques = np.zeros((n_ques_rows, input_seq_len), dtype=int)\nbatch_parts = np.zeros( (n_ques_rows, input_seq_len), dtype=int)\n# ...\n\nk = 0\nfor row in ques_df.itertuples():\n    u_feats = self.user_feats[u_id]\n    ques_len = len(u_feats.last_ques)\n    batch_ques[k, :ques_len] = u_feats.last_ques\n    batch_ques[k, ques_len] = q_id\n    # ...\n    k += 1\n\ninputs = {\n    \"ques\": batch_ques,\n    \"parts\": batch_parts,\n    # ...\n}\n\nouts = model(**inputs)\n</code></pre>",
  "messages": [
    {
      "id": 1143137,
      "postDate": "2021-01-07T19:12:01.967Z",
      "content": "<p>I saw many public notebooks are using <code>np.append</code> to update the user's last N sequence for SAKT/RAINT, the complexity of this operation is O(N), where N is the current sequence length, Instead of using <code>np.append</code>, we can use <code>deque</code>. Another benefit of <code>deque</code> is that it has a <code>maxlen</code> parameter, by setting this parameter to the length of the <code>MAX_SEQ</code> of our model, we don't need to trim it manually. I share my implementation here, hope it's useful.</p>\n<pre><code>class _UserFeats(object):\n    __slots__ = [\n        \"last_ques\", # last questions\n        \"last_ets\",  # last elapsed_times\n        \"last_lts\",  # last lag times\n        \"last_ans\",  # last answers\n    ]\n\n    def __init__(self, max_len, last_seq=None):\n        self.last_ques = deque(maxlen=max_len)\n        self.last_ets = deque(maxlen=max_len)\n        self.last_lts = deque(maxlen=max_len)\n        self.last_ans = deque(maxlen=max_len)\n\n        if last_seq is not None:\n            self.init(last_seq)\n\n    def init(self, last_seq):\n        (\n            last_ques,\n            last_ets,\n            last_lts,\n            last_ans,\n        ) = last_seq\n        self.last_ques.extend(last_ques)\n        self.last_ets.extend(last_ets)\n        self.last_lts.extend(last_lts)\n        self.last_ans.extend(last_ans)\n\n    def add_ques(self, ques):\n        self.last_ques.append(ques)\n\n    def add_et(self, et):\n        self.last_ets.append(et)\n\n    def add_lt(self, lt):\n        self.last_lts.append(lt)\n\n    def add_ans(self, ans):\n        self.last_ans.append(ans)\n\n    def __repr__(self):\n        return \"_UserFeats: last_ques={}, last_ets={}, last_lts={}, last_ans={}\".format(\n            self.last_ques,\n            self.last_ets,\n            self.last_lts,\n            self.last_ans,\n        )\n</code></pre>\n<p>You can store users' features in <code>defaultdict(lambda: _UserFeats)</code> format, and you can loop the <code>test_df</code> to generate the batch inputs for the Transformer model as following:</p>\n<pre><code>user_feats = defaultdict(lambda x: _UserFeats())\n\n# Inint with last N seq of each user in train.\n\n\nques_df = test_df[test_df[\"content_type_id\"] == 0]\nn_ques_rows = ques_df.shape[0]\nbatch_ques = np.zeros((n_ques_rows, input_seq_len), dtype=int)\nbatch_parts = np.zeros( (n_ques_rows, input_seq_len), dtype=int)\n# ...\n\nk = 0\nfor row in ques_df.itertuples():\n    u_feats = self.user_feats[u_id]\n    ques_len = len(u_feats.last_ques)\n    batch_ques[k, :ques_len] = u_feats.last_ques\n    batch_ques[k, ques_len] = q_id\n    # ...\n    k += 1\n\ninputs = {\n    \"ques\": batch_ques,\n    \"parts\": batch_parts,\n    # ...\n}\n\nouts = model(**inputs)\n</code></pre>",
      "rawMarkdown": "I saw many public notebooks are using `np.append` to update the user's last N sequence for SAKT/RAINT, the complexity of this operation is O(N), where N is the current sequence length, Instead of using `np.append`, we can use `deque`. Another benefit of `deque` is that it has a `maxlen` parameter, by setting this parameter to the length of the `MAX_SEQ` of our model, we don't need to trim it manually. I share my implementation here, hope it's useful.\n\n``` Python\nclass _UserFeats(object):\n    __slots__ = [\n        \"last_ques\", # last questions\n        \"last_ets\",  # last elapsed_times\n        \"last_lts\",  # last lag times\n        \"last_ans\",  # last answers\n    ]\n\n    def __init__(self, max_len, last_seq=None):\n        self.last_ques = deque(maxlen=max_len)\n        self.last_ets = deque(maxlen=max_len)\n        self.last_lts = deque(maxlen=max_len)\n        self.last_ans = deque(maxlen=max_len)\n\n        if last_seq is not None:\n            self.init(last_seq)\n\n    def init(self, last_seq):\n        (\n            last_ques,\n            last_ets,\n            last_lts,\n            last_ans,\n        ) = last_seq\n        self.last_ques.extend(last_ques)\n        self.last_ets.extend(last_ets)\n        self.last_lts.extend(last_lts)\n        self.last_ans.extend(last_ans)\n\n    def add_ques(self, ques):\n        self.last_ques.append(ques)\n\n    def add_et(self, et):\n        self.last_ets.append(et)\n\n    def add_lt(self, lt):\n        self.last_lts.append(lt)\n\n    def add_ans(self, ans):\n        self.last_ans.append(ans)\n\n    def __repr__(self):\n        return \"_UserFeats: last_ques={}, last_ets={}, last_lts={}, last_ans={}\".format(\n            self.last_ques,\n            self.last_ets,\n            self.last_lts,\n            self.last_ans,\n        )\n```\nYou can store users' features in `defaultdict(lambda: _UserFeats)` format, and you can loop the `test_df` to generate the batch inputs for the Transformer model as following:\n\n```Python\nuser_feats = defaultdict(lambda x: _UserFeats())\n\n# Inint with last N seq of each user in train.\n\n\nques_df = test_df[test_df[\"content_type_id\"] == 0]\nn_ques_rows = ques_df.shape[0]\nbatch_ques = np.zeros((n_ques_rows, input_seq_len), dtype=int)\nbatch_parts = np.zeros( (n_ques_rows, input_seq_len), dtype=int)\n# ...\n\nk = 0\nfor row in ques_df.itertuples():\n    u_feats = self.user_feats[u_id]\n    ques_len = len(u_feats.last_ques)\n    batch_ques[k, :ques_len] = u_feats.last_ques\n    batch_ques[k, ques_len] = q_id\n    # ...\n    k += 1\n\ninputs = {\n    \"ques\": batch_ques,\n    \"parts\": batch_parts,\n    # ...\n}\n\nouts = model(**inputs)\n```",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1143137": "I saw many public notebooks are using `np.append` to update the user's last N sequence for SAKT/RAINT, the complexity of this operation is O(N), where N is the current sequence length, Instead of using `np.append`, we can use `deque`. Another benefit of `deque` is that it has a `maxlen` parameter, by setting this parameter to the length of the `MAX_SEQ` of our model, we don't need to trim it manually. I share my implementation here, hope it's useful.\n\n``` Python\nclass _UserFeats(object):\n    __slots__ = [\n        \"last_ques\", # last questions\n        \"last_ets\",  # last elapsed_times\n        \"last_lts\",  # last lag times\n        \"last_ans\",  # last answers\n    ]\n\n    def __init__(self, max_len, last_seq=None):\n        self.last_ques = deque(maxlen=max_len)\n        self.last_ets = deque(maxlen=max_len)\n        self.last_lts = deque(maxlen=max_len)\n        self.last_ans = deque(maxlen=max_len)\n\n        if last_seq is not None:\n            self.init(last_seq)\n\n    def init(self, last_seq):\n        (\n            last_ques,\n            last_ets,\n            last_lts,\n            last_ans,\n        ) = last_seq\n        self.last_ques.extend(last_ques)\n        self.last_ets.extend(last_ets)\n        self.last_lts.extend(last_lts)\n        self.last_ans.extend(last_ans)\n\n    def add_ques(self, ques):\n        self.last_ques.append(ques)\n\n    def add_et(self, et):\n        self.last_ets.append(et)\n\n    def add_lt(self, lt):\n        self.last_lts.append(lt)\n\n    def add_ans(self, ans):\n        self.last_ans.append(ans)\n\n    def __repr__(self):\n        return \"_UserFeats: last_ques={}, last_ets={}, last_lts={}, last_ans={}\".format(\n            self.last_ques,\n            self.last_ets,\n            self.last_lts,\n            self.last_ans,\n        )\n```\nYou can store users' features in `defaultdict(lambda: _UserFeats)` format, and you can loop the `test_df` to generate the batch inputs for the Transformer model as following:\n\n```Python\nuser_feats = defaultdict(lambda x: _UserFeats())\n\n# Inint with last N seq of each user in train.\n\n\nques_df = test_df[test_df[\"content_type_id\"] == 0]\nn_ques_rows = ques_df.shape[0]\nbatch_ques = np.zeros((n_ques_rows, input_seq_len), dtype=int)\nbatch_parts = np.zeros( (n_ques_rows, input_seq_len), dtype=int)\n# ...\n\nk = 0\nfor row in ques_df.itertuples():\n    u_feats = self.user_feats[u_id]\n    ques_len = len(u_feats.last_ques)\n    batch_ques[k, :ques_len] = u_feats.last_ques\n    batch_ques[k, ques_len] = q_id\n    # ...\n    k += 1\n\ninputs = {\n    \"ques\": batch_ques,\n    \"parts\": batch_parts,\n    # ...\n}\n\nouts = model(**inputs)\n```"
  }
}