{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Transformer Encoder model"},{"metadata":{},"cell_type":"markdown","source":"In this notebook I will share a chunk of my solution, using the encoder of a Transformer. My solution isn't exactly like this but very close, I'm still trying to improve it. ~~~This notebook will only cover the training part, but it won't be hard to adpat the inference part. If it gets attention ;) I will add the inference portion of the code.~~\n\nI'll try to make things easy to understand, meaning that I will copy-paste almost everything of the code from public resources where you can refer to. Also, the little code I'll develop will be both simple and scalable for you to be able to make something great from it.\n\nIf you have any suggestions I will be grateful :).\n\nThe majority of the model stuff is already implemented in multiple frameworks. We will use here some code in Tensorflow from [the official Transformer tutorial](https://www.tensorflow.org/tutorials/text/transformer).\n\nWe are taking into account the information given in our host's papers ([https://arxiv.org/abs/2002.07033](https://arxiv.org/abs/2002.07033), [https://arxiv.org/abs/2010.12042](https://arxiv.org/abs/2010.12042)), implementing a light version of their model. \n\nWe will use embeddings for the following columns: \n\n- answered_correctly\n- content_id\n- task_container_id\n- prior_elapsed_time (this one as continuous encoding)\n- part\n\nAnd also we will add some information about the position in a fixed sinusoidal fashion.\n\n\nEDIT: As promised, I'm adding the inference portion of the code. Thanks for paying attention to my work."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dtype = {\n    'answered_correctly': 'int8',\n    # 'row_id': 'int64',\n    # 'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n    # 'content_type_id': 'int8',\n    'task_container_id': 'int16',\n    # 'user_answer': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    # 'prior_question_had_explanation': 'boolean'\n}\n\ndtype_questions = {\n    'question_id': 'int32',\n    # 'bundle_id': 'int32',\n    # 'correct_answer': 'int8',\n    'part': 'int8',\n    # 'tags': 'object',\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data"},{"metadata":{},"cell_type":"markdown","source":"We drop lectures, we are not using them here. Also we drop those interactions above 1500th. This can be changed if pleased."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\n    '/kaggle/input/riiid-test-answer-prediction/train.csv',\n    usecols=dtype.keys(),\n    dtype=dtype,\n    nrows=10**6\n)\ndf = df[df.answered_correctly!=-1]\ndf = df.groupby('user_id').head(1500)\n\nquestions = pd.read_csv(\n    '/kaggle/input/riiid-test-answer-prediction/questions.csv', \n    dtype=dtype_questions,\n    usecols=dtype_questions.keys(),\n    index_col='question_id'\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Basic transformation functions"},{"metadata":{},"cell_type":"markdown","source":"We'll need to add one unit to the future embedding columns which has zero in them, as we are going to use the zero as the padding token. Then some non-fancy things."},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform_questions(questions):\n  part_ids = questions.part.max()+1\n  return questions, part_ids\n\n\ndef transform_df(df, questions):\n    df['prior_question_elapsed_time'] = df['prior_question_elapsed_time'].fillna(0).astype(np.float32)/300000\n    content_ids = questions.index.max()+2\n    df = df.join(questions, on='content_id')\n    df['content_id'] += 1\n    df['task_container_id'] += 1\n    task_container_ids = 10001\n    return df, content_ids, task_container_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions, part_ids = transform_questions(questions)\ndf, content_ids, task_container_ids = transform_df(df, questions)\ndf.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Grouping rows by user id and putting them in a hashtable"},{"metadata":{},"cell_type":"markdown","source":"Maybe there's a better way to do this, please tell me!"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = {uid: u.drop(columns='user_id') for uid, u in df.groupby('user_id')}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data generator"},{"metadata":{},"cell_type":"markdown","source":"We need a way to feed the data in the model. I'm subclassing here tf.keras.utils.Sequence. I think it can be faster/better with tf.data but I don't have much experience handling DataFrames with it. You're welcome again to help me here!\n\nEDIT (inference addition): We are reusing the RiidSequence to store last windows_size users' interactions later in the inference part. For that, we need to add to new methods to it. We add a new property too to trace the column answered_correctly, so we can then know where to insert new values during the test iterations (prior_answered_correctly). Notice that we are now dealing directly with numpy arrays instead of DataFrames to inference faster. We transform DataFrames to numpy arrays on the fly when updating the users.\n\nAlso, we need something to shift our answer_correctly and something to make a time series from our user data. \n\nNotice that what we are doing is transforming a 2D tensor (one user interactions) to a 3D tensor by padding and then windowing the user. A user will serve then as a whole batch for the model.\n\nFor example, let's transform a random 2D tensor [5, 4] into a batch with window size of 3 [5, 3, 4]:\n\n1.- User interaction:\n```\narray([[0.32147631, 0.21259791, 0.95322964, 0.82263467],\n       [0.4595167 , 0.69536323, 0.64991227, 0.63098329],\n       [0.34540743, 0.25885106, 0.57766127, 0.38967852],\n       [0.26470928, 0.24314868, 0.86343302, 0.06009916],\n       [0.72159549, 0.96692088, 0.18604403, 0.46528594]])\n```       \n2.- Add pad:\n```\narray([[0.        , 0.        , 0.        , 0.        ],\n       [0.        , 0.        , 0.        , 0.        ],\n       [0.32147631, 0.21259791, 0.95322964, 0.82263467],\n       [0.4595167 , 0.69536323, 0.64991227, 0.63098329],\n       [0.34540743, 0.25885106, 0.57766127, 0.38967852],\n       [0.26470928, 0.24314868, 0.86343302, 0.06009916],\n       [0.72159549, 0.96692088, 0.18604403, 0.46528594]])\n```\n3.- Roll a window to get a batch:\n```\narray([[[0.        , 0.        , 0.        , 0.        ],\n        [0.        , 0.        , 0.        , 0.        ],\n        [0.32147631, 0.21259791, 0.95322964, 0.82263467]],\n\n       [[0.        , 0.        , 0.        , 0.        ],\n        [0.32147631, 0.21259791, 0.95322964, 0.82263467],\n        [0.4595167 , 0.69536323, 0.64991227, 0.63098329]],\n\n       [[0.32147631, 0.21259791, 0.95322964, 0.82263467],\n        [0.4595167 , 0.69536323, 0.64991227, 0.63098329],\n        [0.34540743, 0.25885106, 0.57766127, 0.38967852]],\n\n       [[0.4595167 , 0.69536323, 0.64991227, 0.63098329],\n        [0.34540743, 0.25885106, 0.57766127, 0.38967852],\n        [0.26470928, 0.24314868, 0.86343302, 0.06009916]],\n\n       [[0.34540743, 0.25885106, 0.57766127, 0.38967852],\n        [0.26470928, 0.24314868, 0.86343302, 0.06009916],\n        [0.72159549, 0.96692088, 0.18604403, 0.46528594]]])\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# just some stuff I ctrl C ctrl V from StackOverflow (with little changes)\n# [1,2,3,4] --- w = 2 --[[1,2], [2,3], [3,4]] but 2D to 3D\ndef rolling_window(a, w):\n    s0, s1 = a.strides\n    m, n = a.shape\n    return np.lib.stride_tricks.as_strided(\n        a, \n        shape=(m-w+1, w, n), \n        strides=(s0, s0, s1)\n    )\n\n\ndef make_time_series(x, windows_size):\n  x = np.pad(x, [[ windows_size-1, 0], [0, 0]], constant_values=0)\n  x = rolling_window(x, windows_size)\n  return x\n\n\ndef add_features_to_user(user):\n    # We add one to the column in order to have zeros as padding values\n    # Start Of Sentence (SOS) token will be 3. \n    user['answered_correctly'] = user['answered_correctly'].shift(fill_value=2)+1\n    return user\n\n\nclass RiidSequence(tf.keras.utils.Sequence):\n\n  def __init__(self, \n               users, \n               windows_size,\n               batch_size=256,\n               start=0,\n               end=None):\n    self.users = users # {'user_id': user_df, ...}\n    self.windows_size = windows_size\n    # to convert indices to our keys\n    self.mapper = dict(zip(range(len(users)), users.keys()))\n    # start and end to easy generate training and validation\n    self.start = start\n    self.end = end if end else len(users)\n    # To know where the answered_correctly_column is\n    self.answered_correctly_index = list(self.user_example().columns).index('answered_correctly')\n        \n  def __len__(self):\n    return self.end-self.start\n\n  def __getitem__(self, idx):\n    uid = self.mapper[idx+self.start]\n    user = self.users[uid].copy()\n    y = user['answered_correctly'].to_numpy().copy()\n    x = add_features_to_user(user)\n    return make_time_series(x, self.windows_size), y\n\n  def user_example(self):\n    \"\"\"Just to check what we have till' now.\"\"\"\n    uid = self.mapper[self.start]\n    return add_features_to_user(self.users[uid].copy())\n\n  # INFERENCE PART    \n  def get_user_for_inference(self, user_row):\n    \"\"\"Picks a new user row and concats it to previous interactions \n    if it was already stored.\n    \n    Maybe the biggest trick in the notebook is here. We reuse the user_id column to \n    insert the answered_correctly SOS token because we previously placed the column \n    there on purpose.\n    \n    After it, we roll that column and then crop it if it was bigger than the window\n    size, making the SOS token disapear if out of the sequence.\n    \n    If the sequence if shorter than the window size, then we pad it.\n    \"\"\"\n    uid = user_row[self.answered_correctly_index]\n    user_row[self.answered_correctly_index] = 2 # SOS token\n    user_row = user_row[np.newaxis, ...]\n    if uid in self.users:\n      x = np.concatenate([self.users[uid], user_row])\n      # same as in training, we need to add one!!!\n      x[:, self.answered_correctly_index] = np.roll(x[:, self.answered_correctly_index], 1) + 1\n    else:\n      x = user_row\n     \n    if x.shape[0] < self.windows_size:\n      return np.pad(x, [[self.windows_size-x.shape[0], 0], [0, 0]])\n    elif x.shape[0] > self.windows_size:\n      return x[-self.windows_size:]\n    else:\n      return x\n\n  def update_user(self, uid, user):\n    \"\"\"Concat the new user's interactions to the old ones if already stored.\"\"\"\n    if uid in self.users:\n      self.users[uid] = \\\n        np.concatenate([self.users[uid], user])[-self.windows_size:]\n    else:\n      self.users[uid] = user","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"RiidSequence(df, 64).user_example().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x, y = RiidSequence(df, 64)[0]\nx.shape, y.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So far so good."},{"metadata":{},"cell_type":"markdown","source":"## Modeling (fancy part)"},{"metadata":{},"cell_type":"markdown","source":"I will copy paste some stuff we need from the page I linked at the beginning ([https://www.tensorflow.org/tutorials/text/transformer](https://www.tensorflow.org/tutorials/text/transformer)). I will only clean comments for readability."},{"metadata":{"trusted":true},"cell_type":"code","source":"# POSITION ENCODING\n\ndef get_angles(pos, i, d_model):\n  angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))\n  return pos * angle_rates\n\n\ndef positional_encoding(position, d_model):\n  angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n                          np.arange(d_model)[np.newaxis, :],\n                          d_model)\n  angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])\n  angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])\n\n  pos_encoding = angle_rads[np.newaxis, ...]\n\n  return tf.cast(pos_encoding, dtype=tf.float32)\n\n# NN THINGS\ndef scaled_dot_product_attention(q, k, v, mask):\n  matmul_qk = tf.matmul(q, k, transpose_b=True)\n  dk = tf.cast(tf.shape(k)[-1], tf.float32)\n  scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n  if mask is not None:\n    scaled_attention_logits += (mask * -1e9)  \n  attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)\n  output = tf.matmul(attention_weights, v)\n  return output, attention_weights\n\n    \nclass MultiHeadAttention(tf.keras.layers.Layer):\n  def __init__(self, d_model, num_heads):\n    super(MultiHeadAttention, self).__init__()\n    self.num_heads = num_heads\n    self.d_model = d_model\n    \n    assert d_model % self.num_heads == 0\n    \n    self.depth = d_model // self.num_heads\n    \n    self.wq = tf.keras.layers.Dense(d_model)\n    self.wk = tf.keras.layers.Dense(d_model)\n    self.wv = tf.keras.layers.Dense(d_model)\n    \n    self.dense = tf.keras.layers.Dense(d_model)\n\n  def split_heads(self, x, batch_size):\n    x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n    return tf.transpose(x, perm=[0, 2, 1, 3])\n\n  def call(self, v, k, q, mask):\n    batch_size = tf.shape(q)[0]\n    q = self.wq(q)\n    k = self.wk(k)\n    v = self.wv(v)\n\n    q = self.split_heads(q, batch_size)\n    k = self.split_heads(k, batch_size)\n    v = self.split_heads(v, batch_size)\n\n    scaled_attention, attention_weights = scaled_dot_product_attention(\n        q, k, v, mask)\n\n    scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])\n\n    concat_attention = tf.reshape(scaled_attention, \n                                  (batch_size, -1, self.d_model))\n\n    output = self.dense(concat_attention)\n\n    return output, attention_weights\n\n\ndef point_wise_feed_forward_network(d_model, dff):\n  return tf.keras.Sequential([\n      tf.keras.layers.Dense(dff, activation='relu'),\n      tf.keras.layers.Dense(d_model)\n  ])\n\n\nclass EncoderLayer(tf.keras.layers.Layer):\n  def __init__(self, d_model, num_heads, dff, rate=0.1):\n    super(EncoderLayer, self).__init__()\n\n    self.mha = MultiHeadAttention(d_model, num_heads)\n    self.ffn = point_wise_feed_forward_network(d_model, dff)\n\n    self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n    self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n\n    self.dropout1 = tf.keras.layers.Dropout(rate)\n    self.dropout2 = tf.keras.layers.Dropout(rate)\n\n  def call(self, x, training, mask):\n\n    attn_output, _ = self.mha(x, x, x, mask) \n    attn_output = self.dropout1(attn_output, training=training)\n    out1 = self.layernorm1(x + attn_output) \n\n    ffn_output = self.ffn(out1)\n    ffn_output = self.dropout2(ffn_output, training=training)\n    out2 = self.layernorm2(out1 + ffn_output) \n\n    return out2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a little more tricky. To create padding mask with Multivariate Time Series oriented data (3D) we will slightly modify the function from the reference link."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_padding_mask(seqs):\n  # We mask only those vectors of the sequence in which we have all zeroes \n  # (this is more scalable for some situations).\n  mask = tf.cast(tf.reduce_all(tf.math.equal(seqs, 0), axis=-1), tf.float32)\n  return mask[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And now our model. We are using the functional API for simplicity. We could also subclass tf.keras.Model for it, which is cleaner. \n\nTo implement it, we will recall the columns we had before."},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = list(RiidSequence(df, 64).user_example().columns)\ncolumns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_series_model(\n        n_features,\n        content_ids,\n        task_container_ids,\n        part_ids,\n        windows_size=64,\n        d_model=24,\n        num_heads=4,\n        n_encoder_layers = 2\n    ):\n    # Input\n    inputs = tf.keras.Input(shape=(windows_size, n_features), name='inputs')\n    mask = create_padding_mask(inputs)\n    pos_enc = positional_encoding(windows_size, d_model)    \n    \n    # Divide branches\n    content_id = inputs[..., 0]\n    task_container_id = inputs[..., 1]\n    answered_correctly = inputs[..., 2]\n    elapsed_time = inputs[..., 3]\n    part = inputs[..., 4]\n    \n    # Create embeddings\n    content_embeddings = tf.keras.layers.Embedding(content_ids, d_model)(content_id)\n    task_embeddings = tf.keras.layers.Embedding(task_container_ids, d_model)(task_container_id)\n    answered_correctly_embeddings = tf.keras.layers.Embedding(4, d_model)(answered_correctly)\n    # Continuous! Only a learnable layer for it.\n    elapsed_time_embeddings = tf.keras.layers.Dense(d_model, use_bias=False)(elapsed_time)\n    part_embeddings = tf.keras.layers.Embedding(part_ids, d_model)(part)\n    \n    # Add embeddings\n    x = tf.keras.layers.Add()([\n        pos_enc,\n        content_embeddings,\n        task_embeddings,\n        answered_correctly_embeddings,\n        elapsed_time_embeddings,\n        part_embeddings,\n    ])\n\n    for _ in range(n_encoder_layers):\n        x = EncoderLayer(d_model=d_model, num_heads=num_heads, dff=d_model*4, rate=0.1)(x, mask=mask)\n\n    x = tf.keras.layers.GlobalAveragePooling1D()(x)\n    x = tf.keras.layers.Dropout(0.2)(x)    \n    output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(x)\n    return tf.keras.Model(inputs, output, name='model')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, we have adapted the Encoder to output only one value per sequence, in oposition to the original implementation which outputs an entire sequence per every input sequence. That's why we don't have to add a look ahead mask."},{"metadata":{},"cell_type":"markdown","source":"## TRAIN!"},{"metadata":{},"cell_type":"markdown","source":"In my implementation I have a learning rate scheduler. I removed it for simplicity. Feel free to add it.\n\nFor the example we will have a very little model just to show that it works!"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_idx = int(len(df)*0.8)\nwindows_size = 64\nepochs = 300\npatience = 2\nd_model = 32\nnum_heads = 4\nn_encoder_layers = 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s_train = RiidSequence(df, windows_size, start=0, end=train_idx)\ns_val = RiidSequence(df, windows_size, start=train_idx)\n\nn_features = s_train[0][0].shape[-1]\n\ntf.keras.backend.clear_session()\nmodel = get_series_model(\n        n_features,\n        content_ids,\n        task_container_ids,\n        part_ids,\n        windows_size=windows_size,\n        d_model=d_model,\n        num_heads=num_heads,\n        n_encoder_layers=n_encoder_layers\n    )\n\nmodel.compile(\n    optimizer='adam', \n    loss='binary_crossentropy', \n    metrics=[tf.keras.metrics.AUC(name='AUC'), tf.keras.metrics.BinaryAccuracy(name='acc')]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.keras.utils.plot_model(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(\n    s_train,\n    validation_data=s_val,\n    epochs=epochs,\n    workers=4,\n    shuffle=True,\n    use_multiprocessing=True,\n    callbacks=tf.keras.callbacks.EarlyStopping(patience=patience, monitor='val_AUC', mode='max', restore_best_weights=True),\n    verbose=1\n)\nmodel.save_weights('model.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference"},{"metadata":{},"cell_type":"markdown","source":"First, we load again the training data, but this time we load everything and then tail the users to the last window size interactions.\n\nWe apply the same operations than we did before to the training data in this new DataFrame, and also to the test. Also, we develop a logic to add the previous_answered_correctly. \n\nThe only thing we need to make sure is the user_id column in the test dfs is in the same place than the answered_correctly in the train df to make the trick after it in the RiidSequence."},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del s_val\ndel df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\n    '../input/riiid-test-answer-prediction/train.csv',\n    usecols=dtype.keys(),\n    dtype=dtype,\n    # nrows=10**6\n)\ndf = df[df.answered_correctly!=-1]\ndf = df.groupby('user_id').tail(windows_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df, _, _ = transform_df(df, questions)\ndf = {uid: u.drop(columns='user_id') for uid, u in df.groupby('user_id')}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns[columns.index('answered_correctly')] = 'user_id'\ncolumns = [c for c in columns if c not in questions.columns] + ['row_id']\ncolumns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for test, sample_prediction in iter_test:\n    \n    try:\n        prior_correct = eval(test['prior_group_answers_correct'].iloc[0])\n        prior_correct = [a for a in prior_correct if a != -1]\n    except:\n        prior_correct = []\n    \n    # Add prior correct to test and update stored users\n    if prior_correct:\n        prior_test.insert(s_train.answered_correctly_index, 'answered_correctly', prior_correct)\n        for uid, user in prior_test.groupby('user_id'):\n            s_train.update_user(\n                uid, user.drop(columns='user_id').to_numpy())\n\n    # Filter test\n    test = test.loc[\n        test['content_type_id'] == 0,\n        columns\n    ]\n\n    # Add global features\n    test, _, _ = transform_df(test, questions)\n\n    # Save test for later\n    prior_test = test.drop(columns='row_id').copy()\n\n    # Make x\n    x = np.apply_along_axis(\n        s_train.get_user_for_inference,\n        1,\n        test.drop(columns='row_id').to_numpy()\n    )\n\n    # Predict\n    test['answered_correctly'] = model.predict(x, batch_size=x.shape[0])\n    \n    env.predict(test[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Luckily with this contribution we can make the highest positions more competitive. Feel free to make use of this, fellas. Also, if you find interesting to make a team, just tell me!\n\nIf you feel like there's something I can explain or extend let me know, please, it's a real quick sketch."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}