{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Welcome**\n
\n\nThis notebook will be a detailed guide for impelementing, training and performing inference using the SAINT model. I hope it will useful for participants who failed to get a decent ROC score using it.\n\n\nUsing this implementation, with different hyperparameters and longer training time, I acheieved a 0.804 ROC score (private leaderboard), I ensembled my model along side with LGBM to the rank 39th. I used Tito's validation split to validate the model's score, and to perform inference.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import sys\nimport numpy as np\n\nimport os\nimport glob\nimport time\nfrom os import listdir\nfrom typing import Dict\nimport datatable as dt\n\nimport sklearn.preprocessing as preprocessing\nfrom sklearn.metrics import roc_auc_score\n\nimport pandas as pd\n\nfrom sys import getsizeof\n\n#supress warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport time\nimport gc\nimport math ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the sake of demonstration, I chose smaller hyperparameters, and smaller training data, however you can easily change it from here. For my model, I had the following hyperparameters:\n\n \n* **d_model**: 128\n \n* **batch_size**: 512\n \n* **seq_len**: 100\n \n* **train with whole data**\n \n* **Decoder, and encoder layers**: 4\n \n* **Dropout**: 0.1\n \n\n
\n If I had more computational resources, I would've performed a hyperparameter search, but per my experience, d_model, sequence length and dropout are the most crucial for a quality model. Sequence length is how many previous question answer pairs the model would use for its prediction, the bigger usually the better, but it would take more time to train.\n\n
\n The start token shifts past sequences forward so question answers would leak. For example, if the question answer input pair is ([53,23,15],[0,0,1]) , adding the answer start token woud make it like this: ([53,23,15],[2,0,0]). "},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 512\nseq_len = 100\n\n#If True, the model would be trained on +70 Million rows, 20M otherwise\ntrain_full = False\n\n#Answer start token\ncorrect_start_token = 2\nuser_answer_start_token = 4\n\n\n#Transformer hyperparameter \nd_model = 64\n\ndecoder_layers = 2\nencoder_layers = 2\n\ndropout = 0.1 \nff_model = d_model*4\n\natt_heads = d_model // 32\n\n\n#Loading questions, and every question corresponding part\nque_data = pd.read_csv( \"../input/riiid-test-answer-prediction/questions.csv\")\npart_valus = que_data.part.values\n\nunique_ques = len(que_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load the validation / training splitted data created by Tito. The validation data occur after the training data.\ntrain_data = pd.read_pickle(\"../input/riiid-cross-validation-files/cv1_train.pickle\")\nvalidation = pd.read_pickle(\"../input/riiid-cross-validation-files/cv1_valid.pickle\")\n\n\n#Remove uneeded rows, and drop lectures for the training data\ndel train_data[\"prior_question_had_explanation\"]\ndel train_data[\"prior_question_elapsed_time\"]\ndel train_data[\"max_time_stamp\"]\ndel train_data[\"rand_time_stamp\"]\ndel train_data[\"viretual_time_stamp\"]\n\ndel validation[\"prior_question_had_explanation\"]\ndel validation[\"prior_question_elapsed_time\"]\ndel validation[\"max_time_stamp\"]\ndel validation[\"rand_time_stamp\"]\ndel validation[\"viretual_time_stamp\"]\n\ntrain_data = train_data[train_data.content_type_id == False]\ndel train_data[\"content_type_id\"]\ndel train_data[\"row_id\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This line creates a dictionary containing the last timestamp of every user in the dataset: {user_id: timestamp, .....}. It is going to be useful during inference."},{"metadata":{"trusted":true},"cell_type":"code","source":"last_timestamp = train_data.groupby(\"user_id\")[[\"timestamp\",\"user_id\"]].tail(1).set_index(\"user_id\", drop=True)[\"timestamp\"].to_dict()\ntrain_data.reset_index(drop=True, inplace=True) #Resetting the index frees memory","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compute the difference of time between every two questions of every user in seconds (hence the division)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[\"timestamp\"] = train_data.groupby(\"user_id\")[\"timestamp\"].diff().fillna(0)/1000\ntrain_data[\"timestamp\"] = train_data.timestamp.astype(\"int32\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Encoding the timestamp**: Here, I encoded the timestamp in buckets for embedding, instead of using a continuous embedding. There is 70 possible value ranging from 0 seconds till 604800 second (a week). This feature really made the difference for me in reaching an ROC 0.780"},{"metadata":{"trusted":true},"cell_type":"code","source":"boundaries = [120,600,1800,3600,10800,43200,86400,259200,604800]\nx = train_data.timestamp.copy()\n\nfor i, boundary in enumerate(boundaries):\n \n if i == 0:\n start = 60\n else:\n start = boundaries[i-1]\n \n end = boundary\n \n train_data.loc[(x >= start) & (x < end), \"timestamp\"] = i+60\n \ntrain_data.loc[x >= end, \"timestamp\"] = i+60+1\n\ndel x\ntrain_data[\"timestamp\"] =train_data[\"timestamp\"].astype(\"int8\")\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Every row of the group series contains a tuple of the user's past question, answers, timestamp difference (encoded) and correctness of his answers. It would be used in creating the dataloader."},{"metadata":{"trusted":true},"cell_type":"code","source":"group = train_data[['user_id', 'content_id', 'answered_correctly', 'timestamp',\"user_answer\"]].groupby('user_id').apply(lambda r: (\n r['content_id'].values,\n r['answered_correctly'].values, r['timestamp'].values,r['user_answer'].values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating the training data, and local validation\n\n **Validation data:** Users with history longer than 100 sequence, chosen randomly, and capped off to the last 100 interactions. Note that this validation scheme is a little bit biased, and for that reason, later we are using Tito's validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating the validation data\nuser_counts = group.apply(lambda x: len(x[0])).sort_values(ascending=False)\nuser_counts = user_counts[(user_counts >= seq_len)]\n\naccepted_ids = user_counts.index\nval_group = group.loc[accepted_ids]\n\n\ndef f(x):\n return (x[0][:seq_len], x[1][:seq_len], x[2][:seq_len], x[3][:seq_len])\n\nval_group = val_group.apply(f).sample(frac=0.1)\ngroup = group.drop(index=val_group.index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Training data:** If train_full is set to True, all users with history longer than 100 be selected for training. The sequences length is going to be 100 for every training point, I didn't use padding. \n\nFor a user with 932 sequence for example, I take 9 sequences of 100 interactions and drop the last 32 interactions. Every one of these 9 sequences is independent, meaning the model would base its prediction solely on the 100 previous interaction. It is not the best approach, but it worked well given how large the dataset is. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating sequences of 100 of the all interactions less than 1000\nuser_counts = group.apply(lambda x: len(x[0])).sort_values(ascending=False)\n\nif train_full: #Train with subset of data or not\n user_counts = user_counts[(user_counts >= seq_len)]\nelse:\n user_counts = user_counts[((user_counts >= seq_len) & (user_counts <= 1000))]\n\naccepted_ids = user_counts.index\ngroup = group.loc[accepted_ids]\n\ngroup.index = group.index.astype(\"str\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\n\nauxiliary = []\nk = 0\n \nfor line in tqdm(group):\n \n src, trg, ts, user_answer = line\n chunk_len = seq_len\n i = 0\n \n split_size = src.shape[0] - src.shape[0]%chunk_len\n n_splits = split_size/chunk_len\n \n lst = list(zip(np.split(src[:split_size],n_splits), \n np.split(trg[:split_size], n_splits), \n np.split(ts[:split_size], n_splits), \n np.split(user_answer[:split_size], n_splits),\n ))\n \n auxiliary.extend(lst) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"auxiliary = pd.Series(auxiliary)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.utils.rnn as rnn_utils\nfrom torch.autograd import Variable\n\nimport torch.nn.functional as F\nfrom torch.nn import TransformerEncoder, TransformerEncoderLayer\nfrom torch.nn import TransformerDecoder, TransformerDecoderLayer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TextLoader(torch.utils.data.Dataset):\n def __init__(self, data):\n self.x, self.y, self.ts, self.user_answer = [], [], [], []\n \n for line in data:\n x, y, ts, user_answer = line\n \n self.x.append(x)\n self.y.append(y) \n self.ts.append(ts)\n self.user_answer.append(user_answer)\n\n def __getitem__(self, index):\n return (torch.LongTensor(self.x[index]), \n torch.LongTensor(self.y[index]), \n torch.LongTensor(self.ts[index]),\n torch.LongTensor(self.user_answer[index]))\n\n def __len__(self):\n return len(self.x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TextCollate():\n \n def __call__(self, batch):\n \n x_padded = torch.LongTensor(seq_len, len(batch))\n y_padded = torch.LongTensor(seq_len, len(batch)) \n ts_padded = torch.LongTensor(seq_len, len(batch)) \n user_answer_padded = torch.LongTensor(seq_len, len(batch))\n\n for i in range(len(batch)):\n \n x = batch[i][0]\n x_padded[:x.size(0), i] = x\n \n y = batch[i][1]\n y_padded[:y.size(0), i] = y\n \n ts = batch[i][2]\n ts_padded[:y.size(0),i] = ts\n \n user_answer = batch[i][3]\n user_answer_padded[:y.size(0),i] = user_answer\n \n\n return x_padded, y_padded, ts_padded, user_answer_padded","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pin_memory = True\nnum_workers = 2\n\ntrainset = TextLoader(auxiliary)\nvalset = TextLoader(val_group)\n\ncollate_fn = TextCollate()\n\ntrain_loader = torch.utils.data.DataLoader(trainset, num_workers=num_workers, shuffle=True,\n batch_size=batch_size, pin_memory=pin_memory,\n drop_last=True, collate_fn=collate_fn) #(seq_len, batch_size)\n\nval_loader = torch.utils.data.DataLoader(valset, num_workers=num_workers, shuffle=False,\n batch_size=batch_size, pin_memory=pin_memory,\n drop_last=False, collate_fn=collate_fn)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Creating the model\n\nThis model largely follows the SAINT+ paper. I used Pytorch's `nn.Transformer` for the implementation, and copied some snippets of code from [this](https://nlp.seas.harvard.edu/2018/04/03/attention.html) glorious notebook (The NOAM optimizer and positional encoding). I think the model's implementation is pretty clear, but tell me if any further explanation is needed.\n \nIt is comprised of a encoder and a decoder. The encoder takes lag time, past questions and question part embedding as input; meanwhile the decoder takes the correctness of the user past answer, and past user answers as input. \n \nUsually the model converges in 60 epochs or so, but training it for longer can get you to 0.800 ROC. "},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\npart_valus = torch.LongTensor(part_valus).to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class PositionalEncoding(nn.Module):\n def __init__(self, d_model, dropout=0.1, max_len=5000):\n super(PositionalEncoding, self).__init__()\n self.dropout = nn.Dropout(p=dropout)\n self.scale = nn.Parameter(torch.ones(1))\n\n pe = torch.zeros(max_len, d_model)\n position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n div_term = torch.exp(torch.arange(\n 0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n pe[:, 0::2] = torch.sin(position * div_term)\n pe[:, 1::2] = torch.cos(position * div_term)\n pe = pe.unsqueeze(0).transpose(0, 1)\n self.register_buffer('pe', pe)\n\n def forward(self, x):\n x = x + self.scale * self.pe[:x.size(0), :]\n return self.dropout(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class NoamOpt:\n \"Optim wrapper that implements rate.\"\n def __init__(self, model_size, factor, warmup, optimizer):\n self.optimizer = optimizer\n self._step = 0\n self.warmup = warmup\n self.factor = factor\n self.model_size = model_size\n self._rate = 0\n \n def step(self):\n \"Update parameters and rate\"\n self._step += 1\n rate = self.rate()\n for p in self.optimizer.param_groups:\n p['lr'] = rate\n self._rate = rate\n self.optimizer.step()\n \n def rate(self, step = None):\n \"Implement `lrate` above\"\n if step is None:\n step = self._step\n return self.factor * \\\n (self.model_size ** (-0.5) *\n min(step ** (-0.5), step * self.warmup ** (-1.5)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TransformerModel(nn.Module):\n \n def __init__(self, intoken, hidden, part_arr, enc_layers, dec_layers, dropout, nheads, ff_model, ts_unique=70):\n super(TransformerModel, self).__init__()\n \n self.encoder = nn.Embedding(intoken, hidden)\n self.pos_encoder = PositionalEncoding(hidden, dropout)\n\n self.decoder = nn.Embedding(3, hidden) #0: False , 1: Correct , 3 : Padding\n self.pos_decoder = PositionalEncoding(hidden, dropout)\n \n \n self.transformer = nn.Transformer(d_model=hidden, nhead=nheads, num_encoder_layers=enc_layers, num_decoder_layers=dec_layers, dim_feedforward=ff_model, dropout=dropout, activation='relu')\n self.fc_out = nn.Linear(hidden, 1)\n\n self.src_mask = None\n self.trg_mask = None\n self.memory_mask = None\n \n self.part_embedding = nn.Embedding(7,hidden)\n self.part_arr = part_arr\n \n self.ts_embedding = nn.Embedding(ts_unique, hidden) \n self.user_answer_embedding = nn.Embedding(5, hidden)\n\n \n self.dropout_1 = nn.Dropout(dropout)\n self.dropout_2 = nn.Dropout(dropout)\n self.dropout_3 = nn.Dropout(dropout)\n self.dropout_4 = nn.Dropout(dropout)\n self.dropout_5 = nn.Dropout(dropout)\n self.dropout_6 = nn.Dropout(dropout)\n\n \n def generate_square_subsequent_mask(self, sz, sz1=None):\n \n if sz1 == None:\n mask = torch.triu(torch.ones(sz, sz), 1)\n else:\n mask = torch.triu(torch.ones(sz, sz1), 1)\n \n return mask.masked_fill(mask==1, float('-inf'))\n\n\n def forward(self, src, trg, ts, user_answer):\n\n if self.trg_mask is None or self.trg_mask.size(0) != len(trg):\n self.trg_mask = self.generate_square_subsequent_mask(len(trg)).to(trg.device)\n \n if self.src_mask is None or self.src_mask.size(0) != len(src):\n self.src_mask = self.generate_square_subsequent_mask(len(src)).to(trg.device)\n \n if self.memory_mask is None or self.memory_mask.size(0) != len(trg) or self.memory_mask.size(1) != len(src):\n self.memory_mask = self.generate_square_subsequent_mask(len(trg),len(src)).to(trg.device)\n \n\n \n #Get part, prior, timestamp, task_container and user answer embedding\n part_emb = self.dropout_1(self.part_embedding(self.part_arr[src]-1))\n ts_emb = self.dropout_3(self.ts_embedding(ts))\n user_answer_emb = self.dropout_4(self.user_answer_embedding(user_answer)) \n \n \n #Add embeddings Encoder\n src = self.dropout_5(self.encoder(src)) #Embedding\n src = torch.add(src, part_emb)\n src = torch.add(src, ts_emb) #Last interaction days \n src = self.pos_encoder(src) #Pos embedding\n \n \n #Add embedding decoder\n trg = self.dropout_6(self.decoder(trg))\n trg = torch.add(trg, user_answer_emb)\n trg = self.pos_decoder(trg)\n\n output = self.transformer(src, trg, src_mask=self.src_mask, tgt_mask=self.trg_mask, memory_mask=self.memory_mask)\n \n\n output = self.fc_out(output)\n\n return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"que_emb_size = unique_ques\n\nmodel = TransformerModel(que_emb_size, hidden=d_model,part_arr=part_valus, dec_layers=decoder_layers, enc_layers=encoder_layers, dropout=dropout, nheads=att_heads, ff_model=ff_model).to(device)\n\ndef count_parameters(model):\n return sum(p.numel() for p in model.parameters() if p.requires_grad)\n\nprint(f'The model has {count_parameters(model):,} trainable parameters')\nprint(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.optim as optim\n\n\noptimizer = NoamOpt(d_model, 1, 4000 ,optim.Adam(model.parameters(), lr=0))\n\n#Since the objective of training is a binary classification\ncriterion = nn.BCEWithLogitsLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Add padding to decoder input\ndef add_shift(var, pad):\n \n var_pad = torch.ShortTensor(1, var.shape[1]).to(device)\n var_pad.fill_(pad)\n \n return torch.cat((var_pad, var))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(model, optimizer, criterion, iterator):\n \n model.train()\n \n epoch_loss = 0\n \n for i, batch in enumerate(iterator):\n \n src, trg, ts, user_answer = batch\n src, trg, ts, user_answer = src.to(device), trg.to(device), ts.to(device), user_answer.to(device)\n\n \n trg = add_shift(trg, correct_start_token)\n user_answer = add_shift(user_answer, user_answer_start_token) \n \n optimizer.optimizer.zero_grad()\n output = model(src, trg[:-1,:], ts, user_answer[:-1,:])\n \n loss = criterion(output.squeeze(), trg[1:,:].float())\n\n loss.backward()\n torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n optimizer.step()\n epoch_loss += loss.item()\n \n return epoch_loss / len(iterator)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\n\ndef evaluate(model, criterion, iterator):\n\n model.eval()\n\n epoch_loss = 0\n acc = 0\n \n preds = []\n corr = []\n\n with torch.no_grad(): \n for i, batch in enumerate(iterator):\n\n\n src, trg, ts, user_answer = batch\n src, trg, ts, user_answer = src.to(device), trg.to(device), ts.to(device), user_answer.to(device)\n\n\n trg = add_shift(trg, correct_start_token)\n user_answer = add_shift(user_answer, user_answer_start_token) \n\n optimizer.optimizer.zero_grad()\n output = model(src, trg[:-1,:], ts, user_answer[:-1,:])\n loss = criterion(output.squeeze(), trg[1:,:].float())\n \n preds.extend(F.sigmoid(output).squeeze().reshape(-1).detach().cpu().numpy().tolist())\n corr.extend(trg[1:,:].reshape(-1).detach().cpu().numpy().tolist())\n \n nb_correct = F.sigmoid(output).squeeze().transpose(0, 1).round().reshape(-1) == trg[1:,:].float().transpose(0, 1).reshape(-1)\n accuracy = nb_correct.sum()/float(output.squeeze().transpose(0, 1).round().reshape(-1).shape[0])\n \n \n epoch_loss += loss.item()\n \n acc += accuracy.item()\n \n \n \n return (epoch_loss / len(iterator), acc/len(iterator), roc_auc_score(np.array(corr),np.array(preds)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef epoch_time(start_time, end_time):\n elapsed_time = end_time - start_time\n elapsed_mins = int(elapsed_time / 60)\n elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n return elapsed_mins, elapsed_secs\n\nN_EPOCHS = 60\n\nbest_roc = 0\n\nfor epoch in range(N_EPOCHS):\n print(f'Epoch: {epoch+1:02} ({best_roc})')\n\n start_time = time.time()\n\n train_loss = train(model, optimizer, criterion, train_loader)\n valid_loss, acc, roc = evaluate(model, criterion, val_loader)\n\n epoch_mins, epoch_secs = epoch_time(start_time, time.time())\n\n if roc > best_roc:\n best_roc = roc\n torch.save(model.state_dict(), 'model_best.torch')\n\n print(f'Time: {epoch_mins}m {epoch_secs}s')\n print(f'Train Loss: {train_loss:.3f}')\n print(f'Val Loss: {valid_loss:.3f} Acc {acc:.3f} ROC {roc:.3f}')\n \nprint(best_roc)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inference\n\nTo simulate a real submission, I used the [timeseries API emulator](https://www.kaggle.com/its7171/time-series-api-iter-test-emulator), again created by Tito. I used the same strategy to validate my model, and usually the ROC score I get here matches the leaderboard consistently.\n\n \nI re-created the group sequences of every user, so I have access on their history.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"group = train_data[['user_id', 'content_id', 'answered_correctly', 'timestamp',\"user_answer\"]].groupby('user_id').apply(lambda r: (\n r['content_id'].values,\n r['answered_correctly'].values, r['timestamp'].values,r['user_answer'].values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" The `pred_users` function takes a numpy array of the shape (batch, 4), and it returns the SAINT model's prediction if the user's answer is correct or not. This function:\n\n1. Loops through every item of the input array\n2. Fetch this particular user history, and cap it to the last seq_len -1 sequences\n3. Formats these sequences into Pytorch array and feeds them to the model\n4. Retreive the prediction out of the output.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def pred_users(vals): #Input must be (eval_batch, 3): [\"user_id\", \"content_id\", \"content_type_id\", \"timestamp\"]\n\n eval_batch = vals.shape[0]\n\n tensor_question = np.zeros((eval_batch, seq_len), dtype=np.long)\n tensor_answers = np.zeros((eval_batch, seq_len), dtype=np.long)\n tensor_ts = np.zeros((eval_batch, seq_len), dtype=np.long)\n tensor_user_answer = np.zeros((eval_batch, seq_len), dtype=np.long)\n\n\n val_len = []\n preds = []\n group_index = group.index\n\n for i, line in enumerate(vals):\n\n if line[2] == True:\n val_len.append(0)\n continue\n\n user_id = line[0]\n question_id = line[1]\n timestamp = get_timestamp(line[3], user_id) #Compute timestamp difference correctly\n \n\n que_history = np.array([], dtype=np.int32)\n answers_history = np.array([], dtype=np.int32) \n ts_history = np.array([], dtype=np.int32) \n user_answer_history = np.array([], dtype=np.int32) \n\n if user_id in group_index:\n\n cap = seq_len-1\n que_history, answers_history, ts_history, user_answer_history = group[user_id]\n\n que_history = que_history[-cap:]\n answers_history = answers_history[-cap:]\n ts_history = ts_history[-cap:]\n user_answer_history = user_answer_history[-cap:]\n\n\n #Decoder data, add start token\n answers_history = np.concatenate(([correct_start_token],answers_history))\n user_answer_history = np.concatenate(([user_answer_start_token],user_answer_history))\n\n #Decoder data\n que_history = np.concatenate((que_history, [question_id])) #Add current question\n ts_history = np.concatenate((ts_history, [timestamp])) \n\n tensor_question[i][:len(que_history)] = que_history\n tensor_answers[i][:len(que_history)] = answers_history\n tensor_ts[i][:len(que_history)] = ts_history\n tensor_user_answer[i][:len(que_history)] = user_answer_history\n\n val_len.append(len(que_history))\n\n tensor_question = torch.from_numpy(tensor_question).long().T.to(device)\n tensor_answers = torch.from_numpy(tensor_answers).long().T.to(device)\n tensor_ts = torch.from_numpy(tensor_ts).long().T.to(device)\n tensor_user_answer = torch.from_numpy(tensor_user_answer).long().T.to(device)\n \n with torch.no_grad(): #Disable gradients so prediction runs faster\n out = F.sigmoid(model(tensor_question, tensor_answers, tensor_ts, tensor_user_answer)).squeeze(dim=-1).T\n\n\n for j in range(len(val_len)):\n preds.append(out[j][val_len[j]-1].item())\n\n return preds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" The `update_group_var` function simply updates the `group` Series and the `last_timestamp` to keep track of what the user have learnt"},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_group_var(vals):\n \n global group\n \n for i, line in enumerate(vals):\n \n user_id = line[0]\n question_id = line[1]\n \n content_type_id = line[2]\n ts = get_timestamp(line[3], user_id)\n \n correct = line[4]\n user_answer = line[5]\n \n \n if content_type_id == True:\n continue\n\n if last_timestamp.get(user_id, -1) == -1:\n last_timestamp[user_id] = 0\n else:\n last_timestamp[user_id] = line[3]\n \n if user_id in group.index:\n questions= np.append(group[user_id][0],[question_id])\n answers= np.append(group[user_id][1],[correct])\n ts= np.append(group[user_id][2],[ts])\n user_answer= np.append(group[user_id][3],[user_answer])\n \n group[user_id] = (questions, answers, ts, user_answer)\n else:\n group[user_id] = (np.array([question_id], dtype=np.int32), np.array([correct], dtype=np.int32), np.array([ts], dtype=np.int32)\n ,np.array([user_answer], dtype=np.int32))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Re-creates the timestamp encoding\ndef get_timestamp(ts, user_id):\n \n if last_timestamp.get(user_id, -1) == -1:\n return 0\n \n diff = (ts - last_timestamp[user_id])/1000\n \n if diff < 0:\n return 0\n \n if diff <= 60:\n return int(diff)\n \n for i, boundary in enumerate(boundaries):\n if boundary > diff:\n break\n \n if i == len(boundaries) - 1:\n return 60+i+1\n \n return 60+i","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Tito's iterator: https://www.kaggle.com/its7171/time-series-api-iter-test-emulator\n\nclass Iter_Valid(object):\n def __init__(self, df, max_user=1000):\n df = df.reset_index(drop=True)\n self.df = df\n self.user_answer = df['user_answer'].astype(str).values\n self.answered_correctly = df['answered_correctly'].astype(str).values\n df['prior_group_responses'] = \"[]\"\n df['prior_group_answers_correct'] = \"[]\"\n self.sample_df = df[df['content_type_id'] == 0][['row_id']]\n self.sample_df['answered_correctly'] = 0\n self.len = len(df)\n self.user_id = df.user_id.values\n self.task_container_id = df.task_container_id.values\n self.content_type_id = df.content_type_id.values\n self.max_user = max_user\n self.current = 0\n self.pre_user_answer_list = []\n self.pre_answered_correctly_list = []\n\n def __iter__(self):\n return self\n \n def fix_df(self, user_answer_list, answered_correctly_list, pre_start):\n df= self.df[pre_start:self.current].copy()\n sample_df = self.sample_df[pre_start:self.current].copy()\n df.loc[pre_start,'prior_group_responses'] = '[' + \",\".join(self.pre_user_answer_list) + ']'\n df.loc[pre_start,'prior_group_answers_correct'] = '[' + \",\".join(self.pre_answered_correctly_list) + ']'\n self.pre_user_answer_list = user_answer_list\n self.pre_answered_correctly_list = answered_correctly_list\n return df, sample_df\n\n def __next__(self):\n added_user = set()\n pre_start = self.current\n pre_added_user = -1\n pre_task_container_id = -1\n\n user_answer_list = []\n answered_correctly_list = []\n while self.current < self.len:\n crr_user_id = self.user_id[self.current]\n crr_task_container_id = self.task_container_id[self.current]\n crr_content_type_id = self.content_type_id[self.current]\n if crr_content_type_id == 1:\n # no more than one task_container_id of \"questions\" from any single user\n # so we only care for content_type_id == 0 to break loop\n user_answer_list.append(self.user_answer[self.current])\n answered_correctly_list.append(self.answered_correctly[self.current])\n self.current += 1\n continue\n if crr_user_id in added_user and ((crr_user_id != pre_added_user) or (crr_task_container_id != pre_task_container_id)):\n # known user(not prev user or differnt task container)\n return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n if len(added_user) == self.max_user:\n if crr_user_id == pre_added_user and crr_task_container_id == pre_task_container_id:\n user_answer_list.append(self.user_answer[self.current])\n answered_correctly_list.append(self.answered_correctly[self.current])\n self.current += 1\n continue\n else:\n return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n added_user.add(crr_user_id)\n pre_added_user = crr_user_id\n pre_task_container_id = crr_task_container_id\n user_answer_list.append(self.user_answer[self.current])\n answered_correctly_list.append(self.answered_correctly[self.current])\n self.current += 1\n if pre_start < self.current:\n return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n else:\n raise StopIteration()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = Iter_Valid(validation,max_user=1000)\npredicted = []\ndef set_predict(df):\n predicted.append(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" Finally looping, for d_model 64 it should take approx 1 hour. "},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nimport ast\n\nmodel.eval()\n\npreds = []\npbar = tqdm(total=2500000, position=0, leave=True)\ncheck = None\n\nfor (test_data, current_prediction_df) in iter_test: \n \n if check is not None:\n past_vals = np.array(ast.literal_eval(test_data.iloc[0].prior_group_answers_correct)) \n past_answers = np.array(ast.literal_eval(test_data.iloc[0].prior_group_responses))\n\n past_vals = np.concatenate((vals, past_vals.reshape(len(past_vals),1)), axis=1)\n past_vals = np.concatenate((past_vals, past_answers.reshape(len(past_answers),1)), axis=1)\n\n update_group_var(past_vals) #Update database with the vals of the last batch \n \n vals = test_data[[\"user_id\",\"content_id\",\"content_type_id\",\"timestamp\"]].values\n preds.extend(pred_users(vals))\n \n check = 1\n\n pbar.update(len(test_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = validation.iloc[:len(preds)]\ndf[\"preds\"] = preds\n\ndf = df[df.content_type_id == False]\nprint('Validation ROC:',roc_auc_score(df.answered_correctly, df.preds))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"** Conclusion**\n\n In this notebook I wanted to prove that even with a relatively simple and small model (d_model = 64), you can get a fairly good accuracy, scaling it would for sure increase the score. I also realised how sensitive implementing transformers could be, changing one hyper-parameter can ruin everything. This led me to think that a simple notebook like this one can be super useful for those who want to play with the model and experiment with new ideas, future kagglers learning about this competition, or even for those who are studying transformers.\n\nI hope this notebook was helpful, if more explanation is needed, please feel free to ask in the comments. Also if you think some of my code is innificient, also point it out. \n \n\n\n**PS**: I am currently looking for an entry DS job, if you are hiring or you can recommend me, please check my linkedin profile: https://www.linkedin.com/in/abdessalem-boukil-37923637/ "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}