{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory data analysis\n\nWe first import all the necessary libraries to perform the analysis and fit the first models. We also print the directory structure of the files of the competition:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# General libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Modules of this competition (we don't need them until the creation of the test prediction):\n#\n# import riiideducation\n# env = riiideducation.make_env()\n\n# Printing of all the files in the computetition\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n        \n# Observe it includes all the mentioned csv files as well as the riiideducation module.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The following libraries are need to implement the SAKT model\nimport gc # garbage colector\nimport random # random numbers\nfrom tqdm import tqdm # progress bars (from Arabic taqaddum)\nfrom sklearn.metrics import roc_auc_score # to obtain the AUC of the ROC curve\nfrom sklearn.model_selection import train_test_split # for splitting the dataset in two\n\nimport seaborn as sns  # statistical data visualization\nimport matplotlib.pyplot as plt  # more data visualization\n\nimport torch # Self-explanatory\nimport torch.nn as nn # Building blocks for graphs (of neural networks)\nimport torch.nn.utils.rnn as rnn_utils # I don't know exactly what they include\nfrom torch.autograd import Variable # Automatic differentiation\nfrom torch.utils.data import Dataset, DataLoader # For loading data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load of the dataset\n\nSince the dataset is large, we need to solve the problems related with the loading of the dataset.\n\nIssues we need to solve:\n* How to load the whole dataset (or use it to train any model and for explation).\n* Perform EDA in order to discover the structure of the dataset (max 3 days to do this, preferably half a day)\n* Fit a first model using random forest as \"basis\" score.\n* Program the ROC curve (2 hours max) and using a library that has it programmed. This is important in order to testing and evaluating our model without needing to submit the predictions (which seems to be slow).\n* Review literature about different models. Ideas: XGBoost, ensembles that involve RNN, LSTM, state-of-art NN and trees."},{"metadata":{},"cell_type":"markdown","source":"## Data loading (based on the notebook SAKT1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Hyperparameter of the sequence lenght.\nMAX_SEQ = 100\nepochs = 35","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n# dtype of all columns\n'''\ndtype={'row_id': 'int64',\n    'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n    'content_type_id': 'int8',\n    'task_container_id': 'int16',\n    'user_answer': 'int8', \n    'answered_correctly':'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean',}\n'''\n# dtype of the subsect of columns used in SAKT1\ndtype = {'timestamp': 'int64',\n         'user_id': 'int32',\n         'content_id': 'int16',\n         'content_type_id': 'int8',\n         'answered_correctly':'int8'}\n\n# This loads all the rows but only 5 columns\ntrain_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', usecols=[1, 2, 3, 4, 7], dtype=dtype)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploratory data analysis\n\n* The dataset with only 5 columns has a size of 1.5 Gb, with 101,230,332 rows (100 million!)\n* With respect of the timestamp: the iterations have an average timestamp of 89 days (7.70e+09 seconds) and a maximum timestamp of 1011.87 days (2 years and 9 months; 8.74e+10 miliseconds).\n* The dataset consists of 1,959,032 lectures being watched and 99,171,300 questions posed to students. That is 98.06% of the iterations are questions.\n* Approx. 65.72% of the questions were answered correctly.\n* As for the particular questions (content_id), there are exactly 13,523 different questions in the training dataset.\n* There are 393,656 different students in the training set. \n\n**Note**: we shouldn't use the used_id (maybe the content_id neither) to make our predictions, because we need to predict accurately for the new students.\n\n### The dataset used by SAKT1\n\n* It ignores the lectures, and only uses the information used about the questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df.info()\n#train_df[train_df.answered_correctly != -1].describe()\n\n# Consider only questions as in SAKT1\ntrain_df = train_df[train_df.content_type_id == False]\n\n# The sort the dataframe by the timestamp\ntrain_df = train_df.sort_values(['timestamp'], ascending=True).reset_index(drop = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Research of models used\n\n* Most notebooks use a model called \"Self-Attentive model for Knowledge Tracing\" (SAKT), which is based on this [paper](https://arxiv.org/abs/1907.06837) by Shalini Pandey and George Karypis.\n    - Definition of Knowledge Tracing: the task of modeling each student's mastery of knowledge concepts (KCs) and s(he) engages with a sequence of learning activities.\n    - Recent methods are based on RNN (e.g., \"Deep Knowledge Tracing\" DKT and \"Dynamic Key-Value Memory Network\" DKVMN) and outperformed traditional methods because of their ability to capture complex representation of human learning.\n    - These recent methods have the issue of not generalizing well while dealing with sparse data (which is the case with real-world data as students interact with few KCs).\n    - The proposed approach identifies the KCs from the student's past activities that are relevant to the given KC and predicts his mastery based on the relatively few KCs that it picked. This allows to handle sparse data better compared with RNNs (since it's based on relatively few past activities).\n    - To identify the relevance between KCs, it's proposed a self-attention based approach (which is the SAKT).\n    - The experimentation in the paper outperforms state-of-art models, improving AUC by an average of 4.43%\n    - We describe the structure of the SAKT later.\n    - Most popular public notebooks that use SAKT have a score of 0.773 (which we need to improve).\n    - SAKT was tested (in the paper) on 5 datasets: 4 are real-world and the last one is synthetic. All the datasets contained several thousands of iterations and variable density (defined as (#Unique Iterations)/(#User * #Skill tags) ).\n    - In the paper, the model is evaluated using the area under the ROC curve (AUC). The SAKT was compared with state-of-art models. Trained with 80% of the dataset and tested in the remaining dataset. SAKT was implemented in TensorFlow with an ADAM optimized and a learning rate of 0.001. Dropout rate of 0.2 for big datasets. The maximum length of the sequence $n$ was selected an roughtly proportional to the average exercise tags per student (that is, between 50 to 100).\n    - Results: the score was only slightly better for the SAKT compared to other models. The improvement was of 15.87% for the more sparse model, from 0.727 to 0.854. However, I don't believe this improvement is achived on every sparse dataset (perhaps I'm wrong)."},{"metadata":{},"cell_type":"markdown","source":"## Preprocess of the data (based on SAKT1)\n\n* To preprocess, we first obtain a list with all possible question ids (all the different possible questions).\n* Then we group the dataset by the user_id, where the aggregating function produces a tuple with two entries:\n    - The first entry is a list of all the ids of the questions possed to that student\n    - The second entry is a list of whether the student go the answer right (1) or not (0)\n* **Note**: the preprocessing based on SAKT1 gets rid of the timestamp and the content_type_id. Therefore, we use only two columns to make our predictions (user_id and content_id) and the third column is the output that we use to train. Moreover, only considers users with more than 10 questions answered and limits the historical answers up to 100 questions (hence, the model don't consider your first iterations once you have more than 100 questions answered.)\n* Thus, we end with 312,824 students in the training set and 78204 students in the test set (after a 80/20 split). That it, we ignored only 2,628 users."},{"metadata":{"trusted":true},"cell_type":"code","source":"# First we get the particular ids for each questions\n\nquestions = train_df[\"content_id\"].unique()\nn_questions = len(questions)\nprint(\"number questions:\", len(questions))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This \"groups\" by used id, so the final Series has 393,000 entries. The index is the used id and the value\n# is a tuble with two entries: the first entry contrains all the content_id (in a list) for the questions posed to that\n# student while the second entry contrains if the student answered correctly or not (also in a list)\ngroup = train_df[['user_id', 'content_id', 'answered_correctly']].groupby('user_id').apply(lambda r: (\n            r['content_id'].values,\n            r['answered_correctly'].values))\n\ndel train_df\n# Actually calling the garbage collector.\ngc.collect()\n# for getting the number of users:\n# len(group)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We import random because we need it to randomly select iterations.\nimport random\nrandom.seed(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Notes on the class for the dataset\n\n* It should inherit from `Dataset` class (in `Pytorch.utils.data.Dataset`)\n* The constructor takes the group Series and the n_questions number (the number of different questions), and another parameter called max_seq with default as 100.\n* When called, the constructor initializes three fields: `samples` as the group Series, `n_questions`, and `max_seq`.\n    - Then, it creates a list of all the `user_ids` with more than 10 questions posed (but we don't change the other fields)\n* It defines `len` as the number of user_ids (with +10 questions posed)\n* Finally, it defines the `getitem` method. This method is the method called when we use the brackets. This method gets an index (which is mapped to an used_id in that position) and return three arrays: one contains the question_ids for the user_id (except the first one), the correctness of those questions, and another array what contains the same question_id for questions with wrong answer and questions_id + no_questions for questions with right answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Class for the data set\n# Note: I don't like that it uses too few information. We could improve our model with more features,\n# though we must use feature selection in order to avoid unnecessary noise.\n\n# Note: in first version, MAX_SEQ was 100 and we ignored those\n# users with less than 10 questions answered.\n\nclass SAKTDataset(Dataset):\n    def __init__(self, group, n_questions, max_seq=MAX_SEQ):\n        super(SAKTDataset, self).__init__()\n        self.max_seq = max_seq\n        self.n_questions = n_questions\n        self.samples = group\n        \n        self.user_ids = [] # list of all the user ids\n        for user_id in group.index:\n            # cicles all possible user_ids\n            \n            # gets the questions, answers for that user\n            q, qa = group[user_id]\n            if len(q) < 2:\n                # if we have less than 2 questions answered, then we\n                # ignore the user_id\n                continue\n            # we only append those ids with more than 10 questions possed\n            self.user_ids.append(user_id)\n            \n            # To reduce the memory:\n            #\n            #if len(q)>self.max_seq:\n            #    group[user_id] = (q[-self.max_seq:],qa[-self.max_seq:])\n\n    # we define the len of the dataset as the number of user_ids with more\n    # than 10 questions posed.\n    def __len__(self):\n        return len(self.user_ids)\n\n    # We define the getitem function (indexing using brakets: [])\n    def __getitem__(self, index):\n        # we get the used_id of that index\n        user_id = self.user_ids[index]\n        # then we get the questions and answers for that user id\n        q_, qa_ = self.samples[user_id]\n        # finally, we compute the numbers of questions answered by that student\n        seq_len = len(q_)\n\n        # we create np.ndarrays the length of max_seq\n        q = np.zeros(self.max_seq, dtype=int)\n        qa = np.zeros(self.max_seq, dtype=int)\n        # Originally: In case that q_ and qa_ are larger, we assign to the np.ndarrays the last\n        # max_seq elements.\n        # Then it was defined a random selection of user interactions.\n        if seq_len >= self.max_seq:\n            if random.random()>0.1:\n                # Here we obtain also a random number in (0, 1). Then, if\n                # the number is greater than 0.1 (90% of the time), then\n                # we obtain a random number in (0, k), where k is the\n                # number of observations that are left out of the input\n                # sequence of the neural network. This random number\n                # will be the first element to be considered. This means\n                # that we would consider the last exercises only if the\n                # random number turns to be k.\n                start = random.randint(0,(seq_len-self.max_seq))\n                end = start + self.max_seq\n                q[:] = q_[start:end]\n                qa[:] = qa_[start:end]\n            else:\n                # The previous case.\n                q[:] = q_[-self.max_seq:]\n                qa[:] = qa_[-self.max_seq:]\n            \n        else:\n            if random.random()>0.1:\n                # We obtain a random number. If it's larger than 0.1\n                # then we establish a random subsequence of the list\n                #\n                ############ Why would this improve performance?\n                start = 0\n                end = random.randint(2,seq_len)\n                seq_len = end - start\n                q[-seq_len:] = q_[0:seq_len]\n                qa[-seq_len:] = qa_[0:seq_len]\n            else:\n                # The previous case: we store all the elements of q\n                q[-seq_len:] = q_\n                qa[-seq_len:] = qa_\n                   \n        # we store the all, but the first elements of q, qa in arrays called\n        # target_id (questions) and label (correctness of answer)\n        target_id = q[1:]\n        label = qa[1:]\n        \n        # Finally, x is a ndarray with length of max_seq - 1, and its values are the same\n        # question_id for the questions answered wrong and questions_id + n_questions for the\n        # questions answered correctly (which produces a totally valid new id, at least\n        # restricted on the training dataset, the id produced might be used in a new question\n        # appearing in the testset.)\n        # This x array is created this way because this is how the paper of SAKT creates it.\n        # Still, I don't know why I have to use it like that.\n        x = np.zeros(self.max_seq-1, dtype=int)\n        x = q[:-1].copy()  # we obtain all the questions_ids, except for the last one\n        x += (qa[:-1] == 1) * self.n_questions # then, we for the questions answered correctly\n                                               # we add n_questions\n\n        return x, target_id, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Brief notes on DataLoader of Pytorch:\n\n* The most important argument is the dataset argument, which indicates the data that will be loaded. Pytorch supports:\n    - map-style datasets: the classes that have `__getitem__()` and `__len()` defined.\n    - iterable-style datasets: classes that inherit form `IterableDataset` and implements `__iter__()`.\n* Using num_workers as a positive integer makes the DataLoading parallel (several worker processes).\n* `num_workers`: how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process.\n* `batch_size`: how many samples per batch to load.\n* `shuffle`: set to `True` to have the data reshuffled at every epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"# We could create an «if» statement and use it for different\n# splitings. train/val for the selection of the model and\n# 100% train for doing the last submission.\n\n# This splits our dataset in order to avoid testing on the competition's public test set.\ntrain, val = train_test_split(group, test_size=0.2)\n\n# Initializes the class we defined above.\ntrain_dataset = SAKTDataset(train, n_questions)\n# Uses the DataLoader to load data into Pytorch\ntrain_dataloader = DataLoader(train_dataset, batch_size=2048, shuffle=True, num_workers=8)\n# garbage collector:\ndel train\n\n# The same for the test dataset:\nval_dataset = SAKTDataset(val, n_questions)\nval_dataloader = DataLoader(val_dataset, batch_size=2048, shuffle=True, num_workers=8)\ndel val\n\n# Actually calling the garbage collector.\ngc.collect()\n\nprint(len(train_dataset))\nprint(len(val_dataset))\n\n# The first entry in the out of getitem is the first element\n# of the training dataset.\n#\n# print(val_dataset[0])\n# print(item[1])\n# print(item[2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Definition of the SAKT model (based on SAKT1)\n\nThe process to define a Pytorch model:\n1. First we load and prepare the data. This implies having the data in map-style or iterable-style as described above. Note that we need numerical inputs and numerical outputs. *Did we need to create a class? We could use the same dataframe we used before.*\n2. Then we define the model (structure of the NN). To define a model, we need to extend the nn.Module.\n3. After that, we need to define a loss function and an optimization algorithm. This allows to train the NN.\n4. We train the NN. Then, we test it and finally we make predictions with it.\n\n\n### On how we define the structure of the NN\n\n* We need to inherit from nn.Module.\n* The constructor of the class defines the layers of the NN.\n* The `forward()` function defines how to forward propagate input through the defined layers (overrides predefined method).\n* Among the available layers, there are `Linear` (defines a fully connected layer), `Conv2d` (convolutional), `MaxPool2d`, etcetera. \n* Among the available activation we have `ReLU`, `Softmax` and `Sigmoid`.\n* The following example defines a MLP with a single layer:\n\n\n    class MLP(nn.Module):\n\n        def __init__(self, n_inputs):\n            super(MLP, self).__init__()\n            self.layer = Linear(n_inputs, 1)\n            self.activation = Sigmoid()\n\n        def forward(self, X):\n            X = self.layer(X)\n            X = self.activation(X)\n            return X\n"},{"metadata":{},"cell_type":"markdown","source":"## Description of the structure of the SAKT model\n\n* It consists of several blocks. The last block is a Feed-forward network (MLP with one hidden layer).\n    - The FFN produces the output and receives an already processed input.\n* In the SAKT model, we have the following notation:\n    - $\\mathbf{X} = (\\mathbf{x}_1, \\dots, \\mathbf{x}_n)$ is the student's past iterations (vector).\n    - $\\mathbf{x}_i = (e_i, r_t)$ is the iteration at time $t$, where $e_t$ is the exercised posed to the student and $r_t$ is the correctness of the answer.\n* Thus, our objective is to predict $r_{t + 1}$ or estimate $P[r_{t + 1} = 1 | e_{t + 1}, \\mathbf{X}]$.\n* We must consider the inputs as:--\n* With respect to the layers, we have the following layers:\n    - Embedding layer: We transform the input sequence $y$ into $s$, where $n$ is the maximum length.\n    - Self-attention layer: \n    - FFN: it receives $S$ (the output of the self-attention layer) as input. It has only one hidden layer with ReLU activation functions and one linear output function.\n    - Residual connections: residual connections are applied to both self-attetion and FFN layers. According to the paper, residual connections allow to propagate embeddings of resently solved exercises to the final layer, making it easier for model to leverage the low layer information.\n    - Layer normalization: it's applied to the inputs of the general network, to the input of the self-attention layer and the input of the FFN layer. Accoding to the paper, it stabilizes and accelerates the NN.\n    - \n* Observe we don't put the sigmoid function on the last layer of the SAKT model. Instead, we use only the dot product of the last layer and then use BCEWithLogitsLoss (which incorporates a Sigmoid layer and BCELoss in a single class, as this is more numerically stable because it can use a logarithm trick).\n    \n    \n**Note**: the first snippet of code defines the FFN and the `future_mask`.\n\n**Note**: where is the Self-attention layer? It's only defined the Multihead Attention"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The first class, FFN, defines a feed-forward network (hence, a MLP with dropout).\n# However, the FFN doesn't has the dataset info as input.\n\nclass FFN(nn.Module):\n    def __init__(self, state_size=200):\n        # This NN has only 2 layers. They are densely connected.\n        # The first layer has ReLU as activation function.\n        # The second layer has Dropout.\n        super(FFN, self).__init__()\n        self.state_size = state_size\n            # state_size only refers to the number of inputs. Hence\n            # we have the same number of inputs as of hidden neurons and\n            # output neurons.\n        \n        self.lr1 = nn.Linear(state_size, state_size)\n            # Self-explanatory\n        self.relu = nn.ReLU()\n            # Self-explanatory\n        self.lr2 = nn.Linear(state_size, state_size)\n            # Having the second Linear layer only creates an output layer with\n            # identity activation function\n        self.dropout = nn.Dropout(0.2)\n            # Note on dropout: it randomly zeroes some of the inputs\n            # during training. Observe the shape of input and output is the same.\n\n    def forward(self, x):\n        # we simply activate propagate forward\n        x = self.lr1(x)\n        x = self.relu(x)\n        x = self.lr2(x)\n        return self.dropout(x)\n\n# This creates the mask. I need to read the paper first\ndef future_mask(seq_length):\n    # We first create a matrix of ones. Triu computes the upper triangular matrix\n    # with k = 1 it means that the values below the 1st diagonal are zero.\n    future_mask = np.triu(np.ones((seq_length, seq_length)), k=1).astype('bool')\n    # The we convert the matrix into a tensor (so that PyTorch can process it).\n    return torch.from_numpy(future_mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This defines the whole SAKT model. Observe the last layer is a FFN network.\nclass SAKTModel(nn.Module):\n    def __init__(self, n_questions, max_seq=100, embed_dim=128):\n        # It takes three parameters: n_questions, max_seq and embed_dim\n        super(SAKTModel, self).__init__()\n        # We first initialize every parameter\n        self.n_questions = n_questions\n        self.embed_dim = embed_dim\n        \n        # From docs: A simple lookup table that stores embeddings of a fixed dictionary and size.\n        # First argument `num_embeddings` is the size of the dictionary of embeddings.\n        # The second argument `embedding_dim` is the size of each embedding vector.\n        # This creates the embeddings: observe that all the embedding layers are created with the\n        # torch layer nn.Embedding.\n        # - embeding is a traditional embedding: from dictionary of questions to dimension d.\n        # - pos_embeding is used to ############: from #### to dimension d.\n        #       why we use that?\n        # - e_embedding is similar to the first, but we don't differ using correctness.\n        self.embedding = nn.Embedding(2*n_questions+1, embed_dim)\n        self.pos_embedding = nn.Embedding(max_seq-1, embed_dim)\n        self.e_embedding = nn.Embedding(n_questions+1, embed_dim)\n        \n        # This creates the Attention layer:\n        # This defines the Multihead Attention as in the paper (and I believe is the same definition\n        # as the first paper that defined the Multihead Attention).\n        self.multi_att = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=8, dropout=0.2)\n        \n        # Dropout to the attention output with p = 0.2\n        self.dropout = nn.Dropout(0.2)\n        # Layer normalization (LayerNorm).\n        self.layer_normal = nn.LayerNorm(embed_dim) \n        \n        # Defined FFN:\n        self.ffn = FFN(embed_dim)\n        \n        # The output is the linear of the outputs of FFN\n        # Where's the sigmoid?\n        self.pred = nn.Linear(embed_dim, 1)\n    \n    def forward(self, x, question_ids):\n        # x is a tensor\n        \n        # Calling x.device returns device(type='cuda', index=0) in case it was defined\n        # this way.\n        device = x.device\n        # Transforms the tensor using the embedding.\n        x = self.embedding(x)\n        \n        # This does several things:\n        # Creates a 1D tensor that goes from 0 to k, where k is the second dimension of x\n        # Then inserts the dimension (converts the array into a vector (2d, but only one row))\n        # Finally, moves the tensor to the device.\n        # Thus, the result is [[0, 1, 2, 3, 4, ..., k]] in the device\n        pos_id = torch.arange(x.size(1)).unsqueeze(0).to(device)\n        \n        # This creates the embedding of the position\n        pos_x = self.pos_embedding(pos_id)\n        # We add the position and the input tensor x (Seems like a Res Connection, but pos_x isn't\n        # a layer-modification of x).\n        x = x + pos_x\n        \n        # Finally, we make an embedding of the question_ids.\n        e = self.e_embedding(question_ids)\n        \n        # NOTE: question_ids is pretty similar to that of x. In fact, they are the same when the\n        # question's answer is wrong.\n\n        # Changes the dimensions of the tensor. Generalization of the transpose.\n        # Applied to both x and questions_id.\n        x = x.permute(1, 0, 2) # x: [bs, s_len, embed] => [s_len, bs, embed]\n        e = e.permute(1, 0, 2)\n        \n        # Creates the mask in order to avoid using the future\n        # to predict the past (I think).\n        att_mask = future_mask(x.size(0)).to(device)\n        att_output, att_weight = self.multi_att(e, x, x, attn_mask=att_mask)\n        # Note that «att_output + e» is a residual connection that \"jumps\" the ATT layer.\n        # Then we apply layer normalization to the output of the Res Connection.\n        att_output = self.layer_normal(att_output + e) # Residual connection\n        att_output = att_output.permute(1, 0, 2) # att_output: [s_len, bs, embed] => [bs, s_len, embed]\n\n        # We apply the FFN to the output of the Attention output.\n        x = self.ffn(att_output)\n        # Note that «x + att_output» is a residual connection that \"jumps\" the FFN.\n        # Then we apply layer normalization to the output of the Res Connection.\n        x = self.layer_normal(x + att_output) # Residual connection\n        # Finally, we perform the prediction (dense connection to a single neuron).\n        x = self.pred(x)\n\n        return x.squeeze(-1), att_weight","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With respect of particular hiperparameters:\n* The dimension of embeddings d is 128."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This line initializes the GPU (cuda).\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# This is the constructor of the model we defined.\nmodel = SAKTModel(n_questions, embed_dim=128)\n\n# We construct a optimizer object. In this case: ADAM.\n#\n# optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.99, weight_decay=0.005)\noptimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\ncriterion = nn.BCEWithLogitsLoss()\n\n# This line puts the model in the GPU.\nmodel.to(device)\n# This line puts the criterion in the GPU, also.\ncriterion.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We define a function that defines each train epoch.\ndef train_epoch(model, train_iterator, optim, criterion, device=\"cpu\"):\n    # This sets our NN in the training mode (activates dropout and so on)\n    model.train()\n\n    # This variables are using throughout the training. Self-explanatory\n    train_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n    \n    # Progress bar. Note that train_iterator is an argument of the function.\n    tbar = tqdm(train_iterator)\n    for item in tbar:\n        # Gets the first item of train_iterator and sends it to the device and converts\n        # it to long type.\n        x = item[0].to(device).long()\n        # Similar, but with the second index (of the first item).\n        target_id = item[1].to(device).long()\n        # Idem, but with the third index and converting it to float.\n        label = item[2].to(device).float()\n        # Creates the mask:\n        target_mask = (target_id != 0)\n\n        # We set the gradients to 0.\n        optim.zero_grad()\n        # Get the output and weights of the model.\n        output, atten_weight = model(x, target_id)\n        \n        # Don't know        \n        output = torch.masked_select(output, target_mask)\n        # Don't know\n        label = torch.masked_select(label, target_mask)\n        \n        loss = criterion(output, label)\n        loss.backward()\n        optim.step()\n        train_loss.append(loss.item())\n        pred = (torch.sigmoid(output) >= 0.5).long()\n        \n        num_corrects += (pred == label).sum().item()\n        num_total += len(label)\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(output.view(-1).data.cpu().numpy())\n\n        tbar.set_description('loss - {:.4f}'.format(loss))\n    \n    # After looping, we can finally compute the accuracy and loss:\n    #\n    # acc stands for Accuracy (proportion of right answers)\n    acc = num_corrects / num_total\n    # Area under the ROC curve\n    auc = roc_auc_score(labels, outs)\n    # Loss value using the loss function for training (BCE Loss).\n    loss = np.average(train_loss)\n\n    return loss, acc, auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This defines the epoch of validation.\ndef val_epoch(model, val_iterator, criterion, device=\"cpu\"):\n    model.eval()\n\n    train_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n\n    tbar = tqdm(val_iterator)\n    for item in tbar:\n        x = item[0].to(device).long()\n        target_id = item[1].to(device).long()\n        label = item[2].to(device).float()\n        target_mask = (target_id != 0)\n\n        with torch.no_grad():\n            output, atten_weight = model(x, target_id)\n        \n        output = torch.masked_select(output, target_mask)\n        label = torch.masked_select(label, target_mask)\n\n        loss = criterion(output, label)\n        train_loss.append(loss.item())\n\n        pred = (torch.sigmoid(output) >= 0.5).long()\n        \n        num_corrects += (pred == label).sum().item()\n        num_total += len(label)\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(output.view(-1).data.cpu().numpy())\n\n        tbar.set_description('loss - {:.4f}'.format(loss))\n\n    acc = num_corrects / num_total\n    auc = roc_auc_score(labels, outs)\n    loss = np.average(train_loss)\n\n    return loss, acc, auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"over_fit = 0\nlast_auc = 0\nfor epoch in range(epochs):\n    train_loss, train_acc, train_auc = train_epoch(model, train_dataloader, optimizer, criterion, device)\n    print(\"epoch - {} train_loss - {:.2f} acc - {:.3f} auc - {:.3f}\".format(epoch, train_loss, train_acc, train_auc))\n    \n    val_loss, avl_acc, val_auc = val_epoch(model, val_dataloader, criterion, device)\n    print(\"epoch - {} val_loss - {:.2f} acc - {:.3f} auc - {:.3f}\".format(epoch, val_loss, avl_acc, val_auc))\n    \n    if val_auc > last_auc:\n        last_auc = val_auc\n        over_fit = 0\n    else:\n        over_fit += 1\n        \n    \n    if over_fit >= 2:\n        print(\"early stop epoch \", epoch)\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), \"SAKT-Capo.pt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"#del dataset\n#gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Yield of the predictions for the model\n\n* Since we don't have `answered_correctly` in the test dataset, we have to define a new class for testing the model.\n* In this case, "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Then we create the dataset for the test. This is similar to that of the training, without answers.\nclass TestDataset(Dataset):\n    # Let's see if we do the same.\n    def __init__(self, samples, test_df, questions, max_seq=MAX_SEQ):\n        super(TestDataset, self).__init__()\n        self.samples = samples\n        self.user_ids = [x for x in test_df[\"user_id\"].unique()]\n        self.test_df = test_df\n        self.questions = questions\n        self.n_questions = len(questions)\n        self.max_seq = max_seq\n\n    def __len__(self):\n        return self.test_df.shape[0]\n\n    def __getitem__(self, index):\n        test_info = self.test_df.iloc[index]\n\n        user_id = test_info[\"user_id\"]\n        target_id = test_info[\"content_id\"]\n\n        q = np.zeros(self.max_seq, dtype=int)\n        qa = np.zeros(self.max_seq, dtype=int)\n\n        if user_id in self.samples.index:\n            q_, qa_ = self.samples[user_id]\n            \n            seq_len = len(q_)\n\n            if seq_len >= self.max_seq:\n                q = q_[-self.max_seq:]\n                qa = qa_[-self.max_seq:]\n            else:\n                q[-seq_len:] = q_\n                qa[-seq_len:] = qa_          \n        \n        x = np.zeros(self.max_seq-1, dtype=int)\n        x = q[1:].copy()\n        x += (qa[1:] == 1) * self.n_questions\n        \n        questions = np.append(q[2:], [target_id])\n        \n        return x, questions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We finally import the modules of the competition. They are needed to submit the predictions.\nimport riiideducation\n\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import psutil\n\nmodel.eval()\n\nprev_test_df = None\n\nfor (test_df, sample_prediction_df) in tqdm(iter_test):\n    # What\n    if (prev_test_df is not None) & (psutil.virtual_memory().percent<90):\n        print(psutil.virtual_memory().percent)\n        prev_test_df['answered_correctly'] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        prev_test_df = prev_test_df[prev_test_df.content_type_id == False]\n        prev_group = prev_test_df[['user_id', 'content_id', 'answered_correctly']].groupby('user_id').apply(lambda r: (\n            r['content_id'].values,\n            r['answered_correctly'].values))\n        for prev_user_id in prev_group.index:\n            prev_group_content = prev_group[prev_user_id][0]\n            prev_group_ac = prev_group[prev_user_id][1]\n            if prev_user_id in group.index:\n                group[prev_user_id] = (np.append(group[prev_user_id][0],prev_group_content), \n                                       np.append(group[prev_user_id][1],prev_group_ac))\n \n            else:\n                group[prev_user_id] = (prev_group_content,prev_group_ac)\n            if len(group[prev_user_id][0])>MAX_SEQ:\n                new_group_content = group[prev_user_id][0][-MAX_SEQ:]\n                new_group_ac = group[prev_user_id][1][-MAX_SEQ:]\n                group[prev_user_id] = (new_group_content,new_group_ac)\n\n    prev_test_df = test_df.copy()\n    \n    test_df = test_df[test_df.content_type_id == False]\n    \n    test_dataset = TestDataset(group, test_df, questions)\n    test_dataloader = DataLoader(test_dataset, batch_size=51200, shuffle=False)\n    \n    outs = []\n\n    for item in tqdm(test_dataloader):\n        x = item[0].to(device).long()\n        target_id = item[1].to(device).long()\n\n        with torch.no_grad():\n            output, att_weight = model(x, target_id)\n        \n        \n        output = torch.sigmoid(output)\n        output = output[:, -1]\n\n        # pred = (output >= 0.5).long()\n        # loss = criterion(output, label)\n\n        # val_loss.append(loss.item())\n        # num_corrects += (pred == label).sum().item()\n        # num_total += len(label)\n\n        # labels.extend(label.squeeze(-1).data.cpu().numpy())\n        outs.extend(output.view(-1).data.cpu().numpy())\n        \n    test_df['answered_correctly'] =  outs\n    \n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The following code were celds used to produce the dummy prediction:\n#\n'''\n(test_df, sample_prediction_df) = next(iter_test)\ntest_df\n'''\n#\n'''\nsample_prediction_df\n'''\n#\n'''\nenv.predict(sample_prediction_df)\n'''\n#\n'''\nfor (test_df, sample_prediction_df) in iter_test:\n    test_df['answered_correctly'] = 0.5\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Notes on the score\n\n* The score is based on the area under the ROC curve. At 28/12/2020, the highest (public) score is of 0.816. The lowest possible score to achieve a silve medal is 0.786 and to achieve a bronze medal is 0.783. The public notebook with higher score has a score of 0.783\n\n## Track of attempts\n\n* The first attempt was done by using the dummy prediction (predicting 0.5 for all possible observations). This led to a score of **0.500**. Note this wasn't the worst score, using a poor model could lead to a lower score (there are scores of 0.425 at the time of this writing).\n* The second attempt uses SAKT model based on this [notebook](kaggle.com/wangsg/a-self-attentive-model-for-knowledge-tracing/notebook). Simply replicates the model an gets a public score of **0.764**.\n* The third attempt is based on this [notebook](https://www.kaggle.com/leadbest/sakt-with-randomization-state-updates). It uses random selection of user iterations and small optimization to get a score of 0.771. Since it uses the exact same features as the previous notebook, I'm going to also .... "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}