{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport gc\nimport random\nfrom tqdm import tqdm\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\nimport os\nimport copy\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.utils.rnn as rnn_utils\nfrom torch.autograd import Variable\nfrom torch.utils.data import Dataset, DataLoader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Introduction and Credit\n\nI had a lot of fun with this competition, and I'm disappointed it's over because I was making really good progress towards the end. My final submission got up to 0.798, but it finished scoring after the competition had ended. I'm sure I could have gotten this well above 0.8 if I had the time. There are still a number of ideas I didn't get the chance to implement. \n\nThe following kernels and github were instrumental in building this model.\n\nhttps://www.kaggle.com/wangsg/a-self-attentive-model-for-knowledge-tracing\n\nhttps://www.kaggle.com/mpware/sakt-fork\n\nhttps://github.com/arshadshk/SAINT-pytorch"},{"metadata":{},"cell_type":"markdown","source":"## Load data"},{"metadata":{},"cell_type":"markdown","source":"The poorly named train-df-saint-not-binned contains a pre processed feather version of train_df. In that kernel, I calculate all the interesting features, then create artificial users for any user with more than seq_len interactions. So if a user had 160 interactions, I split this into two users: One with the final 100 interactions, and one with the first 60 interactions. This strategy seemed to work well when applied to my SAKT fork, so I kept it with my SAINT model. As a result of this, instead of 39000 users, the model trains on 120000 \"users\" or so. \n\nGiven more time, I would have liked to experiment with other ways of creating sequences for the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntrain_df = pd.read_feather('../input/train-df-saint-not-binned-everything/train_not_binned')\ntrain_df['prior_question_elapsed_time'].fillna(0, inplace=True)\ntrain_df['prior_question_elapsed_time'] /= 1000\ntrain_df['prior_question_elapsed_time'] = train_df['prior_question_elapsed_time'].round()\ntrain_df['prior_question_elapsed_time'] = train_df['prior_question_elapsed_time'].astype('int16')\n\ntrain_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocess"},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET = 'answered_correctly'\nMAX_SEQ = 100\n# content_ids = train_df[\"content_id\"].unique()\nNUM_QUESTIONS = len(train_df[\"content_id\"].unique()) + 1\nNUM_USERS = len(train_df['user_id'].unique())\nNUM_LAG1S = train_df['lag1'].max() + 1\nNUM_LAG2S = train_df['lag2'].max() + 1\nNUM_LAG3S = train_df['lag3'].max() + 1\nELAPSED_TIMES = train_df['prior_question_elapsed_time'].max() + 1\nMODEL_BEST = 'model_best.pt'\nBS = 1024","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Creates a Series with user_ids as indices and a tuple of all the content_ids and answered_correctlys as lists\ndef create_group(df):\n    return df[['user_id', 'content_id', 'answered_correctly', 'lag1', 'lag2', 'lag3', 'part', 'prior_question_elapsed_time']].groupby('user_id').apply(lambda r: [\n        r['content_id'].values,\n        r['answered_correctly'].values,\n        r['lag1'].values,\n        r['lag2'].values,\n        r['lag3'].values,\n        r['part'].values,\n        r['prior_question_elapsed_time'].values])\n\ntrain_group = create_group(train_df)\n\ndel train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_group = train_group.sample(frac=0.03)\ntrain_group = train_group.drop(valid_group.index).reset_index(drop=True)\nvalid_group.reset_index(drop=True, inplace=True)\ntrain_group.shape, valid_group.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SAINTDataset(Dataset):\n    def __init__(self, user_sequences, num_questions, subset='train', max_seq=100, min_seq=10):\n        super(SAINTDataset, self).__init__()\n        self.max_seq = max_seq\n        self.num_questions = num_questions\n        self.user_sequences = user_sequences\n        self.subset = subset\n\n        self.user_ids = []\n        for user_id in user_sequences.index:\n            q, _, _, _, _, _, _ = user_sequences[user_id]\n            if len(q) < min_seq:\n                continue\n            self.user_ids.append(user_id)\n\n    def __len__(self):\n        return len(self.user_ids)\n\n    def __getitem__(self, index):\n        user_id = self.user_ids[index]\n        # question_id, answered_correctly, lag1, lag2, lag3, part, elapsed_time\n        q_, qa_, l1_, l2_, l3_, p_, el_ = self.user_sequences[user_id]\n        seq_len = len(q_)\n\n        q = np.zeros(self.max_seq, dtype=int)\n        qa = np.zeros(self.max_seq, dtype=int)\n        l1 = np.zeros(self.max_seq, dtype=int)\n        l2 = np.zeros(self.max_seq, dtype=int)\n        l3 = np.zeros(self.max_seq, dtype=int)\n        p = np.zeros(self.max_seq, dtype=int)\n        el = np.zeros(self.max_seq, dtype=int)\n        \n#         # If there are more questions answered than max_seq, take the last max_seq sequences\n        if seq_len >= self.max_seq:\n            q[:] = q_[-self.max_seq:]\n            qa[:] = qa_[-self.max_seq:]\n            l1[:] = l1_[-self.max_seq:]\n            l2[:] = l2_[-self.max_seq:]\n            l3[:] = l3_[-self.max_seq:]\n            p[:] = p_[-self.max_seq:]\n            el[:] = el_[-self.max_seq:]\n        # If not, map our user_sequences to the tail end of q and qa, the start will be padded with zeros\n        else:\n            q[-seq_len:] = q_\n            qa[-seq_len:] = qa_\n            l1[-seq_len:] = l1_\n            l2[-seq_len:] = l2_\n            l3[-seq_len:] = l3_\n            el[-seq_len:] = el_\n        \n        r = np.zeros(self.max_seq, dtype=int)\n        r[1:] = qa[:-1].copy()\n        \n        return q, r, qa, l1, l2, l3, p, el ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define model"},{"metadata":{"trusted":true},"cell_type":"code","source":"class FFN(nn.Module):\n    def __init__(self, dim=128):\n        super().__init__()\n        self.layer1 = nn.Linear(dim, dim)\n        self.layer2 = nn.Linear(dim, dim)\n        self.relu = nn.ReLU()\n    \n    def forward(self, x):\n        return self.layer2(   self.relu(   self.layer1(x)))\n\n    \ndef future_mask(seq_length):\n    future_mask = np.triu(np.ones((seq_length, seq_length)), k=1).astype('bool')\n    return torch.from_numpy(future_mask)\n\n\nclass Encoder(nn.Module):\n    def __init__(self, n_in, seq_len=100, embed_dim=128, nheads=4):\n        super().__init__()\n        self.seq_len = seq_len\n\n        self.part_embed = nn.Embedding(10, embed_dim)\n        \n        self.e_embed = nn.Embedding(n_in, embed_dim)\n        self.e_pos_embed = nn.Embedding(seq_len, embed_dim)\n        self.e_norm = nn.LayerNorm(embed_dim)\n        \n        self.e_multi_att = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=nheads, dropout=0.2)\n        self.m_norm = nn.LayerNorm(embed_dim)\n        self.ffn = FFN(embed_dim)\n    \n    def forward(self, e, p, first_block=True):\n        \n        if first_block:\n            e = self.e_embed(e)\n            p = self.part_embed(p)\n            e = e + p\n         \n        pos = torch.arange(self.seq_len).unsqueeze(0).to(device)\n        e_pos = self.e_pos_embed(pos)\n        e = e + e_pos\n        e = self.e_norm(e)\n        e = e.permute(1,0,2) #[bs, s_len, embed] => [s_len, bs, embed]     \n        n = e.shape[0]\n        \n        att_mask = future_mask(n).to(device)\n        att_out, _ = self.e_multi_att(e, e, e, attn_mask=att_mask)\n        m = e + att_out\n        m = m.permute(1,0,2)\n        \n        o = m + self.ffn(self.m_norm(m))\n        \n        return o\n    \nclass Decoder(nn.Module):\n    def __init__(self, n_in, seq_len=100, embed_dim=128, nheads=4):\n        super().__init__()\n        self.seq_len = seq_len\n        \n        self.r_embed = nn.Embedding(n_in, embed_dim)\n        self.r_pos_embed = nn.Embedding(seq_len, embed_dim)\n        self.r_norm = nn.LayerNorm(embed_dim)\n        \n        self.l1_embed = nn.Embedding(NUM_LAG1S, embed_dim)\n        self.l2_embed = nn.Embedding(NUM_LAG2S, embed_dim)\n        self.l3_embed = nn.Embedding(NUM_LAG3S, embed_dim)\n        self.el_t_embed = nn.Embedding(ELAPSED_TIMES, embed_dim)\n        \n        self.r_multi_att1 = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=4, dropout=0.2)\n        self.r_multi_att2 = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=4, dropout=0.2)\n        self.ffn = FFN(embed_dim)\n        \n        self.r_norm1 = nn.LayerNorm(embed_dim)\n        self.r_norm2 = nn.LayerNorm(embed_dim)\n        self.r_norm3 = nn.LayerNorm(embed_dim)\n\n    \n    def forward(self, r, o, l1, l2, l3, el, first_block=True):\n        \n        if first_block:\n            r = self.r_embed(r)\n            l1 = self.l1_embed(l1)\n            l2 = self.l2_embed(l2)\n            l3 = self.l3_embed(l3)\n            el = self.el_t_embed(el)\n\n            r = r + l1 + l2 + l3 + el\n  \n        pos = torch.arange(self.seq_len).unsqueeze(0).to(device)\n        r_pos_embed = self.r_pos_embed(pos)\n        r = r + r_pos_embed\n        r = self.r_norm1(r) \n        r = r.permute(1,0,2)   \n        n = r.shape[0]\n   \n        att_out1, _ = self.r_multi_att1(r, r, r, attn_mask=future_mask(n).to(device))\n        m1 = r + att_out1\n\n        o = o.permute(1,0,2)\n        o = self.r_norm2(o)\n        att_out2, _ = self.r_multi_att2(m1, o, o, attn_mask=future_mask(n).to(device))\n        \n        m2 = att_out2 + m1\n        m2 = m2.permute(1,0,2)        \n        m2 = self.r_norm3(m2)\n        \n        l = m2 + self.ffn(m2)\n        \n        return l\n\n# This is an altered version from https://github.com/arshadshk/SAINT-pytorch\ndef get_clones(module, N):\n    return nn.ModuleList([copy.deepcopy(module) for i in range(N)])\n\nclass SAINT(nn.Module):\n    def __init__(self, dim_model, num_en, num_de, heads_en, total_ex, total_in, heads_de, seq_len):\n        super().__init__()\n        \n        self.num_en = num_en\n        self.num_de = num_de\n\n        self.encoder = get_clones( Encoder(n_in=total_ex, seq_len=seq_len, embed_dim=dim_model, nheads=heads_en) , num_en)\n        self.decoder = get_clones( Decoder(n_in=total_in, seq_len=seq_len, embed_dim=dim_model, nheads=heads_de) , num_de)\n\n        self.out = nn.Linear(in_features= dim_model , out_features=1)\n    \n    def forward(self, in_ex, in_in, l1, l2, l3, p, el):\n        \n        ## pass through each of the encoder blocks in sequence\n        first_block = True\n        for x in range(self.num_en):\n            if x>=1:\n                first_block = False\n            in_ex = self.encoder[x](in_ex, p, first_block=first_block)\n        \n        ## pass through each decoder blocks in sequence\n        first_block = True\n        for x in range(self.num_de):\n            if x>=1:\n                first_block = False\n            in_in = self.decoder[x]( in_in , in_ex, l1, l2, l3, el, first_block=first_block )\n\n        ## Output layer\n        in_in = torch.sigmoid( self.out( in_in ) )\n        return in_in.squeeze(-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_iterator is our dataloader, criterion is nn.BCEWithLogitsLoss\ndef train_epoch(model, train_iterator, optim, criterion, device=\"cpu\"):\n    model.train()\n\n    train_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n\n    tbar = tqdm(train_iterator)\n    for item in tbar:\n        e = item[0].to(device).long()\n        r = item[1].to(device).long()\n        label = item[2].to(device).float()\n        l1 = item[3].to(device).long()\n        l2 = item[4].to(device).long()\n        l3 = item[5].to(device).long()\n        p = item[6].to(device).long()\n        el = item[7].to(device).long()\n\n        # Zero the gradients in the optimizer\n        optim.zero_grad()\n        # The results of one forward pass\n        output = model(e, r, l1, l2, l3, p, el)\n        # Calculate the loss\n        loss = criterion(output, torch.sigmoid(label))\n        # Calculate the gradients with respect to the loss\n        loss.backward()\n        # Adjust the parameters to minimize the loss based on these gradients\n        optim.step()\n        # Add our loss to the list of losses\n        train_loss.append(loss.item())\n\n        output = output[:, -1]\n        label = label[:, -1] \n        pred = (output >= 0.5).long()\n         \n        num_corrects += (pred == label).sum().item()\n        num_total += len(label)\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(output.view(-1).data.cpu().numpy())\n\n        tbar.set_description('loss - {:.4f}'.format(loss))\n\n    acc = num_corrects / num_total\n    auc = roc_auc_score(labels, outs)\n    loss = np.mean(train_loss)\n\n    return loss, acc, auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/mpware/sakt-fork\ndef valid_epoch(model, valid_iterator, criterion, device=\"cpu\"):\n    model.eval()\n\n    valid_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n\n    #tbar = tqdm(valid_iterator)\n    for item in valid_iterator: # tbar:\n        e = item[0].to(device).long()\n        r = item[1].to(device).long()\n        label = item[2].to(device).float()\n        l1 = item[3].to(device).long()\n        l2 = item[4].to(device).long()\n        l3 = item[5].to(device).long()\n        p = item[6].to(device).long()\n        el = item[7].to(device).long()\n\n        with torch.no_grad():\n            output = model(e, r, l1, l2, l3, p, el)\n        loss = criterion(output, torch.sigmoid(label))\n        valid_loss.append(loss.item())\n\n        output = output[:, -1] # (BS, 1)\n        label = label[:, -1] \n        pred = (output >= 0.5).long()\n        \n        num_corrects += (pred == label).sum().item()\n        num_total += len(label)\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(output.view(-1).data.cpu().numpy())\n\n    acc = num_corrects / num_total\n    auc = roc_auc_score(labels, outs)\n    loss = np.mean(valid_loss)\n\n    return loss, acc, auc\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()\ntrain_dataset = SAINTDataset(train_group, NUM_QUESTIONS, max_seq=MAX_SEQ)\ntrain_dataloader = DataLoader(train_dataset, batch_size=BS, shuffle=True, num_workers=8)\n\nvalid_dataset = SAINTDataset(valid_group, NUM_QUESTIONS, max_seq=MAX_SEQ, subset='valid')\nvalid_dataloader = DataLoader(valid_dataset, batch_size=BS, shuffle=False, num_workers=8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = SAINT(dim_model=128,\n            num_en=2,\n            num_de=2,\n            heads_en=4,\n            heads_de=4,\n            total_ex=NUM_QUESTIONS, \n            total_in=2,\n            seq_len=100\n            )\n\noptimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\ncriterion = nn.BCELoss() \n\nmodel.to(device)\ncriterion.to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()\nepochs = 30\nhistory = []\nauc_max = -np.inf\n\nfor epoch in range(1, epochs+1):\n    train_loss, train_acc, train_auc = train_epoch(model, train_dataloader, optimizer, criterion, device)\n    print(f'Epoch {epoch}, train_loss: {train_loss:5f}, train_acc: {train_acc:5f}, train_auc: {train_auc:5f}')\n    valid_loss, valid_acc, valid_auc = valid_epoch(model, valid_dataloader, criterion, device)\n    print(f'Epoch {epoch}, valid_loss: {valid_loss:5f}, valid_acc: {valid_acc:5f}, valid_auc: {valid_auc:5f}')\n    \n    lr = optimizer.param_groups[0]['lr']\n    history.append({\"epoch\":epoch, \"lr\": lr, **{\"train_auc\": train_auc, \"train_acc\": train_acc}, **{\"valid_auc\": valid_auc, \"valid_acc\": valid_acc}})\n    if valid_auc > auc_max:\n        print(\"Epoch#%s, valid loss %.4f, Metric loss improved from %.4f to %.4f, saving model ...\" % (epoch, valid_loss, auc_max, valid_auc))\n        auc_max = valid_auc\n        torch.save(model.state_dict(), MODEL_BEST)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Conclusion\n\nAlthough I'm happy with how the competition went, I still think I could have improved on my model quite a bit. \n\nI had issues using a continuous representation of the prior_question_elapsed_time feature, so in the end I just left the categorical version. The SAINT+ paper found that this feature worked marginally better with a continuous representation, so I would have liked to get that working.\n\nI'm also sure that my lag features could have been engineered better. Since several questions were given together in a bundle, they had the same timestamps. So in using a simple lag such as lag = t<sub>n</sub> - t<sub>n-1</sub>, you get a lot of lag=0, which doesn't give much signal to the model. It also feeds the model sequences that can change, depending on how the group was formed. So I would have liked to experiment with this a bit.\n\nFinally, there was still a gap between my LB score and my training scores. So I'm sure there were still some issues to resolve on that front.\n\nIf anyone has any comments or questions, I'd love to hear them. Thanks for looking at my kernel!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}