{"cells":[{"metadata":{},"cell_type":"markdown","source":"# RIIID - LightGBM and SAKT Ensemble Inference\n\n### If you like this kernel or forking this kernel, please consider upvoting this and the kernels I copied (acknowledgements) from. It helps them reach more people. \n\n\n## SAKT Model\n- Public Leaderboard Score: 0.773\n- **Pretrained Dataset**: https://www.kaggle.com/manikanthr5/riiid-sakt-model-dataset-public/\n- **Acknowledgement**: All the credits go to this popular notebook https://www.kaggle.com/leadbest/sakt-with-randomization-state-updates which is a modification of https://www.kaggle.com/wangsg/a-self-attentive-model-for-knowledge-tracing. Please show some support to these original work kernels.\n- **Possible Improvements**:\n  - All the data in this notebook is used for training, so create a train and valid dataset for cross validation. Note: For me this degraded my LB score.\n- Some other text book ideas you could try:\n  - Using Label Smoothing\n  - Using Learning Rate Schedulers\n  - Increase the max sequence length and/or embedding dimension\n  - Add more attention layers \n  \n## LightGBM Model\n- Public Leaderboard Score: 0.760\n- **Training Notebook**: https://www.kaggle.com/manikanthr5/riiid-lgbm-single-model-ensembling-training\n- **Inference Notebook**: https://www.kaggle.com/manikanthr5/riiid-lgbm-single-model-ensembling-scoring\n- **Pretrained Dataset**: https://www.kaggle.com/manikanthr5/lgbm-with-loop-feature-engineering-dataset\n- **Acknowledgement**: I have modified this popular notebook https://www.kaggle.com/its7171/lgbm-with-loop-feature-engineering/\n- **Improvement Chances**: \n  - This notebook is pretty simple. Try to add features. Feature Engineering is the key to improving LightGBM Models. By adding good features to the training notebook it is possible to get LB 0.771+.\n  - Beaware of target leakage"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import gc\nimport psutil\nimport joblib\nimport numpy as np\nimport pandas as pd\n\nimport lightgbm as lgb\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set()\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LigthGBM Feature Update"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# funcs for user stats with loop\ndef add_user_feats_without_update(df, answered_correctly_sum_u_dict, count_u_dict):\n    acsu = np.zeros(len(df), dtype=np.int32)\n    cu = np.zeros(len(df), dtype=np.int32)\n    for cnt, row in enumerate(df[['user_id']].values):\n        acsu[cnt] = answered_correctly_sum_u_dict[row[0]]\n        cu[cnt] = count_u_dict[row[0]]\n    user_feats_df = pd.DataFrame({'answered_correctly_sum_u':acsu, 'count_u':cu})\n    user_feats_df['answered_correctly_avg_u'] = user_feats_df['answered_correctly_sum_u'] / user_feats_df['count_u']\n    df = pd.concat([df, user_feats_df], axis=1)\n    return df\n\ndef update_user_feats(df, answered_correctly_sum_u_dict, count_u_dict):\n    for row in df[['user_id','answered_correctly','content_type_id']].values:\n        if row[2] == 0:\n            answered_correctly_sum_u_dict[row[0]] += row[1]\n            count_u_dict[row[0]] += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import RIIID API"},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SAKT Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"MAX_SEQ = 100\n\nclass FFN(nn.Module):\n    def __init__(self, state_size=200):\n        super(FFN, self).__init__()\n        self.state_size = state_size\n\n        self.lr1 = nn.Linear(state_size, state_size)\n        self.relu = nn.ReLU()\n        self.lr2 = nn.Linear(state_size, state_size)\n        self.dropout = nn.Dropout(0.2)\n    \n    def forward(self, x):\n        x = self.lr1(x)\n        x = self.relu(x)\n        x = self.lr2(x)\n        return self.dropout(x)\n\ndef future_mask(seq_length):\n    future_mask = np.triu(np.ones((seq_length, seq_length)), k=1).astype('bool')\n    return torch.from_numpy(future_mask)\n\n\nclass SAKTModel(nn.Module):\n    def __init__(self, n_skill, max_seq=MAX_SEQ, embed_dim=128): #HDKIM 100\n        super(SAKTModel, self).__init__()\n        self.n_skill = n_skill\n        self.embed_dim = embed_dim\n\n        self.embedding = nn.Embedding(2*n_skill+1, embed_dim)\n        self.pos_embedding = nn.Embedding(max_seq-1, embed_dim)\n        self.e_embedding = nn.Embedding(n_skill+1, embed_dim)\n\n        self.multi_att = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=8, dropout=0.2)\n\n        self.dropout = nn.Dropout(0.2)\n        self.layer_normal = nn.LayerNorm(embed_dim) \n\n        self.ffn = FFN(embed_dim)\n        self.pred = nn.Linear(embed_dim, 1)\n    \n    def forward(self, x, question_ids):\n        device = x.device        \n        x = self.embedding(x)\n        pos_id = torch.arange(x.size(1)).unsqueeze(0).to(device)\n\n        pos_x = self.pos_embedding(pos_id)\n        x = x + pos_x\n\n        e = self.e_embedding(question_ids)\n\n        x = x.permute(1, 0, 2) # x: [bs, s_len, embed] => [s_len, bs, embed]\n        e = e.permute(1, 0, 2)\n        att_mask = future_mask(x.size(0)).to(device)\n        att_output, att_weight = self.multi_att(e, x, x, attn_mask=att_mask)\n        att_output = self.layer_normal(att_output + e)\n        att_output = att_output.permute(1, 0, 2) # att_output: [s_len, bs, embed] => [bs, s_len, embed]\n\n        x = self.ffn(att_output)\n        x = self.layer_normal(x + att_output)\n        x = self.pred(x)\n\n        return x.squeeze(-1), att_weight\n    \nclass TestDataset(Dataset):\n    def __init__(self, samples, test_df, skills, max_seq=MAX_SEQ): #HDKIM 100\n        super(TestDataset, self).__init__()\n        self.samples = samples\n        self.user_ids = [x for x in test_df[\"user_id\"].unique()]\n        self.test_df = test_df\n        self.skills = skills\n        self.n_skill = len(skills)\n        self.max_seq = max_seq\n\n    def __len__(self):\n        return self.test_df.shape[0]\n\n    def __getitem__(self, index):\n        test_info = self.test_df.iloc[index]\n\n        user_id = test_info[\"user_id\"]\n        target_id = test_info[\"content_id\"]\n\n        q = np.zeros(self.max_seq, dtype=int)\n        qa = np.zeros(self.max_seq, dtype=int)\n\n        if user_id in self.samples.index:\n            q_, qa_ = self.samples[user_id]\n            \n            seq_len = len(q_)\n\n            if seq_len >= self.max_seq:\n                q = q_[-self.max_seq:]\n                qa = qa_[-self.max_seq:]\n            else:\n                q[-seq_len:] = q_\n                qa[-seq_len:] = qa_          \n        \n        x = np.zeros(self.max_seq-1, dtype=int)\n        x = q[1:].copy()\n        x += (qa[1:] == 1) * self.n_skill\n        \n        questions = np.append(q[2:], [target_id])\n        \n        return x, questions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LightGBM Pretrained Imports\n\nThe pretrained weights are available in this dataset: https://www.kaggle.com/manikanthr5/lgbm-with-loop-feature-engineering-dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"answered_correctly_sum_u_dict = joblib.load(\"../input/lgbm-with-loop-feature-engineering-dataset/answered_correctly_sum_u_dict.pkl.zip\")\ncount_u_dict = joblib.load(\"../input/lgbm-with-loop-feature-engineering-dataset/count_u_dict.pkl.zip\")\n\nquestions_df = pd.read_feather('../input/lgbm-with-loop-feature-engineering-dataset/questions_df.feather')\ncontent_df = pd.read_feather('../input/lgbm-with-loop-feature-engineering-dataset/content_df.feather')\n\nprior_question_elapsed_time_mean = joblib.load(\"../input/lgbm-with-loop-feature-engineering-dataset/prior_question_elapsed_time_mean.pkl.zip\")\n\nTARGET = 'answered_correctly'\nFEATS = ['answered_correctly_avg_u', 'answered_correctly_sum_u', 'count_u', \n         'answered_correctly_avg_c', 'part', 'prior_question_had_explanation', \n         'prior_question_elapsed_time'\n        ]\nlgbm_model = lgb.Booster(model_file=\"../input/lgbm-with-loop-feature-engineering-dataset/fold0_lgb_model.txt\")\nlgbm_model.best_iteration = joblib.load(\"../input/lgbm-with-loop-feature-engineering-dataset/fold0_lgb_model_best_iteration.pkl.zip\")\noptimized_weights = joblib.load(\"../input/lgbm-with-loop-feature-engineering-dataset/optimized_weights.pkl.zip\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance = lgbm_model.feature_importance()\nfeature_importance = pd.DataFrame(\n    {'Features': FEATS, 'Importance': feature_importance}\n).sort_values('Importance', ascending = False)\n\nfig = plt.figure(figsize = (8, 6))\nfig.suptitle('Feature Importance', fontsize = 20)\nplt.tick_params(axis = 'x', labelsize = 12)\nplt.tick_params(axis = 'y', labelsize = 12)\nplt.xlabel('Importance', fontsize = 15)\nplt.ylabel('Features', fontsize = 15)\nsns.barplot(\n    x=feature_importance['Importance'], \n    y=feature_importance['Features'], \n    orient='h'\n)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SAKT Pretrained Model Imports"},{"metadata":{},"cell_type":"markdown","source":"The pretrained weights are from this dataset: https://www.kaggle.com/manikanthr5/riiid-sakt-model-dataset-public/"},{"metadata":{"trusted":true},"cell_type":"code","source":"skills = joblib.load(\"../input/riiid-sakt-model-dataset-public/skills.pkl.zip\")\nn_skill = len(skills)\nprint(\"number skills\", len(skills))\ngroup = joblib.load(\"../input/riiid-sakt-model-dataset-public/group.pkl.zip\")\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nsakt_model = SAKTModel(n_skill, embed_dim=128)\ntry:\n    sakt_model.load_state_dict(torch.load(\"../input/riiid-sakt-model-dataset-public/sakt_model.pt\"))\nexcept:\n    sakt_model.load_state_dict(torch.load(\"../input/riiid-sakt-model-dataset-public/sakt_model.pt\", map_location='cpu'))\n\nsakt_model.to(device)\nsakt_model.eval()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"prev_test_df = None\n\nfor (test_df, sample_prediction_df) in iter_test:\n    if (prev_test_df is not None) & (psutil.virtual_memory().percent < 90):\n        prev_test_df['answered_correctly'] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        prev_test_df = prev_test_df[prev_test_df.content_type_id == False]\n        \n        # This is for SAKT\n        prev_group = prev_test_df[['user_id', 'content_id', 'answered_correctly']].groupby('user_id').apply(lambda r: (\n            r['content_id'].values,\n            r['answered_correctly'].values))\n        for prev_user_id in prev_group.index:\n            if prev_user_id in group.index:\n                group[prev_user_id] = (\n                    np.append(group[prev_user_id][0], prev_group[prev_user_id][0])[-MAX_SEQ:], \n                    np.append(group[prev_user_id][1], prev_group[prev_user_id][1])[-MAX_SEQ:]\n                )\n \n            else:\n                group[prev_user_id] = (\n                    prev_group[prev_user_id][0], \n                    prev_group[prev_user_id][1]\n                )\n                   \n        # This is for LGBM\n        update_user_feats(prev_test_df, answered_correctly_sum_u_dict, count_u_dict)\n\n    prev_test_df = test_df.copy()\n    test_df = test_df[test_df.content_type_id == False]\n    test_df.reset_index(drop=True)\n    \n    # This is for SAKT\n    test_dataset = TestDataset(group, test_df, skills)\n    test_dataloader = DataLoader(test_dataset, batch_size=51200, shuffle=False)\n    \n    sakt_predictions = []\n\n    for item in test_dataloader:\n        x = item[0].to(device).long()\n        target_id = item[1].to(device).long()\n\n        with torch.no_grad():\n            output, att_weight = sakt_model(x, target_id)\n        sakt_predictions.extend(torch.sigmoid(output)[:, -1].view(-1).data.cpu().numpy())\n    \n    # This is for LGBM\n    test_df.reset_index(drop=True, inplace=True)\n    test_df = add_user_feats_without_update(test_df, answered_correctly_sum_u_dict, count_u_dict)\n    test_df = pd.merge(test_df, content_df, on='content_id',  how=\"left\")\n    test_df = pd.merge(test_df, questions_df, left_on='content_id', right_on='question_id', how='left')\n    test_df['prior_question_had_explanation'] = test_df.prior_question_had_explanation.fillna(False).astype('int8')\n    test_df['prior_question_elapsed_time_mean'] = test_df.prior_question_elapsed_time.fillna(prior_question_elapsed_time_mean)\n    lgbm_predictions = lgbm_model.predict(test_df[FEATS], num_iteration=lgbm_model.best_iteration)\n    \n    # Simple Model Score Averaging\n    test_df['answered_correctly'] =  0.5 * np.array(sakt_predictions) + 0.5 * lgbm_predictions\n    env.predict(test_df[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Attention Weight Visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"att_weight = att_weight.detach().cpu().numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attention_weights = att_weight[0]\n\nfig, ax = plt.subplots(figsize=(18, 16))\nmask = np.triu(np.ones_like(attention_weights, dtype=bool))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(attention_weights, mask=mask, cmap=cmap, vmax=.3, center=0,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\nplt.title(\"Attention Weight\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attention_weights = att_weight[1]\n\nfig, ax = plt.subplots(figsize=(18, 16))\nmask = np.triu(np.ones_like(attention_weights, dtype=bool))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(attention_weights, mask=mask, cmap=cmap, vmax=.3,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\nplt.title(\"Attention Weight\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attention_weights = att_weight[10]\n\nfig, ax = plt.subplots(figsize=(18, 16))\nmask = np.triu(np.ones_like(attention_weights, dtype=bool))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(attention_weights, mask=mask, cmap=cmap, vmax=.3,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\nplt.title(\"Attention Weight\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attention_weights = att_weight[16]\n\nfig, ax = plt.subplots(figsize=(18, 16))\nmask = np.triu(np.ones_like(attention_weights, dtype=bool))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(attention_weights, mask=mask, cmap=cmap, vmax=.3,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\nplt.title(\"Attention Weight\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attention_weights = att_weight[32]\n\nfig, ax = plt.subplots(figsize=(18, 16))\nmask = np.triu(np.ones_like(attention_weights, dtype=bool))\n\n# Generate a custom diverging colormap\ncmap = sns.diverging_palette(230, 20, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(attention_weights, mask=mask, cmap=cmap, vmax=.3,\n            square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\nplt.title(\"Attention Weight\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Observations\n\n- Need to use mask and remove the padded cases\n- Attention has some pattern that is repeating every 10 timestamps"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}