{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\nWelcome to the competition '<a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction\">Riiid! Answer Correctness Prediction</a>'.  \nThis Kaggle notebook provides step by step guide for submission the result if you want to join this competition.  \nThis notebook is written and focused on easy to participate in the competition including short EDA.  \n\nThe table of contents is as follows.  \n<a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda/notebook\">Top of Notebook</a>  \n├── <a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda#Environment-Setting\">Environment Setting</a>  \n├── <a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda/notebook#Dataset-Handling-(include-Visualization)\">Dataset Handling</a>  \n├── <a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda/notebook#Neural-Network!\">Neural Network</a>  \n├── <a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda/notebook#Make-the-result\">Make result</a>  \n└── <a href=\"https://www.kaggle.com/yeonghyeon/riiid-step-by-step-guide-for-beginner-with-eda/output\">Output</a>  \n\nTake a look around slowly and enjoy it.  \nAlways good luck to you!\n\n\n<strong>Additional Informations</strong>  \n[1] <a href=\"https://www.riiid.co\">Riiid!</a>  \n[2] <a href=\"https://www.ednetchallenge.ai/\">Riiid AIEd Challenge</a>  "},{"metadata":{},"cell_type":"markdown","source":"# Environment Setting\nThe first procedure is making an environment before all procedure.  \nPlease follow the below steps.  "},{"metadata":{},"cell_type":"markdown","source":"## Riiid Education Package\nFor solving the problem that 'Riiid' prvovides, you need make '<a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/data?select=riiideducation\">riiideducation</a>' environment.  "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import riiideducation\n\ntry: env = riiideducation.make_env()\nexcept: pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Other Packages\nImport other packages what you need.  \nIn this notebook, the following packages are loaded for computation and visualization respectively.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# For computation\nimport os, gc, copy, random\nimport numpy as np\nimport pandas as pd\n\n# For visualization\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# For data analysis\nfrom sklearn import metrics\nfrom sklearn.preprocessing import LabelEncoder\nfrom  sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Package for Deep Learning (DL)\nIn this notebook, the deep learning package '<a href=\"https://pytorch.org/\">Pytorch</a>' is adopted."},{"metadata":{"trusted":true},"cell_type":"code","source":"# For using PyTorch\nimport torch\nimport torch.nn as nn\nimport torch.multiprocessing as mp\n\nfrom torch import optim\nfrom torchvision import models\nfrom torch.nn import functional as F\nfrom torch.utils.data import DataLoader, Dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set static random seed"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.environ['PYTHONHASHSEED'] = str(32)\nrandom.seed(32)\nnp.random.seed(32)\ntorch.random.manual_seed(32)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Handling (include Visualization)\nDataset is one of the most important things for optimizing the Machine Learning (ML) or Deep Learning (DL) model.  \nIn this step, the dataset is mainly handled via the '<a href=\"https://pandas.pydata.org/\">pandas</a>' package.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_dataframe(df, length=5):\n    \n    \"\"\" -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-\n    Define the function for checking a dataframe.\n    This function shows the shape of the dataframe firstly.\n    Then, show the head and tail part of the whole dataframe.\n    -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- \"\"\"\n    \n    print(\"* Key of dataframe\")\n    print(list(df.keys()))\n    print(\"\\n* Shape of dataframe:\", df.shape)\n    print(\"\\n* Head of dataframe\")\n    print(df.head(length))\n    print(\"\\n* Tail of dataframe\")\n    print(df.tail(length))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load and trim the training set\nFirst of all, we need to load the training set named 'train.csv'.  \nNote that, it will consume a lot of time because the file ('train.csv') size is large (5.45GB).  \n(It takes about 3~5 minutes.)  \nAfter loading the 'csv' file as a dataframe, trim the meaningless rows and sorting with ascending order by 'timestamp' column.  \n\n+ For memory efficiency, test set will be handled after training the neural network."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndf_train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', \\\n    usecols=[1, 2, 3, 4, 7, 8, 9], \\\n    dtype={'timestamp': 'int64', 'user_id': 'int32' ,'content_id': 'int16','content_type_id': 'int8', \\\n        'answered_correctly':'int8','prior_question_elapsed_time': 'float32','prior_question_had_explanation': 'boolean'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_dataframe(df_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sub-Step 1. Select the rows where the value of 'content_type_id' is 'False'.\ndf_train = df_train[df_train.content_type_id == False]\n\n# Sub-Step 2. Sort the dataframe with ascending order.\n# The base column for sorting is 'timestamp'.\ndf_train = df_train.sort_values(['timestamp'], ascending=True)\n\n# Sub-Step 3. Then, drop the two columns named 'timestamp' and 'content_type_id'\n# They are not useful anymore.\n# If you do not want to use 'inplace=True', you can use another command as follows.\n# >>> df_train = df_train.drop(['timestamp', 'content_type_id'], axis=1)\ndf_train.drop(['timestamp', 'content_type_id'], axis=1, inplace=True)\n\nprint_dataframe(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate additional meaningful information\nWe should extract meaningful information among the training set.  \nIn another way, we can create additional information from existing data.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sub-Step 1. Calculate the mean and sum of the column 'answered_correctly' group by 'user_id'.\nagg_uid = df_train[['user_id','answered_correctly']].groupby(['user_id']).agg(['mean', 'sum'])\nagg_uid.columns = [\"mean_by_uid\", 'sum_by_uid']\n\nprint_dataframe(agg_uid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sub-Step 2. Calculate the sum of the column 'answered_correctly' group by 'content_id'.\nagg_cid = df_train[['content_id','answered_correctly']].groupby(['content_id']).agg(['mean'])\nagg_cid.columns = [\"mean_by_cid\"]\n\nprint_dataframe(agg_cid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sub-Step 3. Merge the generated informations to training set.\n# In this notebook, the merged dataframe is newly allocated to another variable for maintaining the original dataframe.\n# If you do not need to maintain the original dataframe, you can use the following command.\n# >>> del df_train\n# You can also use the above command with the lack of RAM situation.\nX_tr = pd.merge(df_train, agg_uid, on=['user_id'], how=\"left\")\nX_tr = pd.merge(X_tr, agg_cid, on=['content_id'], how=\"left\")\nX_tr = X_tr[X_tr.answered_correctly != -1]\nX_tr = X_tr.sort_values(['user_id'])\nY_tr = X_tr[['answered_correctly']]\nX_tr = X_tr.drop(['answered_correctly'], axis=1)\n\ndel df_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Dataframe: X_tr\")\nprint_dataframe(X_tr)\n\nprint(\"\\nDataframe: Y_tr\")\nprint_dataframe(Y_tr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_key_tr = list(X_tr.keys())\nfor idx, name_key in enumerate(list_key_tr):\n    if(X_tr[name_key].isna().sum() > 0):\n        try: X_tr[name_key] = X_tr[name_key].fillna(X_tr[name_key].median())\n        except: pass\n    \nprint_dataframe(X_tr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Numerizing the non-numeric column\nFor calculating the non-numeric variable in the optimization procedure, we should digitizing the non-numeric values.  \nIn this notebook, the class named <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\">LabelEncoder</a>' of the package '<a href=\"https://scikit-learn.org/stable/\">scikit-learn</a>' is used."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sub-Step 1. Define and initializing the LabelEncoder class.\nlabel_encoder = LabelEncoder()\n\n# Sub-Step 2. Encode the non-numeric column 'prior_question_had_explanation' via the function 'fit_transform'.\nX_tr['prior_question_had_explanation_enc'] = label_encoder.fit_transform(X_tr['prior_question_had_explanation'])\n\n# Sub-Step 3. Drop the original non-numeric column.\nX_tr.drop(['prior_question_had_explanation'], axis=1, inplace=True)\n\nprint_dataframe(X_tr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization\nVisualization is one other important procedure for achieving insight among the dataset.  \nSome packages named '<a href=\"https://matplotlib.org/\">matplotlib</a>' and '<a href=\"https://seaborn.pydata.org/\">seaborn</a>' will help this procedure.  \nAlso, '<a href=\"https://pandas.pydata.org/\">pandas</a>' includes some of plotting methods such as 'hist'."},{"metadata":{},"cell_type":"markdown","source":"### Plot of Correlation Coefficient (CC)"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 5))\nplt.title(\"Correlation Coefficient Matrix\")\nsns.heatmap(X_tr.corr(), cmap='jet', annot=True, fmt='.3f')\nplt.show()\nplt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plot of Histogram"},{"metadata":{"trusted":true},"cell_type":"code","source":"list_key_tr = list(X_tr.keys())\nfor idx, name_key in enumerate(list_key_tr):\n    plt.figure(figsize=(6, 3))\n    \n    plt.title(name_key.upper())\n    X_tr[name_key].hist() # The pandas supports 'hist' function.\n    plt.ylabel(name_key)\n    \n    plt.grid()\n    plt.tight_layout()\n    plt.show()\n    plt.close()\n    \n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Neural Network!\nIn this section, the classe named with neural network will be defined."},{"metadata":{},"cell_type":"markdown","source":"## Class - Neural Network\nMulti Layer Perceptron (MLP)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class neuralnet(nn.Module):\n    \n    def __init__(self, n_additional_features, n_outputs):\n        \n        super(neuralnet, self).__init__()\n        \n        self.fc0_1 = nn.Linear(n_additional_features, 512) \n        self.fc0_1d = nn.Dropout(0.5)\n        self.fc0_2 = nn.Linear(512, 128) \n        self.fc0_2d = nn.Dropout(0.5)\n        self.fc0_3 = nn.Linear(128, 32) \n        self.fc0_3d = nn.Dropout(0.5)\n        \n        self.fc1 = nn.Linear(32, 64) \n        self.fc1_d = nn.Dropout(0.5)\n        self.fc1_skip = nn.Linear(32, 64) \n        \n        self.fc2 = nn.Linear(64, 128) \n        self.fc2_d = nn.Dropout(0.5)\n        self.fc2_skip = nn.Linear(64, 128) \n        \n        self.fc3 = nn.Linear(192, 256) \n        self.fc3_d = nn.Dropout(0.5)\n        self.fc4 = nn.Linear(384, n_outputs) \n\n    def forward(self, additional_features):\n        \n        out0 = additional_features\n        \n        out0_1 = F.elu(\n            self.fc0_1d(\n                self.fc0_1(out0)\n            )\n        )\n        out0_2 = F.elu(\n            self.fc0_2d(\n                self.fc0_2(out0_1)\n            )\n        )\n        out0_3 = F.elu(\n            self.fc0_3d(\n                self.fc0_3(out0_2)\n            )\n        )\n        \n        out1 = F.elu(\n            self.fc1_d(\n                self.fc1(out0_3)\n            )\n        )\n        out1_s = F.elu(\n            self.fc1_skip(out0_3)\n        )\n        \n        out2 = F.elu(\n            self.fc2_d(\n                self.fc2(out1)\n            )\n        )\n        out2_s = F.elu(\n            self.fc2_skip(out1)\n        )\n        \n        out3 = F.elu(\n            self.fc3_d(\n                self.fc3(\n                    torch.cat((out2, out1_s), 1)\n                )\n            )\n        )\n        out4 = F.relu(\n            self.fc4(\n                torch.cat((out3, out2_s), 1)\n            )\n        )\n        \n        return out4\n    \n    def loss_mse(self, pred, true):\n        \n        return  torch.mean(torch.sum((pred - true)**2, dim=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Hyperparameters\nSet the hyperparameters for optimizing the neural network."},{"metadata":{"trusted":true},"cell_type":"code","source":"ratio_val = 0.1\nbatch_size = 32\nnum_workers = 4\n\nepochs = 10\nlearning_rate = 1e-3\nearly_stopper = False\npatience = 3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset preparing\nFor preparing the dataset, split the training set to the training and validation set.  \nThen, utilize the 'RiiidData' class as defined earlier."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_tr, X_val, Y_tr, Y_val = train_test_split(X_tr, Y_tr, test_size=ratio_val, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel = neuralnet(X_tr.shape[1], 1).to(device)\nprint(\"Number of parameters:\", sum(p.numel() for p in model.parameters() if p.requires_grad))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define the metric function\nDefine the function for mearue metric.  \nIn this problem, the Area Under the Receiver Operating Characteristic Curve (AUROC) is used as a indicator.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"def measure_auroc(label, logit):\n    \n    \"\"\" -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-\n    Define the function for measuring the Area Under the Receiver Operating Characteristic Curve (AUROC).\n    -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- \"\"\"\n    \n    fpr, tpr, thresholds = metrics.roc_curve(label, logit, pos_label=0)\n    auroc = metrics.auc(fpr, tpr)\n    if(auroc < 0.5):\n        fpr, tpr, thresholds = metrics.roc_curve(label, logit, pos_label=1)\n        auroc = metrics.auc(fpr, tpr)\n    return auroc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training!\nNow you have entered the training process.  \nLearning will be progressed as much as the specified epoch through the commands below.  \nTo run completely, remove the break command, which commented in line."},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = optim.Adam(model.parameters(), \\\n    lr=learning_rate, betas=(0.5, 0.999), eps=1e-08, \\\n    weight_decay=learning_rate/10, amsgrad=True)\n\nepoch, loss_min = 0, 1e+100\nepoch_loss_tr, epoch_loss_val = [], []\nepoch_metric_tr, epoch_metric_val = [], []\n\nwhile((epoch<epochs) and not(early_stopper)):\n    epoch += 1\n\n    loss_tr, metric_tr, label, logit = 0, 0, None, None\n    model.train()\n    terminator, idx_s, idx_e, amount = False, 0, batch_size, X_tr.shape[0]\n    cnt_tr = 0\n    while(True):\n        cnt_tr += 1\n        if(idx_s % 65536 == 0): print(\"Run | Epoch [%d/%d], Batch-index [%d~%d / %d]\" %(epoch, epochs, idx_s, idx_e, amount))\n        tmp_x, tmp_y = torch.from_numpy(np.asarray(X_tr[idx_s:idx_e])), torch.from_numpy(np.asarray(Y_tr[idx_s:idx_e]))\n        optimizer.zero_grad()\n        predictions = model(tmp_x.float().to(device))\n        loss_tmp_tr = model.loss_mse(predictions, tmp_y.to(device))\n        loss_tmp_tr.backward()\n        loss_tr += loss_tmp_tr.item()\n        \n        label_tmp = list(np.squeeze(tmp_y.detach().numpy()))\n        logit_tmp = list(np.squeeze(predictions.detach().numpy()))\n        if(label is None): label, logit = label_tmp, logit_tmp\n        else:\n            label.extend(label_tmp)\n            logit.extend(logit_tmp)\n        \n        optimizer.step()\n        gc.collect()\n        \n        break # !!! Remove this break command when you use this notebook in practice. \n        if(terminator): break\n        idx_s, idx_e = idx_e, idx_e+batch_size\n        if(idx_e >= amount): terminator = True\n    \n    print(type(label), type(logit))\n    metric_tr = measure_auroc(label, logit)\n    \n    \n    \"\"\" -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- \"\"\"\n    \n    loss_val, metric_val, label, logit = 0, 0, None, None\n    model.eval()\n    terminator, idx_s, idx_e, amount = False, 0, batch_size, X_val.shape[0]\n    cnt_val = 0\n    with torch.no_grad():\n        while(True):\n            cnt_val += 1\n            if(idx_s % 16384 == 0): print(\"Run | Epoch [%d/%d], Batch-index [%d~%d / %d]\" %(epoch, epochs, idx_s, idx_e, amount))\n            tmp_x, tmp_y = torch.from_numpy(np.asarray(X_val[idx_s:idx_e])), torch.from_numpy(np.asarray(Y_val[idx_s:idx_e]))\n            predictions = model(tmp_x.float().to(device))\n            loss_tmp_val = model.loss_mse(predictions, tmp_y.to(device)).mean()\n            loss_val += loss_tmp_val.item()\n            \n            label_tmp = list(np.squeeze(tmp_y.detach().numpy()))\n            logit_tmp = list(np.squeeze(predictions.detach().numpy()))\n            if(label is None): label, logit = label_tmp, logit_tmp\n            else:\n                label.extend(label_tmp)\n                logit.extend(logit_tmp)\n            \n            break # !!! Remove this break command when you use this notebook in practice. \n            if(terminator): break\n            idx_s, idx_e = idx_e, idx_e+batch_size\n            if(idx_e >= amount): terminator = True\n            \n            gc.collect()\n\n    loss_tr = loss_tr / cnt_tr\n    loss_val = loss_val / cnt_val\n    print(\"Epoch [%d/%d]\" %(epoch, epochs))\n    print(\" Training   | MSE: %.4f   AUROC: %.5f\" %(loss_tr, metric_tr))\n    print(\" Validation | MSE: %.4f   AUROC: %.5f\" %(loss_val, metric_val))\n    \n    epoch_loss_tr.append(loss_tr)\n    epoch_loss_val.append(loss_val)\n    epoch_metric_tr.append(metric_tr)\n    epoch_metric_val.append(metric_val)\n    \n    if(loss_val <= loss_min):\n        loss_min = loss_val\n        best_model = copy.deepcopy(model.state_dict())\n        epochs_no_improve = 0\n\n    else:\n        epochs_no_improve += 1\n        if(epochs_no_improve == patience):\n            print(\"Early stopping!\\n\")\n            early_stopper = True\n            model.load_state_dict(best_model)\n    \n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Show the training results\nThe following block shows the training results via graph.  \nTwo kinds of graph are provided and they represent curve of loss and AUROC respectively.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\n\nplt.subplot(1, 2, 1)\nplt.title('Loss Curve')\nplt.plot(epoch_loss_tr, label='Training')\nplt.plot(epoch_loss_val, label='Validation')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.grid()\nplt.legend(loc='upper right')\n\nplt.subplot(1, 2, 2)\nplt.title('AUROC Curve')\nplt.plot(epoch_metric_tr, label='Training')\nplt.plot(epoch_metric_val, label='Validation')\nplt.ylabel('AUROC')\nplt.xlabel('Epoch')\nplt.grid()\nplt.legend(loc='lower right')\n\nplt.show()\nplt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make the result\nFirst of all, we should remove the training and validation set from the RAM.  \nIf not, we will face the Out Of Memory (OOM) problem.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"del X_tr, X_val, Y_tr, Y_val","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Trim test set\nLoad and trim the test set as same as training set.  \nThe detailed comments are omitted for the processing test set because the content (context) is the same as processing the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_dataframe(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_te = pd.merge(df_test, agg_uid, on=['user_id'], how=\"left\")\nX_te = pd.merge(X_te, agg_cid, on=['content_id'], how=\"left\")\nX_te = X_te.sort_values(['user_id'])\n\ndel df_test\n\nX_te['prior_question_had_explanation_enc'] = label_encoder.fit_transform(X_te['prior_question_had_explanation'])\nX_te.drop(['prior_question_had_explanation'], axis=1, inplace=True)\n\nlist_key_te = list(X_te.keys())\nfor idx, name_key in enumerate(list_key_te):\n    if(X_te[name_key].isna().sum() > 0):\n        try: X_te[name_key] = X_te[name_key].fillna(X_te[name_key].median())\n        except: pass\n    \nprint_dataframe(X_te)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Column Check\nMake the format of the test set as same as the training set via dropping the columns."},{"metadata":{"trusted":true},"cell_type":"code","source":"list_key_te = list(X_te.keys())\nfor name_key in list_key_tr:\n    print(name_key)\n    try: idx_key = list_key_te.index(name_key)\n    except: print(name_key, \"not included\")\n    else: list_key_te.pop(idx_key)\n\nkey4drop = list_key_te\nprint(\"Key for drop\")\nprint(key4drop)\n\nX_te.drop(key4drop, axis=1, inplace=True)\nprint_dataframe(X_te)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.eval()\nlogit = []\nterminator, idx_s, idx_e, amount = False, 0, batch_size, X_te.shape[0]\nwith torch.no_grad():\n    while(True):\n        tmp_x = torch.from_numpy(np.asarray(X_te[idx_s:idx_e]))\n        predictions = model(tmp_x.float().to(device))\n        \n        logit_tmp = list(np.squeeze(predictions.detach().numpy()))\n        if(logit is None): logit = logit_tmp\n        else: logit.extend(logit_tmp)\n            \n        if(terminator): break\n        idx_s, idx_e = idx_e, idx_e+batch_size\n        if(idx_e >= amount): terminator = True\n\n        gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sb = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')\nprint_dataframe(df_sb)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx in range(df_sb.shape[0]):\n    df_sb.loc[idx, 'answered_correctly'] = logit[idx]\nprint(logit[:5])\nprint_dataframe(df_sb)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sb.to_csv('submission.csv', index=False, float_format='%.3f')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}