{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Riiid: Answer Correctness Prediction\n\n\nRiiid is a company whose goal is to imporve quality of education using AI.
\nRiiid wants to make persnolised education better for every student using AI.\n"},{"metadata":{},"cell_type":"markdown","source":"## 1. What is the competition about?๐ก\n\n* Riiid labs made an AI based tutor for south korean students.
\n so they tracked the interaction of the student with the app.
\n and here we have to predict how the student will perform in
\n future interaction.\n \n**Note: Let me know if any information or code is incorrect.
\n and if you find the notebook usefull please UPVOTE**."},{"metadata":{},"cell_type":"markdown","source":"## 2.Metrics: area under ROC curve๐.\n\n\n\nIn order to understand ROC curve we need to understand True Positive Rate(sensitivity) and False Posivtive Rate.\n\n\n**True Positive Rate**:
\n* For a binary classification true positive rate is ratio of predicted samples which are predicted true.
\n* False Positive rate is ratio of samples which are faslely predicted as Positive to all negative samples.\n\n* FPR = F.P / F.P + T.N \n\n\n#### What is ROC (Receiver Operator Characteristic) ?\n\n* ROC curve is the graph of TPR vs FPR over different threshold.
\n It means ROC curve requires to have model predict probability for different classes
\n\n* AUC is the area under that ROC curve.\n\n* more information on roc auc on [this](https://www.dataschool.io/roc-curves-and-auc-explained/) video\n"},{"metadata":{},"cell_type":"markdown","source":"\n### Importing Libraries ๐"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport sys\nimport gc\nimport glob\nimport time\nfrom os import listdir\nimport tqdm\nfrom typing import Dict\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.metrics import roc_auc_score\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplt.style.use(\"fivethirtyeight\")\n\nimport plotly.express as px\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport plotly.figure_factory as ff\n\n#supress warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nfrom colorama import Fore, Back, Style\ny_ = Fore.YELLOW\nr_ = Fore.RED\ng_ = Fore.GREEN\nb_ = Fore.BLUE\nm_ = Fore.MAGENTA\nc_ = Fore.CYAN\nsr_ = Style.RESET_ALL","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Getting data๐ฝ"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"folder_path = '../input/riiid-test-answer-prediction/'\ntrain_csv = folder_path + 'train.csv'\ntest_csv = folder_path + 'example_test.csv'\nlec_csv = folder_path + 'lectures.csv'\nque_csv = folder_path + 'questions.csv'\nsample_csv = folder_path + 'example_sample_submission.csv'\n\ndtype = {'row_id':'int64',\n 'timestemp':'int64',\n 'user_id':'int32',\n 'content_id':'int16',\n 'content_type_id':'int8',\n 'task_container_id':'int16',\n 'user_answer':'int8',\n 'answered_correctly':'int8',\n 'prior_question_elapsed_time':'float32',\n 'prior_question_had_explanation':'boolean'}\n\ntrain_data = pd.read_csv(train_csv,dtype=dtype,nrows=10**6)\ntest_data = pd.read_csv(test_csv)\nlec_data = pd.read_csv(lec_csv)\nque_data = pd.read_csv(que_csv)\nsample = pd.read_csv(sample_csv)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f\"{y_}Number of rows in train data: {r_}{train_data.shape[0]}\\n{y_}Number of columns in train data: {r_}{train_data.shape[1]}\")\nprint(f\"{g_}Number of rows in test data: {r_}{test_data.shape[0]}\\n{g_}Number of columns in test data: {r_}{test_data.shape[1]}\")\nprint(f\"{c_}Number of rows in lecture data: {r_}{lec_data.shape[0]}\\n{c_}Number of columns in lecture data: {r_}{lec_data.shape[1]}\")\nprint(f\"{m_}Number of rows in question data: {r_}{que_data.shape[0]}\\n{m_}Number of columns in question data: {r_}{que_data.shape[1]}\")\nprint(f\"{b_}Number of rows in submission data: {r_}{sample.shape[0]}\\n{b_}Number of columns in submission data:{r_}{sample.shape[1]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head().style.applymap(lambda x:\"background-color:lightgreen\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#looking for null values\ntrain_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lec_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"que_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3 EDA ๐\n\n### 3.1 countplot of user answers"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def countplot(column):\n plt.figure(dpi=100)\n sns.countplot(train_data[column])\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"countplot('user_answer')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.2 count plot of answered correctly"},{"metadata":{"trusted":true},"cell_type":"code","source":"countplot('answered_correctly')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.3 count plot of content_type_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"countplot('content_type_id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Count plot of prior question has explanation"},{"metadata":{"trusted":true},"cell_type":"code","source":"countplot(\"prior_question_had_explanation\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4 Distribution of elapsed timeโ"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(dpi=100)\nsns.distplot(train_data[~train_data[\"prior_question_elapsed_time\"].isna()][\"prior_question_elapsed_time\"],color='yellow')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.5 Distribution of interaction and top 40 of user interaction"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def distribution1(column,color,n=40):\n df = train_data[column].value_counts().reset_index()\n df.columns = [column,'count']\n df[column] = df[column].astype(str) + '-'\n df = df.sort_values(['count'],ascending=False)\n\n plt.figure(figsize=(15,10))\n plt.subplot(121)\n sns.distplot(df['count'],color=color)\n\n plt.subplot(122)\n sns.barplot(x='count',y=column,data=df[:n],orient='h')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"distribution1(\"user_id\",\"purple\") ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.6 Distribution of content_id and top 40 content_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"distribution1(\"content_id\",\"red\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.7 Distribution of task_container_id and top 50 task_container_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"distribution1(\"task_container_id\",\"green\",n=50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.8 Top users who answered correctly\n\nwe will look into users who answered highest percentage of their answer
\ngiven that they have altleast 10 interaction."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"answered_correctly = train_data.groupby(['user_id'])['answered_correctly'].agg(['sum','count']).reset_index()\nanswered_correctly = answered_correctly[answered_correctly['count']>=10]\nanswered_correctly['user_id'] = answered_correctly['user_id'].astype(str) + \"_\"\nanswered_correctly['percentage'] = (answered_correctly['sum'] / answered_correctly['count']) * 100\nanswered_correctly = answered_correctly.sort_values(['percentage'],ascending=False)\n\nplt.figure(figsize=(7,10))\nsns.barplot(x='percentage',y='user_id',data=answered_correctly[:50],orient='h');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.8 Bar Graph of correctly,incorrectly answered user answers"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_correct_user_answers = train_data[train_data['answered_correctly']==1]['user_answer']\ndf_incorrect_user_answers = train_data[train_data['answered_correctly']==0]['user_answer']\n\nplt.figure(figsize=(15,7))\nplt.subplot(121)\nsns.countplot(df_correct_user_answers)\nplt.title(\"Correctly answered user answers\")\nplt.subplot(122)\nsns.countplot(df_incorrect_user_answers)\nplt.title(\"Incorrectl answered user answers\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.9 Mean Responce time for each answers"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sorted_user_id_timestamp = train_data.sort_values(['user_id','timestamp'])\ntrain_data[\"time_required_to_answer\"] = sorted_user_id_timestamp.groupby('user_id')['prior_question_elapsed_time'].shift(periods=-1)\nresponce_time_correct = train_data[train_data['answered_correctly']==1].groupby('user_answer')['time_required_to_answer'].mean()\nresponce_time_incorrect = train_data[train_data['answered_correctly']==0].groupby('user_answer')['time_required_to_answer'].mean()\n\nplt.figure(figsize=(15,7))\nplt.subplot(121)\nsns.barplot(responce_time_incorrect.index,responce_time_correct.values)\nplt.title(\"Responce time for correctly answered answers\")\nplt.subplot(122)\nsns.barplot(responce_time_correct.index,responce_time_correct.values)\nplt.title(\"Responce time for incorrectly answered answers\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How good a user performs will also depend on how much time does he/she spends on the app or content\n\nHere timestamp starts with 0 but some users might have started later so let's see how much time user has spent on the app"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[\"timespend\"]=train_data.groupby('user_id')[\"timestamp\"].transform(lambda x: x.max() - x.min())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.10 Distribution of time spend by user on app.โ"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(dpi=100)\nplt.hist(train_data.timespend,color='red')\nplt.xlabel(\"timespend\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Now let us try to find change in accuracy with timestamp."},{"metadata":{},"cell_type":"markdown","source":"### 3.11 Timestamp vs Accuracy of the user"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data = train_data.sort_values(\"timestamp\").reset_index(drop=True)\ntrain_data['interaction_count'] = 1\ntrain_data['interaction_count'] = train_data.groupby(\"user_id\")['interaction_count'].transform('cumsum')\ntrain_data['correct_answers_till_now'] = train_data.groupby('user_id')['answered_correctly'].transform('cumsum')\ntrain_data['accuracy_per_timestamp'] = train_data['correct_answers_till_now']*100 / train_data['interaction_count']\n\nf = plt.figure(figsize=(7,7))\nsns.set_style(style=\"whitegrid\")\nsns.despine(f, left=True, bottom=True)\nsns.scatterplot(x='timestamp',y='accuracy_per_timestamp',data=train_data,hue='content_type_id');\nplt.xlabel(\"accuracy of user\")\nplt.ylabel(\"timestamp\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's examine user with top interaction to understand all columns of data\n\nuser with most interaction has user_id 7171715"},{"metadata":{"trusted":true},"cell_type":"code","source":"top_user = train_data[train_data.user_id == 7171715]\ntop_user = pd.merge(top_user,que_data,left_on='content_id',right_on='question_id',how='left')\ntop_user = pd.merge(top_user,lec_data,left_on='content_id',right_on='lecture_id',how='left')\ntop_user.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"number of question and lecture attented by user: \",top_user.content_id.nunique())\nprint(\"number of questions attented by user; \",top_user.question_id.nunique())\nprint(\"number of lectures attented by user: \",top_user[top_user.content_type_id==1].content_id.nunique())\nprint(\"number of bundles attented by user: \",top_user.bundle_id.nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.12 Change in accuracy with time of top user\n\nIn starting 100 question acc is high and then it decreases so inorder to see a good graph let's make two graphs
\nfor first 100 and then after 100"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sns.set_style(style=\"darkgrid\")\nplt.figure(figsize=(15,7))\nplt.subplot(121)\nsns.lineplot(x='timestamp',y='accuracy_per_timestamp',data=top_user[:100],color='green')\nplt.subplot(122)\nsns.lineplot(x='timestamp',y='accuracy_per_timestamp',data=top_user[100:],color='green');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Looking at the data this are the possible columns we can use to make prediction.
\n we have to make prediction for all question mean content_type_id == 0.
\n* From answer correctly we can get mean ,accuracy, median of the answers.
\n prior_question_elapsed time may be help to understand how hard previous question was or which question takes more time.
.\n \n* prior_question_had_explanation can be used to see if the user saw the explanatio of previous question.
\n should we shift these two columns ? but it would be difficult beacuse prior questions are grouped by bundle_id.\n* bundle_id can be usefull to determine which question were of same budle means they might belong to same type.
\n we will use part and tags later.\n \n* Our most important columns for training are one which uses answer correctly.\n "},{"metadata":{},"cell_type":"markdown","source":"## 4. Data Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"dtype = {\n 'row_id': 'int64',\n 'timestamp': 'int64',\n 'user_id': 'int32',\n 'content_id': 'int16',\n 'answered_correctly': 'int8',\n 'prior_question_elapsed_time': 'float16',\n 'prior_question_had_explanation': 'boolean'\n}\n\ntrain_data = pd.read_csv(\n '/kaggle/input/riiid-test-answer-prediction/train.csv',\n usecols = dtype.keys(),\n dtype=dtype, \n index_col = 0,\n nrows = 10**7\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# feature_data = train_data.iloc[:int(0.9 * len(train_data))]\n# train_data = train_data.iloc[int(0.9 * len(train_data)):]\n\ntrain_data = train_data.sort_values(\"timestamp\").reset_index(drop=True)\ntrain_data['time_required_to_answer'] = train_data.groupby('user_id')['prior_question_elapsed_time'].shift(-1)\ntrain_data['question_has_explanation'] = train_data.groupby('user_id')['prior_question_had_explanation'].shift(-1)\n\ntag = que_data[\"tags\"].str.split(\" \", n = 10, expand = True) \ntag.columns = ['tags1','tags2','tags3','tags4','tags5','tags6']\n\nque_data = pd.concat([que_data,tag],axis=1).drop(['tags'],axis=1)\nque_data['tags1'] = pd.to_numeric(que_data['tags1'], errors='coerce',downcast='integer').fillna(-1)\nque_data['tags2'] = pd.to_numeric(que_data['tags2'], errors='coerce',downcast='integer').fillna(-1)\nque_data['tags3'] = pd.to_numeric(que_data['tags3'], errors='coerce',downcast='integer').fillna(-1)\n\ntrain_data = pd.merge(train_data,que_data,left_on='content_id',right_on='question_id',how='left')\ntrain_data['timespend'] = train_data.groupby(\"user_id\")[\"timestamp\"].transform(lambda x: (x.max() - x.min())/1000)\ntrain_answered_question = train_data[train_data['answered_correctly']!=-1]\n\ngrouped_by_user_id = train_answered_question.groupby(\"user_id\")\ndf1 = grouped_by_user_id.agg({'answered_correctly':['mean','count','std','median']}).copy()\ndf1.columns = ['mean_user_accuracy', 'questions_answered', 'std_user_accuracy', 'median_user_accuracy']\n\ndel grouped_by_user_id\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_content_id = train_answered_question.groupby(\"content_id\")\ndf2 = grouped_by_content_id.agg({'answered_correctly':['mean','count','std','median']}).copy()\ndf2.columns = ['mean_accuracy', 'questions_asked', 'std_accuracy', 'median_accuracy']\n\n# df3 = grouped_by_content_id.agg({'timespend':['mean','std','median']}).copy()\n# df3.columns = ['mean_time', 'std_time', 'median_time']\n\ndel grouped_by_content_id\ndel train_answered_question\n# del feature_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [\n #numerical columns\n 'mean_user_accuracy', \n 'questions_answered',\n 'std_user_accuracy', \n 'median_user_accuracy',\n 'mean_accuracy', \n 'questions_asked',\n 'std_accuracy', \n 'median_accuracy',\n 'prior_question_elapsed_time', \n 'time_required_to_answer',\n #categorical columns\n 'prior_question_had_explanation',\n 'question_has_explanation',\n 'timespend',\n 'bundle_id',\n 'tags1',\n 'tags2',\n 'tags3',\n# 'mean_time',\n# 'std_time',\n# 'median_time',\n]\ntarget_column = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data[train_data[target_column] != -1]\ntrain_data = train_data.merge(df1, how='left', on='user_id')\ntrain_data = train_data.merge(df2, how='left', on='content_id')\n# train_data = train_data.merge(df3, how='left', on='content_id')\n\ntrain_data['prior_question_had_explanation'] = train_data['prior_question_had_explanation'].fillna(value = False).astype(bool)\ntrain_data['question_has_explanation'] = train_data['question_has_explanation'].fillna(value = False).astype(bool)\n\ntrain_data = train_data.fillna(value = -1)\n\ntarget = train_data[target_column].values\ntrain_data = train_data[features]\ntrain_data = train_data.replace([np.inf, -np.inf], np.nan)\ntrain_data = train_data.fillna(-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scaler = StandardScaler()\ntrain_data = scaler.fit_transform(train_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pytorch Baseline Model ๐ฅ"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import TensorDataset, DataLoader\nfrom sklearn.model_selection import KFold, StratifiedKFold\nfrom sklearn.metrics import roc_auc_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Model(nn.Module):\n def __init__(self,input_size,output_size):\n super(Model,self).__init__()\n self.batch_norm1 = nn.BatchNorm1d(input_size)\n self.dropout1 = nn.Dropout(0.3)\n self.linear1 = nn.utils.weight_norm(nn.Linear(input_size,128))\n \n self.batch_norm2 = nn.BatchNorm1d(128)\n self.dropout2 = nn.Dropout(0.2)\n self.linear2 = nn.utils.weight_norm(nn.Linear(128,32))\n \n self.batch_norm3 = nn.BatchNorm1d(32)\n self.dropout3 = nn.Dropout(0.2)\n self.linear3 = nn.utils.weight_norm(nn.Linear(32,output_size))\n \n def forward(self,xb):\n x = self.batch_norm1(xb)\n x = self.dropout1(x)\n x = F.leaky_relu(self.linear1(x))\n \n x = self.batch_norm2(x)\n x = self.dropout2(x)\n x = F.leaky_relu(self.linear2(x))\n \n x = self.batch_norm3(x)\n x = self.dropout3(x)\n return self.linear3(x)\n\n\n# class Model(nn.Module):\n# def __init__(self,input_dim,output_dim):\n# super(Model,self).__init__()\n# self.layer1 = nn.Linear(input_dim,100)\n# self.layer2 = nn.Linear(100,100)\n# self.layer3 = nn.Linear(100,output_dim)\n \n# def forward(self,xb):\n# x1 = F.leaky_relu(self.layer1(xb))\n# x1 = F.leaky_relu(self.layer2(x1))\n# return self.layer3(x1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"config = {\n \"epochs\":15,\n \"train_batch_size\":50_000,\n \"valid_batch_size\":50_000,\n \"test_batch_size\":50_000,\n \"nfolds\":3,\n \"learning_rate\":0.001,\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run(plot_losses=True):\n \n def train_loop(train_loader,model,loss_fn,device,optimizer,lr_scheduler=None):\n model.train()\n total_loss = 0\n for i, (inputs, targets) in enumerate(train_loader):\n inputs = inputs.to(device)\n targets = targets.to(device)\n \n optimizer.zero_grad()\n outputs = model(inputs)\n\n loss = loss_fn(outputs,targets)\n loss.backward()\n \n total_loss += loss.item()\n\n optimizer.step()\n if lr_scheduler != None:\n lr_scheduler.step(loss.item())\n \n total_loss /= len(train_loader)\n return total_loss\n \n def valid_loop(valid_loader,model,loss_fn,device):\n model.eval()\n total_loss = 0\n predictions = list()\n \n for i, (inputs, targets) in enumerate(valid_loader):\n inputs = inputs.to(device)\n targets = targets.to(device)\n outputs = model(inputs) \n\n loss = loss_fn(outputs,targets)\n predictions.extend(outputs.sigmoid().detach().cpu().numpy())\n \n total_loss += loss.item()\n total_loss /= len(valid_loader)\n \n return total_loss,np.array(predictions) \n \n\n kfold = StratifiedKFold(n_splits=config['nfolds'])\n \n #for storing losses of every fold\n fold_train_losses = list()\n fold_valid_losses = list()\n \n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n print(f\"{device} is used\")\n \n def loss_fn(outputs,targets):\n targets = targets.view(-1,1)\n return nn.BCEWithLogitsLoss()(outputs,targets)\n \n #kfold\n for k , (train_idx,valid_idx) in enumerate(kfold.split(train_data,target)):\n \n x_train,x_valid,y_train,y_valid = train_data[train_idx,:],train_data[valid_idx,:],target[train_idx],target[valid_idx]\n \n input_dim = x_train.shape[1]\n output_dim = 1\n\n model = Model(input_dim,output_dim)\n model.to(device)\n \n train_tensor = torch.tensor(x_train,dtype=torch.float)\n y_train_tensor = torch.tensor(y_train,dtype=torch.float)\n\n train_ds = TensorDataset(train_tensor,y_train_tensor)\n train_dl = DataLoader(train_ds,\n batch_size = config[\"train_batch_size\"],\n shuffle=True,\n num_workers = 4,\n pin_memory=True\n )\n\n valid_tensor = torch.tensor(x_valid,dtype=torch.float)\n y_valid_tensor = torch.tensor(y_valid,dtype=torch.float)\n\n valid_ds = TensorDataset(valid_tensor,y_valid_tensor)\n valid_dl = DataLoader(valid_ds,\n batch_size =config[\"valid_batch_size\"],\n shuffle=False,\n num_workers = 4,\n pin_memory=True,\n )\n \n optimizer = optim.Adam(model.parameters(),lr=config['learning_rate'])\n lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=5, eps=1e-4, verbose=True)\n\n print(f\"Fold {k}\")\n best_loss = 999\n \n train_losses = list()\n valid_losses = list()\n start = time.time()\n for i in range(config[\"epochs\"]):\n train_loss = train_loop(train_dl,model,loss_fn,device,optimizer,lr_scheduler=lr_scheduler)\n valid_loss,predictions = valid_loop(valid_dl,model,loss_fn,device)\n \n train_losses.append(train_loss)\n valid_losses.append(valid_loss)\n end = time.time()\n epoch_time = end - start\n start = end\n \n score = roc_auc_score(y_valid,predictions)\n \n print(f\"epoch:{i} Training loss:{train_loss} | Validation loss:{valid_loss} | Score: {score:.4f} | epoch time {epoch_time:.2f} \")\n \n if valid_loss <= best_loss:\n print(f\"{g_}Validation loss Decreased from {best_loss} to {valid_loss}{sr_}\")\n best_loss = valid_loss\n torch.save(model.state_dict(),f'model{k}.bin')\n \n fold_train_losses.append(train_losses)\n fold_valid_losses.append(valid_losses)\n \n \n if plot_losses == True:\n plt.figure(figsize=(20,14))\n for i, (t,v) in enumerate(zip(fold_train_losses,fold_valid_losses)):\n plt.subplot(2,5,i+1)\n plt.title(f\"Fold {i}\")\n plt.plot(t,label=\"train_loss\")\n plt.plot(v,label=\"valid_loss\")\n plt.legend()\n plt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"run()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"def inference(test):\n device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n all_prediction = np.zeros((test.shape[0],1))\n \n for i in range(config[\"nfolds\"]):\n \n input_dim = test.shape[1]\n output_dim = 1\n model = Model(input_dim,output_dim)\n model.load_state_dict(torch.load(f\"model{i}.bin\"))\n \n predictions = list()\n model.to(device)\n test_tensor = torch.tensor(test,dtype=torch.float)\n test_dl = DataLoader(test_tensor,\n batch_size=config[\"test_batch_size\"],\n shuffle=False)\n \n with torch.no_grad():\n for i, inputs in enumerate(test_dl):\n inputs = inputs.to(device, dtype=torch.float)\n outputs= model(inputs) \n predictions.extend(outputs.sigmoid().cpu().detach().numpy())\n\n all_prediction += np.array(predictions)/config['nfolds']\n \n return all_prediction","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_data,sample_prediction_df) in iter_test:\n test_data = pd.merge(test_data,que_data,left_on='content_id',right_on='question_id',how='left')\n test_data['timespend'] = test_data.groupby(\"user_id\")['timestamp'].transform(lambda x: x.max() - x.min())\n test_data['time_required_to_answer'] = test_data.groupby('user_id')['prior_question_elapsed_time'].shift(-1)\n test_data['question_has_explanation'] = test_data.groupby('user_id')['prior_question_had_explanation'].shift(-1)\n test_data = test_data.merge(df1,how='left',on='user_id')\n test_data = test_data.merge(df2,how='left',on='content_id')\n# test_data = test_data.merge(df3,how='left',on='content_id')\n\n test_data['prior_question_had_explanation'] = test_data['prior_question_had_explanation'].fillna(value = False).astype(bool)\n test_data['question_has_explanation'] = test_data['question_has_explanation'].fillna(value = False).astype(bool)\n\n test_data.fillna(value = -1, inplace = True)\n test_transform = scaler.transform(test_data[features])\n test_data['answered_correctly'] = inference(test_transform)\n env.predict(test_data.loc[test_data['content_type_id']==0,['row_id','answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"./submission.csv\")\nsub.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Work in Progress"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}