{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Welcome!** Here is a baseline model for the Riiid challenge explained:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import riiideducation\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import roc_auc_score\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset for training exceeds the RAM, if you do not use Google Cloud Storage. The dataset for testing, on the other hand, cannot be accessed directly, but the organisers of this competition provide a module for handling the data in batches. It's explained in this [Notebook](https://www.kaggle.com/sohier/competition-api-detailed-introduction). However, there are also more efficient ways to download and store the training data than csv to pandas(See this [Notebook](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid)). Still, we simply resort to using csv to pandas: We load the dataset that contains statistics on one specific answer given by a user to a question. Unfortunately, there are users in the test set for which we do not have data in this dataset:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',\n                   usecols=[1, 2, 3, 4, 5, 7, 8, 9],\n                   dtype={'timestamp': 'int64',\n                          'user_id': 'int32',\n                          'content_id': 'int16',\n                          'content_type_id': 'int8',\n                          'task_container_id': 'int16',\n                          'answered_correctly':'int8',\n                          'prior_question_elapsed_time': 'float32',\n                          'prior_question_had_explanation': 'boolean'}\n                   )\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just as in the test dataset, each row in the training set corresponds to a user's answer to a question. We see that there is information on how often a question is answered correctly in general. Thus, we load the file that contains statistics on each question, as we hope to gain valuable information from it: This is the complete list of questions that appear in the datasets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#reading in question df\nquestions_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv',                         \n                            usecols=[0, 3],\n                            dtype={'question_id': 'int16',\n                              'part': 'int8'}\n                          )\nquestions_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We remove data on lectures. They only represent about 2 percent of this training dataset and they will not be present in the test dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[train.content_type_id == False].sort_values('timestamp').reset_index(drop = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Features\nWe compute the mean time that elapsed while the user answered the previous question:"},{"metadata":{"trusted":true},"cell_type":"code","source":"elapsed_mean = train.prior_question_elapsed_time.mean()\nelapsed_mean","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compute the average amount of questions seen by a user:"},{"metadata":{"trusted":true},"cell_type":"code","source":"group1 = train.loc[(train.content_type_id == False), ['task_container_id', 'user_id']].groupby(['task_container_id']).agg(['count'])\ngroup1.columns = ['avg_questions']\ngroup2 = train.loc[(train.content_type_id == False), ['task_container_id', 'user_id']].groupby(['task_container_id']).agg(['nunique'])\ngroup2.columns = ['avg_questions']\ngroup3 = group1 / group2\ngroup3['avg_questions_seen'] = group3.avg_questions.cumsum()\nprint('The amount of questions seen by the average user:')\ngroup3.iloc[0].avg_questions_seen","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compute the mean accuracy for each user:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_u_final = train.loc[train.content_type_id == False, ['user_id','answered_correctly']].groupby(['user_id']).agg(['mean', 'count'])\nresults_u_final.columns = ['answered_correctly_user','answered_user']\nresults_u_final.answered_correctly_user.describe()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compute the fraction of prior questions that had an explanation for each user:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_u2_final = train.loc[train.content_type_id == False, ['user_id','prior_question_had_explanation']].groupby(['user_id']).agg(['mean'])\nresults_u2_final.columns = ['explanation_mean_user']\nresults_u2_final.explanation_mean_user.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We merge the training and question datasets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.merge(train, questions_df, left_on = 'content_id', right_on = 'question_id', how = 'left')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compute the fraction of correct answers for each question:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_q_final = train.loc[train.content_type_id == False, ['question_id','answered_correctly']].groupby(['question_id']).agg(['mean'])\nresults_q_final.columns = ['quest_pct']\nresults_q_final.quest_pct.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We compute how often each question was asked:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_q2_final = train.loc[train.content_type_id == False, ['question_id','part']].groupby(['question_id']).agg(['count'])\nresults_q2_final.columns = ['count']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We merge the data from the questions.csv and the new question features:"},{"metadata":{"trusted":true},"cell_type":"code","source":"question2 = pd.merge(questions_df, results_q_final, left_on = 'question_id', right_on = 'question_id', how = 'left')\nquestion2 = pd.merge(question2, results_q2_final, left_on = 'question_id', right_on = 'question_id', how = 'left')\nquestion2.quest_pct = round(question2.quest_pct,5)\nquestion2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{},"cell_type":"markdown","source":"We plot the fraction of answers of a user that are correct over the number of question that the respective user answered:"},{"metadata":{"trusted":true},"cell_type":"code","source":"figure=plt.subplots(figsize=(20,20))\nplt.scatter(x = results_u_final.answered_user, y=results_u_final.answered_correctly_user)\nplt.axhline(train['answered_correctly'].mean(), color='k', linestyle='dashed', linewidth=3)\n\nplt.title(\"Fraction of the user's answers that are correct vs. Number of questions answered by the user\", weight='bold')\nplt.text(15000, 0.64, 'Fraction of answers that are correct: {:.2f}'.format(train['answered_correctly'].mean()))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The fraction of first answers that were correct:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[(train.timestamp == 0)].answered_correctly.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The fraction of subsequent answers that were correct:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[(train.timestamp != 0)].answered_correctly.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The likelihood that the average user had an explanation provided with the previous question:"},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_mean_user = results_u2_final.explanation_mean_user.mean()\nprior_mean_user","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We drop the timestamp and the IDs for content question and part from the dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(['timestamp', 'content_type_id', 'question_id', 'part'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Realistic validation \nWe use the most recent five answers of each user as validation set. After all, that's what we would want to predict, if we stopped data collection a bit earlier."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The old length of the training set:')\nprint(len(train))\nvalidation = train.groupby('user_id').tail(10)\ntrain = train[~train.index.isin(validation.index)]\nprint('The length of the training set plus the length of the validation set:')\nprint(len(train) + len(validation))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We again compute the mean accuracy and the fraction of prior questions that had an explanation for each user, but this time without the validation set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_u_val = train[['user_id','answered_correctly']].groupby(['user_id']).agg(['mean'])\nresults_u_val.columns = ['answered_correctly_user']\n\nresults_u2_val = train[['user_id','prior_question_had_explanation']].groupby(['user_id']).agg(['mean'])\nresults_u2_val.columns = ['explanation_mean_user']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We reduce the size of the training set by removing the older answers:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train.groupby('user_id').tail(30)\ntrain = train[~train.index.isin(X.index)]\nprint('The length of the training set plus the length of the validation set plus the length of the set to be discarded:')\nprint(len(X) + len(validation)+ len(train))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We again compute the mean accuracy and the fraction of prior questions that had an explanation for each user, this time for the smaller training set:"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_u_X = train[['user_id','answered_correctly']].groupby(['user_id']).agg(['mean'])\nresults_u_X.columns = ['answered_correctly_user']\n\nresults_u2_X = train[['user_id','prior_question_had_explanation']].groupby(['user_id']).agg(['mean'])\nresults_u2_X.columns = ['explanation_mean_user']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cleaning\nWe remove the oldest part:"},{"metadata":{"trusted":true},"cell_type":"code","source":"del(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We merge the training set with the features that we computed:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = pd.merge(X, group3, left_on=['task_container_id'], right_index= True, how=\"left\")\nX = pd.merge(X, results_u_X, on=['user_id'], how=\"left\")\nX = pd.merge(X, results_u2_X, on=['user_id'], how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We merge the validation set in the same way:"},{"metadata":{"trusted":true},"cell_type":"code","source":"validation = pd.merge(validation, group3, left_on=['task_container_id'], right_index= True, how=\"left\")\nvalidation = pd.merge(validation, results_u_val, on=['user_id'], how=\"left\")\nvalidation = pd.merge(validation, results_u2_val, on=['user_id'], how=\"left\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We replace missing booleans by False. Then, we use an encoder to replace the boolean variables:"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nlb_make = LabelEncoder()\n\nX.prior_question_had_explanation.fillna(False, inplace = True)\nvalidation.prior_question_had_explanation.fillna(False, inplace = True)\n\nvalidation[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(validation[\"prior_question_had_explanation\"])\nX[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(X[\"prior_question_had_explanation\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The mean of the list of fractions of correct answers for questions:"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_mean = question2.quest_pct.mean()\n\nquestion2.quest_pct.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Many questions seem to have been asked few times and answered with an accuracy above average! Let's try to correct the accuracies for questions that have been asked very few times:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#filling questions with no info with a new value\nquestion2.quest_pct = question2.quest_pct.mask((question2['count'] < 3), .65)\n\n\n#filling very hard new questions with a more reasonable value\nquestion2.quest_pct = question2.quest_pct.mask((question2.quest_pct < .2) & (question2['count'] < 21), .2)\n\n#filling very easy new questions with a more reasonable value\nquestion2.quest_pct = question2.quest_pct.mask((question2.quest_pct > .95) & (question2['count'] < 21), .95)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's merge these new features with the training and validation datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"X = pd.merge(X, question2, left_on = 'content_id', right_on = 'question_id', how = 'left')\nvalidation = pd.merge(validation, question2, left_on = 'content_id', right_on = 'question_id', how = 'left')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We define the target and the features for the training and validation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"y = X['answered_correctly']\nX = X.drop(['answered_correctly'], axis=1)\nX.head()\n\ny_val = validation['answered_correctly']\nX_val = validation.drop(['answered_correctly'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We reduce the number of features that we use:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = X[['answered_correctly_user', 'explanation_mean_user', 'quest_pct', 'avg_questions_seen',\n       'prior_question_elapsed_time','prior_question_had_explanation_enc', 'part']]\nX_val = X_val[['answered_correctly_user', 'explanation_mean_user', 'quest_pct', 'avg_questions_seen',\n       'prior_question_elapsed_time','prior_question_had_explanation_enc', 'part']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We replace missing data in the average accuracy of individual users by the rounded average of the overall accuracy. We replace the missing data on the availability of an explanation for the prior question by the the overall liklihood that such an explanation was provided. We replace the missing mean accuracy of questions by the mean of the respective list. We replace the missing part numbers by the middle part. We replace the missing amounts of questions that a user has seen by the average amount of questions that a user has seen. We replace the missing elapsed time data by the mean elapsed time for previous questions. We replace missing information on whether an explanation was provided for the previous question by No."},{"metadata":{"trusted":true},"cell_type":"code","source":"X['answered_correctly_user'].fillna(0.65,  inplace=True)\nX['explanation_mean_user'].fillna(prior_mean_user,  inplace=True)\nX['quest_pct'].fillna(content_mean, inplace=True)\n\nX['part'].fillna(4, inplace = True)\nX['avg_questions_seen'].fillna(1, inplace = True)\nX['prior_question_elapsed_time'].fillna(elapsed_mean, inplace = True)\nX['prior_question_had_explanation_enc'].fillna(0, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We do the same for the validation dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_val['answered_correctly_user'].fillna(0.65,  inplace=True)\nX_val['explanation_mean_user'].fillna(prior_mean_user,  inplace=True)\nX_val['quest_pct'].fillna(content_mean,  inplace=True)\n\nX_val['part'].fillna(4, inplace = True)\nX['avg_questions_seen'].fillna(1, inplace = True)\nX_val['prior_question_elapsed_time'].fillna(elapsed_mean, inplace = True)\nX_val['prior_question_had_explanation_enc'].fillna(0, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Baseline\nWe import the model and define the datasets:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\n\nlgb_train = lgb.Dataset(X, y, categorical_feature = ['part', 'prior_question_had_explanation_enc'],free_raw_data=False)\nlgb_eval = lgb.Dataset(X_val, y_val, categorical_feature = ['part', 'prior_question_had_explanation_enc'], reference=lgb_train, free_raw_data=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We define the objective function and the constraints. Then, we train the model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"### import lightgbm as lgb\nparams = {\n        'num_leaves': 161,\n        'boosting_type': 'gbdt',\n        'max_bin': 890,\n        'objective': 'binary',\n        'metric': 'auc',\n        'max_depth': 12,\n        'min_child_weight': 11,\n        'feature_fraction': 0.6903098140467137,\n        'bagging_fraction': 0.9267405716419829,\n        'bagging_freq': 7,\n        'min_child_samples': 77,\n        'lambda_l1': 0.02267578846472961,\n        'lambda_l2': 9.722845458292198e-08,\n        'early_stopping_rounds': 10\n        }\nlgb_train = lgb.Dataset(X, y, categorical_feature = ['part', 'prior_question_had_explanation_enc'])\nlgb_eval = lgb.Dataset(X_val, y_val, categorical_feature = ['part', 'prior_question_had_explanation_enc'], reference=lgb_train)\nmodel = lgb.train(\n    params, lgb_train,\n    valid_sets=[lgb_train, lgb_eval],\n    verbose_eval=1000,\n    num_boost_round=2000\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Importance\nWe check how relevant the features are in the model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nlgb.plot_importance(model)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict"},{"metadata":{},"cell_type":"markdown","source":"We create an iterator of the test set using the function provided by the compition organiser. For each element in this iterator, we do the following: 1 We add the features that we computed, 2 We replace missing data in the same way that we did it in the training set, 3 We predict the target, and 4 We submit the predicitions with the function that is provided by the compition organisers:"},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()\nfor (test_df, sample_prediction_df) in iter_test:\n    test_df['task_container_id'] = test_df.task_container_id.mask(test_df.task_container_id > 9999, 9999)\n    test_df = pd.merge(test_df, group3, left_on=['task_container_id'], right_index= True, how=\"left\")\n    test_df = pd.merge(test_df, question2, left_on = 'content_id', right_on = 'question_id', how = 'left')\n    test_df = pd.merge(test_df, results_u_final, on=['user_id'],  how=\"left\")\n    test_df = pd.merge(test_df, results_u2_final, on=['user_id'],  how=\"left\")\n    test_df['answered_correctly_user'].fillna(0.65,  inplace=True)\n    test_df['explanation_mean_user'].fillna(prior_mean_user,  inplace=True)\n    test_df['quest_pct'].fillna(content_mean,  inplace=True)\n\n    test_df['part'].fillna(4, inplace = True)\n    test_df['avg_questions_seen'].fillna(1, inplace = True)\n    test_df['prior_question_elapsed_time'].fillna(elapsed_mean, inplace = True)\n    test_df['prior_question_had_explanation'].fillna(False, inplace=True)\n    test_df[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(test_df[\"prior_question_had_explanation\"])\n    \n    test_df['answered_correctly'] =  model.predict(test_df[['answered_correctly_user', 'explanation_mean_user', 'quest_pct', 'avg_questions_seen',\n                                                            'prior_question_elapsed_time','prior_question_had_explanation_enc', 'part']])\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgement\nI am grateful to Takamotoki and Mohammed Abdullah Al Mamun for inspiring me with these notebooks: \nhttps://www.kaggle.com/takamotoki/lgbm-iii-part2\nhttps://www.kaggle.com/mamun18/riiid-lgbm-lii-hyperparameter-tuning-optuna"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}