{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"Riiid Labs provides innovative educational solutions. They have provided AI tutor based on deep-learning algorithms. This is my submission of the tracing knowledge in the _Riiid AIEd Challenge 2020_. The purpose of this notebook is to present: \n* A thorough exploratory data analysis of the student-question interaction dataset. \n* Predict how well a student answers a question. "},{"metadata":{},"cell_type":"markdown","source":"This submission is written in Python. The first thing is to import all the necessary libraries and data into here. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Import libraries\n\nimport riiideducation\n\nimport numpy as np \nimport pandas as pd \n\nfrom pandas_profiling import ProfileReport\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import label_binarize\n\nimport lightgbm as lgb\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nimport xgboost as xgb\nfrom sklearn.ensemble import StackingClassifier\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom scipy.stats import randint as sp_randint\nfrom scipy.stats import uniform as sp_uniform\n\nfrom sklearn.metrics import mean_absolute_error\n\nfrom matplotlib import pyplot as plt \n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import data\n\n# Come from https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908\ntrain_df = pd.DataFrame()\ncounter = 1\nfor chunk in pd.read_csv('../input/riiid-test-answer-prediction/train.csv', chunksize=1000000, low_memory=False):\n    print('Reading chunck {}'.format(counter))\n    # Sample the size as it is too big\n    chunk = chunk.sample(frac=0.1, random_state=1)\n    train_df = pd.concat([train_df, chunk], ignore_index=True)\n    counter += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import other data sets, which are small enough\n\ntest_df = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')\n# questions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\n# lectures_df = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{},"cell_type":"markdown","source":"Let's look at the features in each data frames and see what they mean. "},{"metadata":{},"cell_type":"markdown","source":"Here is the data dictionary of `train.csv`, copied from the introduction. \n\n* `row_id`: (int64) ID code for the row.\n* `timestamp`: (int64) the time between this user interaction and the first event completion from that user.\n* `user_id`: (int32) ID code for the user.\n* `content_id`: (int16) ID code for the user interaction\n* `content_type_id`: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n* `task_container_id`: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n* `user_answer`: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n* `answered_correctly`: (int8) if the user responded correctly. Read -1 as null, for lectures.\n* `prior_question_elapsed_time`: (float32) The average time it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n* `prior_question_had_explanation`: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{},"cell_type":"markdown","source":"Let's look at the first few data and see how does the data frame looks like. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data frame is too large to print out its profiling, and we sought to use the traditional method (to inspect the data frame). The following is from the `.info()` method. It shows the dtype and also how many missing data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f'There are {train_df.shape[0]} rows with {train_df.shape[1]} records in train_df. '","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df['answered_correctly'].isna().sum())\nprint(train_df.shape[0])\nprint('There are {:0.2f}% of missing data in answered_correctly. '.format(train_df['answered_correctly'].isna().sum()/train_df.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df['prior_question_elapsed_time'].isna().sum())\nprint(train_df.shape[0])\nprint('There are {:0.2f}% of missing data in prior_question_elapsed_time. '.format(train_df['prior_question_elapsed_time'].isna().sum()/train_df.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So how many missing records in `'prior_question_had_explanation'`? "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df['prior_question_had_explanation'].isna().sum())\nprint(train_df.shape[0])\nprint('There are {:0.2f}% of missing data in prior_question_had_explanation. '.format(train_df['prior_question_had_explanation'].isna().sum()/train_df.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It might okay to remove the missing values. "},{"metadata":{},"cell_type":"markdown","source":"`test_df` is the validation set and we shall see there are extra features, namely the target variables in the data frame. They are \n* `prior_group_responses` (string) provides all of the `user_answer` entries for previous group in a string representation of a list in the first row of the group. All other rows in each group are null. If you are using Python, you will likely want to call `eval` on the non-null rows. Some rows may be null, or empty lists.\n\n* `prior_group_answers_correct` (string) provides all the `answered_correctly` field for previous group, with the same format and caveats as `prior_group_responses`. Some rows may be null, or empty lists."},{"metadata":{"trusted":true},"cell_type":"code","source":"# ProfileReport(test_df, title=\"`test_df` Profiling Report\", progress_bar=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ProfileReport(questions_df, title=\"`questions_df` Profiling Report\", progress_bar=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's the data dictionary of `questions_df`: \n* `question_id`: foreign key for the train/test content_id column, when the content type is question (0).\n* `bundle_id`: code for which questions are served together.\n* `correct_answer`: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n* `part`: the relevant section of the TOEIC test.\n* `tags`: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"# ProfileReport(lectures_df, title=\"`lectures_df` Profiling Report\", progress_bar=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Participants may watch the lecture and answer the questions, which is listed in `lectures_df` and the details are in below: \n* `lecture_id`: foreign key for the train/test content_id column, when the content type is lecture (1).\n* `part`: top level category code for the lecture.\n* `tag`: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n* `type_of`: brief description of the core purpose of the lecture"},{"metadata":{},"cell_type":"markdown","source":"## Missing Data"},{"metadata":{},"cell_type":"markdown","source":"We now treat the missing data of `train_df` and `test_df` here. From the EDA, we can see that there are 2.3% and 0.3% of missing values in the last two columns of `train_df`. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code from https://www.kaggle.com/dmikar/baseline-for-riiid-lightgbm\nmean_prior = train_df['prior_question_elapsed_time'].astype(\"float64\").mean()\nprint(f'{mean_prior} is filled for the missing data in prior_question_elapsed_time. ')\n\ntrain_df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\ntrain_df['prior_question_had_explanation'].fillna(False, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_prior = test_df['prior_question_elapsed_time'].astype(\"float64\").mean()\nprint(f'{mean_prior} is filled for the missing data in prior_question_elapsed_time. ')\n\ntest_df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\ntest_df['prior_question_had_explanation'].fillna(False, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Correct the data types"},{"metadata":{},"cell_type":"markdown","source":"The LGBM model does not like any dtypes other than `int`, `float` or `bool`. While `prior_question_had_explanation` in `test_df` has a custom data type `boolean`, we will need to change them. "},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['prior_question_had_explanation'].dtype","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['prior_question_had_explanation'].dtype","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train_df['answered_correctly'].to_numpy()\nX = train_df[['user_id', 'content_id', 'content_type_id', 'task_container_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']].to_numpy()\n\nX_train_df, X_val_df, y_train_df, y_val_df = train_test_split(X, y, test_size =0.3, shuffle=False)\n\ndel train_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LightGBM Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_train = lgb.Dataset(X_train_df, y_train_df, categorical_feature = ['prior_question_had_explanation'], free_raw_data=False)\nlgb_eval = lgb.Dataset(X_val_df, y_val_df, categorical_feature = ['prior_question_had_explanation'], free_raw_data=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We now train the model. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# # param values c.f. https://www.kaggle.com/zephyrwang666/riiid-lgbm-bagging2\n# param = {'num_leaves': sp_randint(150, 400), 'max_bin':sp_randint(300, 800), 'min_child_weight': [1e-5, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3, 1e4], \n#          'feature_fraction': sp_uniform(0, 1), 'bagging_fraction': sp_uniform(0, 1), \n#          'objective': ['binary'], 'max_depth': [-1], \n#          'learning_rate': [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09], \"boosting_type\": [\"gbdt\"], \"bagging_seed\": [47], \n#          'eval_metric': ['logloss'], \"verbosity\": [-1], \n#          'reg_alpha': [0, 1e-1, 1, 2, 5, 7, 10, 50, 100], 'reg_lambda': [0, 1e-1, 1, 2, 5, 7, 10, 50, 100], \n#          'random_state': [47]}\n\n# m1 = lgb.LGBMClassifier(valid_sets = [lgb_train, lgb_eval], verbose_eval = 30, num_boost_round = 10000, early_stopping_rounds = 10, n_jobs=4, n_estimators=3000)\n\n# '''\n# Hyperparameter optimisation\n# '''\n# # Code from https://www.kaggle.com/rtatman/lightgbm-hyperparameter-optimisation-lb-0-761#Model-fitting-with-HyperParameter-optimisation\n# #This parameter defines the number of hyperparameter points to be tested\n# n_HP_points_to_test = 5\n\n# gsLGBM = RandomizedSearchCV(\n#     estimator=m1, param_distributions=param, \n#     n_iter=n_HP_points_to_test,\n#     cv=3,\n#     refit=True,\n#     random_state=47,\n#     verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# gsLGBM.fit(X_train_df, y_train_df, eval_set = (X_val_df, y_val_df))\n# print('Best score reached: {} with params: {} '.format(gsLGBM.best_score_, gsLGBM.best_params_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Just in case, the parameters should be printed in here. \n# Score: 0.6788\nopt_parameters_LGBM = {'bagging_fraction': 0.11348847189364952, 'bagging_seed': 47, 'boosting_type': 'gbdt', \n 'eval_metric': 'logloss', 'feature_fraction': 0.9744830944364566, 'learning_rate': 0.09, \n 'max_bin': 479, 'max_depth': -1, 'min_child_weight': 1e-05, 'num_leaves': 173, \n 'objective': 'binary', 'random_state': 47, 'reg_alpha': 0, 'reg_lambda': 50, 'verbosity': -1}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m1 = lgb.LGBMClassifier(valid_sets = [lgb_train, lgb_eval], verbose_eval = 30, num_boost_round = 10000, early_stopping_rounds = 10, n_jobs=4, n_estimators=3000, **opt_parameters_LGBM)\nm1.fit(X_train_df, y_train_df, eval_set = (X_val_df, y_val_df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(f'The mean absolute error of the model is {mean_absolute_error(y_val_df, gsLGBM.predict(X_val_df))}. ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ADABoost"},{"metadata":{"trusted":true},"cell_type":"code","source":"# m2 = AdaBoostClassifier(DecisionTreeClassifier(max_depth=3, min_samples_leaf=1, min_impurity_decrease=10, random_state=47), random_state=47)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# param = {'learning_rate': [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], 'n_estimators': sp_randint(5, 50)}\n\n# '''\n# Hyperparameter optimisation\n# '''\n# # Code from https://www.kaggle.com/rtatman/lightgbm-hyperparameter-optimisation-lb-0-761#Model-fitting-with-HyperParameter-optimisation\n# #This parameter defines the number of HP points to be tested\n# n_HP_points_to_test = 3\n\n# gsADA = RandomizedSearchCV(\n#     estimator=m2, param_distributions=param, \n#     n_iter=n_HP_points_to_test,\n#     cv=3,\n#     refit=True,\n#     random_state=47,\n#     verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# gsADA.fit(X_train_df, y_train_df)\n# print('Best score reached: {} with params: {} '.format(gsADA.best_score_, gsADA.best_params_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Just in case, the parameters should be printed in here. \n# Score: 0.64453\n# opt_parameters_ADA = {'learning_rate': 0.08, 'n_estimators': 11}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ensembling the Models"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Final models from LGBM and ADABoost\n# estimators = [\n#     ('lgbm', lgb.LGBMClassifier(verbose_eval = 30, num_boost_round = 10000, early_stopping_rounds = 10, valid = [lgb_train, lgb_eval],\n#                                 n_jobs=4, n_estimators=3000, metric='multi_logloss', **gsLGBM.best_params_)),\n#     ('ab', AdaBoostClassifier(DecisionTreeClassifier(max_depth=3, min_samples_leaf=1, min_impurity_decrease=10, random_state=47), **gsADA.best_params_))\n# ]\n\n# If anything wrong, uncomment the following: \n# estimators = [\n#     ('lgbm', lgb.LGBMClassifier(verbose_eval = 30, num_boost_round = 10000, \n#                                 n_jobs=4, n_estimators=3000, **opt_parameters_LGBM)),\n#     ('ab', AdaBoostClassifier(DecisionTreeClassifier(max_depth=3, min_samples_leaf=1, min_impurity_decrease=10, random_state=47), **opt_parameters_ADA))\n# ]\n\n# del gsLGBM\n# del gsADA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code from https://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python#Second-Level-Predictions-from-the-First-level-Output\n# gbm = xgb.XGBClassifier(\n#  learning_rate = 0.02,\n#  n_estimators= 5,\n#  max_depth= 4,\n#  min_child_weight= 2,\n#  gamma=0.9,                        \n#  subsample=0.8,\n#  colsample_bytree=0.8,\n#  objective= 'binary',\n#  nthread= -1,\n#  verbosity=2,\n#  scale_pos_weight=1)\n\n# clf = StackingClassifier(\n#     estimators=estimators, final_estimator=gbm\n# )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# clf.fit(X_train_df, y_train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Environment for the comptetition. \n\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    # Repeat of what's written, don't know why the iterator here does not recognise what has been done before. \n    x_columns = ['user_id', 'content_id', 'content_type_id', 'task_container_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']\n    X_train_df = X_train_df\n    test_df = test_df\n    test_df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\n    test_df['prior_question_had_explanation'].fillna(False, inplace = True)\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].astype('bool')\n    test_df['answered_correctly'] = m1.predict(test_df[x_columns])\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}