{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://images.squarespace-cdn.com/content/5eddaaf244c33b340e4f21bd/1600233681723-SBKXS01CLPA4OZFMJYDZ/Screen+Shot+2020-09-16+at+2.21.10+PM.png?content-type=image%2Fpng)","attachments":{}},{"metadata":{},"cell_type":"markdown","source":"# RIIID answer correctness prediction challenge"},{"metadata":{},"cell_type":"markdown","source":"This is my solution notebook to [RIIID answer correctness prediction](https://www.kaggle.com/c/riiid-test-answer-prediction) competition. This notebook is intended for beginners with little coding experience. Please Upvote if you like the notebook, which motivates me for more contributions."},{"metadata":{},"cell_type":"markdown","source":"#### *Reference:* https://www.kaggle.com/datafan07/riiid-challenge-eda-baseline-model"},{"metadata":{},"cell_type":"markdown","source":"###### Loading dependencies"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nnp.random.seed(3)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport riiideducation\nimport gc\n\n%matplotlib inline\n\nimport warnings\nwarnings.simplefilter('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"types = {\n        'row_id': 'int64','timestamp': 'int64','user_id': 'int32','content_id': 'int16','content_type_id': 'int8',\n        'task_container_id': 'int16','user_answer': 'int8','answered_correctly': 'int8', \n        'prior_question_elapsed_time': 'float32','prior_question_had_explanation': 'boolean'\n}\n\ntrain_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',low_memory=False, \n                       nrows=10**6, dtype=types)\n\nquestions_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n\nlectures_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\n\ntest_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\n\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['user_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we have 3824 unique users"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we have few missing values for last tow columns."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['content_type_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we have both lectures and questions"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['answered_correctly'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"there are 3 possible values for answered correctly column"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['content_type_id'] == 1]['user_answer'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"when the content is lecture the ANSWERED_CORRECTLY == -1 (NULL)"},{"metadata":{},"cell_type":"markdown","source":"Files [https://www.kaggle.com/c/riiid-test-answer-prediction/data](http://)\n\ntrain.csv\n\n*    row_id: (int64) ID code for the row.\n*    timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n*    user_id: (int32) ID code for the user.\n*    content_id: (int16) ID code for the user interaction\n*    content_type_id: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n*   task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n*    user_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n*    answered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures.\n*    prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n*    prior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\nlectures.csv: metadata for the lectures watched by users as they progress in their education.\n\n*   lecture_id: foreign key for the train/test content_id column, when the content type is lecture (1).\n*   part: top level category code for the lecture.\n*   tag: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n*   type_of: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"questions.csv: metadata for the questions posed to users.\n\n*     question_id: foreign key for the train/test content_id column, when the content type is question (0).\n*     bundle_id: code for which questions are served together.\n*     correct_answer: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n*     part: the relevant section of the TOEIC test.\n*     tags: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Train data shape:{train_df.shape}')\nprint(f'questions data shape:{questions_df.shape}')\nprint(f'lectures data shape:{lectures_df.shape}')\nprint(f'test data shape:{test_df.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Transform and feature engineering"},{"metadata":{},"cell_type":"markdown","source":"Encoding tags"},{"metadata":{"trusted":true},"cell_type":"code","source":"from category_encoders import HashingEncoder\nenc = HashingEncoder(cols = 'tags', n_components = 6)\nquestions_df = enc.fit_transform(questions_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nlb_make = LabelEncoder()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.loc[train_df['answered_correctly'] != -1].reset_index(drop=True)\ntrain_df = pd.merge(train_df,questions_df[['question_id','part','col_0', 'col_1', 'col_2',\n                                          'col_3', 'col_4', 'col_5']], how='left',\n                    left_on='content_id', right_on='question_id').sort_values('row_id')\ntrain_df['part'] = train_df['part'].astype('int8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"usr_ans = train_df.groupby('user_id').agg({ 'answered_correctly': ['mean','sum', 'count']})\nusr_ans.columns = ['avg_correct_answer','num_of_correct', 'total_answers']\nusr_ans['num_of_correct'] = usr_ans['num_of_correct'].astype('int16')\nusr_ans['total_answers'] = usr_ans['total_answers'].astype('int16')\n\ntrain_df = pd.merge(train_df, usr_ans, how='left', on = 'user_id')\n\ncnt_ans = train_df.groupby('content_id').agg({ 'answered_correctly': ['mean','sum', 'count']})\ncnt_ans.columns = ['avg_correct_answer_c','num_of_correct_c', 'total_answers_c']\ncnt_ans['num_of_correct_c'] = cnt_ans['num_of_correct_c'].astype('int32')\ncnt_ans['total_answers_c'] = cnt_ans['total_answers_c'].astype('int32')\n\ntrain_df = pd.merge(train_df, cnt_ans, how='left', on = 'content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_df.copy()\nX['prior_question_elapsed_time'].fillna(0, inplace=True)\nX['prior_question_had_explanation'] = X['prior_question_had_explanation'].fillna(value = False).astype(bool)\n\ndel train_df\ngc.collect()\n\nX = X.sort_values(['user_id'])\ny = X[[\"answered_correctly\"]]\nX = X.drop([\"answered_correctly\"], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(X[\"prior_question_had_explanation\"])\nX['prior_question_had_explanation_enc'] = X['prior_question_had_explanation_enc'].astype('int8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = X[['user_id','content_id','question_id','avg_correct_answer','task_container_id','num_of_correct','avg_correct_answer','total_answers','avg_correct_answer_c', \n       'num_of_correct_c','total_answers_c','prior_question_elapsed_time','prior_question_had_explanation_enc','part',\n      'col_0', 'col_1', 'col_2','col_3', 'col_4', 'col_5']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training LGBM"},{"metadata":{"trusted":true},"cell_type":"code","source":"from lightgbm import LGBMClassifier\n\nmodel = LGBMClassifier(boosting_type= 'gbdt', colsample_bytree = 0.64, learning_rate= 0.01, objective= 'binary', \n                       random_state= 500, reg_alpha= 0.8, reg_lambda= 1, subsample= 0.6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GridSearchCV"},{"metadata":{"trusted":true},"cell_type":"code","source":"#from sklearn.model_selection import GridSearchCV","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"gridParams = {'learning_rate': [0.01, 0.1],\n              'random_state' : [500], \n              'colsample_bytree' : [0.62, 0.64], \n              'subsample' : [0.6,0.65], \n              'reg_alpha' : [0.8,1], \n              'reg_lambda' : [0.9,1]}\n\ngrid = GridSearchCV(model, gridParams, verbose=1, cv=4, n_jobs=-1)\n\ngrid.fit(X, y)\n\nprint(grid.best_params_)\nprint(grid.best_score_)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cnt_ans=cnt_ans.reset_index()\nusr_ans=usr_ans.reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making predictions on test set"},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()\n\nfor (test_df, sample_prediction_df) in iter_test:\n\n    test_df = test_df.merge(usr_ans, how = 'left', on = 'user_id')\n    test_df = test_df.merge(cnt_ans, how = 'left', on = 'content_id')\n    \n    test_df = pd.merge_ordered(test_df,questions_df[['question_id','part','col_0', 'col_1', 'col_2',\n                                                     'col_3', 'col_4', 'col_5']], how='left', \n                               left_on='content_id', right_on='question_id', fill_method='ffill')\n    \n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df['prior_question_elapsed_time'].fillna(0, inplace=True)\n    test_df['avg_correct_answer'].fillna(0.5, inplace=True)\n    test_df['avg_correct_answer_c'].fillna(0.5, inplace=True)\n    test_df.fillna(value = -1, inplace = True)\n    \n    test_df[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(test_df[\"prior_question_had_explanation\"])\n  \n    y_pred = model.predict_proba(test_df[['user_id','content_id','question_id','avg_correct_answer','task_container_id','num_of_correct',\n                                          'avg_correct_answer','total_answers','avg_correct_answer_c',\n                                          'num_of_correct_c','total_answers_c','prior_question_elapsed_time',\n                                          'prior_question_had_explanation_enc','part','col_0', 'col_1', \n                                          'col_2','col_3', 'col_4', 'col_5']])[:, 1]\n    \n    test_df['answered_correctly'] = y_pred\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Thank you !!! Happy learning :)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}