{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Riiid! Answer Correctness Prediction\n**Concept taken from @Ilia Start Notebook**\n## Introduction\nIn this competition you will predict which questions each student is able to answer correctly. You will loop through a series of batches of questions. Once you make that prediction, you can move on to the next batch.\n\nThis competition is different from most Kaggle Competitions in that:\n* You can only submit from Kaggle Notebooks\n* You must use our custom **`riiideducation`** Python module.  The purpose of this module is to control the flow of information to ensure that you are not using future data to make predictions.  If you do not use this module properly, your code may fail.\n\n## In this Starter Notebook, we'll show how to use the **`riiideducation`** module to get the test features and make predictions.\n## TL;DR: End-to-End Usage Example\n```\nimport riiideducation\nenv = riiideducation.make_env()\n\n# Training data is in the competition dataset as usual\ntrain_df = pd.read_csv('/kaggle/input/riiideducation/train.csv', low_memory=False)\ntrain_my_model(train_df)\n\nfor (test_df, sample_prediction_df) in iter_test:\n    test_df['answered_correctly'] = 0.5\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])```\nNote that `train_my_model` and `make_my_predictions` are functions you need to write for the above example to work."},{"metadata":{},"cell_type":"markdown","source":"## In-depth Introduction\nFirst let's import the module and create an environment."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_types_dict = {\n    'row_id': 'int64',\n    'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n#     'content_type_id': 'int8',\n#     'task_container_id': 'int16',\n#     'user_answer': 'int8',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float16',\n    'prior_question_had_explanation': 'boolean'\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ntrain_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', \n                       nrows=10**7,\n                       usecols = data_types_dict.keys(),\n                       dtype=data_types_dict, \n                       index_col = 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training data is in the competition dataset as usual\nIt's larger than will fit in memory with default settings, so we'll specify more efficient datatypes and only load a subset of the data for now."},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_user_df = train_df.groupby('user_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_user_df.agg({'timestamp': 'max'}).hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Answered correctly¶"},{"metadata":{"trusted":true},"cell_type":"code","source":"(train_df['answered_correctly']==-1).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"~2% of activities are lectures, we should exclude them for answers analysis."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_questions_only_df = train_df[train_df['answered_correctly']!=-1]\ntrain_questions_only_df['answered_correctly'].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On average users answer ~66% questions correctly. Let's look how it is different from user to user."},{"metadata":{},"cell_type":"markdown","source":"**Answers by users**"},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_user_df = train_questions_only_df.groupby('user_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count'] })\n\nuser_answers_df[('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Look's noisy, let's clear it a little bit"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[('answered_correctly','count')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(user_answers_df[('answered_correctly','count')]< 50).mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"54% of users answered less than 50 questions. Let's divide all users into novices and active users"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[user_answers_df[('answered_correctly','count')]< 50][('answered_correctly','mean')].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[user_answers_df[('answered_correctly','count')]< 50][('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[user_answers_df[('answered_correctly','count')] >= 50][('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[user_answers_df[('answered_correctly','count')] >= 50][('answered_correctly','mean')].mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that active users do much better than novices. But anyway average user score is lower than the overall % of correct answers. It means heavy users have even better scores. Let's look at them."},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[user_answers_df[('answered_correctly','count')] >= 500][('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.scatter(x = user_answers_df[('answered_correctly','count')], y=user_answers_df[ ('answered_correctly','mean')])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Timestamp, the average score for the active user, and the number of questions answered can be useful for baseline."},{"metadata":{},"cell_type":"markdown","source":"**Answers by content**"},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_content_df = train_questions_only_df.groupby('content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count'] })","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_answers_df[('answered_correctly','count')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_answers_df[('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Different questions have different popularity and complexity, and it can also be used in the baseline."},{"metadata":{"trusted":true},"cell_type":"code","source":"content_answers_df[content_answers_df[('answered_correctly','count')]>50][('answered_correctly','mean')].hist(bins = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's try to use discovered features and use them in model to predict the right answer probability."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',\n                       usecols = data_types_dict.keys(),\n                       dtype=data_types_dict, \n                       index_col = 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features_part_df = train_df.iloc[:int(9 /10 * len(train_df))]\ntrain_part_df = train_df.iloc[int(9 /10 * len(train_df)):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_questions_only_df = features_part_df[features_part_df['answered_correctly']!=-1]\ngrouped_by_user_df = train_questions_only_df.groupby('user_id')\nuser_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count']}).copy()\nuser_answers_df.columns = ['mean_user_accuracy', 'questions_answered']\n# user_features_dict = user_answers_df.to_dict('index')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features_part_df = train_df.iloc[:int(9 /10 * len(train_df))]\ntrain_part_df = train_df.iloc[int(9 /10 * len(train_df)):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_questions_only_df = features_part_df[features_part_df['answered_correctly']!=-1]\ngrouped_by_user_df = train_questions_only_df.groupby('user_id')\nuser_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count']}).copy()\nuser_answers_df.columns = ['mean_user_accuracy', 'questions_answered']\n# user_features_dict = user_answers_df.to_dict('index')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_content_df = train_questions_only_df.groupby('content_id')\ncontent_answers_df = grouped_by_content_df.agg({'answered_correctly': ['mean', 'count'] }).copy()\ncontent_answers_df.columns = ['mean_accuracy', 'question_asked']\n# user_features_dict = conten_answers_df.to_dict('index')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df\ndel features_part_df\ndel grouped_by_user_df\ndel grouped_by_content_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = ['timestamp','mean_user_accuracy', 'questions_answered','mean_accuracy', 'question_asked', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df = train_part_df[train_part_df[target] != -1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df = train_part_df.merge(user_answers_df, how = 'left', on = 'user_id')\ntrain_part_df = train_part_df.merge(content_answers_df, how = 'left', on = 'content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df['prior_question_had_explanation'] = train_part_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\ntrain_part_df.fillna(value = -1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df = train_part_df[features + [target]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_auc_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from lightgbm import LGBMClassifier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgbm = LGBMClassifier(\n    boosting_type='gbdt', \n    num_leaves=31, \n    max_depth=- 1, \n    n_estimators=60, \n    min_child_samples=1000, \n    subsample=0.6, \n    subsample_freq=1, \n    n_jobs= 2\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgbm.fit(train_part_df[features], train_part_df[target])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_auc_score(train_part_df[target].values, lgbm.predict_proba(train_part_df[features])[:,1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\n\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.merge(user_answers_df, how = 'left', on = 'user_id')\n    test_df = test_df.merge(content_answers_df, how = 'left', on = 'content_id')\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df.fillna(value = -1, inplace = True)\n\n    test_df['answered_correctly'] = lgbm.predict_proba(test_df[features])[:,1]\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}