{"cells":[{"metadata":{},"cell_type":"markdown","source":"Here's a introductive LightGBM model used for predicting correct answers.\nFor some memory issues faced with pandas library, it would be a nice idea to try datatable library known for speed and big data support which uses less memory, for more information about it you can read the following datatable documentation: https://datatable.readthedocs.io/en/latest/start/quick-start.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport datatable as dt\nimport lightgbm as lgb\nfrom matplotlib import pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features =  ['user_id', 'content_id', 'answered_correctly', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntrain_df = dt.fread('../input/riiid-test-answer-prediction/train.csv').to_pandas()\ntrain_df = train_df[features]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing\nLet's add some preprocessing to add more information and features to the training dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['answered_correctly'].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you see in this column 'answered_correctly' there are some indesired values which add noise to the data as well as for the predicted results of the model so it's a good habit to get rid of all empty and inapropriate values within the columns that we need for"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Eliminate rows with -1 values in the target\ntrain_df = train_df[train_df['answered_correctly'] != -1].reset_index(drop=True)\n#Replace null values with FALSE\ntrain_df.fillna(False, inplace=True)\n\ntrain_df['user_id'] = train_df['user_id'].astype('int32')\ntrain_df['content_id'] = train_df['content_id'].astype('int16')\ntrain_df['answered_correctly'] = train_df['answered_correctly'].astype('int8')\ntrain_df['prior_question_elapsed_time'] = train_df['prior_question_elapsed_time'].astype('float32')\ntrain_df['prior_question_had_explanation'] = train_df['prior_question_had_explanation'].astype('bool')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['user'] = train_df.groupby('user_id')['answered_correctly'].shift()\n#Calculate ratio of correct answers of the whole answers provided by the user\ncumulated = train_df.groupby('user_id')['user'].agg(['cumsum', 'cumcount'])\ntrain_df['user_correctness'] = cumulated['cumsum'] / cumulated['cumcount']\ntrain_df.drop(columns=['user'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_agg = train_df.groupby('user_id')['answered_correctly'].agg(['sum', 'count'])\ncontent_agg = train_df.groupby('content_id')['answered_correctly'].agg(['sum', 'count'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.groupby('user_id').tail(60).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')[['question_id', 'part']]\nquestions_df['question_id'] = questions_df['question_id'].astype('int16')\nquestions_df['part'] = questions_df['part'].astype('int8')\n\ntrain_df = pd.merge(train_df, questions_df, left_on='content_id', right_on='question_id', how='left')\ntrain_df.drop(columns=['question_id'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['content_count'] = train_df['content_id'].map(content_agg['count']).astype('int32')\ntrain_df['content_id'] = train_df['content_id'].map(content_agg['sum'] / content_agg['count'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_df = train_df.groupby('user_id').tail(15)\ntrain_df.drop(valid_df.index, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The LightGBM model and training process"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Defining the features to consider after feature engineering\nfeatures = [\n    'content_id',\n    'prior_question_elapsed_time',\n    'prior_question_had_explanation',\n    'user_correctness',\n    'part',\n    'content_count'\n]\n\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Defining LightGBM parameters\nparams = {\n    'objective': 'binary',\n    #'tree_method': 'hist'\n    'seed': 42,\n    'metric': 'auc',\n    'learning_rate': 0.05,\n    'max_bin': 800,\n    'num_leaves': 100\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr_data = lgb.Dataset(train_df[features], label=train_df[target])\nva_data = lgb.Dataset(valid_df[features], label=valid_df[target])\n\n#Training of the model\nmodel = lgb.train(\n    params, \n    tr_data, \n    num_boost_round=10000,\n    valid_sets=[tr_data, va_data], \n    early_stopping_rounds=50,\n    verbose_eval=50\n)\n\n#If you want to save the model\n# model.save_model(f'model.txt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To see how the features selected to the training are valuable it would be nice to plot their importance for the predicition of the correctness of each user's answer"},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb.plot_importance(model, importance_type='gain')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Testing the model via EducationRiid library"},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()\nprior_test_df = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    if prior_test_df is not None:\n        prior_test_df[target] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        prior_test_df = prior_test_df[prior_test_df[target] != -1].reset_index(drop=True)\n        \n        user_ids = prior_test_df['user_id'].values\n        content_ids = prior_test_df['content_id'].values\n        targets = prior_test_df[target].values\n        \n        for user_id, content_id, answered_correctly in zip(user_ids, content_ids, targets):\n            if user_id in user_agg.index:\n                user_agg.loc[user_id, 'sum'] += answered_correctly\n                user_agg.loc[user_id, 'count'] += 1\n            else:\n                user_agg.loc[user_id] = [answered_correctly, 1]\n            \n            if content_id in content_agg.index:\n                content_agg.loc[content_id, 'sum'] += answered_correctly\n                content_agg.loc[content_id, 'count'] += 1\n            else:\n                content_agg.loc[content_id] = [answered_correctly, 1]\n                \n    prior_test_df = test_df.copy()\n    \n    test_df = pd.merge(test_df, questions_df, left_on='content_id', right_on='question_id', how='left')\n    \n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(False).astype('bool')    \n    \n    test_df['user_correctness'] = test_df['user_id'].map(user_agg['sum'] / user_agg['count'])\n    \n    test_df['content_count'] = test_df['content_id'].map(content_agg['count']).fillna(1)\n    test_df['content_id'] = test_df['content_id'].map(content_agg['sum'] / content_agg['count']).fillna(0.7)\n      \n    test_df['answered_correctly'] = model.predict(test_df[features])\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion \n\nThe model is showing a very important precision results but due to the large dataset it's impossible to train it with other features from the proposed datasets (train, question and lectures) also it's only trained on a part of the users (not all of them).\nMaybe it would beneficial to use more hardware resources (RAM !!!) in order to get higher performances."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}