{"cells":[{"metadata":{},"cell_type":"markdown","source":"> <h1>Riiid AIEd Challenge 2020</h1>\n\nFirst contact with competition and <code>riiideducation</code> package. Just have a look at the files and the test prediction iteration method to submit a dummy prediction (all predictions 0.5)."},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport riiideducation\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pickle\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nimport lightgbm as lgb\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '/kaggle/input/riiid-test-answer-prediction'\nTRAIN_PICKLE = '/kaggle/input/riiid-train/train.pkl.gzip'\nWORKING_DIR = '/kaggle/working'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The train data is huge (over 101 million rows). Trying to load it into memory with a plain <code>pd.read_csv</code> leads to kernel crashing. To avoid this, we'll customize the data types used for each of the columns and read the data in chunks (thanks to Sirish for this <a href='https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908'>hint</a>). Also, as it takes more than 9 minutes to load, after reading the train set the first time, I save it as a pickle object, much quicker to load in the future (just a few seconds), and convert the following cell to markdown. After that, I've created a (<a href='https://www.kaggle.com/jcesquiveld/riiid-train'>dataset</a> with the pickle file and added to the data for this notebook."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"```python\n%%time\n\ntypes = {\n    'row_id': 'int64',\n    'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n    'content_type_id': 'boolean',\n    'task_container_id': 'int16',\n    'user_ans**wer': 'int8',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean'\n}\n\n# Load train dataset by chunks\ntrain = pd.DataFrame()\nfor chunk in pd.read_csv(os.path.join(DATA_DIR, 'train.csv'), chunksize=1000000, low_memory=False, dtype=types):\n    train = pd.concat([train, chunk], ignore_index=True)\n    \n# Save train dataset as pickle object, much quicker to load\ntrain.to_pickle(os.path.join(WORKING_DIR, 'train.pkl.gzip'))\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Load the train data set\ntrain_all = pd.read_pickle(TRAIN_PICKLE)\ntrain_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_all.to_pickle(os.path.join(WORKING_DIR, 'train.pkl.gzip'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Data preparation and feature engineering</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Keep only useful columns for this version\n\nTARGET = 'answered_correctly'\ncolumns = ['user_id', 'content_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntrain = train_all.loc[train_all.content_type_id == False, columns + [TARGET]]\ndel train_all\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Calculate user_performance features\nuser_performance = train.groupby('user_id')['answered_correctly'].agg(['sum', 'count'])\nuser_performance['user_percent_correct'] = user_performance['sum'] / user_performance['count']\nuser_performance.drop(columns=['sum'], inplace=True)\nuser_performance.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Calculate question_performance features\nquestion_performance = train.groupby('content_id')['answered_correctly'].agg(['sum', 'count'])\nquestion_performance['question_percent_correct'] = question_performance['sum'] / question_performance['count']\nquestion_performance.drop(columns=['sum', 'count'], inplace=True)\nquestion_performance.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nprior_question_elapsed_time_mean = train.prior_question_elapsed_time.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Use only 1/20 of users for training/validation\n\nnp.random.seed(45)\nusers_ids = train.user_id.unique()\ndata_users_ids = np.random.choice(users_ids, users_ids.shape[0] // 20, replace=False)\n\ndata = train.loc[train.user_id.isin(data_users_ids)]\n\ndel train\n\n_ = gc.collect()\n\ndata.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Expand data with performance features\n\ndata = data.join(user_performance, on='user_id', how='left')\ndata = data.join(question_performance, on='content_id', how='left')\ndata.reset_index(drop=True, inplace=True)\ndata.prior_question_had_explanation = data.prior_question_had_explanation.fillna(False).astype(np.int8)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Train/val split</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# For validation we use the tail with a given threshold of half of those users\n# The threshold is chosen so that aproximately we have a train/val proportion of 80/20\n# This way, there remain users with less than threshold interactions in the train set\n\nhalf_data_users_ids = np.random.choice(data_users_ids, data_users_ids.shape[0] // 2, replace=False)\ndata_val = data.loc[data.user_id.isin(half_data_users_ids)].groupby('user_id').tail(370)\ndata_train = data.drop(data_val.index)\nprint('validation set proportion', data_val.shape[0] / (data_train.shape[0] + data_val.shape[0]))\n\ndel data\n\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_train.columns.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Training</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [\n    'prior_question_elapsed_time',\n    'prior_question_had_explanation',\n    'count',\n    'user_percent_correct',\n    'question_percent_correct'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    'objective': 'binary',\n    'seed': 42,\n    'metric': 'auc',\n    'learning_rate': 0.05,\n    'max_bin': 800,\n    'num_leaves': 75\n}\n\nlgb_train = lgb.Dataset(data_train[features], data_train['answered_correctly'])\nlgb_val = lgb.Dataset(data_val[features], data_val['answered_correctly'])\n\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train classifier\n\nmodel = lgb.train(\n    params,\n    lgb_train,\n    valid_sets=[lgb_train, lgb_val],\n    verbose_eval=100,\n    num_boost_round=5000,\n    early_stopping_rounds=10\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's plot feature importance\n\nlgb.plot_importance(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = ['user_id', 'content_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']\n\n# Create features for user performance as a dict\nuser_performance_dict = {}\nfor key in user_performance.index.values:\n    user_performance_dict[key] = user_performance.loc[key].to_numpy()\n    \n# Create features for question performance as a dict\nquestion_performance_dict = {}\nfor key in question_performance.index.values:\n    question_performance_dict[key] = question_performance.loc[key].to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Prepares batch for prediction using numpy arrays and python dictionaris (no merge)\ndef prepare_test(test_df):\n    \n    test_np = test_df[columns].to_numpy()\n    x_test = np.zeros((len(test_np), len(features)))\n    for i in range(len(test_np)):\n        x_test[i,0:2] = test_np[i,2:]\n        x_test[i,2:4] = user_performance_dict.get(test_np[i][0], [0,0])\n        x_test[i,4:] = question_performance_dict.get(test_np[i][1])\n        \n    \n    return x_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Sanity check - To check we get the same result preprocessing with the prepare_test method\n\ny_val_pred = model.predict(prepare_test(data_val))\ny_val = data_val['answered_correctly']\nroc_auc_score(y_val, y_val_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv(os.path.join(DATA_DIR, 'example_test.csv'))\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%timeit\nx_test = prepare_test(test_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Prediction phase</h2>\n\nOnce we have trained our model(s), we're ready to make predictions. For this, we have to use the <code>riiieducation</code> API."},{"metadata":{"trusted":true},"cell_type":"code","source":"# To avoid running before submitting\npd.DataFrame().to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This has to be called once and only once in a notebook. If called twice by mistake, restart session. \nenv = riiideducation.make_env()\n\n# This is the prediction workflow\n\niter_test = env.iter_test()\nfor (test_df, prediction_df) in iter_test:\n    test_df = test_df.loc[test_df.content_type_id == 0].reset_index(drop=True)\n    x_test = prepare_test(test_df)\n    test_df['answered_correctly'] = model.predict(x_test)   \n    env.predict(test_df[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's all folks"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}