{"cells":[{"metadata":{},"cell_type":"markdown","source":"> <h1>Riiid AIEd Challenge 2020</h1>\n\nFirst contact with competition and <code>riiideducation</code> package. Just have a look at the files and the test prediction iteration method to submit a dummy prediction (all predictions 0.5)."},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport riiideducation\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pickle\nfrom sklearn.model_selection import train_test_split\nimport lightgbm as lgb\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '/kaggle/input/riiid-test-answer-prediction'\nTRAIN_PICKLE = '/kaggle/input/riiid-train/train.pkl.gzip'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The train data is huge (over 101 million rows). Trying to load it into memory with a plain <code>pd.read_csv</code> leads to kernel crashing. To avoid this, we'll customize the data types used for each of the columns and read the data in chunks (thanks to Sirish for this <a href='https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188908'>hint</a>). Also, as it takes more than 9 minutes to load, after reading the train set the first time, I save it as a pickle object, much quicker to load in the future (just a few seconds), and convert the following cell to markdown. After that, I've created a (<a href='https://www.kaggle.com/jcesquiveld/riiid-train'>dataset</a> with the pickle file and added to the data for this notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntypes = {\n    'row_id': 'int64',\n    'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n    'content_type_id': 'boolean',\n    'task_container_id': 'int16',\n    'user_ans**wer': 'int8',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean'\n}\n\n# Load train dataset by chunks\ntrain = pd.DataFrame()\nfor chunk in pd.read_csv(os.path.join(DATA_DIR, 'train.csv'), chunksize=1000000, low_memory=False, dtype=types):\n    train = pd.concat([train, chunk], ignore_index=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"WORKING_DIR=\"/kaggle/working\"\ntrain.to_pickle(os.path.join(WORKING_DIR, 'train.pkl.gzip'))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nWORKING_DIR=\"/kaggle/working\"\nTRAIN_PICKLE=os.path.join(WORKING_DIR, 'train.pkl.gzip')\n# Load the train data set\ntrain_all = pd.read_pickle(TRAIN_PICKLE)\ntrain_all.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Data preparation and feature engineering</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Keep only useful columns for this version\n\nTARGET = 'answered_correctly'\ncolumns = ['user_id', 'content_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntrain = train_all.loc[train_all.content_type_id == False, columns + [TARGET]]\ndel train_all\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Calculate user_performance features\nuser_performance = train.groupby('user_id')['answered_correctly'].agg(['sum', 'count'])\nuser_performance['user_percent_correct'] = user_performance['sum'] / user_performance['count']\nuser_performance.drop(columns=['sum'], inplace=True)\nuser_performance.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Calculate question_performance features\nquestion_performance = train.groupby('content_id')['answered_correctly'].agg(['sum', 'count'])\nquestion_performance['question_percent_correct'] = question_performance['sum'] / question_performance['count']\nquestion_performance.drop(columns=['sum', 'count'], inplace=True)\nquestion_performance.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nprior_question_elapsed_time_mean = train.prior_question_elapsed_time.mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We keep only 10% of data for training\ndata = train.sample(frac=0.1)\ndata.reset_index(drop=True, inplace=True)\n\ndel train\n_ = gc.collect()\n\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add features user features and question features\n\ndata = data.join(user_performance, on='user_id')\ndata = data.join(question_performance, on='content_id')\ndata.reset_index(drop=True, inplace=True)\ndata.prior_question_had_explanation = data.prior_question_had_explanation.fillna(False).astype(np.int8)\ndata.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split into training and validation sets\n\nfeatures = ['user_percent_correct', 'count', 'question_percent_correct','prior_question_elapsed_time', \n            'prior_question_had_explanation']\ndata_train, data_val = train_test_split(data, test_size=0.20)\n\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Training</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n    'objective': 'binary',\n    'seed': 42,\n    'metric': 'auc',\n    'learning_rate': 0.05,\n    'max_bin': 800,\n    'num_leaves': 80\n}\n\nlgb_train = lgb.Dataset(data_train[features], data_train['answered_correctly'])\nlgb_val = lgb.Dataset(data_val[features], data_val['answered_correctly'])\n\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train classifier\n\nmodel = lgb.train(\n    params,\n    lgb_train,\n    valid_sets=[lgb_train, lgb_val],\n    verbose_eval=10,\n    num_boost_round=10,\n    early_stopping_rounds=1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's plot feature importance\n\nlgb.plot_importance(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = ['user_id', 'content_id', 'prior_question_elapsed_time', 'prior_question_had_explanation']\n\ndef prepare_test(test):\n    df = test[columns]\n    df = df.join(user_performance, on='user_id')\n    df = df.join(question_performance, on='content_id')\n    df.prior_question_had_explanation = df.prior_question_had_explanation.fillna(False).astype(np.int8)\n    df.prior_question_elapsed_time = df.prior_question_elapsed_time.fillna(prior_question_elapsed_time_mean)\n    df.fillna(0.5, inplace=True)\n    return df[features]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Prediction phase</h2>\n\nOnce we have trained our model(s), we're ready to make predictions. For this, we have to use the <code>riiieducation</code> API."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This has to be called once and only once in a notebook. If called twice by mistake, restart session. \nenv = riiideducation.make_env()\n\n# This is the prediction workflow\n\niter_test = env.iter_test()\nfor (test_df, prediction_df) in iter_test:\n    test_df = test_df.loc[test_df.content_type_id == 0].reset_index(drop=True)\n    test = prepare_test(test_df)\n    test_df['answered_correctly'] = model.predict(test)   \n    env.predict(test_df[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env.predictions[0].to_csv(\"/kaggle/working/submission.csv\",index=False)\nenv.predictions[0].to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's all folks"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}