{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n### Versions:\n* Version 7: first baseline model\n* Version 8: tuned baseline model (lower score in LB)\n* Version 10: added new features - tunned model (lower in LB)\n* Version 11: no tunning trial\n* Version 13: added standardization, train_test_split and Feature Importance (kept only the important ones)\n* Version 15: removed standardization (lower performance) and added stratified=y in `train_test_split` (shuffle=False)\n* Version 16: adjusted the `train_data` and `features_data`, as there was a leakage","metadata":{}},{"cell_type":"code","source":"from IPython.core.display import HTML\ndef css_styling():\n styles = open(\"../input/riiid-answer-correctness-prediction-rapids/custom.css\", \"r\").read()\n return HTML(\"\")\ncss_styling()\n\n#
\n#

I. Intro: Rapids AI 🏃‍♂️

\n#
","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def notebook_styling():\n styles = open(\"../input/riiid-answer-correctness-prediction-rapids/custom_rapids.css\", \"r\").read()\n return HTML(\"\")\nnotebook_styling()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n\nThe usually used libraries like `numpy`, `pandas` or `scikit-learn` are designed to work on **CPU**. However, this competition is based on a large amount of data (100+ million rows), so these libraries will perform slowly or, even worse, they won't be able to perform at all. Hence, if you have some **GPUs** at hand (especially here on Kaggle), you can put them to work to help you in this competition :)\n\nThankfully, the researchers at Nvidia worked hard and in our favor, developing the `RAPIDS` open-source package.\n\n> RAPIDS is an open-source suite of data processing and machine learning libraries developed by NVIDIA that enables GPU-acceleration for data science workflows.\n\n
\nReferences I used:\n\n
\n\n### Libraries 📚","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Import the Rapids suite here - takes abot 1.5 mins\n\nimport sys\n!cp ../input/rapids/rapids.0.18.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Regular Libraries\nimport os\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.image as mpimg\nfrom IPython.display import display_html\nfrom PIL import Image\nimport gc\nfrom scipy.stats import pearsonr\nimport tqdm\nimport copy\nimport re\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\n\n# Color Palette\ncustom_colors = ['#7400ff', '#a788e4', '#d216d2', '#ffb500', '#36c9dd']\nsns.palplot(sns.color_palette(custom_colors))\n\n# Set Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)\n\n# Set tick size\nplt.rc('xtick',labelsize=12)\nplt.rc('ytick',labelsize=12)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*📌Note: Can't use `Dask-cuDF` because we oly have 1 worker and Memory: 13.96 in the Kaggle GPU Accelerator. If we would have had more than 1 worker, `Dask` would have performed even better :)*","metadata":{}},{"cell_type":"code","source":"# Rapids Imports\nimport cudf\nimport cupy # CuPy is an open-source array library accelerated with NVIDIA CUDA.\n\n\nfrom dask.distributed import Client, wait\nfrom dask_cuda import LocalCUDACluster\n\ncluster = LocalCUDACluster()\nclient = Client(cluster)\nclient","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> [Tesla P-100](https://www.nvidia.com/en-us/data-center/tesla-p100/)","metadata":{}},{"cell_type":"code","source":"!nvidia-smi","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CPU vs GPU comparison\n\n> code source is from RAPIDS demo notebooks: Random Forest Demo","metadata":{}},{"cell_type":"code","source":"# Imports\nimport cudf\nimport numpy as np\nimport pandas as pd\nimport pickle\n\nfrom cuml.ensemble import RandomForestClassifier as curfc\nfrom cuml.metrics import accuracy_score\n\nfrom sklearn.ensemble import RandomForestClassifier as skrfc\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\n\nimport time\n\n\n# Parameters for creating some fake data\nn_samples = 2**15\nn_features = 399\nn_info = 300\ndata_type = np.float32\n\n\n# Make Data\nX,y = make_classification(n_samples=n_samples,\n n_features=n_features,\n n_informative=n_info,\n random_state=123, n_classes=2)\n\nX = pd.DataFrame(X.astype(data_type))\n# cuML Random Forest Classifier requires the labels to be integers\ny = pd.Series(y.astype(np.int32))\n\nX_train, X_test, y_train, y_test = train_test_split(X, y,\n test_size = 0.2,\n random_state=0)\n\n# From CPU to GPU\nX_cudf_train = cudf.DataFrame.from_pandas(X_train)\nX_cudf_test = cudf.DataFrame.from_pandas(X_test)\n\ny_cudf_train = cudf.Series(y_train.values)\n\nprint(\"X shape: \", X.shape, \"\\n\" +\n \"y shape: \", y.shape)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sklearn (CPU)","metadata":{}},{"cell_type":"code","source":"print(\"==== CPU ====\", \"\\n\")\n# ==== Fit ====\n\nstart = time.time()\nsk_model = skrfc(n_estimators=40,\n max_depth=16,\n max_features=1.0,\n random_state=10)\n\nsk_model.fit(X_train, y_train)\nend = time.time()\n\nprint(\"Training time: {} mins\".format(round((end-start)/60, 1)))\n\n# ==== Evaluate ====\nstart = time.time()\nsk_predict = sk_model.predict(X_test)\nsk_acc = accuracy_score(y_test, sk_predict)\nend = time.time()\n\nprint(\"Evaluation time: {} mins\".format(round((end-start)/60, 1)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### CuML (GPU)","metadata":{}},{"cell_type":"code","source":"print(\"==== GPU ====\", \"\\n\")\n# ==== Fit ====\n\nstart = time.time()\ncuml_model = curfc(n_estimators=40,\n max_depth=16,\n max_features=1.0,\n random_state=10,\n n_streams=1) # for reproducibility\n\ncuml_model.fit(X_cudf_train, y_cudf_train)\nend = time.time()\n\nprint(\"Training time: {} mins\".format(round((end-start)/60, 1)))\n\n# ==== Evaluate ====\nstart = time.time()\nfil_preds_orig = cuml_model.predict(X_cudf_test)\n\nfil_acc_orig = accuracy_score(y_test.to_numpy(), fil_preds_orig)\nend = time.time()\n\nprint(\"Evaluation time: {} mins\".format(round((end-start)/60, 1)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

In this experiment, CuML is net superior (MUCH faster) than Sklearn - almost 6 minutes vs 30 seconds.

\n
","metadata":{}},{"cell_type":"markdown","source":"\n\n
\nIn this section we'll use the cudf and cupy libraries provided by RAPIDS, combined with numpy for the plotting part. The notebook runs at the moment in 3 minutes.\n
\n\n# 1. train.csv\n\n* `row_id`: (int64) ID code for the row.\n* `timestamp`: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n* `user_id`: (int32) ID code for the user.\n* `content_id`: (int16) ID code for the user interaction\n* `content_type_id`: (bool) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n* `task_container_id`: (int16) ID code for the *batch of questions or lectures*. (eg. a user might see three questions in a row before seeing the explanations for any of them - those three would all share a task_container_id)\n* `user_answer`: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n* `answered_correctly`: (int8) if the user responded correctly. Read -1 as null, for lectures.\n* `prior_question_elapsed_time`: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between (is null for a user's first question bundle or lecture)\n* `prior_question_had_explanation`: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Read in data\ndtypes = {\n \"row_id\": \"int64\",\n \"timestamp\": \"int64\",\n \"user_id\": \"int32\",\n \"content_id\": \"int16\",\n \"content_type_id\": \"boolean\",\n \"task_container_id\": \"int16\",\n \"user_answer\": \"int8\",\n \"answered_correctly\": \"int8\",\n \"prior_question_elapsed_time\": \"float32\", \n \"prior_question_had_explanation\": \"int8\"\n}\n\ntrain = cudf.read_csv('../input/riiid-test-answer-prediction/train.csv', dtype=dtypes)\n\n# # Drop \"row_id\" column as it doesn't give any information\n# train = train.drop(columns = [\"row_id\"], axis=1, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌Note: The only 2 columns with missing data (explained in documentation - `NULL` values are present for the first question bundle)","metadata":{}},{"cell_type":"code","source":"# Data Information\nprint(\"Rows: {:,}\".format(len(train)), \"\\n\" +\n \"Columns: {}\".format(len(train.columns)))\n\n# Find Missing Data if any\ntotal = len(train)\n\nfor column in train.columns:\n if train[column].isna().sum() != 0:\n print(\"{} has: {:,} ({:.2}%) missing values.\".format(column, train[column].isna().sum(), \n (train[column].isna().sum()/total)*100))\n \n \n# Fill in missing values with \"-1\"\ntrain[\"prior_question_elapsed_time\"] = train[\"prior_question_elapsed_time\"].fillna(-1)\ntrain[\"prior_question_had_explanation\"] = train[\"prior_question_had_explanation\"].fillna(-1)\n\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1 Columns individual analysis\n\n* numerical features (distplot): `timestamp`, `prior_question_elapsed_time`\n* categorical features (distplot): `user_id` count, `content_id` count, `task_container_id` count\n* categorical features (barplot): `user_answer` count, `answered_correctly` count, `prior_question_had_explanation` count\n\n### Predefined functions📂\n\nBecause there is no possibility (yet) to use Rapids for visualization we need to preprocess and convert the data to numpy arrays and plot it afterwards.","metadata":{}},{"cell_type":"code","source":"def distplot_features(df, feature, title, color = custom_colors[4], categorical=True):\n '''Takes a column from the GPU dataframe and plots the distribution (after count).'''\n \n if categorical:\n values = cupy.asnumpy(df[feature].value_counts().values)\n else:\n values = cupy.asnumpy(df[feature].values)\n \n print('Mean: {:,}'.format(np.mean(values)), \"\\n\"\n 'Median: {:,}'.format(np.median(values)), \"\\n\"\n 'Max: {:,}'.format(np.max(values)))\n\n \n plt.figure(figsize = (18, 3))\n \n if categorical:\n sns.distplot(values, hist=False, color = color, kde_kws = {'lw':3})\n else:\n # To speed up the process\n sns.distplot(values[::250000], hist=False, color = color, kde_kws = {'lw':3})\n \n plt.title(title, fontsize=15)\n plt.show();\n \n del values\n gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def barplot_features(df, feature, title, palette = custom_colors[2:]):\n '''Takes the numerical columns (with less than 10 categories) and plots the barplot.'''\n \n # We need to extract both the name of the category and the no. of appearences\n index = cupy.asnumpy(df[feature].value_counts().reset_index()[\"index\"].values)\n values = cupy.asnumpy(df[feature].value_counts().reset_index()[feature].values) \n\n plt.figure(figsize = (18, 3))\n sns.barplot(x = index, y = values, palette = palette)\n plt.title(title, fontsize=15)\n plt.show();\n \n del index, values\n gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inspect numerical features","metadata":{}},{"cell_type":"code","source":"numerical_features = ['timestamp', 'prior_question_elapsed_time']\n\nfor feature in numerical_features:\n distplot_features(train, feature=feature, title = feature + \" distribution\", color = custom_colors[2], categorical=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inspect Categorical Features: many values","metadata":{}},{"cell_type":"code","source":"categorical_features = ['user_id', 'content_id', 'task_container_id']\n\nfor feature in categorical_features:\n distplot_features(train, feature=feature, title = feature + \" countplot distribution\", color = custom_colors[3], categorical=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inspect Categorical Features: fiew values\n\n> There are only a fiew cases where content_type_id is = 1 (meaning lectures) - which is good, we're not supposed to predict those anyways.","metadata":{}},{"cell_type":"code","source":"categorical_for_bar = ['content_type_id', 'user_answer', \n 'answered_correctly', 'prior_question_had_explanation']\n\nfor feature in categorical_for_bar:\n barplot_features(train, feature=feature, title = feature + \" barplot\", palette = custom_colors)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2 Handle Outliers\n\n> 📌Note: The **outliers** might strongly influence the future models. Hence, we need to carefully handle them. However, by trying to erase the outliers we can erase up to 10% of the data, which is valuable information for training our models.\n\nEg.: Below we find the upper boundry for the outliers for the feature `timestamp`. This value is equal to 26,138,936,852 milliseconds, which means 7,700 hours, which means about 320 days.","metadata":{}},{"cell_type":"code","source":"# Total rows we started with\ntotal = len(train)\nfeature = \"timestamp\"\n\n# Compute Outliers\nQ1 = cupy.percentile(train[feature].values, q = 25).item()\nQ3 = cupy.percentile(train[feature].values, q = 75).item()\nIQR = Q3 - Q1\n\n# We'll look only at the upper interval outliers\noutlier_boundry = Q3 + 1.5*IQR\n\nprint('Timestamp: around {:.2}% of the data would be erased.'.format(len(train[train[feature] >= outlier_boundry])/total * 100), \n \"\\n\"+\n 'The outlier boundry is {:,}, which means {:,.5} hrs, which means {:,.5} days.'.format(outlier_boundry, (outlier_boundry / 3.6e+6),\n (outlier_boundry / 3.6e+6)/24))\n\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌Note: However, I would erase all pupils (`user_id`) that have less than 5 appearences in the data (no prediction can be made on these students) ","metadata":{}},{"cell_type":"code","source":"# Select ids to erase\nids_to_erase = train[\"user_id\"].value_counts().reset_index()[train[\"user_id\"].value_counts().reset_index()[\"user_id\"] < 5]\\\n [\"index\"].values\n\n# Erase the ids\nnew_train = train[~train['user_id'].isin(ids_to_erase)]\n\nprint(\"We erased {} rows meaning {:.3}% of all data.\".format(len(train)-len(new_train), (1 - len(new_train)/len(train))*100))\ndel ids_to_erase","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.3 Curiosities\n\nFirst I would like to see if the number of appearences of 1 ID (1 pupil) corresponds to their overall performance (other words, if you assess 1 student multiple times, will they perform better?)","metadata":{}},{"cell_type":"code","source":"# Count how many times the user answered correctly out of all available times\nuser_performance = train.groupby(\"user_id\").agg({ 'row_id': ['count'], 'answered_correctly': ['sum'] }).reset_index()\nuser_performance.columns = [\"user_id\", \"total_count\", \"correct_count\"]\nuser_performance[\"performance\"] = user_performance[\"correct_count\"] / user_performance[\"total_count\"]\n\n# Create intervals for number of appearences\n# between 0 and 1000, 1000 and 2500 and 2500+\ndef condition(x):\n if x <= 1000:\n return 0\n elif (x > 1000) & (x <= 2500):\n return 1\n else:\n return 2\n \nuser_performance[\"total_interval\"] = user_performance[\"total_count\"].applymap(condition)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌Note: So yes, the *average* performance increases along with the number of times one student appears in the data.","metadata":{}},{"cell_type":"code","source":"# Convert to numpy arrays (so we can plot)\nx = cupy.asnumpy(user_performance[\"total_interval\"].values)\ny = cupy.asnumpy(user_performance[\"performance\"].values)\n\n# Plot\nplt.figure(figsize = (18, 4))\nsns.barplot(x = x, y = y, palette = custom_colors[2:])\nplt.title(\"Performance over number of appearences\", fontsize = 15)\nplt.xticks([0, 1, 2], ['<1000', '1000-2500', '2500+']);\n\ndel user_performance","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.4 Save and delete\n\n> To keep the notebook as light as possible and to not overload the memory, we save the `train` data in .feather format (lighter, takes about 7 seconds to upload using `cudf`) and delete the dataframes.\n\n*UPDATE: even better - as Chris Deotte recommends: \"Instead of using .feather use .parquet. RAPIDS cuDF can read parquet 6x faster than feather (and writes parquet 1.1x faster)\"*","metadata":{}},{"cell_type":"code","source":"# First cleanup\ndel train\ngc.collect()\n\n\n# Checkpoint: save to .parquet\nnew_train.to_parquet('new_train.parquet')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Clean the environment\ndel new_train\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. questions.csv\n\n* `question_id`: foreign key for the train/test `content_id` column, when the content type is question (0).\n* `bundle_id`: code for which questions are served together.\n* `correct_answer`: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n* `part`: the relevant section of the TOEIC test.\n* `tags`: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n\n> The Test of English for International Communication (TOEIC) is an international standardized test of English language proficiency for non-native speakers.","metadata":{}},{"cell_type":"code","source":"questions = cudf.read_csv('../input/riiid-test-answer-prediction/questions.csv')\n\n# Data Information\nprint(\"Rows: {:,}\".format(len(questions)), \"\\n\" +\n \"Columns: {}\".format(len(questions.columns)))\n\n# Find Missing Data if any\ntotal = len(questions)\n\nfor column in questions.columns:\n if questions[column].isna().sum() != 0:\n print(\"{} has: {:,} ({:.2}%) missing values.\".format(column, questions[column].isna().sum(), \n (questions[column].isna().sum()/total)*100))\n \n \n# Fill in missing values with \"-1\"\nquestions[\"tags\"] = questions[\"tags\"].fillna(-1)\n\nquestions.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.1 Inspect the columns\n\n* categorical features (distplot): `question_id` count, `bundle_id` count, `tags` count\n* categorical features (barplot): `correct_answer`, `part`","metadata":{}},{"cell_type":"code","source":"# ----- question_id -----\n\n# The table contains an equal number of IDs for each question\nprint('There is a total of {:,} IDs.'.format(len(questions['question_id'].value_counts())), \"\\n\")\n\n# ----- bundle_id -----\nprint('There are {:,} unique bundle IDs.'.format(questions['bundle_id'].nunique()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌Note: majority of the questions are from part 5 - if this distribution doesn't match the `test` set, there might be some issues :)","metadata":{}},{"cell_type":"code","source":"# ----- part & correct_answer -----\n\nfor feature in ['part', 'correct_answer']:\n barplot_features(questions, feature=feature, title=feature + \" - barplot distribution\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ----- tags -----\ndistplot_features(questions, 'tags', title = \"Tags - Count Distribution\", color = custom_colors[0], categorical=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Save and delete","metadata":{}},{"cell_type":"code","source":"# Checkpoint: save to parquet\nquestions.to_parquet('questions.parquet')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del questions\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. lectures.csv\n\n* `lecture_id`: foreign key for the train/test `content_id` column, when the content type is lecture (1).\n* `part`: top level category code for the lecture.\n* `tag`: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n* `type_of`: brief description of the core purpose of the lecture (`string` - so this data needs to be treated a bit different)\n\n*no missing values*","metadata":{}},{"cell_type":"code","source":"lectures = cudf.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\n\n# Encode 'type_of' column\nlectures.type_of,codes = lectures['type_of'].factorize()\n\n# Data Information\nprint(\"Rows: {:,}\".format(len(lectures)), \"\\n\" +\n \"Columns: {}\".format(len(lectures.columns)))\nlectures.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1 Inspect the columns","metadata":{}},{"cell_type":"code","source":"# ----- lecture_id -----\n# The table contains an equal number of IDs for each question\nprint('There is a total of {:,} IDs.'.format(len(lectures['lecture_id'].value_counts())), \"\\n\")\n\n# There are 151 unique tags\nprint('There are a total of {:,} unique tags IDs.'.format(len(lectures['tag'].value_counts())))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Save and delete","metadata":{}},{"cell_type":"code","source":"lectures.to_parquet(\"lectures.parquet\")\n\ndel lectures\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n> Let's look again at the structure of our data:\n\n\n
\n

This section uses the cuML package and XGBoost to compute the predictions.

\n
","metadata":{}},{"cell_type":"code","source":"cudf.set_allocator(\"managed\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Import the data\ntrain = cudf.read_parquet(\"../input/riiid-answer-correctness-prediction-rapids/new_train.parquet\")\nquestions = cudf.read_parquet(\"../input/riiid-answer-correctness-prediction-rapids/questions.parquet\")\n\n# Lectures we won't load, as we are not supposed to predict for these rows","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Let's exclude all observations where (content_type_id = 1) & (answered_correctly = -1)\ntrain = train[train['content_type_id'] != 1]\ntrain = train[train['answered_correctly'] != -1].reset_index(drop=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Feature Engineering","metadata":{}},{"cell_type":"code","source":"# Parameters\ntrain_percent = 0.1\ntotal_len = len(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split data into train data & feature engineering data (to use for past performance)\n# Timestamp is in descending order - meaning that the last 10% observations have\n# the biggest chance of having had some performance recorded before\n# so looking at the performance in the past we'll try to predict the performance now\n\nfeatures_df = train.iloc[ : int(total_len*(1-train_percent))]\ntrain_df = train.iloc[int(total_len*(1-train_percent)) : ]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1 Feature Engineering - Create Data","metadata":{}},{"cell_type":"code","source":"%%time\n# --- STUDENT ANSWERS ---\n# Group by student\nuser_answers = features_df[features_df['answered_correctly']!=-1].\\\n groupby('user_id').\\\n agg({'answered_correctly': ['sum', 'mean', 'min', \n 'max', 'count', 'median', \n 'std', 'var']}).\\\n reset_index()\n\nuser_answers.columns = ['user_id', 'user_sum', 'user_mean', 'user_min', 'user_max', \n 'user_count', 'user_median', 'user_std', 'user_var']\n\n\n# --- CONTENT ID ANSWERS ---\n# Group by content\ncontent_answers = features_df[features_df['answered_correctly']!=-1].\\\n groupby('content_id').\\\n agg({'answered_correctly': ['sum', 'mean', 'min', \n 'max', 'count', 'median', \n 'std', 'var']}).\\\n reset_index()\n\ncontent_answers.columns = ['content_id', 'content_sum', 'content_mean', 'content_min', \n 'content_max', 'content_count', 'content_median', 'content_std', \n 'content_var']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Save FE data; we will use it for the `test` set too :)","metadata":{}},{"cell_type":"code","source":"user_answers.to_parquet('user_answers.parquet')\ncontent_answers.to_parquet('content_answers.parquet')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train, questions\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2 Predefined Functions for Preprocesing¶\n\n> Combine new features with the `train_df`","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Features for ML\nfeatures_to_keep = ['user_sum', 'user_mean', 'user_min', 'user_max', \n 'user_count', 'user_median', 'user_std', 'user_var',\n 'content_sum', 'content_mean', 'content_min', \n 'content_max', 'content_count', 'content_median', 'content_std', \n 'content_var']\ntarget = 'answered_correctly'\n\nall_features = features_to_keep.copy()\nall_features.append(target)\n\n\n# We need to convert True-False variables to integers\ndef to_bool(x):\n '''For the string variables.'''\n if x == False:\n return 0\n else:\n return 1\n\n \ndef combine_features(data = None):\n '''Combine the features with the Train/Test data.'''\n \n # Add \"past\" information\n features_data = data.merge(user_answers, how = 'left', on = 'user_id')\n features_data = features_data.merge(content_answers, how = 'left', on = 'content_id')\n\n # Apply\n features_data['content_type_id'] = features_data['content_type_id'].applymap(to_bool)\n features_data['prior_question_had_explanation'] = features_data['prior_question_had_explanation'].applymap(to_bool)\n\n # Fill in missing spots\n features_data.fillna(value = -1, inplace = True)\n \n return features_data\n\n\n# Scaling the data did not perform as I expected to - so for now we will exclude it\ndef scale_data(features_data=None, train=True, features_to_keep=None, target=None):\n '''Scales the provided data - if the data is for training, excludes the target column.\n It also chooses the features used in the prediction.'''\n \n data_for_standardization = features_data[features_to_keep]\n matrix = data_for_standardization.as_matrix()\n scaled_matrix = StandardScaler().fit_transform(matrix)\n \n scaled_data = cudf.DataFrame(scaled_matrix)\n scaled_data.columns = data_for_standardization.columns\n \n # We don't want to scale the target also\n if train:\n scaled_data[target] = features_data[target]\n \n return scaled_data","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.3 Apply Functions - getting data ready","metadata":{}},{"cell_type":"code","source":"%%time\n\ntrain_df = combine_features(data=train_df)\n# train_df = scale_data(features_data=train_df, train=True, features_to_keep=features_to_keep, target=target)\n\n# Comment this if you're scaling\ntrain_df = train_df[all_features]\n\nprint(\"Observations in train: {:,}\".format(len(train_df)))\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. XGBoost Model\n\n> The [Stratified K Fold](https://nbviewer.jupyter.org/github/daxiongshu/notebooks-extended/blob/kdd_plasticc/competition_notebooks/kaggle/landmark/cudf_stratifiedKfold_1000x_speedup.ipynb) function using `cudf` is from Grandmaster [Jiwei Liu](https://www.kaggle.com/jiweiliu)","metadata":{}},{"cell_type":"code","source":"# RAPIDS roc_auc_score is 16x faster than sklearn. - cdeotte\nfrom cuml.metrics import roc_auc_score\nfrom cuml.preprocessing.model_selection import train_test_split\nimport xgboost\nimport pickle","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Features, target and train/test split\nX = train_df[features_to_keep]\ny = train_df[target]\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, \n shuffle=False, stratify=y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.1 Baseline Model\n\n### Helper Function that runs multiple models","metadata":{}},{"cell_type":"code","source":"def train_xgb_model(X_train, X_test, y_train, y_test, params, details=\"default\", prints=True):\n '''Trains an XGB and returns the trained model + ROC value.'''\n # Create DMatrix - is optimized for both memory efficiency and training speed.\n train_matrix = xgboost.DMatrix(data = X_train, label = y_train)\n \n # Create & Train the model\n model = xgboost.train(params, dtrain = train_matrix)\n\n # Make prediction\n predicts = model.predict(xgboost.DMatrix(X_test))\n roc = roc_auc_score(y_test.astype('int32'), predicts)\n\n if prints:\n print(details + \" - ROC: {:.5}\".format(roc))\n \n return model, roc\n\n\ndef param_tuning_graph(param_values, roc_values):\n '''Represents visually the ROC results for the speciffic parameter tune.'''\n \n plt.figure(figsize=(18, 3))\n ax = sns.barplot(x=param_values, y=roc_values, palette=custom_colors)\n\n for p in ax.patches:\n width = p.get_width()\n height = p.get_height()\n x, y = p.get_xy() \n ax.annotate(f'{height:.5%}', (x + width/2, y + height*1.02), ha='center')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nparams1 = {\n 'max_depth' : 4,\n 'max_leaves' : 2**4,\n 'tree_method' : 'gpu_hist',\n 'objective' : 'reg:logistic',\n 'grow_policy' : 'lossguide'\n}\n\nmodel1, roc1 = train_xgb_model(X_train, X_test, y_train, y_test, \n params1, details=\"baseline model\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

We have a ROC score of 0.71623 in ~14 seconds.

\n

Incredible.

\n
","metadata":{}},{"cell_type":"code","source":"# save model to file\npickle.dump(model1, open(\"xgb_baseline.pickle.dat\", \"wb\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Fine Tuning\n\n> Let's see if we can improve the ROC by only tuning the hyperparameters.","metadata":{}},{"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\netas2 = [0.001, 0.005, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6]\n\nfor eta in etas2:\n params2 = {\n 'max_depth' : 4,\n 'max_leaves' : 2**4,\n 'tree_method' : 'gpu_hist',\n 'objective' : 'reg:logistic',\n 'grow_policy' : 'lossguide',\n 'eta' : eta\n }\n \n _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n params2, details = f\"ETA: {eta}\")\n rocs2.append(roc)\n \n \n# Optional: Further tunning\nmax_depths = [4, 5, 6, 7, 8]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # --- SAVE CHECKPOINT ---\n# params_final = {\n# 'max_depth' : 8,\n# 'max_leaves' : 2**4,\n# 'tree_method' : 'gpu_hist',\n# 'objective' : 'reg:logistic',\n# 'grow_policy' : 'lossguide',\n# 'eta' : 0.5\n# }\n\n# model2, roc2 = train_xgb_model(X_train, X_test, y_train, y_test, \n# params_final, prints=None)\n\n# # save model to file\n# pickle.dump(model2, open(\"baseline_model_tuned.pickle.dat\", \"wb\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4 Feature Importance","metadata":{}},{"cell_type":"code","source":"import shap\n\nexplainer = shap.TreeExplainer(model1)\nshap_values = explainer.shap_values(X_test.to_pandas())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.summary_plot(shap_values, X_test.to_pandas(), plot_type=\"bar\", color=custom_colors[3],\n title = \"\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. LightGBM Model\n\n
\n

If you know any resource that shows how you can use a GPU dataframe to work with Stratified Folding and LGBM please comment :) I've been struggling to find some good tutorials and this is the best I could do so far.

\n
","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold, train_test_split\nfrom sklearn import metrics\nimport lightgbm as lgbm\nfrom sklearn import metrics\nimport gc\nimport pickle","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# We'll do a train | validation | test situation\ntrain, test = train_test_split(train_df, test_size=0.3, shuffle=False)\n\ntrain = train.to_pandas()\ntest = test.to_pandas()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# -----------\nn_splits = 2\n# -----------\n\nskf = StratifiedKFold(n_splits=n_splits, shuffle=True)\n\noof = np.zeros(len(train))\npredictions = np.zeros(len(test))\n\n# Covertion to CPU data\nskf_split = skf.split(X=train[features_to_keep], y=cupy.asnumpy(train[target].values))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"param = {\n 'num_leaves': 80,\n 'max_bin': 250,\n 'min_data_in_leaf': 11,\n 'learning_rate': 0.01,\n 'min_sum_hessian_in_leaf': 0.00245,\n 'bagging_fraction': 1.0, \n 'bagging_freq': 5, \n 'feature_fraction': 0.05,\n 'lambda_l1': 4.972,\n 'lambda_l2': 2.276,\n 'min_gain_to_split': 0.65,\n 'max_depth': 14,\n 'save_binary': True,\n 'seed': 1337,\n 'feature_fraction_seed': 1337,\n 'bagging_seed': 1337,\n 'drop_seed': 1337,\n 'data_random_seed': 1337,\n 'objective': 'binary',\n 'boosting_type': 'gbdt',\n 'verbose': 1,\n 'metric': 'auc',\n 'is_unbalance': True,\n 'boost_from_average': False,\n 'device': 'gpu',\n 'gpu_platform_id': 0,\n 'gpu_device_id': 0\n }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Training Loop\ncounter = 1\n\nfor train_index, valid_index in skf_split:\n print(\"==== Fold {} ====\".format(counter))\n \n lgbm_train = lgbm.Dataset(data = train.iloc[train_index, :][features_to_keep].values,\n label = train.iloc[train_index, :][target].values,\n feature_name = features_to_keep,\n free_raw_data = False)\n \n lgbm_valid = lgbm.Dataset(data = train.iloc[valid_index, :][features_to_keep].values,\n label = train.iloc[valid_index, :][target].values,\n feature_name = features_to_keep,\n free_raw_data = False)\n \n lgbm_2 = lgbm.train(params = param, train_set = lgbm_train, valid_sets = [lgbm_valid],\n early_stopping_rounds = 12, num_boost_round=100, verbose_eval=25)\n \n \n # X_valid to predict\n oof[valid_index] = lgbm_2.predict(train.iloc[valid_index][features_to_keep].values, \n num_iteration = lgbm_2.best_iteration)\n predictions += lgbm_2.predict(test[features_to_keep], \n num_iteration = lgbm_2.best_iteration) / n_splits\n \n counter += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"CV ROC: {:<0.2f}\".format(metrics.roc_auc_score(test[target], predictions)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # save model to file\n# # the model was initially trained on 10,000 num_boost_round\n# # but to keep the notebook running quiqly I modified it in the example to only 100\n# pickle.dump(lgbm_2, open(\"lgbm_10000_12.pickle.dat\", \"wb\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n> This competition has the inference a bit different - for reference please check [this notebook.](https://www.kaggle.com/sohier/competition-api-detailed-introduction/comments) Hence, in the `for` loop we create the actual submission (can be seen in the output).\n\n> Training Part: We don't have access to test set labels because we are predicting before getting labels. That's how it is in real world too: You predict today and get the labels tomorrow. You can use tomorrow's labels to predict the day after. And so on. Per day or per batch. - Vopani\n\nThings we'll have to apply to `test` set:\n* we assume the `test` set DOESN'T have `content_type_id` = 1 NOR `answered_correctly` = -1\n* add information from `user_answers`\n* add information from `content_answers`\n* change True/False columns to 1/0\n* fill missing values with -1\n* normalize the features\n\nBonus: select only the features we want :)","metadata":{}},{"cell_type":"code","source":"lgbm_2 = pickle.load(open('../input/riiid-answer-correctness-prediction-rapids/lgbm_10000_12.pickle.dat', 'rb'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import library and create environment\nimport riiideducation\nenv = riiideducation.make_env()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here you would also add your pretrained model\niter_test = env.iter_test()\n\nfor (test_df, sample_prediction_df) in iter_test:\n test_df = cudf.from_pandas(test_df)\n \n # --- PREPROCESSING ---\n # Here is time to apply the preprocessing to the test_df\n test_df = combine_features(data = test_df)\n #X = scale_data(features_data=test_df, train=False, features_to_keep=features_to_keep).to_pandas()\n X = test_df[features_to_keep].to_pandas()\n \n # --- MODEL ---\n test_df['answered_correctly'] = lgbm_2.predict(X, num_iteration = lgbm_2.best_iteration)\n test_df = test_df.to_pandas()\n env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n# Specs on how I prepped & trained ⌨️🎨\n### (*locally*)\n* Z8 G4 Workstation 🖥\n* 2 CPUs & 96GB Memory 💾\n* NVIDIA Quadro RTX 8000 🎮\n* RAPIDS version 0.17 🏃🏾‍♀️","metadata":{}}]}