{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{},"cell_type":"markdown","source":"*This notebook was forked from Andrada Olteanu's notebook https://www.kaggle.com/andradaolteanu/answer-correctness-rapids-xgb-lgbm/notebook. It was really a big help to me as I'm nowhere near able to really understand how RAPIDS work at the time where I wrote this. Go check her notebook! (It's much more well-written and clear! And it's also contain additionnal information on RAPIDS!)*\n\n\nMy main goal here was to find a way to improve the auc score throught FE or something else by at least a bit. As it's my first competition (and also my first with RAPIDS), I tried to not have too much expectation. \n\nAnyway, it was still a interesting experience. Here are my conclusions, maybe it will help some people:\n* RAPIDS is reeaaally fast! But still don't think that you can do anything. Always delete the things that don't use anymore if you don't want to run into a lot of memory error. I was a bit too much enthusiaste after seeing how fast it fast and didn't pay attention to memory management. Now, I have nightmare about black rectangle with red, green and MemoryError: std::bad_alloc: CUDA in it... And even if you are meticulous, some operations are too costly for Kaggle. \n* The feature engineering made by Andrada Olteanu was already really good. The various indicators (sum, count, mean, std, var, etc. by user and question) capture about 71% of the information. No matter what I tried, it didn't seem  that I could make better features. The first big improvement (and by big, I mean +0.006~...) was simply by adding 'prior_question_had_explanation' to the feature to keep. I also tried to scale the data but adding other variables and scaling was too much for the memory. But even when it worked, it didn't really improve the score.\n* Other than that, I think the main limit here was that some student have \"way too much free time\" like I read it elswhere. Some student appear more than 15000 time. I might be wrong but I think that as they practice a lot, the became more consistant (they give more often the right answer) and thus more predictibles. I tried to make my way around that by removing all rows that have a timestamp superior to the upper outlier boundry for the timestamp and it improve the performance! But not on the hidden test data set... \n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%time\nimport sys\n!cp ../input/rapids/rapids.0.17.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport psutil\nimport gc\n\nimport riiideducation\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# Color Palette\ncustom_colors = ['#7400ff', '#a788e4', '#d216d2', '#ffb500', '#36c9dd']\nsns.palplot(sns.color_palette(custom_colors))\n\n# Set Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)\n\n# Set tick size\nplt.rc('xtick',labelsize=12)\nplt.rc('ytick',labelsize=12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Rapids Imports\nimport cudf\nimport cupy # CuPy is an open-source array library accelerated with NVIDIA CUDA.\n\n\nfrom dask.distributed import Client, wait\nfrom dask_cuda import LocalCUDACluster\n\ncluster = LocalCUDACluster()\nclient = Client(cluster)\nclient","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cudf.set_allocator(\"managed\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Import the data\ntrain = cudf.read_parquet(\"../input/riid-competition-rapids-part-i-eda/clean_train.parquet\")\nquestions = cudf.read_parquet(\"../input/riid-competition-rapids-part-i-eda/questions.parquet\", columns=['question_id','bundle_id', 'part'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train['row_id']\ndel train['task_container_id']\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# I) Cleaning"},{"metadata":{},"cell_type":"markdown","source":"**Removing outliers**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Select ids to erase\nids_to_erase = train[\"user_id\"].value_counts().reset_index()[(train[\"user_id\"].value_counts().reset_index()[\"user_id\"] < 10) |\n                                                            (train[\"user_id\"].value_counts().reset_index()[\"user_id\"] > 15000)]\\\n                                                                                                                [\"index\"].values\nprevious_length = len(train)\n\n# Erase the ids\ntrain = train[~train['user_id'].isin(ids_to_erase)]\n\nprint(\"We erased {} rows meaning {:.3}% of all data.\".format(previous_length-len(train), (1 - len(train)/previous_length)*100))\ndel ids_to_erase, previous_length\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# del train['timestamp'], total, feature, Q1, Q3, Q05, Q95, IQR, upper_outlier_boundry\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# II) Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.merge(questions, how = 'left', left_on = 'content_id', right_on = 'question_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train['question_id']\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters\ntrain_percent = 0.1\ntotal_len = len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split data into train data & feature engineering data\n# The data is ordered by timestamp and user_id, so that the last 10% observations are new observations\n#  is in descending order - meaning that the last 10% observations have\n# the biggest chance of having had some performance recorded before\n# so looking at the performance in the past we'll try to predict the performance now\n\nfeatures_df = train.iloc[ : int(total_len*(1-train_percent))]\ntrain_df = train.iloc[int(total_len*(1-train_percent)) : ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Total rows we started with\ntotal = len(features_df)\nfeature = \"timestamp\"\n\n# Compute Outliers\nQ1 = cupy.percentile(features_df[feature].values, q = 25).item()\nQ3 = cupy.percentile(features_df[feature].values, q = 75).item()\nIQR = Q3 - Q1\n\nupper_outlier_boundry = Q3 + 1.5*IQR\n\nprint('The upper outlier boundry is {:,}, which means {:,.5} hrs, which means {:,.5} days.'.format(upper_outlier_boundry, (upper_outlier_boundry / 3.6e+6),\n                                                                                       (upper_outlier_boundry / 3.6e+6)/24))\n\nprint('Timestamp: around {:.2}% of the data have been erased.'.format((len(features_df[features_df[feature] > upper_outlier_boundry])/total) * 100))\n\n\nfeatures_df = features_df[features_df['timestamp'] <= upper_outlier_boundry]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_lectures=features_df[features_df['answered_correctly']==-1]\nuser_lectures['lec_sum']=user_lectures['answered_correctly']*-1\nuser_lectures=user_lectures[['user_id', 'lec_sum']].groupby('user_id').agg({'lec_sum': 'sum'}).reset_index()\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_lectures.to_parquet('user_lectures.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Let's exclude all observations where (content_type_id = 1) & (answered_correctly = -1)\nfeatures_df = features_df[features_df['content_type_id'] != 1]\nfeatures_df = features_df[features_df['answered_correctly'] != -1].reset_index(drop=True)\nfeatures_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Let's exclude all observations where (content_type_id = 1) & (answered_correctly = -1)\ntrain_df = train_df[train_df['content_type_id'] != 1]\ntrain_df = train_df[train_df['answered_correctly'] != -1].reset_index(drop=True)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- STUDENT ANSWERS ---\n# Group by student\nuser_answers = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby('user_id').\\\n                            agg({'answered_correctly': ['sum', 'mean', 'count', 'std']}).\\\n                            reset_index()\n\nuser_answers.columns = ['user_id', 'user_sum', 'user_mean', \n                        'user_count', 'user_std']\n\nuser_answers['user_percent'] = user_answers['user_sum']/user_answers['user_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- STUDENT ANSWERS ---\n# Group by student and question part\nuser_part_performance = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby(['user_id', 'part']).\\\n                            agg({'answered_correctly': ['sum', 'mean', 'count','std']}).\\\n                            reset_index()\n\nuser_part_performance.columns = ['user_id', 'part', 'user_part_sum', 'user_part_mean', \n                        'user_part_count', 'user_part_std']\n\nuser_part_performance['user_part_percent'] = user_part_performance['user_part_sum']/user_part_performance['user_part_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- CONTENT ID ANSWERS ---\n# Group by student and bundle\nuser_bundle_performance = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby(['user_id', 'bundle_id']).\\\n                            agg({'answered_correctly': ['sum', 'mean','count', 'std']}).\\\n                            reset_index()\n\nuser_bundle_performance.columns = ['user_id', 'bundle_id' , 'user_bundle_sum', 'user_bundle_mean', \n                                     'user_bundle_count', 'user_bundle_std']\n\n\nuser_bundle_performance['userbundle_percent'] = user_bundle_performance['user_bundle_sum']/user_bundle_performance['user_bundle_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- CONTENT ID ANSWERS ---\n# Group by content and questions part\nquestion_part_performance = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby(['content_id', 'part']).\\\n                            agg({'answered_correctly': ['sum', 'mean','count', 'std']}).\\\n                            reset_index()\n\nquestion_part_performance.columns = ['content_id', 'part' , 'question_part_sum', 'question_part_mean', \n                                     'question_part_count', 'question_part_std']\n\n\nquestion_part_performance['question_part_percent'] = question_part_performance['question_part_sum']/question_part_performance['question_part_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- CONTENT ID ANSWERS ---\n# Group by content and bundle\nbundle_performance = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby(['content_id', 'bundle_id']).\\\n                            agg({'answered_correctly': ['sum', 'mean','count', 'std']}).\\\n                            reset_index()\n\nbundle_performance.columns = ['content_id', 'bundle_id' , 'bundle_sum', 'bundle_mean', \n                                     'bundle_count', 'bundle_std']\n\n\nbundle_performance['bundle_percent'] = bundle_performance['bundle_sum']/bundle_performance['bundle_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# --- CONTENT ID ANSWERS ---\n# Group by content\ncontent_answers = features_df[features_df['answered_correctly']!=-1].\\\n                            groupby('content_id').\\\n                            agg({'answered_correctly': ['sum', 'mean', 'count', 'std']}).\\\n                            reset_index()\n\ncontent_answers.columns = ['content_id', 'content_sum', 'content_mean', 'content_count', 'content_std']\n\ncontent_answers['content_percent'] = content_answers['content_sum']/content_answers['content_count']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers.to_parquet('user_answers.parquet')\nuser_part_performance.to_parquet('user_part_performance.parquet')\nuser_bundle_performance.to_parquet('user_bundle_performance.parquet')\nquestion_part_performance.to_parquet('question_part_performance.parquet')\nbundle_performance.to_parquet('bundle_performance.parquet')\ncontent_answers.to_parquet('content_answers.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train, questions, features_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# III) Preprocess"},{"metadata":{"trusted":true},"cell_type":"code","source":"# from sklearn.compose import ColumnTransformer\n\n# from cuml.experimental.preprocessing import MinMaxScaler","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We need to convert True-False variables to integers\ndef to_bool(x):\n    '''For the string variables.'''\n    if x == False:\n        return 0\n    else:\n        return 1\n\n    \ndef combine_features(data = None, add_metadata = False):\n    '''Combine the features with the Train/Test data.'''\n    \n    # Add \"past\" information\n    features_data = data.merge(user_answers, how = 'left', on = 'user_id')\n    features_data = features_data.merge(content_answers, how = 'left', on = 'content_id')\n    \n    if add_metadata==True:\n        features_data = features_data.merge(user_lectures, how = 'left', left_on = ['user_id'], right_on = ['user_id'])\n        features_data['lec_sum'].fillna(0,inplace=True)\n        features_data = features_data.merge(user_part_performance, how = 'left', left_on = ['user_id', 'part'], right_on = ['user_id', 'part'])\n        features_data = features_data.merge(user_bundle_performance, how = 'left', left_on = ['user_id', 'bundle_id'], right_on = ['user_id', 'bundle_id'])\n        features_data = features_data.merge(question_part_performance, how = 'left', left_on = ['content_id', 'part'], right_on = ['content_id', 'part'])\n        features_data = features_data.merge(bundle_performance, how = 'left', left_on = ['content_id', 'bundle_id'], right_on = ['content_id', 'bundle_id'])\n\n    # Apply\n    features_data['content_type_id'] = features_data['content_type_id'].applymap(to_bool)\n    features_data['prior_question_had_explanation'] = features_data['prior_question_had_explanation'].applymap(to_bool)\n\n    # Fill in missing spots\n    features_data.fillna(value = -1, inplace = True)\n    \n    return features_data\n\n\n# def scale_data(features_data=None, train=True, columns=None, target=None):\n#     '''Scales the provided data - if the data is for training, excludes the target column.\n#     It also chooses the features used in the prediction.'''\n    \n#     column_index = [features_data.columns.get_loc(c) for c in columns if c in features_data]\n    \n#     ct = ColumnTransformer([('MinMax', MinMaxScaler(), column_index)], remainder='passthrough')\n#     matrix = features_data.as_matrix()\n#     ct = ct.fit(matrix)\n#     scaled_matrix = ct.transform(matrix)\n#     del ct, column_index, matrix\n    \n#     scaled_data = cudf.DataFrame(scaled_matrix)\n#     del scaled_matrix\n#     scaled_data.columns = features_data.columns\n    \n#     # We don't want to scale the target also\n#     if train:\n#         scaled_data[target] = features_data[target]\n        \n#     return scaled_data\n\n\n\ndef scale_data(features_data=None, train=True, features_to_keep=None, target=None):\n    '''Scales the provided data - if the data is for training, excludes the target column.\n    It also chooses the features used in the prediction.'''\n    \n    data_for_standardization = features_data[features_to_keep]\n    matrix = data_for_standardization.as_matrix()\n    MinMax = MinMaxScaler().fit(matrix)\n    scaled_matrix = MinMax.transform(matrix)\n    del MinMax, matrix\n    \n    scaled_data = cudf.DataFrame(scaled_matrix)\n    scaled_data.columns = data_for_standardization.columns\n    del data_for_standardization\n    \n    # We don't want to scale the target also\n    if train:\n        scaled_data[target] = features_data[target]\n        \n    return scaled_data\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IV) Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"# RAPIDS roc_auc_score is 16x faster than sklearn. - cdeotte\nimport cuml\nimport cupy\nfrom cuml.metrics import roc_auc_score\nfrom cuml.preprocessing.model_selection import train_test_split\nimport xgboost\nfrom xgboost import XGBClassifier\nimport pickle\n\n\nfrom xgboost import plot_importance\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nfrom matplotlib import pyplot\nimport plotly.graph_objects as go","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_version(*x):\n    for i in x:\n        print(i, eval(f'{i}.__version__'))\n        \nprint_version('xgboost', 'cupy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_xgb_model(X_train, X_test, y_train, y_test, params, num_round=10, details = None, prints=True):\n    '''Trains an XGB and returns the trained model + ROC value.'''\n    # Create DMatrix - is optimized for both memory efficiency and training speed.\n    train_matrix = xgboost.DMatrix(data = X_train, label = y_train)\n    \n    \n    # Create & Train the model\n    model = xgboost.train(params, dtrain = train_matrix, \n                          num_boost_round=num_round\n                         )\n\n    # Make prediction\n    predicts = model.predict(xgboost.DMatrix(X_test))\n    roc = roc_auc_score(y_test.astype('int32'), predicts)\n\n    if prints:\n        print(details + \" - ROC: {:.5}\".format(roc))\n    \n    return model, roc\n\n\ndef param_tuning_graph(param_values, roc_values):\n    '''Represents visually the ROC results for the speciffic parameter tune.'''\n    \n    plt.figure(figsize=(18, 3))\n    ax = sns.barplot(x=param_values, y=roc_values, palette=custom_colors)\n\n    for p in ax.patches:\n        width = p.get_width()\n        height = p.get_height()\n        x, y = p.get_xy() \n        ax.annotate(f'{height:.5%}', (x + width/2, y + height*1.02), ha='center')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IV.1) Baseline"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nfeatures_to_keep = ['user_sum', 'user_mean', 'user_count', 'user_std', 'user_percent',\n                    'content_sum', 'content_mean', 'content_count', 'content_std', ]\n\n\n\ntarget = 'answered_correctly'\nall_features = features_to_keep.copy()\nall_features.append(target)\n\n\n\ntrain_df_combined = combine_features(data=train_df)\n\n# Comment this if you're scaling\ntrain_df_combined = train_df_combined[all_features]\n\nprint(\"Observations in train: {:,}\".format(len(train_df)))\ntrain_df_combined.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Features, target and train/test split\nX = train_df_combined[features_to_keep]\ny = train_df_combined[target]\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, \n                                                    shuffle=False, random_state=13, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, y, features_to_keep, target, all_features\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nparams1 = {\n    'max_depth' : 12,\n    'tree_method' : 'gpu_hist',\n    'objective' : 'binary:logistic',\n    'grow_policy' : 'depthwise',\n    'eval_metric': 'auc'\n}\n\n\nmodel1, roc1 = train_xgb_model(X_train, X_test, y_train, y_test, \n                               params1, details=\"baseline model\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = pyplot.subplots(figsize=(12, 8))\n\n\nmodel1.get_score(importance_type='gain')\nplot_importance(model1, ax=ax)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save model to file\npickle.dump(model1, open(\"baseline_model.pickle.dat\", \"wb\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df_combined, model1, roc1, X_train, X_test, y_train, y_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IV.2) Adding data"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Combine with past features\ntrain_df_combined = combine_features(data=train_df, add_metadata = True)\n\n# Features for ML\nfeatures_to_keep = ['timestamp', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'part', 'lec_sum',\n                    'user_sum', 'user_mean', 'user_count', 'user_std', 'user_percent',\n                    'user_part_sum', 'user_part_mean', 'user_part_count', 'user_part_std', 'user_part_percent',\n                    'user_bundle_sum', 'user_bundle_mean', 'user_bundle_count', 'user_bundle_std',\n                    'question_part_sum', 'question_part_mean', 'question_part_count', 'question_part_std', 'question_part_percent',\n                    'bundle_sum', 'bundle_mean', 'bundle_count', 'bundle_std', 'bundle_percent', \n                    'content_sum', 'content_mean', 'content_count', 'content_std', 'question_part_percent']\n\n\ntarget = 'answered_correctly'\nall_features = features_to_keep.copy()\nall_features.append(target)\n\n# Comment this if you're scaling\ntrain_df_combined = train_df_combined[all_features]\n\nprint(\"Observations in train: {:,}\".format(len(train_df)))\ntrain_df_combined.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Features, target and train/test split\nX = train_df_combined[features_to_keep]\ny = train_df_combined[target]\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, \n                                                    shuffle=False, random_state=13, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, y, features_to_keep, target, all_features\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nparams2 = {\n    'max_depth' : 12,\n    'tree_method' : 'gpu_hist',\n    'objective' : 'binary:logistic',\n    'grow_policy' : 'depthwise',\n    'eval_metric': 'auc'\n}\n\n\nmodel2, roc2 = train_xgb_model(X_train, X_test, y_train, y_test, \n                               params2, num_round=10, details=\"added data model\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = pyplot.subplots(figsize=(12, 8))\n\nmodel2.get_score(importance_type='gain')\nplot_importance(model2, ax=ax)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save model to file\npickle.dump(model2, open(\"model2.pickle.dat\", \"wb\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\netas2 = [0.001, 0.005, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1]\n\nresult_etas = {}\n\nfor eta in etas2:\n    params2 = {\n        'tree_method' : 'gpu_hist',\n        'objective' : 'binary:logistic',\n        'grow_policy' : 'depthwise',\n        'eval_metric': 'auc', \n        'eta' : eta\n    }\n\n    _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n                             params2, details = f\"ETA: {eta}\")\n    rocs2.append(roc)\n    result_etas.update({roc: eta})\n\nbest_eta = result_etas[max(rocs2)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\nmax_depths = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]\nresult_max_depths = {}\n\nfor max_depth in max_depths:\n    params2 = {\n        'max_depth' : max_depth,\n        'tree_method' : 'gpu_hist',\n        'objective' : 'binary:logistic',\n        'grow_policy' : 'depthwise',\n        'eval_metric': 'auc'\n    }\n\n    _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n                             params2, details = f\"Max_depth: {max_depth}\")\n    rocs2.append(roc)\n    result_max_depths.update({roc: max_depth})\n\nbest_max_depth = result_max_depths[max(rocs2)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\ngammas = [ 0.0, 0.2 , 0.4, 0.6, 0.8, 1, 2, 4, 6, 8, 10, 20, 50, 100, 200, 500, 1000]\nresult_gamma = {}\n\nfor gamma in gammas:\n    params2 = {\n        'tree_method' : 'gpu_hist',\n        'objective' : 'binary:logistic',\n        'grow_policy' : 'depthwise',\n        'eval_metric': 'auc',\n        'gamma': gamma\n    }\n\n    _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n                             params2, details = f\"Gamma: {gamma}\")\n    rocs2.append(roc)\n    result_gamma.update({roc: gamma})\n\nbest_gamma = result_gamma[max(rocs2)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\ncolsample_bytrees = [ 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 ]\nresult_colsample_bytrees = {}\n\nfor colsample_bytree in colsample_bytrees:\n    params2 = {\n        'tree_method' : 'gpu_hist',\n        'objective' : 'binary:logistic',\n        'grow_policy' : 'depthwise',\n        'eval_metric': 'auc',\n        'colsample_bytree': colsample_bytree\n    }\n\n    _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n                             params2, details = f\"Colsample_bytree: {colsample_bytree}\")\n    rocs2.append(roc)\n    result_colsample_bytrees.update({roc: colsample_bytree})\n\nbest_colsample_bytrees = result_colsample_bytrees[max(rocs2)]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# --- ETA ---\n# aka learning rate\n\nrocs2 = []\nalphas = [0.0, 0.2, 0.4, 0.6, 0.8, 1, 5, 10]\n\nresult_alpha={}\n\nfor alpha in alphas:\n    params2 = {\n        'tree_method' : 'gpu_hist',\n        'objective' : 'binary:logistic',\n        'grow_policy' : 'depthwise',\n        'eval_metric': 'auc', \n        'alpha' : alpha\n    }\n\n    _, roc = train_xgb_model(X_train, X_test, y_train, y_test, \n                             params2, details = f\"alpha: {alpha}\")\n    rocs2.append(roc)\n    result_alpha.update({roc: alpha})\n\nbest_alpha = result_alpha[max(rocs2)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nparams3 = {\n    'max_depth' : best_max_depth,\n    'eta' : best_eta,\n    'gamma': best_gamma,\n    'alpha': best_alpha,\n    'colsample_bytree': best_colsample_bytrees,\n    'tree_method' : 'gpu_hist',\n    'objective' : 'binary:logistic',\n    'grow_policy' : 'depthwise',\n    'eval_metric': 'auc'\n}\n\n\nmodel3, roc3 = train_xgb_model(X_train, X_test, y_train, y_test, \n                               params3, num_round=10, details=\"added data model\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = pyplot.subplots(figsize=(12, 8))\n\nmodel3.get_score(importance_type='gain')\nplot_importance(model3, ax=ax)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save model to file\npickle.dump(model3, open(\"model3.pickle.dat\", \"wb\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance = pd.DataFrame.from_dict(data=model3.get_score(importance_type='gain'), orient='index')\nfeature_importance = feature_importance.sort_values(by=0, ascending=False)\nmost_important_feature = list(feature_importance[0:12].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X_train, X_test, y_train, y_test, model3, roc3\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# IV.3) Feature selection"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df_combined['answered_correctly'] = train_df['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Features for ML\nfeatures_to_keep = most_important_feature\n\ntarget = 'answered_correctly'\n\nall_features = features_to_keep.copy()\nall_features.append(target)\n\n\nprint(\"Observations in train: {:,}\".format(len(train_df)))\ntrain_df_combined.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Features, target and train/test split\nX = train_df_combined[features_to_keep]\ny = train_df_combined[target]\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, \n                                                    shuffle=False, random_state=13, stratify=y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, y, features_to_keep, target, all_features\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nmodel4, roc4 = train_xgb_model(X_train, X_test, y_train, y_test, \n                               params3, num_round=10, details=\"added data model\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = pyplot.subplots(figsize=(12, 8))\n\nmodel4.get_score(importance_type='gain')\nplot_importance(model4, ax=ax)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save model to file\npickle.dump(model4, open(\"model4.pickle.dat\", \"wb\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df_combined, model4, roc4, X_train, X_test, y_train, y_test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submissions"},{"metadata":{"trusted":true},"cell_type":"code","source":"final_model = pickle.load(open('./model3.pickle.dat', 'rb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import library and create environment\nimport riiideducation\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Features for ML\nfeatures_to_keep = ['timestamp', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'part', 'lec_sum', \n                    'user_sum', 'user_mean', 'user_count', 'user_std', 'user_percent',\n                    'user_part_sum', 'user_part_mean', 'user_part_count', 'user_part_std', 'user_part_percent',\n                    'user_bundle_sum', 'user_bundle_mean', 'user_bundle_count', 'user_bundle_std',\n                    'question_part_sum', 'question_part_mean', 'question_part_count', 'question_part_std', 'question_part_percent',\n                    'bundle_sum', 'bundle_mean', 'bundle_count', 'bundle_std', 'bundle_percent', \n                    'content_sum', 'content_mean', 'content_count', 'content_std', 'question_part_percent']\n\n# features_to_keep = most_important_feature","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = cudf.read_parquet(\"../input/riid-competition-rapids-part-i-eda/questions.parquet\", columns=['question_id','bundle_id', 'part'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Here you would also add your pretrained model\niter_test = env.iter_test()\n\nfor (test_df, sample_prediction_df) in iter_test:\n    test_df = cudf.from_pandas(test_df)\n    \n    # --- PREPROCESSING ---\n    # Here is time to apply the preprocessing to the test_df\n    test_df = test_df.merge(questions, how = 'left', left_on = 'content_id', right_on = 'question_id')\n    test_df = combine_features(data = test_df, add_metadata = True)\n    \n    X = test_df[features_to_keep].to_pandas()\n    \n    # --- MODEL ---\n    test_df['answered_correctly'] = final_model.predict(xgboost.DMatrix(X))\n    test_df = test_df.to_pandas()\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# del questions\n# del user_answers, user_part_performance, user_bundle_performance, question_part_performance, bundle_performance, content_answers\n# del model3, roc3\n# gc.collect()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}