{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Riiid! Answer Correctness Prediction"},{"metadata":{},"cell_type":"markdown","source":"The notebook can be treated as a infrence notebook i have commented the training part here if you can find the model here - https://www.kaggle.com/ash1706/model-cat-riid.\n\nThe aim is to use LGBM and catboost and then use thier wieghted average to make the predictions.\n\nI just started with the competetion will work on this as i get more ideas.\n\n## please leave a upvote !!!!!!! 😁🙏"},{"metadata":{},"cell_type":"markdown","source":"I am using the features generated by - https://www.kaggle.com/ldevyataykina/riiid-exploratory-data-analysis-baseline
\nand thanks to this kernel for helping in loading the huge dataset it was a pain - https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets/"},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.simplefilter('ignore')\n\nimport pandas as pd\nfrom pandas.plotting import scatter_matrix\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.metrics import roc_auc_score, confusion_matrix\nfrom sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV, learning_curve\nfrom sklearn.utils import shuffle\nimport lightgbm as lgb\nfrom lightgbm import LGBMClassifier\n\nfrom catboost import CatBoostClassifier\nimport xgboost as xgb\n\n\nimport eli5\n\nimport riiideducation\nimport joblib\nimport time\n\n%matplotlib inline\n# for heatmap and other plots\ncolorMap1 = sns.color_palette(\"RdBu_r\")\n# for countplot and others plots\ncolorMap2 = 'Blues_r'\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input/riiid-test-answer-prediction'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### PATHS"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = \"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\"\nquestions_path = \"../input/riiid-test-answer-prediction/questions.csv\"\nlectures_path = \"../input/riiid-test-answer-prediction/lectures.csv\"\n\ntest = \"../input/riiid-test-answer-prediction/example_test.csv\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TRAIN"},{"metadata":{},"cell_type":"markdown","source":">**row_id**: (int64) ID code for the row.\n\n>**timestamp**: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n>**user_id**: (int32) ID code for the user.\n\n>**content_id**: (int16) ID code for the user interaction\n\n>**content_type_id**: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n>**task_container_id**: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n>**user_answer**: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n>**answered_correctly**: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n>**prior_question_elapsed_time**: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n>**prior_question_had_explanation**: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n%%time\n\ndf = pd.read_pickle(train_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Train shape: {df.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The difference between **timestamp** is not equal to **prior_question_elapsed_time**. Therefore we need to create **timestamp** difference and check it importance"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['timestamp_diff'] = df['timestamp'].diff()\ndf['timestamp_diff'].fillna(0, inplace=True)\ndf.loc[df['timestamp_diff'] < 0, 'timestamp_diff'] = np.median(df['timestamp_diff'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.memory_usage()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Droping **row_id** and **timestamp** "},{"metadata":{"trusted":true},"cell_type":"code","source":"df.drop(['row_id', 'timestamp'], axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Number of unique users: {len(np.unique(df.user_id))}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the missing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df.isnull().sum() / len(df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_answered_tasks = df['task_container_id'].value_counts().reset_index()\nfreq_answered_tasks.columns = [\n 'task_container_id', \n 'freq'\n]\n\ndf['freq_task_id'] = ''\ndf.loc[df['task_container_id'].isin(freq_answered_tasks[freq_answered_tasks['freq'] < 10000]['task_container_id'].values), 'freq_task_id'] = 'very rare answered'\ndf.loc[df['task_container_id'].isin(freq_answered_tasks[freq_answered_tasks['freq'] >= 10000]['task_container_id'].values), 'freq_task_id'] = 'rare answered'\ndf.loc[df['task_container_id'].isin(freq_answered_tasks[freq_answered_tasks['freq'] >= 50000]['task_container_id'].values), 'freq_task_id'] = 'normal answered'\ndf.loc[df['task_container_id'].isin(freq_answered_tasks[freq_answered_tasks['freq'] >= 200000]['task_container_id'].values), 'freq_task_id'] = 'often answered'\ndf.loc[df['task_container_id'].isin(freq_answered_tasks[freq_answered_tasks['freq'] >= 400000]['task_container_id'].values), 'freq_task_id'] = 'very often answered'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Change the -1 values to 0 for lectures in **user_answer** and **answered_correctly** as we saw in the columns' description"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.loc[(df['content_type_id'] == 1) & (df['user_answer'] == -1), 'user_answer'] = 0\ndf.loc[(df['content_type_id'] == 1) & (df['answered_correctly'] == -1), 'answered_correctly'] = 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look where is the most part of the incorrect answers"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.groupby(['content_type_id', 'answered_correctly']).agg({'answered_correctly': 'count'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### QUESTIONS.CSV"},{"metadata":{},"cell_type":"markdown","source":">**question_id**: foreign key for the train/test content_id column, when the content type is question (0).\n\n>**bundle_id**: code for which questions are served together.\n\n>**correct_answer**: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n>**part**: top level category code for the question.\n\n>**tags**: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv(questions_path)\nquestions.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.describe().style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(questions.isnull().sum() / len(questions))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the parts distribution"},{"metadata":{},"cell_type":"markdown","source":"### LECTURES.CSV"},{"metadata":{},"cell_type":"markdown","source":"Metadata for the lectures watched by users as they progress in their education.\n\n>**lecture_id**: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n>**part**: top level category code for the lecture.\n\n>**tag**: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n>**type_of**: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures = pd.read_csv(lectures_path)\nlectures.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures['type_of'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EXAMPLE-TEST.CSV"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_example = pd.read_csv(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_example.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"n = int(df.shape[0] * 0.1)\ntrain = df.sample(n=n, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del questions\ndel lectures\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_characteristics = df.groupby('user_id').agg({'answered_correctly':\n ['mean', 'median', 'std', 'skew', 'count']})\nuser_characteristics.columns = [\n 'mean_user_acc',\n 'median_user_acc',\n 'std_user_acc',\n 'skew_user_acc',\n 'number_of_answered_q'\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We saw earlier some dependencies between **answered_correctly** and the frequency of **task_container_id**. Therefore I want to add some features for the **task_container_id**"},{"metadata":{"trusted":true},"cell_type":"code","source":"task_container_characteristics = df.groupby('task_container_id').agg({'answered_correctly':\n ['mean', 'median', 'std', 'skew', 'count']})\ntask_container_characteristics.columns = [\n 'mean_task_acc',\n 'median_task_acc',\n 'std_task_acc',\n 'skew_task_acc',\n 'number_of_asked_task_containers'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"task_container_characteristics.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_characteristics = df.groupby('content_id').agg({'answered_correctly':\n ['mean', 'median', 'std', 'skew', 'count']})\ncontent_characteristics.columns = [\n 'mean_acc',\n 'median_acc',\n 'std_acc',\n 'skew_acc',\n 'number_of_asked_q'\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# content_characteristics.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train.copy()\ndel train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Merge all of our data"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.merge(user_characteristics, how='left', on='user_id')\ndf = df.merge(task_container_characteristics, how='left', on='task_container_id')\ndf = df.merge(content_characteristics, how='left', on='content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [\n 'prior_question_elapsed_time', \n 'prior_question_had_explanation',\n 'timestamp_diff',\n 'mean_user_acc',\n 'median_user_acc',\n 'std_user_acc',\n 'skew_user_acc',\n 'number_of_answered_q',\n 'mean_task_acc',\n 'median_task_acc',\n 'std_task_acc',\n 'skew_task_acc',\n 'number_of_asked_task_containers',\n 'mean_acc',\n 'median_acc',\n 'std_acc',\n 'skew_acc',\n 'number_of_asked_q'\n]\n\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Drop features that we are not going to use in our model"},{"metadata":{"trusted":true},"cell_type":"code","source":"col_to_drop = set(df.columns.values.tolist()).difference(features + [target])\nfor col in col_to_drop:\n del df[col]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_had_explanation'] = df['prior_question_had_explanation'].fillna(value=False).astype(bool)\ndf = df.fillna(value=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.replace([np.inf, -np.inf], np.nan)\ndf = df.fillna(0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df, test_df, y_train, y_test = train_test_split(df[features], df[target], random_state=777, test_size=0.3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The models are already trained and present in - https://www.kaggle.com/ash1706/model-cat-riid you can add it and play around . the models are basleline though nothing fancy.\nThe params are given in below cells in case you want to train it again"},{"metadata":{"trusted":true},"cell_type":"code","source":"# hyper parametre optimization\n# clf = LGBMClassifier(random_state=777)\n\n# params = {\n# 'n_estimators': [50, 150, 300],\n# 'max_depth': [3, 5, 10],\n# 'num_leaves': [5, 15, 30],\n# 'min_data_in_leaf': [5, 50, 100],\n# 'feature_fraction': [0.1, 0.5, 1.],\n# 'lambda': [0., 0.5, 1.],\n# }\n\n# cv = RandomizedSearchCV(clf, param_distributions=params, cv=5, n_iter=50, verbose=2)\n# cv.fit(df[features], df[target])\n\n# print(cv.best_params_)\n# print(cv.best_score_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It was the best params. Therefore I will use them"},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n 'num_leaves': 10, \n 'n_estimators': 100, \n 'min_data_in_leaf': 10, \n 'max_depth': 5, \n 'lambda': 0.0, \n 'feature_fraction': 1.0 , \n 'device': 'gpu',\n 'gpu_platform_id': 0,\n 'gpu_device_id': 0\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nparams_cat = {\n 'loss_function': 'Logloss',\n 'eval_metric': 'AUC',\n 'task_type': 'GPU' ,\n 'grow_policy': 'Lossguide',\n 'iterations': 2500,\n 'learning_rate': 4e-2,\n 'random_seed': 0,\n 'l2_leaf_reg': 1e-1,\n 'depth': 15,\n 'max_leaves': 10,\n 'border_count': 128,\n 'verbose': 50,\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df.prior_question_had_explanation.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# models lgbm and catboost uncomment to train\nmodel_lbgm = LGBMClassifier(**params)\n# model_lbgm.fit(train_df, y_train)\nmodel = CatBoostClassifier(**params_cat)\n# model_lbgm.fit(train_df, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.save_model(\"model.cbm\") # save model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import joblib\n# joblib.dump(model_lbgm, 'lgb.pkl')\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_model = joblib.load('../input/model-cat-riid/lgb.pkl')# load model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_model('../input/model-cat-riid/model.cbm') #load catboost\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CV"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('LGB ROC-AUC score: ', roc_auc_score(y_test.values, model.predict_proba(test_df)[:, 1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('LGB ROC-AUC score: ', roc_auc_score(y_test.values, lgb_model.predict_proba(test_df)[:, 1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('LGB ROC-AUC score: ', roc_auc_score(y_test.values, model.predict_proba(test_df)[:, 1] * 0.8 + lgb_model.predict_proba(test_df)[:, 1] * 0.2))\n# model.predict_proba(test_df)[:, 1] * 0.8 + lgb_model.predict_proba(test_df)[:, 1] * 0.2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"eli5.show_weights(model, top=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"to many useless features work on this"},{"metadata":{},"cell_type":"markdown","source":"# weight optimization"},{"metadata":{"trusted":true},"cell_type":"code","source":"weights = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\nroc_auc = []\nfor i in weights:\n roc_auc.append(roc_auc_score(y_test.values, model.predict_proba(test_df)[:, 1] * i + lgb_model.predict_proba(test_df)[:, 1] * (1-i)))\n \n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lineplot(x = weights, y = roc_auc )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"0.9 seems to be the best "},{"metadata":{},"cell_type":"markdown","source":"\n\n\n
SUBMISSION PREPARATION
\n
\nplease do leave an upvote!!!!!"},{"metadata":{},"cell_type":"markdown","source":"# to do\n1. Throw SAKT in the mix seems like the way to go\n2. Work on feature eginnering"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}