{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Credits\n\nThis code is almost entirely coming from this one : https://www.kaggle.com/its7171/lgbm-with-loop-feature-engineering\n\nPlease give credit to the original version and upvote @its7171 work.\n\n## What about this notebook?\n\n### How to simply use TabNet\n- It shows that TabNet is as easy to use as LGBM or XGBoost.\n- When using GPU and OneCycleLearningRate you can get decent results in a decent amount of time. Note that this is the worst case scenario to compare training time between boosting algorithm and TabNet as boosting algorithm get very slow with high number of classes while TabNet stays almost as fast. Also the number of features is small, this lower the advantage of using a GPU as even large batches won't fill the GPU.\n- Also this shows how to use embeddings properly as I have seen some mistakes shared on other notebooks.\n\n### How to take advantage of interpretability?\n- I think I haven't emphasize enough the power of explanability given by attention mechanism in my previous posts. I tried here to give a few hits on how explanability could be used to understand the model better.\n- Interpretability is always underestimated in Kaggle Competitions as it won't help for the final score. But in practise, it's always good to be able to give some explanations either in production (to the final user) or before getting to production (to convince the C-level board members). \n\n### Disclaimer\n- I haven't spend much time on this competition so this is a very shallow analysis but I hope it would inspire some people to push this further.\n- I guess the final score could be better with parameter tuning and more features. Not sure that TabNet will outperform LGBM here, could still be a powerful addition in a blend.\n- A new release is coming soon, with a few bugfixes and TabNetPretrainer. Stay tuned!"},{"metadata":{"trusted":true},"cell_type":"code","source":"! pip install /kaggle/input/pytorchtabnet/pytorch_tabnet-2.0.1-py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-12T18:21:01.107461Z","iopub.status.busy":"2020-10-12T18:21:01.1067Z","iopub.status.idle":"2020-10-12T18:21:02.19541Z","shell.execute_reply":"2020-10-12T18:21:02.194464Z"},"lines_to_next_cell":2,"papermill":{"duration":1.131405,"end_time":"2020-10-12T18:21:02.195556","exception":false,"start_time":"2020-10-12T18:21:01.064151","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\nfrom collections import defaultdict\nfrom tqdm.notebook import tqdm\nfrom pytorch_tabnet.tab_model import TabNetClassifier\nimport torch\n\nfrom matplotlib import pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## setting\nCV files are generated by [this notebook](https://www.kaggle.com/its7171/cv-strategy)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_pickle = '../input/riiid-cross-validation-files/cv1_train.pickle'\nvalid_pickle = '../input/riiid-cross-validation-files/cv1_valid.pickle'\nquestion_file = '../input/riiid-test-answer-prediction/questions.csv'\ndebug = False\nvalidaten_flg = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## feature engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"# funcs for user stats with loop\ndef add_user_feats(df, answered_correctly_sum_u_dict, count_u_dict):\n    acsu = np.zeros(len(df), dtype=np.int32)\n    cu = np.zeros(len(df), dtype=np.int32)\n    for cnt,row in enumerate(tqdm(df[['user_id','answered_correctly']].values)):\n        acsu[cnt] = answered_correctly_sum_u_dict[row[0]]\n        cu[cnt] = count_u_dict[row[0]]\n        answered_correctly_sum_u_dict[row[0]] += row[1]\n        count_u_dict[row[0]] += 1\n    user_feats_df = pd.DataFrame({'answered_correctly_sum_u':acsu, 'count_u':cu})\n    user_feats_df['answered_correctly_avg_u'] = user_feats_df['answered_correctly_sum_u'] / user_feats_df['count_u']\n    df = pd.concat([df, user_feats_df], axis=1)\n    return df\n\ndef add_user_feats_without_update(df, answered_correctly_sum_u_dict, count_u_dict):\n    acsu = np.zeros(len(df), dtype=np.int32)\n    cu = np.zeros(len(df), dtype=np.int32)\n    for cnt,row in enumerate(df[['user_id']].values):\n        acsu[cnt] = answered_correctly_sum_u_dict[row[0]]\n        cu[cnt] = count_u_dict[row[0]]\n    user_feats_df = pd.DataFrame({'answered_correctly_sum_u':acsu, 'count_u':cu})\n    user_feats_df['answered_correctly_avg_u'] = user_feats_df['answered_correctly_sum_u'] / user_feats_df['count_u']\n    df = pd.concat([df, user_feats_df], axis=1)\n    return df\n\ndef update_user_feats(df, answered_correctly_sum_u_dict, count_u_dict):\n    for row in df[['user_id','answered_correctly','content_type_id']].values:\n        if row[2] == 0:\n            answered_correctly_sum_u_dict[row[0]] += row[1]\n            count_u_dict[row[0]] += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read data\nfeld_needed = ['row_id', 'user_id', 'content_id', 'content_type_id', 'answered_correctly', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntrain = pd.read_pickle(train_pickle)[feld_needed]\nvalid = pd.read_pickle(valid_pickle)[feld_needed]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read data\nfeld_needed = ['row_id', 'user_id', 'content_id', 'content_type_id', 'answered_correctly', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ntrain = pd.read_pickle(train_pickle)[feld_needed]\nvalid = pd.read_pickle(valid_pickle)[feld_needed]\nif debug:\n    train = train[:1000000]\n    valid = valid[:100000]\nelse:\n    # Not using all training data as I came across memory issues\n    train = train[:10000000]\n    valid = valid[:100000]\ntrain = train.loc[train.content_type_id == False].reset_index(drop=True)\nvalid = valid.loc[valid.content_type_id == False].reset_index(drop=True)\n\n# answered correctly average for each content\ncontent_df = train[['content_id','answered_correctly']].groupby(['content_id']).agg(['mean']).reset_index()\ncontent_df.columns = ['content_id', 'answered_correctly_avg_c']\ntrain = pd.merge(train, content_df, on=['content_id'], how=\"left\")\nvalid = pd.merge(valid, content_df, on=['content_id'], how=\"left\")\n\n# user stats features with loops\nanswered_correctly_sum_u_dict = defaultdict(int)\ncount_u_dict = defaultdict(int)\ntrain = add_user_feats(train, answered_correctly_sum_u_dict, count_u_dict)\nvalid = add_user_feats(valid, answered_correctly_sum_u_dict, count_u_dict)\n\n# fill with mean value for prior_question_elapsed_time\n# note that `train.prior_question_elapsed_time.mean()` dose not work!\n# please refer https://www.kaggle.com/its7171/can-we-trust-pandas-mean for detail.\nprior_question_elapsed_time_mean = train.prior_question_elapsed_time.dropna().values.mean()\ntrain['prior_question_elapsed_time_mean'] = train.prior_question_elapsed_time.fillna(prior_question_elapsed_time_mean)\nvalid['prior_question_elapsed_time_mean'] = valid.prior_question_elapsed_time.fillna(prior_question_elapsed_time_mean)\n\n# use only last 30M training data for limited memory on kaggle env.\n#train = train[-30000000:]\n\n# part\nquestions_df = pd.read_csv(question_file)\ntrain = pd.merge(train, questions_df[['question_id', 'part']], left_on = 'content_id', right_on = 'question_id', how = 'left')\nvalid = pd.merge(valid, questions_df[['question_id', 'part']], left_on = 'content_id', right_on = 'question_id', how = 'left')\n\n# changing dtype to avoid lightgbm error\ntrain['prior_question_had_explanation'] = train.prior_question_had_explanation.fillna(False).astype('int8')\nvalid['prior_question_had_explanation'] = valid.prior_question_had_explanation.fillna(False).astype('int8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Defining features and categorical features"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"TARGET = 'answered_correctly'\nFEATS = ['answered_correctly_avg_u',\n         'answered_correctly_sum_u',\n         'count_u',\n         'answered_correctly_avg_c',\n         'part',\n         'prior_question_had_explanation',\n         'prior_question_elapsed_time']\n\n# Define categorical features\nCAT_FEATS = ['part']\n\ncat_idxs = []\ncat_dims = []\nfor cat_feat in CAT_FEATS:\n    cat_idx = FEATS.index(cat_feat)\n    cat_dim = train[cat_feat].nunique()\n    cat_idxs.append(cat_idx)\n    cat_dims.append(cat_dim)\n    \n# Label encode categorical features\nlabel_encoders = {}\nfor cat_feat in CAT_FEATS:\n    l_enc = LabelEncoder()\n    train.loc[:, cat_feat] = l_enc.fit_transform(train.loc[:, cat_feat].values.reshape(-1, 1))\n    valid.loc[:, cat_feat] = l_enc.transform(valid.loc[:, cat_feat].values.reshape(-1, 1))\n    label_encoders[cat_feat] = l_enc\n\ndro_cols = list(set(train.columns) - set(FEATS))\ny_tr = train[TARGET]\ny_va = valid[TARGET]\ntrain.drop(dro_cols, axis=1, inplace=True)\nvalid.drop(dro_cols, axis=1, inplace=True)\n_=gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, y_train = train[FEATS].values, y_tr.values\nX_valid, y_valid = valid[FEATS].values, y_va.values\n\n# TabNet does not allow Nan values\n# A better fillna method might improve scores\nX_train = np.nan_to_num(X_train, nan=-1)\nX_valid = np.nan_to_num(X_valid, nan=-1)\n\ndel train, y_tr\n_=gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## modeling"},{"metadata":{"trusted":true},"cell_type":"code","source":"BS = 2**12\n\n# Training for more epoch might improve the model performance\n# at the cost of longer training time\nMAX_EPOCH = 10\n\n# Defining TabNet model\nmodel = TabNetClassifier(n_d=32, n_a=32, n_steps=3, gamma=1.2,\n                         n_independent=2, n_shared=2,\n                         lambda_sparse=0., seed=0,\n                         clip_value=1,\n                         cat_idxs=cat_idxs,\n                         cat_dims=cat_dims,\n                         cat_emb_dim=1,\n                         mask_type='entmax',\n                         device_name='auto',\n                         optimizer_fn=torch.optim.Adam,\n                         optimizer_params=dict(lr=2e-2),\n                         scheduler_params=dict(max_lr=0.05,\n                                               steps_per_epoch=int(X_train.shape[0] / BS),\n                                               epochs=MAX_EPOCH,\n                                               #final_div_factor=100,\n                                               is_batch_level=True),\n                         scheduler_fn=torch.optim.lr_scheduler.OneCycleLR,\n                         verbose=1,)\n\nmodel.fit(X_train=X_train, y_train=y_train,\n          eval_set=[(X_valid, y_valid)],\n          eval_name=[\"valid\"],\n          eval_metric=[\"auc\"],\n          batch_size=BS,\n          virtual_batch_size=256,\n          max_epochs=MAX_EPOCH,\n          drop_last=True,\n          pin_memory=True\n         )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_preds = model.predict_proba(X_valid)[:, -1]\nprint('auc:', roc_auc_score(y_va, val_preds))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Basic Features importance\n\nThis is what you find with almost all machine learning models.\n\nIt's useful to discard features and also to understand in a very high level point of view what's important for the model.\n\nSince TabNet is selecting features on the fly, I'm not sure it's very useful to discard features though."},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_importances = model.feature_importances_\nindices = np.argsort(feat_importances)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure()\nplt.title(\"Feature importances\")\nplt.barh(range(len(feat_importances)), feat_importances[indices],\n       color=\"r\", align=\"center\")\n# If you want to define your own labels,\n# change indices to a list of labels on the following line.\nplt.yticks(range(len(feat_importances)), [FEATS[idx] for idx in indices])\nplt.ylim([-1, len(feat_importances)])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Understanding the model better"},{"metadata":{"trusted":true},"cell_type":"code","source":"LIMIT_EXPLAIN = 100000\nexplain_mat, masks = model.explain(X_valid[:LIMIT_EXPLAIN, :])\n# Normalize the importance by sample\nnormalized_explain_mat = np.divide(explain_mat, explain_mat.sum(axis=1).reshape(-1, 1))\n\n# Add prediction to better understand correlation between features and predictions\nexplain_and_preds = np.hstack([normalized_explain_mat, val_preds[:LIMIT_EXPLAIN].reshape(-1, 1)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\n\npx.imshow(explain_and_preds[:200, :],\n          labels=dict(x=\"Features\", y=\"Samples\", color=\"Importance\"),\n          x=FEATS+[\"prediction\"],\n          title=\"Sample wise feature importance (reality is more complex than global feature importance)\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"correlation_importance = np.corrcoef(explain_and_preds.T)\n\npx.imshow(correlation_importance,\n          labels=dict(x=\"Features\", y=\"Features\", color=\"Correlation\"),\n          x=FEATS+[\"prediction\"], y=FEATS+[\"prediction\"],\n          title=\"Correlation between attention mechanism for each feature and predictions\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How to read this?\n\nUnderstanding a complex Machine Learning algorithm is a hard task. Here I'm just giving some hints on what could be done to understand the results of TabNet, taking advantage of the attention mechanism.\n\n### What can we say here ?\n\nIt seems that `prior_question_had_explanation` is totally useless for the model, previous question does not seem to be relevant to predict success on the current question. It also seems that the model does not care about `answered_correctly_avg_u` this might come from the fact that using `answered_correctly_avg_u` is enough to take a decision.\n\nFor example, you can see a high negative correlation between `answered_correctly_avg_c` and `answered_correctly_avg_u`. In the meantime you see a positive correlation between `prediction` and `answered_correctly_avg_c` while a (small) negative correlation between `prediction` and `answered_correctly_avg_u`.\n\nSo maybe one way to say this in English : When the model sees an easy question (high `answered_correctly_avg_c` ) it does not need to look at how good the student is. When it needs to look at harder questions, then it matters to see if the student was doing good previously or not.\n\nThose sentences are partially false of course, but can give a general statement about what the model is looking at. This can help users trust the ML model and also select input features so that the reasonning align more with something accpetable. A more in-depth analysis would give better explanations at a question/student level.\n\n### Disclaimer : results changes at each epoch\n\nIt's actually funny to see how those results can change even for the same model at different epochs. The model changes its mind at each epoch, for example after one epoch it still looks sometimes at `prior_question_had_explanation`, but after a few epochs it stops completely and starts disregarding this feature."},{"metadata":{},"cell_type":"markdown","source":"## inference"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-12T18:42:48.029704Z","iopub.status.busy":"2020-10-12T18:42:48.028875Z","iopub.status.idle":"2020-10-12T18:42:48.032215Z","shell.execute_reply":"2020-10-12T18:42:48.031426Z"},"papermill":{"duration":0.07062,"end_time":"2020-10-12T18:42:48.032349","exception":false,"start_time":"2020-10-12T18:42:47.961729","status":"completed"},"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class Iter_Valid(object):\n    def __init__(self, df, max_user=1000):\n        df = df.reset_index(drop=True)\n        self.df = df\n        self.user_answer = df['user_answer'].astype(str).values\n        self.answered_correctly = df['answered_correctly'].astype(str).values\n        df['prior_group_responses'] = \"[]\"\n        df['prior_group_answers_correct'] = \"[]\"\n        self.sample_df = df[df['content_type_id'] == 0][['row_id']]\n        self.sample_df['answered_correctly'] = 0\n        self.len = len(df)\n        self.user_id = df.user_id.values\n        self.task_container_id = df.task_container_id.values\n        self.content_type_id = df.content_type_id.values\n        self.max_user = max_user\n        self.current = 0\n        self.pre_user_answer_list = []\n        self.pre_answered_correctly_list = []\n\n    def __iter__(self):\n        return self\n    \n    def fix_df(self, user_answer_list, answered_correctly_list, pre_start):\n        df= self.df[pre_start:self.current].copy()\n        sample_df = self.sample_df[pre_start:self.current].copy()\n        df.loc[pre_start,'prior_group_responses'] = '[' + \",\".join(self.pre_user_answer_list) + ']'\n        df.loc[pre_start,'prior_group_answers_correct'] = '[' + \",\".join(self.pre_answered_correctly_list) + ']'\n        self.pre_user_answer_list = user_answer_list\n        self.pre_answered_correctly_list = answered_correctly_list\n        return df, sample_df\n\n    def __next__(self):\n        added_user = set()\n        pre_start = self.current\n        pre_added_user = -1\n        pre_task_container_id = -1\n        pre_content_type_id = -1\n        user_answer_list = []\n        answered_correctly_list = []\n        while self.current < self.len:\n            crr_user_id = self.user_id[self.current]\n            crr_task_container_id = self.task_container_id[self.current]\n            crr_content_type_id = self.content_type_id[self.current]\n            if crr_user_id in added_user and (crr_user_id != pre_added_user or (crr_task_container_id != pre_task_container_id and crr_content_type_id == 0 and pre_content_type_id == 0)):\n                # known user(not prev user or (differnt task container and both question))\n                return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n            if len(added_user) == self.max_user:\n                if  crr_user_id == pre_added_user and (crr_task_container_id == pre_task_container_id or crr_content_type_id == 1):\n                    user_answer_list.append(self.user_answer[self.current])\n                    answered_correctly_list.append(self.answered_correctly[self.current])\n                    self.current += 1\n                    continue\n                else:\n                    return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n            added_user.add(crr_user_id)\n            pre_added_user = crr_user_id\n            pre_task_container_id = crr_task_container_id\n            pre_content_type_id = crr_content_type_id\n            user_answer_list.append(self.user_answer[self.current])\n            answered_correctly_list.append(self.answered_correctly[self.current])\n            self.current += 1\n        if pre_start < self.current:\n            return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n        else:\n            raise StopIteration()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# You can debug your inference code to reduce \"Submission Scoring Error\" with `validaten_flg = True`.\n# Please refer https://www.kaggle.com/its7171/time-series-api-iter-test-emulator about Time-series API (iter_test) Emulator.\n\nif validaten_flg:\n    target_df = pd.read_pickle(valid_pickle)\n    if debug:\n        target_df = target_df[:10000]\n    iter_test = Iter_Valid(target_df,max_user=1000)\n    predicted = []\n    def set_predict(df):\n        predicted.append(df)\n    # reset answered_correctly_sum_u_dict and count_u_dict\n    answered_correctly_sum_u_dict = defaultdict(int)\n    count_u_dict = defaultdict(int)\n    train = pd.read_pickle(train_pickle)[['user_id','answered_correctly','content_type_id']]\n    if debug:\n        train = train[:1000000]\n    train = train[train.content_type_id == False].reset_index(drop=True)\n    update_user_feats(train, answered_correctly_sum_u_dict, count_u_dict)\n    del train\nelse:\n    import riiideducation\n    env = riiideducation.make_env()\n    iter_test = env.iter_test()\n    set_predict = env.predict","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-10-12T18:42:48.174757Z","iopub.status.busy":"2020-10-12T18:42:48.173901Z","iopub.status.idle":"2020-10-12T18:42:49.112276Z","shell.execute_reply":"2020-10-12T18:42:49.111472Z"},"papermill":{"duration":1.016814,"end_time":"2020-10-12T18:42:49.112404","exception":false,"start_time":"2020-10-12T18:42:48.09559","status":"completed"},"scrolled":true,"tags":[],"trusted":true},"cell_type":"code","source":"previous_test_df = None\nfor (test_df, sample_prediction_df) in iter_test:\n    if previous_test_df is not None:\n        previous_test_df[TARGET] = eval(test_df[\"prior_group_answers_correct\"].iloc[0])\n        update_user_feats(previous_test_df, answered_correctly_sum_u_dict, count_u_dict)\n    previous_test_df = test_df.copy()\n    test_df = test_df[test_df['content_type_id'] == 0].reset_index(drop=True)\n    test_df = add_user_feats_without_update(test_df, answered_correctly_sum_u_dict, count_u_dict)\n    test_df = pd.merge(test_df, content_df, on='content_id',  how=\"left\")\n    test_df = pd.merge(test_df, questions_df, left_on='content_id', right_on='question_id', how='left')\n    test_df['prior_question_had_explanation'] = test_df.prior_question_had_explanation.fillna(False).astype('int8')\n    test_df['prior_question_elapsed_time_mean'] = test_df.prior_question_elapsed_time.fillna(prior_question_elapsed_time_mean)\n    \n    for cat_feat in CAT_FEATS:\n        l_enc = label_encoders[cat_feat]\n        test_df.loc[:, cat_feat] = l_enc.fit_transform(test_df.loc[:, cat_feat].values.reshape(-1, 1))\n    \n    X_test = np.nan_to_num(test_df[FEATS].values, nan=-1)\n    test_df[TARGET] =  model.predict_proba(X_test)[:, -1]\n    set_predict(test_df[['row_id', TARGET]])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if validaten_flg:\n    y_true = target_df[target_df.content_type_id == 0].answered_correctly\n    y_pred = pd.concat(predicted).answered_correctly\n    print(roc_auc_score(y_true, y_pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### You've reach the end! Congrats!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}