{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Brief description"},{"metadata":{},"cell_type":"markdown","source":"Our approach is based on single LGBM model. This model gives 79.9 auc on validation, 79.7 on public, 79.9 on private leaderboard. \n\nWe tried to stack model with XGBoost trained on the same features, but it gives us no significant boost. Unfortunately, we didn' t succeed in building strong SAKT or SAINT+ model :(\n\nThis notebook contains only training part of our solution. We upload the inference part later."},{"metadata":{},"cell_type":"markdown","source":"In the description we use the term \"accuracy\" instead of \"answered correctly\". \n\nFeatures that we use in the model can be divided into groups: \n\n\n* *Raw features:*\n\n    1) prior_question_elapsed_time;\n    \n    2) prior_question_had_explanation.\n\n\n* *Question statistics without update*:\n\n    1) count/average/std of accuracy for the current question;\n    \n    2) average/std of elapsed time;\n    \n    3) average/std of \"had_explanation\" flag;\n    \n    4) 15 question \"embeddings\", obtained through ALS factorization of (user_id - content_id) matrix.\n   \n    To get \"elapsed time\" and \"had explanation\" for the current question, we shifted \"prior_question_elapsed_time\" and \"prior_question_had_explanation\" fields by one position back grouping by each user_id.\n    \n    For ALS factorization, we used only the train part of the data to prevent overfitting.\n    \n    \n* *User statistics with update:*\n\n    All statistics are calculated for all user history between beginning (*timestamp = 0*) and current moment.\n\n    1) count/avg of accuracy for the current user;\n    \n    2) avg of *(accuracy - question_accuracy_mean)*;\n    \n    3) avg of *(1 - accuracy) * question_accuracy_mean*;\n    \n    4) avg of \"prior_question_elapsed_time\" and \"prior_question had_explanation\";\n    \n    The main idea of 2 - 3 is to estimate user abilities, taking into consideration not only the rate of correct answers, but also the difficulty of questions he succeeded/not succeeded to answer correctly.\n    \n\n* *Timedelta features:*\n\n    1) Time since previous question for the current user;\n    \n    2) Time since previous question with correct/incorrect answer;\n    \n    3) Time since 2 previous questions/3 previous questions.\n    \n    \n* *User-session statistics with update:*\n\n    The idea is to calculate user statistics only for last user session. To start a new session for each user, we waited for the timedelta between current question and previous question exceed fixed period of time (in final solution we use hour and week as periods).\n\n    1) count/avg of accuracy, avg of *(accuracy - question_accuracy_mean)* for user within 1 hour session;\n    \n    2) count/avg of accuracy, avg of *(accuracy - question_accuracy_mean)* for user within 1 week session;\n    \n    \n* *User-window statistics with update:*\n    \n    The idea is the same as in the previous item, but instead of sessions here we used simple rolling windows:\n    \n    1) sum of *(accuracy - question_accuracy_mean)* for user within window with depth 5/10/15\n    \n    \n* *User-part statistics with update:*\n\n    The statistics here are the same as user statistics, but instead of grouping by \"user_id\" field, we grouped by \"user_id\" and \"part\" fields.\n    \n    \n* *User-als statistics with update:*\n\n    For this group of features, we calculated vectors of each question through ALS-factorization of (user_id - content_id) matrix (the same way as in the first item, but instead of dim=15 we used dim=500). Then we clusterized these vectors with K-means and got cluster for each content_id of question (\"als_cluster\" field). Finally we calculated some statistics grouping by \"user_id\" and \"als_cluster\" fields:\n    \n    1) count/avg of *(accuracy - question_accuracy_mean)*;\n    \n    2) time since previous question for the \"user_id\" - \"als_cluster\" aggregation.\n    \n    \n* *User-question statistics with update:*\n    \n    The statistics here are the same as in the previous item, but instead of grouping by \"user_id\" and \"als_cluster\" fields, we simply grouped by \"user_id\" and \"content_id\" fields. This helps a lot in the situations, when user is proposed to answer the question he answered earlier.\n\n    \n* *Other features:*\n\n    1) avg of accuracy for the current part;\n    \n    2) avg of accuracy for the current als cluster."},{"metadata":{},"cell_type":"markdown","source":"Thanks to organizers for this interesting competition and thanks to [@tito](https://www.kaggle.com/its7171) for his brilliant notebooks with baseline solution and validation strategy!"},{"metadata":{"lines_to_next_cell":2,"papermill":{"duration":1.131405,"end_time":"2020-10-12T18:21:02.195556","exception":false,"start_time":"2020-10-12T18:21:01.064151","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport gc\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom tqdm import tqdm\nimport lightgbm as lgb\nimport pickle\nimport sys\nimport importlib\nimport matplotlib.pyplot as plt\nfrom collections import defaultdict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Functions"},{"metadata":{},"cell_type":"markdown","source":"Here we declare some basic functions for generating features with updates."},{"metadata":{"trusted":true},"cell_type":"code","source":"def calc_arrays(df, dict_agg_user,\n                dict_agg_user_part,\n                dict_agg_user_als,\n                dict_agg_uq,\n                df_type='train',\n                max_td=6048000, \n                windows_list=[5, 10, 15]):\n    \n    if df_type == 'train':\n        shape = df[df.segment == 1].shape[0]\n    else:\n        shape = df.shape[0]\n        \n    ## const\n    HOUR = 3600*10\n    WEEK = 24 * 7 * HOUR\n    USER_DIM = max(windows_list)\n    \n    ## user\n    user_array = np.zeros((shape, 11), dtype='int32')\n    user_sess_array = np.zeros((shape, 6), dtype='int32')\n    user_window_array = np.zeros((shape, len(windows_list)), dtype='int16')\n    \n    ## user-part/als/question\n    user_part_array = np.zeros((shape, 6), dtype='int32')\n    user_als_array = np.zeros((shape, 3), dtype='int32')\n    uq_array = np.zeros((shape, 3), dtype='int32')\n    \n    i = -1\n    for row in tqdm(df[['user_id', 'answered_correctly', 'timestamp', 'content_id', \n                        'part', 'als_cluster', 'question_acc_mean', \n                        'prior_question_elapsed_time', \n                        'prior_question_had_explanation', 'segment']].values):\n        user_id = int(row[0])\n        answered_correctly = int(row[1])\n        timestamp = int(row[2])\n        content_id = int(row[3])\n        part = row[4]\n        als_cluster = row[5]\n        q_level = row[6]\n        elaps_time = row[7]\n        had_expl = row[8]\n        segment = row[9]\n        \n        acc_diff_q = int(answered_correctly * 1000 - q_level * 1000)\n        neg_acc_mul_q = int((1 - answered_correctly) * q_level * 1000)\n        \n        if (df_type == 'train') and (segment == 1):\n            i += 1\n        elif df_type == 'valid':\n            i += 1\n        \n        ## user_id\n        user_row = list(dict_agg_user[user_id])\n        \n        # window features\n        if (df_type == 'valid') or (segment == 1):\n            for k, j in enumerate(windows_list):\n                user_window_array[i, k] = user_row[7] - user_row[16+j] \n        \n        for j in range(USER_DIM - 1, 0, -1):\n            user_row[17 + j] = user_row[16 + j]\n        user_row[17] = user_row[7]\n        \n        # main features\n        user_row[9] += elaps_time\n        user_row[10] += had_expl\n        timedelta = timestamp - user_row[1]\n        timedelta = timedelta if timedelta <= max_td else max_td\n        timedelta_neg = timestamp - user_row[2] if user_row[2] >= 0 else 0\n        timedelta_neg = timedelta_neg if timedelta_neg <= max_td else max_td\n        timedelta_pos = timestamp - user_row[3] if user_row[3] >= 0 else 0\n        timedelta_pos = timedelta_pos if timedelta_pos <= max_td else max_td\n        timedelta_2 = timestamp - user_row[4]\n        timedelta_2 = timedelta_2 if timedelta_2 <= max_td else max_td\n        timedelta_3 = timestamp - user_row[5]\n        timedelta_3 = timedelta_3 if timedelta_3 <= max_td else max_td\n        \n        if (df_type == 'valid') or (segment == 1):\n            user_array[i, 0] = user_row[0]\n            user_array[i, 1] = timedelta\n            user_array[i, 2] = timedelta_neg\n            user_array[i, 3] = timedelta_pos\n            user_array[i, 4] = timedelta_2\n            user_array[i, 5] = timedelta_3\n            for j in range(5):\n                user_array[i, j+6] = user_row[j+6]\n        \n        user_row[0] += 1\n        user_row[5] = user_row[4]\n        user_row[4] = user_row[1]\n        user_row[1] = timestamp\n        user_row[2] = timestamp if answered_correctly == 0 else user_row[2]\n        user_row[3] = timestamp if answered_correctly == 1 else user_row[3]\n        user_row[6] += int(answered_correctly)\n        user_row[7] += acc_diff_q\n        user_row[8] += neg_acc_mul_q\n        \n        # session features\n        if timedelta >= HOUR:\n            user_row[11] = 0\n            user_row[13] = 0\n            user_row[15] = 0\n        if timedelta >= WEEK:\n            user_row[12] = 0\n            user_row[14] = 0 \n            user_row[16] = 0\n            \n        if (df_type == 'valid') or (segment == 1):\n            for j in range(6):\n                user_sess_array[i, j] = user_row[11+j]\n         \n        for j in range(2):\n            user_row[11+j] += 1\n            user_row[13+j] += int(answered_correctly)\n            user_row[15+j] += acc_diff_q \n        \n        dict_agg_user[user_id] = tuple(user_row)\n        \n        ## user-part\n        user_id_part = int(user_id*10 + part)\n        user_part_row = list(dict_agg_user_part[user_id_part])\n        user_part_row[4] += elaps_time\n        user_part_row[5] += had_expl\n        \n        if (df_type == 'valid') or (segment == 1):\n            user_part_array[i, 0] = user_part_row[0]\n            for j in range(4):\n                user_part_array[i, j+1] = user_part_row[j+1]\n        \n        user_part_row[0] += 1\n        user_part_row[1] += int(answered_correctly)\n        user_part_row[2] += acc_diff_q\n        user_part_row[3] += neg_acc_mul_q\n        dict_agg_user_part[user_id_part] = tuple(user_part_row)\n        \n        ## user-als\n        user_id_als = int(user_id*1000 + als_cluster)\n        user_als_row = list(dict_agg_user_als[user_id_als])\n        timedelta = timestamp - user_als_row[1]\n        timedelta = timedelta if timedelta <= max_td else max_td\n        if (df_type == 'valid') or (segment == 1):\n            user_als_array[i, 0] = user_als_row[0]\n            user_als_array[i, 1] = timedelta\n            user_als_array[i, 2] = user_als_row[2]\n            \n        user_als_row[0] += 1\n        user_als_row[1] = timestamp\n        user_als_row[2] += acc_diff_q\n        dict_agg_user_als[user_id_als] = tuple(user_als_row)\n        \n        ## user-question\n        user_content_id = int(user_id*100000 + content_id)\n        uq_row = list(dict_agg_uq[user_content_id])\n        timedelta = timestamp - uq_row[1]\n        timedelta = timedelta if timedelta <= max_td else max_td\n        if (df_type == 'valid') or (segment == 1):\n            uq_array[i, 0] = uq_row[0]\n            uq_array[i, 1] = timedelta\n            uq_array[i, 2] = uq_row[2]\n            \n        uq_row[0] += 1\n        uq_row[1] = timestamp\n        uq_row[2] += acc_diff_q\n        dict_agg_uq[user_content_id] = tuple(uq_row)\n        \n    return user_array, user_sess_array, user_window_array, user_part_array, user_als_array, uq_array","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_user_features(df_, user_array, user_sess_array, \n                      user_window_array, user_part_array, \n                      user_als_array, uq_array, \n                      windows_list=[5, 10, 20]):    \n    target_names = ['acc', 'acc_diff_q_level', 'neg_acc_mul_q_level']\n    \n    # user\n    df_['user_id_cumcount'] = user_array[:, 0]\n    df_['user_id_time_since_last_ans'] = user_array[:, 1]\n    df_['user_id_time_since_last_incorr_ans'] = user_array[:, 2]\n    df_['user_id_time_since_last_corr_ans'] = user_array[:, 3]\n    df_['user_id_time_since_last_ans_2'] = user_array[:, 4]\n    df_['user_id_time_since_last_ans_3'] = user_array[:, 5]\n    df_['user_id_elapsed_time_mean'] = (user_array[:, 9]/(1+df_['user_id_cumcount'])).astype('float32')\n    df_['user_id_had_expl_mean'] = (user_array[:, 10]/(1+df_['user_id_cumcount'])).astype('float32')\n    for j, name in enumerate(target_names):\n        df_['user_id_' + name + '_mean'] = (user_array[:, j+6]/df_['user_id_cumcount']).astype('float32')\n    \n    # user-window\n    for k, j in enumerate(windows_list):\n        df_['user_id_acc_diff_q_level_window_'+str(j)] = user_window_array[:, k]\n    \n    # user-session\n    for j, p in enumerate(['hour', 'week']): \n        df_['user_session_cumcount_'+p] = user_sess_array[:, j]\n        df_['user_session_acc_mean_'+p] = (user_sess_array[:, j+2]/df_['user_session_cumcount_'+p]).astype('float32')\n        df_['user_session_acc_diff_q_level_mean_'+p] = (user_sess_array[:, j+4]/df_['user_session_cumcount_'+p]).astype('float32')\n    \n    # user-part\n    df_['user_id_part_cumcount'] = user_part_array[:, 0]\n    df_['user_id_part_acc_mean'] = user_part_array[:, 1]\n    df_['user_id_part_elapsed_time_mean'] = (user_part_array[:, 4]/(1+df_['user_id_part_cumcount'])).astype('float32')\n    df_['user_id_part_had_expl_mean'] = (user_part_array[:, 5]/(1+df_['user_id_part_cumcount'])).astype('float32')\n    for j, name in enumerate(target_names[1:]):\n        df_['user_id_part_' + name + '_mean'] = (user_part_array[:, j+2]/df_['user_id_part_cumcount']).astype('float32')\n    \n    # user-als\n    df_['user_id_als_cumcount'] = user_als_array[:, 0]\n    df_['user_id_als_timedelta'] = user_als_array[:, 1]\n    df_['user_id_als_acc_diff_q_level_mean'] = (user_als_array[:, 2]/df_['user_id_als_cumcount']).astype('float32')\n    \n    # user-content\n    df_['user_content_id_cumcount'] = uq_array[:, 0]\n    df_['user_content_id_timedelta'] = uq_array[:, 1]\n    df_['user_content_id_acc_diff_q_level_mean'] = (uq_array[:, 2]/df_['user_content_id_cumcount']).astype('float32')\n    return df_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read & preprocess data"},{"metadata":{},"cell_type":"markdown","source":"Here we use train and valid dataframes, generated by this notebook by [@tito](https://www.kaggle.com/its7171): https://www.kaggle.com/its7171/cv-strategy.\n\nNote that the notebook requires significant amount of memory (we had 64 GB of RAM on our machine), so it is impossible to reproduce it in the Kaggle environment (to run all cells, you can set \"IS_KAGGLE\" param to True). "},{"metadata":{"trusted":true},"cell_type":"code","source":"## paths\ninput_path = '../input/riiid-test-answer-prediction'\ncv_path = '../input/riiid-cross-validation-files'\nexternal_path = '../input/external-data'\n\ntrain_pickle = f'{cv_path}/cv1_train.pickle'\nvalid_pickle = f'{cv_path}/cv1_valid.pickle'\n\nquestion_file = f'{input_path}/questions.csv'\n\nembeddings_file = f'{external_path}/df_content_emb.csv'\nals_clusters_file = f'{external_path}/als_clusters.csv'\nfeld_needed = ['row_id', 'user_id', 'content_id', 'content_type_id', 'answered_correctly', \n               'prior_question_elapsed_time', 'prior_question_had_explanation', 'timestamp']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## params\n\n# Kaggle environment\nIS_KAGGLE = True\n\n# dimension of question \"embeddings\"\nDIM = 15\n\n# maximum value of timedelta for clipping and threshold to determine outdated \"user-question\" pairs\nMAX_TD = 6048000\n\n# subsample of train data, actually used for training\nTRAIN_FRAC = 0.75\n\n# rolling windows for user statistics\nWINDOWS_LIST = [5, 10, 15]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_pickle(train_pickle)[feld_needed]\ndf_valid = pd.read_pickle(valid_pickle)[feld_needed]\n\nif IS_KAGGLE:\n    df_train = df_train.iloc[:1000000].reset_index(drop=True)\n\n## divide timestamp by 100 to save memory for dictionaries\ndf_train['timestamp'] = (df_train.timestamp/1e2).astype('int64')\ndf_valid['timestamp'] = (df_valid.timestamp/1e2).astype('int64')\n\ndf_train = df_train.loc[df_train.content_type_id == False].reset_index(drop=True)\ndf_valid = df_valid.loc[df_valid.content_type_id == False].reset_index(drop=True)\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['prior_question_had_explanation'] = df_train['prior_question_had_explanation'].fillna(False).astype('int8')\ndf_valid['prior_question_had_explanation'] = df_valid['prior_question_had_explanation'].fillna(False).astype('int8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Due to memory issues, we couldn't train our model on full *df_train*. So, we decided to train it only on 75% of the data (segment == 1):  "},{"metadata":{"trusted":true},"cell_type":"code","source":"## segments\ntrain_ids = df_train[['row_id']].sample(frac=TRAIN_FRAC, random_state=0).row_id.unique()\ndf_train['segment'] = np.where(df_train.row_id.isin(train_ids), 1, 0).astype('int8')\ndf_valid['segment'] = np.ones(df_valid.shape[0], dtype='int8')\ndf_train.segment.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate question features"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['elapsed_time'] = df_train.groupby('user_id').prior_question_elapsed_time.shift(periods=-1)\ndf_train['had_explanation'] = df_train.groupby('user_id').prior_question_had_explanation.shift(periods=-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Adding main question statistics:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_question_mean = (\n    df_train[['content_id', 'answered_correctly', \n              'elapsed_time', 'had_explanation']]\n    .groupby(['content_id'])\n    .agg({'answered_correctly': ['count', 'mean', 'std'], \n          'elapsed_time': ['mean', 'std'], \n          'had_explanation': ['mean', 'std']})\n    .reset_index()\n)\ndf_train.drop(columns=['elapsed_time', 'had_explanation'], inplace=True)\n\ncolumns = ['content_id']\ncolumns += ['question_acc_count', 'question_acc_mean', 'question_acc_std']\ncolumns += ['question_elapsed_time_mean', 'question_elapsed_time_std']\ncolumns += ['question_had_expl_mean', 'question_had_expl_std']\ndf_question_mean.columns = columns\nfor col in columns:\n    df_question_mean[col] = df_question_mean[col].astype('float32')\n    \ndf_train = pd.merge(df_train, df_question_mean, on='content_id', how=\"left\")\ndf_valid = pd.merge(df_valid, df_question_mean, on='content_id', how=\"left\")\ndf_train.fillna(0, inplace=True)\ndf_valid.fillna(0, inplace=True)\n\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Adding question \"embeddings\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_emb = pd.read_csv(embeddings_file).rename(columns={'dim_'+str(i): 'content_id_dim_'+str(i) for i in range(DIM)})\ndf_emb['content_id'] = df_emb.content_id.astype('int32')\nfor f in list(df_emb.columns)[1:]:\n    df_emb[f] = df_emb[f].astype('float16')\nprint(df_emb.shape)\n\ndf_train = pd.merge(df_train, df_emb, on='content_id', how=\"left\")\ndf_valid = pd.merge(df_valid, df_emb, on='content_id', how=\"left\")\n\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Adding \"part\" and \"als_cluster\" fields:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_part = pd.read_csv(question_file)[['question_id', 'part']]\ndf_part.columns = ['content_id', 'part']\ndf_train = pd.merge(df_train, df_part, on = 'content_id', how = 'left')\ndf_valid = pd.merge(df_valid, df_part, on = 'content_id', how = 'left')\n\ndf_als = pd.read_csv(als_clusters_file)[['question_id', 'cluster']]\ndf_als.columns = ['content_id', 'als_cluster']\ndf_train = pd.merge(df_train, df_als, on = 'content_id', how = 'left')\ndf_valid = pd.merge(df_valid, df_als, on = 'content_id', how = 'left')\n\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Adding \"part\" and \"als_cluster\" statistics:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_part_mean = df_train[['part', 'answered_correctly']].groupby(['part']).agg(['mean']).reset_index()\ndf_part_mean.columns = ['part', 'part_acc_mean']\ndf_train = pd.merge(df_train, df_part_mean, on='part', how=\"left\")\ndf_valid = pd.merge(df_valid, df_part_mean, on='part', how=\"left\")\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_als_mean = df_train[['als_cluster', 'answered_correctly']].groupby(['als_cluster']).agg(['mean']).reset_index()\ndf_als_mean.columns = ['als_cluster', 'als_acc_mean']\ndf_train = pd.merge(df_train, df_als_mean, on='als_cluster', how=\"left\")\ndf_valid = pd.merge(df_valid, df_als_mean, on='als_cluster', how=\"left\")\ndf_train.shape, df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Saving all necessary files (doesn't make sence on Kaggle environment):"},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    df_part_als = df_part.merge(df_als, on='content_id')\n    df_part_als.to_csv(f'{external_path}/df_part_als.csv', index=False)\n    df_question_mean.to_csv(f'{external_path}/df_question_mean.csv', index=False)\n    df_part_mean.to_csv(f'{external_path}/df_part_mean.csv', index=False)\n    df_als_mean.to_csv(f'{external_path}/df_als_mean.csv', index=False)\n\n    del df_part_als, df_question_mean, df_part_mean, df_als_mean\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.memory_usage(deep=True).sum()/2**30","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate user features"},{"metadata":{},"cell_type":"markdown","source":"### 1) train"},{"metadata":{},"cell_type":"markdown","source":"To make update features on train/valid/inference, we use simple defaultdicts from \"collections\" library:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dict_agg_user = defaultdict(lambda: tuple([0, 0, -1, -1] + [0]*(13+max(WINDOWS_LIST))))\ndict_agg_user_part = defaultdict(lambda: (0, 0, 0, 0, 0, 0))\ndict_agg_user_als = defaultdict(lambda: (0, 0, 0))\ndict_agg_uq = defaultdict(lambda: (0, 0, 0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calculating arrays with features for train data with *segment == 1* and updating dictionaries:"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nuser_array, user_sess_array, \\\nuser_window_array, user_part_array, \\\nuser_als_array, uq_array = calc_arrays(df_train, \n                                       dict_agg_user,\n                                       dict_agg_user_part,\n                                       dict_agg_user_als,\n                                       dict_agg_uq, \n                                       df_type='train', \n                                       windows_list=WINDOWS_LIST)\n\ndel dict_agg_uq; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_ = df_train[df_train.segment == 1].reset_index(drop=True)\ndf_train_.memory_usage(deep=True).sum()/2**30","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One of the dictionaries (dict_agg_uq) is used to create \"user-question\" statistics. This dictionary itself is very large, so we remove it and create new one, without outdated \"user-question\" pairs. \n\nWe consider \"user-question\" pair outdated, if the following condition is met:\n\n    max(user_timestamp) - user_timestamp >= MAX_TD"},{"metadata":{"trusted":true},"cell_type":"code","source":"## correct user-content dict\ndf_train = df_train[['user_id', 'content_id', 'timestamp', 'answered_correctly', 'question_acc_mean']].reset_index(drop=True)\ndf_train['max_user_tmp'] = df_train.groupby('user_id').timestamp.transform('max')\ndf_train = df_train.query(f\"max_user_tmp - timestamp  < {MAX_TD}\").drop(columns=['max_user_tmp']).reset_index(drop=True)\ndf_train['user_content_id'] = (df_train.user_id*100000 + df_train.content_id).astype('int64')\ndf_train['acc_diff_q_level'] = (1000 * df_train.answered_correctly - 1000 * df_train.question_acc_mean).astype('int32')\n        \ndf_agg_uq = df_train.groupby('user_content_id').agg({'user_id': 'count', 'timestamp': 'max', 'acc_diff_q_level': 'sum'})\ndel df_train; gc.collect()\ndf_agg_uq.columns = ['count', 'timestamp', 'acc_diff_q_level_mean'] \ndf_agg_uq['acc_diff_q_level_mean'] /= df_agg_uq['count']\ndict_agg_uq = defaultdict(lambda: (0, 0, 0), df_agg_uq.apply(tuple, axis=1).to_dict())\ndel df_agg_uq; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, generating features for train data with *segment == 1*:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_ = add_user_features(df_train_, user_array, user_sess_array, \n                              user_window_array, user_part_array, \n                              user_als_array, uq_array, \n                              windows_list=WINDOWS_LIST)\n\ndel user_array, user_sess_array, user_window_array, user_part_array, user_als_array, uq_array; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Saving dictionaries for debug (doesn't make sence on Kaggle environment):"},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    ## user dict\n    pickle.dump(dict(dict_agg_user), open(f\"{external_path}/user_dict_debug.pickle.dat\", \"wb\"))\n    print(\"User dict saved!\", len(dict_agg_user))\n\n    ## user part dict\n    pickle.dump(dict(dict_agg_user_part), open(f\"{external_path}/user_part_dict_debug.pickle.dat\", \"wb\"))\n    print(\"User part dict saved!\", len(dict_agg_user_part))\n\n    ## user als dict\n    pickle.dump(dict(dict_agg_user_als), open(f\"{external_path}/user_als_dict_debug.pickle.dat\", \"wb\"))\n    print(\"User als dict saved!\", len(dict_agg_user_als))\n\n    ## user-question dict\n    pickle.dump(dict(dict_agg_uq), open(f\"{external_path}/uq_dict_debug.pickle.dat\", \"wb\"))\n    print(\"User-questions dict saved!\", len(dict_agg_uq))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2) valid"},{"metadata":{},"cell_type":"markdown","source":"Calculating arrays with features for valid data and updating dictionaries:"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nuser_array, user_sess_array, \\\nuser_window_array, user_part_array, \\\nuser_als_array, uq_array = calc_arrays(df_valid, \n                                       dict_agg_user,\n                                       dict_agg_user_part,\n                                       dict_agg_user_als,\n                                       dict_agg_uq, \n                                       df_type='valid', \n                                       windows_list=WINDOWS_LIST)\ndf_valid.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generating features for valid data:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_valid = add_user_features(df_valid, user_array, user_sess_array, \n                             user_window_array, user_part_array, \n                             user_als_array, uq_array, \n                             windows_list=WINDOWS_LIST)\n\ndel user_array, user_sess_array, user_window_array, user_part_array, user_als_array, uq_array; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Saving final dictionaries (doesn't make sence on Kaggle environment):"},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    ## user dict\n    pickle.dump(dict(dict_agg_user), open(f\"{external_path}/user_dict.pickle.dat\", \"wb\"))\n    print(\"User dict saved!\", len(dict_agg_user))\n\n    ## user part dict\n    pickle.dump(dict(dict_agg_user_part), open(f\"{external_path}/user_part_dict.pickle.dat\", \"wb\"))\n    print(\"User dict saved!\", len(dict_agg_user_part))\n\n    ## user als dict\n    pickle.dump(dict(dict_agg_user_als), open(f\"{external_path}/user_als_dict.pickle.dat\", \"wb\"))\n    print(\"User dict saved!\", len(dict_agg_user_als))\n\n    ## user-question dict\n    pickle.dump(dict(dict_agg_uq), open(f\"{external_path}/uq_dict.pickle.dat\", \"wb\"))\n    print(\"User-questions dict saved!\", len(dict_agg_uq))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del dict_agg_uq, dict_agg_user, dict_agg_user_part, dict_agg_user_als\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling"},{"metadata":{},"cell_type":"markdown","source":"List of used features:"},{"metadata":{"trusted":true},"cell_type":"code","source":"selected_features = [\n ## raw features\n 'prior_question_elapsed_time',\n 'prior_question_had_explanation',\n \n ## question statistics\n 'question_acc_count',\n 'question_acc_mean',\n 'question_acc_std',  \n 'question_elapsed_time_mean', \n 'question_elapsed_time_std',\n 'question_had_expl_mean', \n 'question_had_expl_std',\n\n ## user statistics\n 'user_id_cumcount',  \n 'user_id_acc_mean',\n 'user_id_acc_diff_q_level_mean',\n 'user_id_neg_acc_mul_q_level_mean',\n 'user_id_elapsed_time_mean',\n 'user_id_had_expl_mean',\n    \n ## timedelta features\n 'user_id_time_since_last_ans',\n 'user_id_time_since_last_corr_ans',\n 'user_id_time_since_last_incorr_ans',\n 'user_id_time_since_last_ans_2',\n 'user_id_time_since_last_ans_3', \n\n ## user-session statistics\n 'user_session_cumcount_hour',\n 'user_session_acc_mean_hour',\n 'user_session_acc_diff_q_level_mean_hour',   \n 'user_session_cumcount_week',\n 'user_session_acc_mean_week',\n 'user_session_acc_diff_q_level_mean_week',\n  \n ## user-part statistics\n 'user_id_part_cumcount',\n 'user_id_part_acc_mean',\n 'user_id_part_acc_diff_q_level_mean',\n 'user_id_part_neg_acc_mul_q_level_mean',\n 'user_id_part_elapsed_time_mean',\n\n ## user-als statistics\n 'user_id_als_cumcount',\n 'user_id_als_timedelta', \n 'user_id_als_acc_diff_q_level_mean', \n    \n ## user-question statistics\n 'user_content_id_cumcount',\n 'user_content_id_timedelta', \n 'user_content_id_acc_diff_q_level_mean',\n    \n ## other features\n 'part_acc_mean',\n 'als_acc_mean'\n] \n\n## question \"embeddings\"\nselected_features += ['content_id_dim_' + str(i) for i in range(DIM)]\n\n## user-window statistics\nselected_features += [f'user_id_acc_diff_q_level_window_{j}' for j in WINDOWS_LIST]\n\nlen(selected_features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Writing train and valid data to csv (doesn't make sence on kaggle environment):"},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    df_train_.to_csv(f'{external_path}/df_train.csv', index=False)\n    df_valid.to_csv(f'{external_path}/df_valid.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    get_type = lambda f: 'float32' if 'dim' not in f else 'float16'\n\n    df_train_ = pd.read_csv(f'{external_path}/df_train.csv',\n                            usecols=['row_id', 'answered_correctly'] + selected_features,\n                            dtype={f: get_type(f) for f in selected_features})\n    df_valid = pd.read_csv(f'{external_path}/df_valid.csv',\n                           usecols=['row_id', 'answered_correctly'] + selected_features,\n                           dtype={f: get_type(f) for f in selected_features})\n\n    df_train_.memory_usage(deep=True).sum()/2**30","execution_count":null,"outputs":[]},{"metadata":{"jupyter":{"outputs_hidden":true},"trusted":true},"cell_type":"code","source":"%%time\n\nlgb_train = lgb.Dataset(df_train_[selected_features].values.astype('float32'), df_train_.answered_correctly)\nlgb_valid = lgb.Dataset(df_valid[selected_features].values.astype('float32'), df_valid.answered_correctly)\n#del df_train_; gc.collect()\n\nmodel = lgb.train(\n    {'objective': 'binary'},\n    lgb_train,\n    valid_sets=[lgb_train, lgb_valid],\n    verbose_eval=50,\n    num_boost_round=15000,\n    early_stopping_rounds=40\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nif IS_KAGGLE == False:\n    df_valid['score'] = model.predict(df_valid[selected_features])\n    print(roc_auc_score(df_valid.answered_correctly, df_valid.score))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Saving scored validation sample and model to csv (doesn't make sence on kaggle environment):"},{"metadata":{"trusted":true},"cell_type":"code","source":"if IS_KAGGLE == False:\n    df_valid.to_csv(f\"{external_path}/df_valid_scored.csv\", index=False)\n    pickle.dump(model, open(f\"{external_path}/model_v21.pickle.dat\", \"wb\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature importances table (note that with flag *IS_KAGGLE = True* this table will be different from the result of training on all data):"},{"metadata":{"jupyter":{"outputs_hidden":true},"trusted":true},"cell_type":"code","source":"df_importances = pd.DataFrame(selected_features, columns=['feature_name'])\ndf_importances['gain'] = model.feature_importance('gain')\ndf_importances = df_importances.sort_values('gain', ascending=False).reset_index(drop=True)\ndf_importances.head(50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}