{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Welcome!\n\nThis is the final version of [this](https://www.kaggle.com/markwijkhuizen/riiid-training-and-prediction-using-a-state) notebook, which I shared at the start of this competition.\n\nSeveral features have been added since then and some RAM efficiency improvements have been made.\n\nI learned a lot during this competition from all the public notebooks and discussions and references to notebooks and discussions I got valuable ideas from will be mentioned throughout this notebook.\n\nWhenever you have questions just leave a comment and I will answer it soon."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport lightgbm as lgb\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.metrics import roc_auc_score\nfrom multiprocessing import cpu_count\nfrom tqdm.notebook import tqdm\nfrom memory_profiler import memory_usage\nfrom cairosvg import svg2png\nfrom PIL import Image\nfrom io import BytesIO\nfrom collections import deque\nfrom glob import glob\nfrom statistics import mean\n\nimport gc\nimport os\nimport sys\nimport re\nimport pickle\nimport math","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"VERSION = 'V1G'\nNUM_BOOST_ROUND = 5000\nVERBOSE_EVAL = 10\nMETRICS = ['auc']\nN_ROWS = 99271300\n\n# used for setting the index of a new DataFrame\ndef get_index_np():\n    return np.arange(N_ROWS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function returns a feature\n# A list of indices and a data type can be passed to retrieve spcific training/validation rows as float32\ndef load_feature(feature, idxs=None, dtype=None):\n    file_path = f'/kaggle/input/riiid-answer-correctness-prediction-features-temp/FEATURES_{VERSION}/{feature}.npz'\n    if idxs is None and dtype is None:\n        return np.load(file_path, allow_pickle=True)['v']\n    elif idxs is not None and dtype is not None:\n        return np.load(file_path, allow_pickle=True)['v'][idxs].astype(dtype)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This simply shows all features\nfor file_path in glob(f'/kaggle/input/riiid-answer-correctness-prediction-features-temp/FEATURES_{VERSION}/*.npz'):\n    print(re.findall('(?<=.\\/)([a-z_0-9]*)(.npz)', file_path)[0][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# All features and a short description\n\n# Given features\n\n**prior_question_elapsed_time:** given feature\n\n**prior_question_had_explanation:** given feature\n\n**bundle_id:** given feature retrieved by merging questions df with train df, categorical feature\n\n# User features\n\n**mean_user_accuracy:** expanding mean user accuracy\n\n**mean_accuracy_diff:** expanding mean of the difference between the mean user accuracy and mean content accuracy\n\n**mean_user_content_accuracy:** expanding mean of the mean content accuracy of questions a user answered\n\n**answered_correctly_user:** amount of questions a user answered correctly\n\n**answered_user:** amount of questions a user answered\n    \n**mean_user_accuracy_r10:** rolling mean user accuracy with window size 10\n\n**mean_user_accuracy_r25:** rolling mean user accuracy with window size 25\n    \n**mean_user_accuracy_r100:** rolling mean user accuracy with window size 100 \n    \n# Content features\n    \n**mean_content_accuracy:** mean content accuracy given retry and prior_question_had_explanation (explained later)\n\n**content_id_pos_diff:** difference between mean position question and actual position of question.\nIf on average a question is asked as 100st question and a user get the question as 10th question this feature is 10-100=-90\n\n\n**content_count:** amount of times a question is asked\n\n# Part features\n    \n**part:** part the question belongs to, categorical feature\n\n**mean_user_part_accuracy:** expanding mean accuracy for a user for the part the question belongs to\n\n**answered_part_user:** amount of questions a user answered in the part the question belongs to\n    \n# Tag features\n    \n**tag_1:** first tag of the question, categorical feature\n    \n**tag_2:** second tag of the question, categorical feature, -1 if question has no second part\n\n# User content features\n    \n**hmean_user_content_accuracy:** harmonic mean of the mean_user_accuracy and mean_content_accuracy, based on [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189569) discussion\n\n# Last interaction elapsed time\n\nAll three features are based on [this](https://www.kaggle.com/ragnar123/riiid-model-lgbm) notebook\n\n**last_interaction_elapsed_time_l1:** timestamp difference with last interaction\n\n**last_interaction_elapsed_time_l2:** timestamp difference with two interactions earlier\n\n\n**last_interaction_elapsed_time_l3:** timestamp difference with three interactions earlier\n\n\n# Other\n\n**attempt:** amount of times a user has attempted the question, based on [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/194266) discussion\n\n**lectures_seen:** amount of lectures a user has seen\n\n\n# Lastly addded feature\n\n**last_question_mean_acc_diff:** difference in mean content accuracy between the current and last feature\n\n**user_part_answered_consecutive:** amount of consecutive answers of the current part.\n\n**mean_user_mean_content_acc_diff:** absolute difference between the mean_user_accuracy and mean_content_accuracy\n\n* The next two questions are based on [this](https://www.kaggle.com/ragnar123/riiid-model-lgbm) notebook\n\n**last_correct_time_elapsed:** timestamp difference between current question and last correctly answered question\n\n**last_incorrect_time_elapsed:** timestamp difference between current question and last incorrectly answered question"},{"metadata":{"trusted":true},"cell_type":"code","source":"# this are given features, bundle_id is retrieved by merging the questions df with the train df\ngiven_features = [\n    'prior_question_elapsed_time',\n    'prior_question_had_explanation',\n    'bundle_id',\n]\n\ndeduced_features = [\n    # user features\n    'mean_user_accuracy',\n    'mean_accuracy_diff',\n    'mean_user_content_accuracy',\n    'answered_correctly_user',\n    'answered_user',\n    'mean_user_accuracy_r10',\n    'mean_user_accuracy_r25',\n    'mean_user_accuracy_r100',\n    # content features\n    'mean_content_accuracy',\n    'content_id_pos_diff',\n    'content_count',\n    # part features\n    'part',\n    'mean_user_part_accuracy',\n    'answered_part_user',\n    # tag features\n    'tag_1',\n    'tag_2',\n    # user content features\n    'hmean_user_content_accuracy',\n    # last interaction elapsed time\n    'last_interaction_elapsed_time_l1',\n    'last_interaction_elapsed_time_l2',\n    'last_interaction_elapsed_time_l3',\n    # other\n    'attempt',\n    'lectures_seen',\n    \n    # lastly added features\n    'last_question_mean_acc_diff',\n    'user_part_answered_consecutive',\n    'mean_user_mean_content_acc_diff',\n    'last_correct_time_elapsed',\n    'last_incorrect_time_elapsed',\n]\n\nfeatures = given_features + deduced_features\n\nfeatures_df_cols = [\n    'user_id', 'content_id', 'part', 'tags', # merge keys\n    'tags_label', 'answered_user', # deduced data\n    'answered_correctly_user', 'mean_user_accuracy', 'mean_content_accuracy', # deduced features\n]\n\ntarget = 'answered_correctly'\n\n# add indices of categorical features as data will be fed as numpy array to LightGBM and not as DataFrame\n# in DataFrame all categoricaly set columns are automatically interpreted as categorical features\n# This is not the case with a numpy array, as it is a simle matrix\n# We thus need to specify the indices of the columns with categorical features to LightGBM\ncategorical_feature = ['part', 'retry', 'prior_question_had_explanation', 'bundle_id', 'tag_1', 'tag_2']\ncategorical_feature_idxs = []\nfor v in categorical_feature:\n    try:\n        categorical_feature_idxs.append(features.index(v))\n    except:\n        pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make train and validation datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_train_val_idxs(TRAIN_SIZE, VAL_SIZE):\n    train_idxs = []\n    val_idxs = []\n    NEW_USER_FRAC = 1/4 # fraction of new users, 25% of validation rows are new users\n    np.random.seed(42)\n    \n    # create df with user_ids and indices\n    df = pd.DataFrame(index=get_index_np())\n    for feature in ['user_id']:\n        df[feature] = load_feature(feature)\n\n    df['index'] = df.index.values.astype(np.uint32)\n    user_id_index = df.groupby('user_id')['index'].apply(np.array)\n    \n    # iterate over users in random order\n    for indices in user_id_index.sample(user_id_index.size, random_state=42):\n        if len(train_idxs) > TRAIN_SIZE:\n            break\n\n        # fill validation data\n        if len(val_idxs) < VAL_SIZE:\n            # add new user\n            if np.random.rand() < NEW_USER_FRAC:\n                val_idxs += list(indices)\n            # randomly split user between train and val\n            else:\n                offset = np.random.randint(0, indices.size)\n                train_idxs += list(indices[:offset])\n                val_idxs += list(indices[offset:])\n        else:\n            train_idxs += list(indices)\n    \n    train_idxs = np.array(train_idxs, dtype=np.uint32)\n    val_idxs = np.array(val_idxs, dtype=np.uint32)\n    \n    return train_idxs, val_idxs\n\ntrain_idxs, val_idxs = get_train_val_idxs(int(96.5e6), 2.5e6)\nprint(f'len train_idxs: {len(train_idxs)}, len validation_idxs: {len(val_idxs)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_x_y(train_idxs, val_idxs):\n    # create numpy arrays\n    X_train = np.ndarray(shape=(len(train_idxs), len(features)), dtype=np.float32)\n    X_val = np.ndarray(shape=(len(val_idxs), len(features)), dtype=np.float32)\n    \n    # now fill them up column wise to reduce memory usage\n    # features are loaded from disk as npz files (compressed numpy arrays)\n    for idx, feature in enumerate(tqdm(features)):\n        X_train[:,idx] = load_feature(feature, train_idxs, np.float32)\n        X_val[:,idx] = load_feature(feature, val_idxs, np.float32)\n    \n    y_train = load_feature(target, train_idxs, np.int8)\n    y_val = load_feature(target, val_idxs, np.int8)\n                         \n    return X_train, y_train, X_val, y_val\n    \nX_train, y_train, X_val, y_val = make_x_y(train_idxs, val_idxs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'X_train.shape: {X_train.shape}\\t y_train.shape: {y_train.shape}')\nprint(f'X_val.shape: {X_val.shape}\\t y_val.shape: {y_val.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next DataFrame gives an overview of the features, note all features are type float32 as LightGBM will use the data without copying. When feeding the features as DataFrame LightGBM will convert the DataFrame to float64, causing a massive RAM usage spike.\n\nPassing the features as numpy array of type float32 will prevent LightGBM from copying the data and numpy arrays are by default much more memory efficient than DataFrames."},{"metadata":{"trusted":true},"cell_type":"code","source":"# show train features\ndisplay(pd.DataFrame(X_train[:25], columns=features))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check the target (answered correctly) as sanity check\ndisplay(y_train[:25])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# make train and validation dataset\n# Do not specify the categorical feature here, but only for training\n# If we do specify it here LightGBM will throw a warning for overriding the categorical features\ntrain_data = lgb.Dataset(\n    data = X_train,\n    label = y_train,\n    categorical_feature = None,\n)\n\nval_data = lgb.Dataset(\n    data = X_val,\n    label = y_val,\n    categorical_feature = None,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Free up RAM\ndel X_train, y_train, X_val, y_val, train_idxs, val_idxs\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Simple LightGBM parameters\n# There is some room for improvement here, any suggestions are welcome!\nlgbm_params = {\n    'objective': 'binary',\n    'metric': METRICS,\n    'num_leaves': 200,\n    'learning_rate': 0.1,\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef train():\n    evals_result = {}\n    model = lgb.train(\n        params = lgbm_params,\n        train_set = train_data,\n        valid_sets = [val_data],\n        num_boost_round = NUM_BOOST_ROUND,\n        verbose_eval = VERBOSE_EVAL,\n        evals_result = evals_result,\n        early_stopping_rounds = 10,\n        categorical_feature = categorical_feature_idxs,\n        feature_name = features,\n    )\n\n    # save model\n    model.save_model(f'model_{VERSION}_{NUM_BOOST_ROUND}.lgb')\n    \n    return model, evals_result\n    \nmodel, evals_result = train()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training History"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_history(evals_result):\n    for metric in METRICS:\n        plt.figure(figsize=(20,8))\n        \n        for key in evals_result.keys():\n            history_len = len(evals_result.get(key)[metric])\n            history = evals_result.get(key)[metric]\n            x_axis = np.arange(1, history_len + 1)\n            plt.plot(x_axis, history, label=key)\n        \n        x_ticks = list(filter(lambda e: (e % (history_len // 100 * 10) == 0) or e == 1, x_axis))\n        plt.xticks(x_ticks, fontsize=12)\n        plt.yticks(fontsize=12)\n\n        plt.title(f'{metric.upper()} History of training', fontsize=18);\n        plt.xlabel('EPOCH', fontsize=16)\n        plt.ylabel(metric.upper(), fontsize=16)\n        \n        if metric in ['auc']:\n            plt.legend(loc='upper left', fontsize=14)\n        else:\n            plt.legend(loc='upper right', fontsize=14)\n        plt.grid()\n        plt.show()\n\nplot_history(evals_result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_feature_importances(model, importance_type, max_num_features=10**10):\n    feature_importances = pd.DataFrame()\n    feature_importances['feature'] = features\n    feature_importances['value'] = pd.DataFrame(model.feature_importance(importance_type))\n    feature_importances = feature_importances.sort_values(by='value', ascending=False) # sort feature importance\n    feature_importances.to_csv(f'feature_importances_{importance_type}.csv') # write feature importance to csv\n    feature_importances = feature_importances[:max_num_features] # only show max_num_features\n    \n    plt.figure(figsize=(18, 8))\n    plt.xlim([0, feature_importances.value.max()*1.1])\n    plt.title(f'Feature {importance_type}', fontsize=18);\n    sns.barplot(data=feature_importances, x='value', y='feature', palette='rocket');\n    for idx, v in enumerate(feature_importances.value):\n        plt.text(v, idx, \"  {:.2e}\".format(v))\n\nshow_feature_importances(model, 'gain')\nshow_feature_importances(model, 'split')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show tree and save as png\ndef save_tree_diagraph(model):\n    tree_digraph = lgb.create_tree_digraph(model, show_info=['split_gain', 'internal_count'])\n\n    tree_png = svg2png(tree_digraph._repr_svg_(), output_width=3840)\n    tree_png = Image.open(BytesIO(tree_png))\n\n    tree_png.save('create_tree_digraph.png')\n\n    display(tree_png)\n    \nsave_tree_diagraph(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# remove train and validation data to free memory before prediction phase\ndel train_data, val_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction Preparation"},{"metadata":{},"cell_type":"markdown","source":"The next dataframe is used for merging with the test_df to get some non-user dependent content, part and tag features"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_features_questions_df():\n    # create DataFrame of features\n    features_questions_df = pd.DataFrame(index=get_index_np())\n    cols = [\n        'content_id',\n        'part',\n        'tag_1',\n        'tag_2',\n        'content_count',\n        'mean_content_id_pos',\n        'bundle_id',\n    ]\n    for feature in tqdm(cols):\n        features_questions_df[feature] = load_feature(feature)\n\n    # content features\n    features_questions_df.drop_duplicates(subset='content_id', inplace=True)\n    features_questions_df.sort_values('content_id', inplace=True)\n    features_questions_df.reset_index(drop=True, inplace=True)\n    \n    return features_questions_df\n    \nfeatures_questions_df = get_features_questions_df()\nprint(f'features_questions_df, rows: {features_questions_df.shape[0]}')\ndisplay(features_questions_df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# STATE\n\nThis next function is the beating heart of my prediction phase, a massive dictionary to keep track of all features of all users and update them with every interaction.\n\nI agree, the code is somewhat unreadable, but the basic idea is as follows:\n\nCompute features over all user data (mean_user_accuracy, answered_user, answered_correctly_user) \nas these features have a lag of 1\n\nGet the last data point for other features (lecturs seen, mean_content_accuracy, etc)\n\nCreate a dictionary where for each user all features are kept track of, an example of a user is shown below the function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_state():\n    # create DataFrame of features\n    features_df = pd.DataFrame(index=get_index_np())\n    cols = [\n        'user_id', 'content_id', 'answered_correctly','lectures_seen',\n        'mean_user_accuracy', 'mean_content_accuracy',\n        'last_correct_time_elapsed', 'last_incorrect_time_elapsed', 'part', 'user_part_answered_consecutive',\n        'timestamp', 'last_interaction_elapsed_time_l1', 'last_interaction_elapsed_time_l2', 'last_interaction_elapsed_time_l3',\n    ]\n    for f in tqdm(cols):\n        features_df[f] = load_feature(f)\n        \n    # get last features\n    last_features = features_df.groupby('user_id')[[\n        'timestamp', 'lectures_seen', 'mean_content_accuracy', 'part', 'user_part_answered_consecutive',\n        'last_correct_time_elapsed', 'last_incorrect_time_elapsed',\n        'last_interaction_elapsed_time_l1', 'last_interaction_elapsed_time_l2', 'last_interaction_elapsed_time_l3',\n    ]].last()\n    \n    # last correct/incorrect time elapsed\n    last_correct_features = features_df.groupby(['user_id', 'answered_correctly'])['timestamp'].last()\n    \n    # drop features only used for last feature computation\n    features_df.drop([\n        'timestamp', 'last_interaction_elapsed_time_l1', 'last_interaction_elapsed_time_l2', 'last_interaction_elapsed_time_l3',\n        'last_correct_time_elapsed', 'last_incorrect_time_elapsed', 'user_part_answered_consecutive',\n    ], axis=1, inplace=True)\n        \n    # compute user features over all train data\n    features_df_grouped_by_user = features_df[['user_id', 'answered_correctly']].groupby('user_id')['answered_correctly']\n    mean_user_accuracy = features_df_grouped_by_user.mean().values.astype(np.float32)\n    answered_correctly_user = features_df_grouped_by_user.sum().values.astype(np.uint16)\n    answered_user = features_df_grouped_by_user.count().values.astype(np.uint16)\n    last_questions_answered_100 = features_df.groupby('user_id')[['user_id', 'answered_correctly']].tail(100).groupby('user_id')['answered_correctly'].apply(np.array).apply(np.flip)\n    # user_mean_content_accuracy_sum for computing mean_user_content_accuracy\n    mean_content_accuracy_sum = features_df.groupby('user_id')['mean_content_accuracy'].sum().values\n    # mean_accuracy_diff\n    mean_accuracy_diff_sum = (features_df.groupby('user_id')['mean_user_accuracy'].sum() - features_df.groupby('user_id')['mean_content_accuracy'].sum()).values\n    # user part features (count, sum)\n    user_part_features = features_df.groupby(['user_id', 'part'])['answered_correctly'].agg(['count', 'sum']).reset_index().astype({'user_id': int, 'part': int})\n    \n    del features_df_grouped_by_user, features_df\n    gc.collect()\n    \n    # get state with precomputed attempts\n    with open('/kaggle/input/riiid-answer-correctness-prediction-features/state.pkl', 'rb') as state_pickle_file:\n         state = pickle.load(state_pickle_file)\n    \n    # add all features to state\n    for idx, user_id in tqdm(enumerate(state.keys()), total=len(state)):\n        state[user_id]['mean_user_accuracy'] = mean_user_accuracy[idx]\n        state[user_id]['answered_correctly_user'] = answered_correctly_user[idx]\n        state[user_id]['answered_user'] = answered_user[idx]\n        state[user_id]['mean_content_accuracy_sum'] = mean_content_accuracy_sum[idx]\n        state[user_id]['mean_accuracy_diff_sum'] = mean_accuracy_diff_sum[idx]\n        # last features\n        state[user_id]['lectures_seen'] = last_features.loc[user_id, 'lectures_seen']\n        state[user_id]['timestamp'] = last_features.loc[user_id, 'timestamp']\n        state[user_id]['last_mean_content_accuracy'] = last_features.loc[user_id, 'mean_content_accuracy']\n        state[user_id]['last_part'] = last_features.loc[user_id, 'part']\n        state[user_id]['user_part_answered_consecutive'] = last_features.loc[user_id, 'user_part_answered_consecutive']\n        state[user_id]['last_correct_timestamp'] = last_correct_features.loc[user_id, True] if (user_id, True) in last_correct_features else np.nan\n        state[user_id]['last_incorrect_timestamp'] = last_correct_features.loc[user_id, False] if (user_id, False) in last_correct_features else np.nan\n        state[user_id]['last_interaction_elapsed_time_l1'] = last_features.loc[user_id, 'last_interaction_elapsed_time_l1']\n        state[user_id]['last_interaction_elapsed_time_l2'] = last_features.loc[user_id, 'last_interaction_elapsed_time_l2']\n        state[user_id]['last_interaction_elapsed_time_l3'] = last_features.loc[user_id, 'last_interaction_elapsed_time_l3']\n        state[user_id]['mean_user_accuracy_r10'] = deque(last_questions_answered_100[user_id][:10], maxlen=10)\n        state[user_id]['mean_user_accuracy_r25'] = deque(last_questions_answered_100[user_id][:25], maxlen=25)\n        state[user_id]['mean_user_accuracy_r100'] = deque(last_questions_answered_100[user_id], maxlen=100)\n            \n    # add user part features\n    for idx, (user_id, part, part_count, part_sum) in tqdm(user_part_features.iterrows(), total=user_part_features.index.size):\n        state[user_id][f'part_{part}_count'] = part_count\n        state[user_id][f'part_{part}_sum'] = part_sum\n                \n    return state\n\nstate = get_state()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Example of the state for the famous user 115\ndisplay(state[115])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gives the state for a new user with all default values\ndef get_new_user(row):\n    return {\n        'mean_user_accuracy': 0.680,\n        'answered_correctly_user': 0,\n        'answered_user': 0,\n        'user_content_attempts': dict(),\n        'lectures_seen': 0,\n        'timestamp': row['timestamp'],\n        'last_mean_content_accuracy': 0,\n        'last_part': int(row['part']),\n        'user_part_answered_consecutive': 0,\n        'last_correct_timestamp': np.nan,\n        'last_incorrect_timestamp': np.nan,\n        'last_interaction_elapsed_time_l1': 0,\n        'last_interaction_elapsed_time_l2': 0,\n        'last_interaction_elapsed_time_l3': 0,\n        'mean_user_accuracy_r10': deque([], maxlen=10),\n        'mean_user_accuracy_r25': deque([], maxlen=25),\n        'mean_user_accuracy_r100': deque([], maxlen=100),\n        'mean_accuracy_diff_sum': 0,\n        'mean_content_accuracy_sum': 0,\n    }\n\n# returns a dictionary with a list for all user features\ndef get_user_data_dict():\n    return {\n        'mean_user_accuracy': [],\n        'answered_correctly_user': [],\n        'answered_user': [],\n        'lectures_seen': [],\n        'last_interaction_elapsed_time_l1': [],\n        'last_interaction_elapsed_time_l2': [],\n        'last_interaction_elapsed_time_l3': [],\n        'mean_user_accuracy_r10': [],\n        'mean_user_accuracy_r25': [],\n        'mean_user_accuracy_r100': [],\n        'mean_user_part_accuracy': [],\n        'answered_part_user': [],\n        'mean_accuracy_diff': [],\n        'mean_user_content_accuracy': [],\n        'last_question_mean_acc_diff': [],\n        'user_part_answered_consecutive': [],\n        'mean_user_mean_content_acc_diff': [],\n        'last_correct_time_elapsed': [],\n        'last_incorrect_time_elapsed': [],\n    }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next function retrieves features for all user from the state and returns a dictionary with all features as shown above."},{"metadata":{"trusted":true},"cell_type":"code","source":"# dictionary which converts a content_id to a tag\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv', dtype={ 'tag': np.uint8 })[['lecture_id', 'tag']]\ncontent_id2tag = pd.Series(lectures['tag'].values, index=lectures['lecture_id'].values)\n\ndef get_user_data(state, test_df):\n    # updated data\n    user_data = get_user_data_dict()\n    \n    # mean first question part accuracies\n    part_first_question_mean_accuracy_dict = {1: 0.75, 2: 0.60, 3: 0.49, 4: 0.41, 5: 0.52, 6: 0.51, 7: 0.47}\n    cols = ['user_id', 'content_id', 'content_type_id', 'timestamp', 'part', 'mean_content_accuracy']\n    \n    for idx, (user_id, content_id, is_lecture, timestamp, part, mean_content_accuracy) in test_df[cols].iterrows():\n        # LECTURE\n        if is_lecture:\n            state[user_id]['lectures_seen'] += 1\n            \n            # fill user data with dummy value\n            for key in user_data.keys():\n                user_data[key].append(0)\n            \n        # QUESTION\n        else:\n            part = int(part)\n            \n            # update last interaction elapsed time\n            state[user_id]['last_interaction_elapsed_time_l3'] = state[user_id]['last_interaction_elapsed_time_l2']\n            state[user_id]['last_interaction_elapsed_time_l2'] = state[user_id]['last_interaction_elapsed_time_l1']\n            if timestamp != state[user_id]['timestamp']:\n                state[user_id]['last_interaction_elapsed_time_l1'] = timestamp - state[user_id]['timestamp']\n                state[user_id]['timestamp'] = timestamp\n            \n            # add various features\n            cols = [\n                'mean_user_accuracy', 'answered_correctly_user', 'answered_user', 'lectures_seen',\n                'last_interaction_elapsed_time_l1', 'last_interaction_elapsed_time_l2', 'last_interaction_elapsed_time_l3',\n            ]\n            for feature in cols:\n                user_data[feature].append(state[user_id][feature])\n            \n            # branch on first question\n            if state[user_id]['answered_user'] > 0:\n                user_data['mean_accuracy_diff'].append(state[user_id]['mean_accuracy_diff_sum'] / state[user_id]['answered_user'])\n                user_data['mean_user_content_accuracy'].append(state[user_id]['mean_content_accuracy_sum'] / state[user_id]['answered_user'])\n                user_data['mean_user_accuracy_r10'].append(np.array(state[user_id]['mean_user_accuracy_r10']).mean())\n                user_data['mean_user_accuracy_r25'].append(np.array(state[user_id]['mean_user_accuracy_r25']).mean())\n                user_data['mean_user_accuracy_r100'].append(np.array(state[user_id]['mean_user_accuracy_r100']).mean())\n            else:\n                user_data['mean_accuracy_diff'].append(0)\n                user_data['mean_user_content_accuracy'].append(0)\n                user_data['mean_user_accuracy_r10'].append(0.68)\n                user_data['mean_user_accuracy_r25'].append(0.68)\n                user_data['mean_user_accuracy_r100'].append(0.68)\n            \n            # last question mean acc diff\n            if state[user_id]['last_mean_content_accuracy'] != 0:\n                user_data['last_question_mean_acc_diff'].append(mean_content_accuracy - state[user_id]['last_mean_content_accuracy'])\n            else:\n                user_data['last_question_mean_acc_diff'].append(0)\n            \n            state[user_id]['last_mean_content_accuracy'] = mean_content_accuracy\n            \n            # user part features\n            if f'part_{part}_count' in state[user_id]:\n                if state[user_id][f'part_{part}_sum'] == 0:\n                    user_data['mean_user_part_accuracy'].append(0)\n                else:\n                    user_data['mean_user_part_accuracy'].append(state[user_id][f'part_{part}_sum'] / state[user_id][f'part_{part}_count'])\n                \n                user_data['answered_part_user'].append(state[user_id][f'part_{part}_count'])\n            else:\n                state[user_id][f'part_{part}_sum'] = 0\n                state[user_id][f'part_{part}_count'] = 0\n                user_data['mean_user_part_accuracy'].append(part_first_question_mean_accuracy_dict[part])\n                user_data['answered_part_user'].append(0)\n            \n            # user_part_answered_consecutive\n            if part == state[user_id]['last_part']:\n                state[user_id]['user_part_answered_consecutive'] += 1\n            else:\n                state[user_id]['last_part'] = part\n                state[user_id]['user_part_answered_consecutive'] = 0\n                \n            user_data['user_part_answered_consecutive'].append(state[user_id]['user_part_answered_consecutive'])\n            \n            # mean_user_mean_content_acc_diff\n            user_data['mean_user_mean_content_acc_diff'].append(mean_content_accuracy - state[user_id]['mean_user_accuracy'])\n            \n            # last correct/incorrect time elapsed\n            user_data['last_correct_time_elapsed'].append(timestamp - state[user_id]['last_correct_timestamp'])\n            user_data['last_incorrect_time_elapsed'].append(timestamp - state[user_id]['last_incorrect_timestamp'])\n    \n    return user_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# adds all new users to the state with default values\ndef add_new_users(test_df):\n    for idx, row in test_df.iterrows():\n        # check if user exists\n        if not row['user_id'] in state:\n            state[row['user_id']] = get_new_user(row)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# adds the attempt and retry feature to the test_df\ndef add_attempt_retry(test_df):\n    attempt = []\n    for user_id, content_id in test_df[['user_id', 'content_id']].itertuples(name=None, index=False):\n        if content_id in state[user_id]['user_content_attempts']:\n            state[user_id]['user_content_attempts'][content_id] += 1\n        else:\n            state[user_id]['user_content_attempts'][content_id] = 0\n\n        attempt.append(state[user_id]['user_content_attempts'][content_id])\n    \n    test_df['attempt'] = attempt\n    test_df['retry'] = test_df['attempt'] > 0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This next function adds the mean_content_accuracy taking the prior_question_had_explanation and retry feature into account. an example of the effect of prior_question_had_explanation and retry are given below."},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('/kaggle/input/riiid-answer-correctness-prediction-features/mean_content_accuracy_cases_dict.pickle', 'rb') as f:\n    mean_content_accuracy_cases_dict = pickle.load(f)\n\ndef add_mean_content_accuracy(test_df):\n    mean_content_accuracy = []\n    for key in test_df[['content_id', 'prior_question_had_explanation', 'retry']].itertuples(name=None, index=False):\n        # get mean content accuracy\n        if key in mean_content_accuracy_cases_dict:\n            mean_content_accuracy.append(mean_content_accuracy_cases_dict[key])\n        else:\n            mean_content_accuracy.append(0)\n        \n    test_df['mean_content_accuracy'] = mean_content_accuracy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Example of the mean_content_accuracy for quesiton 6116, the most answered question\nfor (content_id, prior_question_had_explanation, retry), mean_content_accuracy in mean_content_accuracy_cases_dict.items():\n    if content_id == 6116:\n        print(f'content_id {content_id}, prior_question_had_explanation: {prior_question_had_explanation}, retry: {retry}, mean_content_accuracy: {mean_content_accuracy}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After each prediction iteration the user features are updated"},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_user_data(state, prev_test_df):\n    for idx, row in prev_test_df.iterrows():\n        if not row['content_type_id']:\n            answered_correctly = row['answered_correctly']\n            user_id = row['user_id']\n            part = int(row['part'])\n            # update user features\n            state[user_id]['answered_correctly_user'] += answered_correctly\n            state[user_id]['answered_user'] += 1\n            state[user_id]['mean_user_accuracy'] = state[user_id]['answered_correctly_user'] / state[user_id]['answered_user']\n            state[user_id]['mean_user_accuracy_r10'].appendleft(answered_correctly)\n            state[user_id]['mean_user_accuracy_r25'].appendleft(answered_correctly)\n            state[user_id]['mean_user_accuracy_r100'].appendleft(answered_correctly)\n            # add user part features, initialize if no part answered yet\n            state[user_id][f'part_{part}_sum'] += answered_correctly\n            state[user_id][f'part_{part}_count'] += 1\n            # update other user features\n            state[user_id]['mean_accuracy_diff_sum'] += row['mean_user_accuracy'] - row['mean_content_accuracy']\n            state[user_id]['mean_content_accuracy_sum'] += row['mean_content_accuracy']\n            # last correct/incorrect time elapsed\n            if answered_correctly:\n                state[user_id]['last_correct_timestamp'] = row['timestamp']\n            else:\n                state[user_id]['last_incorrect_timestamp'] = row['timestamp']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make actual prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\n\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.display.max_columns = 99\nprev_test_df = None\n\nfor idx, (test_df, _) in tqdm(enumerate(iter_test)):\n    # from 2nd iteration, update user data\n    if prev_test_df is not None:\n        prev_test_df['answered_correctly'] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        update_user_data(state, prev_test_df)\n        if idx < 4:\n            display(test_df)\n            display(prev_test_df)\n            \n    # fill prior question had explenation\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df['prior_question_elapsed_time'].fillna(23916, inplace=True)\n            \n    # merge with all features\n    test_df = features_questions_df.merge(test_df, how='right', on='content_id')\n    \n    # add new users to state\n    add_new_users(test_df)\n    \n    # add attempt, retry and mean_content_accuracy\n    add_attempt_retry(test_df)\n    add_mean_content_accuracy(test_df)\n    \n    # get user data from state and update attempt\n    user_data = get_user_data(state, test_df)\n    for feature, values in user_data.items():\n        test_df[feature] = values\n    \n    # add harmonic mean\n    test_df['hmean_user_content_accuracy'] = 2 * (\n        (test_df['mean_user_accuracy'] * test_df['mean_content_accuracy']) /\n        (test_df['mean_user_accuracy'] + test_df['mean_content_accuracy'])\n    )\n\n    # add content_d position difference\n    test_df['content_id_pos_diff'] = (test_df['answered_user'] - test_df['mean_content_id_pos']).values.astype(np.int16)\n\n    test_df['answered_correctly'] = model.predict(test_df[features])\n\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n\n    # set previour test_df\n    prev_test_df = test_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('./submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show the first 5 predictions\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That was all of it, hope you enjoyed this notebook and thanks for taking a look at it!\n\nAs I am quite new to the machine learning community any tips and tricks are welcome!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}