{"cells":[{"metadata":{},"cell_type":"markdown","source":"## general\n\nThis is our final inference pipeline. Unfortunately we did not have time to blend anything but this LGB can achieve .7936 CV/.794 Private LB on its own.\n\nThe full training code can be found here:\nhttps://github.com/nicohrubec/riid_solution\n\n**Thanks to @gmilosev for the cool collaboration. This is a joint effort.**"},{"metadata":{},"cell_type":"markdown","source":"## features\n\nThe final model uses only 35 features. Some of the most important were:\n- mean answered_correctly for user\n- rolling mean answered_correctly for user\n- mean answered_correctly for user on question\n- mean answered_correctly for user on part\n- last time the user has seen the current question\n- last time the user was seen\n- last n times the user was seen\n- user count\n- user count for question\n- bin questions to difficulty level and compute mean user answered_correctly per difficulty bin\n\n\n## memory management\nSince we do not see all users we have in train in the test set it does not make sense to precompute the feature dictionaries for all users seen in train especially since we only have 16 GB of ram. Here it is solved by on the fly dictionary computation as follows:\n1. init empty feature dictionaries before inference\n2. check for each user we encounter if the user can already be found in the feature dictionaries\n3. if not: check if the user is in train\n4. if we dont have the user in the dicts and it can be found in train: query the train data and update the feature dicts with the train data for this user.\n\nIn the beginning I was struggling a lot with kernel memory restrictions when adding new features until I found this strategy. After we switched to this strategy we never had any problems."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport pickle\nimport lightgbm as lgb\nimport riiideducation\nfrom sklearn.metrics import roc_auc_score\nfrom tqdm import tqdm\nimport gc\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TARGET = 'answered_correctly'\nfeats = ['content_id', 'task_container_id', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'part', 'content_id_target_mean', 'user_count', 'user_correct_mean', 'user_question_count', 'user_question_correct_mean', 'last_time_user', 'last_time_question_user', 'last_time_user_inter', 'user_last_n_correct', 'answer1', 'answer2', 'answer3', 'answer4', 'user_last_n_time', 'user_last_n_time2', 'user_last_n_time3', 'user_part_count', 'user_part_correct_mean', 'user_part1_mean', 'user_part2_mean', 'user_part3_mean', 'user_part4_mean', 'user_part5_mean', 'user_part6_mean', 'user_part7_mean', 'task_container_eq1', 'task_container_eq2', 'user_hardness_count', 'user_hardness_mean', 'user_hardness_inter']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = '../input/lgb-baseline-riid/lgb_0.7936991.dat'\nquestions_file = '../input/riiid-test-answer-prediction/questions.csv'\nlectures_file = '../input/riiid-test-answer-prediction/lectures.csv'\ntrain_file = '../input/riiid-test-answer-prediction/train.csv'\ntest_file = '../input/riiid-test-answer-prediction/example_test.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_and_merge_feat(trn, target, feat):\n    # compute target mean and merge on train\n    feat_name = '{}_target_mean'.format(feat)\n    mean = trn[[feat, target]].groupby([feat]).agg(['mean'])\n    mean.columns = [feat_name]\n    trn = pd.merge(trn, mean, on=feat, how='left')\n\n    # transform df to dict for test merge\n    feat_dict = mean.astype('float32').to_dict()[feat_name]\n\n    return trn, feat_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_answer_feats(trn):\n    answer_counts = trn.groupby('content_id')['user_answer'].value_counts(normalize=True)\n\n    answer_counts_unstack = answer_counts.unstack().reset_index(drop=True).astype(np.float32)\n    answer_counts_unstack.columns = ['answer1', 'answer2', 'answer3', 'answer4']\n    answer_counts_unstack = answer_counts_unstack.rename_axis('content_id').reset_index()\n    answers = answer_counts_unstack.values[:, -4:].astype(np.float32)\n    answers.sort(axis=1)\n    answer_counts_unstack[['answer1', 'answer2', 'answer3', 'answer4']] = answers\n    answer_counts_unstack = answer_counts_unstack.astype(np.float32)\n\n    return answer_counts_unstack","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_q_hardness_bins(df):\n    df['question_hardness'] = 0\n    df.question_hardness.values[df.content_id_target_mean >= 0.9] = -10  # very easy\n    df.question_hardness.values[(df.content_id_target_mean >= 0.7) & (df.content_id_target_mean < 0.9)] = -11  # easier\n    df.question_hardness.values[(df.content_id_target_mean >= 0.5) & (df.content_id_target_mean < 0.7)] = -12  # harder\n    df.question_hardness.values[df.content_id_target_mean < 0.5] = -13  # hard\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"needed_cols = ['user_id', 'content_id', 'timestamp', 'part', 'task_container_id', TARGET, 'user_answer']\n\nprint(\"Get content mean feature ...\")\ntrain = pd.read_pickle('../input/local-training-file-pickled/train_all.pkl')[needed_cols]\ntrain = train[train.answered_correctly != -1] # exclude lectures\ntrain, content_dict = get_and_merge_feat(train, TARGET, 'content_id')  # get question target mean\ntrain = get_q_hardness_bins(train)  # bin questions into target mean bins --> question difficulty clusters\n\nprint(\"Get question answer distribution ...\")\nanswer_counts = get_answer_feats(train) # answer distribution for each question\n\ndel train['user_answer']\ngc.collect()\n\nmodel = pickle.load(open(model_path, \"rb\"))\nuser_idx = pd.concat([ train[['user_id']].drop_duplicates(keep='first'), train[['user_id']].drop_duplicates(keep='last'), ]).reset_index().sort_values(['user_id', 'index']).groupby(['user_id'])['index'].agg([list]).to_dict()['list']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert answer distribution to {question: distribution} dictionary\nanswer_dist_dict = {}\n\nfor idx, row in enumerate(answer_counts.values):\n    answer_dist_dict[int(row[0])] = [row[1], row[2], row[3], row[4]]\n\ndel answer_counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_dicts(row, count_dict, correct_dict, time_dict, last_n_dict):\n    # get dictionary keys for each feature\n    user = int(row[0])\n    question = int(row[1])\n    timestamp = int(row[2])\n    part = int(-row[3])\n    task_id = int(row[4])\n    hardness = int(row[5])\n    correct = int(row[6])\n\n    if user in count_dict:  # known user\n        # overall user features\n        count_dict[user]['sum'] += 1\n        correct_dict[user]['sum'] += correct\n        time_dict[user]['last'] = timestamp\n        last_n_dict[user]['last_task2'] = last_n_dict[user]['last_task']\n        last_n_dict[user]['last_task'] = task_id\n\n        # update rolling answered correct for user\n        last_n_dict[user]['last_n'].append(correct)\n        correction = last_n_dict[user]['last_n'].pop(0)\n        last_n_dict[user]['sum'] -= correction\n        last_n_dict[user]['sum'] += correct\n\n        # update last n timestamps for user\n        last_n_dict[user]['last_n_time'].append(timestamp)\n        last_n_dict[user]['time_sum'] = last_n_dict[user]['last_n_time'].pop(0)\n        last_n_dict[user]['time_sum2'] = last_n_dict[user]['last_n_time'][9]\n        last_n_dict[user]['time_sum3'] = last_n_dict[user]['last_n_time'][14]\n\n        # update question difficulty specific features\n        if hardness in count_dict[user]:\n            count_dict[user][hardness] += 1\n            correct_dict[user][hardness] += correct\n        else:\n            count_dict[user][hardness] = 1\n            correct_dict[user][hardness] = correct\n\n        # update part specific features\n        if part in count_dict[user]:\n            count_dict[user][part] += 1\n            correct_dict[user][part] += correct\n        else:\n            count_dict[user][part] = 1\n            correct_dict[user][part] = correct\n\n        # update question specific features\n        if question in count_dict[user]:  # known question for this user\n            count_dict[user][question] += 1\n            correct_dict[user][question] += correct\n            time_dict[user][question] = timestamp\n        else:  # unknown question for this user\n            count_dict[user][question] = 1\n            correct_dict[user][question] = correct\n            time_dict[user][question] = timestamp\n\n    else:  # unknown user create new entry\n        count_dict[user] = {'sum': 1, question: 1, part: 1, hardness: 1}\n        correct_dict[user] = {'sum': correct, question: correct, part: correct, hardness: correct}\n        time_dict[user] = {'last': timestamp, question: timestamp}\n        last_n_dict[user] = {'sum': correct, 'time_sum': 0, 'time_sum2': 0, 'time_sum3': 0,\n                             'last_n': [0, 0, 0, 0, correct], 'last_task': task_id, 'last_task2': np.nan,\n                             'last_n_time': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, timestamp]}\n\n    return count_dict, correct_dict, time_dict, last_n_dict\n\n\ndef get_row_values(row, count_dict, correct_dict, time_dict, last_n_dict):\n    feats = np.full(23, fill_value=np.nan, dtype=np.float32)  # row storage\n\n    # get dictionary keys\n    user = int(row[0])\n    question = int(row[1])\n    timestamp = int(row[2])\n    part = int(-row[3])\n    task_id = int(row[4])\n    hardness = int(row[5])\n\n    if user in count_dict:  # known user\n        # get overall user specific features\n        feats[0] = count_dict[user]['sum']\n        feats[1] = correct_dict[user]['sum']\n        feats[4] = timestamp - time_dict[user]['last']\n        feats[6] = last_n_dict[user]['sum']\n        feats[7] = timestamp - last_n_dict[user]['time_sum']\n        feats[8] = timestamp - last_n_dict[user]['time_sum2']\n        feats[9] = timestamp - last_n_dict[user]['time_sum3']\n        feats[19] = task_id - last_n_dict[user]['last_task']\n        feats[20] = task_id - last_n_dict[user]['last_task2']\n\n        # get question difficulty specific features\n        if hardness in count_dict[user]:\n            feats[21] = count_dict[user][hardness]\n            feats[22] = correct_dict[user][hardness]\n\n        # get part specific features\n        if part in count_dict[user]:\n            feats[10] = count_dict[user][part]\n            feats[11] = correct_dict[user][part]\n\n        # add part features for all other parts as well\n        for i, p in enumerate([-1, -2, -3, -4, -5, -6, -7]):\n            if p in count_dict[user]:\n                feats[12 + i] = correct_dict[user][p] / count_dict[user][p]\n\n        # add question specific features\n        if question in count_dict[user]:  # known question for this user\n            feats[2] = count_dict[user][question]\n            feats[3] = correct_dict[user][question]\n            feats[5] = timestamp - time_dict[user][question]\n\n    return feats\n\n\ndef get_user_feats(trn, val, save_dicts=False):\n    # build up the feature dicts row wise on train and compute train features\n    trn, count_dict, correct_dict, time_dict, last_n_dict = calc_dicts_and_add(trn)\n    # add to precomputed train feature dicts and compute inference features\n    val, count_dict, correct_dict, time_dict, last_n_dict = calc_dicts_and_add(val, count_dict, correct_dict,\n                                                                               time_dict, last_n_dict)\n\n    if save_dicts:\n        print(\"Save count dict ...\")\n        with open(configs.count_dict_path, 'wb') as f:\n            pickle.dump(count_dict, f, pickle.HIGHEST_PROTOCOL)\n        print(\"Save correct dict ...\")\n        with open(configs.correct_dict_path, 'wb') as f:\n            pickle.dump(correct_dict, f, pickle.HIGHEST_PROTOCOL)\n        print(\"Save time dict ...\")\n        with open(configs.time_dict_path, 'wb') as f:\n            pickle.dump(time_dict, f, pickle.HIGHEST_PROTOCOL)\n        print(\"Save last n dict ...\")\n        with open(configs.last_n_dict_path, 'wb') as f:\n            pickle.dump(last_n_dict, f, pickle.HIGHEST_PROTOCOL)\n\n    del count_dict, correct_dict, time_dict, last_n_dict\n    gc.collect()\n\n    return trn, val\n\n\ndef calc_feats_from_stats(df, user_feats):\n    # compute correctness means\n    user_feats[:, 1] = user_feats[:, 1] / user_feats[:, 0]\n    user_feats[:, 3] = user_feats[:, 3] / user_feats[:, 2]\n    user_feats[:, 11] = user_feats[:, 11] / user_feats[:, 10]\n    user_feats[:, 22] = user_feats[:, 22] / user_feats[:, 21]\n\n    # assign computed features to new columns in the df\n    df['user_count'] = user_feats[:, 0].astype(np.float32)\n    df['user_correct_mean'] = user_feats[:, 1].astype(np.float32)\n    df['user_question_count'] = user_feats[:, 2].astype(np.float32)\n    user_feats[:, 3][user_feats[:, 3] == -np.inf] = 0\n    df['user_question_correct_mean'] = user_feats[:, 3].astype(np.float32)\n    df['last_time_user'] = user_feats[:, 4].astype(np.float32)\n    df['last_time_question_user'] = user_feats[:, 5].astype(np.float32)\n    df['last_time_user_inter'] = df['last_time_user'].astype(np.float32) - df['last_time_question_user'].astype(\n        np.float32)\n    df['user_last_n_correct'] = user_feats[:, 6].astype(np.float32)\n    df['user_last_n_time'] = user_feats[:, 7].astype(np.float32)\n    df['user_last_n_time2'] = user_feats[:, 8].astype(np.float32)\n    df['user_last_n_time3'] = user_feats[:, 9].astype(np.float32)\n    df['user_part_count'] = user_feats[:, 10].astype(np.float32)\n    df['user_part_correct_mean'] = user_feats[:, 11].astype(np.float32)\n    df['user_part1_mean'] = user_feats[:, 12].astype(np.float32)\n    df['user_part2_mean'] = user_feats[:, 13].astype(np.float32)\n    df['user_part3_mean'] = user_feats[:, 14].astype(np.float32)\n    df['user_part4_mean'] = user_feats[:, 15].astype(np.float32)\n    df['user_part5_mean'] = user_feats[:, 16].astype(np.float32)\n    df['user_part6_mean'] = user_feats[:, 17].astype(np.float32)\n    df['user_part7_mean'] = user_feats[:, 18].astype(np.float32)\n    df['task_container_eq1'] = user_feats[:, 19].astype(np.float32)\n    df['task_container_eq2'] = user_feats[:, 20].astype(np.float32)\n    df['user_hardness_count'] = user_feats[:, 21].astype(np.float32)\n    df['user_hardness_mean'] = user_feats[:, 22].astype(np.float32)\n    df['user_hardness_inter'] = df['user_hardness_mean'] - df['content_id_target_mean']\n\n    return df\n\n\ndef calc_dicts_and_add(df, count_dict=None, correct_dict=None, time_dict=None, last_n_dict=None):\n    # init empty dicts if nothing is provided else use precomputed dicts\n    # count_dict: user specific counts for questions, parts, question difficulty etc.\n    # correct_dict: user specific counts of correct answers for questions, parts, question difficulty etc.\n    # time dict: last time we saw the user, last time user answered this question etc.\n    # last n dict: last n time user correct, last n time user question correct, last n time seen user etc.\n    if not count_dict:\n        count_dict = {}\n    if not correct_dict:\n        correct_dict = {}\n    if not time_dict:\n        time_dict = {}\n    if not last_n_dict:\n        last_n_dict = {}\n\n    # init numpy storage for all features and create row iterator\n    user_feats = np.full((len(df), 23), fill_value=np.nan, dtype=np.float32)\n    prev_row = None\n    feat_iterator = df[['user_id', 'content_id', 'timestamp', 'part', 'task_container_id', 'question_hardness',\n                        'answered_correctly']].values\n    del df['timestamp']\n    del df['user_id']\n    del df['row_id']\n\n    for row_id, curr_row in enumerate(tqdm(feat_iterator)):\n        if prev_row is not None:\n            # increment user information incrementally with newly gained information (previous row)\n            count_dict, correct_dict, time_dict, last_n_dict = update_dicts(prev_row, count_dict, correct_dict,\n                                                                            time_dict, last_n_dict)\n\n        # obtain feature values for current row\n        user_row_values = get_row_values(curr_row, count_dict, correct_dict, time_dict, last_n_dict)\n        user_feats[row_id] = user_row_values\n\n        prev_row = curr_row\n\n    # calculate and add features from preprocessed state dicts to data\n    del feat_iterator\n    df = calc_feats_from_stats(df, user_feats)\n\n    return df, count_dict, correct_dict, time_dict, last_n_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# methods to prepare and merge the meta file to the main data\n\ndef prepare_questions():\n    questions = pd.read_csv(questions_file,\n                            dtype={\n                                'question_id': 'int16',\n                                'bundle_id': 'int16',\n                                'correct_answer': 'int8',\n                                'part': 'int8'\n                            })\n\n    questions.drop(['bundle_id', 'correct_answer'], axis=1, inplace=True)\n    questions.rename(columns={'question_id': 'content_id'}, inplace=True)\n    questions['content_type_id'] = 0\n\n    return questions\n\ndef prepare_lectures():\n    lectures = pd.read_csv(lectures_file,\n                           dtype={\n                               'lecture_id': 'int16',\n                               'tag': 'int16',\n                               'part': 'int8'\n                           })\n\n    lectures.drop(['type_of'], axis=1, inplace=True)\n    lectures.rename(columns={'lecture_id': 'content_id', 'tag': 'tags'}, inplace=True)\n    lectures['content_type_id'] = 1\n\n    return lectures\n\ndef prepare_meta_feats():\n    questions = prepare_questions()\n    lectures = prepare_lectures()\n    meta = questions.append(lectures)\n    \n    return meta","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def replace_bools(df):\n    df.loc[:, 'prior_question_had_explanation'] = df.loc[:, 'prior_question_had_explanation'].map(\n        {False: 0, True: 1}\n    )\n    df['prior_question_had_explanation'] = df['prior_question_had_explanation'].astype(np.float16)\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def merge_feat_val(key_feat, feat_dict):\n    add_feat = np.zeros((len(key_feat)))\n    key_feat = key_feat.values\n\n    for row_id, row in enumerate(key_feat):\n        key = key_feat[row_id]\n\n        if key in feat_dict:\n            add_feat[row_id] = feat_dict[key]\n        else:\n            add_feat[row_id] = -1\n\n    return add_feat","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_answer_feats(row, ans_dict):\n    feats = np.zeros(4)\n    question = int(row[1])\n    \n    feats[0] = ans_dict[question][0]\n    feats[1] = ans_dict[question][1]\n    feats[2] = ans_dict[question][2]\n    feats[3] = ans_dict[question][3]\n    \n    return feats\n\ndef add_answer_feats(df, ans_feats):\n    df['answer1'] = ans_feats[:, 0].astype(np.float32)\n    df['answer2'] = ans_feats[:, 1].astype(np.float32)\n    df['answer3'] = ans_feats[:, 2].astype(np.float32)\n    df['answer4'] = ans_feats[:, 3].astype(np.float32)\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"meta_features = prepare_meta_feats()\nenv = riiideducation.make_env()\nset_predict = env.predict\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# init empty feature dictionaries which will be build up on the fly during inference\ncount_dict = {}\ncorrect_dict = {}\ntime_dict = {}\nlast_n_dict = {}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# row train iterator for feature engineering\ntrain = train[['user_id', 'content_id', 'timestamp', 'part', 'task_container_id', 'question_hardness', TARGET]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nprev_test_df = None\nfor (test_df, sample_prediction_df) in iter_test:\n    if prev_test_df is not None:\n        prev_test_df[TARGET] = eval(test_df[\"prior_group_answers_correct\"].iloc[0])\n        for row_id, prev_row in enumerate(prev_test_df[['user_id', 'content_id', 'timestamp', 'part', 'task_container_id', 'question_hardness', TARGET, 'content_type_id']].values):\n            if prev_row[7]==0: # only if not lecture\n                # update state dicts\n                count_dict, correct_dict, time_dict, last_n_dict = update_dicts(prev_row, count_dict, correct_dict, time_dict, last_n_dict)\n    \n    test_df = pd.merge(test_df, meta_features, on=['content_type_id', 'content_id'], how='left') # merge meta data\n    test_df['content_id_target_mean'] = merge_feat_val(test_df['content_id'], content_dict) # mean encode questions\n    test_df = get_q_hardness_bins(test_df) # get content difficulty bins\n    prev_test_df = test_df.copy()  # copy for updates in next iteration\n    test_df = test_df.sort_values(['user_id','timestamp'], ascending=False)\n    test_df = replace_bools(test_df)\n    test_df = test_df[test_df['content_type_id'] == 0].reset_index(drop=True)  # remove lectures\n    \n    # init empty numpy arrays --> storage for row features\n    user_feats = np.full((len(test_df), 23), dtype=np.float32, fill_value=np.nan)\n    answer_feats = np.zeros((len(test_df), 4), dtype=np.float32)\n    \n    # get features for each row\n    for row_id, curr_row in enumerate(test_df[['user_id', 'content_id', 'timestamp', 'part', 'task_container_id', 'question_hardness']].values):\n        user = curr_row[0]\n        \n        # update dicts on the fly if we encounter a test user which is seen in the train set\n        # check for each row if we have already queried the train for the current user\n        if user not in count_dict:\n            if user in user_idx: # we have not queried train yet so check if we even have the user in train\n                # query train data for seen test user and update dictionaries with the train data\n                for train_row_id, train_curr_row in enumerate(train[train.user_id==user].values):\n                    count_dict, correct_dict, time_dict, last_n_dict = update_dicts(train_curr_row, count_dict, correct_dict, time_dict, last_n_dict)\n        \n        # get features values and fill numpy arrays\n        user_feats[row_id] = get_row_values(curr_row, count_dict, correct_dict, time_dict, last_n_dict)\n        answer_feats[row_id] = get_answer_feats(curr_row, answer_dist_dict)\n    \n    # calc ratios and assign computed features to new columns in the df\n    test_df = calc_feats_from_stats(test_df, user_feats)\n    test_df = add_answer_feats(test_df, answer_feats)\n    \n    test_preds = model.predict(test_df[feats])\n    test_df[TARGET] = test_preds\n    set_predict(test_df[['row_id', TARGET]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}