{"cells":[{"metadata":{},"cell_type":"markdown","source":"I was a bit late with this Solution. My 1st submission was on the last day of Competition. So I wasn't able to fix all bugs with 5 attempts.\nNevertheless Solution is not a fork of any other Kernel (ideas/implementation is 100% independent of other works. Model was submitted before Competition ended).\nFinally I was able to see the Scores as a Late Submission results:\n* public - 0.807 (which would be 33-rd position in leaderbord)\n* private - 0.808 (which would be 35-th position in leaderbord)\nTake into account that this was my 1st submission and not all bugs fixed and I didn't have time to improve the score by implementing all ideas that I had.\nAlso sorry that code is awfull & unreadable. (I was hurrying up to submit).\n\nSome Details:\nModel is Autoregressive Transformer (Encoder only) Architecture (512 dimensional Embeddings, 8 heads, 6 layers, No Positional Encoding (but instead have learned bundle positions Embedding))\nEmbeddings that are fed (as a sum) at input layer.\n\nMaximum Sequence Length is 512. But each Question consumes 2 positions in this sequence:\n* 1st one is for a prediction which doesn't contain any info that if from future (not to leak info for prediction).\n* 2nd one is same Embeddings + Embeddings of the Question answering results which doesn't affect predictions (since they are done at previous positions) but will be available to next Questions. (Here there is a bug since I realized that same bundle Questions can access outcomes of previous Questions from same Bundle during train time, so I expect that score could be improved if fix future masking in attention properly)\n\nDescriptions of Embeddings (features):\n* question_postfactumness_embedding (3 Categories) - Since each Question consumes 2 positions in sequence this Embedding is used to distinguish them (Category 1 - this position is for prediction, Category 2 - this position contains additional info from future to be used for next Questions, Special Category 0 - means that this Embedding info is irrelevant (used for positions past the last Question))\n* question_pre_lecture_length_idxs_embedding (32 Categories) - means if prior to answering this question user watched a Lecture then this is the time-lenth code of the watching time (31 gradations + 0th meaning there was no Lecture)\n* question_embedding (13524 Categories) - Question index (13523 possible values) + special No Question Category\n* day_part_embedding (240 Categories) - This is Question timestamp modulus by day length in milliseconds (86400000) then divided by 360000 to form an integer from 0 to 239 (here I forgot adding special No day_part Category to be consistent with other Embeddings but it doesn't affect the score though)\n* local_bundle_ids_embedding (257 Categories) - this is logical bundle offset in the Questions sequence (example: 1, 2, 3, 3, 3, 4, ... - where offsets may be repeating if several Questions reside same Bundle). Special 0 Category means - no position.\n* prior_question_elapsed_times_idxs_embedding (64 Categories) - a code for prior_question_elapsed_time (63 gradations + 0th meaning there is no prior_question_elapsed_time)\n* question_from_prev_timestamps_idxs_embedding (32 Categories) - a code for prior_question_elapsed_time which is the time from previous interaction timestamp (31 gradations + 0th meaning there is no prior interaction)\n* correct_answer_embedding (5 Categories) - correct answer id (1, 2, 3, 4) plus special No Answer Category\n* post_question_elapsed_times_idxs_embedding (64 Categories) - same as a code for prior_question_elapsed_time but for current Question. This Embedding is not available while doing prediction for Question but available for further Questions\n* post_question_has_explanation_idxs_embedding (3 Categories) - means if question had expanation (0 - Explanation is irrelevant, 1 - No, 2 - Yes). This Embedding is not available while doing prediction for Question but available for further Questions\n* question_was_answered_correctly_idxs_embedding (3 Categories) - means if question was answered correctly (0 - Answer is irrelevant, 1 - No, 2 - Yes). This Embedding is not available while doing prediction for Question but available for further Questions\n\n\nIf You are interested in novel AI techniques I do brief overviews of them at https://twitter.com/AiParticles"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%%time\n\nimport pandas as pd\nimport numpy as np\n\nprint('Starting...')\ntrain_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', usecols=['timestamp', 'user_id', 'content_id', 'content_type_id', 'task_container_id', 'user_answer', 'prior_question_elapsed_time', 'prior_question_had_explanation',\n                                                   \n                                                   \n                                                   'answered_correctly',\n                                                   ],  #low_memory=False, #nrows=10**5, \n                       dtype={'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n                              'task_container_id': 'int16', 'user_answer': 'int8', 'prior_question_elapsed_time': 'float32', \n                              'prior_question_had_explanation': 'boolean',\n                              \n                              \n                              'answered_correctly': 'int8',\n                             }\n#                        dtype={'row_id': 'int32', 'timestamp': 'int32', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n#                               'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n#                               'prior_question_had_explanation': 'boolean',\n#                              }\n                       #,nrows=10000\n                      )\n#train_df.drop(['row_id', 'answered_correctly'], axis=1, inplace=True)\nprint('Sorting...')\n#train_df.sort_values([#'user_id',\n#    'timestamp'], ascending=True).reset_index(drop=True)\nprint('Filling...')\ntrain_df['prior_question_had_explanation'] = train_df['prior_question_had_explanation'].astype(np.float).fillna(-1).astype(np.int8)\nprint('Filling...')\ntrain_df['prior_question_elapsed_time'] = train_df['prior_question_elapsed_time'].fillna(-1000)\nprint(train_df.head())\n#print(train_df['prior_question_had_explanation'])\nprint('Done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_df.to_pickle('./train_df.pickle')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import pandas as pd\n# import numpy as np\n# import gc\n# gc.collect()\n\n# train_df = pd.read_pickle('./train_df.pickle')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimport gc\ngc.collect()\ndef compose_single_user_data(r):\n    timestamps = r['timestamp'].values\n    prev_timestamps = np.pad(timestamps[:-1], (1, 0), 'constant', constant_values=0).astype(np.int64)\n#     timestamp_phases = np.float64(timestamps % 86400000) * (np.float64(2) * np.pi / 86400000)\n#     timestamp_sins = np.sin(timestamp_phases).astype(np.float32)\n#     timestamp_coss = np.cos(timestamp_phases).astype(np.float32)\n#     timestamp_cumul_sins = np.cumsum(timestamp_sins)\n#     timestamp_cumul_coss = np.cumsum(timestamp_coss)\n    #timestamp_phases = np.int32((timestamps % 86400000) * (np.float32(6) / 86400000))\n    #timestamp_bins = np.zeros((len(timestamp_phases), 6), dtype=np.int32)\n    #timestamp_bins[np.arange(len(timestamp_phases)), timestamp_phases] = 1\n    #timestamp_bins_cumuls = np.cumsum(timestamp_bins, axis=0)\n    content_ids = r['content_id'].values\n    content_type_ids = r['content_type_id'].values\n\n    is_question = (content_type_ids == 0)\n    question_timestamps = timestamps[is_question]\n    question_prev_timestamps = prev_timestamps[is_question]\n#     question_timestamp_cumul_sins = timestamp_cumul_sins[is_question]\n#     question_timestamp_cumul_coss = timestamp_cumul_coss[is_question]\n    \n    #question_timestamp_bins_cumuls = timestamp_bins_cumuls[is_question]\n    question_ids = content_ids[is_question]\n    question_task_container_ids = r['task_container_id'].values[is_question]\n    user_answers = r['user_answer'].values[is_question]\n    prior_question_elapsed_times = r['prior_question_elapsed_time'].values[is_question]\n    prior_question_had_explanation_idxs = r['prior_question_had_explanation'].values[is_question] + 1\n    answered_correctly_idxs = r['answered_correctly'].values[is_question] + 1\n\n    prev_was_lecture = np.pad(content_type_ids[:-1], (1, 0), 'constant', constant_values=0).astype(np.int8)\n    prev_prev_timestamps = np.pad(prev_timestamps[:-1], (1, 0), 'constant', constant_values=0).astype(np.int64)\n    question_prev_was_lecture = prev_was_lecture[is_question]\n    question_prev_prev_timestamps = prev_prev_timestamps[is_question]\n    \n#    is_lecture = (content_type_ids == 1)\n#    lecture_timestamps = timestamps[is_lecture]\n#    lecture_ids = content_ids[is_lecture]\n    \n    d = {#'user_id': r['user_id'].iloc[0],\n        #'questions_count': len(question_timestamps),\n        'question_timestamps': question_timestamps,\n        'question_prev_timestamps': question_prev_timestamps,\n        #'question_timestamp_cumul_sins': question_timestamp_cumul_sins,\n        #'question_timestamp_cumul_coss': question_timestamp_cumul_coss,\n        #'question_timestamp_bins_cumuls': question_timestamp_bins_cumuls,\n        'question_ids': question_ids,\n        'question_task_container_ids': question_task_container_ids,\n        'user_answers': user_answers,\n        'prior_question_elapsed_times': prior_question_elapsed_times,\n        'prior_question_had_explanation_idxs': prior_question_had_explanation_idxs,\n        'answered_correctly_idxs': answered_correctly_idxs,\n\n        #'lectures_count': len(lecture_timestamps),\n        #'lecture_timestamps': lecture_timestamps,\n        #'lecture_ids': lecture_ids,\n        'question_prev_was_lecture': question_prev_was_lecture,\n        'question_prev_prev_timestamps': question_prev_prev_timestamps,\n    }\n    return d\nprint('Grouping...')\ntrain_df = train_df.groupby('user_id')\nprint('Applying...')\ntrain_df = train_df.apply(compose_single_user_data)\nprint('Done')\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_rows',100)\ntrain_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_idxs_by_ids = dict(zip(train_df.index, range(1, len(train_df) + 1)))\nuser_ids_by_idxs = np.array([-1] + list(train_df.index), dtype=np.int32)\nuser_data_by_idxs = [None] + list(train_df)\ndel train_df\nuser_data_questions_counts_by_idxs = [len(user_data['question_ids']) if user_data is not None else -1 for user_data in user_data_by_idxs]\n#user_data_lectures_counts_by_idxs = [user_data['lectures_count'] if user_data is not None else -1 for user_data in user_data_by_idxs]\n\n#idx = 10\n#user_id = train_df.index[idx]\nuser_id = 115\nuser_idx = user_idxs_by_ids[user_id]\nprint(f'user_id={user_id} user_idx={user_idx} user_ids_by_idxs[user_idxs_by_ids[user_id]]={user_ids_by_idxs[user_idxs_by_ids[user_id]]}')\nuser_data = user_data_by_idxs[user_idx]\nprint(f'user_data={str(user_data)[:1000]}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import pickle\n\n# filehandler = open(\"./train.obj\",\"wb\")\n# pickle.dump([user_idxs_by_ids, user_ids_by_idxs, user_data_by_idxs, user_data_questions_counts_by_idxs], filehandler)\n# filehandler.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import pandas as pd\n# import numpy as np\n# import gc\n# gc.collect()\n\n# import pickle\n# file = open(\"./train.obj\",'rb')\n# user_idxs_by_ids, user_ids_by_idxs, user_data_by_idxs, user_data_questions_counts_by_idxs = pickle.load(file)\n# file.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv',\n    dtype={'question_id': 'int16', 'bundle_id': 'int32', 'correct_answer': 'int8', 'part': 'int8', 'tags': 'object'})\nquestions_df.fillna('', inplace=True)\ndef extract_tags_list(x):\n    tag_idxs = [0, 0, 0, 0, 0, 0]\n    if type(x) is not str:\n        print('not str:', type(x), x)\n    else:\n        if x != '':\n            for i, part in enumerate(x.split(' ')):\n                tag = int(part)\n                if tag < 0 or tag > 187:\n                    print('bad:', tag, x)\n                tag_idxs[i] = tag + 1\n    #if len(tags) > 1:\n    #    tags = sorted(tags)[::-1]\n    return tag_idxs\nquestions_df['tags_list'] = questions_df['tags'].apply(extract_tags_list)\nquestions_df.head(10)\n#print(f\"{questions_df['tags_list'].max()}\")\nprint('unique question_ids:', len(questions_df['question_id'].unique()))\nmin_question_id = questions_df['question_id'].min()\nprint('min question_id:', min_question_id)\nmax_question_id = questions_df['question_id'].max()\nprint('max question_id:', max_question_id)\n\nprint('unique bundle_ids:', len(questions_df['bundle_id'].unique()))\nprint('min bundle_id:', questions_df['bundle_id'].min())\nprint('max bundle_id:', questions_df['bundle_id'].max())\nprint('unique correct_answers:', len(questions_df['correct_answer'].unique()))\nprint('min correct_answer:', questions_df['correct_answer'].min())\nprint('max correct_answer:', questions_df['correct_answer'].max())\nprint('unique parts:', len(questions_df['part'].unique()))\nprint('min part:', questions_df['part'].min())\nprint('max part:', questions_df['part'].max())\n\nquestions = questions_df.to_dict()\nquestion_idxs_by_ids_dict = {}\nquestion_idxs_by_ids = np.zeros((max_question_id+1,), dtype=np.int16)\nfor i, question_id in questions['question_id'].items():\n    question_idxs_by_ids_dict[question_id] = i + 1\n    question_idxs_by_ids[question_id] = i + 1\nquestion_ids_by_idxs = np.array([-1] + list(questions['question_id']), dtype=np.int16)\nquestion_bundle_ids_by_idxs = np.array([-1] + list(questions['bundle_id'].values()), dtype=np.int32)\nquestion_correct_answers_by_idxs = np.array([-1] + list(questions['correct_answer'].values()), dtype=np.int8)\n\ndel questions\ndel questions_df\ngc.collect()\n\nquestion_id = 13522\nquestion_idx = question_idxs_by_ids[question_id]\nprint(f'question_id={question_id} question_idx={question_idx} question_ids_by_idxs[question_idxs_by_ids[question_id]]={question_ids_by_idxs[question_idxs_by_ids[question_id]]}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# user_idxs_sampling_list = []\nnonzero_user_idxs = np.arange(1, len(user_ids_by_idxs), dtype=np.int32)\nprint(f'nonzero_user_idxs: {len(nonzero_user_idxs)}')\n\n# test_set_user_idxs_sampling_list = []\ntest_set_fraction = 0.1\ntest_set_user_idxs = nonzero_user_idxs.copy()\nnp.random.shuffle(test_set_user_idxs)\ntest_set_user_idxs = np.array(sorted(test_set_user_idxs[:int(len(test_set_user_idxs) * test_set_fraction * 2)].tolist()))\ntest_set_user_idxs_set = set(test_set_user_idxs)\nprint(f'test_set_user_idxs: {len(test_set_user_idxs)} {list(test_set_user_idxs)[:100]}')\n\nmax_user_data_questions_count = 0\nfor user_idx in nonzero_user_idxs:\n    user_data_questions_count = user_data_questions_counts_by_idxs[user_idx]\n    if user_data_questions_count > max_user_data_questions_count:\n        max_user_data_questions_count = user_data_questions_count\nprint(f'max_user_data_questions_count: {max_user_data_questions_count}')\n\ndef sample_test(max_seq_len):\n    test_set_start_positions_by_user_idxs = [-1]\n    test_user_idxs = []\n    test_start_poss = []\n    test_lengths = []\n    test_roc_auc_start_poss = []\n    for user_idx in nonzero_user_idxs:\n        user_data_questions_count = user_data_questions_counts_by_idxs[user_idx]\n        if user_idx in test_set_user_idxs_set:\n            test_set_start_position_by_user_idx = np.random.randint(low=0, high=user_data_questions_count)\n        else:\n            test_set_start_position_by_user_idx = user_data_questions_count\n        test_set_start_positions_by_user_idxs.append(test_set_start_position_by_user_idx)\n\n        if user_idx in test_set_user_idxs_set:\n            excess = (user_data_questions_count - test_set_start_position_by_user_idx) % max_seq_len\n            if excess > 0:\n                test_roc_auc_start_pos = test_set_start_position_by_user_idx\n                start_pos = test_roc_auc_start_pos + excess - max_seq_len\n                if start_pos < 0:\n                    length = max_seq_len - (-start_pos)\n                    start_pos = 0\n                else:\n                    length = max_seq_len\n                test_user_idxs.append(user_idx)\n                test_start_poss.append(start_pos)\n                test_roc_auc_start_poss.append(test_roc_auc_start_pos)\n                assert length > 0\n                test_lengths.append(length)\n            while start_pos < user_data_questions_count:\n                length = min(user_data_questions_count - start_pos, max_seq_len)\n                test_user_idxs.append(user_idx)\n                test_start_poss.append(start_pos)\n                test_roc_auc_start_poss.append(start_pos)\n                assert length > 0\n                test_lengths.append(length)\n                start_pos += length\n    return test_set_start_positions_by_user_idxs, test_user_idxs, test_start_poss, test_lengths, test_roc_auc_start_poss\ntest_set_start_positions_by_user_idxs, test_user_idxs, test_start_poss, test_lengths, test_roc_auc_start_poss = sample_test(200)\nprint(f'test_user_idxs: {len(test_user_idxs)} test_start_poss: {len(test_start_poss)} test_roc_auc_start_poss: {len(test_roc_auc_start_poss)} test_lengths: {len(test_lengths)}')\n\ndef sample_train(max_seq_len):\n    train_user_idxs = []\n    train_start_poss = []\n    train_lengths = []\n    train_samples_counts_by_user_idxs = [-1]\n    for user_idx in nonzero_user_idxs:\n        user_data_questions_count = user_data_questions_counts_by_idxs[user_idx]\n        test_set_start_position_by_user_idx = test_set_start_positions_by_user_idxs[user_idx]\n\n        start_pos = 0\n        train_samples_count_by_user_idx = 0\n        small_amount_of_data_for_user = (test_set_start_position_by_user_idx <= max_seq_len)\n        while start_pos < test_set_start_position_by_user_idx:\n            if (start_pos == 0):# and (not small_amount_of_data_for_user):\n                high = min(test_set_start_position_by_user_idx, max_seq_len)\n                if high == 1:\n                    length = 1\n                else:\n                    length = np.random.randint(low=1, high=high)\n            else:\n                length = min(test_set_start_position_by_user_idx - start_pos, max_seq_len)\n            if (length >= max_seq_len * 0.75) or (np.random.randint(low=0, high=100) >= 90):\n                train_samples_count_by_user_idx += 1\n                train_user_idxs.append(user_idx)\n                train_start_poss.append(start_pos)\n                assert length > 0\n                train_lengths.append(length)\n            start_pos += length\n        train_samples_counts_by_user_idxs.append(train_samples_count_by_user_idx)\n    train_indexes = np.arange(len(train_lengths))\n    np.random.shuffle(train_indexes)\n    return train_user_idxs, train_start_poss, train_lengths, train_indexes, train_samples_counts_by_user_idxs\n\n\ntrain_user_idxs, train_start_poss, train_lengths, train_indexes, train_samples_counts_by_user_idxs = sample_train(200)\nusers_with_much_data_count = 0\nfor train_samples_count_by_user_idx in train_samples_counts_by_user_idxs:\n    if train_samples_count_by_user_idx >= 5:\n        users_with_much_data_count += 1\nprint(f'train_user_idxs: {len(train_user_idxs)} train_start_poss: {len(train_start_poss)} train_lengths: {len(train_lengths)} train_indexes: {len(train_indexes)} users_with_much_data_count: {users_with_much_data_count}')\ndel train_user_idxs\ndel train_start_poss\ndel train_lengths\ndel train_indexes\ndel train_samples_counts_by_user_idxs\n\ndel test_set_start_positions_by_user_idxs\ndel test_user_idxs\ndel test_start_poss\ndel test_lengths\ndel test_roc_auc_start_poss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nimport time \n\n\ntrain_samples_counts_by_user_idxs = None\nclass RiiidDataset(Dataset):\n    def __init__(self, is_train_set=True, max_seq_len=128):\n        super(RiiidDataset, self).__init__()\n        self.is_train_set = is_train_set\n        self.max_seq_len = max_seq_len\n        \n    def prepare(self):\n        if self.is_train_set:\n            global train_samples_counts_by_user_idxs\n            self.set_user_idxs, self.set_start_poss, self.set_lengths, self.set_indexes, train_samples_counts_by_user_idxs_ = sample_train(self.max_seq_len)\n            if train_samples_counts_by_user_idxs is None:\n                train_samples_counts_by_user_idxs = train_samples_counts_by_user_idxs_\n            self.set_roc_auc_start_poss = self.set_start_poss\n        else:\n            self.set_user_idxs, self.set_start_poss, self.set_lengths, self.set_roc_auc_start_poss = test_user_idxs, test_start_poss, test_lengths, test_roc_auc_start_poss\n        \n    def __len__(self):\n        samples_count = len(self.set_lengths)                                                 #//500\n        return samples_count\n\n    def __getitem__(self, index):\n        if index < 0:\n            user_idx = -index\n            user_data = user_data_by_idxs[user_idx]\n            seq_len = len(user_data['question_ids'])\n            if seq_len >= self.max_seq_len:\n                start_pos = seq_len - self.max_seq_len\n                seq_len = self.max_seq_len\n            else:\n                start_pos = 0\n            set_roc_auc_start_poss = start_pos\n            \n            user_data_test_start_pos = 0\n        else:\n            if self.is_train_set:\n                index = self.set_indexes[index]\n            user_idx = self.set_user_idxs[index]\n            user_data = user_data_by_idxs[user_idx]\n            user_data_test_start_pos = test_set_start_positions_by_user_idxs[user_idx]\n            seq_len = self.set_lengths[index]\n            start_pos = self.set_start_poss[index]\n            set_roc_auc_start_poss = self.set_roc_auc_start_poss[index]\n        #print(f'start_pos={start_pos}, seq_len={seq_len}, seq_test_start_pos={seq_test_start_pos}, user_data_len={user_data_len}, user_data_test_start_pos={user_data_test_start_pos}, max_train_prefix_len={max_train_prefix_len}, min_start_pos={min_start_pos}, max_start_pos={max_start_pos}')\n        \n\n#         user_idxs_ = np.zeros(self.max_seq_len, dtype=np.int32)\n#         much_train_data_for_user = int(train_samples_counts_by_user_idxs[user_idx] >= 5)\n#         if self.is_train_set:\n#             user_idxs_[:seq_len] = np.full((seq_len,), fill_value=user_idx * np.random.randint(low=0, high=2) * much_train_data_for_user)\n#         else:\n#             if much_train_data_for_user:\n#                 user_idxs_[:seq_len] = np.full((seq_len,), fill_value=user_idx)\n                \n        \n        user_data_seq = user_data['question_timestamps']\n        question_timestamps = np.zeros(self.max_seq_len, dtype=user_data_seq.dtype)\n        question_timestamps_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        question_timestamps[:seq_len] = question_timestamps_without_padding\n        \n        user_data_seq = user_data['question_prev_timestamps']\n        question_from_prev_timestamps_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        question_prev_timestamps_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        question_from_prev_timestamps_idxs[:seq_len] = np.minimum(np.log(np.maximum(question_timestamps_without_padding - question_prev_timestamps_without_padding, 0) / 1000.0 + 2.0) / np.log(1.5), 31).astype(np.int8)\n        if start_pos == 0:\n            question_from_prev_timestamps_idxs[0] = 0\n        \n        user_data_seq = user_data['question_ids']\n        question_idxs = np.zeros(self.max_seq_len, dtype=np.int32)\n        question_idxs_without_padding = question_idxs_by_ids[user_data_seq[start_pos:start_pos+seq_len]]\n        question_idxs[:seq_len] = question_idxs_without_padding\n        \n#         question_parts = np.zeros(self.max_seq_len, dtype=np.int8)\n#         question_parts[:seq_len] = question_parts_by_idxs[question_idxs_without_padding]\n        \n#         question_tag_idx0s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx0s[:seq_len] = question_tag_idx0s_by_idxs[question_idxs_without_padding]\n#         question_tag_idx1s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx1s[:seq_len] = question_tag_idx1s_by_idxs[question_idxs_without_padding]\n#         question_tag_idx2s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx2s[:seq_len] = question_tag_idx2s_by_idxs[question_idxs_without_padding]\n#         question_tag_idx3s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx3s[:seq_len] = question_tag_idx3s_by_idxs[question_idxs_without_padding]\n#         question_tag_idx4s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx4s[:seq_len] = question_tag_idx4s_by_idxs[question_idxs_without_padding]\n#         question_tag_idx5s = np.zeros(self.max_seq_len, dtype=np.int16)\n#         question_tag_idx5s[:seq_len] = question_tag_idx5s_by_idxs[question_idxs_without_padding]\n      \n        \n        #user_data_seq = user_data['lecture_ids']\n        #lecture_idxs_without_padding = lecture_idxs_by_ids[user_data_seq]\n        #lecture_idxs = -1\n        user_data_seq = user_data['question_prev_was_lecture']\n        #question_from_prev_timestamps_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        question_prev_was_lecture_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        user_data_seq = user_data['question_prev_prev_timestamps']\n        question_prev_prev_timestamps_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        question_pre_lecture_length_idxs_without_padding = np.minimum(np.log(np.maximum(question_prev_timestamps_without_padding - question_prev_prev_timestamps_without_padding, 0) / 1000.0 + 2.0) / np.log(1.5), 31).astype(np.int8)\n        question_pre_lecture_length_idxs_without_padding = question_pre_lecture_length_idxs_without_padding * question_prev_was_lecture_without_padding\n        question_pre_lecture_length_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        question_pre_lecture_length_idxs[:seq_len] = question_pre_lecture_length_idxs_without_padding\n        if start_pos == 0:\n            question_pre_lecture_length_idxs[0:2] = 0\n        elif start_pos == 1:\n            question_pre_lecture_length_idxs[0:1] = 0\n\n\n        #lecture_parts = np.zeros(self.max_seq_len, dtype=np.uint8)\n        #lecture_parts[:seq_len] = lecture_parts_by_idxs[lecture_idxs_without_padding]\n\n        user_data_seq = user_data['question_task_container_ids']\n        question_task_container_ids = np.zeros(self.max_seq_len, dtype=user_data_seq.dtype)\n        question_task_container_ids_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        is_bundle_id_changed = (question_task_container_ids_without_padding != np.pad(question_task_container_ids_without_padding[:-1], (1, 0), 'constant', constant_values=-1)).astype(np.int32)\n        local_bundle_ids_without_padding = np.cumsum(is_bundle_id_changed)\n        local_bundle_ids = np.zeros(self.max_seq_len, dtype=np.int32)\n        local_bundle_ids[:seq_len] = local_bundle_ids_without_padding\n        last_bundle_mask = np.zeros(self.max_seq_len, dtype=np.int8)\n        last_bundle_id = local_bundle_ids_without_padding[-1]\n        k = len(local_bundle_ids_without_padding) - 1\n        while (k >= 0) and (local_bundle_ids_without_padding[k] == last_bundle_id):\n            last_bundle_mask[k] = 1\n            k -= 1\n\n        user_data_seq = user_data['prior_question_elapsed_times']\n        prior_question_elapsed_times_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        prior_question_elapsed_times_idxs_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        prior_question_elapsed_times_idxs_without_padding = np.power(np.minimum(prior_question_elapsed_times_idxs_without_padding // 1000, 300) + 1, 0.726).astype(np.int8)\n        prior_question_elapsed_times_idxs[:seq_len] = prior_question_elapsed_times_idxs_without_padding\n        bundle_elapsed_times_idxs = np.zeros(self.max_seq_len + 2, dtype=np.int8)\n        bundle_elapsed_times_idxs[local_bundle_ids_without_padding - 1 + 1] = prior_question_elapsed_times_idxs_without_padding\n        bundle_elapsed_times_idxs = bundle_elapsed_times_idxs[1:]\n        #print(f'mm: {local_bundle_ids_without_padding.min()} {local_bundle_ids_without_padding.max()}')\n        question_elapsed_times_idxs_without_padding = bundle_elapsed_times_idxs[local_bundle_ids_without_padding]\n        post_question_elapsed_times_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        post_question_elapsed_times_idxs[:seq_len] = question_elapsed_times_idxs_without_padding\n        \n        user_data_seq = user_data['prior_question_had_explanation_idxs']\n        prior_question_had_explanation_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        prior_question_had_explanation_idxs_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        prior_question_had_explanation_idxs[:seq_len] = prior_question_had_explanation_idxs_without_padding\n        bundle_has_explanation_idxs = np.zeros(self.max_seq_len + 2, dtype=np.int8)\n        bundle_has_explanation_idxs[local_bundle_ids_without_padding - 1 + 1] = prior_question_had_explanation_idxs_without_padding\n        bundle_has_explanation_idxs = bundle_has_explanation_idxs[1:]\n        question_has_explanation_idxs_without_padding = bundle_has_explanation_idxs[local_bundle_ids_without_padding]\n        post_question_has_explanation_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        post_question_has_explanation_idxs[:seq_len] = question_has_explanation_idxs_without_padding\n        \n        \n        user_data_seq = user_data['user_answers']\n        user_answers_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        user_answers_idxs[:seq_len] = user_data_seq[start_pos:start_pos+seq_len] + 1\n        \n        correct_answers_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        correct_answers_idxs[:seq_len] = (question_correct_answers_by_idxs + 1)[question_idxs_without_padding]\n\n        user_data_seq = user_data['answered_correctly_idxs']\n        answered_correctly_idxs = np.zeros(self.max_seq_len, dtype=user_data_seq.dtype)\n        answered_correctly_idxs_without_padding = user_data_seq[start_pos:start_pos+seq_len]\n        answered_correctly_idxs[:seq_len] = answered_correctly_idxs_without_padding\n        \n        prior_question_was_answered_correctly_idxs = np.zeros(self.max_seq_len, dtype=np.int8)\n        prior_question_was_answered_correctly_idxs[1:seq_len] = answered_correctly_idxs_without_padding[:-1]\n\n        mask = np.zeros(self.max_seq_len, dtype=np.int8)\n        mask[:seq_len] = 1\n\n        if self.is_train_set:\n            participates_in_roc_auc = mask\n        else:\n            participates_in_roc_auc = np.zeros(self.max_seq_len, dtype=np.int8)\n            if set_roc_auc_start_poss - start_pos < seq_len - 1:\n                participates_in_roc_auc[set_roc_auc_start_poss - start_pos:seq_len - 1] = 1 ####### hardcoded a patch (different from train variant)\n        \n        \n        return (last_bundle_mask,#user_idxs_,\n                question_timestamps,\n                \n                \n                question_pre_lecture_length_idxs,\n                question_idxs,\n                \n                \n#                 question_parts,\n#                 question_tag_idx0s,\n#                 question_tag_idx1s,\n#                 question_tag_idx2s,\n#                 question_tag_idx3s,\n#                 question_tag_idx4s,\n#                 question_tag_idx5s,\n                local_bundle_ids, prior_question_elapsed_times_idxs, prior_question_had_explanation_idxs,\n                question_from_prev_timestamps_idxs,\n                prior_question_was_answered_correctly_idxs,\n                post_question_elapsed_times_idxs, post_question_has_explanation_idxs,\n                \n                \n                user_answers_idxs,\n                \n                correct_answers_idxs,\n                answered_correctly_idxs, participates_in_roc_auc, mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f'device: {device}')\n\n\nimport math\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import TransformerEncoder, TransformerEncoderLayer\n\n\ndef _get_activation_fn(activation):\n    if activation == \"relu\":\n        return F.relu\n    elif activation == \"gelu\":\n        return F.gelu\n\n    raise RuntimeError(\"activation should be relu/gelu, not {}\".format(activation))\n\nfrom typing import Optional\n\nuse_reformer_self_attn = False\nclass TransformerEncoderLayer(nn.Module):\n    r\"\"\"TransformerEncoderLayer is made up of self-attn and feedforward network.\n    This standard encoder layer is based on the paper \"Attention Is All You Need\".\n    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\n    Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in\n    Neural Information Processing Systems, pages 6000-6010. Users may modify or implement\n    in a different way during application.\n\n    Args:\n        d_model: the number of expected features in the input (required).\n        nhead: the number of heads in the multiheadattention models (required).\n        dim_feedforward: the dimension of the feedforward network model (default=2048).\n        dropout: the dropout value (default=0.1).\n        activation: the activation function of intermediate layer, relu or gelu (default=relu).\n\n    Examples::\n        >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)\n        >>> src = torch.rand(10, 32, 512)\n        >>> out = encoder_layer(src)\n    \"\"\"\n\n    def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=\"relu\"):\n        super(TransformerEncoderLayer, self).__init__()\n        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)\n        # Implementation of Feedforward model\n        self.linear1 = nn.Linear(d_model, dim_feedforward)\n        self.dropout = nn.Dropout(dropout)\n        self.linear2 = nn.Linear(dim_feedforward, d_model)\n\n        self.norm1 = nn.LayerNorm(d_model)\n        self.norm2 = nn.LayerNorm(d_model)\n        self.dropout1 = nn.Dropout(dropout)\n        self.dropout2 = nn.Dropout(dropout)\n\n        self.activation = _get_activation_fn(activation)\n\n    def __setstate__(self, state):\n        if 'activation' not in state:\n            state['activation'] = F.relu\n        super(TransformerEncoderLayer, self).__setstate__(state)\n\n    def forward(self, src: torch.Tensor, src_mask: Optional[torch.Tensor] = None, src_key_padding_mask: Optional[torch.Tensor] = None) -> torch.Tensor:\n        r\"\"\"Pass the input through the encoder layer.\n\n        Args:\n            src: the sequence to the encoder layer (required).\n            src_mask: the mask for the src sequence (optional).\n            src_key_padding_mask: the mask for the src keys per batch (optional).\n\n        Shape:\n            see the docs in Transformer class.\n        \"\"\"\n        if use_reformer_self_attn:\n            src2 = self.self_attn(src.transpose(0, 1).contiguous()#, attn_mask=src_mask, key_padding_mask=src_key_padding_mask\n                )\n            src = src + self.dropout1(src2.transpose(0, 1).contiguous())\n        else:\n            src2 = self.self_attn(src, src, src, attn_mask=src_mask,\n                                  key_padding_mask=src_key_padding_mask)[0]\n            src = src + self.dropout1(src2)\n        src = self.norm1(src)\n        src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))\n        src = src + self.dropout2(src2)\n        src = self.norm2(src)\n        return src\n\n\nclass TransformerEncoderOnlyModel(nn.Module):\n\n    def __init__(self, d_model, nhead, dim_feedforward, num_layers, max_seq_len, dropout=0.1, use_performer=True, feature_type='sqr', compute_type='iter', on_gptln=True):\n        super(TransformerEncoderOnlyModel, self).__init__()\n\n        self.d_model = d_model\n        self.nhead = nhead\n        self.max_seq_len = max_seq_len\n        self.model_type = 'Transformer'\n        \n        self.use_performer = use_performer\n        self.feature_type = feature_type\n        self.compute_type = compute_type\n        self.on_gptln = on_gptln\n        \n        \n        #user_idxs_cnt = len(nonzero_user_idxs) + 1\n        #self.user_embedding = nn.Embedding(user_idxs_cnt, d_model)\n        \n        self.day_part_embedding = nn.Embedding(240, d_model)\n\n#         lecture_idxs_cnt = len(lecture_ids_by_idxs)\n#         self.lecture_embedding = nn.Embedding(lecture_idxs_cnt, d_model)\n        self.question_pre_lecture_length_idxs_embedding = nn.Embedding(32, d_model)\n\n        question_idxs_cnt = len(question_ids_by_idxs)\n        self.question_embedding = nn.Embedding(question_idxs_cnt, d_model)\n        \n        self.question_postfactumness_embedding = nn.Embedding(3, d_model)\n        \n#         self.parts_embedding = nn.Embedding(8, d_model)\n#         self.question_tag_idx0s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n#         self.question_tag_idx1s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n#         self.question_tag_idx2s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n#         self.question_tag_idx3s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n#         self.question_tag_idx4s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n#         self.question_tag_idx5s_embedding = nn.Embedding(len(question_tag_idxs), d_model)\n        self.local_bundle_ids_embedding = nn.Embedding(max_seq_len + 1, d_model)\n        self.prior_question_elapsed_times_idxs_embedding = nn.Embedding(64, d_model)\n        self.prior_question_had_explanation_idxs_embedding = nn.Embedding(3, d_model)\n        self.question_from_prev_timestamps_idxs_embedding = nn.Embedding(32, d_model)\n        self.prior_question_was_answered_correctly_idxs_embedding = nn.Embedding(3, d_model)\n        self.post_question_elapsed_times_idxs_embedding = nn.Embedding(64, d_model)\n        self.post_question_has_explanation_idxs_embedding = nn.Embedding(3, d_model)\n        \n        self.question_was_answered_correctly_idxs_embedding = nn.Embedding(3, d_model)\n        self.correct_answer_embedding = nn.Embedding(5, d_model)\n        \n        #self.encoder_bottom = embeddings\n        \n        #self.positional_encoding = PositionalEncoding(d_model, dropout, max_seq_len=max_seq_len)\n        #self.dynamic_positional_encoding = DynamicPositionalEncoding(d_model, dropout)\n        #self.dynamic_positional_encoding = DynamicPositionalEncoding2(d_model, dropout)\n\n        if self.use_performer:\n            vocab_size = None\n            self.transformer_encoder = slim_performer_model.SLiMPerformer(vocab_size, d_model,\n                                             num_layers, dim_feedforward,\n                                             nhead, feature_type,\n                                             compute_type,\n                                             on_gptln)\n        else:\n            encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout, activation='gelu')\n            self.transformer_encoder = TransformerEncoder(encoder_layer, num_layers)\n\n#         encoder_top_linear = nn.Linear(d_model, 3)\n#         encoder_top_activation = nn.Sigmoid()\n#         self.encoder_top = nn.Sequential(\n#             encoder_top_linear,\n#             #encoder_top_activation,\n#         )\n\n        self.encoder_top_linear_correct_answer_idx = nn.Linear(d_model, 1)\n        self.encoder_top_linear_user_answer_idx = nn.Linear(d_model, 5)\n        \n        self.init_weights()\n\n    def generate_square_subsequent_mask(self, seq_len):\n        mask = (torch.triu(torch.ones(seq_len, seq_len)) == 1).transpose(0, 1)\n        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n        return mask\n\n    def init_weights(self):\n        initrange = 0.00001\n        \n        #self.user_embedding.weight.data.uniform_(-initrange, initrange)\n        \n        self.day_part_embedding.weight.data.uniform_(-initrange, initrange)\n        \n        #self.lecture_embedding.weight.data.uniform_(-initrange, initrange)\n        self.question_pre_lecture_length_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.question_embedding.weight.data.uniform_(-initrange, initrange)\n        \n        self.question_postfactumness_embedding.weight.data.uniform_(-initrange, initrange)\n\n        \n#         self.parts_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx0s_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx1s_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx2s_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx3s_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx4s_embedding.weight.data.uniform_(-initrange, initrange)\n#         self.question_tag_idx5s_embedding.weight.data.uniform_(-initrange, initrange)\n        self.local_bundle_ids_embedding.weight.data.uniform_(-initrange, initrange)\n        self.prior_question_elapsed_times_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.prior_question_had_explanation_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.question_from_prev_timestamps_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.prior_question_was_answered_correctly_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.post_question_elapsed_times_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.post_question_has_explanation_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        \n        self.question_was_answered_correctly_idxs_embedding.weight.data.uniform_(-initrange, initrange)\n        self.correct_answer_embedding.weight.data.uniform_(-initrange, initrange)\n        #self.encoder_top_linear.bias.data.zero_()\n        #self.encoder_top_linear.weight.data.uniform_(-initrange, initrange)\n\n    def forward(self,\n                user_idxs,\n                timestamps,\n\n                #lecture_idxs,\n                question_pre_lecture_length_idxs,\n                question_idxs,\n                \n                \n#                 parts,\n#                 question_tag_idx0s,\n#                 question_tag_idx1s,\n#                 question_tag_idx2s,\n#                 question_tag_idx3s,\n#                 question_tag_idx4s,\n#                 question_tag_idx5s,\n                local_bundle_ids,\n                prior_question_elapsed_times_idxs,\n                prior_question_had_explanation_idxs,\n                question_from_prev_timestamps_idxs,\n                prior_question_was_answered_correctly_idxs,\n                post_question_elapsed_times_idxs,\n                post_question_has_explanation_idxs,\n                \n                question_was_answered_correctly_idxs,\n                correct_answers_idxs,\n                \n                mask,\n                \n                attn_mask):\n        \n        #user_embedding = self.user_embedding(user_idxs)\n\n        day_part_embedding = self.day_part_embedding(((timestamps % 86400000) * (240.0 / 86400000)).long())\n        \n        #lecture_embedding = self.lecture_embedding(lecture_idxs)\n        question_pre_lecture_length_idxs_embedding = self.question_pre_lecture_length_idxs_embedding(question_pre_lecture_length_idxs)\n        question_embedding = self.question_embedding(question_idxs)# * math.sqrt(self.d_model)\n        \n        question_postfactumness_embedding_x2 = self.question_postfactumness_embedding(mask.unsqueeze(-1) * torch.tensor([1, 2]).to(device))\n        \n#         parts_embedding = self.parts_embedding(parts)\n#         question_tag_idx0s_embedding = self.question_tag_idx0s_embedding(question_tag_idx0s)\n# #         question_tag_idx1s_embedding = self.question_tag_idx1s_embedding(question_tag_idx1s)\n# #         question_tag_idx2s_embedding = self.question_tag_idx2s_embedding(question_tag_idx2s)\n# #         question_tag_idx3s_embedding = self.question_tag_idx3s_embedding(question_tag_idx3s)\n# #         question_tag_idx4s_embedding = self.question_tag_idx4s_embedding(question_tag_idx4s)\n# #         question_tag_idx5s_embedding = self.question_tag_idx5s_embedding(question_tag_idx5s)\n#         question_tag_idx1s_embedding = self.question_tag_idx0s_embedding(question_tag_idx1s)\n#         question_tag_idx2s_embedding = self.question_tag_idx0s_embedding(question_tag_idx2s)\n#         question_tag_idx3s_embedding = self.question_tag_idx0s_embedding(question_tag_idx3s)\n#         question_tag_idx4s_embedding = self.question_tag_idx0s_embedding(question_tag_idx4s)\n#         question_tag_idx5s_embedding = self.question_tag_idx0s_embedding(question_tag_idx5s)\n        local_bundle_ids_embedding = self.local_bundle_ids_embedding(local_bundle_ids)\n        prior_question_elapsed_times_idxs_embedding = self.prior_question_elapsed_times_idxs_embedding(prior_question_elapsed_times_idxs)\n        prior_question_had_explanation_idxs_embedding = self.prior_question_had_explanation_idxs_embedding(prior_question_had_explanation_idxs)\n        question_from_prev_timestamps_idxs_embedding = self.question_from_prev_timestamps_idxs_embedding(question_from_prev_timestamps_idxs)\n        prior_question_was_answered_correctly_idxs_embedding = self.prior_question_was_answered_correctly_idxs_embedding(prior_question_was_answered_correctly_idxs)\n        post_question_elapsed_times_idxs_embedding = self.post_question_elapsed_times_idxs_embedding(post_question_elapsed_times_idxs)\n        post_question_has_explanation_idxs_embedding = self.post_question_has_explanation_idxs_embedding(post_question_has_explanation_idxs)\n        \n        question_was_answered_correctly_idxs_embedding = self.question_was_answered_correctly_idxs_embedding(question_was_answered_correctly_idxs)\n        correct_answer_embedding = self.correct_answer_embedding(correct_answers_idxs)\n        \n        \n        #positional_encoding = self.positional_encoding(question_idxs)\n        #positional_encoding = self.dynamic_positional_encoding(timestamps)\n        #pos = torch.arange(self.max_seq_len).to(device)\n        #positional_encoding = self.dynamic_positional_encoding(pos, timestamps) * mask.unsqueeze(-1)\n        #positional_encoding = self.positional_encoding(mask)\n        \n        \n        question_related_embeddings = (#user_embedding +\n            #lecture_embedding +\n            question_pre_lecture_length_idxs_embedding\n            + question_embedding\n            \n            + day_part_embedding\n\n\n#             + parts_embedding\n#             + question_tag_idx0s_embedding\n#             + question_tag_idx1s_embedding\n#             + question_tag_idx2s_embedding\n#             + question_tag_idx3s_embedding\n#             + question_tag_idx4s_embedding\n#             + question_tag_idx5s_embedding\n            + local_bundle_ids_embedding\n            #+ prior_question_elapsed_times_idxs_embedding\n            #+ prior_question_had_explanation_idxs_embedding\n            + question_from_prev_timestamps_idxs_embedding\n            #+ prior_question_was_answered_correctly_idxs_embedding\n\n        \n            + correct_answer_embedding\n                        \n            #+  positional_encoding\n        )# * mask.unsqueeze(-1)\n        question_related_embeddings_x2 = question_related_embeddings.unsqueeze(-2) * torch.tensor([[1.], [1.]]).to(device)\n        \n        posts_x2 = (post_question_elapsed_times_idxs_embedding + post_question_has_explanation_idxs_embedding + question_was_answered_correctly_idxs_embedding).unsqueeze(-2) * torch.tensor([[0.], [1.]]).to(device)\n        \n        encoder_input_x2 = question_postfactumness_embedding_x2 + question_related_embeddings_x2 + posts_x2\n        encoder_input = encoder_input_x2.view([encoder_input_x2.size()[0], encoder_input_x2.size()[1] * encoder_input_x2.size()[2], encoder_input_x2.size()[3]])\n        \n        if not self.use_performer:\n            encoder_input = encoder_input.transpose(0, 1)\n            output = self.transformer_encoder(encoder_input, attn_mask)#, src_key_padding_mask=(1-mask).bool())\n        else:\n            output = self.transformer_encoder.full_forward(encoder_input)\n        output0 = self.encoder_top_linear_correct_answer_idx(output)\n        if not self.use_performer:\n            output0 = output0.view([output0.size()[0] // 2, 2, output0.size()[1], output0.size()[2]])[:,0,:,:]\n            output0 = output0.transpose(0, 1).contiguous()\n        else:\n            output0 = output0.view([output0.size()[0], output0.size()[1] // 2, 2, output0.size()[2]])[:,:,0,:]\n        output1 = self.encoder_top_linear_user_answer_idx(output)\n        if not self.use_performer:\n            output1 = output1.view([output1.size()[0] // 2, 2, output1.size()[1], output1.size()[2]])[:,0,:,:]\n            output1 = output1.transpose(0, 1).contiguous()\n        else:\n            output1 = output1.view([output1.size()[0], output1.size()[1] // 2, 2, output1.size()[2]])[:,:,0,:]\n        return output0, output1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from datetime import datetime\n\n\n\n\nfrom torch.optim.lr_scheduler import LambdaLR\nimport math\n\n\nclass CustomBCELoss(nn.Module):\n    def __init__(self):\n        super().__init__()\n        \n    def forward(self, output, target, **kwargs):\n        loss = ((target - 1) * torch.log(output) + (2 - target) * torch.log(1.0 - output)) * (target != 0).float()\n\n        loss = torch.neg(torch.mean(loss))\n\n        return loss\n\n\nWARMUP_EPOCHS = 1.0 / 4.0\nepochs = 5\nlr = 0.0005\ndropout = 0.0\nMAX_SEQ_LEN = 256\nbatch_size = 64\nvalid_batch_size = 384\n\n\nmodel = TransformerEncoderOnlyModel(d_model=512, nhead=8, dim_feedforward=384, num_layers=6, max_seq_len=MAX_SEQ_LEN*2, dropout=dropout, use_performer=False, feature_type='relu', compute_type='iter', on_gptln=True)\n#model = TransformerEncoderOnlyModel(d_model=1024, nhead=8, dim_feedforward=1024, num_layers=6, max_seq_len=MAX_SEQ_LEN*2, dropout=dropout, use_performer=True, feature_type='sqr', compute_type='iter', on_gptln=True)\n#model = TransformerEncoderOnlyModel(d_model=256, nhead=8, dim_feedforward=256, num_layers=3, max_seq_len=MAX_SEQ_LEN*2, dropout=dropout, use_performer=True, feature_type='sqr', compute_type='iter', on_gptln=True)\ncriteria = [#nn.CrossEntropyLoss(ignore_index=0, weight=torch.tensor([0, 0.5, 0.5,])).to(device),#nn.BCEWithLogitsLoss().to(device),\n            CustomBCELoss().to(device),\n            nn.CrossEntropyLoss(ignore_index=0, weight=torch.tensor([0, 0.25, 0.25, 0.25, 0.25,])).to(device),\n           ]\n\nbest_model_file_path = '../input/model4pth/prf_wue0.25_sl256_lr0.0005_e5_of7_trainauc0.8082896856958341.pth'\nprint('Loading Model:', best_model_file_path)\nmodel = torch.load(best_model_file_path)\nmodel.to(device)\n\n\ntest_set_start_positions_by_user_idxs, test_user_idxs, test_start_poss, test_lengths, test_roc_auc_start_poss = sample_test(MAX_SEQ_LEN)\n\nvalid_dataset = RiiidDataset(is_train_set=False, max_seq_len=MAX_SEQ_LEN)\n\nprint('MAX_SEQ_LEN:', MAX_SEQ_LEN, model.max_seq_len)\nprint('epochs:', epochs)\nprint('WARMUP_EPOCHS:', WARMUP_EPOCHS)\nprint('lr:', lr)\nprint('dropout:', dropout)\nprint('batch_size:', batch_size)\nprint('valid_batch_size:', valid_batch_size)\nprint('nhead:', model.nhead)\nprint(model)\n\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# target_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', usecols=['row_id', 'timestamp', 'user_id', 'content_id', 'content_type_id', 'task_container_id', 'user_answer', 'prior_question_elapsed_time', 'prior_question_had_explanation',\n                                                   \n                                                   \n#                                                    'answered_correctly',\n#                                                    ],  #low_memory=False, #nrows=10**5, \n#                        dtype={'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n#                               'task_container_id': 'int16', 'user_answer': 'int8', 'prior_question_elapsed_time': 'float32', \n#                               'prior_question_had_explanation': 'boolean',\n                              \n                              \n#                               'answered_correctly': 'int8',\n#                              }\n#                        ,nrows=20000)\n# print(target_df)\n# target_df = target_df[target_df.index >= 10000]\n# target_df = target_df.sort_values(['timestamp', 'user_id', 'task_container_id'], ascending=True).reset_index(drop=True)\n# print(target_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Iter_Valid(object):\n    def __init__(self, df, max_user=1000):\n        df = df.reset_index(drop=True)\n        self.df = df\n        self.user_answer = df['user_answer'].astype(str).values\n        self.answered_correctly = df['answered_correctly'].astype(str).values\n        df['prior_group_responses'] = \"[]\"\n        df['prior_group_answers_correct'] = \"[]\"\n        self.sample_df = df[df['content_type_id'] == 0][['row_id']]\n        self.sample_df['answered_correctly'] = 0\n        self.len = len(df)\n        self.user_id = df.user_id.values\n        self.task_container_id = df.task_container_id.values\n        self.content_type_id = df.content_type_id.values\n        self.max_user = max_user\n        self.current = 0\n        self.pre_user_answer_list = []\n        self.pre_answered_correctly_list = []\n\n    def __iter__(self):\n        return self\n    \n    def fix_df(self, user_answer_list, answered_correctly_list, pre_start):\n        df= self.df[pre_start:self.current].copy()\n        sample_df = self.sample_df[pre_start:self.current].copy()\n        df.loc[pre_start,'prior_group_responses'] = '[' + \",\".join(self.pre_user_answer_list) + ']'\n        df.loc[pre_start,'prior_group_answers_correct'] = '[' + \",\".join(self.pre_answered_correctly_list) + ']'\n        print('lll',df.loc[pre_start,'prior_group_answers_correct'])\n        self.pre_user_answer_list = user_answer_list\n        self.pre_answered_correctly_list = answered_correctly_list\n        return df, sample_df\n\n    def __next__(self):\n        found_good = False\n        \n#         df0s = []\n#         df1s = []\n        \n#         l = np.random.randint(100)\n        while not found_good:\n            added_user = set()\n            pre_start = self.current\n            pre_added_user = -1\n            pre_task_container_id = -1\n\n            user_answer_list = []\n            answered_correctly_list = []\n            result = None\n            while self.current < self.len:\n                crr_user_id = self.user_id[self.current]\n                crr_task_container_id = self.task_container_id[self.current]\n                crr_content_type_id = self.content_type_id[self.current]\n                if crr_content_type_id == 1:\n                    # no more than one task_container_id of \"questions\" from any single user\n                    # so we only care for content_type_id == 0 to break loop\n                    #user_answer_list.append(self.user_answer[self.current])\n                    #answered_correctly_list.append(self.answered_correctly[self.current])\n                    self.current += 1\n                    continue\n                if crr_user_id in added_user and ((crr_user_id != pre_added_user) or\n                                                  (crr_task_container_id != pre_task_container_id)):\n                    # known user(not prev user or differnt task container)\n                    result = self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n                    break\n                if len(added_user) == self.max_user:\n                    if  crr_user_id == pre_added_user and crr_task_container_id == pre_task_container_id:\n                        user_answer_list.append(self.user_answer[self.current])\n                        answered_correctly_list.append(self.answered_correctly[self.current])\n                        self.current += 1\n                        continue\n                    else:\n                        result = self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n                        break\n                added_user.add(crr_user_id)\n                pre_added_user = crr_user_id\n                pre_task_container_id = crr_task_container_id\n                user_answer_list.append(self.user_answer[self.current])\n                answered_correctly_list.append(self.answered_correctly[self.current])\n                self.current += 1\n            if result is None:\n                if pre_start < self.current:\n                    result = self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n                else:\n                    raise StopIteration()\n\n            dff = result[0]\n            if len(dff[dff['content_type_id'] == 0]) > 0:\n                found_good = True\n                return dff, result[1]\n            else:\n                found_good = False\n\n#         for dff in df0s:\n#             dff.reset_index(drop=True, inplace=True)\n#         for dff in df1s:\n#             dff.reset_index(drop=True, inplace=True)\n\n#         df0 = pd.concat(df0s).reset_index()\n#         df1 = pd.concat(df1s).reset_index()\n        \n#         return df0, df1\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# iter_test = Iter_Valid(target_df,max_user=1000)\n# predicted = []\n# class Envir:\n#     def predict(self, df):\n#         predicted.append(df)\n# env = Envir()\n\n\n# from tqdm.notebook import tqdm\n# import time\n\n# pbar = tqdm(total=len(target_df))\n# previous_test_df = None\n# for (current_test, current_prediction_df) in iter_test:\n#     if previous_test_df is not None:\n#         answers = eval(current_test[\"prior_group_answers_correct\"].iloc[0])\n#         responses = eval(current_test[\"prior_group_responses\"].iloc[0])\n#         previous_test_df['answered_correctly'] = answers\n#         previous_test_df['user_answer'] = responses\n#         # your feature extraction and model training code here\n#     previous_test_df = current_test.copy()\n#     current_test = current_test[current_test.content_type_id == 0]\n#     # your prediction code here\n#     current_test['answered_correctly'] = 0.5\n#     env.predict(current_test.loc[:,['row_id', 'answered_correctly']])\n#     pbar.update(len(current_test))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# y_true = target_df[target_df.content_type_id == 0].answered_correctly\n# y_pred = pd.concat(predicted).answered_correctly\n# print('validation auc:',roc_auc_score(y_true, y_pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\n\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# iter_test = Iter_Valid(target_df,max_user=1000)\n# predicted = []\n# class Envir:\n#     def predict(self, df):\n#         predicted.append(df)\n# env = Envir()\n\n\n\n\n\n\n\n\n\n\n\n\nfrom tqdm.notebook import tqdm\n\n\nuser_data_by_idxs = list(user_data_by_idxs)\nuser_ids_by_idxs = list(user_ids_by_idxs)\n#print(user_ids_by_idxs[-10:])\n\nfake_dataset = RiiidDataset(is_train_set=False, max_seq_len=MAX_SEQ_LEN)\n#fake_dataset.prepare()\n\n# item = fake_dataset.__getitem__(-115)\n# print(len(item))\n\n\n# user_idxs_for_batch = range(1, 1 + 10)\n\n# batch_samples = [[] for i in range(16)]\n# batch_positions_of_interest = []\n# for user_idx_for_batch in user_idxs_for_batch:\n#     #print(user_idx_for_batch)\n#     print('u_idx:', -user_idx_for_batch)\n#     sample = fake_dataset.__getitem__(-user_idx_for_batch)\n#     mask = sample[-1]\n#     position_of_interest = np.nonzero(mask)[0][-1]\n#     #print(position_of_interest)\n#     batch_positions_of_interest.append(position_of_interest)\n#     assert len(sample) == 16\n#     for i in range(len(sample)):\n#         batch_samples[i].append(sample[i])\n# batch = [torch.tensor(batch_samples[i]) for i in range(len(batch_samples))]\n\n\n# print(f'batch_positions_of_interest: {len(batch_positions_of_interest)}')\n# print(f'batch: {len(batch[0])}')\n# #print('batch:', batch)\n\n\n\nmodel.eval()\n\n\ndef process_batch(batch_as_item, batch_positions_of_interest, batch_id):\n    item = batch_as_item\n    \n    losses = []\n    losses_sum = 0.0\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outputs = []\n    predictions = []\n\n    with torch.no_grad():\n        attn_mask = model.generate_square_subsequent_mask(seq_len=model.max_seq_len).to(device)\n        \n        user_idxs = item[0].to(device).long()\n        \n        timestamps = item[1].to(device).double()\n        \n        \n        #lecture_idxs = item[2].to(device).long()\n        question_pre_lecture_length_idxs = item[2].to(device).long()\n        question_idxs = item[3].to(device).long()\n\n        \n#         parts = item[4].to(device).long()\n#         question_tag_idx0s = item[5].to(device).long()\n#         question_tag_idx1s = item[6].to(device).long()\n#         question_tag_idx2s = item[7].to(device).long()\n#         question_tag_idx3s = item[8].to(device).long()\n#         question_tag_idx4s = item[9].to(device).long()\n#         question_tag_idx5s = item[10].to(device).long()\n        local_bundle_ids = item[4].to(device).long()\n        prior_question_elapsed_times_idxs = item[5].to(device).long()\n        prior_question_had_explanation_idxs = item[6].to(device).long()\n        question_from_prev_timestamps_idxs = item[7].to(device).long()\n        prior_question_was_answered_correctly_idxs = item[8].to(device).long()\n        post_question_elapsed_times_idxs = item[9].to(device).long()\n        post_question_has_explanation_idxs = item[10].to(device).long()\n\n        \n        label_user_answers_idxs = item[11].to(device).long()\n        \n        correct_answers_idxs = item[12].to(device).long()\n        label_answered_correctly_idxs = item[-3].to(device).long()\n        participates_in_roc_auc_long = item[-2]\n        participates_in_roc_auc_long = participates_in_roc_auc_long.to(device).long()\n        participates_in_roc_auc = participates_in_roc_auc_long.float()\n        mask = item[-1].to(device).long()\n        \n        \n        output0, output1 = model(user_idxs,\n            timestamps,\n\n            #lecture_idxs,\n            question_pre_lecture_length_idxs,\n            question_idxs,\n\n\n#             parts,\n#             question_tag_idx0s,\n#             question_tag_idx1s,\n#             question_tag_idx2s,\n#             question_tag_idx3s,\n#             question_tag_idx4s,\n#             question_tag_idx5s,\n            local_bundle_ids, prior_question_elapsed_times_idxs, prior_question_had_explanation_idxs,\n            question_from_prev_timestamps_idxs,\n            prior_question_was_answered_correctly_idxs,\n            post_question_elapsed_times_idxs,\n            post_question_has_explanation_idxs,\n\n            label_answered_correctly_idxs,\n            correct_answers_idxs,\n            mask,\n            attn_mask)\n        output0 = output0.squeeze(-1)\n        #loss0 = criteria[0](output0.view(-1, 3), label_answered_correctly_idxs.view(-1))\n        #pred = (output0[:,:,2] > 0).long()\n        output0 = torch.sigmoid(output0)\n        pred = (output0 >= 0.5).long()\n        pred = pred + 1\n        #print(f'len(batch_positions_of_interest)={len(batch_positions_of_interest)} output0.size()={output0.size()}')\n        idx0 = torch.arange(len(batch_positions_of_interest)).to(device)\n        #print(f'idx0.size()={idx0.size()}')\n        idx1 = torch.tensor(batch_positions_of_interest).to(device)\n        #print(f'idx1.size()={idx1.size()}')\n        batch_predictions = output0[idx0, idx1]\n        batch_predictions = batch_predictions.detach()\n        batch_predictions = batch_predictions.cpu()\n        batch_predictions = batch_predictions.numpy().astype(np.float64)\n        loss0 = criteria[0](output0.view(-1), label_answered_correctly_idxs.float().view(-1))\n\n        #loss0 = roc_star_loss(output0[:,:,2], preds, epoch_gamma, last_epoch_y_t, last_epoch_y_pred)\n        #output1 = output1.transpose(0, 1).contiguous()\n        loss1 = criteria[1](output1.reshape(-1, 5), label_user_answers_idxs.view(-1))\n        #loss_normalizer = participates_in_roc_auc.sum().item()\n        loss = (loss0 + loss1) #/ loss_normalizer\n\n        loss_val = loss.detach().item()\n        losses.append(loss_val)\n        losses_sum += loss_val\n\n#         mask = participates_in_roc_auc.bool()\n#         output = torch.masked_select(output, mask)\n#         label_answered_correctly = torch.masked_select(label_answered_correctly, mask)\n#         pred_answered_correctly = (output >= 0.5).long()\n\n#         num_corrects += (pred_answered_correctly == label_answered_correctly).sum().item()\n#         num_total += len(label_answered_correctly)\n\n#         #auc_roc_applicable_positions_mask_flattened = participates_in_roc_auc.bool().view(-1).cpu().numpy()\n#         #label_answered_correctly_flattened = label_answered_correctly.view(-1).cpu().numpy()\n#         labels.extend(label_answered_correctly)\n#         outputs.extend(output)\n\n        #num_corrects += ((torch.argmax(output0, dim=-1) == label_answered_correctly_idxs) * participates_in_roc_auc).sum().item()\n        num_corrects += ((pred == label_answered_correctly_idxs) * participates_in_roc_auc).sum().item()\n        num_total += participates_in_roc_auc.sum().item()\n\n        auc_roc_applicable_positions_mask_flattened = (participates_in_roc_auc * label_answered_correctly_idxs).bool().view(-1).cpu().numpy()#######\n        label_answered_correctly_idxs_flattened = (label_answered_correctly_idxs - 1).view(-1).cpu().numpy()\n        labels.extend(label_answered_correctly_idxs_flattened[auc_roc_applicable_positions_mask_flattened])\n        #output_flattened = output0.view(-1, 3).data.cpu().numpy()[:,2]\n        output_flattened = output0.view(-1).data.cpu().numpy()\n        outputs.extend(output_flattened[auc_roc_applicable_positions_mask_flattened])\n\n    if batch_id % 100 == 0:\n        if len(outputs) > 0:\n            acc = num_corrects / max(0.000001, num_total)\n            if np.max(labels) != np.min(labels):\n                auc = roc_auc_score(labels, outputs)\n            else:\n                auc = 0.0\n            loss = np.mean(losses)  #TODO: need direct mean, not mean of means\n        else:\n            acc = 0.0\n            auc = 0.0\n            loss = np.mean(losses)  #TODO: need direct mean, not mean of means\n        #print(f'outputs', len(outputs), 'labels', len(labels))\n        #print(f'labels', np.array(labels).min(), np.array(labels).max())\n        #if len(outputs) > 0:\n        #    print(f'outputs[0]', outputs[0], outputs[-1])\n        #    print(f'labels[0]', labels[0], labels[-1])\n\n        print(f'{datetime.now().isoformat()}: batch_id={batch_id} batch_len={len(batch)} loss={loss}, acc={acc}, auc={auc}')\n        \n    return batch_predictions\n\n    \n#batch_predictions = process_batch(batch, batch_positions_of_interest, 0)\n#print(batch_predictions)\n\n\n\n\n\ndef update_prev_group(prior_group_answers_correct, prior_group_responses, prior_group_user_idxs):\n    \n    prior_group_answers_correct_idxs = np.array([int(s) + 1 for s in (prior_group_answers_correct[1:-1].replace(' ', '').split(',') if prior_group_answers_correct != '[]' else [])], dtype=np.int8)\n    prior_group_responses = np.array([int(s) for s in (prior_group_responses[1:-1].replace(' ', '').split(',') if prior_group_responses != '[]' else [])], dtype=np.int8)\n    assert len(prior_group_answers_correct_idxs) == len(prior_group_responses)\n    assert len(prior_group_answers_correct_idxs) == len(prior_group_user_idxs), f'len(prior_group_answers_correct_idxs) == len(prior_group_user_idxs): {len(prior_group_answers_correct_idxs)} {len(prior_group_user_idxs)}'\n\n    user_data_skips_by_idxs = np.zeros(len(user_data_by_idxs), dtype='int')\n    for i in range(len(prior_group_responses) - 1, -1, -1):\n        user_idx = prior_group_user_idxs[i]\n        if user_idx != 0:\n            skip = user_data_skips_by_idxs[user_idx]\n            user_data_skips_by_idxs[user_idx] += 1\n            #print(f'patching user_idx {user_idx}')\n            #print(f'******************* prior_group_answers_correct_idxs {prior_group_answers_correct_idxs}')\n            #print(f'******************* patching prior_group_answers_correct_idxs {prior_group_answers_correct_idxs}')\n            user_data = user_data_by_idxs[user_idx]\n            user_answers = user_data['user_answers']\n            assert len(user_answers) > skip, f'len(user_answers) >= skip: {len(user_answers)} > {skip}'\n            user_answers[-skip-1] = prior_group_responses[i]\n\n            answered_correctly_idxs = user_data['answered_correctly_idxs']\n            assert len(answered_correctly_idxs) > skip, f'len(answered_correctly_idxs) > skip: {len(answered_correctly_idxs)} >= {skip}'\n            answered_correctly_idxs[-skip-1] = prior_group_answers_correct_idxs[i]\n    \n    \n    \n#pbar = tqdm(total=len(target_df))\n\ngroup_user_idxs = []\nbatch_id = 0\nprint(f'{datetime.now().isoformat()}: Starting dfs')\nfor (test_df, sample_prediction_df) in iter_test:\n    \n    dbg_cond = batch_id < 10\n    \n    if dbg_cond:\n        print(f'{datetime.now().isoformat()}: Starting df of size {len(test_df)} ({len(test_df[test_df[\"content_type_id\"] == 0])}q {len(test_df[test_df[\"content_type_id\"] == 1])}l) sample_prediction_df: {sample_prediction_df}')\n    \n#     test_df = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv',\n#                        dtype={'group_num': 'int32', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n#                               'task_container_id': 'int16', 'user_answer': 'int8', 'prior_question_elapsed_time': 'float32', \n#                               'prior_question_had_explanation': 'boolean',\n#                              })\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].astype(np.float).fillna(-1).astype(np.int8)\n    test_df['prior_question_elapsed_time'] = test_df['prior_question_elapsed_time'].fillna(-1000)\n    if dbg_cond:\n        #print(f'df columns: {test_df.columns}')\n        #print(f'df dtypes: {test_df.dtypes}')\n        print(test_df.head(10))\n\n    answered_correctly = [0.0 for l in range(len(test_df))]\n    \n    prior_group_user_idxs = group_user_idxs\n    group_user_idxs = []\n    \n    \n    prev_group_num = -1\n    batch_samples = [[] for i in range(16)]\n    batch_positions_to_df_positions = []\n    batch_positions_of_interest = []\n    #for df_position, row in enumerate(test_df[list(test_df.columns)].itertuples(index=True)):\n    for df_position, (group_num, row) in enumerate(test_df.iterrows()):\n        #group_num, row_id, timestamp, user_id, content_id, content_type_id, task_container_id, prior_question_elapsed_time, prior_question_had_explanation, prior_group_answers_correct, prior_group_responses = row\n        timestamp = row['timestamp']\n        user_id = row['user_id']\n        content_id = row['content_id']\n        content_type_id = row['content_type_id']\n        task_container_id = row['task_container_id']\n        prior_question_elapsed_time = row['prior_question_elapsed_time']\n        prior_question_had_explanation = row['prior_question_had_explanation']\n        prior_group_answers_correct = row['prior_group_answers_correct']\n        prior_group_responses = row['prior_group_responses']\n        \n        assert content_type_id == 0 or content_type_id == 1\n        \n        if dbg_cond:\n            print('row, group_num:', row, group_num)\n        \n        #if group_num != prev_group_num:\n        if df_position == 0:\n            update_prev_group(prior_group_answers_correct, prior_group_responses, prior_group_user_idxs)\n\n        if user_id in user_idxs_by_ids:\n            user_idx = user_idxs_by_ids[user_id]\n            user_data = user_data_by_idxs[user_idx]\n        else:\n            user_data = {\n                'question_timestamps': np.array([], dtype=np.int64),\n                'question_prev_timestamps': np.array([], dtype=np.int64),\n\n                'question_ids': np.array([], dtype=np.int16),\n                'question_task_container_ids': np.array([], dtype=np.int16),\n                'user_answers': np.array([], dtype=np.int8),\n                'prior_question_elapsed_times': np.array([], dtype=np.float32),\n                'prior_question_had_explanation_idxs': np.array([], dtype=np.int8),\n                'answered_correctly_idxs': np.array([], dtype=np.int8),\n\n                'question_prev_was_lecture': np.array([], dtype=np.int8),\n                'question_prev_prev_timestamps': np.array([], dtype=np.int64),\n            }\n            user_idx = len(user_data_by_idxs)\n            user_data_by_idxs.append(user_data)\n            user_idxs_by_ids[user_id] = user_idx\n            user_data_questions_counts_by_idxs.append(0)\n            user_ids_by_idxs.append(user_id)\n\n        if content_type_id == 1:\n            user_data['reminder__question_prev_lecture_timestamp'] = (timestamp)\n            group_user_idxs.append(0)\n        else:\n\n            if 'reminder__question_prev_lecture_timestamp' in user_data:\n                reminder__question_prev_lecture_timestamp = user_data['reminder__question_prev_lecture_timestamp']\n                reminder__prev_was_lecture = 1\n                del user_data['reminder__question_prev_lecture_timestamp']\n            else:\n                reminder__question_prev_lecture_timestamp = None\n                reminder__prev_was_lecture = 0\n\n\n            user_data_questions_counts_by_idxs[user_idx] += 1\n\n\n            question_timestamps = user_data['question_timestamps']\n            question_timestamps = np.append(question_timestamps, timestamp)\n            if len(question_timestamps) > MAX_SEQ_LEN + 2:\n                question_timestamps = question_timestamps[-MAX_SEQ_LEN-2:]\n                user_data_questions_counts_by_idxs[user_idx] = MAX_SEQ_LEN + 2\n            user_data['question_timestamps'] = question_timestamps\n\n            question_prev_timestamps = user_data['question_prev_timestamps']\n            if reminder__question_prev_lecture_timestamp is not None:\n                question_prev_timestamp = reminder__question_prev_lecture_timestamp\n            else:\n                if len(question_timestamps) >= 2:\n                    question_prev_timestamp = question_timestamps[-2]\n                else:\n                    question_prev_timestamp = 0\n            question_prev_timestamps = np.append(question_prev_timestamps, question_prev_timestamp)\n            if len(question_prev_timestamps) > MAX_SEQ_LEN + 2:\n                question_prev_timestamps = question_prev_timestamps[-MAX_SEQ_LEN-2:]\n            user_data['question_prev_timestamps'] = question_prev_timestamps\n\n            question_ids = user_data['question_ids']\n            question_ids = np.append(question_ids, content_id)\n            if len(question_ids) > MAX_SEQ_LEN + 2:\n                question_ids = question_ids[-MAX_SEQ_LEN-2:]\n            user_data['question_ids'] = question_ids\n\n            question_task_container_ids = user_data['question_task_container_ids']\n            question_task_container_ids = np.append(question_task_container_ids, task_container_id)\n            if len(question_task_container_ids) > MAX_SEQ_LEN + 2:\n                question_task_container_ids = question_task_container_ids[-MAX_SEQ_LEN-2:]\n            user_data['question_task_container_ids'] = question_task_container_ids\n\n            user_answers = user_data['user_answers']\n            user_answers = np.append(user_answers, 0)\n            if len(user_answers) > MAX_SEQ_LEN + 2:\n                user_answers = user_answers[-MAX_SEQ_LEN-2:]\n            user_data['user_answers'] = user_answers\n\n            prior_question_elapsed_times = user_data['prior_question_elapsed_times']\n            prior_question_elapsed_times = np.append(prior_question_elapsed_times, prior_question_elapsed_time)\n            if len(prior_question_elapsed_times) > MAX_SEQ_LEN + 2:\n                prior_question_elapsed_times = prior_question_elapsed_times[-MAX_SEQ_LEN-2:]\n            user_data['prior_question_elapsed_times'] = prior_question_elapsed_times\n\n            prior_question_had_explanation_idxs = user_data['prior_question_had_explanation_idxs']\n            prior_question_had_explanation_idxs = np.append(prior_question_had_explanation_idxs, prior_question_had_explanation + 1)\n            if len(prior_question_had_explanation_idxs) > MAX_SEQ_LEN + 2:\n                prior_question_had_explanation_idxs = prior_question_had_explanation_idxs[-MAX_SEQ_LEN-2:]\n            user_data['prior_question_had_explanation_idxs'] = prior_question_had_explanation_idxs\n\n            answered_correctly_idxs = user_data['answered_correctly_idxs']\n            answered_correctly_idxs = np.append(answered_correctly_idxs, 0)\n            if len(answered_correctly_idxs) > MAX_SEQ_LEN + 2:\n                answered_correctly_idxs = answered_correctly_idxs[-MAX_SEQ_LEN-2:]\n            user_data['answered_correctly_idxs'] = answered_correctly_idxs\n\n            question_prev_was_lecture = user_data['question_prev_was_lecture']\n            question_prev_was_lecture = np.append(question_prev_was_lecture, reminder__prev_was_lecture)\n            if len(question_prev_was_lecture) > MAX_SEQ_LEN + 2:\n                question_prev_was_lecture = question_prev_was_lecture[-MAX_SEQ_LEN-2:]\n            user_data['question_prev_was_lecture'] = question_prev_was_lecture\n\n            question_prev_prev_timestamps = user_data['question_prev_prev_timestamps']\n            if len(question_prev_timestamps) >= 2:\n                question_prev_prev_timestamp = question_prev_timestamps[-2]\n            else:\n                question_prev_prev_timestamp = 0\n            question_prev_prev_timestamps = np.append(question_prev_prev_timestamps, question_prev_prev_timestamp)\n            if len(question_prev_prev_timestamps) > MAX_SEQ_LEN + 2:\n                question_prev_prev_timestamps = question_prev_prev_timestamps[-MAX_SEQ_LEN-2:]\n            user_data['question_prev_prev_timestamps'] = question_prev_prev_timestamps\n\n            #         print(f'user_id={user_id} user_idx={user_idx} user_idxs_by_ids[user_id]={user_idxs_by_ids[user_id]} {len(user_ids_by_idxs)}')\n            #         print(f'user_id={user_id} user_idx={user_idx} user_ids_by_idxs[user_idxs_by_ids[user_id]]={user_ids_by_idxs[user_idxs_by_ids[user_id]]}')\n            #         print(f'user_data={str(user_data)[:1000]}')\n\n            if dbg_cond:\n                print('u_idx:', user_idx, batch_id)\n            group_user_idxs.append(user_idx)\n            sample = fake_dataset.__getitem__(-user_idx)\n            mask = sample[-1]\n            positions_of_interest = np.nonzero(mask)[0]\n            if len(positions_of_interest) > 0:\n                position_of_interest = positions_of_interest[-1]\n            else:  # Actually this should never happen\n                position_of_interest = 0\n\n            batch_positions_to_df_positions.append(df_position)\n            batch_positions_of_interest.append(position_of_interest)\n            for i in range(len(sample)):\n                batch_samples[i].append(sample[i])\n\n        need_to_update_batch_id = False\n        if (len(batch_samples[0]) == valid_batch_size) or (df_position == len(test_df) - 1):\n\n            if len(batch_samples[0]) > 0:\n                batch = [torch.tensor(batch_samples[i]) for i in range(len(batch_samples))]\n\n                if dbg_cond:\n                    print(f'{datetime.now().isoformat()}: Predicting batch {batch_id}')\n                    for ii in range(16):\n                        print(batch_samples[ii])\n                batch_predictions = process_batch(batch, batch_positions_of_interest, batch_id)\n                for batch_pos, batch_position_to_df_position in enumerate(batch_positions_to_df_positions):\n                    answered_correctly[batch_position_to_df_position] = batch_predictions[batch_pos]\n                #if dbg_cond:\n                #    print(f'{datetime.now().isoformat()}: batch_predictions {batch_id}: {batch_predictions}')\n\n                batch_samples = [[] for i in range(16)]\n                batch_positions_to_df_positions = []\n                batch_positions_of_interest = []\n                need_to_update_batch_id = True\n\n        if df_position == len(test_df) - 1:\n            assert len(test_df) == len(answered_correctly)\n            if dbg_cond:\n                print(f'{datetime.now().isoformat()}: Sending df predictions')\n            #test_df['answered_correctly'] = 0.5\n            #if dbg_cond:\n            #    print(f'dtypes: {test_df.dtypes}')\n            test_df['answered_correctly'] = np.array(answered_correctly)\n#             if dbg_cond:\n#                 print(f'dtypes: {test_df.dtypes}')\n#                 print(f'dtype: {np.array(answered_correctly).dtype}')\n#                 print(f'arr: {np.array(answered_correctly)}')\n#                 print(f'{datetime.now().isoformat()}: pre - Sent df predictions: {test_df.loc[test_df[\"content_type_id\"] == 0, [\"row_id\", \"answered_correctly\"]]}')\n            env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n            \n            #pbar.update(len(test_df))\n            if dbg_cond:\n                print(f'{datetime.now().isoformat()}: Sent df predictions')\n\n\n        if need_to_update_batch_id:\n            batch_id += 1\n\n\n        #prev_group_num = group_num\n\n    if dbg_cond:\n        print(f'{datetime.now().isoformat()}: Finished df')\nprint(f'{datetime.now().isoformat()}: Done')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}