{"cells":[{"metadata":{"papermill":{"duration":0.050085,"end_time":"2021-01-01T18:19:11.163556","exception":false,"start_time":"2021-01-01T18:19:11.113471","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# SAINT\n* With this notebook, we would like to send our special thanks to @Yih-Dar SHIEH for his/her dedicate explanations, we studied his/her notebook (https://www.kaggle.com/yihdarshieh/tpu-track-knowledge-states-of-1m-students) a lot, and gained many insights. Please upvote his/her notebook first;\n* Beside, the model is mostly copied from https://github.com/seewoo5/KT with some personal adjustments in designing masks. The masking scheme is also inspired by @Yih-Dar SHIEH's notebook;\n* Add content_difficulty and user_correctness, the content_difficulty is in the encoder part meanwhile the user_correctness is in the decoder part. They are both computed before splitting the data into training and validating sets;\n* Continue to train the model, reset the learning rate to 1e-5 initially, the model was trained twice before. The first stage was trained in https://www.kaggle.com/shinomoriaoshi/riiid-saint-randomization-inference/output?scriptVersionId=51035364, the last two stages were trained on QBlocks platform. The learning rate scheduler is, the 30 first epochs with the initial learning rate of 1e-3, the next 30 epochs with the inital learning rate of 1e-5, the final 25 epochs with the intial learning rate of 1e-7. All there training stages use Noam scheduler. Best CV after the training process is 0.7825;\n* There are a lot of missing parts due to the limited time, one of them is lag time, which I believe could boost the result quite a lot (according to what I observed from the thread of SAINT benchmark);\n* Because we know the SAINT model has much more potential than what we got here with the LB score, the main reason we share our work is that people can read and discuss what we are missing such that they can boost the model performance, we welcome all discussions :D."},{"metadata":{"papermill":{"duration":0.04809,"end_time":"2021-01-01T18:19:11.260397","exception":false,"start_time":"2021-01-01T18:19:11.212307","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* First setting"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:11.397995Z","iopub.status.busy":"2021-01-01T18:19:11.397094Z","iopub.status.idle":"2021-01-01T18:19:11.400668Z","shell.execute_reply":"2021-01-01T18:19:11.401186Z"},"papermill":{"duration":0.09128,"end_time":"2021-01-01T18:19:11.401308","exception":false,"start_time":"2021-01-01T18:19:11.310028","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"r'''!pip install sklearn\n!pip install datatable\n!pip install seaborn\n!pip install kaggle\n!mkdir ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!chmod 600 /home/qblocks/.kaggle/kaggle.json\n!kaggle datasets download -d shinomoriaoshi/riiid-chunking\n!kaggle competitions download -c riiid-test-answer-prediction\n!kaggle datasets download -d yihdarshieh/r3id-info-public\n!unzip riiid-chunking.zip\n!unzip riiid-test-answer-prediction.zip\n!unzip r3id-info-public.zip'''","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2021-01-01T18:19:11.508643Z","iopub.status.busy":"2021-01-01T18:19:11.507955Z","iopub.status.idle":"2021-01-01T18:19:14.034789Z","shell.execute_reply":"2021-01-01T18:19:14.033871Z"},"papermill":{"duration":2.584051,"end_time":"2021-01-01T18:19:14.034916","exception":false,"start_time":"2021-01-01T18:19:11.450865","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import gc, sys, os\nimport random, math\nfrom tqdm.notebook import tqdm\n\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nimport multiprocessing\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\n\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.nn.utils.rnn as rnn_utils\nfrom torch.autograd import Variable\nfrom torch.utils.data import Dataset, DataLoader\n\nimport seaborn as sns\nsns.set()\nDEFAULT_FIG_WIDTH = 20\nsns.set_context(\"paper\", font_scale = 1.2)\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:14.14318Z","iopub.status.busy":"2021-01-01T18:19:14.142263Z","iopub.status.idle":"2021-01-01T18:19:14.147428Z","shell.execute_reply":"2021-01-01T18:19:14.146792Z"},"papermill":{"duration":0.060579,"end_time":"2021-01-01T18:19:14.147556","exception":false,"start_time":"2021-01-01T18:19:14.086977","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"print('Python     : ' + sys.version.split('\\n')[0])\nprint('Numpy      : ' + np.__version__)\nprint('Pandas     : ' + pd.__version__)\nprint('PyTorch    : ' + torch.__version__)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:14.623741Z","iopub.status.busy":"2021-01-01T18:19:14.620752Z","iopub.status.idle":"2021-01-01T18:19:14.629272Z","shell.execute_reply":"2021-01-01T18:19:14.628716Z"},"papermill":{"duration":0.429337,"end_time":"2021-01-01T18:19:14.629381","exception":false,"start_time":"2021-01-01T18:19:14.200044","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"DEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') # if IS_TPU == False else xm.xla_device()\nprint('Running on device: {}'.format(DEVICE))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:14.743939Z","iopub.status.busy":"2021-01-01T18:19:14.743179Z","iopub.status.idle":"2021-01-01T18:19:14.746323Z","shell.execute_reply":"2021-01-01T18:19:14.745839Z"},"papermill":{"duration":0.062996,"end_time":"2021-01-01T18:19:14.746426","exception":false,"start_time":"2021-01-01T18:19:14.68343","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def seed_everything(s):\n    random.seed(s)\n    os.environ['PYTHONHASHSEED'] = str(s)\n    np.random.seed(s)\n    # Torch\n    torch.manual_seed(s)\n    torch.cuda.manual_seed(s)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    if torch.cuda.is_available():\n        torch.cuda.manual_seed_all(s)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:14.860226Z","iopub.status.busy":"2021-01-01T18:19:14.859374Z","iopub.status.idle":"2021-01-01T18:19:14.862484Z","shell.execute_reply":"2021-01-01T18:19:14.86189Z"},"papermill":{"duration":0.061879,"end_time":"2021-01-01T18:19:14.862576","exception":false,"start_time":"2021-01-01T18:19:14.800697","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"HOME =  \"./\"\nDATA_HOME = r'../input/riiid-test-answer-prediction'\nMODEL_NAME = \"SAKT-v1\"\nMODEL_PATH = HOME + MODEL_NAME\nSTAGE = \"stage1\"\nMODEL_BEST = 'model_best.pt'\nFOLD = 1\nMAX_SEQ = 256\n    \nCONTENT_TYPE_ID = \"content_type_id\"\nCONTENT_ID = \"content_id\"\nTARGET = \"answered_correctly\"\nUSER_ID = \"user_id\"\nTASK_CONTAINER_ID = \"task_container_id\"\nTIMESTAMP = \"timestamp\"\nPART = \"part\"\nELAPSE = \"prior_question_elapsed_time\"\nDIFF = 'content_difficulty'\nCORR = \"user_correctness\"","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.054443,"end_time":"2021-01-01T18:19:14.978239","exception":false,"start_time":"2021-01-01T18:19:14.923796","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Load data"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:19:15.11887Z","iopub.status.busy":"2021-01-01T18:19:15.111614Z","iopub.status.idle":"2021-01-01T18:21:19.151193Z","shell.execute_reply":"2021-01-01T18:21:19.15062Z"},"papermill":{"duration":124.114039,"end_time":"2021-01-01T18:21:19.151319","exception":false,"start_time":"2021-01-01T18:19:15.03728","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#%%time\nUSE_PICKLE = True\n\ndtype = {\n    USER_ID: 'int32', \n    CONTENT_ID: 'int16',\n    CONTENT_TYPE_ID: 'bool',\n    TARGET:'int8',\n}\ntrain_df = dt.fread(os.path.join(DATA_HOME, 'train.csv'), columns = set(dtype.keys())).to_pandas()\ntrain_df = train_df[train_df[CONTENT_TYPE_ID] == False].reset_index(drop = True)\ntrain_df.head()\n\n# Overall user correctness\nuser_agg = train_df.groupby(USER_ID)[TARGET].agg(['sum', 'count']).astype('int16')\n# Overall difficulty of questions\ncontent_agg = train_df.groupby(CONTENT_ID)[TARGET].agg(['sum', 'count'])\n\ndel train_df; gc.collect()\n    \nif not USE_PICKLE:\n    # How hard are questions in each content ID?\n    train_df[DIFF] = train_df[CONTENT_ID].map(content_agg['sum'] / content_agg['count'])\n\n    train_df.fillna(0, inplace = True)\n\n    skills = train_df[CONTENT_ID].unique()\n    n_skill = 13523 # len(skills)\n    print(\"Number of skills\", n_skill)\n\n    elapse = train_df[ELAPSE].unique()\n    n_elapse = 300 # max(elapse)\n    print(\"Number of skills\", n_elapse)\n    \n    batch_size = 100_000\n    batches = []\n    for i in range(int(train_df[USER_ID].nunique()//batch_size)+1):\n        batch = [i * batch_size, min((i + 1) * batch_size, train_df[USER_ID].nunique())]\n        batches.append(batch)\n\n\n    picked_user = []\n    for i, batch in enumerate(batches):\n        u_s = train_df[USER_ID].unique()[batch[0]]\n        u_e = train_df[USER_ID].unique()[batch[1]-1]\n        picked_user.append([u_s, u_e])\n\n    indx = []\n    for i, batch in enumerate(picked_user):\n        idx_s = list(train_df[train_df[USER_ID] == batch[0]].index)[0]\n        idx_e = list(train_df[train_df[USER_ID] == batch[1]].index)[-1]\n        indx.append([idx_s, idx_e])\n\n    for i, batch in enumerate(indx):\n        sub_df = train_df.loc[batch[0]:batch[1]][[USER_ID, CONTENT_ID, \n                                                  TASK_CONTAINER_ID, \n                                                  PART, DIFF, ELAPSE, \n                                                  CORR, TARGET]].groupby(USER_ID).apply(lambda r: (r[CONTENT_ID].values, \n                                                                                                   r[TASK_CONTAINER_ID].values, \n                                                                                                   r[PART].values, \n                                                                                                   r[DIFF].values, \n                                                                                                   r[ELAPSE].values, \n                                                                                                   r[CORR].values, \n                                                                                                   r[TARGET].values))\n        sub_df.to_pickle(f'train_group_{i}.pkl')\n        del sub_df\n\n    del train_df; gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:19.26078Z","iopub.status.busy":"2021-01-01T18:21:19.260053Z","iopub.status.idle":"2021-01-01T18:21:47.654936Z","shell.execute_reply":"2021-01-01T18:21:47.654167Z"},"papermill":{"duration":28.451669,"end_time":"2021-01-01T18:21:47.655052","exception":false,"start_time":"2021-01-01T18:21:19.203383","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train_group = []\nfor i in range(4):\n    train_group.append(pd.read_pickle(os.path.join(r'../input/riiid-chunking', f'train_group_{i}.pkl')))\n    \ntrain_group = pd.concat(train_group)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:47.765906Z","iopub.status.busy":"2021-01-01T18:21:47.7653Z","iopub.status.idle":"2021-01-01T18:21:47.782633Z","shell.execute_reply":"2021-01-01T18:21:47.782132Z"},"papermill":{"duration":0.075712,"end_time":"2021-01-01T18:21:47.782767","exception":false,"start_time":"2021-01-01T18:21:47.707055","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv(\n    '../input/riiid-test-answer-prediction/questions.csv', \n    usecols = [0, 3], \n    dtype = {'question_id': 'int16', 'part': 'int8'}\n)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.051209,"end_time":"2021-01-01T18:21:47.887385","exception":false,"start_time":"2021-01-01T18:21:47.836176","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Training/Validation split strategy with r3id_info_public"},{"metadata":{"papermill":{"duration":0.051372,"end_time":"2021-01-01T18:21:47.99298","exception":false,"start_time":"2021-01-01T18:21:47.941608","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Define some special tokens"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:48.101962Z","iopub.status.busy":"2021-01-01T18:21:48.101249Z","iopub.status.idle":"2021-01-01T18:21:48.105157Z","shell.execute_reply":"2021-01-01T18:21:48.105614Z"},"papermill":{"duration":0.061276,"end_time":"2021-01-01T18:21:48.105745","exception":false,"start_time":"2021-01-01T18:21:48.044469","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"PAD_TOKEN = -1\nSTART_TOKEN = -2\nMASK_TOKEN = -3\n\nspecial_token = [PAD_TOKEN, START_TOKEN, MASK_TOKEN]\n\nPAD_ID = 0\nSTART_ID = 1\nMASK_ID = 2\nRESPONSE_FALSE_ID = 3\nRESPONSE_TRUE_ID = 4","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.051743,"end_time":"2021-01-01T18:21:48.211274","exception":false,"start_time":"2021-01-01T18:21:48.159531","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Mapping for old responses"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:48.319525Z","iopub.status.busy":"2021-01-01T18:21:48.318917Z","iopub.status.idle":"2021-01-01T18:21:48.322984Z","shell.execute_reply":"2021-01-01T18:21:48.322374Z"},"papermill":{"duration":0.060193,"end_time":"2021-01-01T18:21:48.323099","exception":false,"start_time":"2021-01-01T18:21:48.262906","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"responses_total_tokens = np.concatenate((special_token, [0, 1]))\nquestions_encoding_mapper = dict(zip(responses_total_tokens, np.arange(len(responses_total_tokens))))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:48.432243Z","iopub.status.busy":"2021-01-01T18:21:48.43157Z","iopub.status.idle":"2021-01-01T18:21:48.448492Z","shell.execute_reply":"2021-01-01T18:21:48.448Z"},"papermill":{"duration":0.07338,"end_time":"2021-01-01T18:21:48.448579","exception":false,"start_time":"2021-01-01T18:21:48.375199","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Import the pre-defined indexes\nimport json\nwith open(os.path.join(r'../input/r3id-info-public', 'train_valid_split_indices_fold_1.json')) as json_file:\n    splitting_index = json.load(json_file)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.051637,"end_time":"2021-01-01T18:21:48.552072","exception":false,"start_time":"2021-01-01T18:21:48.500435","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Add the START_TOKEN"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:48.661408Z","iopub.status.busy":"2021-01-01T18:21:48.660527Z","iopub.status.idle":"2021-01-01T18:21:54.000103Z","shell.execute_reply":"2021-01-01T18:21:54.000767Z"},"papermill":{"duration":5.397177,"end_time":"2021-01-01T18:21:54.000948","exception":false,"start_time":"2021-01-01T18:21:48.603771","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def add_start_token(x):\n    return (*x, np.append([START_TOKEN], x[-1][:-1]))\ntrain_group = train_group.apply(add_start_token)\ntrain_group","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:21:54.120403Z","iopub.status.busy":"2021-01-01T18:21:54.119498Z","iopub.status.idle":"2021-01-01T18:22:00.694506Z","shell.execute_reply":"2021-01-01T18:22:00.694009Z"},"papermill":{"duration":6.635367,"end_time":"2021-01-01T18:22:00.694617","exception":false,"start_time":"2021-01-01T18:21:54.05925","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Extracting the train_df\ndef train_extraction(user):\n    if str(user) not in splitting_index.keys():\n        return train_group[user]\n    else:\n        if splitting_index[str(user)] == 0:\n            return np.nan\n        else:\n            return tuple([seq[:splitting_index[str(user)]] for seq in train_group[user]])\n        \n# Apply\nsub_train_ = train_group.copy()\nfor i in tqdm(train_group.index):\n    sub_train_[i] = train_extraction(i)\nsub_train_.dropna(inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:00.807815Z","iopub.status.busy":"2021-01-01T18:22:00.806939Z","iopub.status.idle":"2021-01-01T18:22:00.826426Z","shell.execute_reply":"2021-01-01T18:22:00.825927Z"},"papermill":{"duration":0.077463,"end_time":"2021-01-01T18:22:00.826522","exception":false,"start_time":"2021-01-01T18:22:00.749059","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"sub_train_[115]","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:00.945498Z","iopub.status.busy":"2021-01-01T18:22:00.944454Z","iopub.status.idle":"2021-01-01T18:22:05.300072Z","shell.execute_reply":"2021-01-01T18:22:05.299403Z"},"papermill":{"duration":4.41965,"end_time":"2021-01-01T18:22:05.30021","exception":false,"start_time":"2021-01-01T18:22:00.88056","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Extracting the valid_df\ndef valid_extraction(user):\n    if str(user) not in splitting_index.keys():\n        return np.nan\n    else:\n        if splitting_index[str(user)] == 0:\n            return train_group[user]\n        else:\n            return tuple([seq[splitting_index[str(user)]:] for seq in train_group[user]])\n\n# Apply\nvalid_df = train_group.copy()\nfor i in tqdm(train_group.index):\n    valid_df[i] = valid_extraction(i)\nvalid_df.dropna(inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:05.428242Z","iopub.status.busy":"2021-01-01T18:22:05.427505Z","iopub.status.idle":"2021-01-01T18:22:05.431116Z","shell.execute_reply":"2021-01-01T18:22:05.430612Z"},"papermill":{"duration":0.073031,"end_time":"2021-01-01T18:22:05.431219","exception":false,"start_time":"2021-01-01T18:22:05.358188","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def valid_2_sequence(user, valid_seqs, idx, train_seqs = None):\n    # Take the task_seq, determine the positional index and the number of blocks\n    task_seq = valid_seqs[1]\n    if train_seqs is not None:\n        seq_len_train = len(train_seqs[0])\n    pos_idx = np.cumsum([True] + [i != j for i, j in zip(task_seq, task_seq[1:])])\n    num_blocks = pos_idx[-1]\n    \n    # For how many blocks, create that number of subsequences, each subsequence contains only one block\n    user_val_seq = []\n    idx_end = np.where(pos_idx == (idx + 1))[0][-1] + 1\n    if train_seqs is not None:\n        user_val_seq.append(tuple([np.concatenate((train_seq, valid_seq[:idx_end])) \n                                   for train_seq, valid_seq in zip(train_seqs, valid_seqs)]))\n        old_res_mask = np.array([1] * len(train_seqs[0]) + [0] * len(valid_seqs[0][:idx_end]))\n    else:\n        user_val_seq.append(tuple([np.array(valid_seq[:idx_end]) for valid_seq in valid_seqs]))\n        old_res_mask = np.zeros(len(valid_seqs[0][:idx_end]))\n        \n    return user_val_seq[0], old_res_mask","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.056409,"end_time":"2021-01-01T18:22:05.545191","exception":false,"start_time":"2021-01-01T18:22:05.488782","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:05.66411Z","iopub.status.busy":"2021-01-01T18:22:05.663353Z","iopub.status.idle":"2021-01-01T18:22:05.668299Z","shell.execute_reply":"2021-01-01T18:22:05.667807Z"},"papermill":{"duration":0.066427,"end_time":"2021-01-01T18:22:05.668401","exception":false,"start_time":"2021-01-01T18:22:05.601974","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"user_val_seq, old_res_mask = valid_2_sequence(115, valid_df[115], 19, train_seqs = sub_train_[115])","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:05.788145Z","iopub.status.busy":"2021-01-01T18:22:05.787204Z","iopub.status.idle":"2021-01-01T18:22:05.796704Z","shell.execute_reply":"2021-01-01T18:22:05.796039Z"},"papermill":{"duration":0.070987,"end_time":"2021-01-01T18:22:05.796798","exception":false,"start_time":"2021-01-01T18:22:05.725811","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"user_val_seq","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:05.918615Z","iopub.status.busy":"2021-01-01T18:22:05.917814Z","iopub.status.idle":"2021-01-01T18:22:05.921251Z","shell.execute_reply":"2021-01-01T18:22:05.92181Z"},"papermill":{"duration":0.067258,"end_time":"2021-01-01T18:22:05.921937","exception":false,"start_time":"2021-01-01T18:22:05.854679","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"old_res_mask","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:06.054612Z","iopub.status.busy":"2021-01-01T18:22:06.053726Z","iopub.status.idle":"2021-01-01T18:22:06.056518Z","shell.execute_reply":"2021-01-01T18:22:06.056044Z"},"papermill":{"duration":0.076576,"end_time":"2021-01-01T18:22:06.056615","exception":false,"start_time":"2021-01-01T18:22:05.980039","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def num_subsequence_each_user(user, valid_seqs):\n    # Take the task_seq, determine the positional index and the number of blocks\n    task_seq = valid_seqs[1]\n    pos_idx = np.cumsum([True] + [i != j for i, j in zip(task_seq, task_seq[1:])])\n    num_blocks = pos_idx[-1]\n    return num_blocks","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.057977,"end_time":"2021-01-01T18:22:06.173112","exception":false,"start_time":"2021-01-01T18:22:06.115135","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:06.295511Z","iopub.status.busy":"2021-01-01T18:22:06.294546Z","iopub.status.idle":"2021-01-01T18:22:06.300336Z","shell.execute_reply":"2021-01-01T18:22:06.299709Z"},"papermill":{"duration":0.068958,"end_time":"2021-01-01T18:22:06.300443","exception":false,"start_time":"2021-01-01T18:22:06.231485","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"num_subsequence_each_user(115, valid_df[115])","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:06.424706Z","iopub.status.busy":"2021-01-01T18:22:06.424056Z","iopub.status.idle":"2021-01-01T18:22:06.428567Z","shell.execute_reply":"2021-01-01T18:22:06.428068Z"},"papermill":{"duration":0.069054,"end_time":"2021-01-01T18:22:06.428677","exception":false,"start_time":"2021-01-01T18:22:06.359623","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def data_len_calculator(valid_df):\n    return np.cumsum([num_subsequence_each_user(i, valid_df[i]) for i in valid_df.index])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.058333,"end_time":"2021-01-01T18:22:06.545564","exception":false,"start_time":"2021-01-01T18:22:06.487231","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:07.055387Z","iopub.status.busy":"2021-01-01T18:22:07.002765Z","iopub.status.idle":"2021-01-01T18:22:07.417043Z","shell.execute_reply":"2021-01-01T18:22:07.416242Z"},"papermill":{"duration":0.812686,"end_time":"2021-01-01T18:22:07.417154","exception":false,"start_time":"2021-01-01T18:22:06.604468","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"data_len_calculator(valid_df)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.059928,"end_time":"2021-01-01T18:22:07.537715","exception":false,"start_time":"2021-01-01T18:22:07.477787","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Design some masking functions"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:07.668882Z","iopub.status.busy":"2021-01-01T18:22:07.668258Z","iopub.status.idle":"2021-01-01T18:22:07.672418Z","shell.execute_reply":"2021-01-01T18:22:07.67195Z"},"papermill":{"duration":0.074015,"end_time":"2021-01-01T18:22:07.672516","exception":false,"start_time":"2021-01-01T18:22:07.598501","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def position(task, pad_include = True, bundle_ignore = True):\n    if not bundle_ignore:\n        if pad_include:\n            num_padded = len(task[task == PAD_TOKEN])\n            task = task[task != PAD_TOKEN]\n        else:\n            num_padded = 0\n        # Position ids\n        # It depends on the task container\n        pos = np.cumsum([0] * num_padded + [True] + [i != j for i, j in zip(task, task[1:])])\n    else:\n        if pad_include:\n            num_padded = len(task[task == PAD_TOKEN])\n            task = task[task != PAD_TOKEN]\n        else:\n            num_padded = 0\n        pos = np.array([0] * num_padded + list(range(1, len(task) + 1)))\n    return pos","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.059396,"end_time":"2021-01-01T18:22:07.792385","exception":false,"start_time":"2021-01-01T18:22:07.732989","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Prediction mask"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:07.92116Z","iopub.status.busy":"2021-01-01T18:22:07.919176Z","iopub.status.idle":"2021-01-01T18:22:07.921956Z","shell.execute_reply":"2021-01-01T18:22:07.922427Z"},"papermill":{"duration":0.07007,"end_time":"2021-01-01T18:22:07.92254","exception":false,"start_time":"2021-01-01T18:22:07.85247","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def prediction_mask(pos):\n    # This return an array, where 0 shows the position that we don't predict and 1 is the position we do prediction\n    pred_mask = 1 - np.sign(np.cumsum([False] + [i != j for i, j in zip(pos[::-1][1:], pos[::-1])]))[::-1]\n    return pred_mask","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.059895,"end_time":"2021-01-01T18:22:08.042478","exception":false,"start_time":"2021-01-01T18:22:07.982583","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:08.170923Z","iopub.status.busy":"2021-01-01T18:22:08.170109Z","iopub.status.idle":"2021-01-01T18:22:08.175981Z","shell.execute_reply":"2021-01-01T18:22:08.176485Z"},"papermill":{"duration":0.074682,"end_time":"2021-01-01T18:22:08.176598","exception":false,"start_time":"2021-01-01T18:22:08.101916","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"task = user_val_seq[1]\nprint(f'Position ids: {position(task, pad_include = False, bundle_ignore = False)}')\nprint(f'Prediction_mask: {prediction_mask(position(task, pad_include = False, bundle_ignore = False))}')\n\nprint(f'Position ids with bundle ignore: {position(task, pad_include = False)}')\nprint(f'Prediction_mask with bundle ignore: {prediction_mask(position(task, pad_include = False))}')","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:08.305241Z","iopub.status.busy":"2021-01-01T18:22:08.304524Z","iopub.status.idle":"2021-01-01T18:22:08.311025Z","shell.execute_reply":"2021-01-01T18:22:08.311753Z"},"papermill":{"duration":0.073499,"end_time":"2021-01-01T18:22:08.311893","exception":false,"start_time":"2021-01-01T18:22:08.238394","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"task = np.concatenate(([-1] * 5, user_val_seq[1]))\nprint(f'Position ids: {position(task, pad_include = True, bundle_ignore = False)}')\nprint(f'Prediction_mask: {prediction_mask(position(task, pad_include = True, bundle_ignore = False))}')\n\nprint(f'Position ids with bundle ignore: {position(task, pad_include = True)}')\nprint(f'Prediction_mask with bundle ignore: {prediction_mask(position(task, pad_include = True))}')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.059018,"end_time":"2021-01-01T18:22:08.431622","exception":false,"start_time":"2021-01-01T18:22:08.372604","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Old responses to index"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:08.559028Z","iopub.status.busy":"2021-01-01T18:22:08.558272Z","iopub.status.idle":"2021-01-01T18:22:08.562002Z","shell.execute_reply":"2021-01-01T18:22:08.562444Z"},"papermill":{"duration":0.071533,"end_time":"2021-01-01T18:22:08.562567","exception":false,"start_time":"2021-01-01T18:22:08.491034","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def old_response_to_index(old_res, old_res_mask, use_mask = True):\n    if use_mask:\n        masked_seq = old_res * old_res_mask\n    else:\n        masked_seq = old_res\n    # Start padding\n    pad_mask = (masked_seq == PAD_TOKEN).astype(int) * PAD_ID\n    start_mask = (masked_seq == START_TOKEN).astype(int) * START_ID\n    false_mask = (masked_seq == 0).astype(int) * RESPONSE_FALSE_ID\n    true_mask = (masked_seq == 1).astype(int) * RESPONSE_TRUE_ID\n    if use_mask:\n        return (pad_mask + start_mask + false_mask + true_mask) * old_res_mask + (1 - old_res_mask) * MASK_ID\n    else:\n        return pad_mask + start_mask + false_mask + true_mask","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.060195,"end_time":"2021-01-01T18:22:08.682191","exception":false,"start_time":"2021-01-01T18:22:08.621996","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check: validation"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:08.808785Z","iopub.status.busy":"2021-01-01T18:22:08.807974Z","iopub.status.idle":"2021-01-01T18:22:08.811792Z","shell.execute_reply":"2021-01-01T18:22:08.811307Z"},"papermill":{"duration":0.068457,"end_time":"2021-01-01T18:22:08.811891","exception":false,"start_time":"2021-01-01T18:22:08.743434","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"sample_seq = user_val_seq[-1]\nold_response_to_index(sample_seq, old_res_mask)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:08.943827Z","iopub.status.busy":"2021-01-01T18:22:08.943022Z","iopub.status.idle":"2021-01-01T18:22:08.946362Z","shell.execute_reply":"2021-01-01T18:22:08.94681Z"},"papermill":{"duration":0.074532,"end_time":"2021-01-01T18:22:08.946935","exception":false,"start_time":"2021-01-01T18:22:08.872403","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"user_val_seq[-1]","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:09.074297Z","iopub.status.busy":"2021-01-01T18:22:09.073465Z","iopub.status.idle":"2021-01-01T18:22:09.076826Z","shell.execute_reply":"2021-01-01T18:22:09.07734Z"},"papermill":{"duration":0.070454,"end_time":"2021-01-01T18:22:09.077457","exception":false,"start_time":"2021-01-01T18:22:09.007003","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Check with train dataset\nsample_seq = sub_train_[115][-1]\nold_res_mask = np.ones(len(sub_train_[115][-1]))\n\nold_response_to_index(sample_seq, old_res_mask)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.061018,"end_time":"2021-01-01T18:22:09.19934","exception":false,"start_time":"2021-01-01T18:22:09.138322","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Datasets"},{"metadata":{"papermill":{"duration":0.060975,"end_time":"2021-01-01T18:22:09.321457","exception":false,"start_time":"2021-01-01T18:22:09.260482","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Training Dataset"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:09.448194Z","iopub.status.busy":"2021-01-01T18:22:09.447292Z","iopub.status.idle":"2021-01-01T18:22:09.483526Z","shell.execute_reply":"2021-01-01T18:22:09.48305Z"},"papermill":{"duration":0.100893,"end_time":"2021-01-01T18:22:09.483626","exception":false,"start_time":"2021-01-01T18:22:09.382733","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class RIIID_Train_Dataset(Dataset):\n    def __init__(self, group, max_seq = 100):\n        self.group = group\n        self.max_seq = max_seq\n        \n        # Discard users with too short sequences\n        self.user_ids = []\n        for i, user_id in enumerate(group.index):\n            q = group[user_id][0]\n            if len(q) < 2: # 10 interactions minimum\n                continue\n            self.user_ids.append(user_id)\n            \n    def __len__(self):\n        return len(self.user_ids)\n    \n    def __getitem__(self, idx):\n        # Pick user\n        user_id = self.user_ids[idx]\n        \n        # Unpack sequences\n        ques_, task_, part_, diff_, elapse_, corr_, target_, old_res_ = self.group[user_id]\n        \n        # Old responses mask\n        old_res_mask_ = np.ones(len(old_res_))\n        pred_mask_ = old_res_mask_\n        old_res_ = old_response_to_index(old_res_, old_res_mask_)\n        \n        seq_len = len(ques_)\n        \n        # Position\n        pos_ = np.arange(seq_len)\n        \n        # Create arrays to pad the sequences\n        ques = np.zeros(self.max_seq, dtype = int) - 1\n        task = np.zeros(self.max_seq, dtype = int) - 1\n        part = np.zeros(self.max_seq, dtype = int)\n        diff = np.zeros(self.max_seq, dtype = float)\n        elapse = np.zeros(self.max_seq, dtype = float)\n        corr = np.zeros(self.max_seq, dtype = float)\n        target = np.zeros(self.max_seq, dtype = int)\n        old_res = np.zeros(self.max_seq, dtype = int)\n        pred_mask = np.zeros(self.max_seq, dtype = int)\n        \n        if seq_len >= self.max_seq:\n            if seq_len > self.max_seq:\n                # For the training set, if the sequences are longer than the max_seq, random sampling a sub-sequence\n                start_index = np.random.randint(seq_len - self.max_seq)\n                ques = ques_[start_index:(start_index + self.max_seq)]\n                task = task_[start_index:(start_index + self.max_seq)]\n                part = part_[start_index:(start_index + self.max_seq)]\n                diff = diff_[start_index:(start_index + self.max_seq)]\n                elapse = elapse_[start_index:(start_index + self.max_seq)]\n                corr = corr_[start_index:(start_index + self.max_seq)]\n                target = target_[start_index:(start_index + self.max_seq)]\n                old_res = old_res_[start_index:(start_index + self.max_seq)]\n                pred_mask = pred_mask_[start_index:(start_index + self.max_seq)]\n            else:\n                ques = ques_[-self.max_seq:]\n                task = task_[-self.max_seq:]\n                part = part_[-self.max_seq:]\n                diff = diff_[-self.max_seq:]\n                elapse = elapse_[-self.max_seq:]\n                corr = corr_[-self.max_seq:]\n                target = target_[-self.max_seq:]\n                old_res = old_res_[-self.max_seq:]\n                pred_mask = pred_mask_[-self.max_seq:]\n        else:\n            ques[-seq_len:] = ques_\n            task[-seq_len:] = task_\n            part[-seq_len:] = part_\n            diff[-seq_len:] = diff_\n            elapse[-seq_len:] = elapse_\n            corr[-seq_len:] = corr_\n            target[-seq_len:] = target_\n            old_res[-seq_len:] = old_res_\n            pred_mask[-seq_len:] = pred_mask_\n        \n        task_pos = position(task, bundle_ignore = False)\n        \n        e_pos = position(task)\n        d_pos = np.concatenate(([0], e_pos[:-1]))\n        \n        # Return the sequences\n        return {\n            'user': user_id, \n            'content_id': torch.tensor(ques, dtype = torch.long) + 1, \n            'task_container_id': torch.tensor(task, dtype = torch.long) + 1, \n            'part_id': torch.tensor(part, dtype = torch.long), \n            'diff_id': torch.tensor(diff, dtype = torch.float32), \n            'prior_elapsed_time_id': torch.tensor(elapse, dtype = torch.float32), \n            'user_correctness_id': torch.tensor(corr, dtype = torch.float32), \n            'old_response_id': torch.tensor(old_res, dtype = torch.long), \n            'encoder_position_id': torch.tensor(e_pos, dtype = torch.long), \n            'decoder_position_id': torch.tensor(d_pos, dtype = torch.long), \n            'task_position_id': torch.tensor(task_pos, dtype = torch.long), \n            'prediction_mask': torch.tensor(pred_mask, dtype = torch.long), \n            'target':  torch.tensor(target, dtype = torch.float32)\n        }","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.061151,"end_time":"2021-01-01T18:22:09.606712","exception":false,"start_time":"2021-01-01T18:22:09.545561","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Validation dataset"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:09.759349Z","iopub.status.busy":"2021-01-01T18:22:09.749012Z","iopub.status.idle":"2021-01-01T18:22:09.768181Z","shell.execute_reply":"2021-01-01T18:22:09.767527Z"},"papermill":{"duration":0.100436,"end_time":"2021-01-01T18:22:09.768284","exception":false,"start_time":"2021-01-01T18:22:09.667848","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class RIIID_Valid_Dataset(Dataset):\n    def __init__(self, valid, train, max_seq = 100):\n        self.valid_df = valid\n        self.train_df = train\n        self.max_seq = max_seq\n        self.users = list(self.valid_df.index)\n        self.len_valid = np.array([0] + data_len_calculator(self.valid_df).tolist())\n        \n    def __len__(self):\n        return self.len_valid[-1]\n    \n    def __getitem__(self, idx):\n        # Idea:\n        # For the index (idx), determine which user that index shows\n        # E.g. The user number 115 has 20 subsequences, if the index is 7, then it will return 0 (the index of user 115)\n        user_idx = np.argmax(self.len_valid[self.len_valid <= idx])\n        user = self.users[user_idx]\n        user_seq_valid = self.valid_df[user]\n        if user in self.train_df.index:\n            user_seq_train = self.train_df[user]\n        else:\n            user_seq_train = None\n        # Extract the valid sequence\n        \n        begin_idx = self.len_valid[self.len_valid <= idx][-1]\n        idx_in_user_batch = idx - begin_idx\n        \n        # Unpack sequences\n        (ques_, task_, part_, diff_, elapse_, corr_, target_, old_res_), old_res_mask_ = valid_2_sequence(user, user_seq_valid, idx_in_user_batch, \n                                                                                                          train_seqs = user_seq_train)\n        # Old response to index\n        old_res_ = old_response_to_index(old_res_, old_res_mask_, use_mask = False)\n        \n        seq_len = len(ques_)\n        \n        # Position\n        pos_ = np.arange(seq_len)\n        \n        # Create arrays to pad the sequences\n        ques = np.zeros(self.max_seq, dtype = int) - 1\n        task = np.zeros(self.max_seq, dtype = int) - 1\n        part = np.zeros(self.max_seq, dtype = int)\n        diff = np.zeros(self.max_seq, dtype = float)\n        elapse = np.zeros(self.max_seq, dtype = float)\n        corr = np.zeros(self.max_seq, dtype = float)\n        target = np.zeros(self.max_seq, dtype = int)\n        old_res = np.zeros(self.max_seq, dtype = int)\n        \n        if seq_len >= self.max_seq:\n            ques = ques_[-self.max_seq:]\n            task = task_[-self.max_seq:]\n            part = part_[-self.max_seq:]\n            diff = diff_[-self.max_seq:]\n            elapse = elapse_[-self.max_seq:]\n            corr = corr_[-self.max_seq:]\n            target = target_[-self.max_seq:]\n            old_res = old_res_[-self.max_seq:]\n        else:\n            ques[-seq_len:] = ques_\n            task[-seq_len:] = task_\n            part[-seq_len:] = part_\n            diff[-seq_len:] = diff_\n            elapse[-seq_len:] = elapse_\n            corr[-seq_len:] = corr_\n            target[-seq_len:] = target_\n            old_res[-seq_len:] = old_res_\n            \n        # Prediction mask\n        task_pos = position(task, pad_include = True, bundle_ignore = False)\n        pred_mask = prediction_mask(task_pos)\n        #pred_mask = (old_res == MASK_ID).astype(int)\n        \n        e_pos = position(task)\n        d_pos = np.concatenate(([0], e_pos[:-1]))\n        \n        return {\n            'user': user, \n            'content_id': torch.tensor(ques, dtype = torch.long) + 1, \n            'task_container_id': torch.tensor(task, dtype = torch.long) + 1, \n            'part_id': torch.tensor(part, dtype = torch.long), \n            'diff_id': torch.tensor(diff, dtype = torch.float32), \n            'prior_elapsed_time_id': torch.tensor(elapse, dtype = torch.float32), \n            'user_correctness_id': torch.tensor(corr, dtype = torch.float32), \n            'old_response_id': torch.tensor(old_res, dtype = torch.long), \n            'encoder_position_id': torch.tensor(e_pos, dtype = torch.long), \n            'decoder_position_id': torch.tensor(d_pos, dtype = torch.long), \n            'task_position_id': torch.tensor(task_pos, dtype = torch.long), \n            'prediction_mask': torch.tensor(pred_mask, dtype = torch.long), \n            'target':  torch.tensor(target, dtype = torch.float32)\n        }","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.065884,"end_time":"2021-01-01T18:22:09.900917","exception":false,"start_time":"2021-01-01T18:22:09.835033","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:10.06403Z","iopub.status.busy":"2021-01-01T18:22:10.053736Z","iopub.status.idle":"2021-01-01T18:22:13.300175Z","shell.execute_reply":"2021-01-01T18:22:13.299632Z"},"papermill":{"duration":3.337496,"end_time":"2021-01-01T18:22:13.300293","exception":false,"start_time":"2021-01-01T18:22:09.962797","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train_dataset = RIIID_Train_Dataset(sub_train_, 10)\ntrain_dataloader = DataLoader(train_dataset, batch_size = 5, shuffle = False, num_workers = 4)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:13.431799Z","iopub.status.busy":"2021-01-01T18:22:13.431133Z","iopub.status.idle":"2021-01-01T18:22:13.969413Z","shell.execute_reply":"2021-01-01T18:22:13.968926Z"},"papermill":{"duration":0.607378,"end_time":"2021-01-01T18:22:13.969518","exception":false,"start_time":"2021-01-01T18:22:13.36214","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"next(iter(train_dataloader))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:14.902408Z","iopub.status.busy":"2021-01-01T18:22:14.901744Z","iopub.status.idle":"2021-01-01T18:22:14.941231Z","shell.execute_reply":"2021-01-01T18:22:14.940697Z"},"papermill":{"duration":0.891367,"end_time":"2021-01-01T18:22:14.941344","exception":false,"start_time":"2021-01-01T18:22:14.049977","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"valid_dataset = RIIID_Valid_Dataset(valid_df, sub_train_, max_seq = 10)\nvalid_dataloader = DataLoader(valid_dataset, batch_size = 5, shuffle = False, num_workers = 0)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:15.072398Z","iopub.status.busy":"2021-01-01T18:22:15.071134Z","iopub.status.idle":"2021-01-01T18:22:15.195125Z","shell.execute_reply":"2021-01-01T18:22:15.195613Z"},"papermill":{"duration":0.191444,"end_time":"2021-01-01T18:22:15.195757","exception":false,"start_time":"2021-01-01T18:22:15.004313","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"next(iter(valid_dataloader))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.064234,"end_time":"2021-01-01T18:22:15.327219","exception":false,"start_time":"2021-01-01T18:22:15.262985","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Model"},{"metadata":{"papermill":{"duration":0.064128,"end_time":"2021-01-01T18:22:15.455912","exception":false,"start_time":"2021-01-01T18:22:15.391784","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Model Utils"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:15.614971Z","iopub.status.busy":"2021-01-01T18:22:15.614194Z","iopub.status.idle":"2021-01-01T18:22:15.617136Z","shell.execute_reply":"2021-01-01T18:22:15.617687Z"},"papermill":{"duration":0.097644,"end_time":"2021-01-01T18:22:15.617821","exception":false,"start_time":"2021-01-01T18:22:15.520177","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import copy\n\nclass ScheduledOptim():\n    '''A simple wrapper class for learning rate scheduling'''\n    def __init__(self, optimizer, d_model, n_warmup_steps):\n        self._optimizer = optimizer\n        self.n_warmup_steps = n_warmup_steps\n        self.n_current_steps = 0\n        self.init_lr = np.power(d_model, -0.5)\n\n    def step_and_update_lr(self):\n        \"Step with the inner optimizer\"\n        self._update_learning_rate()\n        self._optimizer.step()\n\n    def zero_grad(self):\n        \"Zero out the gradients by the inner optimizer\"\n        self._optimizer.zero_grad()\n\n    def _get_lr_scale(self):\n        return np.min([\n            np.power(self.n_current_steps, -0.5),\n            np.power(self.n_warmup_steps, -1.5) * self.n_current_steps\n        ])\n\n    def _update_learning_rate(self):\n        ''' Learning rate scheduling per step '''\n\n        self.n_current_steps += 1\n        lr = self.init_lr * self._get_lr_scale()\n\n        for param_group in self._optimizer.param_groups:\n            param_group['lr'] = lr\n\n\nclass NoamOpt:\n    \"Optim wrapper that implements rate.\"\n\n    def __init__(self, model_size, factor, warmup, optimizer):\n        self.optimizer = optimizer\n        self._step = 0\n        self.warmup = warmup\n        self.factor = factor\n        self.model_size = model_size\n        self._rate = 0\n\n    def zero_grad(self):\n        self.optimizer.zero_grad()\n\n    def step(self):\n        \"Update parameters and rate\"\n        self._step += 1\n        rate = self.rate()\n        for p in self.optimizer.param_groups:\n            p['lr'] = rate\n        self._rate = rate\n        self.optimizer.step()\n\n    def rate(self, step=None):\n        \"Implement `lrate` above\"\n        if step is None:\n            step = self._step\n        return self.factor * \\\n               (self.model_size ** (-0.5) *\n                min(step ** (-0.5), step * self.warmup ** (-1.5)))\n\nclass NoamOptimizer:\n    def __init__(self, model, lr, model_size, warmup):\n        self._adam = torch.optim.Adam(model.parameters(), lr=lr)\n        self._opt = NoamOpt(\n            model_size=model_size, factor=1, warmup=warmup, optimizer=self._adam)\n\n    def step(self, loss):\n        self._opt.zero_grad()\n        loss.backward()\n        self._opt.step()\n\n# For Transformer-based models\ndef get_pad_mask(seq, pad_idx):\n    return (seq != pad_idx).unsqueeze(-2)\n\ndef get_subsequent_mask_3d(seq, only_before = True):\n    batch_size, seq_len = seq.shape\n    a = seq.unsqueeze(-1).expand(batch_size, seq_len, seq_len)\n    b = seq.unsqueeze(1).expand(batch_size, seq_len, seq_len) + int(only_before)\n    return (a >= b)\n\ndef get_subsequent_mask(seq):\n    ''' For masking out the subsequent info. '''\n    sz_b, len_s = seq.size()\n    subsequent_mask = (1 - torch.triu(torch.ones((1, len_s, len_s), device=seq.device), diagonal=1)).bool()\n    return subsequent_mask\n\ndef get_masks(seq, pad_idx, only_before = True):\n    encoder_mask = (get_pad_mask(seq, pad_idx) & get_subsequent_mask_3d(seq, only_before = only_before))\n    decoder_mask = (get_pad_mask(seq, pad_idx) & get_subsequent_mask(seq))\n    encoder_decoder_mask = encoder_mask\n    return encoder_mask, decoder_mask, encoder_decoder_mask\n\ndef clones(module, N):\n    \"Produce N identical layers.\"\n    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.064617,"end_time":"2021-01-01T18:22:15.747328","exception":false,"start_time":"2021-01-01T18:22:15.682711","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Check"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:15.883827Z","iopub.status.busy":"2021-01-01T18:22:15.882832Z","iopub.status.idle":"2021-01-01T18:22:15.891027Z","shell.execute_reply":"2021-01-01T18:22:15.890356Z"},"papermill":{"duration":0.07865,"end_time":"2021-01-01T18:22:15.891141","exception":false,"start_time":"2021-01-01T18:22:15.812491","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"pos = torch.tensor([[0,0,0,1,2,2,2,3,4,5],\n                    [0,0,0,0,1,2,3,3,4,4]])\n\nget_subsequent_mask_3d(pos)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:16.027279Z","iopub.status.busy":"2021-01-01T18:22:16.025951Z","iopub.status.idle":"2021-01-01T18:22:16.031779Z","shell.execute_reply":"2021-01-01T18:22:16.031217Z"},"papermill":{"duration":0.074807,"end_time":"2021-01-01T18:22:16.031889","exception":false,"start_time":"2021-01-01T18:22:15.957082","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"get_pad_mask(pos, 0)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:16.168768Z","iopub.status.busy":"2021-01-01T18:22:16.168123Z","iopub.status.idle":"2021-01-01T18:22:16.187185Z","shell.execute_reply":"2021-01-01T18:22:16.186679Z"},"papermill":{"duration":0.089652,"end_time":"2021-01-01T18:22:16.18729","exception":false,"start_time":"2021-01-01T18:22:16.097638","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"get_masks(pos, pad_idx = 0)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.065934,"end_time":"2021-01-01T18:22:16.319573","exception":false,"start_time":"2021-01-01T18:22:16.253639","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Building-block layers"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:16.489027Z","iopub.status.busy":"2021-01-01T18:22:16.475996Z","iopub.status.idle":"2021-01-01T18:22:16.491328Z","shell.execute_reply":"2021-01-01T18:22:16.491788Z"},"papermill":{"duration":0.106334,"end_time":"2021-01-01T18:22:16.491909","exception":false,"start_time":"2021-01-01T18:22:16.385575","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def attention(query, key, value, mask=None, dropout=None):\n    \"Compute 'Scaled Dot Product Attention'\"\n    d_k = query.size(-1)\n    scores = torch.matmul(query, key.transpose(-2, -1)) \\\n             / math.sqrt(d_k)\n    if mask is not None:\n        scores = scores.masked_fill(mask == 0, -1e9)\n    p_attn = F.softmax(scores, dim=-1)\n    if dropout is not None:\n        p_attn = dropout(p_attn)\n    return torch.matmul(p_attn, value), p_attn\n\n\nclass MultiHeadedAttention(nn.Module):\n    def __init__(self, h, d_model, dropout=0.1):\n        \"Take in model size and number of heads.\"\n        super(MultiHeadedAttention, self).__init__()\n        assert d_model % h == 0\n        # We assume d_v always equals d_k\n        self.d_k = d_model // h\n        self.h = h\n        self.linears = clones(nn.Linear(d_model, d_model, bias=False), 4) # Q, K, V, last\n        self.attn = None\n        self.dropout = nn.Dropout(p=dropout)\n\n    def forward(self, query, key, value, mask=None):\n        \"Implements Figure 2\"\n        if mask is not None:\n            # Same mask applied to all h heads.\n            mask = mask.unsqueeze(1)\n        nbatches = query.size(0)\n\n        # 1) Do all the linear projections in batch from d_model => h x d_k\n        query, key, value = \\\n            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n             for l, x in zip(self.linears, (query, key, value))]\n\n        # 2) Apply attention on all the projected vectors in batch.\n        x, self.attn = attention(query, key, value, mask=mask,\n                                 dropout=self.dropout)\n\n        # 3) \"Concat\" using a view and apply a final linear.\n        x = x.transpose(1, 2).contiguous() \\\n            .view(nbatches, -1, self.h * self.d_k)\n        return self.linears[-1](x)\n\n\nclass PositionwiseFeedForward(nn.Module):\n    \"Implements FFN equation.\"\n    def __init__(self, d_model, d_ff, dropout=0.1):\n        super(PositionwiseFeedForward, self).__init__()\n        self.w_1 = nn.Linear(d_model, d_ff)\n        self.w_2 = nn.Linear(d_ff, d_model)\n        self.dropout = nn.Dropout(dropout)\n\n    def forward(self, x):\n        return self.w_2(self.dropout(F.relu(self.w_1(x))))\n    \nclass SAINTLayer_encoder(nn.Module):\n    \"\"\"\n    Single Encoder block of SAINT\n    \"\"\"\n    def __init__(self, hidden_dim, num_head, dropout):\n        super().__init__()\n        self._self_attn = MultiHeadedAttention(num_head, hidden_dim, dropout)\n        self._ffn = PositionwiseFeedForward(hidden_dim, hidden_dim, dropout)\n        self._layernorms = clones(nn.LayerNorm(hidden_dim, eps=1e-6), 2)\n        self._dropout = nn.Dropout(dropout)\n\n    def forward(self, src, mask = None):\n        \"\"\"\n        query: question embeddings\n        key: interaction embeddings\n        \"\"\"\n        # self-attention block\n        src2 = self._self_attn(query=src, key=src, value=src, mask=mask)\n        src = src + self._dropout(src2)\n        src = self._layernorms[0](src)\n        src2 = self._ffn(src)\n        src = src + self._dropout(src2)\n        src = self._layernorms[1](src)\n        return src\n    \nclass SAINTLayer_decoder(nn.Module):\n    \"\"\"\n    Single Encoder block of SAINT\n    \"\"\"\n    def __init__(self, hidden_dim, num_head, dropout):\n        super().__init__()\n        self._self_attn_decoder = MultiHeadedAttention(num_head, hidden_dim, dropout)\n        self._self_attn_encoder_decoder = MultiHeadedAttention(num_head, hidden_dim, dropout)\n        self._ffn = PositionwiseFeedForward(hidden_dim, hidden_dim, dropout)\n        self._layernorms = clones(nn.LayerNorm(hidden_dim, eps=1e-6), 3)\n        self._dropout = nn.Dropout(dropout)\n\n    def forward(self, tgt, memory, encoder_decoder_mask = None, decoder_mask = None):\n        \"\"\"\n        query: question embeddings\n        key: interaction embeddings\n        \"\"\"\n        # self-attention block\n        tgt2 = self._self_attn_decoder(query=tgt, key=tgt, value=tgt, mask=decoder_mask)\n        tgt = tgt + self._dropout(tgt2)\n        tgt = self._layernorms[0](tgt)\n        tgt2 = self._self_attn_encoder_decoder(query=tgt, key=memory, value=memory, mask=encoder_decoder_mask)\n        tgt = tgt + self._dropout(tgt2)\n        tgt = self._layernorms[1](tgt)\n        tgt2 = self._ffn(tgt)\n        tgt = tgt + self._dropout(tgt2)\n        tgt = self._layernorms[2](tgt)\n        return tgt","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.066233,"end_time":"2021-01-01T18:22:16.624236","exception":false,"start_time":"2021-01-01T18:22:16.558003","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Main model"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:16.779312Z","iopub.status.busy":"2021-01-01T18:22:16.778613Z","iopub.status.idle":"2021-01-01T18:22:16.783046Z","shell.execute_reply":"2021-01-01T18:22:16.782445Z"},"papermill":{"duration":0.092309,"end_time":"2021-01-01T18:22:16.783156","exception":false,"start_time":"2021-01-01T18:22:16.690847","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class SAINT_plus(nn.Module):\n    def __init__(self, question_num, task_num, max_seq = 100, d_model = 128, nhead = 8, dropout = 0.1, num_layers = 2):\n        super().__init__()\n        self.question_num = question_num\n        self.question_num = task_num\n        self.max_seq = max_seq\n        self.d_model = d_model\n        self.nhead = nhead\n        self.dropout = dropout\n        \n        # Construct embedding layers\n        # For categorical features\n        self._positional_embedding = nn.Embedding(self.max_seq + 1, d_model)\n        self._question_embedding = nn.Embedding(question_num + 1, d_model)\n        self._task_embedding = nn.Embedding(task_num + 1, d_model)\n        self._part_embedding = nn.Embedding(9, d_model)\n        self._target_embedding = nn.Embedding(len(responses_total_tokens), d_model)\n        \n        # For continuous features\n        self._diff_embedding = nn.Linear(1, d_model)\n        self._elapse_embedding = nn.Linear(1, d_model)\n        self._corr_embedding = nn.Linear(1, d_model)\n        \n        # Blocks\n        self.encoder_layers = clones(SAINTLayer_encoder(d_model, nhead, dropout), num_layers)\n        self.decoder_layers = clones(SAINTLayer_decoder(d_model, nhead, dropout), num_layers)\n\n        # Prediction layer\n        self._prediction = nn.Linear(d_model, 1)\n        \n    def forward(self, question, task, part, diff, elapse, corr, target, e_pos, d_pos, task_pos):\n        device = question.device\n        # Embed the continuous features first\n        diff = self._diff_embedding(diff.unsqueeze(-1))\n        elapse = self._elapse_embedding(elapse.unsqueeze(-1))\n        corr = self._corr_embedding(corr.unsqueeze(-1))\n        \n        # Masks\n        encoder_mask, decoder_mask, encoder_decoder_mask = get_masks(task_pos, 0, only_before = False)\n        encoder_mask = encoder_mask.to(device)\n        decoder_mask = decoder_mask.to(device)\n        encoder_decoder_mask = encoder_decoder_mask.to(device)\n        \n        # Embed positions\n        e_pos = self._positional_embedding(e_pos)\n        d_pos = self._positional_embedding(d_pos)\n        # Embed question\n        question = self._question_embedding(question)\n        # Embed task\n        task = self._task_embedding(task)\n        # Embed task\n        part = self._part_embedding(part)\n        # Embed task\n        target = self._target_embedding(target)\n        \n        # Aggregate\n        # Encoder is the information of questions themselves\n        encoder = question + task + part + e_pos + diff\n        # Encoder is the information of answers themselves\n        decoder = target + elapse + d_pos + corr\n        \n        # Feed into the transformer\n        # Encoder\n        for layer in self.encoder_layers:\n            encoder = layer(encoder, mask = encoder_mask)\n        \n        # Decoder\n        for layer in self.decoder_layers:\n            decoder = layer(decoder, encoder, encoder_decoder_mask = encoder_decoder_mask, \n                            decoder_mask = decoder_mask)\n            \n        output = self._prediction(decoder)\n        return output.squeeze(-1)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.06703,"end_time":"2021-01-01T18:22:16.917006","exception":false,"start_time":"2021-01-01T18:22:16.849976","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Training"},{"metadata":{"papermill":{"duration":0.066347,"end_time":"2021-01-01T18:22:17.049716","exception":false,"start_time":"2021-01-01T18:22:16.983369","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Training utils"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:18.190957Z","iopub.status.busy":"2021-01-01T18:22:18.189105Z","iopub.status.idle":"2021-01-01T18:22:18.191712Z","shell.execute_reply":"2021-01-01T18:22:18.192172Z"},"papermill":{"duration":0.090345,"end_time":"2021-01-01T18:22:18.192293","exception":false,"start_time":"2021-01-01T18:22:18.101948","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def train_epoch(model, train_iterator, criterion, optim, device = 'cpu'):\n    model.train()\n    \n    train_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n    masks = []\n\n    tbar = tqdm(train_iterator)\n    for item in tbar:\n        x_ques = item['content_id'].to(device)\n        x_task = item['task_container_id'].to(device)\n        x_part = item['part_id'].to(device)\n        x_diff = item['diff_id'].to(device)\n        x_elapse = item['prior_elapsed_time_id'].to(device)\n        x_corr = item['user_correctness_id'].to(device)\n        x_ans = item['old_response_id'].to(device)\n        x_e_pos = item['encoder_position_id'].to(device)\n        x_d_pos = item['decoder_position_id'].to(device)\n        x_task_pos = item['task_position_id'].to(device)\n        mask = item['prediction_mask'].to(device)\n        label = item['target'].to(device)\n        \n        # Define loss\n        # criterion = nn.BCEWithLogitsLoss(reduction = 'sum')\n        \n        with torch.set_grad_enabled(True):\n            output = model(x_ques, x_task, x_part, x_diff, x_elapse, x_corr, x_ans, x_e_pos, x_d_pos, x_task_pos)\n            # Choose output and label\n            output = torch.masked_select(output, mask.bool())\n            label = torch.masked_select(label, mask.bool())\n            # Forward loss\n            loss = criterion(output, label)# / torch.sum(mask)\n            # Optimization\n            optim.step(loss)\n            \n        train_loss.append(loss.item())\n        \n        pred = (torch.sigmoid(output) >= 0.5).long()\n        \n        num_corrects += (pred == label).sum().item()\n        num_total += mask.sum().item()\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(torch.sigmoid(output).view(-1).data.cpu().numpy())\n        batch_auc = roc_auc_score(label.view(-1).data.cpu().numpy(), torch.sigmoid(output).view(-1).data.cpu().numpy())\n                           \n        tbar.set_description('loss - {:.4f} || auc - {:.4f}'.format(loss, batch_auc))\n    \n    acc = num_corrects / num_total\n    auc = roc_auc_score(labels, outs)\n    loss = np.mean(train_loss)\n\n    return loss, acc, auc","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:18.345969Z","iopub.status.busy":"2021-01-01T18:22:18.333973Z","iopub.status.idle":"2021-01-01T18:22:18.348737Z","shell.execute_reply":"2021-01-01T18:22:18.348229Z"},"papermill":{"duration":0.089554,"end_time":"2021-01-01T18:22:18.348829","exception":false,"start_time":"2021-01-01T18:22:18.259275","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def valid_epoch(model, valid_iterator, criterion, device = 'cpu'):\n    model.eval()\n\n    valid_loss = []\n    num_corrects = 0\n    num_total = 0\n    labels = []\n    outs = []\n    masks = []\n    \n    tbar = tqdm(valid_iterator)\n    for item in tbar:\n        x_ques = item['content_id'].to(device)\n        x_task = item['task_container_id'].to(device)\n        x_part = item['part_id'].to(device)\n        x_diff = item['diff_id'].to(device)\n        x_elapse = item['prior_elapsed_time_id'].to(device)\n        x_corr = item['user_correctness_id'].to(device)\n        x_ans = item['old_response_id'].to(device)\n        x_e_pos = item['encoder_position_id'].to(device)\n        x_d_pos = item['decoder_position_id'].to(device)\n        x_task_pos = item['task_position_id'].to(device)\n        mask = item['prediction_mask'].to(device)\n        label = item['target'].to(device)\n        \n        # Define loss\n        #criterion = nn.BCEWithLogitsLoss(reduction = 'sum')\n        \n        with torch.no_grad():\n            output = model(x_ques, x_task, x_part, x_diff, x_elapse, x_corr, x_ans, x_e_pos, x_d_pos, x_task_pos)\n        # Choose output and label\n        output = torch.masked_select(output, mask.bool())\n        label = torch.masked_select(label, mask.bool())\n        # Forward loss\n        loss = criterion(output, label)# / torch.sum(mask)\n        \n        valid_loss.append(loss.item())        \n        \n        pred = (torch.sigmoid(output) >= 0.5).long()\n        \n        num_corrects += (pred == label).sum().item()\n        num_total += mask.sum().item()\n\n        labels.extend(label.view(-1).data.cpu().numpy())\n        outs.extend(torch.sigmoid(output).view(-1).data.cpu().numpy())\n        batch_auc = roc_auc_score(label.view(-1).data.cpu().numpy(), torch.sigmoid(output).view(-1).data.cpu().numpy())\n\n        tbar.set_description('loss - {:.4f} || auc - {:.4f}'.format(loss, batch_auc))\n\n    acc = num_corrects / num_total\n    auc = roc_auc_score(labels, outs)\n    loss = np.mean(valid_loss)\n    \n    return loss, acc, auc","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:18.495083Z","iopub.status.busy":"2021-01-01T18:22:18.494357Z","iopub.status.idle":"2021-01-01T18:22:18.497806Z","shell.execute_reply":"2021-01-01T18:22:18.498248Z"},"papermill":{"duration":0.081862,"end_time":"2021-01-01T18:22:18.498405","exception":false,"start_time":"2021-01-01T18:22:18.416543","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class train_config:\n    # Training\n    METRIC_ = 'max'\n    EPOCHS = 30\n    LR = 1e-5\n    WARM_UP = 4000\n    MODE = 'inference'\n    WORKERS = multiprocessing.cpu_count()\n    BATCH_SIZE = 128\n    TRAIN_RATIO = 0.97\n    CRITERION = nn.BCEWithLogitsLoss().to(DEVICE)\n    # Model\n    D_MODEL = 128\n    N_HEADS = 8\n    N_LAYERS = 4\n    N_QUES = 13523\n    DROP = 0.1\n    \n    if torch.cuda.is_available():\n        MAP_LOCATION = lambda storage, loc: storage.cuda()\n    else:\n        MAP_LOCATION = 'cpu'","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:18.661874Z","iopub.status.busy":"2021-01-01T18:22:18.651599Z","iopub.status.idle":"2021-01-01T18:22:18.664545Z","shell.execute_reply":"2021-01-01T18:22:18.664084Z"},"papermill":{"duration":0.099176,"end_time":"2021-01-01T18:22:18.66464","exception":false,"start_time":"2021-01-01T18:22:18.565464","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"if train_config.MODE == 'train':\n    # Pre-setting\n    auc_max = -np.inf\n    history = []\n    es = 0\n    \n    # Define model\n    model = SAINT_plus(train_config.N_QUES, 10000, max_seq = MAX_SEQ, d_model = train_config.D_MODEL, \n                       nhead = train_config.N_HEADS, dropout = train_config.DROP, num_layers = train_config.N_LAYERS).to(DEVICE)\n    # Define optimizer and scheduler\n    optimizer = NoamOptimizer(model, train_config.LR, train_config.D_MODEL, train_config.WARM_UP)\n    \n    # Define validation iterator, first, define the auxilary validation dataset\n    valid_dataset = RIIID_Valid_Dataset(valid_df.sample(frac = 0.5, random_state = 2020, replace = False), sub_train_, max_seq = MAX_SEQ)\n    valid_dataloader = DataLoader(valid_dataset, batch_size = train_config.BATCH_SIZE * 4, \n                                  shuffle = False, num_workers = train_config.WORKERS)\n    \n    for epoch in range(1, train_config.EPOCHS + 1):\n        # Random sampling\n        weights = sub_train_.apply(lambda x: max(min(500, len(x[0])), 5))\n        train_df_ = sub_train_.sample(frac = train_config.TRAIN_RATIO, weights = weights, replace = True,\n                                      random_state = epoch)\n        train_dataset = RIIID_Train_Dataset(train_df_.reset_index(drop = True), MAX_SEQ)\n        train_dataloader = DataLoader(train_dataset, batch_size = train_config.BATCH_SIZE, \n                                  shuffle = True, num_workers = train_config.WORKERS)\n        # Training\n        train_loss, train_acc, train_auc = train_epoch(model, train_dataloader, train_config.CRITERION, optimizer, device = DEVICE)\n        print(\"\\nEpoch#{}, train_loss - {:.4f} acc - {:.4f} auc - {:.4f}\".format(epoch, train_loss, train_acc, train_auc))\n        # Validation\n        valid_loss, valid_acc, valid_auc = valid_epoch(model, valid_dataloader, train_config.CRITERION, device = DEVICE)\n        print(\"Epoch#{}, valid_loss - {:.4f} acc - {:.4f} auc - {:.4f}\".format(epoch, valid_loss, valid_acc, valid_auc))\n        \n        # The current learning rate\n        lr = optimizer._adam.param_groups[0]['lr']\n        # Store training history\n        history.append({\"epoch\": epoch, \"lr\": lr, **{\"train_auc\": train_auc, \"train_acc\": train_acc}, **{\"valid_auc\": valid_auc, \"valid_acc\": valid_acc}})\n        \n        if valid_auc > auc_max:\n            print(\"Epoch#%s, valid loss %.4f, Metric loss improved from %.4f to %.4f, saving model ...\" % (epoch, valid_loss, auc_max, valid_auc))\n            auc_max = valid_auc\n            torch.save(model.state_dict(), MODEL_BEST)\n            es = 0\n        else:\n            es += 1\n        \n        if es > 20:\n            break\n            \n    if history:\n        metric = 'auc'\n        # Plot training history\n        history_pd = pd.DataFrame(history[1:]).set_index(\"epoch\")\n        train_history_pd = history_pd[[c for c in history_pd.columns if \"train_\" in c]]\n        valid_history_pd = history_pd[[c for c in history_pd.columns if \"valid_\" in c]]\n        lr_history_pd = history_pd[[c for c in history_pd.columns if \"lr\" in c]]\n        fig, ax = plt.subplots(1,2, figsize = (DEFAULT_FIG_WIDTH, 6))\n        t_epoch = train_history_pd[\"train_%s\" % metric].argmin() if train_config.METRIC_ == \"min\" else train_history_pd[\"train_%s\" % metric].argmax()\n        v_epoch = valid_history_pd[\"valid_%s\" % metric].argmin() if train_config.METRIC_ == \"min\" else valid_history_pd[\"valid_%s\" % metric].argmax()\n        d = train_history_pd.plot(kind = \"line\", ax = ax[0], title = \"Epoch: %d, Train: %.3f\" % (t_epoch, train_history_pd.iloc[t_epoch,:][\"train_%s\" % metric]))\n        d = lr_history_pd.plot(kind = \"line\", ax = ax[0], secondary_y = True)\n        d = valid_history_pd.plot(kind = \"line\", ax = ax[1], title = \"Epoch: %d, Valid: %.3f\" % (v_epoch, valid_history_pd.iloc[v_epoch,:][\"valid_%s\" % metric]))\n        d = lr_history_pd.plot(kind = \"line\", ax = ax[1], secondary_y = True)\n        plt.savefig(\"train.png\", bbox_inches = 'tight')\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.067351,"end_time":"2021-01-01T18:22:18.798935","exception":false,"start_time":"2021-01-01T18:22:18.731584","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Inference"},{"metadata":{"papermill":{"duration":0.068733,"end_time":"2021-01-01T18:22:18.934892","exception":false,"start_time":"2021-01-01T18:22:18.866159","status":"completed"},"tags":[]},"cell_type":"markdown","source":"* Test dataset"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:19.098456Z","iopub.status.busy":"2021-01-01T18:22:19.093132Z","iopub.status.idle":"2021-01-01T18:22:19.112971Z","shell.execute_reply":"2021-01-01T18:22:19.112461Z"},"papermill":{"duration":0.110761,"end_time":"2021-01-01T18:22:19.113075","exception":false,"start_time":"2021-01-01T18:22:19.002314","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class RIIID_Test_Dataset(Dataset):\n    def __init__(self, group, test_df, max_seq = 100):\n        self.group = group\n        self.test_df = test_df\n        self.max_seq = max_seq\n        self.pos = np.cumsum([False] + [i != j for i, j in zip(test_df[USER_ID].values, test_df[USER_ID].values[1:])])\n        \n    def __len__(self):\n        return len(np.unique(self.pos))\n    \n    def __getitem__(self, idx):\n        # Determine bundle\n        user_bundle = self.test_df.iloc[self.pos == idx]\n        user_id = user_bundle.user_id.unique()[0]\n        \n        # Extract the sequences\n        new_ques_ = user_bundle[CONTENT_ID].values\n        new_task_ = user_bundle[TASK_CONTAINER_ID].values\n        new_part_ = user_bundle[PART].values\n        new_diff_ = user_bundle[DIFF].values\n        new_elapse_ = user_bundle[ELAPSE].values\n        new_corr_ = user_bundle[CORR].values\n        \n        # New sequence length\n        new_seq_len = len(new_ques_)\n        \n        # Create arrays to pad the sequences\n        ques = np.zeros(self.max_seq, dtype = int) - 1\n        task = np.zeros(self.max_seq, dtype = int) - 1\n        part = np.zeros(self.max_seq, dtype = int)\n        diff = np.zeros(self.max_seq, dtype = float)\n        elapse = np.zeros(self.max_seq, dtype = float)\n        corr = np.zeros(self.max_seq, dtype = float)\n        old_res = np.zeros(self.max_seq, dtype = int)\n        pred_mask = np.zeros(self.max_seq, dtype = int)\n        \n        if user_id in self.group.index:\n            # If the user is already defined in the training group\n            # Unpack the old information\n            ques_, task_, part_, diff_, elapse_, corr_, target_, old_res_ = self.group[user_id]\n            \n            # Append the new information with the old one\n            ques_ = np.append(ques_, new_ques_)\n            task_ = np.append(task_, new_task_)\n            part_ = np.append(part_, new_part_)\n            diff_ = np.append(diff_, new_diff_)\n            elapse_ = np.append(elapse_, new_elapse_)\n            corr_ = np.append(corr_, new_corr_)\n            \n            # For old responses, we copy the last information of the old information and distribute it to the entire bundle\n            old_res_ = np.append(old_res_, target_[-1] * np.ones(new_seq_len))\n            \n            # Old responses mask\n            old_res_mask_ = np.ones(len(old_res_))\n            old_res_ = old_response_to_index(old_res_, old_res_mask_, use_mask = False)\n\n            seq_len = len(ques_)\n            \n            if seq_len >= self.max_seq:\n                ques = ques_[-self.max_seq:]\n                task = task_[-self.max_seq:]\n                part = part_[-self.max_seq:]\n                diff = diff_[-self.max_seq:]\n                elapse = elapse_[-self.max_seq:]\n                corr = corr_[-self.max_seq:]\n                old_res = old_res_[-self.max_seq:]\n            else:\n                ques[-seq_len:] = ques_\n                task[-seq_len:] = task_\n                part[-seq_len:] = part_\n                diff[-seq_len:] = diff_\n                elapse[-seq_len:] = elapse_\n                corr[-seq_len:] = corr_\n                old_res[-seq_len:] = old_res_\n        else:\n            # Else, if the user is new\n            # Append the new information into the array of 0\n            ques = np.append(ques, new_ques_)\n            task = np.append(task, new_task_)\n            part = np.append(part, new_part_)\n            diff = np.append(diff, new_diff_)\n            elapse = np.append(elapse, new_elapse_)\n            corr = np.append(corr, new_corr_)\n            if new_seq_len == 1:\n                old_res = np.append(old_res, START_ID)\n            else:\n                old_res = np.concatenate((old_res, [START_ID], [MASK_ID] * (new_seq_len - 1)))\n            \n            # Slice to obtain the max_seq length\n            ques = ques[-self.max_seq:]\n            task = task[-self.max_seq:]\n            part = part[-self.max_seq:]\n            diff = diff[-self.max_seq:]\n            elapse = elapse[-self.max_seq:]\n            corr = corr[-self.max_seq:]\n            old_res = old_res[-self.max_seq:]\n            \n        # Position\n        task_pos = position(task, pad_include = True, bundle_ignore = False)\n        pred_mask[-new_seq_len:] = np.ones(new_seq_len)\n        \n        e_pos = position(task)\n        d_pos = np.concatenate(([0], e_pos[:-1]))\n        \n        return {\n            'user': user_id, \n            'content_id': torch.tensor(ques, dtype = torch.long) + 1, \n            'task_container_id': torch.tensor(task, dtype = torch.long) + 1, \n            'part_id': torch.tensor(part, dtype = torch.long), \n            'diff_id': torch.tensor(diff, dtype = torch.float32), \n            'prior_elapsed_time_id': torch.tensor(elapse, dtype = torch.float32), \n            'user_correctness_id': torch.tensor(corr, dtype = torch.float32), \n            'old_response_id': torch.tensor(old_res, dtype = torch.long), \n            'encoder_position_id': torch.tensor(e_pos, dtype = torch.long), \n            'decoder_position_id': torch.tensor(d_pos, dtype = torch.long), \n            'task_position_id': torch.tensor(task_pos, dtype = torch.long), \n            'prediction_mask': torch.tensor(pred_mask, dtype = torch.long)\n        }","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.06689,"end_time":"2021-01-01T18:22:19.247057","exception":false,"start_time":"2021-01-01T18:22:19.180167","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Load the model"},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:19.388564Z","iopub.status.busy":"2021-01-01T18:22:19.38799Z","iopub.status.idle":"2021-01-01T18:22:25.162826Z","shell.execute_reply":"2021-01-01T18:22:25.161852Z"},"papermill":{"duration":5.849012,"end_time":"2021-01-01T18:22:25.16294","exception":false,"start_time":"2021-01-01T18:22:19.313928","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"model = SAINT_plus(train_config.N_QUES, 10000, max_seq = MAX_SEQ, d_model = train_config.D_MODEL, \n                   nhead = train_config.N_HEADS, dropout = train_config.DROP, num_layers = train_config.N_LAYERS).to(DEVICE)\n\nif train_config.MODE == 'train':\n    resume_path = MODEL_BEST\nelif train_config.MODE == 'inference':\n    resume_path = r'../input/riiid-saint-best-models/model_best_user_correctness_included_v7.pt'\n    \nmodel.load_state_dict(torch.load(resume_path, map_location = train_config.MAP_LOCATION))\n\nmodel.to(DEVICE)\nmodel.eval()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:25.305524Z","iopub.status.busy":"2021-01-01T18:22:25.304665Z","iopub.status.idle":"2021-01-01T18:22:25.307509Z","shell.execute_reply":"2021-01-01T18:22:25.307043Z"},"papermill":{"duration":0.075531,"end_time":"2021-01-01T18:22:25.307608","exception":false,"start_time":"2021-01-01T18:22:25.232077","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"EMULATION = False\n\nif EMULATION:\n    target_val = pd.read_pickle('../input/riiid-cross-validation-files/cv1_valid.pickle')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.execute_input":"2021-01-01T18:22:25.468989Z","iopub.status.busy":"2021-01-01T18:22:25.462287Z","iopub.status.idle":"2021-01-01T18:22:25.471456Z","shell.execute_reply":"2021-01-01T18:22:25.470992Z"},"papermill":{"duration":0.095745,"end_time":"2021-01-01T18:22:25.471551","exception":false,"start_time":"2021-01-01T18:22:25.375806","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class Iter_Valid(object):\n    def __init__(self, df, max_user=1000):\n        df = df.reset_index(drop=True)\n        self.df = df\n        self.user_answer = df['user_answer'].astype(str).values\n        self.answered_correctly = df['answered_correctly'].astype(str).values\n        df['prior_group_responses'] = \"[]\"\n        df['prior_group_answers_correct'] = \"[]\"\n        self.sample_df = df[df['content_type_id'] == 0][['row_id']]\n        self.sample_df['answered_correctly'] = 0\n        self.len = len(df)\n        self.user_id = df.user_id.values\n        self.task_container_id = df.task_container_id.values\n        self.content_type_id = df.content_type_id.values\n        self.max_user = max_user\n        self.current = 0\n        self.pre_user_answer_list = []\n        self.pre_answered_correctly_list = []\n\n    def __iter__(self):\n        return self\n    \n    def fix_df(self, user_answer_list, answered_correctly_list, pre_start):\n        df= self.df[pre_start:self.current].copy()\n        sample_df = self.sample_df[pre_start:self.current].copy()\n        df.loc[pre_start,'prior_group_responses'] = '[' + \",\".join(self.pre_user_answer_list) + ']'\n        df.loc[pre_start,'prior_group_answers_correct'] = '[' + \",\".join(self.pre_answered_correctly_list) + ']'\n        self.pre_user_answer_list = user_answer_list\n        self.pre_answered_correctly_list = answered_correctly_list\n        return df, sample_df\n\n    def __next__(self):\n        added_user = set()\n        pre_start = self.current\n        pre_added_user = -1\n        pre_task_container_id = -1\n\n        user_answer_list = []\n        answered_correctly_list = []\n        while self.current < self.len:\n            crr_user_id = self.user_id[self.current]\n            crr_task_container_id = self.task_container_id[self.current]\n            crr_content_type_id = self.content_type_id[self.current]\n            if crr_content_type_id == 1:\n                # no more than one task_container_id of \"questions\" from any single user\n                # so we only care for content_type_id == 0 to break loop\n                user_answer_list.append(self.user_answer[self.current])\n                answered_correctly_list.append(self.answered_correctly[self.current])\n                self.current += 1\n                continue\n            if crr_user_id in added_user and ((crr_user_id != pre_added_user) or (crr_task_container_id != pre_task_container_id)):\n                # known user(not prev user or differnt task container)\n                return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n            if len(added_user) == self.max_user:\n                if  crr_user_id == pre_added_user and crr_task_container_id == pre_task_container_id:\n                    user_answer_list.append(self.user_answer[self.current])\n                    answered_correctly_list.append(self.answered_correctly[self.current])\n                    self.current += 1\n                    continue\n                else:\n                    return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n            added_user.add(crr_user_id)\n            pre_added_user = crr_user_id\n            pre_task_container_id = crr_task_container_id\n            user_answer_list.append(self.user_answer[self.current])\n            answered_correctly_list.append(self.answered_correctly[self.current])\n            self.current += 1\n        if pre_start < self.current:\n            return self.fix_df(user_answer_list, answered_correctly_list, pre_start)\n        else:\n            raise StopIteration()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:25.614932Z","iopub.status.busy":"2021-01-01T18:22:25.614195Z","iopub.status.idle":"2021-01-01T18:22:25.648821Z","shell.execute_reply":"2021-01-01T18:22:25.648339Z"},"papermill":{"duration":0.108934,"end_time":"2021-01-01T18:22:25.648923","exception":false,"start_time":"2021-01-01T18:22:25.539989","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"if EMULATION:\n    iter_test = Iter_Valid(target_val, max_user = 1000)\n    predicted = []\n    def set_predict(df):\n        predicted.append(df)\nelse:\n    import riiideducation\n    env = riiideducation.make_env()\n    iter_test = env.iter_test()\n    set_predict = env.predict\n\ntest_train_group = train_group.copy()\nprev_test_df = None","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:25.792802Z","iopub.status.busy":"2021-01-01T18:22:25.792203Z","iopub.status.idle":"2021-01-01T18:22:26.173507Z","shell.execute_reply":"2021-01-01T18:22:26.172904Z"},"papermill":{"duration":0.456298,"end_time":"2021-01-01T18:22:26.173624","exception":false,"start_time":"2021-01-01T18:22:25.717326","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import psutil\nfrom collections import defaultdict\n\nuser_sum_dict = user_agg['sum'].astype('int32').to_dict(defaultdict(int))\nuser_count_dict = user_agg['count'].astype('int32').to_dict(defaultdict(int))\n\ncontent_sum_dict = content_agg['sum'].astype('int32').to_dict(defaultdict(int))\ncontent_count_dict = content_agg['count'].astype('int32').to_dict(defaultdict(int))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-01-01T18:22:26.37562Z","iopub.status.busy":"2021-01-01T18:22:26.355046Z","iopub.status.idle":"2021-01-01T18:22:27.44885Z","shell.execute_reply":"2021-01-01T18:22:27.448176Z"},"papermill":{"duration":1.206476,"end_time":"2021-01-01T18:22:27.448992","exception":false,"start_time":"2021-01-01T18:22:26.242516","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"for ii, (test_df, sample_prediction_df) in enumerate(tqdm(iter_test)):\n    print('*' * 50)\n    if (prev_test_df is not None) & (psutil.virtual_memory().percent < 90):\n        print(psutil.virtual_memory().percent)\n        prev_test_df[TARGET] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        prev_test_df = prev_test_df[prev_test_df.content_type_id == False].reset_index(drop = True)\n        \n        # Position\n        pos = np.cumsum([False] + [i != j for i, j in zip(prev_test_df[USER_ID].values, prev_test_df[USER_ID].values[1:])])\n        \n        # Update content tables (table about questions and correct answers of questions)\n        content_ids = prev_test_df[CONTENT_ID].values\n        user_ids = prev_test_df[USER_ID].values\n        targets = prev_test_df[TARGET].values\n        \n        for content_id, user_id, answered_correctly in zip(content_ids, user_ids, targets):\n            user_sum_dict[user_id] += answered_correctly\n            user_count_dict[user_id] += 1\n            content_sum_dict[content_id] += answered_correctly\n            content_count_dict[content_id] += 1\n        \n        # Because there will be cases that a user appears multiple times in the test_df, but not in a same bundle,\n        # ...we can not use the groupby() method\n        \n        # Update train_group\n        for prev_user_id in np.unique(pos):\n            sub_user_df = prev_test_df[pos == prev_user_id]\n            user = sub_user_df[USER_ID].unique()[0]\n            \n            prev_group_content = sub_user_df[CONTENT_ID].values\n            prev_group_task = sub_user_df[TASK_CONTAINER_ID].values\n            prev_group_part = sub_user_df[PART].values\n            prev_group_diff = sub_user_df[DIFF].values\n            prev_group_elapse = sub_user_df[ELAPSE].values\n            prev_group_correct = sub_user_df[CORR].values\n            prev_group_target = sub_user_df[TARGET].values\n            \n            if user in test_train_group.index:\n                # Update old response if the user is already define in the test_train_group\n                prev_group_old_res = np.concatenate(([test_train_group[user][6][-1]], prev_group_target))[:-1]\n                test_train_group[user] = (np.append(test_train_group[user][0], prev_group_content), \n                                          np.append(test_train_group[user][1], prev_group_task), \n                                          np.append(test_train_group[user][2], prev_group_part), \n                                          np.append(test_train_group[user][3], prev_group_diff), \n                                          np.append(test_train_group[user][4], prev_group_elapse), \n                                          np.append(test_train_group[user][5], prev_group_correct), \n                                          np.append(test_train_group[user][6], prev_group_target), \n                                          np.append(test_train_group[user][7], prev_group_old_res))\n            else:\n                # If the user is new, create a new sequence\n                prev_group_old_res = np.array([1] * len(prev_group_content))\n                test_train_group[user] = (prev_group_content, prev_group_task, prev_group_part, \n                                          prev_group_diff, prev_group_elapse, prev_group_correct, \n                                          prev_group_target, prev_group_old_res)\n            \n            if len(test_train_group[user][0]) > MAX_SEQ:\n                new_group_content = test_train_group[user][0][-MAX_SEQ:]\n                new_group_task = test_train_group[user][1][-MAX_SEQ:]\n                new_group_part = test_train_group[user][2][-MAX_SEQ:]\n                new_group_diff = test_train_group[user][3][-MAX_SEQ:]\n                new_group_elapse = test_train_group[user][4][-MAX_SEQ:]\n                new_group_correct = test_train_group[user][5][-MAX_SEQ:]\n                new_group_target = test_train_group[user][6][-MAX_SEQ:]\n                new_group_old_res = test_train_group[user][7][-MAX_SEQ:]\n                \n                test_train_group[user] = (new_group_content, new_group_task, new_group_part, \n                                          new_group_diff, new_group_elapse, new_group_correct,\n                                          new_group_target, new_group_old_res)\n                \n    # Merge with question data\n    test_df = pd.merge(test_df, questions_df, left_on = CONTENT_ID, right_on = 'question_id', how = 'left')\n    \n    # Compute the content_difficulty\n    user_sum = np.zeros(len(test_df), dtype = np.int16)\n    user_count = np.zeros(len(test_df), dtype = np.int16)\n    content_sum = np.zeros(len(test_df), dtype = np.int32)\n    content_count = np.zeros(len(test_df), dtype = np.int32)\n\n    for i, (user_id, content_id) in enumerate(zip(test_df[USER_ID].values, test_df[CONTENT_ID].values)):\n        user_sum[i] = user_sum_dict[user_id]\n        user_count[i] = user_count_dict[user_id]\n        content_sum[i] = content_sum_dict[content_id]\n        content_count[i] = content_count_dict[content_id]\n    \n    test_df[DIFF] = content_sum / content_count\n    test_df[CORR] = user_sum / user_count\n    test_df[ELAPSE] = test_df[ELAPSE] / 300000 * 100\n    \n    # Mapping the elapse time into categories\n    test_df.fillna(0, inplace = True)\n    \n    # Copy the test_df to prev_test_df\n    prev_test_df = test_df.copy()\n    \n    # Drop lecture observations\n    test_df = test_df[test_df.content_type_id == False].reset_index(drop = True)\n    \n    test_dataset = RIIID_Test_Dataset(test_train_group, test_df)\n    test_dataloader = DataLoader(test_dataset, batch_size = train_config.BATCH_SIZE, shuffle = False, drop_last = False)\n    \n    outs = []\n\n    for item in test_dataloader:\n        x_ques = item['content_id'].to(DEVICE)\n        x_task = item['task_container_id'].to(DEVICE)\n        x_part = item['part_id'].to(DEVICE)\n        x_diff = item['diff_id'].to(DEVICE)\n        x_elapse = item['prior_elapsed_time_id'].to(DEVICE)\n        x_corr = item['user_correctness_id'].to(DEVICE)\n        x_ans = item['old_response_id'].to(DEVICE)\n        x_e_pos = item['encoder_position_id'].to(DEVICE)\n        x_d_pos = item['decoder_position_id'].to(DEVICE)\n        x_task_pos = item['task_position_id'].to(DEVICE)\n        mask = item['prediction_mask'].to(DEVICE)\n\n        with torch.no_grad():\n            output = model(x_ques, x_task, x_part, x_diff, x_elapse, x_corr, x_ans, x_e_pos, x_d_pos, x_task_pos)\n        output = torch.masked_select(output, mask.bool())    \n        outs.extend(torch.sigmoid(output).view(-1).data.cpu().numpy())\n    \n    test_df['answered_correctly'] = outs\n    set_predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}