{"cells":[{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# NOTES"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"1/2 BS\n\nINFERENCE\n- Pipeline\n - History: Keep last 100 interactions per user in memory\n - Build test sequence on-fly from current test_df and history for each user\n - Avoid workers > 0 in Pytorch dataloader, batches are small due to API\n - safeguard against a batch that just has a sinle lecture in it with `\n score = model.predict(inputs) if X_tes.shape[0] > 0 else np.array([], dtype = np.float32)`\n - build pipeline with all train data, not just the data we trained on!!!\n- power siamese network ensembling\n\nLAST DITCH\n- NoamOpt\n - https://lab-ml.com/labml_nn/optimizers/noam.html\n - https://www.kaggle.com/adityaecdrid/fork-of-saint-inference-ea970c/comments\n- collapse infrequent content_id embedding\n- verify prior_question_had_explanation has signal or does it just cause overfit??\n- float32 to speed things up?\n- 1/2 BS\n- drop all continuous features into a single subnet before feeding to rnn\n- ++emb/++rnn dropout\n- pretrain: ?\n- userid model when we have 10 or more interactions :-) and non-userid model\n- ffn encoder layer\n- content id mean ordering vs user ordering difference\n\n```\nexplanation_agg = train_df.groupby('user_id')['prior_question_had_explanation'].agg(['sum', 'count'])\nexplanation_agg=explanation_agg.astype('int16')\ncum = train_df.groupby('user_id')['prior_question_had_explanation'].agg(['cumsum', 'cumcount'])\ncum['cumcount']=cum['cumcount']+1\ntrain_df['explanation_mean'] = cum['cumsum'] / cum['cumcount']\n```\n\n**NOTES**\n- Host's papers (https://arxiv.org/abs/2002.07033, https://arxiv.org/abs/2010.12042)."},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# Reports"},{"metadata":{"heading_collapsed":true,"hidden":true},"cell_type":"markdown","source":"## Findings v2"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- [conjecture] we believe recent correct/incorrect streak has benefit\n- [conjecture] all time correct/incorrect streaks bias the user too much?\n - recommend dropping same response streak\n - recommend dropping alltime correct/incorrect streaks\n - short term streak also absorbs the last 3,5,7 TCID mean feature\n- [fail] try adding encoder into decoder embedding as well..\n- [no change] no attention encoder, e.g. just a FFN/MLP and heavier decoder - surprisingly, this doesnt actually reduce the model's performance but 0.0001 if even.\n- [useless] content id mean ordering vs user ordering\n- [useless] user_answered longest same value streak - looks like reduces overfitting a bit :-)\n - seems to be a strong regularizer\n - stops our overfitting but doesn't increase score\n - the hit we take is 0.00025\n - but maybe too strong\n- [good] post training user embedding fine tuning provides 0.0001 boost =E\n- [meh] pretrain all network except exerciseid against target, then fine tune exercise id\n- [meh] content_id rank diff causes slow overfit\n- [superbad] SQRT(LEN//2) for TCID Shuffling\n- [horrible] linear capped times rather than log scaled\n- [good] slight tcid shuffling and no preferential sampling for tail of long users\n- [no change] experience = cumtime spent answering, log encoded\n- [no change] cumulative time spent answering for the session\n- [no change] log(ms) instead of log(s)\n- [bad] history moved to the embedder side...\n- [bad] history merged into MEMORY (after embedder) going into decoder\n- [okay] target clustered diagnostic features barely move the envelope\n- [no diff] 2,3, vs 4,6 vs 6,9\n- [bad] target mask off training\n- [okay] other folds have similar scores....\n- [good] inplace intra tcid shuffling augmentation works without issue, slight boost\n-- [**great**] maasking and nan removal of history (correctness)\n- [nogood] testing 128 w/o dropout; but train score caps out at 7990 ish.....\n- [bad] tsvd/pca/clustering on tag_group_id\n- [bad] all forms of difficulty\n- BNorm [bad] on Linear inputs w/o Bias\n- [bad] 256 w 4layer 2epoch warmup\n- [nochange] dim_feedforward=d_model x 2 rather than just d_model\n- [nochange] xent instead of bce\nTested\n- prior_question_elapsed_time_embeddings shifted decoder\n- prior_question_had_explanation_embeddings shifted decoder\n- pos_embedding shifted decoder\n- user_correctness patch for lectures\n- ts embeddings shifted decoder <-- this hurt score a lot so rejected\n- filtering <5 TCID users, and do trimming before processing\n- attention masks: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206620\n- why is difficulty shifted???????\n - dunno, but it was giving us better score\n - currently not shifted\n- correctness_u_recency and difficulty are on different scales than the rest of the features flowing into the layer\n- perhaps a layer norm first.."},{"metadata":{"heading_collapsed":true,"hidden":true},"cell_type":"markdown","source":"### Findings"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- [**great**] Smaller BS (64)—maybe even no dropout. But this makes iterating slow... maybe dual gpu...\n- [**great**] stock transformerencoder/decoder layers are better\n- [**great**] larger ffn dim, at least d_model...\n- [**great**] sin pos embedder\n- [**great**] bundle id embedding: bundle groups the questions. task_container_id is a subset of q's in a bundle which are selected to be asked.\n- [**great**] diff1 and diff2 interaction ts, no differentiation between Q & L\n- [**great**] 20% encoder dropout. we need to test further increases\n- [**great**] previous question correctness embedding\n- [**great**] content type id embedding\n- [**great**] data parallel\n- [**great**] patched float64 stdscaler overflow issues\n- [**great**] everything on the encoder except correctness and pos emb - still using IRT\n- [**great**] no decoder needed it seems\n- [**great**] larger CAP_SAMPLING_LENGTH_TRN seems to help, which makes sense since those longer users will be part of subset. test against lb though to optimize for this\n- [**great**] user correctness mean\n- [**great**] fp16 double speed\n- [**great**] pre-sorting parquets to save time...\n- [**great**] CV from: `kaggle kernels output marisakamozz/cv-strategy-in-the-kaggle-environment`\n---\n- [good] x2 = 8 layer stack more important than x2 = 512 dimension model...\n- [good] important to seed numpy/random in different dataset loader threads\n- [good] swapping memory+target on the decoder is competitive and might make for good ensemble\n- [good] static pos embedding\n- [good] train sampling: put more weight on further sequence to balance start sequences\n- [good] big model dim, 256 or 512 even\n- [good] question features on encoder, response features on decoder\n- [good] heavier tail is marginally better... have to use dropout there though\n- [good] updated to unbiased CV: https://www.kaggle.com/marisakamozz/cv-strategy-in-the-kaggle-environment\n- [good] start_offset prefers later user interaction sequences\n- [good] incorrect rank\n- [good] label smoothing - cuts down speeds in 1/2 though....\n- [good] bxe w/ dense targets\n- [good] masked lectures...\n- [good] fixed bug in embedding incorrect_rank (+3)\n- [good] padding_idx=0 for all embeddings\n- [good] tag community network clustering\n- [good] actually, looks like relus beat tanh/gelu\n---\n- [meh] timestamp_u_incorrect_recency, timestamp_u_correct_recency features. I'm keeping them for now but they literally did nothing for AUC but may make us start to overfit.\n- [meh] task_container_id doesn't really seem to move the needle; neither embedding nor log transform (NEEDS RETESTING)\n- [meh] weight decay slows down convergence by 25% but seems to help sliiiiightly marginally? disabling for now we can re-enable in the future\n- [meh] filling padded targets with mean; pretty sure this does nothing\n- [meh] adding ALL tag embeddings (eats a lot of memory too) and the advantage is epsilon\n- [meh] changing dropout {0, 0.1, 0.2} doesn't seem to make a difference..\n- [meh] consolidation of continuous features into a single dense layer expended to d_model w/ tanh activation. Results actually degraded slightly; but this gives us the ability to further process through, e.g. BatchNorm, dropout, linear, before feeding into the Transformer.\n- [meh] incorrect rank is now the target, xent loss, and weight tieing. Results aren't bad except with weight typing, they drop 0.0001. But without tieing, might be good for ensemble..\n- [meh] adamw and adam preform similarly. asgd/sgd w/ large momentum, converge too slowly but converg'ish...\n- [meh] item response theory output (alpha and beta) didn't really move the needle vs sigmoid\n- [meh] ffn on exercise embedding / full src embedding before feeding into stack, marginal changes\n- [meh] user_part_cummeans no real impact\n- [meh] cumulative mean based on adjusted correctness (by question difficulty) \n---\n- [bad] removing bias from linear embedder layers\n- [bad] ts and its transforms no predictive power. HOWEVER, we might be able to break into two groups... explore..\n- [bad] first bundle id no predictive power\n- [bad] ts offset had no predictive power\n- [bad] tags have no predictive power...\n- [bad] weight decay, even only on non bias/norm layers...\n- [bad] num_workers>0 in dataloader forces a seed, which isn't desired since we have random sampling augmentation. also inference batches are small so >n workers slows things down due to overhead.\n- [bad] padding in collater has diminishing returns due to sampling preferring full length batches; not even useful in inference, since we cannot arrange users by length due to API ordering\n- [bad] move continuous features to the decoder\n- [bed] dense: BN,LinearNoBias,ReLU,Dropout,Linear,TanH\n- [bad] swap memory + input for decoder\n- [bad] container id embeddings\n- [bad] IRT correct for `no_responses`\n- [bad] user_lecture_lv: which is cumsum lectures / content\n- [bad] user_lecture_ts: timestamp / cumcount(content)\n- [bad] ts_since_lecture doesnt move the needle\n- [bad] mean difficulty, even lagged.\n- [bad] community and tag do nothing..."},{"metadata":{"heading_collapsed":true,"hidden":true},"cell_type":"markdown","source":"### Sampling Strategy"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"- Take all user ids in training data.\n- Compute their lengths and cap the result to 500.\n- Scale the prior to make them sum 1 (probability distribution).\n- Sample N ids with replacement with previous computed probabilities. N in my case is the same as different user ids you have in the training data. The result will have repeated ids.\n- Take a random sequence for every id. It may lead to repeated sequences with low probability. For example, if you have id 115 repeated 7 times, you will take 7 random sequences for the user 115.\n- Repeat each epoch."},{"metadata":{"papermill":{"duration":0.016225,"end_time":"2020-11-12T09:45:16.415602","exception":false,"start_time":"2020-11-12T09:45:16.399377","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Setup"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-12T09:45:16.452401Z","iopub.status.busy":"2020-11-12T09:45:16.451753Z","iopub.status.idle":"2020-11-12T09:45:17.939066Z","shell.execute_reply":"2020-11-12T09:45:17.938339Z"},"papermill":{"duration":1.510191,"end_time":"2020-11-12T09:45:17.939185","exception":false,"start_time":"2020-11-12T09:45:16.428994","status":"completed"},"tags":[],"trusted":false},"cell_type":"code","source":"PRELOAD = False\n\nimport numpy as np\nimport pandas as pd\nimport gc, joblib, random\n\nimport matplotlib.pyplot as plt\nfrom tqdm.auto import tqdm\nfrom pathlib import Path\nfrom typing import List\nfrom collections import defaultdict#,deque, namedtuple, Counter\nfrom bitarray import bitarray\n\nfrom time import time\n\n# import itertools\n# from contextlib import contextmanager\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport scipy\n\nfrom einops import repeat, rearrange\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom prettytable import PrettyTable\n\nTRAIN_DTYPES = {\n # 'row_id': np.uint32,\n 'timestamp': np.uint64,\n 'user_id': np.uint32,\n 'content_id': np.uint16,\n 'content_type_id': np.uint8,\n 'task_container_id': np.uint16,\n 'user_answer': np.int8,\n 'answered_correctly': np.int8,\n 'prior_question_elapsed_time': np.float32,\n 'prior_question_had_explanation': 'boolean'\n}\n\nDATA_DIR = Path('../input/riiid-test-answer-prediction')\nTRAIN_PATH = DATA_DIR / 'train.csv'\nQUESTIONS_PATH = DATA_DIR / 'questions.csv'\nLECTURES_PATH = DATA_DIR / 'lectures.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"DEVICE = 'cuda'\nN_EPOCHS = 50\nBATCH_SIZE = 512 #// 2\n\n# Must be >= window_size\nCAP_SAMPLING_LENGTH_TRN = 2000\nCAP_SAMPLING_LENGTH_VAL = 500\n\n# this parameter denotes how many last seen content_ids I am going to consider\n# .\nWINDOW_SIZE = 128\n\nPAD = 0\n\nGREAT_SEED = 1337\n\ndef seed_everything(seed_value):\n random.seed(seed_value)\n np.random.seed(seed_value)\n torch.manual_seed(seed_value)\n\n if torch.cuda.is_available(): \n torch.cuda.manual_seed(seed_value)\n torch.cuda.manual_seed_all(seed_value)\n torch.backends.cudnn.deterministic = False\n torch.backends.cudnn.benchmark = False\n \nseed_everything(GREAT_SEED)\ngc.enable()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Start col id labeling from this row below\nCOL_OFFSET_CUT = 1\n\ndataset_cols = [\n # Leave user_id first, so we can truncate it\n 'timestamp',\n 'user_id',\n 'task_container_id',\n \n #############\n # We start indexing from here, since the rest get cut off:\n 'content_id',\n 'part_id',\n 'prior_question_elapsed_time',\n 'prior_question_had_explanation',\n 'incorrect_rank', # decoder feature that needs help..........\n 'content_type_id',\n 'bundle_id',\n \n # only these of the 4 have signal (0-index)\n 'answer_ratio1',\n 'answer_ratio2',\n 'correct_streak_u',\n 'incorrect_streak_u',\n 'correct_streak_alltime_u',\n 'incorrect_streak_alltime_u',\n \n # our session information + a lifetime field (elapsed time sum on platform—probably should be capped)\n 'session_content_num_u',\n 'session_duration_u',\n 'session_ans_duration_u',\n 'lifetime_ans_duration_u',\n \n # First block of continuous - timestamp intervals\n 'lag_ts_u_recency',\n 'last_ts_u_recency1',\n 'last_ts_u_recency2',\n 'last_ts_u_recency3',\n 'last_correct_ts_u_recency',\n 'last_incorrect_ts_u_recency',\n \n # Second block of continuous - how often user is right on global/part/session basis\n 'correctness_u_recency',\n 'part_correctness_u_recency',\n 'session_correctness_u',\n \n # Have we been served this content before?\n 'encountered',\n \n # Average score of most frequent asked questions\n 'diagnostic_u_recency_1',\n 'diagnostic_u_recency_2',\n 'diagnostic_u_recency_3',\n 'diagnostic_u_recency_4',\n 'diagnostic_u_recency_5',\n 'diagnostic_u_recency_6',\n\n 'answered_correctly',\n]\n\nCOL_USER_ID = dataset_cols.index('user_id') - COL_OFFSET_CUT\nCOL_TASK_CONTAINER_ID = dataset_cols.index('task_container_id') - COL_OFFSET_CUT\nCOL_CONTENT_ID = dataset_cols.index('content_id') - COL_OFFSET_CUT\nCOL_PART_ID = dataset_cols.index('part_id') - COL_OFFSET_CUT\nCOL_PRIOR_QUESTION_ELAPSED_TIME = dataset_cols.index('prior_question_elapsed_time') - COL_OFFSET_CUT\nCOL_PRIOR_QUESTION_EXPLANATION = dataset_cols.index('prior_question_had_explanation') - COL_OFFSET_CUT\nCOL_INCORRECT_RANK = dataset_cols.index('incorrect_rank') - COL_OFFSET_CUT\nCOL_CONTENT_TYPE_ID = dataset_cols.index('content_type_id') - COL_OFFSET_CUT\nCOL_BUNDLE_ID = dataset_cols.index('bundle_id') - COL_OFFSET_CUT\n\nCOL_ANSWER_RATIO1 = dataset_cols.index('answer_ratio1') - COL_OFFSET_CUT\nCOL_ANSWER_RATIO2 = dataset_cols.index('answer_ratio2') - COL_OFFSET_CUT\nCOL_CORRECT_STREAK_U = dataset_cols.index('correct_streak_u') - COL_OFFSET_CUT\nCOL_INCORRECT_STREAK_U = dataset_cols.index('incorrect_streak_u') - COL_OFFSET_CUT\nCOL_CORRECT_STREAK_ALLTIME_U = dataset_cols.index('correct_streak_alltime_u') - COL_OFFSET_CUT\nCOL_INCORRECT_STREAK_ALLTIME_U = dataset_cols.index('incorrect_streak_alltime_u') - COL_OFFSET_CUT\n \nCOL_SESSION_CONTENT_NUM_U = dataset_cols.index('session_content_num_u') - COL_OFFSET_CUT\nCOL_SESSION_DURATION_U = dataset_cols.index('session_duration_u') - COL_OFFSET_CUT\nCOL_SESSION_CORRECTNESS_U = dataset_cols.index('session_correctness_u') - COL_OFFSET_CUT\nCOL_SESSION_ANS_DURATION_U = dataset_cols.index('session_ans_duration_u') - COL_OFFSET_CUT\nCOL_LIFETIME_ANS_DURATION_U = dataset_cols.index('lifetime_ans_duration_u') - COL_OFFSET_CUT\n\nCOL_LAG_TS_RECENCY = dataset_cols.index('lag_ts_u_recency') - COL_OFFSET_CUT\nCOL_LAST_TS_RECENCY1 = dataset_cols.index('last_ts_u_recency1') - COL_OFFSET_CUT\nCOL_LAST_TS_RECENCY2 = dataset_cols.index('last_ts_u_recency2') - COL_OFFSET_CUT\nCOL_LAST_TS_RECENCY3 = dataset_cols.index('last_ts_u_recency3') - COL_OFFSET_CUT\nCOL_CORRECTNESS_U_RECENCY = dataset_cols.index('correctness_u_recency') - COL_OFFSET_CUT\nCOL_PART_CORRECTNESS_U_RECENCY = dataset_cols.index('part_correctness_u_recency') - COL_OFFSET_CUT\nCOL_LAST_CORRECT_TS_U_RECENCY = dataset_cols.index('last_correct_ts_u_recency') - COL_OFFSET_CUT\nCOL_LAST_INCORRECT_TS_U_RECENCY = dataset_cols.index('last_incorrect_ts_u_recency') - COL_OFFSET_CUT\n\nCOL_ENCOUNTERED = dataset_cols.index('encountered') - COL_OFFSET_CUT\n\nCOL_DIAGNOSTIC_RECENCY_1 = dataset_cols.index('diagnostic_u_recency_1') - COL_OFFSET_CUT\nCOL_DIAGNOSTIC_RECENCY_2 = dataset_cols.index('diagnostic_u_recency_2') - COL_OFFSET_CUT\nCOL_DIAGNOSTIC_RECENCY_3 = dataset_cols.index('diagnostic_u_recency_3') - COL_OFFSET_CUT\nCOL_DIAGNOSTIC_RECENCY_4 = dataset_cols.index('diagnostic_u_recency_4') - COL_OFFSET_CUT\nCOL_DIAGNOSTIC_RECENCY_5 = dataset_cols.index('diagnostic_u_recency_5') - COL_OFFSET_CUT\nCOL_DIAGNOSTIC_RECENCY_6 = dataset_cols.index('diagnostic_u_recency_6') - COL_OFFSET_CUT\n\nCOL_ANSWERED_CORRECTLY = dataset_cols.index('answered_correctly') - COL_OFFSET_CUT\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class UserDF():\n def pop_if_needed(self):\n # Ensure we're only keeping the final 100:\n if self.lensize[0] < 128:\n return True\n \n self.user_id.pop(0)\n self.task_container_id.pop(0)\n\n #############\n # We start indexing from here, since the rest get cut off:\n self.content_id.pop(0)\n self.part_id.pop(0)\n self.prior_question_elapsed_time.pop(0)\n self.prior_question_had_explanation.pop(0)\n self.incorrect_rank.pop(0) # decoder feature that needs help..........\n self.content_type_id.pop(0)\n self.bundle_id.pop(0)\n\n # only these of the 4 have signal (0-index)\n self.answer_ratio1.pop(0)\n self.answer_ratio2.pop(0)\n self.correct_streak_u.pop(0)\n self.incorrect_streak_u.pop(0)\n self.correct_streak_alltime_u.pop(0)\n self.incorrect_streak_alltime_u.pop(0)\n\n # our session information + a lifetime field (elapsed time sum on platform—probably should be capped)\n self.session_content_num_u.pop(0)\n self.session_duration_u.pop(0)\n self.session_ans_duration_u.pop(0)\n self.lifetime_ans_duration_u.pop(0)\n\n # First block of continuous - timestamp intervals\n self.lag_ts_u_recency.pop(0)\n self.last_ts_u_recency1.pop(0)\n self.last_ts_u_recency2.pop(0)\n self.last_ts_u_recency3.pop(0)\n self.last_correct_ts_u_recency.pop(0)\n self.last_incorrect_ts_u_recency.pop(0)\n\n # Second block of continuous - how often user is right on global/part/session basis\n self.correctness_u_recency.pop(0)\n self.part_correctness_u_recency.pop(0)\n self.session_correctness_u.pop(0)\n\n # Have we been served this content before?\n self.encountered.pop(0)\n\n # Average score of most frequent asked questions\n self.diagnostic_u_recency_1.pop(0)\n self.diagnostic_u_recency_2.pop(0)\n self.diagnostic_u_recency_3.pop(0)\n self.diagnostic_u_recency_4.pop(0)\n self.diagnostic_u_recency_5.pop(0)\n self.diagnostic_u_recency_6.pop(0)\n\n self.answered_correctly.pop(0)\n return False\n \n def __init__(self):\n # TODO: We may need to add some variables to keep track of how many predictions\n # And where in test_df they are\n self.lensize = [0]\n \n ##########\n self.user_id = []\n self.task_container_id = []\n\n #############\n # We start indexing from here, since the rest get cut off:\n self.content_id = []\n self.part_id = []\n self.prior_question_elapsed_time = []\n self.prior_question_had_explanation = []\n self.incorrect_rank = [] # decoder feature that needs help..........\n self.content_type_id = []\n self.bundle_id = []\n\n # only these of the 4 have signal (0-index)\n self.answer_ratio1 = []\n self.answer_ratio2 = []\n self.correct_streak_u = []\n self.incorrect_streak_u = []\n self.correct_streak_alltime_u = []\n self.incorrect_streak_alltime_u = []\n\n # our session information + a lifetime field (elapsed time sum on platform—probably should be capped)\n self.session_content_num_u = []\n self.session_duration_u = []\n self.session_ans_duration_u = []\n self.lifetime_ans_duration_u = []\n\n # First block of continuous - timestamp intervals\n self.lag_ts_u_recency = []\n self.last_ts_u_recency1 = []\n self.last_ts_u_recency2 = []\n self.last_ts_u_recency3 = []\n self.last_correct_ts_u_recency = []\n self.last_incorrect_ts_u_recency = []\n\n # Second block of continuous - how often user is right on global/part/session basis\n self.correctness_u_recency = []\n self.part_correctness_u_recency = []\n self.session_correctness_u = []\n\n # Have we been served this content before?\n self.encountered = []\n\n # Average score of most frequent asked questions\n self.diagnostic_u_recency_1 = []\n self.diagnostic_u_recency_2 = []\n self.diagnostic_u_recency_3 = []\n self.diagnostic_u_recency_4 = []\n self.diagnostic_u_recency_5 = []\n self.diagnostic_u_recency_6 = []\n\n self.answered_correctly = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class UserFeatures():\n # lagtime = previous bundle's tsdiff - previousquestionelapsedtime * previousbundlesize\n # account for lectures.\n \n def __init__(self):\n # Everything needs to be a list so we can pass references..\n # Idx=0 current ts tcid, Idx=1 start idx of this TCID, Idx=2 counter of items in TCID\n # Idx=0 current ts tcid, Idx=1 counter, Idx=2 is [idx,idx,idx]\n self.cached_task_container_id_u = [np.nan]\n\n # These two features are only used to compute the lag feature.\n # Index: 0 = current, 1=previous\n self.task_container_num_questions_u = [0,0] # Only updated on question_id's!\n # Index: 0 = ts, 1=per-bundle1, 1 = per-bundle2,\n self.last_question_ts_u_recency2 = [np.nan,np.nan,np.nan]\n\n # Idx0=ts, Idx1=CorrectTS, Idx2=CountTS, Idx3=CorrectBundleNoLeak, Idx4=CountBundleBundleNoLeak\n # Idx5=CumAnsTimeSessionNoLeak, Idx6=CumAnsTimePlatformNoLeak\n self.session_u = [np.nan, 2,3, 2,3, 0,0]\n\n # Index: 0=ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3, ...\n self.last_ts_u = [np.nan,np.nan,np.nan,np.nan]\n\n # Index: 0 = per-ts, 1 = per-bundle\n self.last_correct_ts_u = [0,0] #[np.nan,np.nan]\n\n # Index: 0,1 = per-ts, 2,3 = per-bundle. 0=correct qs, 1=total qs\n self.last_incorrect_ts_u = [0,0] #[np.nan,np.nan]\n\n # Index: 0,1 = per-ts, 2,3 = per-bundle. 0=correct qs, 1=total qs\n self.correctness_u = [2,3,2,3] # 4right,2wrong=66.7%, mean target\n\n # 7*4, [7parts][0=correct_ts,1=count_ts,2=correct_bundle,3=count_bundle]\n self.part_correctness_u = 7 * [[2,3,2,3]]\n\n # diagnostic_content_id_order_map\n # diagnostic_content_id_mean_map\n # diagnostic_content_id_mean_map_values\n self.diagnostic_u_1 = [\n # Target Grouping rather than association grouping for now:\n #26 \t6911 \t0.818782\n #33 \t7901 \t0.824779\n\n # First two indices are 'update indicators', last two are default values that are set at end of each TCID\n np.nan, np.nan, 0.818782, 0.824779,\n\n # Last value is default mean\n 0.8217805\n ]\n\n self.diagnostic_u_2 = [\n # 30 \t7219 \t0.594133\n # 25 \t6910 \t0.602537\n # 4 \t2066 \t0.614043\n # 21 \t6879 \t0.619330\n # 3 \t2065 \t0.621588\n # 1 \t1279 \t0.635821\n np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,\n 0.594133, 0.602537, 0.614043, 0.619330, 0.621588, 0.635821,\n\n 0.6145753333333334\n ]\n\n self.diagnostic_u_3 = [\n # 12 \t3365 \t0.532004\n # 16 \t4697 \t0.545093\n # 5 \t2594 \t0.557091\n np.nan, np.nan, np.nan,\n 0.532004, 0.545093, 0.557091,\n\n 0.5447293333333334\n ]\n\n self.diagnostic_u_4 = [\n # 20 \t6878 \t0.430956\n # 27 \t6912 \t0.431397\n # 15 \t4493 \t0.440798\n # 7 \t2596 \t0.468018\n # 29 \t7218 \t0.485773\n np.nan, np.nan, np.nan, np.nan, np.nan,\n 0.430956, 0.431397, 0.440798, 0.468018, 0.485773,\n\n 0.4513884\n ]\n\n self.diagnostic_u_5 = [\n # 24 \t6909 \t0.390869\n # 32 \t7877 \t0.393220\n # 22 \t6880 \t0.404521\n # 28 \t7217 \t0.405293\n # 6 \t2595 \t0.419734\n np.nan, np.nan, np.nan, np.nan, np.nan,\n 0.390869, 0.393220, 0.404521, 0.405293, 0.419734,\n 0.4027274\n ]\n\n self.diagnostic_u_6 = [\n # 10 \t2949 \t0.211211\n # 9 \t2948 \t0.226543\n # 14 \t4121 \t0.230986\n # 23 \t6881 \t0.252001\n # 18 \t6174 \t0.255422\n # 31 \t7220 \t0.262637\n # 11 \t3364 \t0.268132\n np.nan, np.nan, np.nan, np.nan, np.nan, np.nan, np.nan,\n 0.211211, 0.226543, 0.230986, 0.252001, 0.255422, 0.262637, 0.268132,\n 0.24384742857142858,\n ]\n \n #user_answer_streak_map\n self.answer_ratio = [\n # Idx0 = last user answer, Idx1 = streak count\n # We don't have to limit to bundles since we have access to this at read time\n 1,1, # UA=1 counter, UA=1 bundle \n 1,1, # UA=2 counter, UA=2 bundle\n 4,4, # total q asked\n ]\n \n self.streak_u = [\n 0,0,0, # correct ts, bundle, all-time bundle\n 0,0,0, # incorrect ts, bundle, all-time bundle\n ]\n\n # We don't need to update @ Bundle because we'll only see a question once in a TCID\n self.content_encounter = bitarray(13550+425, endian='little')\n self.content_encounter.setall(0)\n \n def get_features(self):\n return (\n self.cached_task_container_id_u,\n self.task_container_num_questions_u,\n self.last_question_ts_u_recency2,\n self.session_u,\n self.last_ts_u,\n self.last_correct_ts_u,\n self.last_incorrect_ts_u,\n self.correctness_u,\n self.part_correctness_u,\n self.diagnostic_u_1,\n self.diagnostic_u_2,\n self.diagnostic_u_3,\n self.diagnostic_u_4,\n self.diagnostic_u_5,\n self.diagnostic_u_6,\n self.answer_ratio,\n self.streak_u,\n self.content_encounter,\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# For pickling\ndef newUserFeatures(): return UserFeatures()\ndef newUserDF(): return UserDF()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def prepare_training_data(fold_number=0):\n t = time()\n \n # +1 for padding idx\n df_lectures = pd.read_csv(LECTURES_PATH)\n df_questions = pd.read_csv(QUESTIONS_PATH)\n #df_questions_lsi = pd.read_csv('./genism_question_lsi20.csv')[['content_id', 'tags_lsi']]\n\n # Since we increment content_id...\n #df_questions_lsi.content_id += 1\n df_questions.question_id += 1\n df_lectures.lecture_id += 1\n \n mqid = df_questions.question_id.max() + 1\n lecture_ids_map = {v: i+mqid for i,v in enumerate(df_lectures.lecture_id.unique()) }\n df_lectures.lecture_id = df_lectures.lecture_id.map(lecture_ids_map)\n \n\n # Parts are: [1, 2, 3, 4, 5, 6, 7]\n # So 0 already reserved (and not mapped); 7+1 total\n # No join needed:\n # NOTE: This is mapping only the first part for questions!!!!!!!!!!!!!!!!!!!!\n part_ids_map = dict(zip(df_questions.question_id, df_questions.part))\n part_ids_map.update(dict(zip(df_lectures.lecture_id, df_lectures.part)))\n\n# df_train = pd.read_parquet(f'../input/uthman-cv/cv{fold_number+1}_train.parquet')\n# df_valid = pd.read_parquet(f'../input/uthman-cv/cv{fold_number+1}_valid.parquet')\n \n df_train = pd.read_parquet(f'../input/cv-strategy-in-the-kaggle-environment/cv{fold_number+1}_train_sorted.parquet')[TRAIN_DTYPES.keys()]\n df_valid = pd.read_parquet(f'../input/cv-strategy-in-the-kaggle-environment/cv{fold_number+1}_valid_sorted.parquet')[TRAIN_DTYPES.keys()] \n\n# df_train = pd.read_pickle(f'../input/riiid-cross-validation-files/cv{fold_number+1}_train.pickle')\n# df_valid = pd.read_pickle(f'../input/riiid-cross-validation-files/cv{fold_number+1}_valid.pickle')\n# df_train.sort_values(['user_id','timestamp'], inplace=True)\n# df_valid.sort_values(['user_id','timestamp'], inplace=True)\n \n# # # NOTE: REMOVE:\n# df_train = df_train.iloc[:5000000]\n# df_valid = df_valid.iloc[:5000000]\n \n #Filter\n df_train = df_train.groupby('user_id', sort=False).head(CAP_SAMPLING_LENGTH_TRN)\n df_valid = df_valid.groupby('user_id', sort=False).head(CAP_SAMPLING_LENGTH_VAL)\n # NOTE: User IDs already mapped.... but we add 1 so that we can use 0 for unknown users\n embedded_user_ids_map = joblib.load('../input/uthman-cv/mapped_embedded_user_ids.jlib')\n embedded_user_ids_map = {i:v+1 for i,v in embedded_user_ids_map.items()}\n \n \n # Be careful, anything we do that df = assigns into the df, or ['creates'] a new column\n # will result in a new object, besides the original when in a for loop!!!\n for df in df_train, df_valid:\n # 0 = PAD/NA, 1 = STARTER, 2=False, 3=True\n # TODO: Keep on for tito and tito+ cv\n df.user_id = df.user_id.map(embedded_user_ids_map).astype(np.uint32)\n \n df.prior_question_had_explanation = (df.prior_question_had_explanation.astype(np.float16).fillna(-2) + 2).astype(np.uint8)\n #df.prior_question_had_explanation = df.prior_question_had_explanation.astype(np.float16).fillna(0).astype(np.uint8)\n \n # NOTE: Pre-sorted the parquets:\n # df.sort_values(['user_id', 'timestamp'], inplace=True)\n \n # SPECIAL NOTE / TODO:\n # Valid set has some new users — about 1/3rd in our case\n # We want its user_ids to map to 0 = pad, to simulate us not\n # having any history for them. We do this in the dataloader\n # I guess....\n mask_user_ids = set(df_train.user_id.unique()) - set(df_valid.user_id.unique())\n \n \n gc.collect()\n \n \n #########################################\n# diagnostic_content_id_mean_map = df_train[['content_type_id', 'task_container_id', 'user_id', 'content_id', 'prior_question_had_explanation', 'answered_correctly']][\n# (df_train.content_type_id==0) &\n# (df_train.task_container_id<40)\n# ]\n\n# diagnostics_content_id = []\n# current_user_id = None\n# add = True\n# for user_id, content_id, prior_question_had_explanation in tqdm(diagnostic_content_id_mean_map[[\n# 'user_id','content_id','prior_question_had_explanation'\n# ]].values): \n# # run until we find prior_question_had_explanation\n# if current_user_id != user_id:\n# current_user_id = user_id\n# add = True\n\n# if prior_question_had_explanation==3:\n# add = False\n\n# if add:\n# diagnostics_content_id.append(content_id)\n\n# #10% of the population\n# diagnostics_content_id = pd.Series(diagnostics_content_id).value_counts()\n# diagnostics_content_id = diagnostics_content_id[diagnostics_content_id>40000].index.tolist()\n# diagnostic_content_id_mean_map = diagnostic_content_id_mean_map[['content_id','answered_correctly']][\n# diagnostic_content_id_mean_map.content_id.isin(diagnostics_content_id)\n# ].groupby('content_id').answered_correctly.mean().to_dict()\n# diagnostic_content_id_mean_map_values = list(diagnostic_content_id_mean_map.values())\n #########################################\n\n \n df_train.reset_index(inplace=True, drop=True)\n df_valid.reset_index(inplace=True, drop=True)\n \n # Cannot be in loop since we're creating a new feature:\n df_train.content_id += 1\n df_valid.content_id += 1\n \n# # NOTE: If we remove the dropping of bad_users w/ TCID count < 5\n# # Then this block can be removed and we can move the diagnostic building here\n# diagnostic_content_id_mean_map = {\n# # The mapping of the features\n# k+1:v for k,v in diagnostic_content_id_mean_map.items()\n# }\n# diagnostic_content_id_order_map = {\n# # The order of the features in our storage\n# k:i for i,k in enumerate(diagnostic_content_id_mean_map)\n# }\n \n # Fix lecture content_ids\n mask = (df_train.content_type_id == 1)\n df_train.loc[mask, 'content_id'] = df_train[mask].content_id.map(lecture_ids_map)\n df_train.loc[mask, 'answered_correctly'] = 1 # default for lectures, loss be masked anyway\n mask = (df_valid.content_type_id == 1)\n df_valid.loc[mask, 'content_id'] = df_valid[mask].content_id.map(lecture_ids_map)\n df_valid.loc[mask, 'answered_correctly'] = 1 # default for lectures, loss be masked anyway\n \n df_train['part_id'] = df_train.content_id.map(part_ids_map).astype(np.uint8)\n df_valid['part_id'] = df_valid.content_id.map(part_ids_map).astype(np.uint8)\n \n # Bundle IDs; 0=pad, 1=lecture\n bundle_id_map = {v:i+2 for i,v in enumerate(df_questions.bundle_id.unique())}\n df_train['bundle_id'] = df_train.content_id.map(bundle_id_map).fillna(0).astype(np.uint16)\n df_valid['bundle_id'] = df_valid.content_id.map(bundle_id_map).fillna(0).astype(np.uint16)\n \n #df_train['tag_group_id'] = df_train.content_id.map(tag_group_mapper).astype(np.uint16)\n #df_valid['tag_group_id'] = df_valid.content_id.map(tag_group_mapper).astype(np.uint16)\n \n ################################################################\n ## IMPORTANT : Question Incorrect Rank, we basically need this for the entire dataset\n ## And it should be calculated a-priori\n \n # How we calculated it:\n# question_incorrect_ranks = df[['content_id', 'user_answer', 'answered_correctly']]\n# question_incorrect_ranks = question_incorrect_ranks[question_incorrect_ranks.answered_correctly == 0]\n# question_incorrect_ranks = question_incorrect_ranks.groupby(['content_id','user_answer'], sort=False).count().reset_index()\n# question_incorrect_ranks.columns = ['content_id', 'user_answer', 'incorrect_rank']\n \n# # NOTE: We can also force ensure we have all {question_id,user_answer} pairs here............. so we dont have to fillna(0)....\n# # 0=pad, 1=starter, 2=correct, 3=wrong1, 4=wrong2, 5=wrong3\n# question_incorrect_ranks.sort_values(['content_id', 'incorrect_rank'], inplace=True)\n# question_incorrect_ranks['incorrect_rank'] = 3 + question_incorrect_ranks.groupby('content_id', sort=False).cumcount()\n\n# question_incorrect_ranks.content_id = question_incorrect_ranks.content_id.astype(np.int32)\n# question_incorrect_ranks.user_answer = question_incorrect_ranks.user_answer.astype(np.int8)\n# joblib.dump(question_incorrect_ranks, f'./question_incorrect_ranks.pkl')\n\n question_incorrect_ranks = joblib.load(f'./question_incorrect_ranks.pkl')\n \n df_train = df_train.merge(question_incorrect_ranks, how='left', on=['user_answer','content_id'])\n df_train.loc[df_train.incorrect_rank.isna(), 'incorrect_rank'] = 2 # fillna w/ correct\n df_train.incorrect_rank = df_train.incorrect_rank.astype(np.uint8)\n \n df_valid = df_valid.merge(question_incorrect_ranks, how='left', on=['user_answer','content_id'])\n df_valid.loc[df_valid.incorrect_rank.isna(), 'incorrect_rank'] = 2 # fillna w/ correct\n df_valid.incorrect_rank = df_valid.incorrect_rank.astype(np.uint8)\n ################################################################\n \n gc.collect()\n\n ###############################################################\n ######## IMPORTANT: NOTE: FOR VALID SET, WE SHOULD BE PROVIDING THE TRAINING SET HISTORY........\n ######## JUST LIKE IN INFERENCE; BUT IT SHOULDN'T COUNT.......... towards accuracy . loss\n # Sampling Strategy (for train)\n # Take first 500 records of all user ids in training data\n user_record_count = df_train.groupby('user_id', sort=False).size()\n\n # Scale the prior to make them sum 1 (probability distribution).\n user_record_count_priors = (user_record_count / user_record_count.sum())#.cumsum()\n Ntrn = user_record_count_priors.shape[0]\n\n num_slots = (user_record_count_priors / user_record_count_priors.min()).astype(np.uint16)\n train_user_id_sampling = np.concatenate([\n np.array([user_id] * slots, dtype=np.uint32)\n for user_id, slots in num_slots.to_dict().items()\n ])\n del user_record_count, user_record_count_priors, num_slots\n\n # Sampling Strategy (for valid)\n # Take first 500 records of all user ids in training data\n user_record_count = df_valid.groupby('user_id', sort=False).size()\n\n # Scale the prior to make them sum 1 (probability distribution).\n user_record_count_priors = (user_record_count / user_record_count.sum())#.cumsum()\n Nval = user_record_count_priors.shape[0]\n\n num_slots = (user_record_count_priors / user_record_count_priors.min()).astype(np.uint16)\n valid_user_id_sampling = np.concatenate([\n np.array([user_id] * slots, dtype=np.uint32)\n for user_id, slots in num_slots.to_dict().items()\n ])\n del user_record_count, user_record_count_priors, num_slots\n \n gc.collect()\n \n # Sample N ids with replacement with previous computed probabilities.\n # N in my case is the same as different user ids you have in the training data.\n # The result will have repeated ids.\n #samples = np.random.randint(0,user_id_sampling.shape[0], size=N)\n #samples = user_id_sampling[samples]\n ###############################################################\n \n user_features = defaultdict(newUserFeatures())\n \n # Index: 0 = per-ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3 \n session_content_num_u = np.zeros(df_train.shape[0], dtype=np.float32)\n session_duration_u = np.zeros(df_train.shape[0], dtype=np.float64)\n session_correctness_u = np.zeros(df_train.shape[0], dtype=np.float32)\n session_ans_duration_u= np.zeros(df_train.shape[0], dtype=np.float64)\n lifetime_ans_duration_u= np.zeros(df_train.shape[0], dtype=np.float64)\n lag_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency1 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency2 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency3 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_incorrect_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n last_correct_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n correctness_u_recency = np.zeros(df_train.shape[0], dtype=np.float32)\n part_correctness_u_recency = np.zeros(df_train.shape[0], dtype=np.float32)\n diagnostic_u_recency = np.zeros((df_train.shape[0],6), dtype=np.float32)\n answer_ratio1 = np.zeros(df_train.shape[0], dtype=np.float32)\n answer_ratio2 = np.zeros(df_train.shape[0], dtype=np.float32)\n correct_streak_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n incorrect_streak_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n correct_streak_alltime_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n incorrect_streak_alltime_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n encountered = np.zeros(df_train.shape[0], dtype=np.uint8)\n \n for idx, (_, user_id, content_id, task_container_id, content_type_id, part_id, timestamp, prior_question_elapsed_time, answered_correctly, user_answer) in enumerate(tqdm(df_train[[\n 'user_id', 'content_id', 'task_container_id', 'content_type_id', 'part_id', 'timestamp', 'prior_question_elapsed_time', 'answered_correctly', 'user_answer'\n ]].itertuples())):\n # Adjustments\n part_id -= 1 # So we can index at 0\n (\n cached_task_container_id_u,\n task_container_num_questions_u,\n last_question_ts_u_recency2,\n session_u,\n last_ts_u,\n last_correct_ts_u,\n last_incorrect_ts_u,\n correctness_u,\n part_correctness_u,\n diagnostic_u_1,\n diagnostic_u_2,\n diagnostic_u_3,\n diagnostic_u_4,\n diagnostic_u_5,\n diagnostic_u_6,\n answer_ratio,\n streak_u,\n content_encounter,\n ) = user_features[user_id].get_features()\n part_correctness_u__part = part_correctness_u[part_id]\n \n # Step 1) Bundle Alignment operation - in kernel submission, this will run each time\n if cached_task_container_id_u[0] != task_container_id:\n # Initialize:\n cached_task_container_id_u[0] = task_container_id\n \n # Index: 0 = per-ts, 1 = per-bundle\n last_correct_ts_u[1] = last_correct_ts_u[0]\n last_incorrect_ts_u[1] = last_incorrect_ts_u[0]\n\n # Index: 0,1 = per-ts, 2,3 = per-bundle. 0=correct qs, 1=total qs\n correctness_u[2] = correctness_u[0]\n correctness_u[3] = correctness_u[1]\n\n for part_id in range(7):\n # 7*4, [7parts][0=correct_ts,1=count_ts,2=correct_bundle,3=count_bundle]\n part_correctness_u[part_id][2] = part_correctness_u[part_id][0]\n part_correctness_u[part_id][3] = part_correctness_u[part_id][1]\n \n # Index: 0=current-ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3 \n # Order is important\n last_ts_u[3] = last_ts_u[2]\n last_ts_u[2] = last_ts_u[1]\n last_ts_u[1] = last_ts_u[0]\n last_ts_u[0] = timestamp\n \n # Streak\n streak_u[1] = streak_u[0] # Update bundle\n streak_u[4] = streak_u[3] # Update bundle\n if streak_u[2] < streak_u[1]: streak_u[2] = streak_u[1] # Update all-time\n if streak_u[5] < streak_u[4]: streak_u[5] = streak_u[4] # Update all-time\n \n # Answer Ratios\n answer_ratio[1] = answer_ratio[0]\n answer_ratio[3] = answer_ratio[2]\n answer_ratio[5] = answer_ratio[4]\n \n if task_container_id < 40:\n # Diagnostic Updates\n for i in range(2):\n if np.isnan(diagnostic_u_1[i]): continue\n diagnostic_u_1[i+2] = diagnostic_u_1[i]\n diagnostic_u_1[i] = np.nan\n\n for i in range(6):\n if np.isnan(diagnostic_u_2[i]): continue\n diagnostic_u_2[i+6] = diagnostic_u_2[i]\n diagnostic_u_2[i] = np.nan\n\n for i in range(3):\n if np.isnan(diagnostic_u_3[i]): continue\n diagnostic_u_3[i+3] = diagnostic_u_3[i]\n diagnostic_u_3[i] = np.nan\n\n for i in range(5):\n if not np.isnan(diagnostic_u_4[i]):\n diagnostic_u_4[i+5] = diagnostic_u_4[i]\n diagnostic_u_4[i] = np.nan\n if not np.isnan(diagnostic_u_5[i]):\n diagnostic_u_5[i+5] = diagnostic_u_5[i]\n diagnostic_u_5[i] = np.nan\n\n for i in range(7):\n if np.isnan(diagnostic_u_6[i]): continue\n diagnostic_u_6[i+7] = diagnostic_u_6[i]\n diagnostic_u_6[i] = np.nan\n \n # The means:\n diagnostic_u_1[-1] = np.mean(diagnostic_u_1[2:-1])\n diagnostic_u_2[-1] = np.mean(diagnostic_u_2[6:-1])\n diagnostic_u_3[-1] = np.mean(diagnostic_u_3[3:-1])\n diagnostic_u_4[-1] = np.mean(diagnostic_u_4[5:-1])\n diagnostic_u_5[-1] = np.mean(diagnostic_u_5[5:-1])\n diagnostic_u_6[-1] = np.mean(diagnostic_u_6[7:-1])\n # End diagnostic features\n ###\n \n ###\n # Lag features\n if content_type_id==0:\n # We only reset for questions; lectures will ffill()\n task_container_num_questions_u[1] = task_container_num_questions_u[0]\n task_container_num_questions_u[0] = 0\n \n # Same claculation as last_ts_u_recency2\n last_question_ts_u_recency2[2] = last_question_ts_u_recency2[1]\n last_question_ts_u_recency2[1] = last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[0] = timestamp\n \n if not np.isnan(prior_question_elapsed_time):\n session_u[5] += prior_question_elapsed_time * task_container_num_questions_u[1]\n session_u[6] += prior_question_elapsed_time * task_container_num_questions_u[1]\n else:\n # Entering a lecture:\n diff = timestamp - last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[2] += diff\n last_question_ts_u_recency2[1] += diff\n last_question_ts_u_recency2[0] = timestamp\n # End lag features\n \n # Copy the session stuff over\n session_u[3] = session_u[1]\n session_u[4] = session_u[2]\n \n ###\n \n # Bake values - note, none of these change per task_container_id,\n # even though we're recalculating them all every iteration......\n lag_ts_u_recency[idx] = last_question_ts_u_recency2[1] - last_question_ts_u_recency2[2] - prior_question_elapsed_time * task_container_num_questions_u[1]\n last_ts_u_recency1[idx] = timestamp - last_ts_u[1]\n last_ts_u_recency2[idx] = last_ts_u[1] - last_ts_u[2]\n last_ts_u_recency3[idx] = last_ts_u[2] - last_ts_u[3]\n last_correct_ts_u_recency[idx] = timestamp - last_correct_ts_u[1]\n last_incorrect_ts_u_recency[idx] = timestamp - last_incorrect_ts_u[1]\n correctness_u_recency[idx] = correctness_u[2] / correctness_u[3]\n part_correctness_u_recency[idx] = part_correctness_u__part[2] / part_correctness_u__part[3]\n session_content_num_u[idx] = session_u[4] - 2 # we start at 6....\n session_duration_u[idx] = 0 if np.isnan(session_u[0]) else timestamp - session_u[0]\n session_correctness_u[idx] = session_u[3] / session_u[4]\n session_ans_duration_u[idx] = session_u[5]\n lifetime_ans_duration_u[idx] = session_u[6]\n diagnostic_u_recency[idx,0] = diagnostic_u_1[-1]\n diagnostic_u_recency[idx,1] = diagnostic_u_2[-1]\n diagnostic_u_recency[idx,2] = diagnostic_u_3[-1]\n diagnostic_u_recency[idx,3] = diagnostic_u_4[-1]\n diagnostic_u_recency[idx,4] = diagnostic_u_5[-1]\n diagnostic_u_recency[idx,5] = diagnostic_u_6[-1]\n answer_ratio1[idx] = answer_ratio[1] / answer_ratio[5]\n answer_ratio2[idx] = answer_ratio[3] / answer_ratio[5]\n correct_streak_u[idx] = streak_u[1]\n incorrect_streak_u[idx] = streak_u[4]\n correct_streak_alltime_u[idx] = streak_u[2]\n incorrect_streak_alltime_u[idx] = streak_u[5]\n \n \n # Step 0) Handle session stuff\n if np.isnan(last_ts_u[0]) or timestamp - last_ts_u[0] > 3600000 // 2:\n # Reset the session\n session_u[0] = timestamp\n session_u[1] = 2 # correct\n session_u[2] = 3 # count\n session_u[3] = 2 # correct\n session_u[4] = 3 # count\n session_u[5] = 0 # CumAnsTimeSessionNoLeak\n \n if content_encounter[content_id] == 1:\n encountered[idx] = 2\n else:\n # 0 for pad\n encountered[idx] = 1\n \n if content_type_id==0:\n # Lag: Increase for each question we see\n task_container_num_questions_u[0] += 1\n answer_ratio[5] += 1\n \n # Answer Selection Ratio for questions only\n if user_answer==1:\n answer_ratio[0] += 1\n elif user_answer==2:\n answer_ratio[2] += 1\n\n # Content Counters\n correctness_u[1] += 1\n part_correctness_u__part[1] += 1\n session_u[2] += 1\n content_encounter[content_id] = 1\n \n if answered_correctly==0:\n last_incorrect_ts_u[0] = timestamp\n streak_u[0] = 0 # reset correct counter\n streak_u[3] += 1 # increment incorrect counter\n else:\n last_correct_ts_u[0] = timestamp\n streak_u[0] += 1 # increment correct counter\n streak_u[3] = 0 # reset incorrect counter\n\n # Correct Content Counters\n correctness_u[0] += 1\n part_correctness_u__part[0] += 1\n session_u[1] += 1\n \n # Diagnostic Features\n if task_container_id < 40:\n if content_id == 6911: diagnostic_u_1[0] = answered_correctly\n elif content_id == 7901: diagnostic_u_1[1] = answered_correctly\n elif content_id == 7219: diagnostic_u_2[0] = answered_correctly\n elif content_id == 6910: diagnostic_u_2[1] = answered_correctly\n elif content_id == 2066: diagnostic_u_2[2] = answered_correctly\n elif content_id == 6879: diagnostic_u_2[3] = answered_correctly\n elif content_id == 2065: diagnostic_u_2[4] = answered_correctly\n elif content_id == 1279: diagnostic_u_2[5] = answered_correctly\n elif content_id == 3365: diagnostic_u_3[0] = answered_correctly\n elif content_id == 4697: diagnostic_u_3[1] = answered_correctly\n elif content_id == 2594: diagnostic_u_3[2] = answered_correctly\n elif content_id == 6878: diagnostic_u_4[0] = answered_correctly\n elif content_id == 6912: diagnostic_u_4[1] = answered_correctly\n elif content_id == 4493: diagnostic_u_4[2] = answered_correctly\n elif content_id == 2596: diagnostic_u_4[3] = answered_correctly\n elif content_id == 7218: diagnostic_u_4[4] = answered_correctly\n elif content_id == 6909: diagnostic_u_5[0] = answered_correctly\n elif content_id == 7877: diagnostic_u_5[1] = answered_correctly\n elif content_id == 6880: diagnostic_u_5[2] = answered_correctly\n elif content_id == 7217: diagnostic_u_5[3] = answered_correctly\n elif content_id == 2595: diagnostic_u_5[4] = answered_correctly\n elif content_id == 2949: diagnostic_u_6[0] = answered_correctly\n elif content_id == 2948: diagnostic_u_6[1] = answered_correctly\n elif content_id == 4121: diagnostic_u_6[2] = answered_correctly\n elif content_id == 6881: diagnostic_u_6[3] = answered_correctly\n elif content_id == 6174: diagnostic_u_6[4] = answered_correctly\n elif content_id == 7220: diagnostic_u_6[5] = answered_correctly\n elif content_id == 3364: diagnostic_u_6[6] = answered_correctly\n\n df_train['lag_ts_u_recency'] = np.log1p(lag_ts_u_recency.clip(0,86400000) / 1000)\n df_train['last_ts_u_recency1'] = np.log1p(last_ts_u_recency1.clip(650,1209600000))\n df_train['last_ts_u_recency2'] = np.log1p(last_ts_u_recency2.clip(650,1209600000))\n df_train['last_ts_u_recency3'] = np.log1p(last_ts_u_recency3.clip(650,1209600000))\n df_train['last_correct_ts_u_recency'] = np.log1p(last_correct_ts_u_recency.clip(650,1209600000))\n df_train['last_incorrect_ts_u_recency'] = np.log1p(last_incorrect_ts_u_recency.clip(650,1209600000))\n df_train['correctness_u_recency'] = correctness_u_recency\n df_train['part_correctness_u_recency'] = part_correctness_u_recency\n df_train['session_content_num_u'] = np.log1p(session_content_num_u)\n df_train['session_duration_u'] = np.log1p(session_duration_u.clip(4100,18000000))\n df_train['session_correctness_u'] = session_correctness_u\n df_train['session_ans_duration_u'] = np.log1p(session_ans_duration_u.clip(4100,18000000))\n df_train['lifetime_ans_duration_u'] = np.log1p(lifetime_ans_duration_u.clip(4100,None))\n df_train['diagnostic_u_recency_1'] = diagnostic_u_recency[:, 0]\n df_train['diagnostic_u_recency_2'] = diagnostic_u_recency[:, 1]\n df_train['diagnostic_u_recency_3'] = diagnostic_u_recency[:, 2]\n df_train['diagnostic_u_recency_4'] = diagnostic_u_recency[:, 3]\n df_train['diagnostic_u_recency_5'] = diagnostic_u_recency[:, 4]\n df_train['diagnostic_u_recency_6'] = diagnostic_u_recency[:, 5]\n df_train['answer_ratio1'] = answer_ratio1\n df_train['answer_ratio2'] = answer_ratio2\n df_train['correct_streak_u'] = np.log1p(correct_streak_u)\n df_train['incorrect_streak_u'] = np.log1p(incorrect_streak_u)\n df_train['correct_streak_alltime_u'] = np.log1p(correct_streak_alltime_u)\n df_train['incorrect_streak_alltime_u'] = np.log1p(incorrect_streak_alltime_u)\n df_train['encountered'] = encountered\n \n lag_ts_u_recency_mean, lag_ts_u_recency_std = df_train.lag_ts_u_recency.mean(), df_train.lag_ts_u_recency.std()\n last_ts_u_recency1_mean, last_ts_u_recency1_std = df_train.last_ts_u_recency1.mean(), df_train.last_ts_u_recency1.std() \n last_ts_u_recency2_mean, last_ts_u_recency2_std = df_train.last_ts_u_recency2.mean(), df_train.last_ts_u_recency2.std() \n last_ts_u_recency3_mean, last_ts_u_recency3_std = df_train.last_ts_u_recency3.mean(), df_train.last_ts_u_recency3.std() \n last_incorrect_ts_u_recency_mean, last_incorrect_ts_u_recency_std = df_train.last_incorrect_ts_u_recency.mean(), df_train.last_incorrect_ts_u_recency.std() \n last_correct_ts_u_recency_mean, last_correct_ts_u_recency_std = df_train.last_correct_ts_u_recency.mean(), df_train.last_correct_ts_u_recency.std() \n correctness_u_recency_mean, correctness_u_recency_std = df_train.correctness_u_recency.mean(), df_train.correctness_u_recency.std() \n part_correctness_u_recency_mean, part_correctness_u_recency_std = df_train.part_correctness_u_recency.mean(), df_train.part_correctness_u_recency.std()\n session_content_num_u_mean, session_content_num_u_std = df_train.session_content_num_u.mean(), df_train.session_content_num_u.std()\n session_duration_u_mean, session_duration_u_std = df_train.session_duration_u.mean(), df_train.session_duration_u.std()\n session_correctness_u_mean, session_correctness_u_std = df_train.session_correctness_u.mean(), df_train.session_correctness_u.std() \n session_ans_duration_u_mean, session_ans_duration_u_std = df_train.session_ans_duration_u.mean(), df_train.session_ans_duration_u.std()\n lifetime_ans_duration_u_mean, lifetime_ans_duration_u_std = df_train.lifetime_ans_duration_u.mean(), df_train.lifetime_ans_duration_u.std()\n answer_ratio1_mean, answer_ratio1_std = df_train.answer_ratio1.mean(), df_train.answer_ratio1.std()\n answer_ratio2_mean, answer_ratio2_std = df_train.answer_ratio2.mean(), df_train.answer_ratio2.std()\n correct_streak_u_mean, correct_streak_u_std = df_train.correct_streak_u.mean(), df_train.correct_streak_u.std()\n incorrect_streak_u_mean, incorrect_streak_u_std = df_train.incorrect_streak_u.mean(), df_train.incorrect_streak_u.std()\n correct_streak_alltime_u_mean, correct_streak_alltime_u_std = df_train.correct_streak_alltime_u.mean(), df_train.correct_streak_alltime_u.std()\n incorrect_streak_alltime_u_mean, incorrect_streak_alltime_u_std = df_train.incorrect_streak_alltime_u.mean(), df_train.incorrect_streak_alltime_u.std()\n diagnostic_u_recency_1_mean, diagnostic_u_recency_1_std = df_train.diagnostic_u_recency_1.mean(), df_train.diagnostic_u_recency_1.std()\n diagnostic_u_recency_2_mean, diagnostic_u_recency_2_std = df_train.diagnostic_u_recency_2.mean(), df_train.diagnostic_u_recency_2.std()\n diagnostic_u_recency_3_mean, diagnostic_u_recency_3_std = df_train.diagnostic_u_recency_3.mean(), df_train.diagnostic_u_recency_3.std()\n diagnostic_u_recency_4_mean, diagnostic_u_recency_4_std = df_train.diagnostic_u_recency_4.mean(), df_train.diagnostic_u_recency_4.std()\n diagnostic_u_recency_5_mean, diagnostic_u_recency_5_std = df_train.diagnostic_u_recency_5.mean(), df_train.diagnostic_u_recency_5.std()\n diagnostic_u_recency_6_mean, diagnostic_u_recency_6_std = df_train.diagnostic_u_recency_6.mean(), df_train.diagnostic_u_recency_6.std()\n \n df_train.lag_ts_u_recency = ((df_train.lag_ts_u_recency - lag_ts_u_recency_mean) / lag_ts_u_recency_std).astype(np.float32) \n df_train.last_ts_u_recency1 = ((df_train.last_ts_u_recency1 - last_ts_u_recency1_mean) / last_ts_u_recency1_std).astype(np.float32) \n df_train.last_ts_u_recency2 = ((df_train.last_ts_u_recency2 - last_ts_u_recency2_mean) / last_ts_u_recency2_std).astype(np.float32) \n df_train.last_ts_u_recency3 = ((df_train.last_ts_u_recency3 - last_ts_u_recency3_mean) / last_ts_u_recency3_std).astype(np.float32) \n df_train.last_incorrect_ts_u_recency = ((df_train.last_incorrect_ts_u_recency - last_incorrect_ts_u_recency_mean) / last_incorrect_ts_u_recency_std).astype(np.float32)\n df_train.last_correct_ts_u_recency = ((df_train.last_correct_ts_u_recency - last_correct_ts_u_recency_mean ) / last_correct_ts_u_recency_std).astype(np.float32) \n df_train.correctness_u_recency = ((df_train.correctness_u_recency - correctness_u_recency_mean ) / correctness_u_recency_std).astype(np.float32) \n df_train.part_correctness_u_recency = ((df_train.part_correctness_u_recency - part_correctness_u_recency_mean ) / part_correctness_u_recency_std).astype(np.float32) \n df_train.session_content_num_u = ((df_train.session_content_num_u - session_content_num_u_mean ) / session_content_num_u_std).astype(np.float32) \n df_train.session_duration_u = ((df_train.session_duration_u - session_duration_u_mean ) / session_duration_u_std).astype(np.float32) \n df_train.session_correctness_u = ((df_train.session_correctness_u - session_correctness_u_mean ) / session_correctness_u_std).astype(np.float32) \n df_train.session_ans_duration_u = ((df_train.session_ans_duration_u - session_ans_duration_u_mean ) / session_ans_duration_u_std).astype(np.float32) \n df_train.lifetime_ans_duration_u= ((df_train.lifetime_ans_duration_u - lifetime_ans_duration_u_mean) / lifetime_ans_duration_u_std).astype(np.float32) \n df_train.diagnostic_u_recency_1 = ((df_train.diagnostic_u_recency_1 - diagnostic_u_recency_1_mean ) / diagnostic_u_recency_1_std).astype(np.float32) \n df_train.diagnostic_u_recency_2 = ((df_train.diagnostic_u_recency_2 - diagnostic_u_recency_2_mean ) / diagnostic_u_recency_2_std).astype(np.float32) \n df_train.diagnostic_u_recency_3 = ((df_train.diagnostic_u_recency_3 - diagnostic_u_recency_3_mean ) / diagnostic_u_recency_3_std).astype(np.float32) \n df_train.diagnostic_u_recency_4 = ((df_train.diagnostic_u_recency_4 - diagnostic_u_recency_4_mean ) / diagnostic_u_recency_4_std).astype(np.float32) \n df_train.diagnostic_u_recency_5 = ((df_train.diagnostic_u_recency_5 - diagnostic_u_recency_5_mean ) / diagnostic_u_recency_5_std).astype(np.float32) \n df_train.diagnostic_u_recency_6 = ((df_train.diagnostic_u_recency_6 - diagnostic_u_recency_6_mean ) / diagnostic_u_recency_6_std).astype(np.float32) \n df_train.answer_ratio1 = ((df_train.answer_ratio1 - answer_ratio1_mean ) / answer_ratio1_std).astype(np.float32) \n df_train.answer_ratio2 = ((df_train.answer_ratio2 - answer_ratio2_mean ) / answer_ratio2_std).astype(np.float32) \n df_train.correct_streak_u = ((df_train.correct_streak_u - correct_streak_u_mean ) / correct_streak_u_std).astype(np.float32) \n df_train.incorrect_streak_u = ((df_train.incorrect_streak_u - incorrect_streak_u_mean ) / incorrect_streak_u_std).astype(np.float32) \n df_train.correct_streak_alltime_u = ((df_train.correct_streak_alltime_u - correct_streak_alltime_u_mean ) / correct_streak_alltime_u_std).astype(np.float32) \n df_train.incorrect_streak_alltime_u = ((df_train.incorrect_streak_alltime_u - incorrect_streak_alltime_u_mean ) / incorrect_streak_alltime_u_std).astype(np.float32) \n\n \n df_train.loc[df_train.lag_ts_u_recency.isna(), 'lag_ts_u_recency'] = 0\n df_train.loc[df_train.last_ts_u_recency1.isna(), 'last_ts_u_recency1'] = 0\n df_train.loc[df_train.last_ts_u_recency2.isna(), 'last_ts_u_recency2'] = 0\n df_train.loc[df_train.last_ts_u_recency3.isna(), 'last_ts_u_recency3'] = 0\n df_train.loc[df_train.last_incorrect_ts_u_recency.isna(), 'last_incorrect_ts_u_recency'] = 0\n df_train.loc[df_train.last_correct_ts_u_recency.isna(), 'last_correct_ts_u_recency'] = 0\n df_train.loc[df_train.correctness_u_recency.isna(), 'correctness_u_recency'] = 0.653417715747257 # mean answered correctly\n df_train.loc[df_train.part_correctness_u_recency.isna(), 'part_correctness_u_recency'] = 0.653417715747257 # mean answered correctly across dset \n df_train.loc[df_train.session_content_num_u.isna(), 'session_content_num_u'] = 0\n df_train.loc[df_train.session_duration_u.isna(), 'session_duration_u'] = 0\n df_train.loc[df_train.session_correctness_u.isna(), 'session_correctness_u'] = 0.653417715747257\n df_train.loc[df_train.session_ans_duration_u.isna(), 'session_ans_duration_u'] = 0\n df_train.loc[df_train.lifetime_ans_duration_u.isna(), 'lifetime_ans_duration_u'] = 0\n df_train.loc[df_train.diagnostic_u_recency_1.isna(), 'diagnostic_u_recency_1'] = 0\n df_train.loc[df_train.diagnostic_u_recency_2.isna(), 'diagnostic_u_recency_2'] = 0\n df_train.loc[df_train.diagnostic_u_recency_3.isna(), 'diagnostic_u_recency_3'] = 0\n df_train.loc[df_train.diagnostic_u_recency_4.isna(), 'diagnostic_u_recency_4'] = 0\n df_train.loc[df_train.diagnostic_u_recency_5.isna(), 'diagnostic_u_recency_5'] = 0\n df_train.loc[df_train.diagnostic_u_recency_6.isna(), 'diagnostic_u_recency_6'] = 0\n df_train.loc[df_train.answer_ratio1.isna(), 'answer_ratio1'] = 0\n df_train.loc[df_train.answer_ratio2.isna(), 'answer_ratio2'] = 0\n df_train.loc[df_train.correct_streak_u.isna(), 'correct_streak_u'] = 0\n df_train.loc[df_train.incorrect_streak_u.isna(), 'incorrect_streak_u'] = 0\n df_train.loc[df_train.correct_streak_alltime_u.isna(), 'correct_streak_alltime_u'] = 0\n df_train.loc[df_train.incorrect_streak_alltime_u.isna(), 'incorrect_streak_alltime_u'] = 0\n \n ###\n \n # For val set, we continue adding onto existing dictionaries\n # But set the vectors = 0\n session_content_num_u = np.zeros(df_valid.shape[0], dtype=np.float32)\n session_duration_u = np.zeros(df_valid.shape[0], dtype=np.float64)\n session_correctness_u = np.zeros(df_valid.shape[0], dtype=np.float32)\n session_ans_duration_u= np.zeros(df_valid.shape[0], dtype=np.float64)\n lifetime_ans_duration_u= np.zeros(df_valid.shape[0], dtype=np.float64)\n lag_ts_u_recency = np.zeros(df_valid.shape[0], dtype=np.float64)\n last_ts_u_recency1 = np.zeros(df_valid.shape[0], dtype=np.float64)\n last_ts_u_recency2 = np.zeros(df_valid.shape[0], dtype=np.float64)\n last_ts_u_recency3 = np.zeros(df_valid.shape[0], dtype=np.float64)\n last_incorrect_ts_u_recency = np.zeros(df_valid.shape[0], dtype=np.float64)\n last_correct_ts_u_recency = np.zeros(df_valid.shape[0], dtype=np.float64)\n correctness_u_recency = np.zeros(df_valid.shape[0], dtype=np.float32)\n part_correctness_u_recency = np.zeros(df_valid.shape[0], dtype=np.float32)\n diagnostic_u_recency = np.zeros((df_valid.shape[0],6), dtype=np.float32)\n answer_ratio1 = np.zeros(df_valid.shape[0], dtype=np.float32)\n answer_ratio2 = np.zeros(df_valid.shape[0], dtype=np.float32)\n correct_streak_u = np.zeros(df_valid.shape[0], dtype=np.uint16)\n incorrect_streak_u = np.zeros(df_valid.shape[0], dtype=np.uint16)\n correct_streak_alltime_u = np.zeros(df_valid.shape[0], dtype=np.uint16)\n incorrect_streak_alltime_u = np.zeros(df_valid.shape[0], dtype=np.uint16)\n encountered = np.zeros(df_valid.shape[0], dtype=np.uint8)\n \n for idx, (_, user_id, content_id, task_container_id, content_type_id, part_id, timestamp, prior_question_elapsed_time, answered_correctly, user_answer) in enumerate(tqdm(df_valid[[ \n 'user_id', 'content_id', 'task_container_id', 'content_type_id', 'part_id', 'timestamp', 'prior_question_elapsed_time', 'answered_correctly', 'user_answer'\n ]].itertuples())):\n # Adjustments\n part_id -= 1 # So we can index at 0\n (\n cached_task_container_id_u,\n task_container_num_questions_u,\n last_question_ts_u_recency2,\n session_u,\n last_ts_u,\n last_correct_ts_u,\n last_incorrect_ts_u,\n correctness_u,\n part_correctness_u,\n diagnostic_u_1,\n diagnostic_u_2,\n diagnostic_u_3,\n diagnostic_u_4,\n diagnostic_u_5,\n diagnostic_u_6,\n answer_ratio,\n streak_u,\n content_encounter,\n ) = user_features[user_id].get_features()\n part_correctness_u__part = part_correctness_u[part_id]\n \n # Step 1) Bundle Alignment operation - in kernel submission, this will run each time\n if cached_task_container_id_u[0] != task_container_id:\n # Initialize:\n cached_task_container_id_u[0] = task_container_id\n \n # Index: 0 = per-ts, 1 = per-bundle\n last_correct_ts_u[1] = last_correct_ts_u[0]\n last_incorrect_ts_u[1] = last_incorrect_ts_u[0]\n\n # Index: 0,1 = per-ts, 2,3 = per-bundle. 0=correct qs, 1=total qs\n correctness_u[2] = correctness_u[0]\n correctness_u[3] = correctness_u[1]\n\n for part_id in range(7):\n # 7*4, [7parts][0=correct_ts,1=count_ts,2=correct_bundle,3=count_bundle]\n part_correctness_u[part_id][2] = part_correctness_u[part_id][0]\n part_correctness_u[part_id][3] = part_correctness_u[part_id][1]\n \n # Index: 0=current-ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3 \n # Order is important\n last_ts_u[3] = last_ts_u[2]\n last_ts_u[2] = last_ts_u[1]\n last_ts_u[1] = last_ts_u[0]\n last_ts_u[0] = timestamp\n \n # Streak\n streak_u[1] = streak_u[0] # Update bundle\n streak_u[4] = streak_u[3] # Update bundle\n if streak_u[2] < streak_u[1]: streak_u[2] = streak_u[1] # Update all-time\n if streak_u[5] < streak_u[4]: streak_u[5] = streak_u[4] # Update all-time\n \n # Answer Ratios\n answer_ratio[1] = answer_ratio[0]\n answer_ratio[3] = answer_ratio[2]\n answer_ratio[5] = answer_ratio[4]\n \n if task_container_id < 40:\n # Diagnostic Updates\n for i in range(2):\n if np.isnan(diagnostic_u_1[i]): continue\n diagnostic_u_1[i+2] = diagnostic_u_1[i]\n diagnostic_u_1[i] = np.nan\n\n for i in range(6):\n if np.isnan(diagnostic_u_2[i]): continue\n diagnostic_u_2[i+6] = diagnostic_u_2[i]\n diagnostic_u_2[i] = np.nan\n\n for i in range(3):\n if np.isnan(diagnostic_u_3[i]): continue\n diagnostic_u_3[i+3] = diagnostic_u_3[i]\n diagnostic_u_3[i] = np.nan\n\n for i in range(5):\n if not np.isnan(diagnostic_u_4[i]):\n diagnostic_u_4[i+5] = diagnostic_u_4[i]\n diagnostic_u_4[i] = np.nan\n if not np.isnan(diagnostic_u_5[i]):\n diagnostic_u_5[i+5] = diagnostic_u_5[i]\n diagnostic_u_5[i] = np.nan\n\n for i in range(7):\n if np.isnan(diagnostic_u_6[i]): continue\n diagnostic_u_6[i+7] = diagnostic_u_6[i]\n diagnostic_u_6[i] = np.nan\n \n # The means:\n diagnostic_u_1[-1] = np.mean(diagnostic_u_1[2:-1])\n diagnostic_u_2[-1] = np.mean(diagnostic_u_2[6:-1])\n diagnostic_u_3[-1] = np.mean(diagnostic_u_3[3:-1])\n diagnostic_u_4[-1] = np.mean(diagnostic_u_4[5:-1])\n diagnostic_u_5[-1] = np.mean(diagnostic_u_5[5:-1])\n diagnostic_u_6[-1] = np.mean(diagnostic_u_6[7:-1])\n # End diagnostic features\n ###\n \n ###\n # Lag features\n if content_type_id==0:\n # We only reset for questions; lectures will ffill()\n task_container_num_questions_u[1] = task_container_num_questions_u[0]\n task_container_num_questions_u[0] = 0\n \n # Same claculation as last_ts_u_recency2\n last_question_ts_u_recency2[2] = last_question_ts_u_recency2[1]\n last_question_ts_u_recency2[1] = last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[0] = timestamp\n \n if not np.isnan(prior_question_elapsed_time):\n session_u[5] += prior_question_elapsed_time * task_container_num_questions_u[1]\n session_u[6] += prior_question_elapsed_time * task_container_num_questions_u[1]\n else:\n # Entering a lecture:\n diff = timestamp - last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[2] += diff\n last_question_ts_u_recency2[1] += diff\n last_question_ts_u_recency2[0] = timestamp\n # End lag features\n \n # Copy the session stuff over\n session_u[3] = session_u[1]\n session_u[4] = session_u[2]\n \n ###\n \n # Bake values - note, none of these change per task_container_id,\n # even though we're recalculating them all every iteration......\n lag_ts_u_recency[idx] = last_question_ts_u_recency2[1] - last_question_ts_u_recency2[2] - prior_question_elapsed_time * task_container_num_questions_u[1]\n last_ts_u_recency1[idx] = timestamp - last_ts_u[1]\n last_ts_u_recency2[idx] = last_ts_u[1] - last_ts_u[2]\n last_ts_u_recency3[idx] = last_ts_u[2] - last_ts_u[3]\n last_correct_ts_u_recency[idx] = timestamp - last_correct_ts_u[1]\n last_incorrect_ts_u_recency[idx] = timestamp - last_incorrect_ts_u[1]\n correctness_u_recency[idx] = correctness_u[2] / correctness_u[3]\n part_correctness_u_recency[idx] = part_correctness_u__part[2] / part_correctness_u__part[3]\n session_content_num_u[idx] = session_u[4] - 2 # we start at 6....\n session_duration_u[idx] = 0 if np.isnan(session_u[0]) else timestamp - session_u[0]\n session_correctness_u[idx] = session_u[3] / session_u[4]\n session_ans_duration_u[idx] = session_u[5]\n lifetime_ans_duration_u[idx] = session_u[6]\n diagnostic_u_recency[idx,0] = diagnostic_u_1[-1]\n diagnostic_u_recency[idx,1] = diagnostic_u_2[-1]\n diagnostic_u_recency[idx,2] = diagnostic_u_3[-1]\n diagnostic_u_recency[idx,3] = diagnostic_u_4[-1]\n diagnostic_u_recency[idx,4] = diagnostic_u_5[-1]\n diagnostic_u_recency[idx,5] = diagnostic_u_6[-1]\n answer_ratio1[idx] = answer_ratio[1] / answer_ratio[5]\n answer_ratio2[idx] = answer_ratio[3] / answer_ratio[5]\n correct_streak_u[idx] = streak_u[1]\n incorrect_streak_u[idx] = streak_u[4]\n correct_streak_alltime_u[idx] = streak_u[2]\n incorrect_streak_alltime_u[idx] = streak_u[5]\n \n \n # Step 0) Handle session stuff\n if np.isnan(last_ts_u[0]) or timestamp - last_ts_u[0] > 3600000 // 2:\n # Reset the session\n session_u[0] = timestamp\n session_u[1] = 2 # correct\n session_u[2] = 3 # count\n session_u[3] = 2 # correct\n session_u[4] = 3 # count\n session_u[5] = 0 # CumAnsTimeSessionNoLeak\n \n if content_encounter[content_id] == 1:\n encountered[idx] = 2\n else:\n # 0 for pad\n encountered[idx] = 1\n \n if content_type_id==0:\n # Lag: Increase for each question we see\n task_container_num_questions_u[0] += 1\n answer_ratio[5] += 1\n \n # Answer Selection Ratio for questions only\n if user_answer==1:\n answer_ratio[0] += 1\n elif user_answer==2:\n answer_ratio[2] += 1\n\n # Content Counters\n correctness_u[1] += 1\n part_correctness_u__part[1] += 1\n session_u[2] += 1\n content_encounter[content_id] = 1\n \n if answered_correctly==0:\n last_incorrect_ts_u[0] = timestamp\n streak_u[0] = 0 # reset correct counter\n streak_u[3] += 1 # increment incorrect counter\n else:\n last_correct_ts_u[0] = timestamp\n streak_u[0] += 1 # increment correct counter\n streak_u[3] = 0 # reset incorrect counter\n\n # Correct Content Counters\n correctness_u[0] += 1\n part_correctness_u__part[0] += 1\n session_u[1] += 1\n \n # Diagnostic Features\n if task_container_id < 40:\n if content_id == 6911: diagnostic_u_1[0] = answered_correctly\n elif content_id == 7901: diagnostic_u_1[1] = answered_correctly\n elif content_id == 7219: diagnostic_u_2[0] = answered_correctly\n elif content_id == 6910: diagnostic_u_2[1] = answered_correctly\n elif content_id == 2066: diagnostic_u_2[2] = answered_correctly\n elif content_id == 6879: diagnostic_u_2[3] = answered_correctly\n elif content_id == 2065: diagnostic_u_2[4] = answered_correctly\n elif content_id == 1279: diagnostic_u_2[5] = answered_correctly\n elif content_id == 3365: diagnostic_u_3[0] = answered_correctly\n elif content_id == 4697: diagnostic_u_3[1] = answered_correctly\n elif content_id == 2594: diagnostic_u_3[2] = answered_correctly\n elif content_id == 6878: diagnostic_u_4[0] = answered_correctly\n elif content_id == 6912: diagnostic_u_4[1] = answered_correctly\n elif content_id == 4493: diagnostic_u_4[2] = answered_correctly\n elif content_id == 2596: diagnostic_u_4[3] = answered_correctly\n elif content_id == 7218: diagnostic_u_4[4] = answered_correctly\n elif content_id == 6909: diagnostic_u_5[0] = answered_correctly\n elif content_id == 7877: diagnostic_u_5[1] = answered_correctly\n elif content_id == 6880: diagnostic_u_5[2] = answered_correctly\n elif content_id == 7217: diagnostic_u_5[3] = answered_correctly\n elif content_id == 2595: diagnostic_u_5[4] = answered_correctly\n elif content_id == 2949: diagnostic_u_6[0] = answered_correctly\n elif content_id == 2948: diagnostic_u_6[1] = answered_correctly\n elif content_id == 4121: diagnostic_u_6[2] = answered_correctly\n elif content_id == 6881: diagnostic_u_6[3] = answered_correctly\n elif content_id == 6174: diagnostic_u_6[4] = answered_correctly\n elif content_id == 7220: diagnostic_u_6[5] = answered_correctly\n elif content_id == 3364: diagnostic_u_6[6] = answered_correctly\n\n \n # No need to recalculate mean, std - just use train sets\n df_valid['lag_ts_u_recency'] = np.log1p(lag_ts_u_recency.clip(0,86400000) / 1000)\n df_valid['last_ts_u_recency1'] = np.log1p(last_ts_u_recency1.clip(650,1209600000))\n df_valid['last_ts_u_recency2'] = np.log1p(last_ts_u_recency2.clip(650,1209600000))\n df_valid['last_ts_u_recency3'] = np.log1p(last_ts_u_recency3.clip(650,1209600000))\n df_valid['last_correct_ts_u_recency'] = np.log1p(last_correct_ts_u_recency.clip(650,1209600000))\n df_valid['last_incorrect_ts_u_recency'] = np.log1p(last_incorrect_ts_u_recency.clip(650,1209600000))\n df_valid['correctness_u_recency'] = correctness_u_recency\n df_valid['part_correctness_u_recency'] = part_correctness_u_recency\n df_valid['session_content_num_u'] = np.log1p(session_content_num_u)\n df_valid['session_duration_u'] = np.log1p(session_duration_u.clip(4100,18000000))\n df_valid['session_correctness_u'] = session_correctness_u\n df_valid['session_ans_duration_u'] = np.log1p(session_ans_duration_u.clip(4100,18000000))\n df_valid['lifetime_ans_duration_u'] = np.log1p(lifetime_ans_duration_u.clip(4100,None))\n df_valid['diagnostic_u_recency_1'] = diagnostic_u_recency[:, 0]\n df_valid['diagnostic_u_recency_2'] = diagnostic_u_recency[:, 1]\n df_valid['diagnostic_u_recency_3'] = diagnostic_u_recency[:, 2]\n df_valid['diagnostic_u_recency_4'] = diagnostic_u_recency[:, 3]\n df_valid['diagnostic_u_recency_5'] = diagnostic_u_recency[:, 4]\n df_valid['diagnostic_u_recency_6'] = diagnostic_u_recency[:, 5]\n df_valid['answer_ratio1'] = answer_ratio1\n df_valid['answer_ratio2'] = answer_ratio2\n df_valid['correct_streak_u'] = np.log1p(correct_streak_u)\n df_valid['incorrect_streak_u'] = np.log1p(incorrect_streak_u)\n df_valid['correct_streak_alltime_u'] = np.log1p(correct_streak_alltime_u)\n df_valid['incorrect_streak_alltime_u'] = np.log1p(incorrect_streak_alltime_u)\n df_valid['encountered'] = encountered\n \n df_valid.lag_ts_u_recency = ((df_valid.lag_ts_u_recency - lag_ts_u_recency_mean) / lag_ts_u_recency_std).astype(np.float32) \n df_valid.last_ts_u_recency1 = ((df_valid.last_ts_u_recency1 - last_ts_u_recency1_mean) / last_ts_u_recency1_std).astype(np.float32) \n df_valid.last_ts_u_recency2 = ((df_valid.last_ts_u_recency2 - last_ts_u_recency2_mean) / last_ts_u_recency2_std).astype(np.float32) \n df_valid.last_ts_u_recency3 = ((df_valid.last_ts_u_recency3 - last_ts_u_recency3_mean) / last_ts_u_recency3_std).astype(np.float32) \n df_valid.last_incorrect_ts_u_recency = ((df_valid.last_incorrect_ts_u_recency - last_incorrect_ts_u_recency_mean) / last_incorrect_ts_u_recency_std).astype(np.float32)\n df_valid.last_correct_ts_u_recency = ((df_valid.last_correct_ts_u_recency - last_correct_ts_u_recency_mean ) / last_correct_ts_u_recency_std).astype(np.float32) \n df_valid.correctness_u_recency = ((df_valid.correctness_u_recency - correctness_u_recency_mean ) / correctness_u_recency_std).astype(np.float32) \n df_valid.part_correctness_u_recency = ((df_valid.part_correctness_u_recency - part_correctness_u_recency_mean ) / part_correctness_u_recency_std).astype(np.float32) \n df_valid.session_content_num_u = ((df_valid.session_content_num_u - session_content_num_u_mean ) / session_content_num_u_std).astype(np.float32) \n df_valid.session_duration_u = ((df_valid.session_duration_u - session_duration_u_mean ) / session_duration_u_std).astype(np.float32) \n df_valid.session_correctness_u = ((df_valid.session_correctness_u - session_correctness_u_mean ) / session_correctness_u_std).astype(np.float32) \n df_valid.session_ans_duration_u = ((df_valid.session_ans_duration_u - session_ans_duration_u_mean ) / session_ans_duration_u_std).astype(np.float32) \n df_valid.lifetime_ans_duration_u= ((df_valid.lifetime_ans_duration_u - lifetime_ans_duration_u_mean) / lifetime_ans_duration_u_std).astype(np.float32) \n df_valid.diagnostic_u_recency_1 = ((df_valid.diagnostic_u_recency_1 - diagnostic_u_recency_1_mean ) / diagnostic_u_recency_1_std).astype(np.float32) \n df_valid.diagnostic_u_recency_2 = ((df_valid.diagnostic_u_recency_2 - diagnostic_u_recency_2_mean ) / diagnostic_u_recency_2_std).astype(np.float32) \n df_valid.diagnostic_u_recency_3 = ((df_valid.diagnostic_u_recency_3 - diagnostic_u_recency_3_mean ) / diagnostic_u_recency_3_std).astype(np.float32) \n df_valid.diagnostic_u_recency_4 = ((df_valid.diagnostic_u_recency_4 - diagnostic_u_recency_4_mean ) / diagnostic_u_recency_4_std).astype(np.float32) \n df_valid.diagnostic_u_recency_5 = ((df_valid.diagnostic_u_recency_5 - diagnostic_u_recency_5_mean ) / diagnostic_u_recency_5_std).astype(np.float32) \n df_valid.diagnostic_u_recency_6 = ((df_valid.diagnostic_u_recency_6 - diagnostic_u_recency_6_mean ) / diagnostic_u_recency_6_std).astype(np.float32) \n df_valid.answer_ratio1 = ((df_valid.answer_ratio1 - answer_ratio1_mean ) / answer_ratio1_std).astype(np.float32) \n df_valid.answer_ratio2 = ((df_valid.answer_ratio2 - answer_ratio2_mean ) / answer_ratio2_std).astype(np.float32) \n df_valid.correct_streak_u = ((df_valid.correct_streak_u - correct_streak_u_mean ) / correct_streak_u_std).astype(np.float32) \n df_valid.incorrect_streak_u = ((df_valid.incorrect_streak_u - incorrect_streak_u_mean ) / incorrect_streak_u_std).astype(np.float32) \n df_valid.correct_streak_alltime_u = ((df_valid.correct_streak_alltime_u - correct_streak_alltime_u_mean ) / correct_streak_alltime_u_std).astype(np.float32) \n df_valid.incorrect_streak_alltime_u = ((df_valid.incorrect_streak_alltime_u - incorrect_streak_alltime_u_mean ) / incorrect_streak_alltime_u_std).astype(np.float32) \n \n df_valid.loc[df_valid.lag_ts_u_recency.isna(), 'lag_ts_u_recency'] = 0\n df_valid.loc[df_valid.last_ts_u_recency1.isna(), 'last_ts_u_recency1'] = 0\n df_valid.loc[df_valid.last_ts_u_recency2.isna(), 'last_ts_u_recency2'] = 0\n df_valid.loc[df_valid.last_ts_u_recency3.isna(), 'last_ts_u_recency3'] = 0\n df_valid.loc[df_valid.last_incorrect_ts_u_recency.isna(), 'last_incorrect_ts_u_recency'] = 0\n df_valid.loc[df_valid.last_correct_ts_u_recency.isna(), 'last_correct_ts_u_recency'] = 0\n df_valid.loc[df_valid.correctness_u_recency.isna(), 'correctness_u_recency'] = 0.653417715747257 # mean answered correctly\n df_valid.loc[df_valid.part_correctness_u_recency.isna(), 'part_correctness_u_recency'] = 0.653417715747257 # mean answered correctly across dset TODO This is probably wrong\n df_valid.loc[df_valid.session_content_num_u.isna(), 'session_content_num_u'] = 0\n df_valid.loc[df_valid.session_duration_u.isna(), 'session_duration_u'] = 0\n df_valid.loc[df_valid.session_correctness_u.isna(), 'session_correctness_u'] = 0.653417715747257\n df_valid.loc[df_valid.session_ans_duration_u.isna(), 'session_ans_duration_u'] = 0\n df_valid.loc[df_valid.lifetime_ans_duration_u.isna(), 'lifetime_ans_duration_u'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_1.isna(), 'diagnostic_u_recency_1'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_2.isna(), 'diagnostic_u_recency_2'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_3.isna(), 'diagnostic_u_recency_3'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_4.isna(), 'diagnostic_u_recency_4'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_5.isna(), 'diagnostic_u_recency_5'] = 0\n df_valid.loc[df_valid.diagnostic_u_recency_6.isna(), 'diagnostic_u_recency_6'] = 0\n df_valid.loc[df_valid.answer_ratio1.isna(), 'answer_ratio1'] = 0\n df_valid.loc[df_valid.answer_ratio2.isna(), 'answer_ratio2'] = 0\n df_valid.loc[df_valid.correct_streak_u.isna(), 'correct_streak_u'] = 0\n df_valid.loc[df_valid.incorrect_streak_u.isna(), 'incorrect_streak_u'] = 0\n df_valid.loc[df_valid.correct_streak_alltime_u.isna(), 'correct_streak_alltime_u'] = 0\n df_valid.loc[df_valid.incorrect_streak_alltime_u.isna(), 'incorrect_streak_alltime_u'] = 0\n \n ################################################################\n for df in df_train, df_valid:\n # This part is important though:\n # Convert to seconds for embedding, then mean fill across entire datasert\n # 0 = pad/NA\n # 1 = starter\n # 2 = ....\n #df.prior_question_elapsed_time.fillna(MEAN_prior_question_elapsed_time, inplace=True)\n df.prior_question_elapsed_time //= 1000 # for embedding 0-300, looks already capped\n df.loc[df.prior_question_elapsed_time.isna(), 'prior_question_elapsed_time'] = -2\n df.prior_question_elapsed_time += 2\n \n ###############################################################\n # Since there are users in train that are also in val,\n # we need to extend into val the train indices (historical data).\n # But we only want to _evaluate_ on the new valid portion.\n # How the hell we do that? For now, we'll just keep them entirely separate.\n train_prev_user_id, train_user_id_indices = None, {}\n \n for idx, user_id in enumerate(tqdm(df_train.user_id.values)):\n if train_prev_user_id != user_id:\n if train_prev_user_id != None:\n # End: Not inclusive\n train_user_id_indices[train_prev_user_id].append(idx)\n\n # Start: Inclusive\n train_user_id_indices[user_id] = [idx]\n train_prev_user_id = user_id\n \n train_user_id_indices[train_prev_user_id].append(df_train.shape[0])\n \n\n valid_prev_user_id, valid_user_id_indices = None, {}\n for idx, user_id in enumerate(tqdm(df_valid.user_id.values)):\n if valid_prev_user_id != user_id:\n if valid_prev_user_id != None:\n # End: Not inclusive\n valid_user_id_indices[valid_prev_user_id].append(idx)\n\n # Start: Inclusive\n valid_user_id_indices[user_id] = [idx]\n valid_prev_user_id = user_id\n\n valid_user_id_indices[valid_prev_user_id].append(df_valid.shape[0])\n ###############################################################\n \n df_train = df_train[dataset_cols]\n df_valid = df_valid[dataset_cols]\n \n print(time()-t, 'seconds')\n return (\n df_train, df_valid, Ntrn, Nval,\n embedded_user_ids_map, mask_user_ids, part_ids_map, lecture_ids_map, bundle_id_map,\n train_user_id_sampling, valid_user_id_sampling,\n train_user_id_indices, valid_user_id_indices,\n question_incorrect_ranks,\n answer_ratio1_mean, answer_ratio1_std,\n answer_ratio2_mean, answer_ratio2_std,\n correct_streak_u_mean, correct_streak_u_std,\n incorrect_streak_u_mean, incorrect_streak_u_std,\n correct_streak_alltime_u_mean, correct_streak_alltime_u_std,\n incorrect_streak_alltime_u_mean, incorrect_streak_alltime_u_std,\n diagnostic_u_recency_1_mean, diagnostic_u_recency_1_std,\n diagnostic_u_recency_2_mean, diagnostic_u_recency_2_std,\n diagnostic_u_recency_3_mean, diagnostic_u_recency_3_std,\n diagnostic_u_recency_4_mean, diagnostic_u_recency_4_std,\n diagnostic_u_recency_5_mean, diagnostic_u_recency_5_std,\n diagnostic_u_recency_6_mean, diagnostic_u_recency_6_std,\n session_content_num_u_mean, session_content_num_u_std,\n session_duration_u_mean, session_duration_u_std,\n session_correctness_u_mean, session_correctness_u_std,\n session_ans_duration_u_mean, session_ans_duration_u_std,\n lifetime_ans_duration_u_mean, lifetime_ans_duration_u_std,\n correctness_u_recency_mean, correctness_u_recency_std,\n part_correctness_u_recency_mean, part_correctness_u_recency_std,\n lag_ts_u_recency_mean, lag_ts_u_recency_std,\n last_ts_u_recency1_mean, last_ts_u_recency1_std,\n last_ts_u_recency2_mean, last_ts_u_recency2_std,\n last_ts_u_recency3_mean, last_ts_u_recency3_std,\n last_incorrect_ts_u_recency_mean, last_incorrect_ts_u_recency_std,\n last_correct_ts_u_recency_mean, last_correct_ts_u_recency_std,\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def newUserFeatures(): return UserFeatures()\ndef newUserDF(): return UserDF()\ndef prepare_sub_data(fold_number):\n # TODO: Create the prior_elapsed_time_cont feature!!!!!!!\n \n # TODO: Load up our mappings.\n # For ensemble, unfortunatley we have to use the same fold, no time for dual feature standarizations...\n \n # user_features = joblib.load('../input/uthman-riiid/user_features_full.jlib')\n # user_states = joblib.load('../input/uthman-riiid/user_states_full.jlib') # last 128 records dataset format\n t = time()\n\n # TODO: Test with a small one first... \n df_train = pd.read_parquet(f'../input/cv-strategy-in-the-kaggle-environment/cv{fold_number+1}_train_sorted.parquet')[TRAIN_DTYPES.keys()]\n df_valid = pd.read_parquet(f'../input/cv-strategy-in-the-kaggle-environment/cv{fold_number+1}_valid_sorted.parquet')[TRAIN_DTYPES.keys()] \n df_train = df_train.append(df_valid, sort=False)\n df_train.reset_index(inplace=True, drop=True)\n del df_valid\n df_train.sort_values(['user_id','timestamp'], inplace=True)\n \n # Optional\n #df_train.user_id = df_train.user_id.map(embedded_user_ids_map).astype(np.uint32)\n df_train.prior_question_had_explanation = (df_train.prior_question_had_explanation.astype(np.float16).fillna(-2) + 2).astype(np.uint8)\n \n df_train.content_id += 1\n \n mask = (df_train.content_type_id == 1)\n df_train.loc[mask, 'content_id'] = df_train[mask].content_id.map(lecture_ids_map)\n df_train.loc[mask, 'answered_correctly'] = 1 # default for lectures, loss be masked anyway\n \n df_train['part_id'] = df_train.content_id.map(part_ids_map).astype(np.uint8)\n df_train['bundle_id'] = df_train.content_id.map(bundle_id_map).fillna(0).astype(np.uint16)\n #question_incorrect_ranks = joblib.load(f'./question_incorrect_ranks.pkl')\n #df_train = df_train.merge(question_incorrect_ranks, how='left', on=['user_answer','content_id'])\n #df_train.loc[df_train.incorrect_rank.isna(), 'incorrect_rank'] = 2 # fillna w/ correct\n #df_train.incorrect_rank = df_train.incorrect_rank.astype(np.uint8)\n \n gc.collect()\n \n user_features = defaultdict(newUserFeatures)\n \n # Index: 0 = per-ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3 \n session_content_num_u = np.zeros(df_train.shape[0], dtype=np.float32)\n session_duration_u = np.zeros(df_train.shape[0], dtype=np.float64)\n session_correctness_u = np.zeros(df_train.shape[0], dtype=np.float32)\n session_ans_duration_u= np.zeros(df_train.shape[0], dtype=np.float64)\n lifetime_ans_duration_u= np.zeros(df_train.shape[0], dtype=np.float64)\n lag_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency1 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency2 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_ts_u_recency3 = np.zeros(df_train.shape[0], dtype=np.float64)\n last_incorrect_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n last_correct_ts_u_recency = np.zeros(df_train.shape[0], dtype=np.float64)\n correctness_u_recency = np.zeros(df_train.shape[0], dtype=np.float32)\n part_correctness_u_recency = np.zeros(df_train.shape[0], dtype=np.float32)\n diagnostic_u_recency = np.zeros((df_train.shape[0],6), dtype=np.float32)\n answer_ratio1 = np.zeros(df_train.shape[0], dtype=np.float32)\n answer_ratio2 = np.zeros(df_train.shape[0], dtype=np.float32)\n correct_streak_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n incorrect_streak_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n correct_streak_alltime_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n incorrect_streak_alltime_u = np.zeros(df_train.shape[0], dtype=np.uint16)\n encountered = np.zeros(df_train.shape[0], dtype=np.uint8)\n \n for idx, (_, user_id, content_id, task_container_id, content_type_id, part_id, timestamp, prior_question_elapsed_time, answered_correctly, user_answer) in enumerate(tqdm(df_train[[\n 'user_id', 'content_id', 'task_container_id', 'content_type_id', 'part_id', 'timestamp', 'prior_question_elapsed_time', 'answered_correctly', 'user_answer'\n ]].itertuples())):\n # Adjustments\n part_id -= 1 # So we can index at 0\n (\n cached_task_container_id_u,\n task_container_num_questions_u,\n last_question_ts_u_recency2,\n session_u,\n last_ts_u,\n last_correct_ts_u,\n last_incorrect_ts_u,\n correctness_u,\n part_correctness_u,\n diagnostic_u_1,\n diagnostic_u_2,\n diagnostic_u_3,\n diagnostic_u_4,\n diagnostic_u_5,\n diagnostic_u_6,\n answer_ratio,\n streak_u,\n content_encounter,\n ) = user_features[user_id].get_features()\n part_correctness_u__part = part_correctness_u[part_id]\n \n # Step 1) Bundle Alignment operation - in kernel submission, this will run each time\n if cached_task_container_id_u[0] != task_container_id:\n # Initialize:\n cached_task_container_id_u[0] = task_container_id\n \n # Index: 0 = per-ts, 1 = per-bundle\n last_correct_ts_u[1] = last_correct_ts_u[0]\n last_incorrect_ts_u[1] = last_incorrect_ts_u[0]\n\n # Index: 0,1 = per-ts, 2,3 = per-bundle. 0=correct qs, 1=total qs\n correctness_u[2] = correctness_u[0]\n correctness_u[3] = correctness_u[1]\n\n for part_id in range(7):\n # 7*4, [7parts][0=correct_ts,1=count_ts,2=correct_bundle,3=count_bundle]\n part_correctness_u[part_id][2] = part_correctness_u[part_id][0]\n part_correctness_u[part_id][3] = part_correctness_u[part_id][1]\n \n # Index: 0=current-ts, 1 = per-bundle1, 2 = per-bundle2, 3 = per-bundle3 \n # Order is important\n last_ts_u[3] = last_ts_u[2]\n last_ts_u[2] = last_ts_u[1]\n last_ts_u[1] = last_ts_u[0]\n last_ts_u[0] = timestamp\n \n # Streak\n streak_u[1] = streak_u[0] # Update bundle\n streak_u[4] = streak_u[3] # Update bundle\n if streak_u[2] < streak_u[1]: streak_u[2] = streak_u[1] # Update all-time\n if streak_u[5] < streak_u[4]: streak_u[5] = streak_u[4] # Update all-time\n \n # Answer Ratios\n answer_ratio[1] = answer_ratio[0]\n answer_ratio[3] = answer_ratio[2]\n answer_ratio[5] = answer_ratio[4]\n \n if task_container_id < 40:\n # Diagnostic Updates\n for i in range(2):\n if np.isnan(diagnostic_u_1[i]): continue\n diagnostic_u_1[i+2] = diagnostic_u_1[i]\n diagnostic_u_1[i] = np.nan\n\n for i in range(6):\n if np.isnan(diagnostic_u_2[i]): continue\n diagnostic_u_2[i+6] = diagnostic_u_2[i]\n diagnostic_u_2[i] = np.nan\n\n for i in range(3):\n if np.isnan(diagnostic_u_3[i]): continue\n diagnostic_u_3[i+3] = diagnostic_u_3[i]\n diagnostic_u_3[i] = np.nan\n\n for i in range(5):\n if not np.isnan(diagnostic_u_4[i]):\n diagnostic_u_4[i+5] = diagnostic_u_4[i]\n diagnostic_u_4[i] = np.nan\n if not np.isnan(diagnostic_u_5[i]):\n diagnostic_u_5[i+5] = diagnostic_u_5[i]\n diagnostic_u_5[i] = np.nan\n\n for i in range(7):\n if np.isnan(diagnostic_u_6[i]): continue\n diagnostic_u_6[i+7] = diagnostic_u_6[i]\n diagnostic_u_6[i] = np.nan\n \n # The means:\n diagnostic_u_1[-1] = np.mean(diagnostic_u_1[2:-1])\n diagnostic_u_2[-1] = np.mean(diagnostic_u_2[6:-1])\n diagnostic_u_3[-1] = np.mean(diagnostic_u_3[3:-1])\n diagnostic_u_4[-1] = np.mean(diagnostic_u_4[5:-1])\n diagnostic_u_5[-1] = np.mean(diagnostic_u_5[5:-1])\n diagnostic_u_6[-1] = np.mean(diagnostic_u_6[7:-1])\n # End diagnostic features\n ###\n \n ###\n # Lag features\n if content_type_id==0:\n # We only reset for questions; lectures will ffill()\n task_container_num_questions_u[1] = task_container_num_questions_u[0]\n task_container_num_questions_u[0] = 0\n \n # Same claculation as last_ts_u_recency2\n last_question_ts_u_recency2[2] = last_question_ts_u_recency2[1]\n last_question_ts_u_recency2[1] = last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[0] = timestamp\n \n if not np.isnan(prior_question_elapsed_time):\n session_u[5] += prior_question_elapsed_time * task_container_num_questions_u[1]\n session_u[6] += prior_question_elapsed_time * task_container_num_questions_u[1]\n else:\n # Entering a lecture:\n diff = timestamp - last_question_ts_u_recency2[0]\n last_question_ts_u_recency2[2] += diff\n last_question_ts_u_recency2[1] += diff\n last_question_ts_u_recency2[0] = timestamp\n # End lag features\n \n # Copy the session stuff over\n session_u[3] = session_u[1]\n session_u[4] = session_u[2]\n \n ###\n \n # Bake values - note, none of these change per task_container_id,\n # even though we're recalculating them all every iteration......\n lag_ts_u_recency[idx] = last_question_ts_u_recency2[1] - last_question_ts_u_recency2[2] - prior_question_elapsed_time * task_container_num_questions_u[1]\n last_ts_u_recency1[idx] = timestamp - last_ts_u[1]\n last_ts_u_recency2[idx] = last_ts_u[1] - last_ts_u[2]\n last_ts_u_recency3[idx] = last_ts_u[2] - last_ts_u[3]\n last_correct_ts_u_recency[idx] = timestamp - last_correct_ts_u[1]\n last_incorrect_ts_u_recency[idx] = timestamp - last_incorrect_ts_u[1]\n correctness_u_recency[idx] = correctness_u[2] / correctness_u[3]\n part_correctness_u_recency[idx] = part_correctness_u__part[2] / part_correctness_u__part[3]\n session_content_num_u[idx] = session_u[4] - 2 # we start at 6....\n session_duration_u[idx] = 0 if np.isnan(session_u[0]) else timestamp - session_u[0]\n session_correctness_u[idx] = session_u[3] / session_u[4]\n session_ans_duration_u[idx] = session_u[5]\n lifetime_ans_duration_u[idx] = session_u[6]\n diagnostic_u_recency[idx,0] = diagnostic_u_1[-1]\n diagnostic_u_recency[idx,1] = diagnostic_u_2[-1]\n diagnostic_u_recency[idx,2] = diagnostic_u_3[-1]\n diagnostic_u_recency[idx,3] = diagnostic_u_4[-1]\n diagnostic_u_recency[idx,4] = diagnostic_u_5[-1]\n diagnostic_u_recency[idx,5] = diagnostic_u_6[-1]\n answer_ratio1[idx] = answer_ratio[1] / answer_ratio[5]\n answer_ratio2[idx] = answer_ratio[3] / answer_ratio[5]\n correct_streak_u[idx] = streak_u[1]\n incorrect_streak_u[idx] = streak_u[4]\n correct_streak_alltime_u[idx] = streak_u[2]\n incorrect_streak_alltime_u[idx] = streak_u[5]\n \n \n # Step 0) Handle session stuff\n if np.isnan(last_ts_u[0]) or timestamp - last_ts_u[0] > 3600000 // 2:\n # Reset the session\n session_u[0] = timestamp\n session_u[1] = 2 # correct\n session_u[2] = 3 # count\n session_u[3] = 2 # correct\n session_u[4] = 3 # count\n session_u[5] = 0 # CumAnsTimeSessionNoLeak\n \n if content_encounter[content_id] == 1:\n encountered[idx] = 2\n else:\n # 0 for pad\n encountered[idx] = 1\n \n if content_type_id==0:\n # Lag: Increase for each question we see\n task_container_num_questions_u[0] += 1\n answer_ratio[5] += 1\n \n # Answer Selection Ratio for questions only\n if user_answer==1:\n answer_ratio[0] += 1\n elif user_answer==2:\n answer_ratio[2] += 1\n\n # Content Counters\n correctness_u[1] += 1\n part_correctness_u__part[1] += 1\n session_u[2] += 1\n content_encounter[content_id] = 1\n \n if answered_correctly==0:\n last_incorrect_ts_u[0] = timestamp\n streak_u[0] = 0 # reset correct counter\n streak_u[3] += 1 # increment incorrect counter\n else:\n last_correct_ts_u[0] = timestamp\n streak_u[0] += 1 # increment correct counter\n streak_u[3] = 0 # reset incorrect counter\n\n # Correct Content Counters\n correctness_u[0] += 1\n part_correctness_u__part[0] += 1\n session_u[1] += 1\n \n # Diagnostic Features\n if task_container_id < 40:\n if content_id == 6911: diagnostic_u_1[0] = answered_correctly\n elif content_id == 7901: diagnostic_u_1[1] = answered_correctly\n elif content_id == 7219: diagnostic_u_2[0] = answered_correctly\n elif content_id == 6910: diagnostic_u_2[1] = answered_correctly\n elif content_id == 2066: diagnostic_u_2[2] = answered_correctly\n elif content_id == 6879: diagnostic_u_2[3] = answered_correctly\n elif content_id == 2065: diagnostic_u_2[4] = answered_correctly\n elif content_id == 1279: diagnostic_u_2[5] = answered_correctly\n elif content_id == 3365: diagnostic_u_3[0] = answered_correctly\n elif content_id == 4697: diagnostic_u_3[1] = answered_correctly\n elif content_id == 2594: diagnostic_u_3[2] = answered_correctly\n elif content_id == 6878: diagnostic_u_4[0] = answered_correctly\n elif content_id == 6912: diagnostic_u_4[1] = answered_correctly\n elif content_id == 4493: diagnostic_u_4[2] = answered_correctly\n elif content_id == 2596: diagnostic_u_4[3] = answered_correctly\n elif content_id == 7218: diagnostic_u_4[4] = answered_correctly\n elif content_id == 6909: diagnostic_u_5[0] = answered_correctly\n elif content_id == 7877: diagnostic_u_5[1] = answered_correctly\n elif content_id == 6880: diagnostic_u_5[2] = answered_correctly\n elif content_id == 7217: diagnostic_u_5[3] = answered_correctly\n elif content_id == 2595: diagnostic_u_5[4] = answered_correctly\n elif content_id == 2949: diagnostic_u_6[0] = answered_correctly\n elif content_id == 2948: diagnostic_u_6[1] = answered_correctly\n elif content_id == 4121: diagnostic_u_6[2] = answered_correctly\n elif content_id == 6881: diagnostic_u_6[3] = answered_correctly\n elif content_id == 6174: diagnostic_u_6[4] = answered_correctly\n elif content_id == 7220: diagnostic_u_6[5] = answered_correctly\n elif content_id == 3364: diagnostic_u_6[6] = answered_correctly\n\n df_train['lag_ts_u_recency'] = np.log1p(lag_ts_u_recency.clip(0,86400000) / 1000)\n df_train['last_ts_u_recency1'] = np.log1p(last_ts_u_recency1.clip(650,1209600000))\n df_train['last_ts_u_recency2'] = np.log1p(last_ts_u_recency2.clip(650,1209600000))\n df_train['last_ts_u_recency3'] = np.log1p(last_ts_u_recency3.clip(650,1209600000))\n df_train['last_correct_ts_u_recency'] = np.log1p(last_correct_ts_u_recency.clip(650,1209600000))\n df_train['last_incorrect_ts_u_recency'] = np.log1p(last_incorrect_ts_u_recency.clip(650,1209600000))\n df_train['correctness_u_recency'] = correctness_u_recency\n df_train['part_correctness_u_recency'] = part_correctness_u_recency\n df_train['session_content_num_u'] = np.log1p(session_content_num_u)\n df_train['session_duration_u'] = np.log1p(session_duration_u.clip(4100,18000000))\n df_train['session_correctness_u'] = session_correctness_u\n df_train['session_ans_duration_u'] = np.log1p(session_ans_duration_u.clip(4100,18000000))\n df_train['lifetime_ans_duration_u'] = np.log1p(lifetime_ans_duration_u.clip(4100,None))\n df_train['diagnostic_u_recency_1'] = diagnostic_u_recency[:, 0]\n df_train['diagnostic_u_recency_2'] = diagnostic_u_recency[:, 1]\n df_train['diagnostic_u_recency_3'] = diagnostic_u_recency[:, 2]\n df_train['diagnostic_u_recency_4'] = diagnostic_u_recency[:, 3]\n df_train['diagnostic_u_recency_5'] = diagnostic_u_recency[:, 4]\n df_train['diagnostic_u_recency_6'] = diagnostic_u_recency[:, 5]\n df_train['answer_ratio1'] = answer_ratio1\n df_train['answer_ratio2'] = answer_ratio2\n df_train['correct_streak_u'] = np.log1p(correct_streak_u)\n df_train['incorrect_streak_u'] = np.log1p(incorrect_streak_u)\n df_train['correct_streak_alltime_u'] = np.log1p(correct_streak_alltime_u)\n df_train['incorrect_streak_alltime_u'] = np.log1p(incorrect_streak_alltime_u)\n df_train['encountered'] = encountered\n \n df_train.lag_ts_u_recency = ((df_train.lag_ts_u_recency - lag_ts_u_recency_mean) / lag_ts_u_recency_std).astype(np.float32) \n df_train.last_ts_u_recency1 = ((df_train.last_ts_u_recency1 - last_ts_u_recency1_mean) / last_ts_u_recency1_std).astype(np.float32) \n df_train.last_ts_u_recency2 = ((df_train.last_ts_u_recency2 - last_ts_u_recency2_mean) / last_ts_u_recency2_std).astype(np.float32) \n df_train.last_ts_u_recency3 = ((df_train.last_ts_u_recency3 - last_ts_u_recency3_mean) / last_ts_u_recency3_std).astype(np.float32) \n df_train.last_incorrect_ts_u_recency = ((df_train.last_incorrect_ts_u_recency - last_incorrect_ts_u_recency_mean) / last_incorrect_ts_u_recency_std).astype(np.float32)\n df_train.last_correct_ts_u_recency = ((df_train.last_correct_ts_u_recency - last_correct_ts_u_recency_mean ) / last_correct_ts_u_recency_std).astype(np.float32) \n df_train.correctness_u_recency = ((df_train.correctness_u_recency - correctness_u_recency_mean ) / correctness_u_recency_std).astype(np.float32) \n df_train.part_correctness_u_recency = ((df_train.part_correctness_u_recency - part_correctness_u_recency_mean ) / part_correctness_u_recency_std).astype(np.float32) \n df_train.session_content_num_u = ((df_train.session_content_num_u - session_content_num_u_mean ) / session_content_num_u_std).astype(np.float32) \n df_train.session_duration_u = ((df_train.session_duration_u - session_duration_u_mean ) / session_duration_u_std).astype(np.float32) \n df_train.session_correctness_u = ((df_train.session_correctness_u - session_correctness_u_mean ) / session_correctness_u_std).astype(np.float32) \n df_train.session_ans_duration_u = ((df_train.session_ans_duration_u - session_ans_duration_u_mean ) / session_ans_duration_u_std).astype(np.float32) \n df_train.lifetime_ans_duration_u= ((df_train.lifetime_ans_duration_u - lifetime_ans_duration_u_mean) / lifetime_ans_duration_u_std).astype(np.float32) \n df_train.diagnostic_u_recency_1 = ((df_train.diagnostic_u_recency_1 - diagnostic_u_recency_1_mean ) / diagnostic_u_recency_1_std).astype(np.float32) \n df_train.diagnostic_u_recency_2 = ((df_train.diagnostic_u_recency_2 - diagnostic_u_recency_2_mean ) / diagnostic_u_recency_2_std).astype(np.float32) \n df_train.diagnostic_u_recency_3 = ((df_train.diagnostic_u_recency_3 - diagnostic_u_recency_3_mean ) / diagnostic_u_recency_3_std).astype(np.float32) \n df_train.diagnostic_u_recency_4 = ((df_train.diagnostic_u_recency_4 - diagnostic_u_recency_4_mean ) / diagnostic_u_recency_4_std).astype(np.float32) \n df_train.diagnostic_u_recency_5 = ((df_train.diagnostic_u_recency_5 - diagnostic_u_recency_5_mean ) / diagnostic_u_recency_5_std).astype(np.float32) \n df_train.diagnostic_u_recency_6 = ((df_train.diagnostic_u_recency_6 - diagnostic_u_recency_6_mean ) / diagnostic_u_recency_6_std).astype(np.float32) \n df_train.answer_ratio1 = ((df_train.answer_ratio1 - answer_ratio1_mean ) / answer_ratio1_std).astype(np.float32) \n df_train.answer_ratio2 = ((df_train.answer_ratio2 - answer_ratio2_mean ) / answer_ratio2_std).astype(np.float32) \n df_train.correct_streak_u = ((df_train.correct_streak_u - correct_streak_u_mean ) / correct_streak_u_std).astype(np.float32) \n df_train.incorrect_streak_u = ((df_train.incorrect_streak_u - incorrect_streak_u_mean ) / incorrect_streak_u_std).astype(np.float32) \n df_train.correct_streak_alltime_u = ((df_train.correct_streak_alltime_u - correct_streak_alltime_u_mean ) / correct_streak_alltime_u_std).astype(np.float32) \n df_train.incorrect_streak_alltime_u = ((df_train.incorrect_streak_alltime_u - incorrect_streak_alltime_u_mean ) / incorrect_streak_alltime_u_std).astype(np.float32) \n\n df_train.loc[df_train.lag_ts_u_recency.isna(), 'lag_ts_u_recency'] = 0\n df_train.loc[df_train.last_ts_u_recency1.isna(), 'last_ts_u_recency1'] = 0\n df_train.loc[df_train.last_ts_u_recency2.isna(), 'last_ts_u_recency2'] = 0\n df_train.loc[df_train.last_ts_u_recency3.isna(), 'last_ts_u_recency3'] = 0\n df_train.loc[df_train.last_incorrect_ts_u_recency.isna(), 'last_incorrect_ts_u_recency'] = 0\n df_train.loc[df_train.last_correct_ts_u_recency.isna(), 'last_correct_ts_u_recency'] = 0\n df_train.loc[df_train.correctness_u_recency.isna(), 'correctness_u_recency'] = 0.653417715747257 # mean answered correctly\n df_train.loc[df_train.part_correctness_u_recency.isna(), 'part_correctness_u_recency'] = 0.653417715747257 # mean answered correctly across dset \n df_train.loc[df_train.session_content_num_u.isna(), 'session_content_num_u'] = 0\n df_train.loc[df_train.session_duration_u.isna(), 'session_duration_u'] = 0\n df_train.loc[df_train.session_correctness_u.isna(), 'session_correctness_u'] = 0.653417715747257\n df_train.loc[df_train.session_ans_duration_u.isna(), 'session_ans_duration_u'] = 0\n df_train.loc[df_train.lifetime_ans_duration_u.isna(), 'lifetime_ans_duration_u'] = 0\n df_train.loc[df_train.diagnostic_u_recency_1.isna(), 'diagnostic_u_recency_1'] = 0\n df_train.loc[df_train.diagnostic_u_recency_2.isna(), 'diagnostic_u_recency_2'] = 0\n df_train.loc[df_train.diagnostic_u_recency_3.isna(), 'diagnostic_u_recency_3'] = 0\n df_train.loc[df_train.diagnostic_u_recency_4.isna(), 'diagnostic_u_recency_4'] = 0\n df_train.loc[df_train.diagnostic_u_recency_5.isna(), 'diagnostic_u_recency_5'] = 0\n df_train.loc[df_train.diagnostic_u_recency_6.isna(), 'diagnostic_u_recency_6'] = 0\n df_train.loc[df_train.answer_ratio1.isna(), 'answer_ratio1'] = 0\n df_train.loc[df_train.answer_ratio2.isna(), 'answer_ratio2'] = 0\n df_train.loc[df_train.correct_streak_u.isna(), 'correct_streak_u'] = 0\n df_train.loc[df_train.incorrect_streak_u.isna(), 'incorrect_streak_u'] = 0\n df_train.loc[df_train.correct_streak_alltime_u.isna(), 'correct_streak_alltime_u'] = 0\n df_train.loc[df_train.incorrect_streak_alltime_u.isna(), 'incorrect_streak_alltime_u'] = 0\n \n df_train['prior_question_elapsed_time_cont'] = df_train.prior_question_elapsed_time\n df_train.prior_question_elapsed_time //= 1000 # for embedding 0-300, looks already capped\n df_train.loc[df_train.prior_question_elapsed_time.isna(), 'prior_question_elapsed_time'] = -2\n df_train.prior_question_elapsed_time += 2\n \n # Now, keep just the lasrt 128\n df_train = df_train.groupby('user_id', sort=False).tail(128)\n \n # Create a default dictionary for user dataframe cache\n user_dfs = defaultdict(newUserDF)\n \n # Save it out\n for idx, (_, user_id, task_container_id, content_id, part_id, prior_question_elapsed_time, prior_question_had_explanation, incorrect_rank, content_type_id, bundle_id, answer_ratio1, answer_ratio2, correct_streak_u, incorrect_streak_u, correct_streak_alltime_u, incorrect_streak_alltime_u, session_content_num_u, session_duration_u, session_ans_duration_u, lifetime_ans_duration_u, lag_ts_u_recency, last_ts_u_recency1, last_ts_u_recency2, last_ts_u_recency3, last_correct_ts_u_recency, last_incorrect_ts_u_recency, correctness_u_recency, part_correctness_u_recency, session_correctness_u, encountered, diagnostic_u_recency_1, diagnostic_u_recency_2, diagnostic_u_recency_3, diagnostic_u_recency_4, diagnostic_u_recency_5, diagnostic_u_recency_6, answered_correctly) \\\n in enumerate(tqdm(df_train[dataset_cols[COL_OFFSET_CUT:]].itertuples())):\n state = user_dfs[user_id]\n \n state.lensize[0] += 1\n state.user_id.append(user_id)\n state.task_container_id.append(task_container_id)\n state.content_id.append(content_id)\n state.part_id.append(part_id)\n state.prior_question_elapsed_time.append(prior_question_elapsed_time)\n state.prior_question_had_explanation.append(prior_question_had_explanation)\n state.incorrect_rank.append(incorrect_rank) # decoder feature that needs help..........\n state.content_type_id.append(content_type_id)\n state.bundle_id.append(bundle_id)\n\n # only these of the 4 have signal (0-index)\n state.answer_ratio1.append(answer_ratio1)\n state.answer_ratio2.append(answer_ratio2)\n state.correct_streak_u.append(correct_streak_u)\n state.incorrect_streak_u.append(incorrect_streak_u)\n state.correct_streak_alltime_u.append(correct_streak_alltime_u)\n state.incorrect_streak_alltime_u.append(incorrect_streak_alltime_u)\n\n # our session information + a lifetime field (elapsed time sum on platform—probably should be capped)\n state.session_content_num_u.append(session_content_num_u)\n state.session_duration_u.append(session_duration_u)\n state.session_ans_duration_u.append(session_ans_duration_u)\n state.lifetime_ans_duration_u.append(lifetime_ans_duration_u)\n\n # First block of continuous - timestamp intervals\n state.lag_ts_u_recency.append(lag_ts_u_recency)\n state.last_ts_u_recency1.append(last_ts_u_recency1)\n state.last_ts_u_recency2.append(last_ts_u_recency2)\n state.last_ts_u_recency3.append(last_ts_u_recency3)\n state.last_correct_ts_u_recency.append(last_correct_ts_u_recency)\n state.last_incorrect_ts_u_recency.append(last_incorrect_ts_u_recency)\n\n # Second block of continuous - how often user is right on global/part/session basis\n state.correctness_u_recency.append(correctness_u_recency)\n state.part_correctness_u_recency.append(part_correctness_u_recency)\n state.session_correctness_u.append(session_correctness_u)\n\n # Have we been served this content before?\n state.encountered.append(encountered)\n\n # Average score of most frequent asked questions\n state.diagnostic_u_recency_1.append(diagnostic_u_recency_1)\n state.diagnostic_u_recency_2.append(diagnostic_u_recency_2)\n state.diagnostic_u_recency_3.append(diagnostic_u_recency_3)\n state.diagnostic_u_recency_4.append(diagnostic_u_recency_4)\n state.diagnostic_u_recency_5.append(diagnostic_u_recency_5)\n state.diagnostic_u_recency_6.append(diagnostic_u_recency_6)\n\n state.answered_correctly.append(answered_correctly)\n \n joblib.dump(user_features, './user_features_full.jlib')\n joblib.dump(user_dfs, './user_dfs_full.jlib')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"gc.collect()\nprepare_sub_data(4)","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# Dataset"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-12T09:45:31.865771Z","iopub.status.busy":"2020-11-12T09:45:31.865087Z","iopub.status.idle":"2020-11-12T09:45:31.867231Z","shell.execute_reply":"2020-11-12T09:45:31.867729Z"},"hidden":true,"papermill":{"duration":0.026841,"end_time":"2020-11-12T09:45:31.867849","exception":false,"start_time":"2020-11-12T09:45:31.841008","status":"completed"},"scrolled":true,"tags":[],"trusted":false},"cell_type":"code","source":"class Riiid(torch.utils.data.Dataset):\n def __init__(self, npdata, N, user_id_indices, user_id_sampling, mask_user_ids, augment=True):\n # df_train[dataset_cols[3:]].values\n self.npdata = npdata\n self.N = N\n self.user_id_indices = user_id_indices\n self.user_id_sampling = user_id_sampling\n self.CONTENT_MASK_TOKEN = 13941+2 - 1\n self.mask_user_ids = mask_user_ids\n self.augment = augment\n \n def __len__(self):\n return self.N\n \n def __getitem__(self, idx=None):\n # Take a random sequence for every id. It may lead to repeated sequences with\n # low probability. For example, if you have id 115 repeated 7 times, you will\n # take 7 random sequences for the user 115.\n cont_feats = [[0,0, 0,0, 0,0, 0,0, 0]]\n sess_feats = [[0,0,0,0]]\n diag_feats = [[0,0,0,0,0,0]]\n strk_feats = [[0,0,0,0,0,0]]\n \n # With replacement, so we ignore idx\n random_user_id = self.user_id_sampling[\n np.random.randint(0, self.user_id_sampling.shape[0])\n ]\n \n # NOTE:\n # These uesr ids are expected to already be mapped\n row_start, row_end = self.user_id_indices[random_user_id]\n row_len = row_end - row_start\n rows = self.npdata[row_start:row_end]\n \n if row_len <= WINDOW_SIZE:\n pad_len = WINDOW_SIZE - row_len\n padding = [PAD] * pad_len\n neg_pad = [-2] * pad_len\n\n # Augment indices order within Task Container ID, since it literally doesn't matter\n indices = np.arange(row_len, dtype=np.uint16)\n if self.augment:\n cuts = np.concatenate((\n np.nonzero(np.concatenate([[1],np.diff(rows[:, COL_TASK_CONTAINER_ID])]))[0],\n [row_len]\n ))\n indices = [ indices[cuts[i]:cuts[i+1]] for i in range(len(cuts) - 1) ]\n for block in indices: np.random.shuffle(block)\n if len(indices) > 6:\n ii = np.random.randint(0,len(indices)-1,(2,))\n indices[ii[0]], indices[ii[0]+1] = indices[ii[0]+1], indices[ii[0]] \n indices[ii[1]], indices[ii[1]+1] = indices[ii[1]+1], indices[ii[1]] \n indices = np.concatenate(indices)\n \n if pad_len==0:\n continuous = rows[indices, COL_LAG_TS_RECENCY:COL_SESSION_CORRECTNESS_U+1]\n session = rows[indices, COL_SESSION_CONTENT_NUM_U:COL_LIFETIME_ANS_DURATION_U+1]\n diagnostic = rows[indices, COL_DIAGNOSTIC_RECENCY_1:COL_DIAGNOSTIC_RECENCY_6+1]\n streak = rows[indices, COL_ANSWER_RATIO1:COL_INCORRECT_STREAK_ALLTIME_U+1]\n\n else:\n continuous = np.vstack((\n rows[indices, COL_LAG_TS_RECENCY:COL_SESSION_CORRECTNESS_U+1],\n cont_feats * pad_len\n ))\n \n session = np.vstack((\n rows[indices, COL_SESSION_CONTENT_NUM_U:COL_LIFETIME_ANS_DURATION_U+1],\n sess_feats * pad_len\n ))\n \n diagnostic = np.vstack((\n rows[indices, COL_DIAGNOSTIC_RECENCY_1:COL_DIAGNOSTIC_RECENCY_6+1],\n diag_feats * pad_len\n ))\n \n streak = np.vstack((\n rows[indices, COL_ANSWER_RATIO1:COL_INCORRECT_STREAK_ALLTIME_U+1],\n strk_feats * pad_len\n ))\n \n if self.mask_user_ids:\n user_ids = np.zeros(WINDOW_SIZE, dtype=int)\n else:\n user_ids = np.concatenate((rows[indices, COL_USER_ID], padding)).astype(int)\n \n return (\n idx,\n user_ids,\n np.concatenate((rows[indices, COL_CONTENT_ID], padding)).astype(int),\n np.concatenate((rows[indices, COL_CONTENT_TYPE_ID], padding)).astype(int),\n np.concatenate((rows[indices, COL_PART_ID], padding)).astype(int),\n np.concatenate((rows[indices, COL_ENCOUNTERED], padding)).astype(int),\n\n np.concatenate((\n rows[indices, COL_PRIOR_QUESTION_ELAPSED_TIME],\n padding\n )).astype(int),\n \n np.concatenate((\n rows[indices, COL_PRIOR_QUESTION_EXPLANATION],\n padding\n )).astype(int),\n \n # correctness_id\n np.concatenate((\n [1], # starter\n #2 + rows[:-1, COL_ANSWERED_CORRECTLY],\n rows[indices[:-1], COL_INCORRECT_RANK],\n padding\n )).astype(int),\n \n # Bundle\n np.concatenate((rows[indices, COL_BUNDLE_ID], padding)).astype(int),\n streak.astype(np.float32),\n \n # Continuous Features\n continuous.astype(np.float32),\n session.astype(np.float32),\n diagnostic.astype(np.float32),\n \n # padding mask:\n np.concatenate(([False]*(row_end - row_start), [True]*pad_len)).astype(np.uint8),\n \n # lecture mask\n np.concatenate((1==rows[indices, COL_CONTENT_TYPE_ID], [False]*pad_len)).astype(np.uint8),\n \n # labels:\n np.concatenate((rows[indices, COL_ANSWERED_CORRECTLY], padding)).astype(np.float32),\n \n # task container id:\n np.concatenate((rows[indices, COL_TASK_CONTAINER_ID], neg_pad)).astype(np.int),\n )\n \n # Return random WINDOW_SIZE crop from the range of this users inputs;\n # We also want later values to be samples more often that earlier samples\n# start_offset = max(\n# # No preference to further items, just let ranadomness do its thing with our data:\n# np.random.randint(0, row_len-WINDOW_SIZE),\n# #np.random.randint(0, row_len-WINDOW_SIZE),\n# #np.random.randint(0, row_len-WINDOW_SIZE),\n# )\n start_offset = np.random.randint(0, row_len-WINDOW_SIZE)\n\n indices = np.arange(start_offset, start_offset+WINDOW_SIZE, dtype=np.uint16)\n if self.augment:\n cuts = np.concatenate((\n np.nonzero(np.concatenate([[1],np.diff(rows[start_offset:start_offset+WINDOW_SIZE, COL_TASK_CONTAINER_ID])]))[0],\n [WINDOW_SIZE]\n ))\n indices = [ indices[cuts[i]:cuts[i+1]] for i in range(len(cuts) - 1) ]\n for block in indices: np.random.shuffle(block)\n if len(indices) > 6:\n ii = np.random.randint(0,len(indices)-1,(2,))\n indices[ii[0]], indices[ii[0]+1] = indices[ii[0]+1], indices[ii[0]] \n indices[ii[1]], indices[ii[1]+1] = indices[ii[1]+1], indices[ii[1]] \n indices = np.concatenate(indices)\n \n if self.mask_user_ids:\n user_ids = np.zeros(WINDOW_SIZE, dtype=int)\n else:\n user_ids = rows[indices, COL_USER_ID].astype(int)\n \n return (\n idx,\n user_ids,\n rows[indices, COL_CONTENT_ID].astype(int),\n rows[indices, COL_CONTENT_TYPE_ID].astype(int),\n rows[indices, COL_PART_ID].astype(int),\n rows[indices, COL_ENCOUNTERED].astype(int),\n \n np.concatenate((\n #[1], # NEW Shifted decoder stuff:\n #rows[start_offset:start_offset+WINDOW_SIZE-1, COL_PRIOR_QUESTION_ELAPSED_TIME],\n rows[indices, COL_PRIOR_QUESTION_ELAPSED_TIME],\n )).astype(int),\n\n np.concatenate((\n #[1], # NEW Shifted decoder stuff:\n #rows[start_offset:start_offset+WINDOW_SIZE-1, COL_PRIOR_QUESTION_EXPLANATION],\n rows[indices, COL_PRIOR_QUESTION_EXPLANATION],\n )).astype(int),\n\n # correctness_id\n np.concatenate((\n [1], # starter\n rows[indices[:-1], COL_INCORRECT_RANK],\n #2+rows[start_offset:start_offset+WINDOW_SIZE-1, COL_ANSWERED_CORRECTLY],\n )).astype(int),\n \n # Bundle\n rows[indices, COL_BUNDLE_ID].astype(int),\n \n # STREAK:\n rows[indices, COL_ANSWER_RATIO1:COL_INCORRECT_STREAK_ALLTIME_U+1].astype(np.float32),\n \n # Continuous Features:\n rows[indices, COL_LAG_TS_RECENCY:COL_SESSION_CORRECTNESS_U+1].astype(np.float32),\n rows[indices, COL_SESSION_CONTENT_NUM_U:COL_LIFETIME_ANS_DURATION_U+1].astype(np.float32),\n rows[indices, COL_DIAGNOSTIC_RECENCY_1:COL_DIAGNOSTIC_RECENCY_6+1].astype(np.float32),\n \n # padding mask:\n np.array([False]*WINDOW_SIZE).astype(np.uint8),\n \n # lecture mask\n (1 == rows[indices, COL_CONTENT_TYPE_ID]).astype(np.uint8),\n\n # labels:\n rows[indices, COL_ANSWERED_CORRECTLY].astype(np.float32),\n \n # task container id:\n rows[indices, COL_TASK_CONTAINER_ID].astype(np.int),\n )","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def collate_fn(batch):\n (\n _, user_id, content_id, content_type_id, part_id, encountered_id, prior_question_elapsed_time,\n prior_question_had_explanation, correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, lecture_mask, labels, task_container_id\n ) = zip(*batch)\n \n user_id = torch.LongTensor(user_id)\n content_id = torch.LongTensor(content_id)\n \n content_type_id = torch.LongTensor(content_type_id)\n part_id = torch.LongTensor(part_id)\n encountered_id = torch.LongTensor(encountered_id)\n prior_question_elapsed_time = torch.LongTensor(prior_question_elapsed_time)\n prior_question_had_explanation = torch.LongTensor(prior_question_had_explanation)\n correctness_id = torch.LongTensor(correctness_id)\n bundle_id = torch.LongTensor(bundle_id)\n cont_streaks = torch.FloatTensor(cont_streaks)\n continuous = torch.FloatTensor(continuous)\n diagnostic = torch.FloatTensor(diagnostic)\n session = torch.FloatTensor(session)\n mask = torch.BoolTensor(mask)\n lecture_mask = torch.BoolTensor(lecture_mask)\n \n labels = torch.FloatTensor(labels)\n task_container_id = torch.LongTensor(task_container_id)\n \n # remember the order\n return (\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, lecture_mask, labels, task_container_id\n )\n\n","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# Model"},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def count_parameters(model):\n table = PrettyTable([\"Modules\", \"Parameters\"])\n total_params = 0\n for name, parameter in model.named_parameters():\n if not parameter.requires_grad: continue\n param = parameter.numel()\n table.add_row([name, param])\n total_params+=param\n print(table)\n print(f\"Total Trainable Params: {total_params}\")\n return total_params","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class ItemResponseTheory(nn.Module):\n '''\n chance = probability of random guess = 0.25\n a = concept ability\n B = exercise difficulty (can be the mean)\n '''\n def __init__(self, chance=0.25):\n super(ItemResponseTheory, self).__init__()\n self.chance = chance\n \n def forward(self, input, no_responses):\n # p(correct|a,B) = c + (1-c) / (1 + np.exp(B-a)), where:\n \n alpha = input[:,:,0]\n Beta = input[:,:,1]\n \n no_responses = no_responses.permute(1,0)\n return 1/no_responses + (1-1/no_responses) / (1 + torch.exp(Beta-alpha))\n #return self.chance + (1-chance) / (1 + torch.exp(Beta-alpha))","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class BartSinusoidalPositionalEmbedding(nn.Embedding):\n \"\"\"This module produces sinusoidal positional embeddings of any length.\"\"\"\n\n def __init__(self, num_positions: int, embedding_dim: int, padding_idx=None):\n super().__init__(num_positions+1, embedding_dim)\n self.weight = self._init_weight(self.weight)\n self.positions = None\n \n @staticmethod\n def _init_weight(out: nn.Parameter):\n \"\"\"\n Identical to the XLM create_sinusoidal_embeddings except features are not interleaved. The cos features are in\n the 2nd half of the vector. [dim // 2:]\n \"\"\"\n n_pos, dim = out.shape\n position_enc = np.array(\n [[pos / np.power(10000, 2 * (j // 2) / dim) for j in range(dim)] for pos in range(n_pos)]\n )\n out.requires_grad = False # set early to avoid an error in pytorch-1.8+\n sentinel = dim // 2 if dim % 2 == 0 else (dim // 2) + 1\n out[:, :sentinel] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))\n out[:, sentinel:] = torch.FloatTensor(np.cos(position_enc[:, 1::2]))\n \n # Scale this bad boy\n out *= 0.1 # 10% of whatever it is\n \n out.detach_()\n return out\n\n @torch.no_grad()\n def forward(self, start:int=0):\n if self.positions is None:\n self.positions = torch.arange(\n 0, WINDOW_SIZE+1,\n dtype=torch.long,\n device=self.weight.device\n )\n \n return super().forward(self.positions)[start:start+WINDOW_SIZE]","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class LambdaLayer(nn.Module):\n def __init__(self, lambd):\n super(LambdaLayer, self).__init__()\n self.lambd = lambd\n def forward(self, x):\n return self.lambd(x)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class TransposeBatchNorm(nn.Module):\n def __init__(self, num_features):\n super(TransposeBatchNorm, self).__init__()\n self.bn = nn.BatchNorm1d(num_features)\n def forward(self, x):\n return self.bn(x.transpose(1,2)).transpose(1,2)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def generate_mask(size, diagonal=1): \n return torch.triu(torch.ones(size, size)==1, diagonal=diagonal)\n\ndef tasks_mask(tasks, seq_length, diagonal=1):\n future_mask = generate_mask(seq_length, diagonal=diagonal).to(tasks.device)\n container_mask= torch.ones((seq_length, seq_length)).to(tasks.device)\n container_mask=(container_mask*tasks.reshape(1,-1))==(container_mask*tasks.reshape(-1,1))\n future_mask=future_mask+container_mask\n future_mask = future_mask.fill_diagonal_(False)\n return future_mask\n\ndef tasks_3d_mask(tasks, seq_length=WINDOW_SIZE, nhead=1, diagonal=1):\n #https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/206620\n mask_3d = [tasks_mask(t, seq_length, diagonal=diagonal) for t in tasks]\n mask_3d = torch.stack(mask_3d, dim=0)\n # Need BS*num_heads shape\n repeat_3d = [mask_3d for t in range(nhead)]\n repeat_3d = torch.cat(repeat_3d)\n return repeat_3d","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class ThinTransformerEncoderLayer(nn.Module):\n def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=\"relu\"):\n super(ThinTransformerEncoderLayer, self).__init__()\n # Implementation of Feedforward model\n self.linear1 = nn.Linear(d_model, dim_feedforward)\n self.dropout = nn.Dropout(dropout)\n self.linear2 = nn.Linear(dim_feedforward, d_model)\n\n self.norm1 = nn.LayerNorm(d_model)\n self.norm2 = nn.LayerNorm(d_model)\n self.dropout2 = nn.Dropout(dropout)\n\n\n def forward(self, src, src_mask = None, src_key_padding_mask = None):\n r\"\"\"Pass the input through the encoder layer.\n\n Args:\n src: the sequence to the encoder layer (required).\n src_mask: the mask for the src sequence (optional).\n src_key_padding_mask: the mask for the src keys per batch (optional).\n\n Shape:\n see the docs in Transformer class.\n \"\"\"\n src = self.norm1(src)\n src2 = self.linear2(self.dropout(F.relu(self.linear1(src))))\n src = src + self.dropout2(src2)\n src = self.norm2(src)\n \n # Mask Out:\n src[src_key_padding_mask.transpose(0,1)[:,:,None].expand(src.shape)] = 0\n return src","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"class SaintTransformerModel(nn.Module):\n def __init__(\n self,\n d_model:int=32,\n nhead:int=2,\n num_users:int=393657+1,\n num_layers:int=2,\n num_exercises:int=13941+2, # NOTE: 1+total number of QUESTIONS+LECTURES + Mask at the end\n num_answer_streaks:int=8+1,\n num_content_types:int=2,\n num_parts:int=7+1, # TOTAL\n num_bundles:int=9765+2,\n num_prior_question_elapsed_time:int=301+2,\n num_explanable:int=2+2,\n num_max_seq_len:int=WINDOW_SIZE, # TODO, figure this out, do we want to extend past?\n num_encountered:int=2+1,\n dropout:float=0.1,\n emb_dropout:float=0.2,\n initrange:float = 0.02\n ):\n super(SaintTransformerModel, self).__init__()\n\n self.d_model = d_model\n self.nhead = nhead\n self.emb_dropout = nn.Dropout(p=emb_dropout)\n \n# # These two have double dropout, due to their cardinality:\n# self.user_embeddings = nn.Sequential(\n# nn.Embedding(num_users, 16, padding_idx=0),\n# #nn.Linear(16, d_model, bias=False),\n# #nn.Dropout(p=emb_dropout)\n# )\n# self.answer_streak_embeddings = nn.Embedding(num_answer_streaks, d_model, padding_idx=0)\n self.exercise_embeddings = nn.Embedding(num_exercises, d_model, padding_idx=0)\n self.bundle_embeddings = nn.Embedding(num_bundles, d_model, padding_idx=0)\n self.content_type_embeddings = nn.Embedding(num_content_types, d_model, padding_idx=0)\n self.sin_pos_embedder = BartSinusoidalPositionalEmbedding(num_positions=WINDOW_SIZE, embedding_dim=d_model, padding_idx=0) \n self.part_embeddings = nn.Embedding(num_parts, d_model, padding_idx=0)\n self.prior_question_elapsed_time_embeddings = nn.Embedding(num_prior_question_elapsed_time, d_model, padding_idx=0) \n self.prior_question_had_explanation_embeddings = nn.Embedding(num_explanable, d_model, padding_idx=0)\n self.correctness_embeddings = nn.Embedding(4+2, d_model, padding_idx=0)\n self.encountered_embeddings = nn.Embedding(num_encountered, d_model, padding_idx=0)\n\n self.cont_mlp = nn.Sequential(\n # LN ? BN\n nn.Linear(25, d_model//2),\n nn.Dropout(p=emb_dropout),\n nn.ReLU(inplace=True),\n nn.Linear(d_model//2, d_model, bias=False),\n nn.ReLU(inplace=True),\n )\n \n self.transformer_encoder = nn.TransformerEncoder(\n nn.TransformerEncoderLayer(\n d_model, nhead, dim_feedforward=d_model,#2048,\n dropout=dropout, activation='relu'\n ),\n num_layers\n )\n \n self.transformer_history = nn.TransformerEncoder(\n nn.TransformerEncoderLayer(\n d_model, nhead*2, dim_feedforward=d_model,#2048,\n dropout=dropout, activation='relu'\n ),\n 1 # history only has 1 layer\n )\n \n self.transformer_decoder = nn.TransformerDecoder(\n nn.TransformerDecoderLayer(\n d_model, nhead, dim_feedforward=d_model,#2048,\n dropout=dropout, activation='relu'\n ),\n num_layers\n )\n \n self.decoder = nn.Linear(d_model,1)\n self.init_weights(initrange)\n\n def init_weights(self, initrange:float = 0.02) -> None:\n self.content_type_embeddings.weight.data.normal_(0, initrange)\n self.encountered_embeddings.weight.data.normal_(0, initrange)\n self.exercise_embeddings.weight.data.normal_(0, initrange)\n self.bundle_embeddings.weight.data.normal_(0, initrange)\n self.part_embeddings.weight.data.normal_(0, initrange)\n self.prior_question_elapsed_time_embeddings.weight.data.normal_(0, initrange)\n self.prior_question_had_explanation_embeddings.weight.data.normal_(0, initrange)\n self.correctness_embeddings.weight.data.normal_(0, initrange)\n #self.decoder.bias.data.zero_()\n self.decoder.weight.data.normal_(0, initrange)\n \n @torch.cuda.amp.autocast()\n def forward(\n self,\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, padding_mask, task_container_id\n ):\n batch_size = content_id.shape[0]\n seq_len = content_id.shape[-1]\n \n future_mask = torch.ones([seq_len, seq_len], device=task_container_id.device, dtype=torch.uint8)\n future_mask = future_mask.triu_(1).view(seq_len, seq_len).bool()\n history_mask = tasks_3d_mask(task_container_id, seq_length=WINDOW_SIZE, nhead=self.nhead*2)\n \n ##########\n embedded_src = self.emb_dropout(\n self.content_type_embeddings(content_type_id)\n + self.part_embeddings(part_id)\n + self.bundle_embeddings(bundle_id)\n + self.exercise_embeddings(content_id)\n + self.sin_pos_embedder(start=1)\n ).transpose(0, 1) # (S, N, E)\n \n output_src = self.emb_dropout(\n # Shifted by BUNDLE already\n self.prior_question_elapsed_time_embeddings(prior_question_elapsed_time)\n + self.prior_question_had_explanation_embeddings(prior_question_had_explanation)\n + self.encountered_embeddings(encountered_id)\n \n + self.cont_mlp(\n torch.cat([\n cont_streaks,\n diagnostic,\n session,\n continuous,\n ], dim=-1)\n )\n \n + self.sin_pos_embedder(start=0)\n ).transpose(0, 1) # (S, N, E)\n\n history_src = self.emb_dropout(\n self.correctness_embeddings(correctness_id)\n + self.sin_pos_embedder(start=1)\n ).transpose(0, 1)\n \n history = self.transformer_history(\n src=history_src,\n mask=history_mask,\n src_key_padding_mask=padding_mask,\n )\n with torch.no_grad():\n history[torch.isnan(history)] = 0\n\n memory = self.transformer_encoder(\n src=embedded_src,\n mask=future_mask,\n src_key_padding_mask=padding_mask,\n )\n\n output = self.transformer_decoder(\n # TODO: Try adding memory into here? so we have a full 'interaction' embedding\n tgt = output_src + history, #embedded_src\n memory = memory,\n tgt_mask = future_mask,\n memory_mask = future_mask,\n tgt_key_padding_mask = padding_mask,\n memory_key_padding_mask = padding_mask,\n )\n \n # TODO: Maybe we can concatenate instead of add...\n output = self.decoder(output).transpose(1, 0)\n return output","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class SiameseNetwork(nn.Module):\n def __init__(self, networks):\n super(SiameseNetwork, self).__init__()\n self.networks = nn.ModuleList(networks)\n self.power = nn.Parameter(torch.FloatTensor([1]))\n \n @torch.cuda.amp.autocast()\n def forward(\n self,\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, padding_mask, task_container_id\n ):\n futures = [\n torch.jit.fork(\n model,\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, padding_mask, task_container_id\n ) for model in self.networks\n ]\n outputs = [torch.jit.wait(fut) for fut in futures]\n return (torch.stack(outputs) ** self.power).mean(dim=0)","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# Gym"},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class LinearWarmupStepSchedule(torch.optim.lr_scheduler.LambdaLR):\n def __init__(self, optimizer, warmup_steps):\n self.warmup_steps = warmup_steps\n super(LinearWarmupStepSchedule, self).__init__(optimizer, self.lr_lambda, last_epoch=-1)\n\n def lr_lambda(self, step):\n if step < self.warmup_steps:\n return (step+1) / (max(1., self.warmup_steps))\n\n return 1","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n self.vals = []\n\n def update(self, val, n=1):\n self.vals.append(val)\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n \n def draw(self, epoch, score, prefix=''):\n plt.title(f'{prefix} E{epoch+1}, Loss: {self.avg:.5f}, AUC: {score:.5f}')\n plt.plot(self.vals, linewidth=0.2)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def draw(trn_loss, trn_auc, val_loss, val_auc):\n slope_trn, intercept_trn = scipy.stats.linregress(np.arange(len(trn_loss)), trn_loss)[:2]\n \n plt.title(f'E{epoch+1}. L: {np.mean(trn_loss):.5f}, {np.mean(val_loss):.5f}. AUC: {trn_auc:.5f}, {val_auc:.5f}')\n plt.plot(trn_loss, linewidth=0.2, label=f'Trn {slope_trn}')\n plt.plot(val_loss, linewidth=0.2, label=f'Val')\n plt.plot([0,len(trn_loss)], [intercept_trn, intercept_trn + slope_trn*len(trn_loss)], c='blue', linewidth=1)\n plt.legend()\n plt.show()\n \n print(f'- L: {np.mean(trn_loss):.4f}, {np.mean(val_loss):.4f}')\n print(f'- A: {trn_auc:.4f}, {val_auc:.4f}')\n \n return [np.mean(trn_loss), np.mean(val_loss), trn_auc, val_auc]","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"class RocAucMeter(object):\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.y_true = None\n self.y_pred = None\n self.all_preds = None # Not just the masked ones\n self.all_masks = None # ..\n self.score = 0\n\n def update(self, y_true, y_pred, mask):\n # NOTE: 1 = CORRECT!!!!!!\n y_true = y_true.detach().cpu().numpy()\n #y_true = y_true.detach().cpu().argmax(-1).numpy()\n #y_true = y_true.cpu().numpy().argmax(axis=1).clip(min=0, max=1).astype(int)\n \n #y_pred = torch.softmax(y_pred.detach().float(), dim=2).data.cpu().numpy()[:,:,1]\n #y_pred = torch.sigmoid(y_pred.detach()).data.cpu().numpy()[:,:,1] # 0 = correct xent incorrect actuakky\n \n y_pred = torch.sigmoid(y_pred.detach()).data.cpu().numpy() # 0 = correct bce\n #y_pred = torch.softmax(y_pred.detach(), dim=2).data.cpu().numpy()[:,:,1]\n \n # IRT\n #y_pred = y_pred.detach().data.cpu().numpy() # 0 = correct\n\n mask = mask.detach().cpu().numpy()\n \n if self.y_true is None:\n self.y_true = y_true[~mask]\n self.y_pred = y_pred[~mask]\n #self.all_masks = mask[None]\n #self.all_preds = y_pred[None]\n else:\n self.y_true = np.concatenate((self.y_true, y_true[~mask]))\n self.y_pred = np.concatenate((self.y_pred, y_pred[~mask]))\n \n # We can add these later if we need them...\n #print(self.all_preds.shape, y_pred[None].shape)\n #self.all_preds = np.vstack((self.all_preds, y_pred[None]))\n #self.all_masks = np.vstack((self.all_masks, mask[None]))\n\n @property\n def avg(self):\n return roc_auc_score(self.y_true.flatten(), self.y_pred.flatten())\n #return auc(self.y_true.flatten(), self.y_pred.flatten())","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def save(path, model, optimizer, best_loss, epoch, step_scheduler=None, scheduler=None, lean=False):\n model.eval()\n \n if lean:\n torch.save(model.state_dict(), path)\n return\n\n params = {\n 'model_state_dict': model.state_dict(),\n 'optimizer_state_dict': optimizer.state_dict(),\n 'best_loss': best_loss,\n 'epoch': epoch,\n }\n if scheduler is not None: params['scheduler_state_dict'] = scheduler.state_dict()\n if step_scheduler is not None: params['step_scheduler_state_dict'] = step_scheduler.state_dict()\n torch.save(params, path)\n\ndef load(path, model, optimizer, step_scheduler=None, scheduler=None):\n checkpoint = torch.load(path)\n \n if 'step_scheduler' in checkpoint:\n step_scheduler.load_state_dict(checkpoint['step_scheduler_state_dict'])\n scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n\n model.load_state_dict(checkpoint['model_state_dict'])\n optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n \n return (\n model, optimizer,\n checkpoint['best_loss'], checkpoint['epoch'],\n step_scheduler, scheduler\n )","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def train_fn(model, optimizer, step_scheduler, loss_fn, dataloader, device):\n model.train()\n final_loss = 0\n \n scaler = torch.cuda.amp.GradScaler()\n meter = RocAucMeter()\n losses = AverageMeter()\n \n pbar = tqdm(dataloader)\n for idx, batch in enumerate(pbar):\n (\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, lecture_mask, labels, task_container_id\n ) = batch\n\n user_id = user_id.to(device)\n content_id = content_id.to(device)\n part_id = part_id.to(device)\n encountered_id = encountered_id.to(device)\n content_type_id = content_type_id.to(device)\n prior_question_elapsed_time = prior_question_elapsed_time.to(device)\n prior_question_had_explanation = prior_question_had_explanation.to(device)\n correctness_id = correctness_id.to(device)\n bundle_id = bundle_id.to(device)\n cont_streaks = cont_streaks.to(device)\n continuous = continuous.to(device)\n session = session.to(device)\n diagnostic = diagnostic.to(device)\n mask = mask.to(device)\n lecture_mask = lecture_mask.to(device)\n labels = labels.to(device)\n task_container_id = task_container_id.to(device)\n \n optimizer.zero_grad()\n with torch.cuda.amp.autocast():\n output = model(\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, task_container_id\n )\n with torch.no_grad(): masked = mask | lecture_mask\n loss = loss_fn(output[~masked].view(-1), labels[~masked].view(-1))\n #\n \n scaler.scale(loss).backward() # loss.backward()\n scaler.step(optimizer) # optimizer.step()\n scaler.update()\n if step_scheduler is not None: step_scheduler.step()\n \n with torch.no_grad():\n losses.update(loss.item())\n meter.update(labels, output, mask|lecture_mask)\n pbar.set_description(f'{loss.item():.4f}')\n \n return losses, meter","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"def valid_fn(model, scheduler, loss_fn, dataloader, device):\n model.eval()\n \n meter = RocAucMeter()\n losses = AverageMeter()\n valid_preds = []\n \n with torch.no_grad():\n pbar = tqdm(dataloader)\n for idx, batch in enumerate(pbar):\n (\n user_id, content_id, content_type_id, part_id, encountered_id, \n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, lecture_mask, labels, task_container_id\n ) = batch\n\n user_id = user_id.to(device)\n content_id = content_id.to(device)\n content_type_id = content_type_id.to(device)\n part_id = part_id.to(device)\n encountered_id = encountered_id.to(device)\n prior_question_elapsed_time = prior_question_elapsed_time.to(device)\n prior_question_had_explanation = prior_question_had_explanation.to(device)\n correctness_id = correctness_id.to(device)\n bundle_id = bundle_id.to(device)\n cont_streaks = cont_streaks.to(device)\n continuous = continuous.to(device)\n session = session.to(device)\n diagnostic = diagnostic.to(device)\n mask = mask.to(device)\n lecture_mask = lecture_mask.to(device)\n labels = labels.to(device)\n task_container_id = task_container_id.to(device)\n \n with torch.cuda.amp.autocast():\n output = model(\n user_id, content_id, content_type_id, part_id, encountered_id,\n prior_question_elapsed_time, prior_question_had_explanation,\n correctness_id, bundle_id, cont_streaks,\n continuous, session, diagnostic, mask, task_container_id\n )\n with torch.no_grad(): masked = mask | lecture_mask\n loss = loss_fn(output[~masked].view(-1), labels[~masked].view(-1))\n\n losses.update(loss.item())\n meter.update(labels, output, mask|lecture_mask)\n pbar.set_description(f'{loss.item():.4f}')\n\n if scheduler is not None: scheduler.step(losses.avg)\n \n return losses, meter","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"# def inference_fn(model, dataloader, device):\n# model.eval()\n# preds = []\n \n# for data in dataloader:\n# inputs = data['x'].to(device)\n\n# with torch.no_grad():\n# outputs = model(inputs)\n \n# preds.append(outputs.sigmoid().detach().cpu().numpy())\n \n# preds = np.concatenate(preds)\n# return preds","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# Build out and cache data"},{"metadata":{"hidden":true,"scrolled":true,"trusted":false},"cell_type":"code","source":"# Prepare the dataset\nPREFIX = 'cvfold4'\n!mkdir -p {PREFIX}\n\n# TODO: Red the order on all of these to ensure correct\nif PRELOAD:\n (\n df_train, df_valid, Ntrn, Nval,\n train_user_id_sampling, valid_user_id_sampling,\n train_user_id_indices, valid_user_id_indices,\n ) = joblib.load('./preload.jlib')\n \n (\n embedded_user_ids_map, mask_user_ids, part_ids_map, lecture_ids_map, bundle_id_map,\n question_incorrect_ranks,\n answer_ratio1_mean, answer_ratio1_std,\n answer_ratio2_mean, answer_ratio2_std,\n correct_streak_u_mean, correct_streak_u_std,\n incorrect_streak_u_mean, incorrect_streak_u_std,\n correct_streak_alltime_u_mean, correct_streak_alltime_u_std,\n incorrect_streak_alltime_u_mean, incorrect_streak_alltime_u_std,\n diagnostic_u_recency_1_mean, diagnostic_u_recency_1_std,\n diagnostic_u_recency_2_mean, diagnostic_u_recency_2_std,\n diagnostic_u_recency_3_mean, diagnostic_u_recency_3_std,\n diagnostic_u_recency_4_mean, diagnostic_u_recency_4_std,\n diagnostic_u_recency_5_mean, diagnostic_u_recency_5_std,\n diagnostic_u_recency_6_mean, diagnostic_u_recency_6_std,\n session_content_num_u_mean, session_content_num_u_std,\n session_duration_u_mean, session_duration_u_std,\n session_correctness_u_mean, session_correctness_u_std,\n session_ans_duration_u_mean, session_ans_duration_u_std,\n lifetime_ans_duration_u_mean, lifetime_ans_duration_u_std,\n correctness_u_recency_mean, correctness_u_recency_std,\n part_correctness_u_recency_mean, part_correctness_u_recency_std,\n lag_ts_u_recency_mean, lag_ts_u_recency_std,\n last_ts_u_recency1_mean, last_ts_u_recency1_std,\n last_ts_u_recency2_mean, last_ts_u_recency2_std,\n last_ts_u_recency3_mean, last_ts_u_recency3_std,\n last_incorrect_ts_u_recency_mean, last_incorrect_ts_u_recency_std,\n last_correct_ts_u_recency_mean, last_correct_ts_u_recency_std,\n ) = joblib.load('./mappings.jlib')\n \nelse:\n (\n df_train, df_valid, Ntrn, Nval,\n embedded_user_ids_map, mask_user_ids, part_ids_map, lecture_ids_map, bundle_id_map,\n train_user_id_sampling, valid_user_id_sampling,\n train_user_id_indices, valid_user_id_indices,\n \n question_incorrect_ranks,\n answer_ratio1_mean, answer_ratio1_std,\n answer_ratio2_mean, answer_ratio2_std,\n correct_streak_u_mean, correct_streak_u_std,\n incorrect_streak_u_mean, incorrect_streak_u_std,\n correct_streak_alltime_u_mean, correct_streak_alltime_u_std,\n incorrect_streak_alltime_u_mean, incorrect_streak_alltime_u_std,\n diagnostic_u_recency_1_mean, diagnostic_u_recency_1_std,\n diagnostic_u_recency_2_mean, diagnostic_u_recency_2_std,\n diagnostic_u_recency_3_mean, diagnostic_u_recency_3_std,\n diagnostic_u_recency_4_mean, diagnostic_u_recency_4_std,\n diagnostic_u_recency_5_mean, diagnostic_u_recency_5_std,\n diagnostic_u_recency_6_mean, diagnostic_u_recency_6_std,\n session_content_num_u_mean, session_content_num_u_std,\n session_duration_u_mean, session_duration_u_std,\n session_correctness_u_mean, session_correctness_u_std,\n session_ans_duration_u_mean, session_ans_duration_u_std,\n lifetime_ans_duration_u_mean, lifetime_ans_duration_u_std,\n correctness_u_recency_mean, correctness_u_recency_std,\n part_correctness_u_recency_mean, part_correctness_u_recency_std,\n lag_ts_u_recency_mean, lag_ts_u_recency_std,\n last_ts_u_recency1_mean, last_ts_u_recency1_std,\n last_ts_u_recency2_mean, last_ts_u_recency2_std,\n last_ts_u_recency3_mean, last_ts_u_recency3_std,\n last_incorrect_ts_u_recency_mean, last_incorrect_ts_u_recency_std,\n last_correct_ts_u_recency_mean, last_correct_ts_u_recency_std,\n ) = prepare_training_data(fold_number=4)\n \n \n joblib.dump((\n df_train, df_valid, Ntrn, Nval,\n train_user_id_sampling, valid_user_id_sampling,\n train_user_id_indices, valid_user_id_indices,\n \n ), './preload.jlib')\n joblib.dump((\n embedded_user_ids_map, mask_user_ids, part_ids_map, lecture_ids_map, bundle_id_map,\n question_incorrect_ranks,\n answer_ratio1_mean, answer_ratio1_std,\n answer_ratio2_mean, answer_ratio2_std,\n correct_streak_u_mean, correct_streak_u_std,\n incorrect_streak_u_mean, incorrect_streak_u_std,\n correct_streak_alltime_u_mean, correct_streak_alltime_u_std,\n incorrect_streak_alltime_u_mean, incorrect_streak_alltime_u_std,\n diagnostic_u_recency_1_mean, diagnostic_u_recency_1_std,\n diagnostic_u_recency_2_mean, diagnostic_u_recency_2_std,\n diagnostic_u_recency_3_mean, diagnostic_u_recency_3_std,\n diagnostic_u_recency_4_mean, diagnostic_u_recency_4_std,\n diagnostic_u_recency_5_mean, diagnostic_u_recency_5_std,\n diagnostic_u_recency_6_mean, diagnostic_u_recency_6_std,\n session_content_num_u_mean, session_content_num_u_std,\n session_duration_u_mean, session_duration_u_std,\n session_correctness_u_mean, session_correctness_u_std,\n session_ans_duration_u_mean, session_ans_duration_u_std,\n lifetime_ans_duration_u_mean, lifetime_ans_duration_u_std,\n correctness_u_recency_mean, correctness_u_recency_std,\n part_correctness_u_recency_mean, part_correctness_u_recency_std,\n lag_ts_u_recency_mean, lag_ts_u_recency_std,\n last_ts_u_recency1_mean, last_ts_u_recency1_std,\n last_ts_u_recency2_mean, last_ts_u_recency2_std,\n last_ts_u_recency3_mean, last_ts_u_recency3_std,\n last_incorrect_ts_u_recency_mean, last_incorrect_ts_u_recency_std,\n last_correct_ts_u_recency_mean, last_correct_ts_u_recency_std,\n ),\n './mappings.jlib'\n )\n\n# TODO: Remove this: temp measure...\n#df_train.loc[df_train.lag_ts_u_recency.isna(),'lag_ts_u_recency'] = -999\n#df_valid.loc[df_valid.lag_ts_u_recency.isna(),'lag_ts_u_recency'] = -999\n# 80488335/? [08:34<00:00, 156373.86it/s]\n# 80488335/? [07:54<00:00, 169760.91it/s]\n# 80488335/? [07:43<00:00, 173648.30it/s] <-- questions only\n# 80488335/? [07:54<00:00, 169693.34it/s] <-- all content_id\n# 80488335/? [08:11<00:00, 163828.23it/s] <-- whatever we're doing now\ndf_train.head(50)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"# print(d[0]) # user_id 115; I encourage you to match the same with the dataframes.\n# z = next(iter(loader))\n# len(z)","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"df_train.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"df_valid.shape","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"trusted":false},"cell_type":"code","source":"df_train.columns","execution_count":null,"outputs":[]},{"metadata":{"hidden":true,"scrolled":false,"trusted":false},"cell_type":"code","source":"# # z = df_train#.sample(5000000)\n# # y = df_valid#.sample(5000000)\n\n# for f in [\n# 'lag_ts_u_recency'\n# # 'session_content_num_u', 'session_duration_u',\n# # 'last_ts_u_recency1','last_ts_u_recency2','last_ts_u_recency3',\n# # 'last_correct_ts_u_recency','last_incorrect_ts_u_recency',\n# # 'correctness_u_recency','part_correctness_u_recency',\n# ]:\n# ax = plt.subplots(2,2, figsize=(14,10))[1]\n# ax[0,0].set_title(f'{f} train')\n# ax[1,0].set_title(f'{f} valid')\n# ax[0,0].hist(np.log1p(df_train[f]), 30)\n# ax[1,0].hist(np.log1p(df_valid[f]), 30)\n \n\n# ax[0,1].set_title(f'{f} train')\n# ax[1,1].set_title(f'{f} valid')\n# ax[0,1].hist(np.log1p(df_train[f]/1000), 30)\n# ax[1,1].hist(np.log1p(df_valid[f]/1000), 30)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Regular Training"},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"gc.collect()\n\ndef worker_init(worker_id):\n weak_seed = torch.utils.data.get_worker_info().seed % 10240\n random.seed(weak_seed)\n np.random.seed(weak_seed)\n \n# Truncate Timestamp\ntrain_dataset = Riiid(\n npdata=df_train[dataset_cols[COL_OFFSET_CUT:]].values,\n N=Ntrn, #df_train.user_id.nunique(),\n user_id_indices=train_user_id_indices,\n user_id_sampling=train_user_id_sampling,\n mask_user_ids=mask_user_ids\n)\n\ntrain_loader = torch.utils.data.DataLoader(\n dataset=train_dataset,\n batch_size=BATCH_SIZE,\n collate_fn=collate_fn,\n num_workers=4,\n worker_init_fn=worker_init,\n pin_memory=True,\n shuffle=True # doesn't really matter since it'll be random anywaay\n)\n\n\nvalid_dataset = Riiid(\n npdata=df_valid[dataset_cols[COL_OFFSET_CUT:]].values,\n # multiplying by 5 just so we get some coverage....\n N=Nval*5, #df_valid.user_id.nunique(),\n user_id_indices=valid_user_id_indices,\n user_id_sampling=valid_user_id_sampling,\n mask_user_ids=mask_user_ids,\n augment=False\n)\n\nvalid_loader = torch.utils.data.DataLoader(\n dataset=valid_dataset,\n batch_size=BATCH_SIZE,\n collate_fn=collate_fn,\n num_workers=4,\n worker_init_fn=worker_init,\n pin_memory=True,\n shuffle=True\n)\n_ = gc.collect()\n\nprint(valid_dataset[0]) # sample dataset","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"execution":{"iopub.execute_input":"2020-11-12T09:45:32.266755Z","iopub.status.busy":"2020-11-12T09:45:32.266165Z","iopub.status.idle":"2020-11-12T09:45:32.303969Z","shell.execute_reply":"2020-11-12T09:45:32.303466Z"},"papermill":{"duration":0.059005,"end_time":"2020-11-12T09:45:32.304076","exception":false,"start_time":"2020-11-12T09:45:32.245071","status":"completed"},"scrolled":true,"tags":[],"trusted":false},"cell_type":"code","source":"gc.collect()\n\n# TODO: 1e-4 - 5e-5\nlr = 2.5e-4 # SAINT+ paper = 1e-3\nlr = 1e-3\nWEIGHT_DECAY = 0.0000001 #0.00001 # significant...\n\n# creating the model\nmodel = SaintTransformerModel(\n d_model=128, # minimally\n nhead=4, # leave it\n num_layers=2, # maybe increase it 2-4\n dropout=0.05, # can be increased....\n emb_dropout=0.05,\n initrange=0.02\n)\n# lr=2e-3\n# model = SiameseNetwork([\n# SaintTransformerModel(\n# d_model=128, # minimally\n# nhead=4, # leave it\n# num_layers=2, # maybe increase it 2-4\n# dropout=0.1, # can be increased....\n# emb_dropout=0.15,\n# initrange=0.02\n# ),\n# SaintTransformerModel(\n# d_model=128, # minimally\n# nhead=4, # leave it\n# num_layers=2, # maybe increase it 2-4\n# dropout=0.15, # can be increased....\n# emb_dropout=0.1,\n# initrange=0.02\n# )\n# ])\n\n# model = SaintTransformerModel(\n# d_model=256,\n# nhead=4,\n# num_layers=4,\n# dropout=0.1,\n# emb_dropout=0.1,\n# initrange=0.02\n# )\n\n# # createing the mdoel\n# model = SaintTransformerModel(\n# d_model=256, # minimally\n# nhead=4, # leave it\n# num_layers=2, # maybe increase it 2-4\n# dropout=0, # can be increased....\n# emb_dropout=0,\n# initrange=0.02\n# )\n\n# # Finetuning: Load ML-Model:\n# lr = 1e-4\n# # state = torch.load(f'{PREFIX}/pretrain_bundle_best-loss-leanmodel.pth')\n# state = torch.load(f'45-cvfold0/best-loss-leanmodel.pth')\n# for key in list(state.keys()):\n# if 'user_embeddings' in key:\n# state.pop(key)\n# continue\n# state[key[len('module.'):]] = state[key]\n# del state[key]\n# print(model.load_state_dict(state, strict=False))\n\n# # Patch the decoder to accept our user embedding, which is new...\n# # NOTE: We really hsould create 2 sepaarate model classes:\n# for name, param in model.named_parameters(): \n# if name.startswith('decoder'): continue\n# if 'user_embeddings' in name: continue\n# param.requires_grad = False\n\n# decoder = torch.zeros([1,128+16], dtype=torch.float32)\n# decoder.data[:,:128] = model.decoder.weight.data # x.clone().detach()\n# model.decoder.weight.data = decoder.data\n# model.decoder.in_features = 128+16\n# # self.weight = Parameter(torch.Tensor(out_features, in_features))\n\nmodel.to(DEVICE) # look into it!\nmodel = nn.DataParallel(model)\n\nloss_fn_tr = nn.BCEWithLogitsLoss()\nloss_fn_vl = nn.BCEWithLogitsLoss()\n\n# remove biases and layernorm from weight decay and add weight decay for regularization\nparam_optimizer = list(model.named_parameters())\nno_decay = ['bias', 'norm']\noptimizer_grouped_parameters = [\n # NOTE: LayerNorm's weights + biases are both included in the do not decay list\n {'params': filter(lambda p: p.requires_grad, [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)]), 'weight_decay': WEIGHT_DECAY},\n {'params': filter(lambda p: p.requires_grad, [p for n, p in param_optimizer if any(nd in n for nd in no_decay)]), 'weight_decay': 0.0}\n]\noptimizer = torch.optim.Adam(optimizer_grouped_parameters, lr=lr)\n\n\nwarmup_step_scheduler = None\n# warmup_step_scheduler = LinearWarmupStepSchedule(\n# optimizer=optimizer,\n# warmup_steps=len(train_loader)//4 # first two epochs? //3*2 # first ~67%\n# )\n\nlr_redux_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n optimizer=optimizer,\n factor=0.5,\n patience=1,\n verbose=True, \n threshold_mode='abs',\n min_lr=1e-7,\n eps=1e-7\n)\n\n#_ = count_parameters(model)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"# adjusted initrange of ts and removed decoder bias 0 fill\nbest_loss = 999\npatience = 4\nlog = []\ngc.collect()\ntorch.cuda.empty_cache()\n\nprint('Full training progress...')\nfor epoch in tqdm(range(N_EPOCHS)):\n trn_losses, trn_meter = train_fn(\n model,\n optimizer,\n step_scheduler=warmup_step_scheduler,\n loss_fn=loss_fn_tr,\n dataloader=train_loader,\n device=DEVICE\n )\n #trn_losses.draw(epoch, trn_meter.avg, 'Train')\n \n val_losses, val_meter = valid_fn(\n model,\n scheduler=lr_redux_scheduler,\n loss_fn=loss_fn_vl,\n dataloader=valid_loader,\n device=DEVICE\n )\n #val_losses.draw(epoch, val_meter.avg, 'Valid')\n log.append(draw(trn_losses.vals, trn_meter.avg, val_losses.vals, val_meter.avg))\n \n \n if val_losses.avg >= best_loss:\n ez_out -= 1\n print(f'Loss not better... {ez_out} more chances.')\n if ez_out <= 0:\n print('Breaking!')\n break\n \n else:\n # Save model...\n best_loss = val_losses.avg\n ez_out = patience\n \n save(\n f'{PREFIX}/epoch{epoch}-checkpoint.pth',\n model, optimizer,\n best_loss,\n epoch,\n step_scheduler=warmup_step_scheduler,\n scheduler=lr_redux_scheduler,\n lean=False\n )\n \n save(\n f'{PREFIX}/best-loss-leanmodel.pth',\n model, None,\n None,\n None,\n step_scheduler=None,\n scheduler=None,\n lean=True\n )\n \n pd.DataFrame(log, columns=['TrnLoss','ValLoss','TrnAUC','ValAUC']).to_csv(f'{PREFIX}/log.csv', index=False)\n\n# Best Val Loss\npd.DataFrame(log, columns=['TrnLoss','ValLoss','TrnAUC','ValAUC'])#.sort_values('ValLoss')\n# A: 0.7422, 0.7829 full layered history aadded to decoder input\n# now testing full layer history added to clasasificaation layer","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# NOTE: Lag feature is completely removed......\n\nz=pd.DataFrame(log, columns=['TrnLoss','ValLoss','TrnAUC','ValAUC'])\nstr(np.round(z.ValAUC,4).tolist()).replace(', 0.', ',')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"7795,7844,7861,7885,7886,7899,7899,7915,7920 <-- attempt_no with part_correctness_u merged into correctness MARK_A\n7776,7826,7850,7878,7890,7892,7908,7914,7912,7921,7927,7931 <--- using correct ts for last_incorrect_ts_u/last_correct_ts_u (lectures dont count) 7884 train auc\n7780,7828,7851,7880,7892,7892,7912,7918,7918,7926,7931,7933,7934,7935,7939 <-- lectures count as \"questions\". ts and ratio features build off of all content, and lectures are counted as always answered correctly (7901)\n7793,7843,7873,7887,7898,7903,7911,7911,7923,7929,7931,7940,7937,7941, <-- same as above but corrected last_ts_u_recency2/3_mean s.t. not based on current ts(!) (7910 train)\n7809,7849,7873,7888,7897,7904,7908,7915,7923,7926,7932,7938,7942,7942,7946 (7919) <-- refactored dataset building, trained to early stop\n7783,7826,7843,7860,7865,7864,7882,7887,7889,7880,7886,7891,7894,7892,7898 (7907) <-- abs log encoded lag_ts_u_recency\n7789,7846,7872,7888,7893,7912,7904,7920,7926,7929,7936,7933,7936,7933,7941,7940,7940 (7915) <-- {0,1440} clipped log encoded lag_ts_u_recency \n7796,7849,7870,7889,7896,7909,7904,7927,7923,7932,7931,7931,7936,7939,7942,7940 (7910) <--- raw lag_ts_u_recency std scaled\n7810,7848,7873,7891,7895,7910,7919,7917,7931,7927,7933,7937,7940 (7903) <-- tag_group_id\n7822,7871,7897,7911,7920,7928,7941,7941,7946,7954,7952,7953,7959,7957 (7925) <-- 2 layer masked history\n7833,7884,7902,7917,7927,7930,7939,7944,7944,7955,7954,7956,7962,7965,7965,7971,7971,7969 (7964)<-- 1 layer masked history w/ pos emb (start=0)\n7830,7881,7911,7918,7925,7931,7939,7940,7952,7952,7958,7960,7957,7964 (7951) <-- tcid_size_id\n7836,7878,7905,7909,7924,7930,7941,7943,7953,7952,7964,7966,7971,7969,7974,7977 (7962) <-- session_content_num_u & session_duration_u continuous linear embedded\n7839,7884,7903,7918,7930,7934,7950,7949,7957,7957,7961,7965,7969 (7943) <-- session_correctness_u\nbad <-- session_correctness_u as above, with batchnorm on linears and all linear biases corrected to =False\nno change <-- 256 w 4layer\n7839,7881,7907,7920,7936,7941,7949,7951,7957,7964,7958,7968,7970,7967,7968,7971,7976,7974,7977,7974,7976 (7968) <-- 2,3\n7838,7881,7906,7919,7935,7940,7950,7951,7956,7963,7957,7968,7970,7967,7969,7971,7976,7974,7977,7975,7976 (7967) <-- 6,9\n7838,7883,7905,7921,7933,7940,7946,7947,7957,7958,7960,7968,7972,7972,7971,7981 (7971) <-- 2,3 with 5 diagnostic features target clustered\n7817,7874,7899,7905,7924,7922,7927,7933,7927,7928,7932,7936,7933,7937,7936,7933,7933,7934 <-- 256 w/o no dropout at all\n7843,7890,7909,7923,7939,7948,7961,7962,7982,7981,7983,7988,7992,7988,7992,7992,7995,7994,7997 (7973) <-- titos cv fold 1 w/ TCID internal shuffle aug and 5 diagnostic features..\n7833,7888,7904,7922,7936,7946,7948,7954,7959,7962,7967,7972,7980,7979,7983,7982,7987 (7977) <-- session_ans_duration_u, lifetime_ans_duration_u\n7833,7889,7905,7921,7937,7945,7948,7954,7971,7972,7975,7975,7980,7980,7981,7979,7984 (7970) <-- same as above but with log(ms) instead of log(s)\n7832,7889,7907,7922,7935,7948,7956,7961,7955,7973,7975,7977,7981,7982,7983,7989,7990,7992,7992,7993,7996,7995,7999,7998 (8004) <-- same as above but sampling doesnt prefer tail end 3x and inter-user tcid pair-swapping\n7971,7980,7976,7977,7985,7984,7993,7992,7989,7990,7993 (7985) <-- pretraining of everything except exercise embedding and then finetuning exercise embedding\n7842,7889,7914,7925,7934,7944,7953,7958,7962,7965,7965,7972,7971,7972,7975,7982,7977,7981,7987,7990,7992,7990,7996 (8006) <-- user_id embedding\n7829,7881,7903,7927,7938,7946,7951,7956,7965,7968,7968,7970,7984,7987,7983,7990,7987,7993,7990,7993,7996,7998,7999,7998,7999,8001,8000,7997,8004,8007,8005 [8015],8005,7999,8001,8003 Base Model (no user_id) \n7836,7892,7916,7926,7935,7939,7952,7965,7963,7965,7974,7983,7979,7983,7984,7986,7984,7979,[7995],7993,7990,7995,7989 (7985) <-- longest same answer streak\n7826,7877,7905,7918,7935,7938,7949,7950,7957,7961,7962,7970,7967,7968,7982,7986,7986,7985,7993,7991,7996,7993,7994,7999,[7998],8001,8003,8002,8000, (8006) <-- encoder FFN no attention\n[redo] <-- memory added as input into decoder\n7842,7893,7909,7931,7939,7945,7949,7955,7953,7963,7967,7966,7965,7973,7980,7988,7991,7985,7985,7997,7995,7990,[7998],7997,7999,8000,7998 (8016) <-- 2 streak features, but im uncertain I rebuilt the dataset =( so gonna rerun again...\n7836,7886,7905,7920,7932, 7937,7947,7954,7955,7962, 7962,7961,7965,7968,7974, 7973,7975,7976,7977,7983, 7974,7975,7987,7993,7994, 7992,[7995],7996,7989,7997,7997 (8007) no streaaks of any kind but lag features added\n7822,7883,7906,7917,7930, 7940,7947,7952,7952,7965, 7971,7972,7973,7979,7978, 7978,7984,7988,7979,7982, 7986,7989,7989,7989,7992,7995,7992,7999,7998,8,7998,7994<-- full history encoder stack mixes in encoder base features which is added to penultimate output\n\nwe can try same as above but using the same weights...\n\nWhat were on:\n7831,7880,7904,7922,7929, 7940,7953,7953,7960,7960, 7972,7974,7974,7985,7988, 7988,7992,7997,7994,7999,7999,7997,8,8002,8002,8003,8002,8004,801,8004,8006,8002,8006]'\n","execution_count":null,"outputs":[]},{"metadata":{"heading_collapsed":true},"cell_type":"markdown","source":"# History"},{"metadata":{"hidden":true},"cell_type":"markdown","source":"Mine E7:\n- L: 0.497, 0.467\n- A: 0.751, 0.758\n\nMine E12 w/ WD:\n- L: 0.500, 0.467\n- A: 0.746, 0.757\n\nMine E8 w/ Incorrect Rank:\n- L: 0.498, 0.467\n- A: 0.753, 0.759\n\nMine E12 w/ BXE\n- L: 0.491, 0.466\n- A: 0.759, 0.760\n\nLectures E5\n- L: 0.485, 0.457\n- A: 0.753, 0.761\n\nStatic Pos Emb E3\n- L: 0.479, 0.455\n- A: 0.763, 0.764\n\nSin Pos Emb E11\n- L: 0.474, 0.453\n- A: 0.769, 0.767\n\nTS Features E11\n- L: 0.472, 0.452\n- A: 0.772, 0.769\n\nBundleID + Linear TS Diff E9:\n- L: 0.442, 0.423\n- A: 0.793, 0.787\n\nEncoderQuestion, DecoderResponse\n- L: 0.4436, 0.4216\n- A: 0.7903, 0.7874\n\nSwap Memory and Output on Decoder E6 <-- good, we'll ensemble with this\n- L: 0.4456, 0.4215\n- A: 0.7888, 0.7878\n\nEncoder Dropout @20% E8 <-- good, we need to increase dropout\n- L: 0.4456, 0.4206\n- A: 0.7888, 0.7879\n\nPQExplanation and ContentTypeId E8 <-- not saturated due to dropout\n- L: 0.4454, 0.4201\n- A: 0.7890, 0.7885\n\nSame w/ DataParallel to effectively half BS, E11\n- L: 0.4425, 0.4202\n- A: 0.7917, 0.7892\n\nItemResponseTheory instead of sigmoid E9\n- L: 0.4465, 0.4204\n- A: 0.7893, 0.7884\n\nIRT w/ HeavyEncoder (all inputs except correctness+pos) E9\n- L: 0.4453, 0.4197\n- A: 0.7908, 0.7893\n\nHeaviestEncoder = as above, no decoder and 4 encoder layers E7\n- L: 0.4462, 0.4203\n- A: 0.7892, 0.7892\n\n8 Layer Encoder Stack E10 (HeaviestEncoder) trained with 50% LR\n- L: 0.4440, 0.4189\n- A: 0.7919, 0.7913\n\n512 Dim Encoder Stack (HeaviestEncoder) E8\n- L: 0.4447, 0.4199\n- A: 0.7912, 0.7903 <-- standard embedded dropout = 0.2\n\n- L: 0.4418, 0.4201\n- A: 0.7948, 0.7909 <-- when we increase embedded dropout = 0.4\n\nTS Offset (256dim 4layer-encoder only) E10\n- L: 0.4425, 0.4201\n- A: 0.7937, 0.7894\n- this is identical to HeaviestEncoder but with the TSOffset Input. It doesn't seem to help..\n\nExtended Train CAP_SAMPLING_LENGTH E9 (HeaviestEncoder)\n- L: 0.4644, 0.4178\n- A: 0.7885, 0.7914\n\nUser Correctness CumMean (Based on above model) E17\n- L: 0.4574, 0.4164\n- A: 0.7965, 0.7949\n\nTag0 - Once again, raw tags are useless\n- L: 0.4600, 0.4158\n- A: 0.7934, 0.7945\n\nTag Community Feature E18\n- L: 0.4563, 0.4157\n- A: 0.7978, 0.7959\n\nTimestamp Embedding E12\n- L: 0.4596, 0.4165\n- A: 0.7940, 0.7945\n\nFull Embedding FFN w/o timestamp/tag\n- L: 0.4597, 0.4160\n- A: 0.7943, 0.7951\n\n384 Emb size w/ 8layer stack - current whatever E21 with lr//5\n- L: 0.4571, 0.4139\n- A: 0.7956, 0.7958\n\nfixed std,mean and added adaptive chance for IPT\n\nRegular size. With bad std/mean for user part correctness. E 10\n- L: 0.4602, 0.4165\n- A: 0.7932, 0.7939\n\nUsing user part correctness adjusted per question difficulty\n- ...\n\nLag Feature\n- Hurts...\n\nNo_Responses\n- L: 0.4555, 0.4133\n- A: 0.7975, 0.7973\n\nNo_Responses Categorical\n- L: 0.4583, 0.4141\n- A: 0.7944, 0.7967\n\ntimestamp_u_incorrect_recency, timestamp_u_correct_recency features\n- L: 0.4563, 0.4129\n- A: 0.7967, 0.7974\n\npatched masking\n- L: 0.5052, 0.5121\n- A: 0.7961, 0.7977"},{"metadata":{},"cell_type":"markdown","source":"# Fin~"},{"metadata":{},"cell_type":"raw","source":"2) OOM =Out of memory reason. If you are using merge for test data to join with the meta data of question,lecture etc , use copy=False which might save some crucial memory. 3) Do left join and assign meta feature null values to default values. \n\n\npd.merge(test_df,questions_df[['question_id','XXXXX']],how='left',copy=False) \n\npickling dicts: https://www.kaggle.com/syxuming/two-ways-to-reduce-memory-about-load-dict-pkl"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}