{"cells":[{"metadata":{},"cell_type":"markdown","source":"Leaving this information after collecting it from docs and multiple discussions etc,\n\n- **src (torch.Tensor)** – the sequence to the encoder (required). shape: (𝑁,𝑆,𝐸).\n- **tgt (torch.Tensor)** – the sequence to the decoder (required). shape: (𝑁,𝑇,𝐸).\n- **src_mask (torch.Tensor, optional)** – the additive mask for the src sequence (optional). shape: (𝑆,𝑆).\n- **tgt_mask (torch.Tensor, optional)** – the additive mask for the tgt sequence (optional). shape: (𝑇,𝑇).\n- **memory_mask (torch.Tensor, optional)** – the additive mask for the encoder output (optional). shape: (𝑇,𝑆).\n- **src_key_padding_mask (torch.Tensor, optional)** – the ByteTensor mask for src keys per batch (optional). shape: (𝑁,𝑆)\n- **tgt_key_padding_mask (torch.Tensor, optional)** – the ByteTensor mask for tgt keys per batch (optional). shape: (𝑁,𝑇).\n- **memory_key_padding_mask (torch.Tensor, optional)** – the ByteTensor mask for memory keys per batch (optional). shape” (𝑁,𝑆).\n\n- **output (torch.Tensor)** – The output sequence, shape: (𝑁,𝑇,𝐸).\n\n- **Note ([src/tgt/memory]_mask** should be filled with) float(‘-inf’) for the masked positions and float(0.0) else.\n\n- **These masks ensure that predictions for position i depend only on the unmasked positions j and are applied identically for each sequence in a batch.** \n\n- **[src/tgt/memory]_key_padding_mask should be a ByteTensor where False values are positions that should be masked with float(‘-inf’) and True values will be unchanged. This mask ensures that no information will be taken from position i if it is masked, and has a separate mask for each sequence in a batch.**\n\n\n# # answered correctly is not padded"},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T13:59:33.155576Z","start_time":"2020-11-19T13:59:31.430232Z"},"trusted":true},"cell_type":"code","source":"# import some typical packages and fix random seed\n\nimport numpy as np\nimport pandas as pd\nimport pickle\nimport gc\nimport time\nimport lightgbm as lgbm\nimport itertools\n\n\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import roc_auc_score, auc, roc_curve\nfrom contextlib import contextmanager\nfrom pathlib import Path\nfrom time import time\nfrom pathlib import Path\nfrom collections import namedtuple\n\nRANDOM_SEED = 42\nnp.random.seed(RANDOM_SEED)\ngc.enable()\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport numpy as np\nimport math\n\nfrom torch.nn import TransformerEncoder, TransformerEncoderLayer, TransformerDecoder, TransformerDecoderLayer\nfrom torch.nn.modules.dropout import Dropout\nfrom torch.nn.modules.activation import MultiheadAttention\nfrom torch.nn.modules.normalization import LayerNorm\n\nTRAIN_DTYPES = {\n    # 'row_id': np.uint32,\n    'timestamp': np.uint64,\n    'user_id': np.uint32,\n    'content_id': np.uint16,\n    'content_type_id': np.uint8,\n    'task_container_id': np.uint16,\n    'user_answer': np.int8,\n    'answered_correctly': np.int8,\n    'prior_question_elapsed_time': np.float32,\n    'prior_question_had_explanation': 'boolean'\n}\n\nDATA_DIR = Path('../input/riiid-test-answer-prediction')\nTRAIN_PATH = DATA_DIR / 'train.csv'\nQUESTIONS_PATH = DATA_DIR / 'questions.csv'\nLECTURES_PATH = DATA_DIR / 'lectures.csv'\n\n'''\nMasking in the encoder is required to make sure any padding doesn't contribute to the self-attention mechanism. \nIn Pytorch, this is done by passing src_key_padding_mask to the transformer. \nFor the example, this looks like [False, False, False, False, False, False, False, True, True, True] \nwhere the True positions should be masked. The output of the encoder is called memory.\n''';\n\n'''\nS is the source sequence length, N is the batch size, E is the feature number.\nsrc: (S, N, E) = (source sequence len, batch size, feature number)\ntgt: (T, N, E) = (target sequence len, batch size, feature number)\nsrc_mask: (S, S) = the additive mask for the src sequence\ntgt_mask: (T, T) = (target sequence len, target sequence len)\nsrc_key_padding_mask: (N, S) = (batch size, source sequence len)\ntgt_key_padding_mask: (N, T) = (batch size, target sequence len)\nmemory_key_padding_mask: (N, S) = (batch size, source sequence len)\n\nkey_padding_mask controls how which batch items are allowed to attend to certain key positions. \nThis is most commonly used to avoid attending to padding elements. \n\nattn_mask controls how query positions are allowed to attend to key positions. \nThis is useful for doing left-to-right (causal) attention, where we enforce that query positions are \nonly allowed to attend to keys to their left.\n''';\n\nfrom sklearn.metrics import accuracy_score\n\nclass MaskedCrossEntropyLoss(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.loss_ = nn.CrossEntropyLoss()\n\n    def forward(self, input, target, mask=None):\n        # Flatten to (BS*seq_len)\n        input_ = input.reshape(-1)\n        target_ = target.reshape(-1)\n        if mask is not None:\n            mask_ = mask.reshape(-1)\n            input_ = input_[mask_ == False]\n            target_ = target_[mask_ == False]\n        return self.loss_(input_, target_)\n\n    \ndef roc_auc_compute_fn(y_targets, y_preds):\n    try:\n        from sklearn.metrics import roc_auc_score\n    except ImportError:\n        raise RuntimeError(\"This contrib module requires sklearn to be installed.\")\n    \n\n    y_true = y_targets.detach().cpu()\n    y_pred = y_preds.detach().cpu()\n    \n    return roc_auc_score(y_true, y_pred)\n\n\ndef accuracy_score_compute_fn(y_targets, y_preds):\n    try:\n        from sklearn.metrics import roc_auc_score\n    except ImportError:\n        raise RuntimeError(\"This contrib module requires sklearn to be installed.\")\n    \n    # ON CPU\n    y_true = y_targets.detach().cpu()\n    y_pred = y_preds.detach().cpu()\n#     y_true = y_targets.cpu()\n#     y_pred = y_preds.cpu()\n    return accuracy_score(y_true, y_pred)\n\ntorch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing the Dataset"},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T13:59:40.753401Z","start_time":"2020-11-19T13:59:33.251431Z"},"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n\nCV_NO = 2\nLAST_N = 100\n\nPAD = 0\nSOS_TOKEN = 2\nPAD_ANS = 3\nTIMELINESS_PAD = 2\nCNT = 0\nLAG_PAD = 0\n\ndf_questions = pd.read_csv(QUESTIONS_PATH)\npart_ids_map = dict(zip(df_questions.question_id, df_questions.part))\n\n\n# answered correctly is not padded\nwith open(\"../input/saint-inference-user-id-as-key-pkls/d_train_2_saint_inference_with_lag.pkl/d_train_2_saint_inference_with_lag.pkl\", \"rb\") as fid:\n    d_train = pickle.load(fid)\n\n\n# with open(\"../input/saint-inference-user-id-as-key-pkls/cache/d_valid_2_saint_inference.pkl\", \"rb\") as fid:\n#     d_valid = pickle.load(fid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(d_train[115])","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T13:59:48.96973Z","start_time":"2020-11-19T13:59:48.946906Z"},"code_folding":[1,6,64,89,127,139],"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Source: https://pytorch.org/tutorials/beginner/transformer_tutorial\ndef clones(module, N):\n    \"Produce N identical layers.\"\n    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])\n\n\n# Source: https://pytorch.org/tutorials/beginner/transformer_tutorial\nclass PositionalEncoding(nn.Module):\n    '''\n    A piece of information added to each word about its position in the sentence in Transformers.\n    Even Indices Sine; Oddd Indices Cosine;\n    Had replaced math.log with np.log;\n    '''\n    def __init__(self, d_model, dropout=0.1, max_len=LAST_N):\n        super(PositionalEncoding, self).__init__()\n        self.dropout = nn.Dropout(p=dropout)\n        pe = torch.zeros(max_len, d_model)\n        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-np.log(10000.0) / d_model))\n        pe[:, 0::2] = torch.sin(position * div_term)\n        pe[:, 1::2] = torch.cos(position * div_term)\n        pe = pe.unsqueeze(0).transpose(0, 1)\n        self.register_buffer('pe', pe)\n\n    def forward(self, x):\n        x = x + self.pe[:x.size(0), :]\n        return self.dropout(x)\n\n\n# The below implements the optimizer that SAINT uses.\nclass NoamOpt(object):\n    r'''Noam Optim wrapper that implements rate.\n    Three settings of the lrate hyperparameters.\n    opts = [NoamOpt(512, 1, 4000, None), NoamOpt(512, 1, 8000, None), NoamOpt(256, 1, 4000, None)]\n    plt.plot(np.arange(1, 20000), [[opt.rate(i) for opt in opts] for i in range(1, 20000)])\n    plt.legend([\"512:4000\", \"512:8000\", \"256:4000\"])\n    '''\n    def __init__(self, model_size, factor, warmup, optimizer):\n        self.optimizer = optimizer\n        self._step = 0\n        self.warmup = warmup\n        self.factor = factor\n        self.model_size = model_size\n        self._rate = 0\n        \n    def step(self):\n        \"Update parameters and rate\"\n        self._step += 1\n        rate = self.rate()\n        for p in self.optimizer.param_groups:\n            p['lr'] = rate\n        self._rate = rate\n        self.optimizer.step()\n        \n    def rate(self, step = None):\n        r\"Implement `lrate` above\"\n        if step is None:\n            step = self._step\n        return self.factor * (self.model_size ** (-0.5) * min(step ** (-0.5), step * self.warmup ** (-1.5)))\n\n\ndef grab_optimizer(model):\n    return NoamOpt(model.src_embed[0].d_model, 2, 4000, torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))\n\n\nclass raw_conf:\n\n    mtype = \"SAINT\"\n    backbone = \"transformer\" \n\n    pad_mode = \"random\" # i.e. pad with data from another user\n    seq_len = 100\n    embedding_dim = 256 # embed_dim must be divisible by num_heads\n    exercices_id_size = 13782\n    exercices_part_size = 7\n    response_size = 2 \n    elapsed_time_size = 300 \n    lag_time_size = 720 # It was 1440 in SAINT paper.\n    explanation_size = 2\n    position_encoding_enabled = True\n\n    # Model\n    nhead = 8\n    num_encoder_layers = 4\n    num_decoder_layers = 4\n    dim_feedforward = 2048\n    dropout = 0.1\n    activation = None","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T13:59:50.527764Z","start_time":"2020-11-19T13:59:50.510636Z"},"code_folding":[79,89,118],"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class SAINT(nn.Module):\n\n    def __init__(self,\n                 d_model: int = 64,\n                 nhead: int = 8, \n                 num_encoder_layers: int = 2, num_decoder_layers: int = 2,\n                 dim_feedforward: int = 64, dropout: float = 0.1, activation: str = 'relu',\n                 device=\"cpu\"\n        ):\n        \n        super(SAINT, self).__init__()\n\n        encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout, activation)\n        encoder_norm = LayerNorm(d_model)\n        self.encoder = TransformerEncoder(encoder_layer, num_encoder_layers, encoder_norm)\n        \n        decoder_layer = TransformerDecoderLayer(d_model, nhead, dim_feedforward, dropout, activation)\n        decoder_norm = LayerNorm(d_model)\n        self.decoder = TransformerDecoder(decoder_layer, num_decoder_layers, decoder_norm)\n\n        self._reset_parameters() # inits the weights and all by default\n\n        self.d_model = d_model\n        self.nhead = nhead\n        self.SEQ_LEN = 100\n        self.device = device\n        \n        self.POS_START_TOKEN = 0\n        self.CORRECTNESS_START_TOKEN = 2\n        self.ELAPSED_START_TOKEN = 0\n        self.LAG_START_TOKEN = 1\n        \n        # NB position embeddings weights are tied wrt encoder and decoder.\n        \n        # ---------------#        \n        ## ENCODER SIDE ##\n        # ---------------#\n        \n        self.exercise_embeddings = nn.Embedding(num_embeddings=13524+1, embedding_dim=d_model, padding_idx=0)# Exercise\n        self.pos_embedding = PositionalEncoding(d_model=d_model, max_len=LAST_N) # nn.Embedding(self.SEQ_LEN+1, d_model, padding_idx=0) # Position\n        self.part_embeddings = nn.Embedding(num_embeddings=7+1, embedding_dim=d_model) # Part\n        # self.timeliness_embeddings = nn.Embedding(num_embeddings=2+1, embedding_dim=d_model, padding_idx=2)\n        \n        # ---------------#\n        ## DECODER SIDE ##\n        # ---------------#\n        \n        # Position Embeddings -> shared b/w them\n        self.correctness_embedding = nn.Embedding(num_embeddings=2+1+1, embedding_dim=d_model, padding_idx=3)# Correctness\n        self.prior_question_elapsed_time = nn.Embedding(num_embeddings=301, embedding_dim=d_model)# Elapsed Time\n        self.lag_time = nn.Embedding(num_embeddings=301+1+1, embedding_dim=d_model, padding_idx=0) # Lag Time\n        \n        self.ffn = nn.Linear(d_model, 2)\n        \n    def _reset_parameters(self):\n        r\"\"\"Initiate parameters in the transformer model.\"\"\"\n\n        for p in self.parameters():\n            if p.dim() > 1:\n                nn.init.xavier_uniform_(p)\n\n    def _generate_square_subsequent_mask(self, sz):\n        r\"\"\"\n        For training.\n        The masked positions are filled with float('-inf'). \n        Unmasked positions are filled with float(0.0).\n        \"\"\"\n        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)\n        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n        return mask\n\n    def _generate_mem_mask(self, tgt_sz=LAST_N, src_sz=LAST_N):\n        mask = (torch.triu(torch.ones(src_sz, tgt_sz)) == 1).transpose(0, 1)\n        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n        return mask\n\n    def forward(self,\n                lag=None,\n                content_id=None, part_id=None, prior_question_elapsed_time=None, # timeliness=None,\n                answered_correctly=None, src_key_padding_mask=None, test=False,\n        ):\n        r\"\"\"Take in and process masked source/target sequences.\n\n        Args:\n            src: the sequence to the encoder (required).\n            tgt: the sequence to the decoder (required).\n            src_mask: the additive mask for the src sequence (optional).\n            tgt_mask: the additive mask for the tgt sequence (optional).\n            memory_mask: the additive mask for the encoder output (optional).\n            src_key_padding_mask: the ByteTensor mask for src keys per batch (optional).\n            tgt_key_padding_mask: the ByteTensor mask for tgt keys per batch (optional).\n            memory_key_padding_mask: the ByteTensor mask for memory keys per batch (optional).\n\n        Shape:\n            - src: :math:`(S, N, E)`.\n            - tgt: :math:`(T, N, E)`.\n            - src_mask: :math:`(S, S)`.\n            - tgt_mask: :math:`(T, T)`.\n            - memory_mask: :math:`(T, S)`.\n            - src_key_padding_mask: :math:`(N, S)`.\n            - tgt_key_padding_mask: :math:`(N, T)`.\n            - memory_key_padding_mask: :math:`(N, S)`.\n\n            Note: \n            \n            [src/tgt/memory]_mask ensures that position \"i\" is allowed to attend the unmasked\n            positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend\n            while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``\n            are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor\n            is provided, it will be added to the attention weight.\n            \n            [src/tgt/memory]_key_padding_mask provides specified elements in the key to be ignored by\n            the attention. If a ByteTensor is provided, the non-zero positions will be ignored while the zero\n            positions will be unchanged. If a BoolTensor is provided, the positions with the\n            value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.\n\n            - output: :math:`(T, N, E)`.\n\n            Note: Due to the multi-head attention architecture in the transformer model,\n            the output sequence length of a transformer is same as the input sequence\n            (i.e. target) length of the decode.\n\n            where S is the source sequence length, T is the target sequence length, N is the\n            batch size, E is the feature number\n\n        Examples:\n            >>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)\n        \"\"\"\n        \n        device = self.device\n        # ENCODER SIDE\n        src_mask = self._generate_square_subsequent_mask(self.SEQ_LEN).to(device)\n\n        # using nn.Embedding Only as Positional Embedding\n        # embedded_src = self.exercise_embeddings(content_id) + \\\n        # self.pos_embedding(torch.arange(0, content_id.shape[1]).to(self.device).unsqueeze(0).repeat(content_id.shape[0], 1)) + \\\n        # self.part_embeddings(part_id) + self.prior_question_elapsed_time(prior_question_elapsed_time)\n        \n        # BATCH_SIZE, MAX_SENT_LEN, EMBEDDING_DIM (N, S, E)\n        embedded_src = self.exercise_embeddings(content_id) + \\\n        self.part_embeddings(part_id) + self.prior_question_elapsed_time(prior_question_elapsed_time) # + self.timeliness_embeddings(timeliness)\n        \n        # BATCH_SIZE, SEQ_LEN, EMBEDDING_DIM (N, S, E)\n        embedded_src = self.pos_embedding(embedded_src.permute(1,0,2)) # pos_enc(src.permute(1, 0, 2))\n        \n        embedded_src = embedded_src * np.sqrt(self.d_model)\n        \n        # (S, N, E)\n        _memory = self.encoder(src=embedded_src, mask=src_mask, src_key_padding_mask=src_key_padding_mask)\n        del embedded_src\n        \n        # DECODER SIDE\n        \n        tgt_mask = self._generate_square_subsequent_mask(self.SEQ_LEN).to(device)\n        memory_mask = self._generate_square_subsequent_mask(self.SEQ_LEN).to(device)\n        \n        tgt_key_padding_mask = src_key_padding_mask.clone() # <TO-DO>; here T and S are same length\n        memory_key_padding_mask = src_key_padding_mask.clone()\n        \n        # Correctness\n        x_correctness = answered_correctly.long()\n        # x_correctness =  torch.roll(answered_correctly.long(), shifts=(0, 1, 0), dims=(0, 1, 0)) # Shift right the sequence\n        # x_correctness[:,0] = self.CORRECTNESS_START_TOKEN\n        \n        # Position handled via embeddings.\n        \n        # Elapsed Time\n        x_elapsed = prior_question_elapsed_time\n        # x_elapsed =  torch.roll(prior_question_elapsed_time, shifts=(0, 1, 0), dims=(0, 1, 0))# Shift right the sequence\n        # x_elapsed[:,0] = self.ELAPSED_START_TOKEN\n\n        # Lag Time\n        x_lag = lag\n        # x_lag =  torch.roll(lag, shifts=(0, 1, 0), dims=(0, 1, 0))# Shift right the sequence\n        # x_lag[:,0] = self.LAG_START_TOKEN\n        \n        # BATCH_SIZE, MAX_SENT_LEN, EMBEDDING_DIM (N, S, E)\n        embedded_tgt = self.correctness_embedding(x_correctness) + self.prior_question_elapsed_time(x_elapsed)\n        embedded_tgt = self.pos_embedding(embedded_tgt.permute(1,0,2)) # pos_enc(src.permute(1, 0, 2))\n\n        # (S, N, E)\n        # embedded_tgt = embedded_tgt.transpose(0, 1)\n        embedded_tgt = embedded_tgt * np.sqrt(self.d_model)\n        \n        output = self.decoder(\n            tgt=embedded_tgt, memory=_memory,\n            tgt_mask=None, memory_mask=memory_mask, tgt_key_padding_mask=content_id==0,\n            memory_key_padding_mask=memory_key_padding_mask\n        )\n        \n        output = output.transpose(1, 0) # (N, S, E)\n        output = self.ffn(output)\n        \n        if test == True:\n            # To mask future tokens, you should use src_mask, tgt_mask, memory_mask\n            # output = output.contiguous().view(-1, 2)\n            predicted_probs = torch.softmax(output[:,-1,:], dim=1)[:,1]\n            return predicted_probs\n        return output","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T14:03:08.807689Z","start_time":"2020-11-19T14:03:08.376555Z"},"trusted":true},"cell_type":"code","source":"DEVICE = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")\n\nprint(DEVICE)\n\nmodel = SAINT(\n    d_model = 512,\n    nhead = 4,\n    num_encoder_layers = 4, num_decoder_layers = 4,\n    dim_feedforward = 256, dropout = 0.1, activation = 'relu', device=DEVICE\n)\n\nmodel.to(DEVICE);\nmodel.eval();\n\nBATCH_SIZE = 64 # *64\nlr = 1e-4 # learning rate\noptimizer = torch.optim.Adam(model.parameters(), lr=lr)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2020-11-19T14:06:13.694484Z","start_time":"2020-11-19T14:05:07.487803Z"},"code_folding":[0],"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\nmodel.eval();\nmodel.load_state_dict(torch.load(\"../input/saint-model-weights/Archive 2/saint_2_21_hidden_256_d_model_512_layers_4_with_lag.pt\", map_location=DEVICE));\n# torch.save(model.state_dict(), f'saint_testing.pt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"'''\n    #########################################\n    #thanks to @tksasagi for her tweet#######\n    #########################################\n    print('|￣￣￣￣￣￣￣￣|')\n    print('|    VALIDATION    |') \n    print('|     Fold %i prediction cv AUC: %.5f  |' %(i,cv_auc))\n    print('| ＿＿＿_＿＿＿＿|') \n    print(' (\\__/) ||') \n    print(' (•ㅅ•) || ')\n    print(' / 　 づ')\n''';"},{"metadata":{},"cell_type":"markdown","source":"# INFERENCE ATTEMPT - 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nimport os, copy\nimport multiprocessing\nfrom multiprocessing import Process, Queue, Pool\nimport time\n\nTARGET = 'answered_correctly'\nSTATE = {}\nPART_IDS_MAP = part_ids_map\nPRIOR_QUESTION_ELAPSED_MEAN_FILLNA = 26000\nMAX_SEQ = 100\n\nprint(\"TILL TRAINING: 554169193\",)\n\n\n# with open(\"../input/saint-inference-user-id-as-key-pkls/cache/d_train_2_saint_inference.pkl\", \"rb\") as fid:\n#     STATE = pickle.load(fid)\n\nSTATE = d_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ngc.collect()\n\nimport torch\nimport riiideducation\nfrom time import time\nfrom contextlib import contextmanager\n\nenv = riiideducation.make_env()\niter_test = env.iter_test()\n\nprevious_test_df = None\n\nclass Colors:\n    \"\"\"Defining Color Codes to color the text displayed on terminal.\n    \"\"\"\n\n    blue = \"\\033[94m\"\n    green = \"\\033[92m\"\n    yellow = \"\\033[93m\"\n    red = \"\\033[91m\"\n    end = \"\\033[0m\"\n\n\ndef color(string: str, color: Colors = Colors.yellow) -> str:\n    return f\"{color}{string}{Colors.end}\"\n\n\n@contextmanager\ndef timer(label: str) -> None:\n    \"\"\"compute the time the code block takes to run.\n    \"\"\"\n    start = time()  # Setup - __enter__\n    print(color(f\"{label}: Start at {start}\"))\n    try:\n        yield  # yield to body of `with` statement\n    finally:  # Teardown - __exit__\n        end = time()\n        print(color(f\"{label}: End at {end} ({end - start} elapsed secs)\", color=Colors.red))\n\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nwith open(\"../input/gen-all-users-past-content-ids/last_seen_timestamp.pkl\", \"rb\") as fid:\n    last_seen_timestamp = pickle.load(fid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"last_seen_timestamp[115]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"last_seen_timestamp[115.0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(d_train[554169193])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(d_train[554169193][\"content_id\"]), len(d_train[554169193][\"answered_correctly\"]), len(d_train[554169193]['part_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(STATE[275030867]) # BEFORE ANYTHING","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ngc.collect();\nmodel.eval();\n\nMAX_SEQ = 100\nNEW_IDS = []\n\n\nfor idx, (test_df, sample_prediction_df) in enumerate(iter_test):\n    _content_id, _part_id, _prior_question_elapsed_time, _src_key_padding_mask, _timeliness, _answered_correctly, _lag = [], [], [], [], [], [], []\n    \n    if previous_test_df is not None:\n\n        previous_test_df[TARGET] = eval(test_df[\"prior_group_answers_correct\"].iloc[0])\n        previous_test_df = previous_test_df[previous_test_df['content_type_id'] == 0].reset_index(drop=True)\n        previous_test_df = previous_test_df[['user_id', TARGET]].to_numpy()\n        \n        for (user_id, target) in previous_test_df:\n            # old_user, new_user issue will come here as well but no NEW_USER will be unseen at this stage.\n            _len = STATE[user_id]['len']\n            if _len == 1 and len(set(STATE[user_id][TARGET])) == 1:\n                STATE[user_id][TARGET] = []\n                STATE[user_id][TARGET].append(target)\n            else:\n                if _len >= 100:\n#                     if user_id in [555691277]:\n#                         print(\"doing for\", user_id, target)\n#                         print(\"BEFORE UPDATION\", STATE[user_id])\n                    STATE[user_id][TARGET] = STATE[user_id][TARGET][-99:] + [target]\n#                     if user_id in [555691277]:\n#                         print(\"AFTER UPDATION\", STATE[user_id])\n                else:\n                    STATE[user_id][TARGET].append(target)\n\n    \n    previous_test_df = test_df.copy()\n    test_df = test_df[test_df['content_type_id'] == 0].reset_index(drop=True)\n    test_df[\"part_id\"] = test_df[\"content_id\"].map(PART_IDS_MAP)\n    test_df['prior_question_elapsed_time'] = test_df.prior_question_elapsed_time.fillna(PRIOR_QUESTION_ELAPSED_MEAN_FILLNA) // 1000\n    \n    # test_df[\"timeliness\"] = test_df[[\"part_id\", \"prior_question_elapsed_time\"]].apply(lambda x: get_timeliness(x[\"part_id\"], x[\"prior_question_elapsed_time\"]), axis=1)*1\n    \n    # for lag and content_id, we have added a +2 to them; # one for padding and other for SOS\n    test_df[\"content_id\"] += 2\n    # test_df[\"lag\"] = test_df[\"lag\"].clip(lower=1000, upper=300_000) // 1000\n    # test_df[\"lag\"] = test_df[\"lag\"].fillna(0)\n    # test_df[\"lag\"] += 2\n\n    curr = test_df[[\"user_id\", \"content_id\", \"part_id\", \"prior_question_elapsed_time\", \"timestamp\"]].to_numpy()\n    \n    for (user_id, content_id, part_id, prior_question_elapsed_time, timestamp) in curr:\n        # user_id = user_id\n        if user_id in STATE:\n            # great news for us actually, so, let's add the values and restrict them\n            # using padded sequences of MAX_SEQ as 100\n\n            UPDATE_IDX = STATE[user_id]['len']\n            lag = (np.clip(timestamp - last_seen_timestamp[user_id]['timestamp'], 1_000, 300_000) // 1000) + 2\n            last_seen_timestamp[user_id]['timestamp'] = timestamp\n            \n#             # creating the batch we need here by ouselves\n#             _content_id.append(STATE[user_id]['content_id'])\n#             _part_id.append(STATE[user_id]['part_id'])\n#             _prior_question_elapsed_time.append(STATE[user_id]['prior_question_elapsed_time'])\n#             # _timeliness.append(STATE[user_id]['timeliness'])\n#             _lag.append(STATE[user_id]['lag'])\n#             _src_key_padding_mask.append(STATE[user_id]['src_key_padding_mask'])\n\n#             # padding it on FLY here itself\n#             _answered_correctly.append(STATE[user_id]['answered_correctly'] + [3]*(100 - len(STATE[user_id]['answered_correctly'])))\n            \n            if UPDATE_IDX >= MAX_SEQ:\n                STATE[user_id]['content_id'] = STATE[user_id]['content_id'][-99:] + [content_id]\n                STATE[user_id]['part_id'] = STATE[user_id]['part_id'][-99:] + [part_id]\n                STATE[user_id]['prior_question_elapsed_time'] = STATE[user_id]['prior_question_elapsed_time'][-99:] + [prior_question_elapsed_time]\n                STATE[user_id]['lag'] = STATE[user_id]['lag'][-99:] + [lag]\n                # STATE[user_id]['timeliness'] = STATE[user_id]['timeliness'][-99:] + [timeliness]\n                STATE[user_id]['len'] = MAX_SEQ\n            else:\n                STATE[user_id]['content_id'][UPDATE_IDX] = content_id\n                STATE[user_id]['part_id'][UPDATE_IDX] = part_id\n                STATE[user_id]['prior_question_elapsed_time'][UPDATE_IDX] = prior_question_elapsed_time\n                # STATE[user_id]['timeliness'][UPDATE_IDX] = timeliness\n                STATE[user_id]['lag'][UPDATE_IDX] = lag\n                STATE[user_id]['src_key_padding_mask'][UPDATE_IDX] = False\n\n            # creating the batch we need here by ouselves\n            _content_id.append(STATE[user_id]['content_id'])\n            _part_id.append(STATE[user_id]['part_id'])\n            _prior_question_elapsed_time.append(STATE[user_id]['prior_question_elapsed_time'])\n            # _timeliness.append(STATE[user_id]['timeliness'])\n            _lag.append(STATE[user_id]['lag'])\n            _src_key_padding_mask.append(STATE[user_id]['src_key_padding_mask'])\n\n            # padding it on FLY here itself\n            _answered_correctly.append(STATE[user_id]['answered_correctly'] + [3]*(100 - len(STATE[user_id]['answered_correctly'])))\n\n            # point to next updation place except if at-max, then it's MAX_SEQ always.\n            STATE[user_id]['len'] = min(MAX_SEQ, STATE[user_id]['len'] + 1)\n\n        else:\n            NEW_IDS.append(user_id)\n            # A NEW USER\n            \n#             _content_id.append([0] * (MAX_SEQ))\n#             _part_id.append([0] * (MAX_SEQ))\n#             _prior_question_elapsed_time.append([0] * (MAX_SEQ))\n#             _src_key_padding_mask.append([True] * (MAX_SEQ))\n#             _lag.append([0] * (MAX_SEQ))\n#             _answered_correctly.append([3.] * MAX_SEQ)\n            \n            STATE[user_id] = {\n                'content_id': [content_id] + [0] * (MAX_SEQ-1),\n                'part_id': [part_id] + [0] * (MAX_SEQ-1),\n                'prior_question_elapsed_time': [prior_question_elapsed_time] + [0] * (MAX_SEQ-1),\n                'src_key_padding_mask': [False] + [True] * (MAX_SEQ-1),\n                # \"timeliness\": [timeliness] + [TIMELINESS_PAD] * (MAX_SEQ-1),\n                'answered_correctly': [3.] * MAX_SEQ, # empty basically.\n                \"lag\": [0] + [0] * (MAX_SEQ-1), # don't start WITH 0 ???????\n                'len': 1,\n            }\n            \n            # add it as well\n            _content_id.append(STATE[user_id]['content_id']) # should have start token ?\n            _part_id.append(STATE[user_id]['part_id'])\n            _prior_question_elapsed_time.append(STATE[user_id]['prior_question_elapsed_time'])\n            _src_key_padding_mask.append(STATE[user_id]['src_key_padding_mask'])\n            _lag.append(STATE[user_id]['lag'])\n            _answered_correctly.append(STATE[user_id]['answered_correctly'])\n            \n            # reset it\n            STATE[user_id]['answered_correctly'] = []\n            last_seen_timestamp[user_id] = {'timestamp' : 0}\n            # print(\"NEW ENTRY FOR\", user_id, STATE[user_id])\n    \n    with torch.no_grad():\n        _content_id = torch.Tensor(_content_id).to(DEVICE).long()\n        _part_id = torch.Tensor(_part_id).to(DEVICE).long()\n        _prior_question_elapsed_time = torch.Tensor(_prior_question_elapsed_time).to(DEVICE).long()\n        _src_key_padding_mask = torch.Tensor(_src_key_padding_mask).to(DEVICE).bool()\n        _answered_correctly = torch.Tensor(_answered_correctly).to(DEVICE)\n        _lag = torch.Tensor(_lag).to(DEVICE).long()\n\n        test_df['answered_correctly'] = model(\n                content_id=_content_id, part_id=_part_id, \n                prior_question_elapsed_time=_prior_question_elapsed_time, \n                src_key_padding_mask=_src_key_padding_mask,\n                lag=_lag,\n                answered_correctly=_answered_correctly,\n                test=True,\n        ).cpu()\n\n        env.predict(test_df.loc[:, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(STATE[275030867])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_content_id","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_part_id","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = _content_id == 0\nmask.masked_fill(mask, -np.inf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df[[\"row_id\", \"answered_correctly\"]].values","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}