{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Brief comments\n\nJust copying the discription I wrote on the discussion topic. It's more readable there [link](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/209711)\n\n\nThanks for all the discussion and support from fellow Kagglers. I've learned a lot in the competition and tried a lot of things. Here a simple list of the stuff that I believe worked well. I'll try to cover the whole model but may make updates to clarify and add explanations later.\n\n\n**Architecture**\nMy architecture was just 3 Encoder modules stacked together using 512 as d_model and 4 Encoder layers in each block.\n\nMy Questions, Interactions and Response sequences were all padded on the left by a unique vector made using historical stats from the user. This ensured that all 3 sequences were aligned on the sequence number\n\n1. Questions Only Block (Q-Block):\n This one was just self attention over the questions.\n\n1. Interactions Only Block (I-Block):\n This one was just self attention over the interactions.\n\n1. Questions, Responses, Interaction Block (QRI Block):\n I used output from Q-Block as query, I-Block as keys and Responses/(QRI Block) as values. The residual connection after attention module was made using value vector (instead of the default query vector). Self Attention Mask was used in first 3 layers and for the last layer I used a custom mask that only attended to prior interactions and didn't have the residual connection.\n\nConcatenated the output of Q-Block and QRI-Block, feed it into 3 Linear layers.\n\n**Data Usage**\n\nI split the longer sequences into windows of 256 length and 128 overlap. (0-256, 128-384, 256-512).\nI saved them in tf.records and during inference I took a uniform random 128 window from each 256 length sequences. This ensures any event would be placed at 0-127 location in my training sequence with equal probability (excluding the last 128 events in the last window of a user). For users having smaller sequence length I padded with 0 to make the length equal 128 and used those 128 each time.\n\n**Compute Resources**\n\nI initially only used Kaggle GPUs to train the model. Started setting up a TPU pipeline in the last 3 weeks of the competition and in the end was able to use the TPUs as well. I think I've exhausted my complete TPU quota for last 3 weeks.\n\nI'd like to thank @yihdarshieh (TPU Guru) for this TPU notebook. It helped a lot in setting up the TPU pipeline.\n\n**Sequential Encodings**\n\nI used 2 encodings that captured the sequence of events. I subtracted the first value in each sequence from the rest to ensure that all encodings started with a 0.\n\n**Temporal Encoding**\n\nFor this I converted the timestamp into minutes and used the same scheme as that of positional encoding with power (60 x 24 x 365 = 1 year). I believe this enabled the model to know how far apart in time were 2 questions/interactions. I believe this to be a better implementation of the lag time variable used in SAINT+ as it captures the difference in time between all of the events simultaneously rather than just between 2 adjacent events.\n\n**Positional Encoding**\n\nFor this I used the task_container_id as the position and a power of 10,000.\n\n**Proxy for Knowledge**\n\nInstead of just using 0/1 from responses I added an new feature which is a heuristic for knowledge in case the response was incorrect. If a user selects option 2 when 1 is the correct one. I would calculate (total number of times 2 was selected for that questions)/(total number of times the question has been answered incorrectly). This ratio was maintained and updated during inference as well. I checked the correlation of mean of proxy_knowledge for incorrect answers and overall user accuracy, it was around 0.4.\n\n**Question difficulty**\n\nSimple feature that is calculated as (total number of times the question has been answered correctly)/(total number of times the question has been answered). This ratio was maintained and updated during inference as well.\n\n**Custom Masks**\n\nI used self attention mask that didn't attend on events of the same bundle.\nFor the last layer of QRI block I removed the mask entries along the diagonal to ensure it only attends to prior values.\n\n**Starting Vector**\n\nFor the starting vector I used counts of the time each tag was seen/answered correctly in questions and lectures. Just used a dense layer to encode this into the first vector of the sequence.\n\n**Other Details**\n\n* Used Noam LR provided on the Transformers page on TF documentation.\n* Batch size 1024 (Trained on TPUs - 1 Epoch took 4-6 mins)\n* For Validation I just separated around 3.4% of users initially and used their sequences.\n* Model converged around 20-30 Epochs. (4 Hours training time at max)\n* Sequence length of 128 was used.\n\n**Question Embeddings**\n\nI added up embeddings for content id, part id, (a weighted average of) tags ids and type_of (from lectures), then concatenated it with both Sequential embeddings and then into a dense layer with d_model dimensions.\n\n**Response Embeddings**\n\nI concatenated answered correctly, proxy knowledge, question difficulty, time elapsed and question had explanation and feed into a dense layer with d_model dimensions.\n\n**Interaction Embeddings**\n\nI just took the first d_model//2 units from both Questions/Response embeddings and concatenated them for this."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"max_sequence_length = 128\n\nd_model = 512\nembedding_dims = 512\nmini_d = embedding_dims//4\nnum_layers = 4\nnum_heads = 8\n\ntemporal_encoder_power = 60*24*360\ntask_encoder_power = 10000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for ii in {\"max_sequence_length\":max_sequence_length,\n\"d_model\":d_model,\n\"embedding_dims\":embedding_dims,\n\"mini_d\":mini_d,\n\"num_layers\":num_layers,\n\"num_heads\":num_heads,\n\"temporal_encoder_power\":temporal_encoder_power,\n\"task_encoder_power\":task_encoder_power}.items():\n print(ii)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"len_validation = 31306\nlen_train = 904649","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.utils import Sequence\nimport random\nfrom sklearn import preprocessing\nfrom sklearn.metrics import roc_auc_score\nfrom collections import defaultdict \nfrom itertools import chain\n\n%load_ext tensorboard\n\nimport joblib\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport gc\nimport sys\nfrom tqdm.notebook import tqdm\nfrom IPython.display import SVG","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(tf.__version__)\nprint(tfa.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\ntf.config.set_soft_device_placement(True)\nREPLICAS = strategy.num_replicas_in_sync\nprint(\"REPLICAS: \", REPLICAS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_description_parse = {\n \"e_timestamp\" : tf.io.RaggedFeature(value_key=\"e_timestamp\", dtype=tf.int64),\n \"e_content_id\" : tf.io.RaggedFeature(value_key=\"e_content_id\", dtype=tf.int64),\n \"e_content_type_id\" : tf.io.RaggedFeature(value_key=\"e_content_type_id\", dtype=tf.int64),\n \"r_question_elapsed_time\" : tf.io.RaggedFeature(value_key=\"r_question_elapsed_time\", dtype=tf.int64),\n \"r_question_had_explanation\" : tf.io.RaggedFeature(value_key=\"r_question_had_explanation\", dtype=tf.int64),\n \"e_task_container_id\" : tf.io.RaggedFeature(value_key=\"e_task_container_id\", dtype=tf.int64),\n \"e_part\" : tf.io.RaggedFeature(value_key=\"e_part\", dtype=tf.int64),\n \"e_tags\" : tf.io.RaggedFeature(value_key=\"e_tags\", dtype=tf.int64),\n \"e_type_of\" : tf.io.RaggedFeature(value_key=\"e_type_of\", dtype=tf.int64),\n \"r_answer\" : tf.io.RaggedFeature(value_key=\"r_answer\", dtype=tf.int64),\n \"r_q_difficulty\" : tf.io.RaggedFeature(value_key=\"r_q_difficulty\", dtype=tf.float32),\n \"r_proxy_knowledge\" : tf.io.RaggedFeature(value_key=\"r_proxy_knowledge\", dtype=tf.float32),\n \"h_tags_counts\" : tf.io.FixedLenFeature([1,188], dtype=tf.int64),\n \"h_tags_correct\" : tf.io.FixedLenFeature([1,188], dtype=tf.int64),\n \"h_lecture_counts\" : tf.io.FixedLenFeature([1,188], dtype=tf.int64),\n \"h_timestamp_sums\" : tf.io.FixedLenFeature([1,1], dtype=tf.float32),\n \"h_parts_counts\" : tf.io.FixedLenFeature([1,1], dtype=tf.int64),\n \"user\": tf.io.FixedLenFeature([], dtype=tf.int64),\n \"len_seq\" : tf.io.FixedLenFeature([], dtype=tf.int64),\n \"mask\": tf.io.RaggedFeature(value_key=\"mask\", dtype=tf.int64)\n}\n\n@tf.function\ndef _parse_function_ragged(example_proto):\n _parsed = tf.io.parse_single_example(example_proto, feature_description_parse)\n \n parsed = {}\n len_seq = _parsed['len_seq']\n max_len = tf.constant(128, dtype=tf.dtypes.int64)\n\n if len_seq > max_len:\n start = tf.random.uniform(shape=[], minval=0, maxval=len_seq-max_len, dtype=tf.dtypes.int64)\n else:\n start = tf.constant(0, dtype=tf.dtypes.int64)\n \n len_seq = tf.reduce_sum(tf.ones_like(_parsed['r_answer'], dtype=tf.int64))\n \n for values in feature_description_parse:\n if values in ['len_seq','user',\"h_tags_counts\",\"h_tags_correct\",\"h_lecture_counts\",\"h_timestamp_sums\",\"h_parts_counts\"]:\n parsed[values] = _parsed[values]\n else: \n if values=='e_tags':\n parsed[values] = tf.reshape(_parsed[values][6*start:6*(start+max_len)],shape=(max_len,tf.constant(6, dtype=tf.dtypes.int64)))\n else:\n parsed[values] = tf.reshape(_parsed[values][start:start+max_len],shape=(max_len,))\n \n return parsed,parsed['r_answer'][...,tf.newaxis],parsed['mask']\n\ndef read_process_tfrecords(filenames):\n raw_dataset = tf.data.TFRecordDataset(filenames)\n parsed_dataset = raw_dataset.map(_parse_function_ragged)\n return parsed_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Set the GCP path\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('riiid-random-sequences') \n\nlocal_path_data = '../input/riiid-random-sequences'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = read_process_tfrecords([GCS_DS_PATH+\"/\"+ii for ii in os.listdir(local_path_data) if 'train' in ii])\nvalidation_dataset = read_process_tfrecords([GCS_DS_PATH+\"/\"+'validation_data.tfrecord'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Custom_Mask(tf.keras.layers.Layer):\n def __init__(self, histories = 1, max_sequence_length = max_sequence_length, **kwargs):\n super(Custom_Mask, self).__init__(**kwargs)\n \n self.histories = histories\n self.max_sequence_length = max_sequence_length\n len_mask = self.max_sequence_length+self.histories\n \n self.triu = (tf.keras.backend.cast(tf.linalg.band_part(tf.keras.backend.ones((self.max_sequence_length, self.max_sequence_length)), -1, 0),tf.bool)[tf.newaxis,...])\n self.ones = tf.eye(len_mask, num_columns=None, batch_shape=None, dtype=tf.dtypes.bool)[tf.newaxis,...]\n self.hist_mask = tf.keras.backend.cast(tf.keras.backend.ones(shape=(1,self.histories,self.histories)),tf.bool)\n \n def get_config(self):\n config = super(Custom_Mask, self).get_config()\n config.update({\"histories\": self.histories,\n \"max_sequence_length\":self.max_sequence_length})\n return config\n \n \n \n def call(self,inputs):\n batch_size = tf.math.reduce_sum(tf.ones_like(inputs[:, :1], dtype=tf.int32))\n len_seq = tf.math.reduce_sum(tf.ones_like(inputs[:1, :], dtype=tf.int32))\n x = tf.broadcast_to(inputs[:, :, tf.newaxis], shape=[batch_size, len_seq, len_seq])\n y = tf.broadcast_to(inputs[:, tf.newaxis, :], shape=[batch_size, len_seq, len_seq])\n mask_bundles = tf.not_equal(x,y)\n \n bundles_pad = tf.logical_and(mask_bundles,self.triu[:,:len_seq,:len_seq])\n \n bundles_pad = tf.pad(bundles_pad, [[0,0],[1,0],[0,0]], mode='CONSTANT', constant_values=False)\n bundles_pad = tf.pad(bundles_pad, [[0,0],[0,0],[1,0]], mode='CONSTANT', constant_values=True)\n \n response_mask = bundles_pad\n \n event_mask = tf.logical_or(response_mask,self.ones[:,:len_seq+1,:len_seq+1])\n mask_bundles = tf.logical_not(mask_bundles)\n\n return event_mask,response_mask,mask_bundles\n\n def compute_output_shape(self, input_shape):\n s = input_shape\n h = self.histories\n if s[-1] is not None:\n with_hist = s[0],s[1]+h,s[1]+h\n without_hist = s[0],s[1],s[1]\n else:\n with_hist = None,None,None\n without_hist = with_hist\n return (with_hist),(with_hist),(without_hist)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Temporal_Embedding(tf.keras.layers.Layer):\n\n def __init__(self, temporal_vector_dimensions,temporal_encoder_power, **kwargs):\n \n super(Temporal_Embedding, self).__init__(**kwargs)\n\n \n self.temporal_vector_dimensions = temporal_vector_dimensions\n self.temporal_encoder_power = temporal_encoder_power\n \n i_vector = tf.keras.backend.arange(self.temporal_vector_dimensions//2)\n\n self.factor = tf.cast(1 / (self.temporal_encoder_power ** (2*i_vector/self.temporal_vector_dimensions)), dtype=tf.float32)\n \n \n def get_config(self):\n config = super(Temporal_Embedding, self).get_config()\n config.update({\"temporal_encoder_power\": self.temporal_encoder_power,\n \"temporal_vector_dimensions\":self.temporal_vector_dimensions})\n return config\n \n \n def get_sin(self, x):\n return tf.keras.backend.sin(x)\n \n def get_cos(self, x):\n return tf.keras.backend.cos(x)\n \n def rescale(self, x):\n return tf.cast(x, dtype=tf.float32) * self.factor\n \n\n def call(self, x):\n pos_vector = tf.keras.layers.Lambda(self.rescale)(x)\n pos_vector_sin = tf.keras.layers.Lambda(self.get_sin)(pos_vector)\n pos_vector_cos = tf.keras.layers.Lambda(self.get_cos)(pos_vector)\n pos_vector = tf.keras.layers.concatenate([pos_vector_sin,\n pos_vector_cos], axis=-1)\n return pos_vector\n\n\nclass Masked_Sum(tf.keras.layers.Layer):\n def __init__(self,**kwargs):\n super(Masked_Sum, self).__init__(**kwargs)\n \n def get_config(self):\n config = super(Masked_Sum, self).get_config()\n return config\n \n def call(self, x):\n x, mask = x\n mask = tf.cast(mask, x.dtype)[...,tf.newaxis]\n x = tf.keras.layers.Multiply()([x,mask])\n x = tf.keras.backend.sum(x,axis=-2)\n return x\n \n \n \nclass Elem_Divide(tf.keras.layers.Layer):\n def __init__(self,**kwargs):\n super(Elem_Divide, self).__init__(**kwargs)\n self.e = tf.constant([1e-9],dtype=tf.float32)\n \n def get_config(self):\n config = super(Elem_Divide, self).get_config()\n return config\n \n def call(self,x):\n x,y = x\n x = tf.cast(x, tf.float32)\n y = tf.math.add(tf.cast(y, tf.float32),self.e)\n return tf.divide(x,y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class weighted_embs(tf.keras.layers.Layer):\n def __init__(self,**kwargs):\n super(weighted_embs, self).__init__(**kwargs)\n self.dense_layer = tf.keras.layers.Dense(1,use_bias=False)\n self.permute_layer = tf.keras.layers.Permute((1,3,2))\n \n def get_config(self):\n config = super(weighted_embs, self).get_config()\n return config\n \n def call(self, inputs):\n inputs, mask = inputs\n ones_like = tf.ones_like(inputs)\n ones_like = ones_like*tf.cast(mask,tf.float32)[...,tf.newaxis]\n ones_like = self.permute_layer(ones_like)\n ones_like = self.dense_layer(ones_like)\n inputs = self.permute_layer(inputs)\n inputs = self.dense_layer(inputs)\n out = tf.keras.backend.squeeze(inputs/(ones_like+1e-6),axis=-1)\n return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Hindsight(tf.keras.layers.Layer):\n\n def __init__(self,d_model,num_heads,dff,rate=0.1,add_on = 'q',**kwargs):\n super(Hindsight, self).__init__(**kwargs)\n\n\n self.d_model = d_model\n self.num_heads = num_heads\n self.dff = dff\n self.rate = rate\n self.add_on = add_on\n self.mha = tfa.layers.MultiHeadAttention(head_size = self.d_model//self.num_heads, num_heads=self.num_heads)\n self.dense1 = tf.keras.layers.Dense(self.dff, activation='relu') # (batch_size, seq_len, dff)\n self.dense2 = tf.keras.layers.Dense(self.d_model)\n\n self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n\n self.dropout1 = tf.keras.layers.Dropout(self.rate)\n self.dropout2 = tf.keras.layers.Dropout(self.rate)\n self.dropout3 = tf.keras.layers.Dropout(self.rate)\n\n \n \n def get_config(self):\n config = super(Hindsight, self).get_config()\n config.update({\"d_model\": self.d_model,\n \"num_heads\":self.num_heads,\n \"dff\" : self.dff,\n \"rate\":self.rate,\n 'add_on':self.add_on})\n return config\n \n \n def call(self, x):\n q, v, k, mask = x\n \n attn_output = self.mha([q, k, v], mask=mask) # (batch_size, input_seq_len, d_model)\n attn_output = self.dropout1(attn_output)\n \n if self.add_on == 'q':\n out1 = self.layernorm1(q + attn_output) # (batch_size, input_seq_len, d_model)\n \n elif self.add_on == 'v':\n out1 = self.layernorm1(v + attn_output) # (batch_size, input_seq_len, d_model)\n \n else:\n out1 = self.layernorm1(attn_output) # (batch_size, input_seq_len, d_model)\n \n dense1 = self.dense1(out1)\n dense_1_output = self.dropout2(dense1)\n\n dense2 = self.dense2(dense_1_output) # (batch_size, input_seq_len, d_model)\n dense_2_output = self.dropout3(dense2)\n\n out2 = self.layernorm2(out1 + dense_2_output) # (batch_size, input_seq_len, d_model)\n\n return out2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class drop_timestamp(tf.keras.layers.Layer):\n def __init__(self, rate,**kwargs):\n super(drop_timestamp, self).__init__(**kwargs)\n self.rate = rate\n self.dropout = tf.keras.layers.Dropout(self.rate,noise_shape=(None,None,1))\n \n def get_config(self):\n config = super(drop_timestamp, self).get_config()\n config.update({\"rate\": self.rate})\n return config\n \n \n def call(self, inputs):\n ones = tf.keras.layers.Lambda(lambda x:tf.ones_like(x))(inputs)\n ones = self.dropout(ones)\n mask = tf.keras.layers.Lambda(lambda x:tf.cast(tf.math.not_equal(x, 0),tf.float32))(ones)\n output = tf.keras.layers.Lambda(lambda x:tf.multiply(x[0],x[1]))([inputs,mask])\n return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_embedding_dimension = 189\n\ncontent_id_embedding_dimension = 13942 # len(DM.encoder_content_id.classes_)+1\n\npart_id_embedding_dimension = 8 #DM.contents_df['part'].max()+1\ntype_of_embedding_dimension = 5 #DM.contents_df['type_of'].max()+1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generate_Model(num_layers=num_layers,\n d_model=d_model,\n num_heads=num_heads,\n dff=d_model*2,\n rate=0.1,\n embedding_dimensions = embedding_dims,\n seq_len = max_sequence_length):\n \n mini_d = embedding_dimensions//8\n \n e_timestamp = tf.keras.layers.Input(shape=(seq_len,1),name='e_timestamp')\n e_content_id = tf.keras.layers.Input(shape=(seq_len,),name='e_content_id')\n \n e_content_type_id = tf.keras.layers.Input(shape=(seq_len,),name='e_content_type_id') \n \n e_task_container_id = tf.keras.layers.Input(shape=(seq_len,),name='e_task_container_id')\n e_part = tf.keras.layers.Input(shape=(seq_len,),name='e_part')\n e_tags = tf.keras.layers.Input(shape=(seq_len,6),name='e_tags')\n e_type_of = tf.keras.layers.Input(shape=(seq_len,),name='e_type_of')\n \n r_answer = tf.keras.layers.Input(shape=(seq_len,1),name='r_answer')\n r_question_had_explanation = tf.keras.layers.Input(shape=(seq_len,1),name='r_question_had_explanation')\n r_question_elapsed_time = tf.keras.layers.Input(shape=(seq_len,1),name='r_question_elapsed_time')\n \n r_q_difficulty = tf.keras.layers.Input(shape=(seq_len,1),name='r_q_difficulty')\n r_proxy_knowledge = tf.keras.layers.Input(shape=(seq_len,1),name='r_proxy_knowledge')\n \n h_tags_counts = tf.keras.layers.Input(shape=(1,188),name='h_tags_counts')\n h_tags_correct = tf.keras.layers.Input(shape=(1,188),name='h_tags_correct')\n h_lecture_counts = tf.keras.layers.Input(shape=(1,188),name='h_lecture_counts')\n h_timestamp_sums = tf.keras.layers.Input(shape=(1,1),name='h_timestamp_sums')\n h_parts_counts = tf.keras.layers.Input(shape=(1,1),name='h_parts_counts') \n \n temporal_vector = tf.keras.layers.Lambda(lambda x:x-x[:,:1,:])(e_timestamp)\n temporal_vector = tf.keras.layers.Lambda(lambda x:x/600)(temporal_vector)\n temporal_vector = Temporal_Embedding(embedding_dimensions,temporal_encoder_power)(temporal_vector)\n \n \n task_id_encoding_layer = Temporal_Embedding(embedding_dimensions,task_encoder_power)\n task_id_encoding = tf.keras.layers.Lambda(lambda x:tf.keras.backend.expand_dims(x,axis=-1))(e_task_container_id)\n task_id_encoding = tf.keras.layers.Lambda(lambda x:x-x[:,:1,:])(task_id_encoding)\n task_id_encoding = task_id_encoding_layer(task_id_encoding)\n \n \n tags_embedding_layer = tf.keras.layers.Embedding(tag_embedding_dimension,embedding_dimensions,mask_zero=True)\n event_tags_embedding = tags_embedding_layer(e_tags)\n\n event_tags_embedding = weighted_embs()([event_tags_embedding,event_tags_embedding._keras_mask])\n \n event_content_id_embeddings = tf.keras.layers.Embedding(content_id_embedding_dimension,embedding_dimensions)(e_content_id)\n\n event_content_id_embeddings = drop_timestamp(rate=rate)(event_content_id_embeddings)\n \n \n event_content_type_id_embeddings = tf.keras.layers.Lambda(lambda x:tf.keras.backend.expand_dims(x,axis=-1))(e_content_type_id)\n event_content_type_id_embeddings = tf.keras.layers.Dense(embedding_dimensions)(event_content_type_id_embeddings)\n \n \n part_id_embeddings_layer = tf.keras.layers.Embedding(part_id_embedding_dimension,embedding_dimensions)\n event_part_id_embeddings = part_id_embeddings_layer(e_part)\n \n event_type_of_embeddings = tf.keras.layers.Embedding(type_of_embedding_dimension,embedding_dimensions)(e_type_of)\n \n response_answer = tf.keras.layers.Dense(embedding_dimensions)(r_answer)\n \n response_hint = tf.keras.layers.Dense(embedding_dimensions)(r_question_had_explanation)\n \n response_time = tf.keras.layers.Dense(mini_d,activation='relu')(r_question_elapsed_time)\n response_time = tf.keras.layers.Dropout(rate)(response_time)\n response_time = tf.keras.layers.Dense(embedding_dimensions)(response_time)\n \n question_difficulty = tf.keras.layers.Dense(mini_d,activation='relu')(r_q_difficulty)\n question_difficulty = tf.keras.layers.Dropout(rate)(question_difficulty)\n question_difficulty = tf.keras.layers.Dense(embedding_dimensions)(question_difficulty)\n \n proxy_knowledge = tf.keras.layers.Dense(mini_d,activation='relu')(r_proxy_knowledge)\n proxy_knowledge = tf.keras.layers.Dropout(rate)(proxy_knowledge)\n proxy_knowledge = tf.keras.layers.Dense(embedding_dimensions)(proxy_knowledge)\n \n event_content_id_embeddings = tf.keras.layers.Add()([event_content_id_embeddings,\n event_tags_embedding,\n event_part_id_embeddings,\n event_type_of_embeddings,\n question_difficulty,])\n \n \n event_stream = tf.keras.layers.Concatenate(axis=-1)([event_content_id_embeddings,\n temporal_vector,\n task_id_encoding])\n \n event_stream = tf.keras.layers.Dense(d_model)(event_stream)\n \n \n response_stream = tf.keras.layers.Add()([response_answer,\n response_hint,\n response_time,\n proxy_knowledge,\n event_content_type_id_embeddings]) \n\n histories_timestamp = tf.keras.layers.Lambda(lambda x:x/600)(h_timestamp_sums)\n histories_timestamp = Elem_Divide()([histories_timestamp,h_parts_counts])\n histories_temporal = Temporal_Embedding(embedding_dimensions,temporal_encoder_power)(histories_timestamp)\n \n \n histories_stream = tf.keras.layers.Concatenate(axis=-1)([h_tags_counts,\n h_lecture_counts])\n \n histories_stream = tf.keras.layers.GaussianNoise(stddev=0.5)(histories_stream)\n\n histories_stream = tf.keras.layers.Concatenate(axis=-1)([histories_temporal,\n histories_stream])\n \n\n histories_stream = tf.keras.layers.Dropout(rate)(histories_stream)\n histories_stream = tf.keras.layers.Dense(embedding_dimensions,activation='relu')(histories_stream)\n \n histories_stream = tf.keras.layers.Dropout(rate)(histories_stream) \n histories_stream = tf.keras.layers.Dense(d_model)(histories_stream)\n \n histories_response_stream = tf.keras.layers.Concatenate(axis=-2)([h_tags_correct,\n h_tags_counts])\n \n histories_response_stream = tf.keras.layers.GaussianNoise(stddev=0.5)(histories_response_stream)\n histories_response_stream = tf.keras.layers.Permute((2,1))(histories_response_stream)\n \n histories_response_stream = tf.keras.layers.Dense(mini_d,activation='relu')(histories_response_stream)\n histories_response_stream = tf.keras.layers.Dropout(rate)(histories_response_stream)\n \n histories_response_stream = tf.keras.layers.Dense(mini_d,activation='relu')(histories_response_stream)\n histories_response_stream = tf.keras.layers.Dropout(rate)(histories_response_stream)\n \n histories_response_stream = tf.keras.layers.Dense(1,activation='relu')(histories_response_stream)\n histories_response_stream = tf.keras.layers.Permute((2,1))(histories_response_stream)\n \n histories_response_stream = tf.keras.layers.Dropout(rate)(histories_response_stream)\n histories_response_stream = tf.keras.layers.Dense(embedding_dimensions,activation='relu')(histories_response_stream)\n \n histories_response_stream = tf.keras.layers.Dropout(rate)(histories_response_stream)\n histories_response_stream = tf.keras.layers.Dense(d_model)(histories_response_stream)\n \n\n queries_final = tf.keras.layers.Concatenate(axis=1,name='events_with_hist')([histories_stream,event_stream])\n keys_final_0 = tf.keras.layers.Lambda(lambda x:x[...,:d_model//2])(queries_final)\n queries_final = tf.keras.layers.LayerNormalization(epsilon=1e-6)(queries_final)\n queries_final = tf.keras.layers.Dropout(rate)(queries_final)\n \n values_final = tf.keras.layers.Concatenate(axis=1,name='responses_with_hist')([histories_response_stream,response_stream])\n keys_final_1 = tf.keras.layers.Lambda(lambda x:x[...,:d_model//2])(values_final)\n values_final = tf.keras.layers.LayerNormalization(epsilon=1e-6)(values_final) \n values_final = tf.keras.layers.Dropout(rate)(values_final)\n\n \n keys_final = tf.keras.layers.Concatenate(axis=-1)([keys_final_0,\n keys_final_1])\n\n keys_final = tf.keras.layers.Dense(d_model)(keys_final)\n keys_final = tf.keras.layers.LayerNormalization(epsilon=1e-6)(keys_final)\n keys_final = tf.keras.layers.Dropout(rate)(keys_final)\n\n mask = Custom_Mask(histories = 1, max_sequence_length = max_sequence_length)(e_task_container_id)\n\n for ii in range(num_layers):\n keys_final = Hindsight(d_model, num_heads, dff, rate=rate, add_on ='q',name=f'KKK_K_{ii+1}')([keys_final, keys_final, keys_final, mask[0]])\n \n keys_final = drop_timestamp(rate=rate)(queries_final)\n \n for ii in range(num_layers):\n queries_final = Hindsight(d_model, num_heads, dff, rate=rate, add_on ='q',name=f'QQQ_Q_{ii+1}')([queries_final, queries_final, queries_final, mask[0]])\n\n for ii in range(num_layers): \n if ii