{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NLP Course- Transformers With Quora Binary Classification\n\n\nThis is an extension of [Notebook-1](https://www.kaggle.com/abhilash1910/nlp-workshop-ml-india/edit) where we will be focussing on extensive architectures from the transformer family. We will be using variants of the BERT transformer after understanding the fundamental building blocks of transformers and attention mechanisms.\n\nFirstly, we will be using transformers made with the help of the [HuggingFace Repository](https://huggingface.co/) specifically the [DistillBert](https://huggingface.co/transformers/model_doc/distilbert.html) transformer and then understand the inner layers inside the transformer. We will also be exploring the attention mechanisms- Self,Scaled Dot Product,Hierarchical- and other variants.\n\nThough we will be focussing on distillbert, we can also use any other versions as well. This is the [paper](https://paperswithcode.com/paper/attention-is-all-you-need) which started the revolution in NLP space related to attentions and transformers. This [repository](https://github.com/Lsdefine/attention-is-all-you-need-keras/blob/master/transformer.py) contains the original implementation of the transformer with scaled dot product attention.\n\n[hierarchical attention](https://github.com/abhilash1910/MiniAttention) this can be used for Hierarchical Attention.\n\n[Keras Example](https://keras.io/examples/nlp/text_classification_with_transformer/) also provides a good information for creating a custom model.\n\nThe most important architecture which remains the same throughout the Transformers architecture is :\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Some useful resources \n\nThese are certain important resources and kernels which may help in understanding the codes of attention mechanisms with Keras:\n\n1. [Transformer in Keras](https://github.com/Lsdefine/attention-is-all-you-need-keras/blob/master/transformer.py)\n2. [Good Kernel](https://www.kaggle.com/suicaokhoailang/lstm-attention-baseline-0-652-lb)\n3. [Another Kernel](https://www.kaggle.com/shujian/transformer-initial-attempt)\n\nThese kernels are written well and they provide a good outline as to how the attention model is built internally. Through we will be discussin on this.\n"},{"metadata":{},"cell_type":"markdown","source":"## DistilBert \n\nThis contains the architecture of the distilbert transformer model:\n\n\n\nIn this case, the distilbert performs better than Bert in most cases owing to continuous feedback of attention weights from the teacher to the student network. Where the weights change by a large extent in case of Bert, this fails to happen in DistilBert."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating DistillBert with Transformers\n\nWe will be creating the DistillBert transformer and then training it with our own corpus.\nThen we will validate the results with our initial benchmarks from last kernel.\n\nFollow this [link](https://huggingface.co/models) for testing any pre-trained transformer model."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\nfrom tqdm.notebook import tqdm\nfrom tokenizers import BertWordPieceTokenizer\nfrom sklearn.model_selection import train_test_split\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tokenize the Data (Fast Tokenize)\n\nThis will assist in the model tuning stage. We have to divide the data in chunks so that it is simpler for the model to absorb the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Tokenize the data and separate them in chunks of 256 units\n\nmaxlen=512\nchunk_size=256\ndef fast_encode(texts, tokenizer, chunk_size=chunk_size, maxlen=maxlen):\n tokenizer.enable_truncation(max_length=maxlen)\n tokenizer.enable_padding(max_length=maxlen)\n all_ids = []\n #sliding window methodology\n for i in tqdm(range(0, len(texts), chunk_size)):\n text_chunk = texts[i:i+chunk_size].tolist()\n encs = tokenizer.encode_batch(text_chunk)\n all_ids.extend([enc.ids for enc in encs])\n \n return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Building With DistillBert\n\nIn this context, we have to build the model. We are creating a function which would help us to do that .\n\nWe will be using the [Model](https://www.kaggle.com/abhilash1910/nlp-workshop-ml-india/) API from Keras which was mentioned in the session-1, and addionally we will be using the same additional layers as Dense. The activations (sigmoid) also remain pretty much the same."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create the model\n\ndef build_model(transformer, max_len=512):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n #Replaced from the Embedding+LSTM/CoNN layers\n sequence_output = transformer(input_word_ids)[0]\n cls_token = sequence_output[:, 0, :]\n out = Dense(1, activation='sigmoid')(cls_token)\n \n model = Model(inputs=input_word_ids, outputs=out)\n model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n model.summary()\n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Cluster Check\n\nIn this context, we will be using the TPU cluster from the Notebook (Hardware accelerations). TPUs provide a better performance with respect to Tensorflow and Keras computations on tensors against GPUs.But it has to be explicitly called out in the code segment.\n\n[Kaggle Documentation on TPUs](https://www.kaggle.com/docs/tpu) provide an excellent starting point for this.Highly recommend to go through it.\n\nSteps to check and run the TPU cluster:\n\n### detect and init the TPU\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(tpu)\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\n### instantiate a distribution strategy\ntpu_strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n### instantiating the model in the strategy scope creates the model on the TPU\nwith tpu_strategy.scope():\n model = tf.keras.Sequential( … ) # define your model normally\n model.compile( … )\n\n### train model normally\nmodel.fit(training_dataset, epochs=EPOCHS, steps_per_epoch=…)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Detect and deploy\n\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is\n # set: this is always the case on Kaggle.\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Replicas\n\nThe replicas assist in segregating the data in sync, by allowing a faster batch sampling and an equal partition of the set, amongst the different replicas. In this context, it will be partitioned in blocks amongst 8 TPU clusters."},{"metadata":{"trusted":true},"cell_type":"code","source":"#allow experimental tf\nAUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n# Configuration of hyperparameters\nEPOCHS = 3\n#batch size denotes the partitioning amongst the cluster replicas.\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the Tokenizer from DistillBert\n\nIn this case, we will be using the [DistilBertTokenizer](https://huggingface.co/transformers/v2.11.0/model_doc/distilbert.html). This will help us to tokenize the data from 'distillbert-base-multilingual-cased' pre-trained tokenized model.\nIt is recommended to go through the different tokenizers present in [HuggingFace](https://huggingface.co/transformers/main_classes/tokenizer.html)\n\nThe two most important ones for generic language modelling:\n\n1. ByteLevelBPETokenizer\n2. BertWordPieceTokenizer\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_set['question_text'].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['question_text']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tokenize the text samples\n\nIn this case, we apply the fast Tokenizer from Distillbert on the samples."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Datasets \n\nAn important aspect of finetuning Bert variants of transformers is to use a proper tensor slicing mechanism for splitting the training and validation sets. This allows the algorithm to understand the different datasets. This also allows the DistillBert algorithm to download and train the data in the form of \"tf.data.Dataset\", which effectively implies that we have converted the dataset to be compatible with tensorflow datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Build the transformer model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFDistilBertModel\n .from_pretrained('distilbert-base-multilingual-cased')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Parameters in DistillBert\n\nAs it is visible the number of parameters have greatly increased and now it has approx. 10.34 million parameters (1 crore) to train. This increase in the number of trainable parameters make the transformer model less susceptible to catastrophic forgetting , which is prevalent in Sequential networks. Moreover, transformers have attention mechanisms which allow the neural network to retrieve certain important weights from the training network. This form of learning is very commonly known as seq2seq modelling (a subset of transfer learning) deviced by Illya Sutskever from OpenAI. Some important resources in seq2seq learning is as follows:\n\n1. [Sutskever's Paper](https://paperswithcode.com/paper/sequence-to-sequence-learning-with-neural)\n2. [DistillBert Paper](https://paperswithcode.com/paper/distilbert-a-distilled-version-of-bert)\n3. [Representational Learning](https://paperswithcode.com/task/representation-learning)\n4. [Transfer Learning](https://paperswithcode.com/task/transfer-learning)\n\nThese resources are for scientific papers which would definitely assist in understanding the core concepts of transfer learning, seq2seq modelling."},{"metadata":{},"cell_type":"markdown","source":"## Train the Transformer\n\nIn this case, we train the DistillBert transformer with our corpus."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validating the Model\n\nAfter saturation of the training metrics(accuracy), we have to validate the model against the validation /testing set. This is similar to the code written above."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_steps = valid_x.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n valid_dataset.repeat(),\n steps_per_epoch=n_steps,\n epochs=EPOCHS*2\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Outline for training with any Transformer\n\nThe following is the outline for training any transformer(HuggingFace) with the Kaggle corpus:\n\n1. Create a function for Fast Tokenization. Provide the maxlen for truncating and chunk size.\n2. Create a function for the Transformer Model. Notice there are no additional Embedding/LSTM layers attached in the model.\n3. Detect and check for TPUs - this will greatly boost the training period for us.Tensor Processing Units are specifically designed for super fast computing of the Tensors (in Tensorflow).\n4. Create Replicas to split the TPU cluster for better performance - recommended.\n5. Load the training and validation datasets into the Fast Tokenization function to encode it with any Transformer (in our case DistillBert). Notice BertWordPieceTokenizer- this is used with any variant of Bert Transformer.\n6. After Tokenization, make the dataset compatible with Tensorflow datasets- (tensorflow.data.Datasets). This gives performance boost on TPU.\n7. Load the Transformer model from The HuggingFace Repository. The code for this should be of the same pattern for any Transformer.\n8. Train the model with the parameters.\n9. Validate it against the validation set.\n\nThis pipeline follows for any classification task with Transformers. For categorical classification, only the change is required in the model building (function) stage - the last Dense layer should have a softmax activation instead of sigmoid. Other than this, no changes are required.\n\nThe list of pre-trained models from the HuggingFace repository can be found here:\n\n1. [Pre-trained Model Names](https://huggingface.co/transformers/pretrained_models.html)"},{"metadata":{},"cell_type":"markdown","source":"## Analysing a different Transformer\n\nNow that we have trained and validated against DistillBert, let us try another Transformer. In this case, we will be using the XLM- Roberta Transformer. Important details on this model can be found here:\n\n1. [XLM-Roberta](https://huggingface.co/transformers/model_doc/xlmroberta.html)\n2. [Paper](https://arxiv.org/abs/1911.02116)\n\nThis is entirely based on the Roberta Transformer by Facebook,and some of the details of that model is present here:\n\n1. [Roberta](https://huggingface.co/transformers/model_doc/roberta.html)\n2. [Paper](https://huggingface.co/transformers/model_doc/roberta.html)\n\nApart from this, there are several in the list which was provided in the tab above. Anyone can be tried out and the performance can be validated against.\n\nFor doing this, we follow the same instructions provided above. Since in this case, we already have done till step 4. \nFrom step 5 we have to change the tokenizer and the model.\n\n The architecture for Roberta (original) is provided below:\n \n "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer\ntokenizer = transformers.XLMRobertaTokenizer.from_pretrained('xlm-roberta-base')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the samples\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Load into Tensorflow compatible datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFRobertaModel\n .from_pretrained('roberta-base')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the Transformer\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Albert- The lightweight Bert\n\nWe have seen 2 different transformers for our use case- DistillBert, XLM-Roberta (Roberta). Now let us see the performance of ALbert from Google Research. It is a lightweight , which uses splits and tensor decomposition of the embedding matrix. Most of the architecture is same as Bert, with the addition of repeated layers which reduces memory consumption. These are the resources for understanding Albert and Bert:\n\n1. [HuggingFace Albert](https://huggingface.co/transformers/model_doc/albert.html)\n2. [Albert Paper](https://arxiv.org/abs/1909.11942)\n\nThe original Bert paper and resources are present in these links:\n\n1. [Traditional BERT](https://huggingface.co/transformers/model_doc/bert.html)\n2. [Paper](https://arxiv.org/abs/1810.04805)\n\nIn this case, we will first use Bert Transformer and then analyse the performance using Albert.\nThis will allow us to see the speed up and the increase in performance in Albert from Bert. \n\nAs usual we will be focussing from Steps 5 to 9."},{"metadata":{},"cell_type":"markdown","source":"## Analysis with Bert Transformer\n\nFirst, let us build a composite Bert model for our use case.\nThe model architecture of Bert is shown from the paper :\n\n\n\n\nThe pathway from pre-training to fine-tuning is shown here:\n\n\n\n\n\nThe full block diagram of internal layers of the Bert model is shown here:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer- Bert\ntokenizer = transformers.BertTokenizer.from_pretrained('bert-base-multilingual-cased')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the datasets\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\nprint(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Create Tensorflow Datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFBertModel\n .from_pretrained('bert-base-multilingual-cased')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the model\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyse the Albert Model\n\nNow with the performance of Bert with us, we will be analysing the Albert model for our use case.\nFor this, we will use the Albert Model from HuggingFace Transformers"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer- Bert\ntokenizer = transformers.AlbertTokenizer.from_pretrained('albert-base-v1')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the datasets\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\nprint(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Create Tensorflow Datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFAlbertModel\n .from_pretrained('albert-base-v1')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the model\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some other Transformers\n\nThere are several transformers we can use for our use cases. We will be focussing on more such transformer architectures.\nThe first one is GPT (OpenAI) and the second is BART (Facebook).\n\nGPT(Generative Pre-training) is a SOTA for generative (transfer learning) based architecture which has achieved great heights with its new language models (GPT2/3). We will be analysing the performance of GPT-2 for our use case. Some important resources on GPT-2:\n\n1. [GPT-2](https://openai.com/blog/better-language-models/)\n2. [HuggingFace GPT-2](https://huggingface.co/transformers/model_doc/gpt2.html)\n\nWe will also be looking into BART, which is another improvisation by Facebook,done by using Bert's left to right encoder and GPT's right to left decoder. Some resources on this can be found at:\n\n1. [BART](https://huggingface.co/transformers/model_doc/bart.html)\n2. [Paper](https://arxiv.org/abs/1910.13461)\n\nFor our usecases, we will just follow the steps from before and build these models on our corpuses. \nSince BART is incompatible with Tensorflow yet, we can try it using Pytorch.\nAnother Transformer architecture which is important is TransformerXL:\n\n1. [TransformerXL](https://huggingface.co/transformers/model_doc/transformerxl.html)\n2. [Paper](https://arxiv.org/abs/1901.02860)\n\nThis covers most of the famous architectures in the Transformer space. A comparative analysis between Bert and Gpt model can be found here:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## GPT-2 Transformer\n\nHere we analyse the GPT-2 Transformer for our use case.\n\nThe pathway for self attention in GPT-2 is shown here:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer- GPT-2\ntokenizer = transformers.GPT2Tokenizer.from_pretrained('gpt2-medium')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the datasets\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\nprint(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Create Tensorflow Datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFGPT2Model\n .from_pretrained('gpt2-medium')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the model\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyse the BART Transformer\n\nHere we will now analyse the BART Transformer and check its performance with respect to other transformers.\nHowever BART from HuggingFace is not compatible with Tensorflow yet. We should use Pytorch as our framework for training BART.\n\nA preview is shown here:\n\n\n\n\nBart uses pretrained encoders to evaluate against the current encodings and validates them before passing in to the transformer network."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer- BART\ntokenizer = transformers.BartTokenizer.from_pretrained('facebook/bart-base')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the datasets\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\nprint(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Create Tensorflow Datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bart (HuggingFace) is incompatible with Tensorflow (for now!)\n\nSo for this, we just need to plug in the Bart tokenizer with any pre-trained model, for instance Albert to form a composite architecture. This is sometimes refered to as an ensemble Transformer architecture."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model- Bart tokenizer + ALbert model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFAlbertModel\n .from_pretrained('albert-base-v1')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the model\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Transformer XL Architecture\n\nThis is another transformer and should be analysed as well. All of the transformers covered are SOTA in their respective tasks and anyone can be used based on certain use cases.\n\nA good analysis between GPT/BERT/Transformer-XL is provided here:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n# First load the real tokenizer- BART\ntokenizer = transformers.TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')\n# Save the loaded tokenizer locally\ntokenizer.save_pretrained('.')\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('vocab.txt', lowercase=False)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 5.\n#Tokenize the datasets\ntrain_x = fast_encode(train_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\nval_x = fast_encode(test_set['question_text'].astype(str), fast_tokenizer, maxlen=MAX_LEN)\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\nprint(train_x.shape)\nprint(train_y.shape)\nprint(val_x.shape)\nprint(val_y.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 6.\n#Create Tensorflow Datasets\ntrain_dataset = (\n tf.data.Dataset\n .from_tensor_slices((train_x, train_y))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((val_x, val_y))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\nprint(train_dataset)\nprint(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 7.\n#Build the transformer model- Bart tokenizer + ALbert model\nwith strategy.scope():\n transformer_layer = (\n transformers.TFTransfoXLModel\n .from_pretrained('transfo-xl-wt103')\n )\n model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 8.\n#Train the model\n\nn_steps = train_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Step 9.\n#Validate the model\n\nn_steps = val_x.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n train_dataset,\n steps_per_epoch=n_steps,\n validation_data=valid_dataset,\n epochs=EPOCHS\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion for Transformers For Industrial Use case\n\nThis concludes the several SOTA Transformers for our usecase and this can also be used for any classification tasks as well.\nIn the next section, let us build a custom transformer to understand the architectures in depth along with the Attention mechanism\n\nThis image shows the variation of Transformer architectures for different NLP tasks from classification to question anwering:\n\n\n\nPlease follow the excellent resources:\n\n1. [Transfer Learning in Transformers](http://jalammar.github.io/illustrated-bert/)\n2. [GPT-2 Enhanced Architecuture](http://jalammar.github.io/illustrated-gpt2/)\n3. [Current SOTA GPT-3](http://jalammar.github.io/how-gpt3-works-visualizations-animations/)\n"},{"metadata":{},"cell_type":"markdown","source":"# Creating a Custom Transformer\n\nThis section will help us in understanding the concepts required for creating a custom transformer.\nThe most important paper for this is present in the [link](https://paperswithcode.com/paper/attention-is-all-you-need) . It was written by Vaswani etal from Google research and is the paper which started the transformer journey.\n\n\n\nTransformers alleviate the use of Convolution or Recurrent Networks by replacing them with attention mechanisms. This is because of the need for parallelization in training as well as to remove recurrence units. Seq2seq transduction models like the ones shown yesterday rely on something called as a Encoder-Decoder architecture. \n\nThe encoder architecture consists of stacked LSTM cells which control and give access to the memory for storing latent features. The decoder is responsible for extracting the outputs from the topmost stacked LSTM cell. While this is a good strategy over traditional models, it gets affected by catastrophic forgetting. \n\nSome excellent sources for understanding seq2seq encoder-decoder model are:\n\n1. [Jason's blog](https://machinelearningmastery.com/develop-encoder-decoder-model-sequence-sequence-prediction-keras/)\n2. [Francois Chollet's Code](https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py)\n\nFrom the encoder -decoder architecture arose the need for memory persistent attention units.\nThe first paper on attention was provided by Bahdanau for [Neural Machine Translation](https://paperswithcode.com/paper/neural-machine-translation-by-jointly).\n\n\nA comparative analysis between Bahdanau Attention and Luong Attention is provided here:\n\n\n\nNeural Machine Translation is an entire different area of research which also uses Encoder decoder architectures coupled with attention mechanisms. A diagram of how this happens is provided below:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Encoder Decoder Architectures\n\nThese architectures are inherent in every transformer. Both of them are effectively similar deep learning models which are used to storing and abstracting data mainly consisting of LSTM units. The block diagram is provided:\n\n\n\n\nThere are different embeddings which are present in both of them (encoder and decoder). The input to the encoder layer is tokenized and then passed in to the positional embeddings layer. Then these are passed into the Deep Learning layers (mostly CNN/LSTM) . Then these are passed through the Attention module which performs tensor computations to assign more weights to certain words in the input (encoded) sentence. These are then passed in to the decoder unit.\n\nDuring indference, the decoder uses the mapping of the attention weights of certain important words to get the correct probabilities of the output words. Essentially the activation used in totality is the softmax as it has been employed across every embedding algorithm ,which relies on joint probability distribution of a word with respect to other words in the sentence."},{"metadata":{},"cell_type":"markdown","source":"## Self Attention in Depth\n\nSelf Attention relies on certain vectors q(query),k(key) and v(value). The vectors q and k are multiplied together and divided by 8(according to the papers) and then passed through a normalized softmax activation unit. This is donw so as to retain the memory of the weights when they are transferred from the encoder to the decoder layer. A series of images help us understand this:\n\n\n\nThis is how the operation is executed:\n\n\n\n\nThe entire Self Attention is done multiple times to get Multi-Head Attention which aids in parallel computing of the weights of the attention layers:\n\n\n\nThis excellent resource from [Jalammar](http://jalammar.github.io/illustrated-transformer/) is highly recommended."},{"metadata":{},"cell_type":"markdown","source":"## Layer Normalization after Multi Headed Self Attention\n\nLayer normalization is an important measure to maintain the mean and variance of each layer. It has its similarities with batch normalization in normal neural networks, but with certain [changes](http://jalammar.github.io/images/t/transformer_resideual_layer_norm_3.png).\n\nThe entire Encoder architecture involving Layer Norm and Self Attention for Transformers is represented here:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Creating a Transformer With Keras\n\nA mini transformer with multi headed attention mechanism from keras for our use cases.\n\nFor this we will be needing a masked Multi Head self attention.The effect of masking in Attention is provided in the image:\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Resources for Transformers\n\nSome of the resources which helped to create this:\n\n1. [Tensorflow Implementation-Google Brain](https://github.com/tensorflow/models/tree/master/official/nlp/transformer)\n2. [Excellent Resource](https://github.com/Lsdefine/attention-is-all-you-need-keras/blob/master/transformer.py)\n3. [Tensor2Tensor from Google](https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb)\n\nThese will be helpful for analysing with other resources provided."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Creating the inputs features\ntrain_df=pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest_df=pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\ntrain_set,test_set=train_test_split(train_df,test_size=0.2,random_state=2017)\nprint(train_set.shape)\nprint(test_set.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Codes from day-1\nmaxlen=1000\nmax_features=5000 \nembed_size=300\n\n#clean some null words or use the previously cleaned & lemmatized corpus\n\ntrain_x=train_set['question_text'].fillna('_na_').values\nval_x=test_set['question_text'].fillna('_na_').values\n\n#Tokenizing steps- must be remembered\ntokenizer=Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_x))\ntrain_x=tokenizer.texts_to_sequences(train_x)\nval_x=tokenizer.texts_to_sequences(val_x)\n\n#Pad the sequence- To allow same length for all vectorized words\ntrain_x=pad_sequences(train_x,maxlen=maxlen)\nval_x=pad_sequences(val_x,maxlen=maxlen)\n\n\n#get the target values - either using values or using Label Encoder\ntrain_y=train_set['target'].values\nval_y=test_set['target'].values\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Codes from Day-1\nEMBEDDING_FILE = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\ndef get_coefs(word,*arr): return word, np.asarray(arr, dtype='float32')\nembeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(EMBEDDING_FILE))\n\nall_embs = np.stack(embeddings_index.values())\nemb_mean,emb_std = all_embs.mean(), all_embs.std()\nembed_size = all_embs.shape[1]\n\nword_index = tokenizer.word_index\nnb_words = min(max_features, len(word_index))\nembedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))\nfor word, i in word_index.items():\n if i >= max_features: continue\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None: embedding_matrix[i] = embedding_vector\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building the actual Model\n\nIn this case, here are the following parts of the model:\n\n1. Add a Layer Normalization Layer\n2. Create a function/class for Self Attention\n3. Join different Self Attention objects together to get Multi Head Attention\n4. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import random, os, sys\nimport numpy as np\nfrom keras.models import *\nfrom keras.layers import *\nfrom keras.callbacks import *\nfrom keras.initializers import *\nimport tensorflow as tf\nfrom keras.engine.topology import Layer\n\ntry:\n from dataloader import TokenList, pad_to_longest\n # for transformer\nexcept: pass\n\n#Layer normalization class\nclass LayerNormalization(Layer):\n def __init__(self, eps=1e-6, **kwargs):\n self.eps = eps\n super(LayerNormalization, self).__init__(**kwargs)\n def build(self, input_shape):\n #Adding custom weights\n self.gamma = self.add_weight(name='gamma', shape=input_shape[-1:],\n initializer=Ones(), trainable=True)\n self.beta = self.add_weight(name='beta', shape=input_shape[-1:],\n initializer=Zeros(), trainable=True)\n super(LayerNormalization, self).build(input_shape)\n def call(self, x):\n mean = K.mean(x, axis=-1, keepdims=True)\n std = K.std(x, axis=-1, keepdims=True)\n return self.gamma * (x - mean) / (std + self.eps) + self.beta\n def compute_output_shape(self, input_shape):\n return input_shape\n\n#Division by 8 (q.k/d^0.5)\nclass ScaledDotProductAttention():\n def __init__(self, d_model, attn_dropout=0.1):\n self.temper = np.sqrt(d_model)\n self.dropout = Dropout(attn_dropout)\n def __call__(self, q, k, v, mask):\n attn = Lambda(lambda x:K.batch_dot(x[0],x[1],axes=[2,2])/self.temper)([q, k])\n if mask is not None:\n mmask = Lambda(lambda x:(-1e+10)*(1-x))(mask)\n attn = Add()([attn, mmask])\n attn = Activation('softmax')(attn)\n attn = self.dropout(attn)\n output = Lambda(lambda x:K.batch_dot(x[0], x[1]))([attn, v])\n return output, attn\n\nclass MultiHeadAttention():\n # mode 0 - big martixes, faster; mode 1 - more clear implementation\n def __init__(self, n_head, d_model, d_k, d_v, dropout, mode=0, use_norm=True):\n self.mode = mode\n self.n_head = n_head\n self.d_k = d_k\n self.d_v = d_v\n self.dropout = dropout\n if mode == 0:\n self.qs_layer = Dense(n_head*d_k, use_bias=False)\n self.ks_layer = Dense(n_head*d_k, use_bias=False)\n self.vs_layer = Dense(n_head*d_v, use_bias=False)\n elif mode == 1:\n self.qs_layers = []\n self.ks_layers = []\n self.vs_layers = []\n for _ in range(n_head):\n self.qs_layers.append(TimeDistributed(Dense(d_k, use_bias=False)))\n self.ks_layers.append(TimeDistributed(Dense(d_k, use_bias=False)))\n self.vs_layers.append(TimeDistributed(Dense(d_v, use_bias=False)))\n #Joining scaled dot product\n self.attention = ScaledDotProductAttention(d_model)\n self.layer_norm = LayerNormalization() if use_norm else None\n self.w_o = TimeDistributed(Dense(d_model))\n\n def __call__(self, q, k, v, mask=None):\n d_k, d_v = self.d_k, self.d_v\n n_head = self.n_head\n\n if self.mode == 0:\n qs = self.qs_layer(q) # [batch_size, len_q, n_head*d_k]\n ks = self.ks_layer(k)\n vs = self.vs_layer(v)\n\n def reshape1(x):\n s = tf.shape(x) # [batch_size, len_q, n_head * d_k]\n x = tf.reshape(x, [s[0], s[1], n_head, d_k])\n x = tf.transpose(x, [2, 0, 1, 3]) \n x = tf.reshape(x, [-1, s[1], d_k]) # [n_head * batch_size, len_q, d_k]\n return x\n qs = Lambda(reshape1)(qs)\n ks = Lambda(reshape1)(ks)\n vs = Lambda(reshape1)(vs)\n\n if mask is not None:\n mask = Lambda(lambda x:K.repeat_elements(x, n_head, 0))(mask)\n head, attn = self.attention(qs, ks, vs, mask=mask) \n \n def reshape2(x):\n s = tf.shape(x) # [n_head * batch_size, len_v, d_v]\n x = tf.reshape(x, [n_head, -1, s[1], s[2]]) \n x = tf.transpose(x, [1, 2, 0, 3])\n x = tf.reshape(x, [-1, s[1], n_head*d_v]) # [batch_size, len_v, n_head * d_v]\n return x\n head = Lambda(reshape2)(head)\n elif self.mode == 1:\n heads = []; attns = []\n for i in range(n_head):\n qs = self.qs_layers[i](q) \n ks = self.ks_layers[i](k) \n vs = self.vs_layers[i](v) \n head, attn = self.attention(qs, ks, vs, mask)\n heads.append(head); attns.append(attn)\n head = Concatenate()(heads) if n_head > 1 else heads[0]\n attn = Concatenate()(attns) if n_head > 1 else attns[0]\n\n outputs = self.w_o(head)\n outputs = Dropout(self.dropout)(outputs)\n if not self.layer_norm: return outputs, attn\n outputs = Add()([outputs, q])\n return self.layer_norm(outputs), attn\n#Feedforward layer using COnv1D and Layer normalization.\nclass PositionwiseFeedForward():\n def __init__(self, d_hid, d_inner_hid, dropout=0.1):\n self.w_1 = Conv1D(d_inner_hid, 1, activation='relu')\n self.w_2 = Conv1D(d_hid, 1)\n self.layer_norm = LayerNormalization()\n self.dropout = Dropout(dropout)\n def __call__(self, x):\n output = self.w_1(x) \n output = self.w_2(output)\n output = self.dropout(output)\n output = Add()([output, x])\n return self.layer_norm(output)\n#Encoder layer containing self/multi head attention with positionwisefeedforward\nclass EncoderLayer():\n def __init__(self, d_model, d_inner_hid, n_head, d_k, d_v, dropout=0.1):\n self.self_att_layer = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)\n self.pos_ffn_layer = PositionwiseFeedForward(d_model, d_inner_hid, dropout=dropout)\n def __call__(self, enc_input, mask=None):\n output, slf_attn = self.self_att_layer(enc_input, enc_input, enc_input, mask=mask)\n output = self.pos_ffn_layer(output)\n return output, slf_attn\n#Decoder layer with same architecture as the encoder.\nclass DecoderLayer():\n def __init__(self, d_model, d_inner_hid, n_head, d_k, d_v, dropout=0.1):\n self.self_att_layer = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)\n self.enc_att_layer = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)\n self.pos_ffn_layer = PositionwiseFeedForward(d_model, d_inner_hid, dropout=dropout)\n def __call__(self, dec_input, enc_output, self_mask=None, enc_mask=None):\n output, slf_attn = self.self_att_layer(dec_input, dec_input, dec_input, mask=self_mask)\n output, enc_attn = self.enc_att_layer(output, enc_output, enc_output, mask=enc_mask)\n output = self.pos_ffn_layer(output)\n return output, slf_attn, enc_attn\n#This is from the paper \"Attention is all you need\" which hypothesizes sin and cosine for positional encoding\ndef GetPosEncodingMatrix(max_len, d_emb):\n pos_enc = np.array([\n [pos / np.power(10000, 2 * (j // 2) / d_emb) for j in range(d_emb)] \n if pos != 0 else np.zeros(d_emb) \n for pos in range(max_len)\n ])\n pos_enc[1:, 0::2] = np.sin(pos_enc[1:, 0::2]) # dim 2i\n pos_enc[1:, 1::2] = np.cos(pos_enc[1:, 1::2]) # dim 2i+1\n return pos_enc\n\n#normal padding class for masking\ndef GetPadMask(q, k):\n ones = K.expand_dims(K.ones_like(q, 'float32'), -1)\n mask = K.cast(K.expand_dims(K.not_equal(k, 0), 1), 'float32')\n mask = K.batch_dot(ones, mask, axes=[2,1])\n return mask\n\ndef GetSubMask(s):\n len_s = tf.shape(s)[1]\n bs = tf.shape(s)[:1]\n mask = K.cumsum(tf.eye(len_s, batch_shape=bs), 1)\n return mask\n\nclass Encoder():\n def __init__(self, d_model, d_inner_hid, n_head, d_k, d_v, \\\n layers=6, dropout=0.1, word_emb=None, pos_emb=None):\n self.emb_layer = word_emb\n self.pos_layer = pos_emb\n self.emb_dropout = Dropout(dropout)\n self.layers = [EncoderLayer(d_model, d_inner_hid, n_head, d_k, d_v, dropout) for _ in range(layers)]\n \n def __call__(self, src_seq, src_pos, return_att=False, active_layers=999):\n x = self.emb_layer(src_seq)\n if src_pos is not None:\n pos = self.pos_layer(src_pos)\n x = Add()([x, pos])\n x = self.emb_dropout(x)\n if return_att: atts = []\n mask = Lambda(lambda x:GetPadMask(x, x))(src_seq)\n for enc_layer in self.layers[:active_layers]:\n x, att = enc_layer(x, mask)\n if return_att: atts.append(att)\n return (x, atts) if return_att else x\n\n\nclass Transformer():\n def __init__(self, len_limit, d_model=embed_size, \\\n d_inner_hid=512, n_head=10, d_k=64, d_v=64, layers=2, dropout=0.1, \\\n share_word_emb=False, **kwargs):\n self.name = 'Transformer'\n self.len_limit = len_limit\n self.src_loc_info = True\n self.d_model = d_model\n self.decode_model = None\n d_emb = d_model\n\n pos_emb = Embedding(len_limit, d_emb, trainable=False, \\\n weights=[GetPosEncodingMatrix(len_limit, d_emb)])\n\n i_word_emb = Embedding(max_features, d_emb, weights=[embedding_matrix]) # Add Kaggle provided embedding here\n\n self.encoder = Encoder(d_model, d_inner_hid, n_head, d_k, d_v, layers, dropout, \\\n word_emb=i_word_emb, pos_emb=pos_emb)\n\n \n def get_pos_seq(self, x):\n mask = K.cast(K.not_equal(x, 0), 'int32')\n pos = K.cumsum(K.ones_like(x, 'int32'), 1)\n return pos * mask\n\n def compile(self, active_layers=999):\n src_seq_input = Input(shape=(None,))\n src_seq = src_seq_input\n src_pos = Lambda(self.get_pos_seq)(src_seq)\n if not self.src_loc_info: src_pos = None\n\n x = self.encoder(src_seq, src_pos, active_layers=active_layers)\n # x = GlobalMaxPool1D()(x) # Not sure about this layer. Just wanted to reduce dimension\n x = GlobalAveragePooling1D()(x)\n outp = Dense(1, activation=\"sigmoid\")(x)\n\n self.model = Model(inputs=src_seq_input, outputs=outp)\n self.model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"seq2seq = Transformer(maxlen, layers=1)\nseq2seq.compile()\nmodel = seq2seq.model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evaluate and Train the Model\n\nmodel.fit(train_x, train_y, batch_size=1024, epochs=1, validation_data=(val_x, val_y))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some important resources from Github\n\nHere are some important resources which will definitely help.\n\n1. [BERT:Google Research](https://github.com/google-research/bert)\n2. [Dale's notebook](https://github.com/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb)\n3. [Albert](https://github.com/google-research/albert)\n4. [GPT-2](https://github.com/openai/gpt-2)\n5. [Roberta](https://github.com/pytorch/fairseq/tree/master/examples/roberta)\n6. [Bart](https://github.com/pytorch/fairseq/tree/master/examples/bart)\n7. [Transformer-XL](https://github.com/kimiyoung/transformer-xl)\n8. [Hugging Face Transformers](https://github.com/huggingface/transformers)\n\n\nThese resources should help. Apart from this, for more details on attention mechanism (for now only Hierarchical) you can refer to [my repo](https://github.com/abhilash1910/MiniAttention). I will be adding more custom attention modules which can be used any sequential network, including a separate transformer architecture. "},{"metadata":{},"cell_type":"markdown","source":"## Graph Learning\n\nThere is another paradigm in NLP which relies on Deep Learning in Graphs, which is beyond today's session. Just as an overview, all the architectures/models which we have built are also applicable for knowledge graphs. These are essentially in huge corpuses of data such as Google search, where scalable transformer architectures are employed on graphs. All social media (twitter,facebook etc) rely on extensive large scale graph transformer networks for classification, question answering, semantic indexing, reference resolution, masking and generative modelling."},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nThis completes today's session and the entire workshop in totality.\nWould appreciate feedbacks as well as follow-ups. Follow me on [github](https://github.com/abhilash1910) for any updates on deep learning or NLP!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}