{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Fine-Tune BERT for Text Classification with TensorFlow

","metadata":{"id":"zGCJYkQj_Uu2"}},{"cell_type":"markdown","source":"
\n \n

Figure 1: BERT Classification Model

\n
","metadata":{"id":"4y2m1S6e12il"}},{"cell_type":"markdown","source":"We will be using GPU accelerated Kernel for this tutorial as we would require a GPU to fine-tune BERT.\n\n## Prerequisites:\n- Willingness to learn: Growth Mindset is all you need \n- Some basic idea about Tensorflow/Keras \n- Some Python to follow along with the code ","metadata":{}},{"cell_type":"markdown","source":"## Initial Set Up","metadata":{"id":"zhV7YowzK-I9"}},{"cell_type":"markdown","source":"### Install TensorFlow and TensorFlow Model Garden","metadata":{"id":"Obch3rAuBVf0"}},{"cell_type":"code","source":"import tensorflow as tf\nprint(tf.version.VERSION)","metadata":{"id":"bUQEY3dFB0jX","outputId":"4de5670e-af17-4b66-cf41-6effd4ddada2","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cloning the Github Repo for tensorflow models \n- –depth 1, during cloning, Git will only get the latest copy of the relevant files. It can save you a lot of space and time.\n\n- -b lets us clone a specific branch only.\n\n**Please match it with your tensorflow 2.x version.**","metadata":{"id":"RAjJnOAL4wrn"}},{"cell_type":"code","source":"!git clone --depth 1 -b v2.4.0 https://github.com/tensorflow/models.git","metadata":{"id":"AFRTC-zwUy6D","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install six\n!pip install google-api-python-client>=1.6.7\n!pip install kaggle>=1.3.9\n!pip install numpy>=1.20\n!pip install oauth2client\n!pip install pandas>=0.22.0\n!pip install psutil>=5.4.3\n!pip install py-cpuinfo>=3.3.0\n!pip install scipy>=0.19.1\n!pip install tensorflow-hub>=0.6.0\n!pip install tensorflow-model-optimization>=0.4.1\n!pip install tensorflow-datasets\n!pip install tensorflow-addons\n!pip install gin-config\n!pip install tf_slim>=1.1.0\n!pip install Cython\n!pip install matplotlib\n!pip install pyyaml>=5.1,<6.0\n!pip install opencv-python-headless\n!pip install Pillow\n!pip install pycocotools\n!pip install seqeval\n!pip install sentencepiece\n!pip install sacrebleu\n!pip install immutabledict","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**: After installing the required Python packages, you'll need to restart the Colab Runtime Engine \n(Run ---> Restart and clear all cell outputs...)","metadata":{"id":"-WlOhTvS54cu"}},{"cell_type":"markdown","source":"**NOTE: ANYTHING BEFORE THIS CELL SHOULD ONLY BE RUN ONCE, ONLY DURING THE INITIAL SETUP**","metadata":{"id":"15Tl8wqi_reD"}},{"cell_type":"markdown","source":"## Some Initial Imports and Checks","metadata":{"id":"KOn8Qr4x_TCc"}},{"cell_type":"markdown","source":"A Healthy practice for any ML practioner is to do a clean experiment tracking such that reasults are reproducable and trackable. For this kernel we will be looking into [Weights and Biases](https://wandb.ai/site) for experiment tracking.\n\nHere are Four main things that W&B offers:\n- Experiment Tracking: Tracking ML Experiments and logging various parameters and metrics on a clean dashboard.\n- Sweeps: Hyper-parameter tuning. You can run multiple experiments with different hyper-parameters and track them.\n- Artifacts: Storing Datasets, models and other files for version tracking.\n- Reports: we can create reports on experiments and project levels.\n\n\nWe will be looking into, **Experiment Trancking** and **Artifacts**","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade -q wandb","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:17:09.424748Z","iopub.execute_input":"2022-12-03T01:17:09.425221Z","iopub.status.idle":"2022-12-03T01:17:19.593435Z","shell.execute_reply.started":"2022-12-03T01:17:09.425089Z","shell.execute_reply":"2022-12-03T01:17:19.592260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Logging into wandb.\nFirst things first, we need to create a free account on [W&B](https://wandb.ai/site)\n\nThen let us access our [authorization API key](https://wandb.ai/authorize) and add it to kaggle's secret key for hassle free authentication.\n\nMore details about Kaggle's Secret key feature on ---> https://www.kaggle.com/product-feedback/114053","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nwandb_api = user_secrets.get_secret(\"wandb\")\n","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:17:24.927778Z","iopub.execute_input":"2022-12-03T01:17:24.928160Z","iopub.status.idle":"2022-12-03T01:17:25.178260Z","shell.execute_reply.started":"2022-12-03T01:17:24.928118Z","shell.execute_reply":"2022-12-03T01:17:25.177389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nfrom wandb.keras import WandbCallback\nwandb.login(key=wandb_api)\n","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:17:28.782312Z","iopub.execute_input":"2022-12-03T01:17:28.782656Z","iopub.status.idle":"2022-12-03T01:17:33.351849Z","shell.execute_reply.started":"2022-12-03T01:17:28.782619Z","shell.execute_reply":"2022-12-03T01:17:33.350940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\nimport tensorflow_hub as hub\nimport sys\nsys.path.append('models')\nfrom official.nlp.data import classifier_data_lib\nfrom official.nlp.bert import tokenization\nfrom official.nlp import optimization\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set()","metadata":{"id":"GmqEylyFYTdP","execution":{"iopub.status.busy":"2022-12-03T01:17:38.386289Z","iopub.execute_input":"2022-12-03T01:17:38.386639Z","iopub.status.idle":"2022-12-03T01:17:39.714104Z","shell.execute_reply.started":"2022-12-03T01:17:38.386599Z","shell.execute_reply":"2022-12-03T01:17:39.713251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"TF Version: \", tf.__version__)\nprint(\"Eager mode: \", tf.executing_eagerly())\nprint(\"Hub version: \", hub.__version__)\nprint(\"GPU is\", \"available\" if tf.config.experimental.list_physical_devices(\"GPU\") else \"NOT AVAILABLE\")","metadata":{"id":"ZuX1lB8pPJ-W","outputId":"02ab3578-d8c7-4c10-b552-7f89caa060fb","execution":{"iopub.status.busy":"2022-12-03T01:17:44.231862Z","iopub.execute_input":"2022-12-03T01:17:44.232242Z","iopub.status.idle":"2022-12-03T01:17:44.437411Z","shell.execute_reply.started":"2022-12-03T01:17:44.232208Z","shell.execute_reply":"2022-12-03T01:17:44.436125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lets Get the Dataset","metadata":{"id":"IMsEoT3Fg4Wg"}},{"cell_type":"markdown","source":"- The data we will use is the dataset provided on the [Quora Insincere Questions Classification competition on Kaggle](https://www.kaggle.com/c/quora-insincere-questions-classification/data). \n\n- Please feel free to download the train set from kaggle or use the link below to download the train.csv from that competition [https://archive.org/download/quora_dataset_train.csv/quora_dataset_train.csv.zip](https://archive.org/download/quora_dataset_train.csv/quora_dataset_train.csv.zip). \n\n- Let us decompress and read the data into a pandas DataFrame.","metadata":{"id":"QtbwpWgyEZg7"}},{"cell_type":"code","source":"import os\nprint(os.listdir(\"../input/entwicklungsdaten\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TO LOAD DATA FROM KAGGLE\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\ndf = pd.read_csv('../input/entwicklungsdaten/Entwicklungsdaten.tsv', sep='\\t', usecols= ['hatespeech','c_text'])\nprint(df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:18:03.361161Z","iopub.execute_input":"2022-12-03T01:18:03.361549Z","iopub.status.idle":"2022-12-03T01:18:03.601441Z","shell.execute_reply.started":"2022-12-03T01:18:03.361515Z","shell.execute_reply":"2022-12-03T01:18:03.599977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(10)\n# label 0 == non toxic\n# label 1 == toxic ","metadata":{"id":"yeHE98KiMvDd","outputId":"ca9c98d7-28cd-4198-c051-1179fe34c34e","execution":{"iopub.status.busy":"2022-12-03T01:18:06.692552Z","iopub.execute_input":"2022-12-03T01:18:06.692932Z","iopub.status.idle":"2022-12-03T01:18:06.713650Z","shell.execute_reply.started":"2022-12-03T01:18:06.692875Z","shell.execute_reply":"2022-12-03T01:18:06.712690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Get to Know your data: Some Basic EDA","metadata":{"id":"yWpRHkVgEtDx"}},{"cell_type":"code","source":"print(df['hatespeech'].value_counts())\ndf['hatespeech'].value_counts().plot.bar()\nplt.yscale('log');\nplt.title('Distribution of Labels')","metadata":{"id":"leRFRWJMocVa","outputId":"139c0c12-979d-4e5c-9a10-32501b1d7e4a","execution":{"iopub.status.busy":"2022-12-03T01:18:11.077929Z","iopub.execute_input":"2022-12-03T01:18:11.078294Z","iopub.status.idle":"2022-12-03T01:18:11.496987Z","shell.execute_reply.started":"2022-12-03T01:18:11.078260Z","shell.execute_reply":"2022-12-03T01:18:11.495974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Average word length of tweets in dataset is {0:.0f}.'.format(np.mean(df['c_text'].apply(lambda x: len(x.split())))))\nprint('Max word length of tweets in dataset is {0:.0f}.'.format(np.max(df['c_text'].apply(lambda x: len(x.split())))))\nprint('Average character length of tweets in dataset is {0:.0f}.'.format(np.mean(df['c_text'].apply(lambda x: len(x)))))","metadata":{"id":"KNYi2D4kBLeR","outputId":"f880db5b-5d50-4b13-b75e-d7b1adc06d7a","execution":{"iopub.status.busy":"2022-12-03T01:18:22.080486Z","iopub.execute_input":"2022-12-03T01:18:22.080835Z","iopub.status.idle":"2022-12-03T01:18:22.131454Z","shell.execute_reply.started":"2022-12-03T01:18:22.080801Z","shell.execute_reply":"2022-12-03T01:18:22.130558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since the dataset is very imbalanced we will keep the same distribution in both train and test set by stratifying it based on the labels\n# using small portions of the data as the over all dataset would take ages to train, feel free to include more data by changing train_size \ntrain_df, remaining = train_test_split(df, random_state=42, train_size=0.9, stratify=df.hatespeech.values)\nvalid_df, _ = train_test_split(remaining, random_state=42, train_size=0.1, stratify=remaining.hatespeech.values)\nprint(train_df.shape)\nprint(valid_df.shape)","metadata":{"id":"fScULIGPwuWk","outputId":"e71c3c2f-a8b6-42d4-81d5-c76fbc391358","execution":{"iopub.status.busy":"2022-12-03T01:18:25.829589Z","iopub.execute_input":"2022-12-03T01:18:25.829949Z","iopub.status.idle":"2022-12-03T01:18:25.845715Z","shell.execute_reply.started":"2022-12-03T01:18:25.829892Z","shell.execute_reply":"2022-12-03T01:18:25.844467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"FOR TRAIN SET\\n\")\nprint('Average word length of questions in train set is {0:.0f}.'.format(np.mean(train_df['c_text'].apply(lambda x: len(x.split())))))\nprint('Max word length of questions in train set is {0:.0f}.'.format(np.max(train_df['c_text'].apply(lambda x: len(x.split())))))\nprint('Average character length of questions in train set is {0:.0f}.'.format(np.mean(train_df['c_text'].apply(lambda x: len(x)))))\nprint('Label Distribution in train set is \\n{}.'.format(train_df['hatespeech'].value_counts()))\nprint(\"\\n\\nFOR VALIDATION SET\\n\")\nprint('Average word length of questions in valid set is {0:.0f}.'.format(np.mean(valid_df['c_text'].apply(lambda x: len(x.split())))))\nprint('Max word length of questions in valid set is {0:.0f}.'.format(np.max(valid_df['c_text'].apply(lambda x: len(x.split())))))\nprint('Average character length of questions in valid set is {0:.0f}.'.format(np.mean(valid_df['c_text'].apply(lambda x: len(x)))))\nprint('Label Distribution in validation set is \\n{}.'.format(valid_df['hatespeech'].value_counts()))","metadata":{"id":"P-MRpoomCHb8","outputId":"0386a834-dc95-44cb-fd12-75a69c5b553a","execution":{"iopub.status.busy":"2022-12-03T01:18:29.238739Z","iopub.execute_input":"2022-12-03T01:18:29.239477Z","iopub.status.idle":"2022-12-03T01:18:29.323591Z","shell.execute_reply.started":"2022-12-03T01:18:29.239432Z","shell.execute_reply":"2022-12-03T01:18:29.322737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So it looks like the train and validation set are similar in terms of class imbalance and the various lengths in the question texts.","metadata":{"id":"GS7wwDvmDwH_"}},{"cell_type":"code","source":"# TRAIN SET \ntrain_df['c_text'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Distribution of tweet text length in words')","metadata":{"id":"tE0eCG8pFYnZ","outputId":"ff243909-2ca1-4c21-8984-0933c098cf07","execution":{"iopub.status.busy":"2022-12-03T01:18:33.308732Z","iopub.execute_input":"2022-12-03T01:18:33.309084Z","iopub.status.idle":"2022-12-03T01:18:33.703805Z","shell.execute_reply.started":"2022-12-03T01:18:33.309050Z","shell.execute_reply":"2022-12-03T01:18:33.702794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# VALIDATION SET\nvalid_df['c_text'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Distribution of tweet text length in words')","metadata":{"id":"ucLYQp-dFc0e","outputId":"79276936-5ded-479e-8019-34c0167fea9f","execution":{"iopub.status.busy":"2022-12-03T01:18:45.606676Z","iopub.execute_input":"2022-12-03T01:18:45.607047Z","iopub.status.idle":"2022-12-03T01:18:45.923587Z","shell.execute_reply.started":"2022-12-03T01:18:45.607010Z","shell.execute_reply":"2022-12-03T01:18:45.922407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TRAIN SET\ntrain_df['c_text'].apply(lambda x: len(x)).plot(kind='hist');\nplt.yscale('log');\nplt.title('Distribution of question text length in characters')","metadata":{"id":"zK4D4acOGFTI","outputId":"ec7cf553-0934-4452-fc99-68ef009b1597","execution":{"iopub.status.busy":"2022-12-03T01:18:49.238984Z","iopub.execute_input":"2022-12-03T01:18:49.239330Z","iopub.status.idle":"2022-12-03T01:18:49.605420Z","shell.execute_reply.started":"2022-12-03T01:18:49.239297Z","shell.execute_reply":"2022-12-03T01:18:49.604539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# VALIDATION SET\nvalid_df['c_text'].apply(lambda x: len(x)).plot(kind='hist');\nplt.yscale('log');\nplt.title('Distribution of question text length in characters')","metadata":{"id":"uqjE6McdGFDg","outputId":"9cbd7b5f-a688-43c1-e305-d8f0a9026338","execution":{"iopub.status.busy":"2022-12-03T01:18:52.894292Z","iopub.execute_input":"2022-12-03T01:18:52.894671Z","iopub.status.idle":"2022-12-03T01:18:53.241100Z","shell.execute_reply.started":"2022-12-03T01:18:52.894633Z","shell.execute_reply":"2022-12-03T01:18:53.240095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Even the distribution of question length in words and characters is very similar. It looks like a good train test split so far.","metadata":{"id":"Gqx5RT7zFiGe"}},{"cell_type":"markdown","source":"# Taming the Data","metadata":{"id":"I46vYYMgG4kh"}},{"cell_type":"code","source":"# we want the dataset to be created and processed on the cpu \nwith tf.device('/cpu:0'):\n train_data = tf.data.Dataset.from_tensor_slices((train_df['c_text'].values, train_df['hatespeech'].values))\n valid_data = tf.data.Dataset.from_tensor_slices((valid_df['c_text'].values, valid_df['hatespeech'].values))\n # lets look at 3 samples from train set\n for text,label in train_data.take(3):\n print(text)\n print(label)\n","metadata":{"id":"qQYMGT5_qLPX","outputId":"06269af9-aae8-4d08-a7c9-e87473836654","execution":{"iopub.status.busy":"2022-12-03T01:19:02.199677Z","iopub.execute_input":"2022-12-03T01:19:02.200046Z","iopub.status.idle":"2022-12-03T01:19:04.088671Z","shell.execute_reply.started":"2022-12-03T01:19:02.200009Z","shell.execute_reply":"2022-12-03T01:19:04.087758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_data))\nprint(len(valid_data))","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:19:13.043606Z","iopub.execute_input":"2022-12-03T01:19:13.043962Z","iopub.status.idle":"2022-12-03T01:19:13.050165Z","shell.execute_reply.started":"2022-12-03T01:19:13.043926Z","shell.execute_reply":"2022-12-03T01:19:13.049234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lets BERT: Get the Pre-trained BERT Model from TensorFlow Hub","metadata":{"id":"e2-ReN88Hvy_"}},{"cell_type":"markdown","source":"We will be using the uncased BERT present in the tfhub. In order to prepare the text to be given to the BERT layer, we need to first tokenize our words. The tokenizer here is present as a model asset and will do uncasing for us as well.","metadata":{"id":"-0anx2AQ69wt"}},{"cell_type":"markdown","source":"Setting all parameters in form of a dictionary so any changes if needed can be made here","metadata":{}},{"cell_type":"code","source":"# Setting some parameters\n\nconfig = {'label_list' : [0, 1], # Label categories\n 'max_seq_length' : 240, # maximum length of (token) input sequences\n 'train_batch_size' : 32,\n 'learning_rate': 2e-5,\n 'epochs': 10,\n 'optimizer': 'adam',\n 'dropout': 0.5,\n 'train_samples': len(train_data),\n 'valid_samples': len(valid_data),\n 'train_split': 0.9,\n 'valid_split': 0.1\n }","metadata":{"id":"fi141L6X7EmH","execution":{"iopub.status.busy":"2022-12-03T01:19:27.390603Z","iopub.execute_input":"2022-12-03T01:19:27.390968Z","iopub.status.idle":"2022-12-03T01:19:27.398049Z","shell.execute_reply.started":"2022-12-03T01:19:27.390931Z","shell.execute_reply":"2022-12-03T01:19:27.396052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get BERT layer and tokenizer:\n# All details here: https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2\n\nbert_layer = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2',\n#bert_layer = hub.KerasLayer('https://huggingface.co/bert-base-german-cased',\n trainable=True)\nvocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\ndo_lower_case = bert_layer.resolved_object.do_lower_case.numpy() # checks if the bert layer we are using is uncased or not\ntokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)","metadata":{"id":"EMb5M86b4-BU","execution":{"iopub.status.busy":"2022-12-03T01:20:28.755479Z","iopub.execute_input":"2022-12-03T01:20:28.755846Z","iopub.status.idle":"2022-12-03T01:20:33.166091Z","shell.execute_reply.started":"2022-12-03T01:20:28.755809Z","shell.execute_reply":"2022-12-03T01:20:33.165170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Checking out some of the training samples and their tokenized ids","metadata":{"id":"-cXVyapO7Oot"}},{"cell_type":"code","source":"input_string = \"hello world, it is a wonderful day for learning\"\nprint(tokenizer.wordpiece_tokenizer.tokenize(input_string))\nprint(tokenizer.convert_tokens_to_ids(tokenizer.wordpiece_tokenizer.tokenize(input_string)))","metadata":{"id":"wEUezMK-zkkI","outputId":"5375cfde-c6e5-4fd5-d1a2-647246cd465f","execution":{"iopub.status.busy":"2022-12-03T01:20:37.595615Z","iopub.execute_input":"2022-12-03T01:20:37.595976Z","iopub.status.idle":"2022-12-03T01:20:37.602337Z","shell.execute_reply.started":"2022-12-03T01:20:37.595935Z","shell.execute_reply":"2022-12-03T01:20:37.601373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lets Get That Data Ready: Tokenize and Preprocess Text for BERT","metadata":{"id":"9QinzNq6OsP1"}},{"cell_type":"markdown","source":"Each line of the dataset is composed of the review text and its label.\nData preprocessing consists of transforming text to BERT input features:\n`input_word_ids`, `input_mask`, `segment_ids/input_type_ids`\n\n- **Input Word Ids:** Output of our tokenizer, converting each sentence into a set of token ids.\n\n- **Input Masks:** Since we are padding all the sequences to 128(max sequence length), it is important that we create some sort of mask to make sure those paddings do not interfere with the actual text tokens. Therefore we need a generate input mask blocking the paddings. The mask has 1 for real tokens and 0 for padding tokens. Only real\ntokens are attended to.\n\n- **Segment Ids:** For out task of text classification, since there is only one sequence, the segment_ids/input_type_ids is essentially just a vector of 0s. \n\n\n\nBert was trained on two tasks:\n- fill in randomly masked words from a sentence.\n- given two sentences, which sentence came first. \n","metadata":{"id":"dZCrWwCCo_82"}},{"cell_type":"code","source":"# This provides a function to convert row to input features and label, \n# this uses the classifier_data_lib which is a class defined in the tensorflow model garden we installed earlier\ndef create_feature(text, label, label_list=config['label_list'], max_seq_length=config['max_seq_length'], tokenizer=tokenizer):\n \"\"\"\n converts the datapoint into usable features for BERT using the classifier_data_lib\n\n Parameters:\n text: Input text string\n label: label associated with the text\n label_list: (list) all possible labels\n max_seq_length: (int) maximum sequence length set for bert\n tokenizer: the tokenizer object instantiated by the files in model assets\n\n Returns:\n feature.input_ids: The token ids for the input text string\n feature.input_masks: The padding mask generated \n feature.segment_ids: essentially here a vector of 0s since classification\n feature.label_id: the corresponding label id from lable_list [0, 1] here\n\n \"\"\"\n # since we only have 1 sentence for classification purpose, textr_b is None\n example = classifier_data_lib.InputExample(guid = None,\n text_a = text.numpy(), \n text_b = None, \n label = label.numpy())\n # since only 1 example, the index=0\n feature = classifier_data_lib.convert_single_example(0, example, label_list,\n max_seq_length, tokenizer)\n\n return (feature.input_ids, feature.input_mask, feature.segment_ids, feature.label_id)","metadata":{"id":"m-21A5aNJM0W","execution":{"iopub.status.busy":"2022-12-03T01:20:48.705929Z","iopub.execute_input":"2022-12-03T01:20:48.706290Z","iopub.status.idle":"2022-12-03T01:20:48.713408Z","shell.execute_reply.started":"2022-12-03T01:20:48.706256Z","shell.execute_reply":"2022-12-03T01:20:48.712101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You want to use [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) to apply this function to each element of the dataset. [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) runs in graph mode.\n\n- Graph tensors do not have a value.\n- In graph mode you can only use TensorFlow Ops and functions.\n\nSo you can't `.map` this function directly: You need to wrap it in a [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function). The [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function) will pass regular tensors (with a value and a `.numpy()` method to access it), to the wrapped python function.","metadata":{"id":"A_HQSsHwWCsK"}},{"cell_type":"markdown","source":"### Wrapping the Python Function into a TensorFlow op for Eager Execution","metadata":{"id":"zaNlkKVfWX0Q"}},{"cell_type":"code","source":"def create_feature_map(text, label):\n \"\"\"\n A tensorflow function wrapper to apply the transformation on the dataset.\n Parameters:\n Text: the input text string.\n label: the classification ground truth label associated with the input string\n\n Returns:\n A tuple of a dictionary and a corresponding label_id with it. The dictionary \n contains the input_word_ids, input_mask, input_type_ids \n \"\"\"\n input_ids, input_mask, segment_ids, label_id = tf.py_function(create_feature, inp=[text, label], \n Tout=[tf.int32, tf.int32, tf.int32, tf.int32])\n max_seq_length = config['max_seq_length']\n\n # py_func doesn't set the shape of the returned tensors.\n input_ids.set_shape([max_seq_length])\n input_mask.set_shape([max_seq_length])\n segment_ids.set_shape([max_seq_length])\n label_id.set_shape([])\n\n x = {\n 'input_word_ids': input_ids,\n 'input_mask': input_mask,\n 'input_type_ids': segment_ids\n }\n return (x, label_id)\n\n # the final datapoint passed to the model is of the format a dictionary as x and labels.\n # the dictionary have keys which should obv match","metadata":{"id":"AGACBcfCWC2O","execution":{"iopub.status.busy":"2022-12-03T01:21:06.890012Z","iopub.execute_input":"2022-12-03T01:21:06.890357Z","iopub.status.idle":"2022-12-03T01:21:06.896624Z","shell.execute_reply.started":"2022-12-03T01:21:06.890324Z","shell.execute_reply":"2022-12-03T01:21:06.895422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let the Data Flow: Creating the final input pipeline using `tf.data`","metadata":{"id":"dhdO6MjTbtn1"}},{"cell_type":"code","source":"# Now we will simply apply the transformation to our train and test datasets\nwith tf.device('/cpu:0'):\n # train\n train_data = (train_data.map(create_feature_map,\n num_parallel_calls=tf.data.experimental.AUTOTUNE)\n\n .shuffle(1000)\n .batch(32, drop_remainder=True)\n .prefetch(tf.data.experimental.AUTOTUNE))\n\n # valid\n valid_data = (valid_data.map(create_feature_map, \n num_parallel_calls=tf.data.experimental.AUTOTUNE)\n .batch(32, drop_remainder=True)\n .prefetch(tf.data.experimental.AUTOTUNE)) ","metadata":{"id":"LHRdiO3dnPNr","execution":{"iopub.status.busy":"2022-12-03T01:21:13.159268Z","iopub.execute_input":"2022-12-03T01:21:13.159601Z","iopub.status.idle":"2022-12-03T01:21:13.259578Z","shell.execute_reply.started":"2022-12-03T01:21:13.159569Z","shell.execute_reply":"2022-12-03T01:21:13.258676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The resulting `tf.data.Datasets` return `(features, labels)` pairs, as expected by [`keras.Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit):","metadata":{"id":"KLUWnfx-YDi2"}},{"cell_type":"code","source":"# train data spec, we can finally see the input datapoint is now converted to the BERT specific input tensor\ntrain_data.element_spec","metadata":{"id":"B0Z2cy9GHQ8x","outputId":"236d77a8-48b5-4377-b613-6d57f874578e","execution":{"iopub.status.busy":"2022-12-03T01:21:23.713182Z","iopub.execute_input":"2022-12-03T01:21:23.713534Z","iopub.status.idle":"2022-12-03T01:21:23.719539Z","shell.execute_reply.started":"2022-12-03T01:21:23.713502Z","shell.execute_reply":"2022-12-03T01:21:23.718677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# valid data spec\nvalid_data.element_spec","metadata":{"id":"DGAH-ycYOmao","outputId":"0c5c836e-f92c-47d4-d62e-10abf4d02ed9","execution":{"iopub.status.busy":"2022-12-03T01:21:29.132183Z","iopub.execute_input":"2022-12-03T01:21:29.132513Z","iopub.status.idle":"2022-12-03T01:21:29.139401Z","shell.execute_reply.started":"2022-12-03T01:21:29.132480Z","shell.execute_reply":"2022-12-03T01:21:29.138180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Lets Model Our Way to Glory!!!","metadata":{"id":"GZxe-7yhPyQe"}},{"cell_type":"markdown","source":"## Create The Model","metadata":{"id":"fB6vR1KHIWrv"}},{"cell_type":"markdown","source":"There are two outputs from the BERT Layer: \n- A pooled_output of shape [batch_size, 768] with representations for the entire input sequences. \n- A sequence_output of shape [batch_size, max_seq_length, 768] with representations for each input token (in context).\n\nFor the classification task, we are only concerned with the pooled_output.","metadata":{"id":"e33vaIFW1yil"}},{"cell_type":"code","source":"# Building the model, input ---> BERT Layer ---> Classification Head\ndef create_model():\n \n input_word_ids = tf.keras.layers.Input(shape=(config['max_seq_length'],), dtype=tf.int32,\n name=\"input_word_ids\")\n input_mask = tf.keras.layers.Input(shape=(config['max_seq_length'],), dtype=tf.int32,\n name=\"input_mask\")\n input_type_ids = tf.keras.layers.Input(shape=(config['max_seq_length'],), dtype=tf.int32,\n name=\"input_type_ids\")\n\n\n pooled_output, sequence_output = bert_layer([input_word_ids, input_mask, input_type_ids])\n # for classification we only care about the pooled-output\n # at this point we can play around with the classification head based on the downstream tasks and its complexity\n\n drop = tf.keras.layers.Dropout(config['dropout'])(pooled_output)\n output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(drop)\n\n # inputs coming from the function\n model = tf.keras.Model(\n inputs={\n 'input_word_ids': input_word_ids,\n 'input_mask': input_mask,\n 'input_type_ids': input_type_ids}, \n outputs=output)\n\n return model\n ","metadata":{"id":"G9il4gtlADcp","execution":{"iopub.status.busy":"2022-12-03T01:21:41.866417Z","iopub.execute_input":"2022-12-03T01:21:41.866760Z","iopub.status.idle":"2022-12-03T01:21:41.875374Z","shell.execute_reply.started":"2022-12-03T01:21:41.866726Z","shell.execute_reply":"2022-12-03T01:21:41.874257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let Us Train !","metadata":{"id":"S6maM-vr7YaJ"}},{"cell_type":"code","source":"# Calling the create model function to get the keras based functional model\nmodel = create_model()","metadata":{"id":"ptCtiiONsBgo","execution":{"iopub.status.busy":"2022-12-03T01:21:49.488612Z","iopub.execute_input":"2022-12-03T01:21:49.488953Z","iopub.status.idle":"2022-12-03T01:21:50.094766Z","shell.execute_reply.started":"2022-12-03T01:21:49.488914Z","shell.execute_reply":"2022-12-03T01:21:50.093891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# using adam with a lr of 2*(10^-5), loss as binary cross entropy as only 2 classes and similarly binary accuracy\nmodel.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=config['learning_rate']),\n loss=tf.keras.losses.BinaryCrossentropy(),\n metrics=[tf.keras.metrics.BinaryAccuracy(),\n tf.keras.metrics.PrecisionAtRecall(0.5),\n tf.keras.metrics.Precision(),\n tf.keras.metrics.Recall()])\nmodel.summary()","metadata":{"id":"6GJaFnkbMtPL","outputId":"293a795e-096b-4fcf-a3d3-73e013f79f12","execution":{"iopub.status.busy":"2022-12-03T01:21:57.763269Z","iopub.execute_input":"2022-12-03T01:21:57.763597Z","iopub.status.idle":"2022-12-03T01:21:57.809233Z","shell.execute_reply.started":"2022-12-03T01:21:57.763565Z","shell.execute_reply":"2022-12-03T01:21:57.808401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One drawback of the tf hub is that we import the entire module as a layer in keras as a result of which we dont see the parameters and layers in the model summary.","metadata":{"id":"fhhVabKn23LY"}},{"cell_type":"code","source":"tf.keras.utils.plot_model(model=model, show_shapes=True, dpi=76, )","metadata":{"id":"WDwqsmSV4l4N","outputId":"414388f1-9f22-49fd-fb63-8df025a0ec4a","execution":{"iopub.status.busy":"2022-12-03T01:23:16.070759Z","iopub.execute_input":"2022-12-03T01:23:16.071143Z","iopub.status.idle":"2022-12-03T01:23:16.623768Z","shell.execute_reply.started":"2022-12-03T01:23:16.071109Z","shell.execute_reply":"2022-12-03T01:23:16.622673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The official tfhub page states that\n\"All parameters in the module are trainable, and fine-tuning all parameters is the recommended practice.\" Therefore we will go ahead and train teh entire model without freezing anything","metadata":{"id":"82ye_lKZ3OA_"}},{"cell_type":"markdown","source":"#### W&B Experiment Tracking\nIn order to start the expirment tracking, we will be creating 'runs' on W&B, \n\n`wandb.init()`: It initializes the run with basic project information\nparameters: \n- project: The project name, this will create a new project tab where all the experiments for this project will be tracked\n- config: A dictionary of all parameters and hyper-parameters we wish to track\n- group: optional, but would help us to group by different parameters later on\n- job_type: to describe the job type, it would help in grouping different experiments later. eg \"train\", \"evaluate\" etc","metadata":{}},{"cell_type":"code","source":"# Update CONFIG dict with the name of the model.\nconfig['model_name'] = 'BERT_EN_UNCASED'\nprint('Training configuration: ', config)\n\n# Initialize W&B run\nrun = wandb.init(project='Finetune-BERT-Text-Classification', \n config=config,\n group='BERT_EN_UNCASED', \n job_type='train')","metadata":{"execution":{"iopub.status.busy":"2022-12-03T01:23:27.776890Z","iopub.execute_input":"2022-12-03T01:23:27.777282Z","iopub.status.idle":"2022-12-03T01:23:32.804433Z","shell.execute_reply.started":"2022-12-03T01:23:27.777246Z","shell.execute_reply":"2022-12-03T01:23:32.803413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to Log all the different metrics we will use a simple callback provided by W&B\n\n`WandCallback()` : https://docs.wandb.ai/guides/integrations/keras\n\nYes, Its as simple as adding a callback :D","metadata":{}},{"cell_type":"code","source":"# Train model\n# setting low epochs as It starts to overfit with this limited data, please feel free to change\nepochs = config['epochs']\nhistory = model.fit(train_data,\n validation_data=valid_data,\n epochs=epochs,\n verbose=1,\n callbacks = [WandbCallback()])\nrun.finish()","metadata":{"id":"OcREcgPUHr9O","outputId":"c123d1a0-3c0c-4465-ce15-bf1e7b3c82c4","execution":{"iopub.status.busy":"2022-12-03T01:24:30.360492Z","iopub.execute_input":"2022-12-03T01:24:30.360852Z","iopub.status.idle":"2022-12-03T02:01:31.884740Z","shell.execute_reply.started":"2022-12-03T01:24:30.360817Z","shell.execute_reply":"2022-12-03T02:01:31.883965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets Evaluate\n\nLet us do an evaluation on the validation set and log the scores using weights and biases.\n\n**wandb.log()**: Log a dict of scalars (metrics like accuracy and loss) and any other type of wandb object.\nHere we will pass the evaluation dictionary as it is and log it.","metadata":{}},{"cell_type":"code","source":"# Initialize a new run for the evaluation-job\nrun = wandb.init(project='Finetune-BERT-Text-Classification', \n config=config,\n group='BERT_EN_UNCASED', \n job_type='evaluate')\n\n\n\n# Model Evaluation on validation set\nevaluation_results = model.evaluate(valid_data,return_dict=True)\n\n# Log scores using wandb.log()\nwandb.log(evaluation_results)\n\n# Finish the run\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2022-12-03T02:04:08.384280Z","iopub.execute_input":"2022-12-03T02:04:08.384609Z","iopub.status.idle":"2022-12-03T02:04:18.657184Z","shell.execute_reply.started":"2022-12-03T02:04:08.384578Z","shell.execute_reply":"2022-12-03T02:04:18.656374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets Look at some Graphs\n\nThese Graphs will mainly be useful when we are training for more epochs and more data.\nAll these graphs are actually directly logged on the [wandb dashboard](https://wandb.ai/akshayuppal12/Finetune-BERT-Text-Classification/runs/29thnm00?workspace=user-akshayuppal12) created for this run. To still give out a method to generate graphs explicitly here is some very basic code.","metadata":{"id":"Hnb8cCI4KQLe"}},{"cell_type":"code","source":"\ndef create_graphs(history):\n train_accuracy = history.history['binary_accuracy']\n val_accuracy = history.history['val_binary_accuracy'],\n train_losses = history.history['loss'],\n val_losses = history.history['val_loss']\n fig1 = plt.figure()\n plt.plot(train_accuracy, 'r', linewidth=3.0, marker='o')\n plt.plot(val_accuracy, 'b', linewidth=3.0, marker='o')\n plt.legend(['Training Accuracy', 'Validation Accuracy'], fontsize=18)\n plt.xlabel('Epochs ', fontsize=16)\n plt.ylabel('Accuracy', fontsize=16)\n plt.title('Accuracy Curves', fontsize=16)\n plt.show()\n\n fig2 = plt.figure()\n plt.plot(train_losses, 'r', linewidth=3.0, marker='o')\n plt.plot(val_losses, 'b', linewidth=3.0, marker='o')\n plt.legend(['Training loss', 'Validation Loss'], fontsize=18)\n plt.xlabel('Epochs ', fontsize=16)\n plt.ylabel('Loss', fontsize=16)\n plt.title('Loss Curves', fontsize=16)\n plt.show()","metadata":{"id":"IttJTcQnI5aL","execution":{"iopub.status.busy":"2022-12-03T02:05:26.150778Z","iopub.execute_input":"2022-12-03T02:05:26.151136Z","iopub.status.idle":"2022-12-03T02:05:26.158250Z","shell.execute_reply.started":"2022-12-03T02:05:26.151104Z","shell.execute_reply":"2022-12-03T02:05:26.157342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we just trained for a very limited data and less epoch these graphs generated here are not as \nrepresenetative, so leaving this here more interms of a place holder for the reader to experiment with.","metadata":{}},{"cell_type":"code","source":"#create_graphs(history)","metadata":{"id":"KXmTwmQlJ_y4","outputId":"fa20263d-4059-4312-9e32-29ae00e43e5c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Saving the models and model Versioning","metadata":{}},{"cell_type":"markdown","source":"#### W&B Artifacts\nFor saving the models and making it easier to track different experiments, I will be using wandb.artifacts.\nW&B Artifacts are a way to save your datasets and models.\n\nWithin a run, there are three steps for creating and saving a model Artifact.\n\n- Create an empty Artifact with wandb.Artifact().\n- Add your model file to the Artifact with wandb.add_file().\n- Call wandb.log_artifact() to save the Artifact\n","metadata":{}},{"cell_type":"code","source":"# Save model\nmodel.save(f\"{config['model_name']}.h5\")\n\n# Initialize a new W&B run for saving the model, changing the job_type\nrun = wandb.init(project='Finetune-BERT-Text-Classification', \n config=config,\n group='BERT_EN_UNCASED', \n job_type='save')\n\n\n# Save model as Model Artifact\nartifact = wandb.Artifact(name=f\"{config['model_name']}\", type='model')\nartifact.add_file(f\"{config['model_name']}.h5\")\nrun.log_artifact(artifact)\n\n# Finish W&B run\nrun.finish()","metadata":{"execution":{"iopub.status.busy":"2022-12-03T02:06:18.624417Z","iopub.execute_input":"2022-12-03T02:06:18.624747Z","iopub.status.idle":"2022-12-03T02:06:50.121601Z","shell.execute_reply.started":"2022-12-03T02:06:18.624714Z","shell.execute_reply":"2022-12-03T02:06:50.120778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[Artifact dashboard](https://wandb.ai/akshayuppal12/Finetune-BERT-Text-Classification/artifacts/model/BERT_EN_UNCASED/48ffa3e14aba242a5113): Model versioning and more.","metadata":{}},{"cell_type":"markdown","source":"### Quick Sneak Peek into the W&B Dashboard\n\n**Things to note:**\n- Grouping of experiments and runs.\n- Visualizations of all training logs and metrics.\n- Visualizations for system metrics, could be useful when training on cloud instances or physical GPU machines\n- Hyperparmeter tracking in the tabular form.\n- Artifacts: Model versioning and storage.\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"Hopefully, this was useful for you, and by now, you have a small kickstart on training and utilizing BERT for a variety of downstream tasks like classification, Named Entity Recognition, Sentence filling, and many more.\n\nYou can check out and get the entire code as a notebook and run it on colab from this [Github Repo](https://github.com/au1206/Fine_Tuning_BERT).\n\nIf this was helpful, consider sharing it with more people so they can also learn about it.\n\nComing up Next:\n- BERT Annotated Paper\n- Write up on Transformers and its workings \n\nFor some annotated, reader-friendly research papers on advanced concepts and tutorials like these please visit https://au1206.github.io/.\n\nIf you made it this far, please consider leaving feedback so I can improve and also if you liked it consider upvoting. \n\n(UPDATE): [BERT Annotated Paper and Beyond...](https://au1206.github.io/annotated%20paper/BERT/)","metadata":{"id":"67dhLzBoKDSR"}},{"cell_type":"markdown","source":"### References:\nW&B usage and intro: https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases ","metadata":{}}]}