{"metadata":{"colab":{"provenance":[],"gpuType":"T4"},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"accelerator":"GPU","language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":10737,"databundleVersionId":290346,"sourceType":"competition"},{"sourceId":7722873,"sourceType":"datasetVersion","datasetId":4511453},{"sourceId":7724047,"sourceType":"datasetVersion","datasetId":4512304},{"sourceId":2644,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":1910}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":" Fine-Tuned BERT for Quora Insincere Questions Classification \n
","metadata":{"id":"zGCJYkQj_Uu2"}},{"cell_type":"markdown","source":"### (1) This notebook is based on Coursera Project \"Fine Tune BERT for Text Classification with TensorFlow\" Tutorial. Link to the Tutorial: https://www.coursera.org/projects/fine-tune-bert-tensorflow","metadata":{"id":"fhJY1a5DXCh3"}},{"cell_type":"markdown","source":"### (2) The pretrained BERT model used in this project is [available on Kaggle](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2).","metadata":{"id":"5yQG5PCO_WFx"}},{"cell_type":"markdown","source":"### (3) The notebook has been trained in the Colab environment because I faced dependency issues on Kaggle which I could not fix. Link to Colab notebook: https://colab.research.google.com/drive/19XwqXJxiGWyR97Q5BDXmqdeVByXZnSi2?usp=sharing\n### After training the BERT model on Colab, I have used the fine-tuned BERT model to generate the predictions on Test data within the same Colab environment only. The generated predictions have been attached as an input file.\n\n___","metadata":{"id":"2NNVfqrND4xN"}},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":" ","metadata":{}},{"cell_type":"markdown","source":"## PART A: Code for BERT Model Training on Colab Tesla T4 GPU Runtime ","metadata":{"id":"RE7zcg5DD4xO"}},{"cell_type":"markdown","source":"\n
\n
\n
\n
\n
\n
\n**Input Mask** - Note that BERT uses Self-Attention Networks to provide\n contextualised embeddings corresponding to each token in the token string i.e., for each word in the string BERT looks to the left and right of it in the sentence so as to find contextual meaning of the word in the sentence (say, if there is a \"the\", then look at the noun to which it points). Now, note that we have padded our token strings with 0s upto the max seq length, but we do not want the padding 0s to influence the contextual information to be derived. The Input Mask is a list of same length as the length of Token Ids (ie the max seq length) where there is a 0 for a padding and 1 for a valid token. The 0s will cancel out the internal multiplications that we perform for capturing the Self Attention for contextual information.
\n\n**Input Type IDs** - Note that originally BERT was pretrained on two tasks, Masked Language Modelling (where random words from the sentence would be masked and it would be the task for the BERT to predict what those masked words are) and the other task was Next Sentence Prediction or NSP (Given two sentences, the BERT has to predict which came first and which came after. The first sentence was given the value 0 and the next sentence was given the value 1).
\n**In Text classification, we are dealing with only 1 sequence at a time, so our input type IDs would just be a vector with all values 0. **\n","metadata":{"id":"shyvv_0JaIzj"}},{"cell_type":"code","source":"# This provides a function to convert row to input features and label\n\ndef to_feature(text, label, label_list=label_list, max_seq_length=max_seq_length, tokenizer=tokenizer):\n example = classifier_data_lib.InputExample(guid = None,\n text_a = text.numpy(),\n text_b = None, # 1 for NSP and None for text classification\n label = label.numpy())\n feature = classifier_data_lib.convert_single_example(0, example, label_list, max_seq_length, tokenizer) # we are converting the 0th example\n return (feature.input_ids, feature.input_mask, feature.segment_ids, feature.label_id)\n\n #returning the Token IDs, Input Masks, Input type IDs and Labels","metadata":{"id":"m-21A5aNJM0W"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You want to use [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) to apply this function to each element of the dataset. [`Dataset.map`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) runs in graph mode.\n\n- Graph tensors do not have a value.\n- In graph mode you can only use TensorFlow Ops and functions.\n\nSo you can't `.map` this function directly: You need to wrap it in a [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function). The [`tf.py_function`](https://www.tensorflow.org/api_docs/python/tf/py_function) will pass regular tensors (with a value and a `.numpy()` method to access it), to the wrapped python function.","metadata":{"id":"A_HQSsHwWCsK"}},{"cell_type":"markdown","source":"### 7. Wrap a Python Function into a TensorFlow op for Eager Execution","metadata":{"id":"zaNlkKVfWX0Q"}},{"cell_type":"code","source":"def to_feature_map(text, label):\n input_ids, input_mask, segment_ids, label_id = tf.py_function(to_feature, inp = [text, label],\n Tout = [tf.int32, tf.int32, tf.int32, tf.int32]) # stands for Tensor output type\n input_ids.set_shape([max_seq_length])\n input_mask.set_shape([max_seq_length])\n segment_ids.set_shape([max_seq_length])\n label_id.set_shape([])\n\n x = {\n 'input_word_ids': input_ids,\n 'input_mask': input_mask,\n 'input_type_ids': segment_ids\n }\n\n return [x, label_id]","metadata":{"id":"AGACBcfCWC2O"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8. Create a TensorFlow Input Pipeline with `tf.data`","metadata":{"id":"dhdO6MjTbtn1"}},{"cell_type":"code","source":"with tf.device('/cpu:0'):\n # train\n train_data = (train_data.map(to_feature_map,\n num_parallel_calls = tf.data.experimental.AUTOTUNE) # The dataset mapping can be parallelised\n # over the CPU cores available, but we do not know\n # how many CPU cores are available, so we let Tf find it out!\n\n .shuffle(1000).batch(32, drop_remainder=True) # If there are more samples left afterwards, drop them\n # so that all batches are equal size\n\n .prefetch(tf.data.experimental.AUTOTUNE)) # when model is training on nth batch, prefetch\n # next batch of input to save time\n\n # valid\n valid_data = (valid_data.map(to_feature_map,\n num_parallel_calls = tf.data.experimental.AUTOTUNE)\n .batch(32, drop_remainder=True)\n .prefetch(tf.data.experimental.AUTOTUNE))","metadata":{"id":"LHRdiO3dnPNr"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The resulting `tf.data.Datasets` return `(features, labels)` pairs, as expected by [`keras.Model.fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit):","metadata":{"id":"KLUWnfx-YDi2"}},{"cell_type":"code","source":"# train data spec\ntrain_data.element_spec","metadata":{"id":"B0Z2cy9GHQ8x","outputId":"b0a414ca-d93d-4e02-918c-97ac3c9f72a4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# valid data spec\nvalid_data.element_spec","metadata":{"id":"DGAH-ycYOmao","outputId":"9abc2cbb-8670-47e4-e263-fd09805eedfa"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9. Add a Classification Head to the BERT Layer","metadata":{"id":"GZxe-7yhPyQe"}},{"cell_type":"markdown","source":"\n
\n
\n