{"cells":[{"metadata":{},"cell_type":"markdown","source":"# BERT Embeddings with TensorFlow 2.0\nWith the new release of TensorFlow, this Notebook aims to show a simple use of the BERT model.\n- See BERT on paper: https://arxiv.org/pdf/1810.04805.pdf\n- See BERT on GitHub: https://github.com/google-research/bert\n- See BERT on TensorHub: https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1\n- See 'old' use of BERT for comparison: https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb"},{"metadata":{},"cell_type":"markdown","source":"## Update TF\nWe need Tensorflow 2.0 and TensorHub 0.7 for this Notebook"},{"metadata":{},"cell_type":"markdown","source":"If TensorFlow Hub is not 0.7 yet on release, use dev.\n\n**Disclaimer**: bert-tensorflow is not the latest version of BERT. To use bert with TF 2.0, you should use tensorflow/models where the model is updated to tf2.0. To resolve this issue, I'll use `tf.gfile = tf.io.gfile` at one point of the code. If you use Google Colab, it uses the latest bert version (no need for `pip install bert-tensorflow`), but I couldn't reproduce the same in Kaggle\n\nIf you know, how to install module from here tf/models, please share in comment!\nThe latest BERT at tf/models: https://github.com/tensorflow/models/tree/master/official/nlp/bert"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install --upgrade tensorflow\n!pip install bert-tensorflow\n!pip install tf-hub-nightly","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow_hub as hub\nprint(\"TF version: \", tf.__version__)\nprint(\"Hub version: \", hub.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import modules"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow_hub as hub\nimport tensorflow as tf\nfrom bert.tokenization import FullTokenizer     # Still from bert module\nfrom tensorflow.keras.models import Model       # Keras is the new high level API for TensorFlow\nimport math","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building model using tf.keras and hub. from sentences to embeddings.\n\nInputs:\n - input token ids (tokenizer converts tokens using vocab file)\n - input masks (1 for useful tokens, 0 for padding)\n - segment ids (for 2 text training: 0 for the first one, 1 for the second one)\n\nOutputs:\n - pooled_output of shape `[batch_size, 768]` with representations for the entire input sequences\n - sequence_output of shape `[batch_size, max_seq_length, 768]` with representations for each input token (in context)"},{"metadata":{"trusted":true},"cell_type":"code","source":"max_seq_length = 1  # Your choice here.\ninput_word_ids = tf.keras.layers.Input(shape=(max_seq_length,),\n                                       dtype=tf.int32,\n                                       name=\"input_word_ids\")\ninput_mask = tf.keras.layers.Input(shape=(max_seq_length,),\n                                   dtype=tf.int32,\n                                   name=\"input_mask\")\nsegment_ids = tf.keras.layers.Input(shape=(max_seq_length,),\n                                    dtype=tf.int32,\n                                    name=\"segment_ids\")\nbert_layer = hub.KerasLayer(\"https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1\",\n                            trainable=True)\npooled_output, sequence_output = bert_layer([input_word_ids, input_mask, segment_ids])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Model(inputs=[input_word_ids, input_mask, segment_ids], outputs=[pooled_output, sequence_output])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generating segments and masks based on the original BERT"},{"metadata":{"trusted":true},"cell_type":"code","source":"# See BERT paper: https://arxiv.org/pdf/1810.04805.pdf\n# And BERT implementation convert_single_example() at https://github.com/google-research/bert/blob/master/run_classifier.py\n\ndef get_masks(tokens, max_seq_length):\n    \"\"\"Mask for padding\"\"\"\n    if len(tokens)>max_seq_length:\n        raise IndexError(\"Token length more than max seq length!\")\n    return [1]*len(tokens) + [0] * (max_seq_length - len(tokens))\n\n\ndef get_segments(tokens, max_seq_length):\n    \"\"\"Segments: 0 for the first sequence, 1 for the second\"\"\"\n    if len(tokens)>max_seq_length:\n        raise IndexError(\"Token length more than max seq length!\")\n    segments = []\n    current_segment_id = 0\n    for token in tokens:\n        segments.append(current_segment_id)\n        if token == \"[SEP]\":\n            current_segment_id = 1\n    return segments + [0] * (max_seq_length - len(tokens))\n\n\ndef get_ids(tokens, tokenizer, max_seq_length):\n    \"\"\"Token ids from Tokenizer vocab\"\"\"\n    token_ids = tokenizer.convert_tokens_to_ids(tokens)\n    input_ids = token_ids + [0] * (max_seq_length-len(token_ids))\n    return input_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Google Colab don't need this. FullTokenizer is not updated to tf2.0 yet\ntf.gfile = tf.io.gfile","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Import tokenizer using the original vocab file"},{"metadata":{"trusted":true},"cell_type":"code","source":"vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\ndo_lower_case = bert_layer.resolved_object.do_lower_case.numpy()\ntokenizer = FullTokenizer(vocab_file, do_lower_case)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test BERT embedding generator model"},{"metadata":{"trusted":true},"cell_type":"code","source":"s0 = \"nice\"\nstokens0 = tokenizer.tokenize(s0)\n\ninput_ids0 = get_ids(stokens0, tokenizer, max_seq_length)\ninput_masks0 = get_masks(stokens0, max_seq_length)\ninput_segments0 = get_segments(stokens0, max_seq_length)\n\nprint(stokens0)\nprint(input_ids0)\nprint(input_masks0)\nprint(input_segments0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s1 = \"nasty\"\nstokens1 = tokenizer.tokenize(s1)\n\ninput_ids1 = get_ids(stokens1, tokenizer, max_seq_length)\ninput_masks1 = get_masks(stokens1, max_seq_length)\ninput_segments1 = get_segments(stokens1, max_seq_length)\n\nprint(stokens1)\nprint(input_ids1)\nprint(input_masks1)\nprint(input_segments1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_ids0 = np.array(input_ids0)\ninput_masks0 = np.array(input_masks0)\ninput_segments0 = np.array(input_segments0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_ids1 = np.array(input_ids1)\ninput_masks1 = np.array(input_masks1)\ninput_segments1 = np.array(input_segments1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(input_masks0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(s0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generate Embeddings using the pretrained model"},{"metadata":{"trusted":true},"cell_type":"code","source":"pool_embs0, all_embs0 = model.predict([[input_ids0],[input_masks0],[input_segments0]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pool_embs1, all_embs1 = model.predict([[input_ids1],[input_masks1],[input_segments1]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pooled embedding vs [CLS] as sentence-level representation\n\nPreviously, the [CLS] token's embedding were used as sentence-level representation (see the original paper). However, here a pooled embedding were introduced. This part is a short comparison of the two embedding using cosine similarity"},{"metadata":{"trusted":true},"cell_type":"code","source":"def square_rooted(x):\n    return math.sqrt(sum([a*a for a in x]))\n\n\ndef cosine_similarity(x,y):\n    numerator = sum(a*b for a,b in zip(x,y))\n    denominator = square_rooted(x)*square_rooted(y)\n    return numerator/float(denominator)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cosine_similarity(pool_embs0[0], all_embs0[0][127])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cosine_similarity(pool_embs1[0], all_embs1[0][127])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cosine_similarity(pool_embs0[0],pool_embs1[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cosine_similarity(all_embs0[0][0],all_embs1[0][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_embs0[0][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pool_embs0.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pool_embs1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_embs1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_embs0.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(pool_embs[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pooled_output[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pool_embs0[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pool_embs0[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(pool_embs1[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(all_embs[0][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_embs[0][1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}