{"cells":[{"metadata":{},"cell_type":"markdown","source":"This is a two part Notebook\n\nNOTE TO SELF: TF2.0 BASELINE W/ BERT NEEDS TO BE CHANGED FIRST\n\n1. Comprehensive BERT Tutorial
\n2. Implementation in Tensorflow 2.0\n
\n\n> **Note:** The main objective of this notebook is to provide a **baseline for this competition with some explanation about BERT**. I decided to wite such a notebook because I didn't find anything quite like this when I started out at NLP Competitions. I hope beginners can benefit from this notebook. Even if you're a non-beginner there might be some elements in this notebook you may be interested in.\n\n
"},{"metadata":{},"cell_type":"markdown","source":"If you like this approach please give this kernel an UPVOTE to show your appreciation"},{"metadata":{},"cell_type":"markdown","source":"# Comprehensive BERT Tutorial\n\n## Introduction\nSo if you're like me just starting out at NLP after spending a few months building Computer Vision models as a beginner then surely this kernel has something in store for you.\n
\n\nSo if you're like me just starting out at NLP after spending a few months building Computer Vision models as a beginner then surely this kernel has something in store for you.\n\nI had a hard time wrapping my head around this all new bleeding-edge, state-of-the-art NLP model BERT, I had to dig through a lot of articles to truly grasp what BERT is all about, I'll share my understanding of BERT in this notebook.\n\n\n\n## References and Credits:\nThis notebook wouldn't have been possible without these amazing resources. Most of the text and figures used in this notebooks are taken from the below mentioned resources, combining everything into one.\n1. [BERT for Dummies step by step tutorial by Michel Kana](https://towardsdatascience.com/bert-for-dummies-step-by-step-tutorial-fb90890ffe03)\n2. [Demystifying BERT: Groundbreaking NLP Framework by Mohd Sanad Zaki Rizvi](https://www.analyticsvidhya.com/blog/2019/09/demystifying-bert-groundbreaking-nlp-framework/)\n3. [A visual guide to using BERT by Jay Alammar](http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/)\n4. [BERT Fine tuning By Chris McCormick and Nick Ryan](https://mccormickml.com/2019/07/22/BERT-fine-tuning/)\n5. [How to use BERT in Kaggle competitions - Reddit Thread](https://www.reddit.com/r/MachineLearning/comments/ao23cp/p_how_to_use_bert_in_kaggle_competitions_a/)\n6. [BERT GitHub repository](https://github.com/google-research/bert)\n7. [BERT - SOTA NLP model Explained by Rani Horev](https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html)\n8. [YOUTUBE - BERT Pretranied Deep Bidirectional Transformers for Language Understanding algorithm by Danny Luo](https://www.youtube.com/watch?v=BhlOGGzC0Q0)\n9. [State-of-the-art pre-training for natural language processing with BERT by Javed Quadrud-Din](https://blog.insightdatascience.com/using-bert-for-state-of-the-art-pre-training-for-natural-language-processing-1d87142c29e7)\n10. [BERT-Explained FAQs by Yashu Seth](https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/)\n\n## Contents\n1. The BERT Landscape \n2. What is BERT? \n3. Why BERT Matters?\n
\n4. How BERT works?
\n 4.1 Architecture of BERT \n 4.2 Preprocessing text for BERT \n 4.3 Pre-training \n5. Fine Tuning Techniques for BERT
\n 5.1 Sequence Classification Tasks \n 5.2 Sentence Pair Classification Tasks \n 5.3 Question Answering Tasks
\n 5.4 Single Sentence Tagging Tasks
\n 5.5 Hyperparameter Tuning
\n6. BERT Benchmarks on Question/Answering Tasks
\n7. Key Takeaways
\n8. Conclusion"},{"metadata":{},"cell_type":"markdown","source":"## 1. The BERT Landscape\n> BERT is a deep learning model that has given state-of-the-art results on a wide variety of natural language processing tasks. It stands for **Bidirectional Encoder Representations for Transformers**. It has been pre-trained on Wikipedia and BooksCorpus and requires (only) task-specific fine-tuning.\n\n It has caused a stir in the Machine Learning community by presenting state-of-the-art results in a wide variety of NLP tasks, including **Question Answering (SQuAD v1.1)**, **Natural Language Inference (MNLI)**, and others.\n \n It’s not an exaggeration to say that BERT has significantly altered the NLP landscape. Imagine using a single model that is trained on a large unlabelled dataset to achieve State-of-the-Art results on 11 individual NLP tasks. And all of this with little fine-tuning. That’s BERT! It’s a tectonic shift in how we design NLP models.\n\nBERT has inspired many recent NLP architectures, training approaches and language models, such as Google’s TransformerXL, OpenAI’s GPT-2, XLNet, ERNIE2.0, RoBERTa, etc."},{"metadata":{},"cell_type":"markdown","source":"## 2. What is BERT?\nIt is basically a bunch of Transformer encoders stacked together (not the whole Transformer architecture but just the encoder). The concept of bidirectionality is the key differentiator between BERT and its predecessor, OpenAI GPT. BERT is bidirectional because its self-attention layer performs self-attention on both directions.\n\nThere are a few things I want to explain in this section.\n\n* First, It’s easy to get that BERT stands for Bidirectional Encoder Representations from Transformers. Each word here has a meaning to it and we will encounter that one by one. For now, **the key takeaway from this line is – BERT is based on the Transformer architecture.** \n\n* Second, BERT is **pre-trained on a large corpus of unlabelled text** including the entire Wikipedia(that’s 2,500 million words!) and Book Corpus (800 million words). This pretraining step is really important for BERT's success. This is because as we train a model on a large text corpus, our model starts to pick up the deeper and intimate understandings of how the language works. This knowledge is the swiss army knife that is useful for almost any NLP task.\n\n* Third, BERT is a **deeply bidirectional** model. Bidirectional means that BERT learns information from both the left and the right side of a token’s context during the training phase.\n\nThis bidirectional understanding is crucial to take NLP models to the next level. Let's see an example to understand what it really means. There may be two sentences having the same word but their meaning may be completely different based on what comes before or after as we can see here below.\n\n\n\nWithout taking these contexts into consideration it's impossible for machines to truly understand meanings and it may throw out trashy responses time and time again which is not really a good thing.\n\nBut BERT fixes this. Yes it does. That was one of the game changing aspect of BERT.\n\n* Fourth, finally the biggest advantage of BERT is it brought about the **ImageNet movement** with it and the most impressive aspect of BERT is that we can fine-tune it by adding just a couple of additional output layers to create state-of-the-art models for a variety of NLP tasks.\n"},{"metadata":{},"cell_type":"markdown","source":"## 3. Why BERT matters?\n\nNow I think it's pretty clear to you why but let's see proof, as we should always do.\n\n\n\nWhile it’s not clear that all GLUE tasks are very meaningful, generic models based on an encoder named Transformer (Open-GPT, BERT and BigBird), closed the gap between task-dedicated models and human performance and within less than a year."},{"metadata":{},"cell_type":"markdown","source":"## 4. How BERT Works?\nLet’s look a bit closely at BERT and understand why it is such an effective method to model language. We’ve already seen what BERT can do earlier – but how does it do it? We’ll answer this pertinent question in this section.\n\n### 1. Architecture of BERT\n\nBERT is a multi-layer bidirectional Transformer encoder. There are two models introduced in the paper.\n\n* BERT base – 12 layers (transformer blocks), 12 attention heads, and 110 million parameters.\n* BERT Large – 24 layers, 16 attention heads and, 340 million parameters.\n\nFor an in-depth understanding of the building blocks of BERT (aka Transformers), you should definitely check [this awesome post](http://jalammar.github.io/illustrated-transformer/) – The Illustrated Transformers.\n\n*Here's a representation of BERT Architecture*\n\n\n### 2. Preprocessing Text for BERT\nThe input representation used by BERT is able to represent a single text sentence as well as a pair of sentences (eg., Question, Answering) in a single sequence of tokens.\n\n* The first token of every input sequence is the special classification token – **[CLS]**. This token is used in classification tasks as an aggregate of the entire sequence representation. It is ignored in non-classification tasks.\n* For single text sentence tasks, this **[CLS]** token is followed by the WordPiece tokens and the separator token – **[SEP]**.\n\n\n\n* For sentence pair tasks, the WordPiece tokens of the two sentences are separated by another [SEP] token. This input sequence also ends with the **[SEP]** token.\n\n* A sentence embedding indicating Sentence A or Sentence B is added to each token. Sentence embeddings are similar to token/word embeddings with a vocabulary of 2.\n\n* A positional embedding is also added to each token to indicate its position in the sequence.\n\nBERT developers have set a a specific set of rules to represent languages before feeding into the model.\n\n\n\nFor starters, every input embedding is a combination of 3 embeddings:\n\n* **Position Embeddings**: BERT learns and uses positional embeddings to express the position of words in a sentence. These are added to overcome the limitation of Transformer which, unlike an RNN, is not able to capture “sequence” or “order” information\n* **Segment Embeddings**: BERT can also take sentence pairs as inputs for tasks (Question-Answering). That’s why it learns a unique embedding for the first and the second sentences to help the model distinguish between them. In the above example, all the tokens marked as EA belong to sentence A (and similarly for EB)\n* **Token Embeddings**: These are the embeddings learned for the specific token from the WordPiece token vocabulary\n\nFor a given token, its input representation is constructed by **summing the corresponding token, segment, and position embeddings**.\n\nSuch a comprehensive embedding scheme contains a lot of useful information for the model.\n\nThese combinations of preprocessing steps make BERT so versatile. This implies that without making any major change in the model’s architecture, we can easily train it on multiple kinds of NLP tasks.\n\n**Tokenization:**\nBERT uses WordPiece tokenization. The vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the existing words in the vocabulary are iteratively added.\n\n\n### 3. Pre Training\nThe model was trained in two tasks simultaneously:\n\n**1. Masked Language Model** \n\n**2. Next Sentence Prediction.**\n\n**Note:** I am not going to go over these two techniques in this notebook. I recommend online reading."},{"metadata":{},"cell_type":"markdown","source":"## 5. Fine Tuning Techniques for BERT\nUsing BERT for a specific task is relatively straightforward. BERT can be used for a wide variety of language tasks, while only adding a small layer to the core model\n\n### 5.1 Sequence Classification Tasks\nThe final hidden state of the **[CLS]** token is taken as the fixed-dimensional pooled representation of the input sequence. This is fed to the classification layer. The classification layer is the only new parameter added and has a dimension of K x H, where K is the number of classifier labels and H is the size of the hidden state. The label probabilities are computed with a standard softmax.\n\n\n\n### 5.2 Sentence Pair Classification Tasks\nThis procedure is exactly similar to the single sequence classification task. The only difference is in the input representation where the two sentences are concatenated together.\n\n\n\n### 5.3 Question-Answering Tasks (Goal of this competition)\nQuestion answering is a prediction task. Given a question and a context paragraph, the model predicts a start and an end token from the paragraph that most likely answers the question.\n\n\n\nJust like sentence pair tasks, the question becomes the first sentence and paragraph the second sentence in the input sequence. There are only two new parameters learned during fine-tuning a start vector and an end vector with size equal to the hidden shape size. The probability of token i being the start of the answer span is computed as – softmax(S . K), where S is the start vector and K is the final transformer output of token i. The same applies to the end token.\n\n\n\n### 5.4 Single Sentence Tagging Tasks\n\nIn single sentence tagging tasks such as named entity recognition, a tag must be predicted for every word in the input. The final hidden states (the transformer output) of every input token is fed to the classification layer to get a prediction for every token. Since WordPiece tokenizer breaks some words into sub-words, the prediction of only the first token of a word is considered.\n\n\n\n### 5.5 Hyperparameter Tuning\nThe optimal hyperparameter values are task-specific. But, the authors found that the following range of values works well across all tasks\n\n* **Dropout** – 0.1\n* **Batch Size** – 16, 32\n* **Learning Rate (Adam)** – 5e-5, 3e-5, 2e-5\n* **Number of epochs** – 3, 4 (yeah you read it right)\n\nThe authors also observed that large datasets (> 100k labeled samples) are less sensitive to hyperparameter choice than smaller datasets.\n"},{"metadata":{},"cell_type":"markdown","source":"## 6. BERT Benchmarks on Question Answering tasks\n\n> The Standford Question Answering Dataset (SQuAD) is a collection of 100k crowdsourced question/answer pairs (Rajpurkar et al., 2016). Given a question and a paragraph from Wikipedia containing the answer, the task is to predict the answer text span in the paragraph.\n\nIn SQUAD the big improvement in performance was achieved by BERT large. The model that achieved the highest score was an ensemble of BERT large models, augmenting the dataset with TriviaQA.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## 7. Key Takeaways\n\n> 1) Model size matters, even at huge scale. BERT_large, with 345 million parameters, is the largest model of its kind. It is demonstrably superior on small-scale tasks to BERT_base, which uses the same architecture with “only” 110 million parameters.\n\n> 2) With enough training data, more training steps == higher accuracy. For instance, on the MNLI task, the BERT_base accuracy improves by 1.0% when trained on 1M steps (128,000 words batch size) compared to 500K steps with the same batch size.\n\n> 3) BERT’s bidirectional approach (MLM) converges slower than left-to-right approaches (because only 15% of words are predicted in each batch) but bidirectional training still outperforms left-to-right training after a small number of pre-training steps.\n\n"},{"metadata":{},"cell_type":"markdown","source":"## 8. Conclusion\n\nBERT is undoubtedly a breakthrough in the use of Machine Learning for Natural Language Processing. The fact that it’s approachable and allows fast fine-tuning will likely allow a wide range of practical applications in the future. In this summary, we attempted to describe the main ideas of the paper while not drowning in excessive technical details. For those wishing for a deeper dive, we highly recommend reading the full article and ancillary articles referenced in it. \n\n> **Feel free to pass on any suggestion to improve this notebook in the comment section (if you have any)**"},{"metadata":{},"cell_type":"markdown","source":"Please give this kernel an UPVOTE to show your appreciation, if you find it useful."},{"metadata":{},"cell_type":"markdown","source":"# Code Implementation in Tensorflow 2.0\n\n> **Note:** The code for this notebook is taken from the [translated version](https://www.kaggle.com/dimitreoliveira/using-tf-2-0-w-bert-on-nq-translated-to-tf2-0) posted by [Dimitre Oliviera](https://www.kaggle.com/dimitreoliveira)"},{"metadata":{},"cell_type":"markdown","source":"**This is a translated version of the baseline [script](https://www.kaggle.com/philculliton/using-tensorflow-2-0-w-bert-on-nq) from the Tensorflow team**\n\n**Oliviera translated the script to the Tensorflow 2.0 version, this way we can take part in the TF2 prizes and may use the version to improve the work.**\n\n**A few notes:**\n- If you want to keep using **flags** and **logging** you will have to use the **absl** lib (this is recommended by the TF team).\n- Since we won't use it with the kernels, he removed most of the **TPU** related stuff to reduce complexity.\n- Tensorflow 2 don't let us use global variables **(tf.compat.v1.trainable_variables())**.\n- If you have experience with Tensorflow 2 or have any correction/improvement, please let him know.\n\n\nIn this notebook, we'll be using the Bert baseline for Tensorflow to create predictions for the Natural Questions test set. Note that this uses a model that has already been pre-trained - we're only doing inference here. A GPU is required, and this should take between 1-2 hours to run.\n\nThe original script can be found [here](https://github.com/google-research/language/blob/master/language/question_answering/bert_joint/run_nq.py).\nThe supporting modules were drawn from the [official Tensorflow model repository](https://github.com/tensorflow/models/tree/master/official). The bert-joint-baseline data is described [here](https://github.com/google-research/language/tree/master/language/question_answering/bert_joint).\n\n**Note:** This baseline uses code that was migrated from TF1.x. Be aware that it contains use of tf.compat.v1, which is not permitted to be eligible for [TF2.0 prizes in this competition](https://www.kaggle.com/c/tensorflow2-question-answering/overview/prizes). It is intended to be used as a starting point, but we're excited to see how much better you can do using TF2.0!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_hub as hub\n# import tf2_0_baseline_w_bert as tf2baseline # old script\nimport tf2_0_baseline_w_bert_translated_to_tf2_0 as tf2baseline # Oliviera's script\nimport bert_modeling as modeling\nimport bert_optimization as optimization\nimport bert_tokenization as tokenization\nimport json\nimport absl\nimport sys\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Tensorflow flags are variables that can be passed around within the TF system. Every flag below has some context provided regarding what the flag is and how it's used.**\n\n**Most of these can be changed as desired, with the exception of the Special Flags at the bottom, which _must_ stay as-is to work with the Kaggle back end.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load ALBERT model from TF_Hub\n\nalbert_module = hub.Module(\"https://tfhub.dev/google/albert_base/3\", trainable=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def del_all_flags(FLAGS):\n flags_dict = FLAGS._flags()\n keys_list = [keys for keys in flags_dict]\n for keys in keys_list:\n FLAGS.__delattr__(keys)\n\ndel_all_flags(absl.flags.FLAGS)\n\nflags = absl.flags\n\nflags.DEFINE_string(\n \"bert_config_file\", \"/kaggle/input/bertjointbaseline/bert_config.json\",\n \"The config json file corresponding to the pre-trained BERT model. \"\n \"This specifies the model architecture.\")\n\nflags.DEFINE_string(\"vocab_file\", \"/kaggle/input/bertjointbaseline/vocab-nq.txt\",\n \"The vocabulary file that the BERT model was trained on.\")\n\nflags.DEFINE_string(\n \"output_dir\", \"outdir\",\n \"The output directory where the model checkpoints will be written.\")\n\nflags.DEFINE_string(\"train_precomputed_file\", None,\n \"Precomputed tf records for training.\")\n\nflags.DEFINE_integer(\"train_num_precomputed\", None,\n \"Number of precomputed tf records for training.\")\n\nflags.DEFINE_string(\n \"output_prediction_file\", \"predictions.json\",\n \"Where to print predictions in NQ prediction format, to be passed to\"\n \"natural_questions.nq_eval.\")\n\nflags.DEFINE_string(\n \"init_checkpoint\", \"/kaggle/input/bertjointbaseline/bert_joint.ckpt\",\n \"Initial checkpoint (usually from a pre-trained BERT model).\")\n\nflags.DEFINE_bool(\n \"do_lower_case\", True,\n \"Whether to lower case the input text. Should be True for uncased \"\n \"models and False for cased models.\")\n\nflags.DEFINE_integer(\n \"max_seq_length\", 384,\n \"The maximum total input sequence length after WordPiece tokenization. \"\n \"Sequences longer than this will be truncated, and sequences shorter \"\n \"than this will be padded.\")\n\nflags.DEFINE_integer(\n \"doc_stride\", 128,\n \"When splitting up a long document into chunks, how much stride to \"\n \"take between chunks.\")\n\nflags.DEFINE_integer(\n \"max_query_length\", 64,\n \"The maximum number of tokens for the question. Questions longer than \"\n \"this will be truncated to this length.\")\n\nflags.DEFINE_bool(\"do_train\", False, \"Whether to run training.\")\n\nflags.DEFINE_bool(\"do_predict\", True, \"Whether to run eval on the dev set.\")\n\nflags.DEFINE_integer(\"train_batch_size\", 32, \"Total batch size for training.\")\n\nflags.DEFINE_integer(\"predict_batch_size\", 8,\n \"Total batch size for predictions.\")\n\nflags.DEFINE_float(\"learning_rate\", 5e-5, \"The initial learning rate for Adam.\")\n\nflags.DEFINE_float(\"num_train_epochs\", 3.0,\n \"Total number of training epochs to perform.\")\n\nflags.DEFINE_float(\n \"warmup_proportion\", 0.1,\n \"Proportion of training to perform linear learning rate warmup for. \"\n \"E.g., 0.1 = 10% of training.\")\n\nflags.DEFINE_integer(\"save_checkpoints_steps\", 1000,\n \"How often to save the model checkpoint.\")\n\nflags.DEFINE_integer(\"iterations_per_loop\", 1000,\n \"How many steps to make in each estimator call.\")\n\nflags.DEFINE_integer(\n \"n_best_size\", 20,\n \"The total number of n-best predictions to generate in the \"\n \"nbest_predictions.json output file.\")\n\nflags.DEFINE_integer(\n \"verbosity\", 1, \"How verbose our error messages should be\")\n\nflags.DEFINE_integer(\n \"max_answer_length\", 30,\n \"The maximum length of an answer that can be generated. This is needed \"\n \"because the start and end predictions are not conditioned on one another.\")\n\nflags.DEFINE_float(\n \"include_unknowns\", -1.0,\n \"If positive, probability of including answers of type `UNKNOWN`.\")\n\nflags.DEFINE_bool(\"use_tpu\", False, \"Whether to use TPU or GPU/CPU.\")\nflags.DEFINE_bool(\"use_one_hot_embeddings\", False, \"Whether to use use_one_hot_embeddings\")\n\nabsl.flags.DEFINE_string(\n \"gcp_project\", None,\n \"[Optional] Project name for the Cloud TPU-enabled project. If not \"\n \"specified, we will attempt to automatically detect the GCE project from \"\n \"metadata.\")\n\nflags.DEFINE_bool(\n \"verbose_logging\", False,\n \"If true, all of the warnings related to data processing will be printed. \"\n \"A number of warnings are expected for a normal NQ evaluation.\")\n\nflags.DEFINE_boolean(\n \"skip_nested_contexts\", True,\n \"Completely ignore context that are not top level nodes in the page.\")\n\nflags.DEFINE_integer(\"task_id\", 0,\n \"Train and dev shard to read from and write to.\")\n\nflags.DEFINE_integer(\"max_contexts\", 48,\n \"Maximum number of contexts to output for an example.\")\n\nflags.DEFINE_integer(\n \"max_position\", 50,\n \"Maximum context position for which to generate special tokens.\")\n\n\n## Special flags - do not change\n\nflags.DEFINE_string(\n \"predict_file\", \"/kaggle/input/tensorflow2-question-answering/simplified-nq-test.jsonl\",\n \"NQ json for predictions. E.g., dev-v1.1.jsonl.gz or test-v1.1.jsonl.gz\")\nflags.DEFINE_boolean(\"logtostderr\", True, \"Logs to stderr\")\nflags.DEFINE_boolean(\"undefok\", True, \"it's okay to be undefined\")\nflags.DEFINE_string('f', '', 'kernel')\nflags.DEFINE_string('HistoryManager.hist_file', '', 'kernel')\n\nFLAGS = flags.FLAGS\nFLAGS(sys.argv) # Parse the flags","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here, we:**\n1. Set up Bert\n2. Read in the test set\n3. Run it past the pre-built Bert model to create embeddings\n4. Use those embeddings to make predictions\n5. Write those predictions to `predictions.json`\n\nFeel free to change the code below. Code for the `tf2baseline.*` functions is included in the `tf2_0_baseline_w_bert` utility script, and can be customized, whether by forking the utility script and updating it, or by creating your own non-`tf2baseline` versions in this kernel.\n\nNote: the `tf2_0_baseline_w_bert` utility script contains code for training your own embeddings. Here that code is removed."},{"metadata":{"trusted":true},"cell_type":"code","source":"bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)\n\ntf2baseline.validate_flags_or_throw(bert_config)\ntf.io.gfile.makedirs(FLAGS.output_dir)\n\ntokenizer = tokenization.FullTokenizer(\n vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)\n\nrun_config = tf.estimator.RunConfig(\n model_dir=FLAGS.output_dir,\n save_checkpoints_steps=FLAGS.save_checkpoints_steps)\n\nnum_train_steps = None\nnum_warmup_steps = None\n\nmodel_fn = tf2baseline.model_fn_builder(\n bert_config=bert_config,\n init_checkpoint=FLAGS.init_checkpoint,\n learning_rate=FLAGS.learning_rate,\n num_train_steps=num_train_steps,\n num_warmup_steps=num_warmup_steps,\n use_tpu=FLAGS.use_tpu,\n use_one_hot_embeddings=FLAGS.use_one_hot_embeddings)\n\nestimator = tf.estimator.Estimator(\n model_fn=model_fn,\n config=run_config,\n params={'batch_size':FLAGS.train_batch_size})\n\n\nif FLAGS.do_predict:\n if not FLAGS.output_prediction_file:\n raise ValueError(\n \"--output_prediction_file must be defined in predict mode.\")\n \n eval_examples = tf2baseline.read_nq_examples(\n input_file=FLAGS.predict_file, is_training=False)\n\n print(\"FLAGS.predict_file\", FLAGS.predict_file)\n\n eval_writer = tf2baseline.FeatureWriter(\n filename=os.path.join(FLAGS.output_dir, \"eval.tf_record\"),\n is_training=False)\n eval_features = []\n\n def append_feature(feature):\n eval_features.append(feature)\n eval_writer.process_feature(feature)\n\n num_spans_to_ids = tf2baseline.convert_examples_to_features(\n examples=eval_examples,\n tokenizer=tokenizer,\n is_training=False,\n output_fn=append_feature)\n eval_writer.close()\n eval_filename = eval_writer.filename\n\n print(\"***** Running predictions *****\")\n print(f\" Num orig examples = %d\" % len(eval_examples))\n print(f\" Num split examples = %d\" % len(eval_features))\n print(f\" Batch size = %d\" % FLAGS.predict_batch_size)\n for spans, ids in num_spans_to_ids.items():\n print(f\" Num split into %d = %d\" % (spans, len(ids)))\n\n predict_input_fn = tf2baseline.input_fn_builder(\n input_file=eval_filename,\n seq_length=FLAGS.max_seq_length,\n is_training=False,\n drop_remainder=False)\n\n all_results = []\n\n for result in estimator.predict(\n predict_input_fn, yield_single_examples=True):\n if len(all_results) % 1000 == 0:\n print(\"Processing example: %d\" % (len(all_results)))\n\n unique_id = int(result[\"unique_ids\"])\n start_logits = [float(x) for x in result[\"start_logits\"].flat]\n end_logits = [float(x) for x in result[\"end_logits\"].flat]\n answer_type_logits = [float(x) for x in result[\"answer_type_logits\"].flat]\n\n all_results.append(\n tf2baseline.RawResult(\n unique_id=unique_id,\n start_logits=start_logits,\n end_logits=end_logits,\n answer_type_logits=answer_type_logits))\n\n print (\"Going to candidates file\")\n\n candidates_dict = tf2baseline.read_candidates(FLAGS.predict_file)\n\n print (\"setting up eval features\")\n\n raw_dataset = tf.data.TFRecordDataset(eval_filename)\n eval_features = []\n for raw_record in raw_dataset:\n eval_features.append(tf.train.Example.FromString(raw_record.numpy()))\n \n print (\"compute_pred_dict\")\n\n nq_pred_dict = tf2baseline.compute_pred_dict(candidates_dict, eval_features,\n [r._asdict() for r in all_results])\n predictions_json = {\"predictions\": list(nq_pred_dict.values())}\n\n print (\"writing json\")\n\n with tf.io.gfile.GFile(FLAGS.output_prediction_file, \"w\") as f:\n json.dump(predictions_json, f, indent=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Now, we turn `predictions.json` into a `submission.csv` file.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_answers_df = pd.read_json(\"/kaggle/working/predictions.json\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Bert model produces a `confidence` score, which the Kaggle metric does not use. You, however, can use that score to determine which answers get submitted. See the limits commented out in `create_short_answer` and `create_long_answer` below for an example.\n\nValues for `confidence` will range between `1.0` and `2.0`."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_short_answer(entry):\n if entry[\"short_answers_score\"] < 1.5:\n return \"\"\n \n answer = [] \n for short_answer in entry[\"short_answers\"]:\n if short_answer[\"start_token\"] > -1:\n answer.append(str(short_answer[\"start_token\"]) + \":\" + str(short_answer[\"end_token\"]))\n if entry[\"yes_no_answer\"] != \"NONE\":\n answer.append(entry[\"yes_no_answer\"])\n return \" \".join(answer)\n\ndef create_long_answer(entry):\n if entry[\"long_answer_score\"] < 1.5:\n return \"\"\n\n answer = []\n if entry[\"long_answer\"][\"start_token\"] > -1:\n answer.append(str(entry[\"long_answer\"][\"start_token\"]) + \":\" + str(entry[\"long_answer\"][\"end_token\"]))\n return \" \".join(answer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_answers_df[\"long_answer_score\"] = test_answers_df[\"predictions\"].apply(lambda q: q[\"long_answer_score\"])\ntest_answers_df[\"short_answer_score\"] = test_answers_df[\"predictions\"].apply(lambda q: q[\"short_answers_score\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_answers_df[\"long_answer_score\"].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"An example of what each sample's answers look like in `prediction.json`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_answers_df.predictions.values[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We re-format the JSON answers to match the requirements for submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_answers_df[\"long_answer\"] = test_answers_df[\"predictions\"].apply(create_long_answer)\ntest_answers_df[\"short_answer\"] = test_answers_df[\"predictions\"].apply(create_short_answer)\ntest_answers_df[\"example_id\"] = test_answers_df[\"predictions\"].apply(lambda q: str(q[\"example_id\"]))\n\nlong_answers = dict(zip(test_answers_df[\"example_id\"], test_answers_df[\"long_answer\"]))\nshort_answers = dict(zip(test_answers_df[\"example_id\"], test_answers_df[\"short_answer\"]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we add them to our sample submission. Recall that each sample has both a `_long` and `_short` entry in the sample submission, one for each type of answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(\"/kaggle/input/tensorflow2-question-answering/sample_submission.csv\")\n\nlong_prediction_strings = sample_submission[sample_submission[\"example_id\"].str.contains(\"_long\")].apply(lambda q: long_answers[q[\"example_id\"].replace(\"_long\", \"\")], axis=1)\nshort_prediction_strings = sample_submission[sample_submission[\"example_id\"].str.contains(\"_short\")].apply(lambda q: short_answers[q[\"example_id\"].replace(\"_short\", \"\")], axis=1)\n\nsample_submission.loc[sample_submission[\"example_id\"].str.contains(\"_long\"), \"PredictionString\"] = long_prediction_strings\nsample_submission.loc[sample_submission[\"example_id\"].str.contains(\"_short\"), \"PredictionString\"] = short_prediction_strings","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally, we write out our submission!"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission.to_csv(\"submission.csv\", index=False)\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please give this kernel an UPVOTE to show your appreciation, if you find it useful.\n
\n
\nAlso don't forget to upvote Dimitre's kernel here"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}