{"cells":[{"metadata":{},"cell_type":"markdown","source":"It's mostly the same BERT-joint [pipeline](https://github.com/google-research/language/blob/master/language/question_answering/bert_joint/run_nq.py) by Google Research team but with some insights from the [paper](https://arxiv.org/abs/1909.05286) by IBM team \"Frustratingly Easy Natural Question Answering\"\n\nMain points:\n1. BERT-large WWM uncased as an initial checkpoint\n1. the model is first fine-tuned with SQuAD 2.0 data achieving 85.2% / 82.2% (F1/exact)\n1. we're returning answer type (null, yes, no, short, long) probabilities as well, that's also used at inference \n1. maximal sequence length is increased to a maximum (for BERT) of 512\n1. maximal query length is lowered to 24 due to short questions in NQ dataset (max 17 words for the dev set). This allows more place for candidate texts\n1. `doc_stride` is set to 192 following the experiments reported in the paper by IBM\n1. tokenization is done faster with LRU cache\n1. the total number of n-best predictions to consider is increased to 20 (fixing a funny [bug](https://github.com/google-research/language/blob/master/language/question_answering/bert_joint/run_nq.py#L1160) in the original code where `n_best_size` is overwritten with a local variable)\n1. thresholds are tuned twice (see below)\n\nActually, we noticed that the metric is not too heavily dependent on score thresholds, small perturbations are fine\n\n<img src=\"https://habrastorage.org/webt/ka/no/b0/kanob0kktor4pnmyy5uaqwfyylu.png\" width=50% />\n\nAnd of course coming up with a right metric was important - actually, it's just `nq_eval` with 1 more line :)\n\n<img src=\"https://habrastorage.org/webt/ar/12/g0/ar12g0cea9fnojk_ghhjs2wyyae.png\" width=70% />\n\n**This approach leads to 68.2 / 56.7 / 63.5 dev scores (long/short/all F1), 65/66 public/private LB.**\n\nThis Notebook describes only our best single model, @ddanevskiy is going to outline our whole solution.\n\nThi is a short Notebook, most of the code lives in the modified `run_nq` script from [this shared Dataset](https://www.kaggle.com/kashnitsky/bert-wwm-063065-checkpoint) that I use in the Notebook. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\nimport json\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport time\nfrom contextlib import contextmanager","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Nice way to report running times**"},{"metadata":{"trusted":true},"cell_type":"code","source":"@contextmanager\ndef timer(name):\n    t0 = time.time()\n    yield\n    print(f'[{name}] done in {time.time() - t0:.0f} s')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Paths to pretrained models, configs, data etc.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# input\nPATH_TO_DATA = '../input/tensorflow2-question-answering/'\nPATH_TO_CUSTOM_BERT = '../input/bert-wwm-063065-checkpoint/'\nPATH_TO_CUSTOM_BERT_WEIGHTS = PATH_TO_CUSTOM_BERT + \\\n    '20200109_bert_joint_wwm_output/20200109_bert_joint_wwm_output/'\nCKPT_NAME = PATH_TO_CUSTOM_BERT_WEIGHTS + '20200109_bert_joint_wwm_output_model.ckpt-15458'\nPATH_TO_TF_WHEELS = PATH_TO_CUSTOM_BERT + \\\n    'tensorflow_gpu_1_13_1_with_deps_whl/tensorflow_gpu_1_13_1_with_deps_whl/'\n\n# output\nOUT_PREDICT_JSON = 'simplified-nq-test-pred.json'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Constants**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# see comments below\nLRU_CACHE_SIZE = 30000\nMAX_SEQ_LEN = 512 \nDOC_STRIDE = 192           \nMAX_QUERY_LEN = 24\n# score thresholds are set with the dev set and `nq_eval`\nLONG_ANS_THRES = 4.4524\nSHORT_ANS_THRES =  7.7251\n# I did two iterations of threshold setting with dev set\nLONG_ANS_THRES_FINAL = 2.1165\nSHORT_ANS_THRES_FINAL =  7.6657","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Adding `document_tokens` to simplified test data so that we can reuse models trained with originally formatted NQ data.**\n\nThis can be done in the `run_nq.py` of course, just didn't refactor this."},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_test_set_for_bert_joint():\n    with open(PATH_TO_DATA + 'simplified-nq-test.jsonl') as f_in, \\\n        open('simplified-nq-test_with_doc_tokens.jsonl', 'w') as f_out:\n            for i, line in tqdm(enumerate(f_in)):\n                json_line = json.loads(line)\n                json_line[\"document_tokens\"] = []\n                for token in json_line['document_text'].split(' '):\n                    json_line[\"document_tokens\"].append({\"token\":token, \n                                             \"start_byte\": -1, \n                                             \"end_byte\": -1, \n                                             \"html_token\": '<' in token})\n                    json_line['annotations'] = []\n                    json_line[\"document_title\"] = \\\n                        json_line[\"document_tokens\"][0][\"token\"]\n\n                f_out.write(json.dumps(json_line) + '\\n')\n             \n    !gzip simplified-nq-test_with_doc_tokens.jsonl","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We'll be using TF 1.13.1, so installing it with dependencies from a dataset**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def setup_tensorflow_1_13_1():\n    # Install `tensorflow-gpu==1.13.1` from pre-downloaded wheels\n    !pip install --no-deps $PATH_TO_TF_WHEELS/*.whl > /dev/null 2>&1\n    # Install custom library google-language + dependencies\n    !cp -r $PATH_TO_CUSTOM_BERT/bert-tensorflow-1.0.1/ . \n    !cd bert-tensorflow-1.0.1/bert-tensorflow-1.0.1/; python setup.py install > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The crucial part - inference**\n\nSorry for this ugly mixture of Python and Bash but it's handy :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_bert_inference():\n    !python $PATH_TO_CUSTOM_BERT/run_nq.py    --logtostderr         \\\n    --bert_config_file=$PATH_TO_CUSTOM_BERT/bert_config.json         \\\n    --vocab_file=$PATH_TO_CUSTOM_BERT/vocab-nq.txt                    \\\n    --tokenizer_cache_size=$LRU_CACHE_SIZE                             \\\n    --max_seq_length=$MAX_SEQ_LEN                                       \\\n    --doc_stride=$DOC_STRIDE                                             \\\n    --max_query_length=$MAX_QUERY_LEN                                     \\\n    --init_checkpoint=$CKPT_NAME                                           \\\n    --predict_file=simplified-nq-test_with_doc_tokens.jsonl.gz              \\\n    --do_predict                                                             \\\n    --output_dir=bert_model_output                                            \\\n    --output_prediction_file=simplified-nq-test-pred.json                      \\\n    > /dev/null 2>&1    \n    \n    # cleaning up\n    !rm -rf bert-tensorflow-1.0.1/\n    !rm simplified-nq-test_with_doc_tokens.jsonl.gz\n    !rm -rf bert_model_output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here we account for answer types and tune thresholds twice - before and after that**\n\nAnswer types:\n\n- 0 - \"no-answer\" otherwise (null instances)\n- 1 - \"yes\" for \"yes\" annotations where the instance contains the long answer span\n- 2 - \"no\" for \"no\" annotations where the instance contains the long answer span\n- 3 - \"short\" for instances that contain all annotated short spans\n- 4 -  \"long\" when the instance contains the long answer span but there is no short or yes/no answer"},{"metadata":{"trusted":true},"cell_type":"code","source":"def postprocess_predictions(pred_json_path=OUT_PREDICT_JSON,\n                            long_thres=LONG_ANS_THRES,\n                            short_thres=SHORT_ANS_THRES):\n    \n    empty_answer = {'candidate_index': -1,\n                'end_byte': -1,\n                'end_token': -1,\n                'start_byte': -1,\n                'start_token': -1}\n    \n    with open(pred_json_path) as f:\n        pred_json = json.load(f)\n    \n    pred_json_processed = {'predictions': []}\n\n    for i, entry in enumerate(pred_json['predictions']):\n\n        entry_copy = entry.copy()\n        ans_type = entry['answer_type']\n\n        if entry['long_answer_score'] < long_thres:\n            entry_copy['long_answer'] = empty_answer\n        if entry['short_answers_score'] < short_thres:\n            entry_copy['short_answers'] = []\n\n        if ans_type== 0: # null\n            entry_copy['long_answer'] = empty_answer\n            entry_copy['short_answers'] = [empty_answer]\n        elif ans_type == 1: # yes\n            entry_copy['yes_no_answer'] = \"YES\"\n            entry_copy['short_answers'] = [empty_answer]\n        elif ans_type== 2: # no\n            entry_copy['yes_no_answer'] = \"NO\"\n            entry_copy['short_answers'] = [empty_answer]\n        elif ans_type == 3: # short\n            entry_copy['yes_no_answer'] = \"NONE\"\n        elif ans_type == 4: # long but no short or yes/no\n            entry_copy['yes_no_answer'] = \"NONE\"\n            entry_copy['short_answers'] = [empty_answer]\n\n        pred_json_processed['predictions'].append(entry_copy)\n        \n    return pred_json_processed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Convert JSON file with prediction into competition submission CSV file**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def form_submission_file(pred_json,\n                         long_thres=LONG_ANS_THRES,\n                         short_thres=SHORT_ANS_THRES):\n\n    example_ids, preds = [], []\n\n    for entry in pred_json['predictions']:\n\n        example_ids.append(str(entry['example_id']) + '_long')\n        example_ids.append(str(entry['example_id']) + '_short')\n\n        score = entry['long_answer_score']\n        \n        if score >= long_thres:\n            long_pred = '{}:{}'.format(entry['long_answer']['start_token'],\n                                   entry['long_answer']['end_token'])\n            if long_pred == '-1:-1':\n                long_pred = \"\"\n        else:\n            long_pred = \"\"\n        \n        if entry['yes_no_answer'] != \"NONE\":\n            short_pred = entry['yes_no_answer']\n        elif score >= short_thres:\n            if entry['short_answers']:\n                short_pred = '{}:{}'.format(entry['short_answers'][0]['start_token'],\n                                   entry['short_answers'][0]['end_token'])\n            else:\n                short_pred = \"\"\n            if short_pred == '-1:-1':\n                short_pred = \"\"\n        else:\n            short_pred = \"\"\n\n        preds.extend([long_pred, short_pred])\n    \n    sub_df = pd.DataFrame({'example_id':example_ids, \n                           'PredictionString': preds})\\\n            .sort_values(by='example_id').reset_index(drop=True)\n    \n    return sub_df  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Assemble it all**"},{"metadata":{"trusted":true},"cell_type":"code","source":"with timer('ALL'):\n    with timer('Processing test set'):\n        process_test_set_for_bert_joint()\n    with timer('Setting up packages'):\n        setup_tensorflow_1_13_1()\n    with timer('Running inference'):\n        run_bert_inference()\n    with timer('Forming final submission file'):\n        pred_json_processed = postprocess_predictions(OUT_PREDICT_JSON,\n                                                     long_thres=LONG_ANS_THRES,\n                                                      short_thres=SHORT_ANS_THRES\n                                                     )\n        sub_df = form_submission_file(pred_json_processed,\n                                      long_thres=LONG_ANS_THRES_FINAL,\n                                      short_thres=SHORT_ANS_THRES_FINAL\n                                     )\n        sub_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}