{"cells":[{"metadata":{},"cell_type":"markdown","source":"Source: https://www.kaggle.com/petrov/first-long-paragraph"},{"metadata":{},"cell_type":"markdown","source":"Loading the required packages"},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nfrom pathlib import Path\nfrom tqdm import tqdm\n\nimport pandas as pd\n\nTEST_TOTAL = 692","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"PATH = Path('/kaggle/input/tensorflow2-question-answering')\n!ls -1 {PATH}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following function returns back the start & end tokens in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_joined_tokens(answer: dict) -> str:\n    return '%d:%d' % (answer['start_token'], answer['end_token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*get_pred*\nThis function will always return back - *YES* for a short_answer"},{"metadata":{},"cell_type":"markdown","source":"As the name of the notebook suggests. The largest answer candidate is chosen OR if no answer candidate is found to be greater than 35 in length then the first answer candidate is picked. \n\n>If start of token as paragraph token \n    \n    and *top_level* is True\n    \n    and len of answer candidate is >35 \n    \n    Then chose the answer \nIf any of the above options do not work then select the first answer candidate\n    "},{"metadata":{"trusted":true},"cell_type":"code","source":"candidates = json_data['long_answer_candidates']\ncandidates","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_pred(json_data: dict) -> dict:\n    ret = {'short': 'YES', 'long': ''}\n    candidates = json_data['long_answer_candidates']\n    \n    paragraphs = []\n    tokens = json_data['document_text'].split(' ')\n    for cand in candidates:\n        start_token = tokens[cand['start_token']]\n        if start_token == '<P>' and cand['top_level'] and cand['end_token']-cand['start_token']>35:\n            break\n    else:\n        cand = candidates[0]\n        \n    ret['long'] = get_joined_tokens(cand)\n    \n    id_ = str(json_data['example_id'])\n    ret = {id_+'_'+k: v for k, v in ret.items()} \n    return ret","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Initialize a dictionary for holding all predictions\n\npreds = dict()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(PATH / 'simplified-nq-test.jsonl', 'r') as f:\n    for line in tqdm(f, total=TEST_TOTAL):\n        json_data = json.loads(line) \n        #break\n        prediction = get_pred(json_data)\n        preds.update(prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"submission = pd.read_csv(PATH / 'sample_submission.csv')\nsubmission['PredictionString'] = submission['example_id'].map(lambda x: preds[x])\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}