{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n“Why is the sky blue?”\n\nThis is a question an open-domain question answering (QA) system should be able to respond to. QA systems emulate how people look for information by reading the web to return answers to common questions. Machine learning can be used to improve the accuracy of these answers.\n\nExisting natural language models have been focused on extracting answers from a short paragraph rather than reading an entire page of content for proper context. As a result, the responses can be complicated or lengthy. A good answer will be both succinct and relevant.\n\nIn this competition, your goal is to predict short and long answer responses to real questions about Wikipedia articles. The dataset is provided by Google's Natural Questions, but contains its own unique private test set. A visualization of examples shows long and—where available—short answers. In addition to prizes for the top teams, there is a special set of awards for using TensorFlow 2.0 APIs.\n\nIf successful, this challenge will help spur the development of more effective and robust QA systems.\n\n## What should I expect the data format to be?\nEach sample contains a Wikipedia article, a related question, and the candidate long form answers. The training examples also provide the correct long and short form answer or answers for the sample, if any exist.\n\n## What am I predicting?\nFor each article + question pair, you must predict / select long and short form answers to the question drawn directly from the article. - A long answer would be a longer section of text that answers the question - several sentences or a paragraph. - A short answer might be a sentence or phrase, or even in some cases a YES/NO. The short answers are always contained within / a subset of one of the plausible long answers. - A given article can (and very often will) allow for both long and short answers, depending on the question.\n\nThere is more detail about the data and what you're predicting [on the Github page](https://github.com/google-research-datasets/natural-questions/blob/master/README.md) for the Natural Questions dataset. This page also contains helpful utilities and scripts. Note that we are using the simplified text version of the data - most of the HTML tags have been removed, and only those necessary to break up paragraphs / sections are included.\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom IPython.core.display import HTML\nimport json\nimport gc\nimport json\nimport subprocess\nfrom tqdm import tqdm_notebook as tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nPATH = '/kaggle/input/tensorflow2-question-answering/'\n!ls {PATH}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Parmeters ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Train & test dataset paths\nPATH_TRAIN = PATH + 'simplified-nq-train.jsonl'\nPATH_TEST = PATH + 'simplified-nq-test.jsonl'\nnrows = 1000 #Number of lines to take from test dataset\n\n#Init parmeters\n\ntrain_ds = []\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extract data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Common way to convert .jsonl file into pd.DataFrame is pd.read_json(FILENAME, orient='records', lines=True), Nontheless, This is humongous train dataset , Kaggle notebook RAM cannot support it. Instead, I'll load train dataset iteratively:","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"\nwith open(PATH+'simplified-nq-train.jsonl', 'rt') as f:\n    for i in range(nrows):\n        train_ds.append(json.loads(f.readline()))\n\ntrain_df = pd.DataFrame(train_ds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['yes_no'] = train_df.annotations.apply(lambda x: x[0]['yes_no_answer'])\ntrain_df['long'] = train_df.annotations.apply(lambda x: [x[0]['long_answer']['start_token'], x[0]['long_answer']['end_token']])\ntrain_df['short'] = train_df.annotations.apply(lambda x: x[0]['short_answers'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[\"annotations\"][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data explained","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"question_text - the question that the user asked (into google engine)\ndocument_url  - link to where answer exists \nannotations - Is it a yes/no answer (short answer-can also be NONE) , if not this is long answer - put start and end token for best answer","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# long_answer_candidates\n\nstart_token: The token position in the text where the answer begins;\n\nend_token: The token position in the text where the answer ends;\n\ntop_level: Whether this answer is contained inside another answer in the text{True/False}\n\n\n*** There can be multiple candidate-answers to a single question","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Get Number of words per train/test sets","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"N_TRAIN_bytes = subprocess.check_output('wc -l {}'.format(PATH_TRAIN), shell=True)\nN_TEST_bytes = subprocess.check_output('wc -l {}'.format(PATH_TEST), shell=True)\n\nN_TRAIN = int(N_TRAIN_bytes.split()[0])\nN_TEST = int(N_TEST_bytes.split()[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Train examples : \" , N_TRAIN)\nprint(\"Test examples : \" , N_TEST)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visulaziation ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"question_text_train_len = np.zeros(N_TRAIN)\ndocument_text_train_len = np.zeros(N_TRAIN)\n\nquestion_text_test_len = np.zeros(N_TEST)\ndocument_text_test_len = np.zeros(N_TEST)\n\nt_yesno_train = []\nn_long_candidates_train = np.zeros(N_TRAIN)\nt_long_train = np.zeros((N_TRAIN,2))\nn_long_candidates_test = np.zeros(N_TEST)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(PATH_TRAIN, 'rt') as f:\n    for train_idx in tqdm(range(N_TRAIN)):\n        dic = json.loads(f.readline())\n        question_text_train_len[train_idx] = len(dic['question_text'].split())\n        document_text_train_len[train_idx] = len(dic['document_text'].split())\n        t_yesno_train.append(dic['annotations'][0]['yes_no_answer'])\n        n_long_candidates_train[train_idx] = len(dic['long_answer_candidates'])\n        t_long_train[train_idx,0] = dic['annotations'][0]['long_answer']['start_token']\n        t_long_train[train_idx,1] = dic['annotations'][0]['long_answer']['end_token']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(PATH_TEST, 'rt') as f:\n    for test_idx in tqdm(range(N_TEST)):\n        dic = json.loads(f.readline())\n        question_text_test_len[test_idx] = len(dic['question_text'].split())\n        document_text_test_len[test_idx] = len(dic['document_text'].split())\n        n_long_candidates_test[test_idx] = len(dic['long_answer_candidates'])\n     ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(question_text_train_len, density=True , alpha=0.9, color='orange', label='train') \nplt.hist(question_text_test_len, density=True, alpha=0.5, color='b', label='test')\nplt.legend()#['test','train']\nplt.title('Question text length Vs Sample proportion')\nplt.xlabel('Question text length')\nplt.ylabel('Sample proportion')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(document_text_train_len, color='orange',density=True, alpha=0.9, label='Train') \nplt.hist(document_text_test_len,  color='b',density=True, alpha=0.5, label='Test') \nplt.legend()\nplt.title('Document text length Vs Sample proportion')\nplt.xlabel('Document text length')\nplt.ylabel('Sample proportion')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(t_yesno_train, bins=[0,1,2,3], align='left', density=True, rwidth=0.5, label='train')\nplt.legend()\nplt.title('yes-no answer sample proportion')\nplt.xlabel('yes-no answer')\nplt.ylabel('sample proportion')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(n_long_candidates_train, alpha=0.5, color='b', label='train') \nplt.legend()\nplt.title('Long answer candidates Vs Samples')\nplt.xlabel('Long answer candidates')\nplt.ylabel('Samples')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check amonut of exists answers  ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train_df.long.apply(lambda x: \"Answer Doesn't exist\" if x[0] == -1 else \"Answer Exists\").value_counts(normalize=True))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Short answers distribution -as we can see less than 50% are yes/no questions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mask_answer_exists = train_df.long.apply(lambda x: \"Answer Doesn't exist\" if x == -1 else \"Answer Exists\") == \"Answer Exists\"\n\nyes_no_dist = train_df[mask_answer_exists].yes_no.value_counts(normalize=True)\nyes_no_dist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"short_dist = train_df[mask_answer_exists].short.apply(lambda x: \"Short answer exists\" if len(x) > 0 else \"Short answer doesn't exist\").value_counts(normalize=True)\nplt.figure(figsize=(8,6))\nsns.barplot(x=short_dist.index,y=short_dist.values).set_title(\"Short answers distribution Vs questions with answers\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"short_size_dist = train_df[mask_answer_exists].short.apply(len).value_counts(normalize=True)\nshort_size_dist_pretty = pd.concat([short_size_dist.loc[[0,1,],], pd.Series(short_size_dist.loc[2:].sum(),index=['>=2'])])\nshort_size_dist_pretty = short_size_dist_pretty.rename(index={0: 'No Short answer',1:\"1 Short answer\",\">=2\":\"More than 1 short answers\"})\nplt.figure(figsize=(12,6))\nsns.barplot(x=short_size_dist_pretty.index,y=short_size_dist_pretty.values).set_title(\"Short Answers Distribution Vs questions with answers\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}