{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport json\n\nfrom IPython.display import display\n\n#local script\nfrom tfutils_py import get_answer, read_sample\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Introduction\nHere's an introduction to the dataset based on the info provided here: https://github.com/google-research-datasets/natural-questions/blob/master/README.md.\n\nThis challenge is actually a specific type of challenge called **Natural Question**. When a user asks a question to a search engine, instead of returning the user a whole document that contains the answer to this question, we want to be able to return the answer directly taken from that document. The motivation is that the user doesn't want to have to manually parse through the document looking for his answer.\n\n## Long Answer Candidates\n\nThe first task in *Natural Question* problems [...] is to identify the **smallest HTML bounding box** that contains all of the information required to infer the answer to a question. The candidates will therefore be bounded by the following html tags:\n- paragraphs;\n- lists;\n    * list items;\n- tables;\n     * table rows. \n\n\nIn this specific problem, **multiple** long-answer-candidates to each question were generated (presumably through an automated process).\n## Annotations\nIt is the goal of the annotators to choose the *best long answer candidate* to answer the question. This *best candidate* is the one that verifies the condition: **The smallest long-answer-candidate that answers the question.** Furthermore, if applicable, the annotators also have to specify the answer (yes or no) in case it is a yes/no question and lastly also specify set of short answers which should answer the question in less words than the **best long-answer-candidate**, if these can be found in the text.\n\n## Challenge\n\nThe train set essentially contains a lot of manual annotations, in this challenge we want to train a model with those annotations so it learns how to annotate by itself on future questions."},{"metadata":{},"cell_type":"markdown","source":"# EDA\n\n"},{"metadata":{},"cell_type":"markdown","source":"I wrote an utility script to make this kernel cleaner, it can be found here: https://www.kaggle.com/snovaisg/tfutils-py"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = read_sample(n=3)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# document_text"},{"metadata":{},"cell_type":"markdown","source":"This is a sample of the first document in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = read_sample(n=10)\ndf.loc[0,'document_text'][:50]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the distribution of document size"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = read_sample(n=1000)\ndoc_text_words = df['document_text'].apply(lambda x: len(x.split(' ')))\nplt.figure(figsize=(12,6))\nsns.distplot(doc_text_words.values,kde=True,hist=False).set_title('Distribution of text word count of 1000 docs')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusions\n\n- The text is in html\n- Most texts in this dataset contain about 5k words."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# long_answer_candidates"},{"metadata":{},"cell_type":"markdown","source":"These are candidates which were selected prior to the annotators giving their answer. They contain the *start* and *end* tokens of the html text which will **always** be html tags."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = read_sample(n=3)\ndf.long_answer_candidates[0][:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can take some conclusions already:\n- **start_token**: The token position in the text where the answer begins;\n- **end_token**: The token position in the text where the answer ends;\n- **top_level**: Whether this answer is contained inside another answer in the text.\n\nFurthermore,\n- There can be multiple candidate-answers to a single question"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sample answer\nsample = df.iloc[0]\nget_answer(sample.document_text, sample.long_answer_candidates[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Annotations\nWe can think of these as our target variables. There are few possibilities for these:\n\n- Long answer;\n- yes or no answer;\n- short answer;\n- Answer doesn't exist in the text;\n\n\nLet's simply understand the distribution of these in our dataset. We'll start by the long_answers because when there isn't a long answer for a question, we can conclude there doesn't exist any form of answer to the question on the wikipedia page.\n\n## long_answers_distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess(n=10):\n    df = read_sample(n=n,ignore_doc_text=True)\n    df['yes_no'] = df.annotations.apply(lambda x: x[0]['yes_no_answer'])\n    df['long'] = df.annotations.apply(lambda x: [x[0]['long_answer']['start_token'], x[0]['long_answer']['end_token']])\n    df['short'] = df.annotations.apply(lambda x: x[0]['short_answers'])\n    return df\ndf = preprocess(5000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(df.long.apply(lambda x: \"Answer Doesn't exist\" if x[0] == -1 else \"Answer Exists\").value_counts(normalize=True))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About half of the questions don't have an answer, that's surprising!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's keep a mask of the answers that exist\nmask_answer_exists = df.long.apply(lambda x: \"Answer Doesn't exist\" if x == -1 else \"Answer Exists\") == \"Answer Exists\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution of Yes and No Answers"},{"metadata":{"trusted":true},"cell_type":"code","source":"yes_no_dist = df[mask_answer_exists].yes_no.value_counts(normalize=True)\nyes_no_dist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Essentially most of the answerable questions aren't a yes/no type.\n\n## Distribution of short answers"},{"metadata":{"trusted":true},"cell_type":"code","source":"short_dist = df[mask_answer_exists].short.apply(lambda x: \"Short answer exists\" if len(x) > 0 else \"Short answer doesn't exist\").value_counts(normalize=True)\nplt.figure(figsize=(8,6))\nsns.barplot(x=short_dist.index,y=short_dist.values).set_title(\"Distribution of short answers in answerable questions\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About 60% of the questions that are answerable don't have a short answer. There can also be multiple short answers to a question so let's check the distribution of that!"},{"metadata":{"trusted":true},"cell_type":"code","source":"short_size_dist = df[mask_answer_exists].short.apply(len).value_counts(normalize=True)\nshort_size_dist_pretty = pd.concat([short_size_dist.loc[[0,1,],], pd.Series(short_size_dist.loc[2:].sum(),index=['>=2'])])\nshort_size_dist_pretty = short_size_dist_pretty.rename(index={0: 'No Short answer',1:\"1 Short answer\",\">=2\":\"More than 1 short answers\"})\nplt.figure(figsize=(12,6))\nsns.barplot(x=short_size_dist_pretty.index,y=short_size_dist_pretty.values).set_title(\"Distribution of Number of Short Answers in answerable questions\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Conclusions:\n- About 60% of the answerable questions don't have a short answer\n- About 30% of the answerable questions have 1 short answer\n- About 10% of the answerable questions have more than 1 possible short answer"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}