{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **Q&A Data Exploratory Data Analysis**"},{"metadata":{},"cell_type":"markdown","source":"This is just a quick EDA I made when I started the competition to give me a feel for the dataset.\nIt is fairly simple and just meant to give a brief overview of the dataset.\nA list of takeaways from this EDA are listed at the bottom that helped me get my initial 0.64 lb."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from collections import Counter\nimport json\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom scipy import stats\nimport seaborn as sns\nimport sentencepiece as spm\nimport tqdm\n\n%matplotlib inline\nsns.set()\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_generator(path, chunk_size=30000):\n    curr_pos = 0\n    last_line = False\n    with open(path, 'rt') as f:\n        while not last_line:\n            df = []\n            for i in range(curr_pos, curr_pos+chunk_size):\n                line = f.readline()\n                if line is None:\n                    last_line = True\n                    break\n                df.append(json.loads(line))\n            curr_pos = i + 1\n            yield pd.DataFrame(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = '/kaggle/input/tensorflow2-question-answering/'\ntrain_path = os.path.join(data_path, 'simplified-nq-train.jsonl')\ntest_path = os.path.join(data_path, 'simplified-nq-test.jsonl')\n\ntrain_gen = data_generator(train_path, chunk_size=5000)\n\ndf = next(train_gen)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Testing Sentencepiece"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Uncomment for a demonstration of trainig and using a Sentencepiece model\n\n# tmp_data_gen = data_generator(train_path, chunk_size=5000)\n# with open('wiki_text.txt', 'w+') as f:\n#     print('Generating corpus for sentencepiece model...')\n#     for i in tqdm.tqdm(range(5)):\n#         docs = next(tmp_data_gen)['document_text']\n#         for text in docs:\n#             f.write(text + '\\n')\n        \n# sp = spm.SentencePieceTrainer.Train('--input=wiki_text.txt --model_prefix=test_sp --vocab_size=8000 --character_coverage=1.0 --model_type=unigram')\n\n# sp = spm.SentencePieceProcessor()\n# sp.Load('test_sp.model');\n\n# print(sp.EncodeAsIds('this is a test'))\n# print(sp.encode_as_pieces('this is a test'))\n# print(sp.encode_as_pieces('that is a test'))\n# print(sp.encode_as_pieces('Some of the tokenizing here is quite strange, but I guess itll be okay :)'))\n# print(sp.decode_ids([1, 2, 3]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split documents up into informal tokens for analysis\ndf['tokens'] = df['document_text'].apply(lambda x: [w.lower() for w in x.split(' ')])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"word_counts = {}\nfor tokens in df['tokens']:\n    for token in tokens:\n        if token in word_counts:\n            word_counts[token] += 1\n        else:\n            word_counts[token] = 1\n            \ntop_word_counts = sorted(word_counts.items(), key=lambda i: i[1], reverse=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.title('Most Common Tokens')\nplt.bar(x=[x[0] for x in top_word_counts[:20]], height=[x[1] for x in top_word_counts[:20]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.title('Words Per Page')\nsns.distplot(df['tokens'].apply(len).values, kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_tags = df['tokens'].apply(lambda x: sum([1 if t.startswith('<') and t.endswith('>') else 0 for t in x])).values\ntag_ratios = n_tags / df['tokens'].apply(len)\n\nplt.figure(figsize=(12, 5))\nplt.title('Tag Percentage Distribution')\nsns.distplot(tag_ratios, kde=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\nplt.title('Number of Long Answers Candidates')\nplt.xlim(0, 1000)\nsns.distplot(df['long_answer_candidates'].apply(len), kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.title('Number of Tokens per Long Answer Candidate')\nsns.distplot(df['tokens'].apply(len) / df['long_answer_candidates'].apply(len), kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.title('Question Text Length')\nsns.distplot(df['question_text'].apply(lambda s: s.split(' ')).apply(len), kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12, 6))\nplt.title('Long Answer Text Log Length')\n\nlong_answer_lengths = df.apply(lambda row: row['annotations'][0]['long_answer']['end_token'] - \\\n                               row['annotations'][0]['long_answer']['start_token'], axis=1).values\nlong_answer_lengths = [x for x in long_answer_lengths if x != 0]\n\n# fig.axes[0].set_xscale('log')\n\nsns.distplot(np.log(long_answer_lengths), kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.title('Short Answer Text Length')\n\nshort_answer_lengths = df.apply(\n    lambda row: [x['end_token'] - x['start_token'] for x in row['annotations'][0]['short_answers']],\n    axis=1).values\nshort_answer_lengths = np.concatenate(short_answer_lengths)\n\nsns.distplot(short_answer_lengths, kde=False);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12, 6))\nplt.title('Short Answer Text Length')\n\nsns.boxplot(short_answer_lengths);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(stats.describe(short_answer_lengths))\nprint(np.quantile(short_answer_lengths, 0.99))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n- HTML tags make up an large proportion of most articles.\n- The sizes of most article pages is small but is right-skewed, meaning there are a small number of very large big articles."},{"metadata":{},"cell_type":"markdown","source":"# Example Annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(10):\n    print(df['annotations'][i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['annotations'][0][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Every annotations entry is a list of rank 1\nsum([1 if x != 1 else 0 for x in df['annotations'].apply(len)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n\n- All annotations are a list with the actual annotation being the first element.\n- When there is no yes/no answer, the value is 'NONE'\n- When there is no long answer, the value of each map entry is -1\n- There can be multiple short answers, but only one long answer\n- When there are no short answers, the value is an empty list"},{"metadata":{},"cell_type":"markdown","source":"# Example Annotations"},{"metadata":{"trusted":true},"cell_type":"code","source":"f = 0\nfor i in range(len(df)):\n    print(i)\n    print(df['annotations'][i][0])\n    f += 1\n    if f >= 5:\n        break\n\nprint('-------------------------------------------')\n\nf = 0\nfor i in range(len(df)):\n    if df['annotations'][i][0]['yes_no_answer'] != 'NONE':\n        print(i)\n        print(df['annotations'][i][0])\n        f += 1\n    if f >= 5:\n        break\n        \nprint('-------------------------------------------')\n        \nf = 0\nfor i in range(len(df)):\n    if len(df['annotations'][i][0]['short_answers']) > 1:\n        print(i)\n        print(df['annotations'][i][0])\n        f += 1\n    if f >= 5:\n        break\n\nprint('-------------------------------------------')\n\n# It looks like a short answer will probably only exist if a long answer also exists\nf = 0\nfor i in range(len(df)):\n    if len(df['annotations'][i][0]['short_answers']) >= 1 and df['annotations'][i][0]['long_answer']['start_token'] == -1:\n        print(i)\n        print(df['annotations'][i][0])\n        f += 1\n    if f >= 5:\n        break\n\nprint('-------------------------------------------')\n\n# It looks like a YES/NO will probably only exist if a long answer also exists\nf = 0\nfor i in range(len(df)):\n    if df['annotations'][i][0]['yes_no_answer'] != 'NONE' and df['annotations'][i][0]['long_answer']['start_token'] == -1:\n        print(i)\n        print(df['annotations'][i][0])\n        f += 1\n    if f >= 5:\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Findings\n\n- A question must have a long answer to have a short answer or yes/no answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_count = 0\nn_count = 0\nla_count = 0\nsa_count = 0\nnone_count = 0\nfor an in df['annotations']:\n    an = an[0]\n    none = True\n    if an['yes_no_answer'] == 'YES':\n        y_count += 1\n        none = False\n    elif an['yes_no_answer'] == 'NO':\n        n_count += 1\n        none = False\n        \n    if an['long_answer']['start_token'] != -1:\n        la_count += 1\n        none = False\n        \n    if len(an['short_answers']) > 0:\n        sa_count += 1\n        none = False\n        \n    if none:\n        none_count += 1\n        \nn = float(len(df))\n\nplt.figure(figsize=(12, 6))\nplt.title('Answer Possibilities by Category')\nplt.bar(x=['% Yes', '% No', '% Long Answer', '% Short Answer', '% No Answer'],\n        height=[y_count/n, n_count/n, la_count/n, sa_count/n, none_count/n]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Findings\n\n- Questions with yes/no answers are VERY RARE.\n- Around half of the questions have a long answer\n- Around 30%-40% of questions have short answers\n- Around half of questions have no answer"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['has_long_answer'] = df['annotations'].apply(lambda x: x[0]['long_answer']['start_token'] > -1)\ndf['has_short_answer'] = df['annotations'].apply(lambda x: len(x[0]['short_answers']) > 0)\ndf['has_yn_answer'] = df['annotations'].apply(lambda x: x[0]['yes_no_answer'] != 'NONE')\n\nla_with_answer_counts = []\nla_no_answer_counts = []\n\nfor i, row in df.iterrows():\n    n_candidates = len(row['long_answer_candidates'])\n    if row['has_long_answer']:\n        la_with_answer_counts.append(len(row['long_answer_candidates']))\n    else:\n        la_no_answer_counts.append(len(row['long_answer_candidates']))\n    \nt, p = stats.ttest_ind(la_with_answer_counts, la_no_answer_counts)\nprint(f'p-val: {p} | t-stat: {t}')\nif p > 0.05:\n    print('No significant difference between distributions')\nelse:\n    print('Significant difference between distributions')\n    \n\nplt.figure(figsize=(6, 10))\nplt.ylim(0, 1200)\nplt.title('Amount of Long Answer Candidates for Questions With and Without Answers')\nplt.boxplot([la_no_answer_counts, la_with_answer_counts]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['n_article_tokens'] = df['tokens'].apply(len)\ndf['n_candidate_long_answers'] = df['long_answer_candidates'].apply(len)\n\ndf['has_long_answer_int'] = df['has_long_answer'].apply(np.int8)\ndf['has_short_answer_int'] = df['has_short_answer'].apply(np.int8)\ndf['has_yn_answer_int'] = df['has_yn_answer'].apply(np.int8)\n\nsns.pairplot(df, vars=['n_article_tokens', 'n_candidate_long_answers', 'has_long_answer_int', 'has_short_answer_int', 'has_yn_answer_int'],\n             kind='scatter', size=3, plot_kws={'alpha': 0.075});","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, row in df.iterrows():\n    if not row['has_short_answer'] or not row['has_long_answer']:\n        continue\n    \n    annotations = row['annotations'][0]\n    long_range = (annotations['long_answer']['start_token'], annotations['long_answer']['end_token'])\n    for short_ans in annotations['short_answers']:\n        if not (short_ans['start_token'] >= long_range[0] and short_ans['end_token'] <= long_range[1]):\n            print(f'**{i}**')\n            print('Short Answer: ' + ' '.join(df['tokens'][short_ans['start_token']:short_ans['end_token']]))\n            print('Long Answer: ' + ' '.join(df['tokens'][long_range[0]:long_range[1]]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Findings\n\n- Questions that have yes/no answers have a much smaller range than questions that done, and thye also tend to have less long answer candidates. This should be a factor included in the final model.\n- Short answers are always a part of the long answers"},{"metadata":{},"cell_type":"markdown","source":"# All Findings\n##### **Note - EDA was done with just a sample of the data, as the entire training file is too large to read to ram*\n\n- HTML tags make up an large proportion of most articles.\n- The sizes of most article pages is small but is right-skewed, meaning there are a small number of very large big articles.\n- All annotations are a list with the actual annotation being the first element.\n- When there is no yes/no answer, the value is 'NONE'\n- When there is no long answer, the value of each map entry is -1\n- There can be multiple short answers, but only one long answer\n- When there are no short answers, the value is an empty list\n- A question must have a long answer to have a short answer or yes/no answer.\n- Questions with yes/no answers are VERY RARE.\n- Around half of the questions have a long answer\n- Around 30%-40% of questions have short answers\n- Around half of questions have no answer\n- Questions that have yes/no answers have a much smaller range than questions that done, and thye also tend to have less long answer candidates. This should be a factor included in the final model.\n- Short answers are always a part of the long answers\n- Check out more info on the [official GitHub page](https://github.com/google-research-datasets/natural-questions/blob/master/README.md)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}