{"cells":[{"metadata":{},"cell_type":"markdown","source":"TensorFlow 2.0 Question Answering Competition Notebook\n==================\n* Develop a more effective and robust QA system\n* This is forked from the following notebook\n* Source: https://www.kaggle.com/jazivxt/on-the-professor-and-the-madman\n\nAdded comments for understanding the code"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Installing the regular pandas & numpy libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport json","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This loads lines from a json dump. The number of lines is provided by the parameter *max_limit* \nThe data is appended to a list & then converted into a dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_lines_m(path, max_limit=4000):\n    rlm = []; ml = max_limit\n    for l in open(path, 'r'):\n        rlm.append(json.loads(l))\n        ml -= 1\n        if ml <= 0: break\n    return pd.DataFrame(rlm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Specifies the path to the input files"},{"metadata":{"trusted":true},"cell_type":"code","source":"p = '../input/tensorflow2-question-answering/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next few lines of code read the input file & prints out the shape & columns in the input file "},{"metadata":{"trusted":true},"cell_type":"code","source":"train = read_lines_m(p + 'simplified-nq-train.jsonl')\n\nprint(train.shape)\nprint(train.columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The column *Annotation* is interesting since it provides possible responses for the given question.\nFor example for the first question in the dataset - *Which is the most common use of opt-in e-mail marketing* \nThe potential start & end tokens of the answers can be found in the *long_answer* & *short_answer* dictionaries "},{"metadata":{"trusted":true},"cell_type":"code","source":"train.question_text[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.annotations[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the long_answer & short answers are extracted from the document_text"},{"metadata":{"trusted":true},"cell_type":"code","source":"document_text_tokens = train.document_text[0].split(' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"' '.join(document_text_tokens[train.annotations[0][0]['long_answer']['start_token']:train.annotations[0][0]['long_answer']['end_token']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"' '.join(document_text_tokens[train.annotations[0][0]['short_answers'][0]['start_token']:train.annotations[0][0]['short_answers'][0]['end_token']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's look at another question\nprint(train.annotations[105])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.question_text[105])\nprint(\"Long Answer:\")\n\ndocument_text_tokens = train.document_text[105].split(' ')\nprint(' '.join(document_text_tokens[train.annotations[105][0]['long_answer']['start_token']:train.annotations[105][0]['long_answer']['end_token']]))\nprint(\"Short Answer:\")\nprint(' '.join(document_text_tokens[train.annotations[105][0]['short_answers'][0]['start_token']:train.annotations[105][0]['short_answers'][0]['end_token']]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well.... Let's get back to the rest of the code\n\nThe next piece of code copies the start_tokens for all the long_answers for all the questions"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['D'] = [t[0]['long_answer']['start_token'] for t in train.annotations]\ntrain['D'].head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The 4th index has a -1, this may indicate that the long answer is missing. Lets check it out "},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.annotations[5])\nprint(train.question_text[5])\n\ndocument_text_tokens = train.document_text[5].split(' ')\nprint(\"Long Answer:\")\nprint(document_text_tokens[train.annotations[5][0]['long_answer']['start_token']:train.annotations[5][0]['long_answer']['end_token']])\nprint(\"Short Answer:\")\nprint(document_text_tokens[train.annotations[5][0]['short_answers'][0]['start_token']:train.annotations[5][0]['short_answers'][0]['end_token']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If the question doesnt have a long answer the start_token is -1. But it can also be that the document doesnt have an relevant answer. \n"},{"metadata":{},"cell_type":"markdown","source":"In the next code block we are removing all questions which do not have a long answer "},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[train['D']>-1].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"test = read_lines_m(p + 'simplified-nq-test.jsonl').reset_index(drop=True)\nsub = pd.read_csv(p + 'sample_submission.csv')\ntrain.shape, test.shape, sub.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i =99\nprint('URL:', train.document_url[i])\nprint(train.question_text[i])\nprint(train.long_answer_candidates[i][0])\nprint('Long Answer')\nprint(' '.join(train.document_text[i].split()[train.annotations[i][0]['long_answer']['start_token'] : train.annotations[i][0]['long_answer']['end_token']]))\nif len(train.annotations[i][0]['short_answers']) > 0:\n    print('Short Answer')\n    print(' '.join(train.document_text[i].split()[train.annotations[i][0]['short_answers'][0]['start_token'] : train.annotations[i][0]['short_answers'][0]['end_token']]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the next code block, we are calculating the span of all the long answers & short answers. \nLater we print out the median of the long answer span & short answer span "},{"metadata":{"trusted":true},"cell_type":"code","source":"la=[t[0]['long_answer']['end_token'] - t[0]['long_answer']['start_token'] for t in train.annotations]\nsa=[t[0]['short_answers'][0]['end_token'] - t[0]['short_answers'][0]['start_token'] for t in train.annotations if len(t[0]['short_answers'])>0]\nnp.median(la), np.median(sa)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Importing the usual nltk related packages"},{"metadata":{"trusted":true},"cell_type":"code","source":"from bs4 import BeautifulSoup as b\nfrom nltk.corpus import stopwords\nimport random, nltk","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's walk through the next code segment - \n\nWe first iterate through each question in the test set \nThe document_text is then parsed using an *html parser* \n> s = b(test.document_text[0], 'html.parser')\n\nWe then extract texts from all paragraphs in the document text into a list. The paragraph text is added only if the text length is greater than 50 \n> p = [p.get_text() for p in s.find_all('p', text=True) if len(p.get_text()) > 50]\n\nFor each paragraph in the above list we run a word match. This is done through the *qa_word_match* function"},{"metadata":{},"cell_type":"markdown","source":"The short answer is randomly chosen \n\n>short_answer=random.choice(['YES','NO'])\n\nOR \n\n> r = random.randrange(r, r + 114)\n\n> short_answer=''.join([str(r),':', str(r + 2)])"},{"metadata":{},"cell_type":"markdown","source":"**Inside the qa_word_match function call**\n\nEach question string is made lowercase & then split at a word level. Then the nltk stopword list is used to remove all unnecessary words\n\n> q = q.lower().split()\n\n\n> q = [q1 for q1 in q if q1 not in list(set(stopwords.words('english')))]\n\nWe then check if any words in the question list matches with the potential answers (*This is the paragraph text that we have extracted*)\nThe string where there is the largest match, we return it back\n> m = np.sum([1 for w in a1.lower().split() if w in q])\n\nThe complete function is defined below"},{"metadata":{"trusted":true},"cell_type":"code","source":"def qa_word_match(q,a):\n    q = q.lower().split()\n    q = [q1 for q1 in q if q1 not in list(set(stopwords.words('english')))]\n    tm = 0\n    a2 = a[0]\n    for a1 in a:\n        m = np.sum([1 for w in a1.lower().split() if w in q])\n        if m > tm:\n            tm = int(m)\n            a2 = str(a1)\n    return a2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result = []\nfor i in range(len(test.example_id)):\n    s = b(test.document_text[i], 'html.parser')\n    p = [p.get_text() for p in s.find_all('p', text=True) if len(p.get_text()) > 50]\n    if len(p)>0:\n        a = qa_word_match(test.question_text[i], p)\n        r = test.document_text[i].find(a)\n        r = len(test.document_text[i][:r].split()) - 1\n        long_answer= ''.join([str(r),':', str(r + len(p[0].split()) + 2)])\n    else:\n        try:\n            r = random.randrange(390, len(test.document_text[i].split()))\n        except:\n            r=7\n        long_answer= ''.join([str(r),':', str(r + 114)])\n\n    if len([q for q in ['am', 'are', 'can', 'could', 'did', 'do', 'does', 'has', 'have', 'is', 'may', 'should', 'was', 'were', 'will'] if q in test.question_text[i].lower().split()])>0:\n        short_answer=random.choice(['YES','NO'])\n    else:\n        r = random.randrange(r, r + 114)\n        short_answer=''.join([str(r),':', str(r + 2)])\n    result.append([test.example_id[i] + '_long', long_answer])\n    result.append([test.example_id[i] + '_short', short_answer])\npd.DataFrame(result,columns=['example_id', 'PredictionString']).to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ｈ𝐀𝑷𝑷𝓎 🇰𝗮𝘨𝘨🇱𝖎Ｎɢ 💯\n=========================="},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}