{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import KFold\nfrom sklearn.preprocessing import LabelEncoder\n\nimport tensorflow as tf\n\nimport json\nimport gc\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## File Description\n- `simplified-nq-train.jsonl` - the training data, in newline-delimited JSON format.\n- `simplified-nq-kaggle-test.jsonl` - the test data, in newline-delimited JSON format.\n- `sample_submission.csv` - a sample submission file in the correct format"},{"metadata":{},"cell_type":"markdown","source":"## Data fields\n- `document_text` - the text of the article in question (with some HTML tags to provide document structure). The text can be tokenized by splitting on whitespace.\n- `question_text` - the question to be answered\n- `long_answer_candidates` - a JSON array containing all of the plausible long answers.\n- `annotations` - a JSON array containing all of the correct long + short answers. Only provided for train.\n- `document_url` - the URL for the full article. Provided for informational purposes only. This is NOT the simplified version of the article so indices from this cannot be used directly. The content may also no longer match the html used to generate document_text. Only provided for train.\n- `example_id` - unique ID for the sample."},{"metadata":{},"cell_type":"markdown","source":"## Submission File\nLet's check the submission file to understand better what we need to predict\nFor each ID in the test set, you must predict \n\na) a set of start:end token indices, \n\nb) a YES/NO answer if applicable (short answers ONLY), or \n\nc) a BLANK answer if no prediction can be made. The file should contain a header and have the following format:\n\n- -7853356005143141653_long,6:18\n- -7853356005143141653_short,YES\n- -545833482873225036_long,105:200\n- -545833482873225036_short,\n- -6998273848279890840_long,\n- -6998273848279890840_short,NO"},{"metadata":{},"cell_type":"markdown","source":"## Load Data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = '/kaggle/input/tensorflow2-question-answering/'\ntrain_path = 'simplified-nq-train.jsonl'\ntest_path = 'simplified-nq-test.jsonl'\nsample_submission_path = 'sample_submission.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_data(path, sample = True, chunksize = 30000):\n    if sample == True:\n        df = []\n        with open(path, 'rt') as reader:\n            for i in range(chunksize):\n                df.append(json.loads(reader.readline()))\n        df = pd.DataFrame(df)\n        print('Our sampled dataset have {} rows and {} columns'.format(df.shape[0], df.shape[1]))\n    else:\n        df = pd.read_json(path, orient = 'records', lines = True)\n        print('Our dataset have {} rows and {} columns'.format(df.shape[0], df.shape[1]))\n        gc.collect()\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = read_data(path+train_path, sample = True)\ntest = read_data(path+test_path, sample = False)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(path + sample_submission_path)\nprint('Our sample submission have {} rows'.format(sample_submission.shape[0]))\nsample_submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Missing Values"},{"metadata":{"trusted":true},"cell_type":"code","source":"def missing_values(df):\n    df = pd.DataFrame(df.isnull().sum()).reset_index()\n    df.columns = ['features', 'n_missing_values']\n    return df\nmissing_values(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"missing_values(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we don't have missing values."},{"metadata":{},"cell_type":"markdown","source":"**Let's explore the first row of out train set to understand the logic of this dataset.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"question_text_0 = train.loc[0, 'question_text']\nquestion_text_0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"document_text_0 = train.loc[0, 'document_text'].split()\n\" \".join(document_text_0[:100])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import HTML,display\ndisplay(HTML(\" \".join(document_text_0)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So in the `document_text` have a huge wikipedia text."},{"metadata":{"trusted":true},"cell_type":"code","source":"long_answer_candidates_0 = train.loc[0, 'long_answer_candidates']\nlong_answer_candidates_0[0:20]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This are all the possibles long answers ranges. In other words they give you the start indices and last indices of all the possibles long answers in the document text columns that could answer the question."},{"metadata":{"trusted":true},"cell_type":"code","source":"annotations_0 = train.loc[0, 'annotations']\nannotations_0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### More will be coming soon..."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}