{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About This Kernel\n\nThis is the training kernel for the [Tensorflow 2.0 Question Answering](https://www.kaggle.com/c/tensorflow2-question-answering) Competition. I also created [an inference kernel](https://www.kaggle.com/xhlulu/tf2-qa-lstm-inference-kernel/) that shows how to use the trained model to make a submission.\n\nMy approach is heavily inspired from [Oleg's kernel](https://www.kaggle.com/opanichev/tf2-0-qa-binary-classification-baseline), i.e. I break down each document into parts corresponding to each of the candidate long answers. Then, I label each of the long answer to be `1` if it is the true long answer, or `0` if not. For each of row, I also include the question and the `example_id`. Then, I train a LSTM to predict that label.\n\nThe sections are broken down in the following way:\n\n1. **Build Dataset:** Two different utility functions for creating ready-to-use dataframes for both training and testing data. The `build_train` function only loads a subset of all training data, and set a sampling rate `S`, such that we only keep `1/S` of all the negative-labelled data (and keep all positive-labelled data).\n\n2. **Preprocessing:** Train a keras `Tokenizer` to encode the text and questions into list of integers (tokenization), then pad them to a fixed length to form a single numpy array for text and one for questions.\n\n3. **Modelling:**\n    * Generate a fasttext embedding (directly using [FAIR's Official Python API](https://github.com/facebookresearch/fastText/tree/master/python)) based on the index of the tokenizer. \n    * Build two 2-layer bidirectional LSTM; one to read the questions, and one to read the text. \n    * Concatenate the output of the LSTM and feed in 2-layer fully-connected neural networks.\n    * Predict the binary output using Sigmoid activation.\n    * Optimize using Adam and binary cross-entropy loss.\n\n4. **Save Model:** Due to the submission time limit, it is better to import the model we just trained in a separate kernel to infer and submit. In the inference kernel, we first remove all the rows with less than 0.5 confidence, then for each `example_id` we only keep the one with the highest confidence to be the output."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport json\nimport gc\nimport pickle\n\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm_notebook as tqdm\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Dense, Embedding, SpatialDropout1D, concatenate, Masking\nfrom tensorflow.keras.layers import LSTM, Bidirectional, GlobalMaxPooling1D, Dropout\nfrom tensorflow.keras.preprocessing import text, sequence\nfrom tqdm import tqdm_notebook as tqdm\nimport fasttext","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Dataset"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def build_train(train_path, n_rows=200000, sampling_rate=15):\n    with open(train_path) as f:\n        processed_rows = []\n\n        for i in tqdm(range(n_rows)):\n            line = f.readline()\n            if not line:\n                break\n\n            line = json.loads(line)\n\n            text = line['document_text'].split(' ')\n            question = line['question_text']\n            annotations = line['annotations'][0]\n\n            for i, candidate in enumerate(line['long_answer_candidates']):\n                label = i == annotations['long_answer']['candidate_index']\n\n                start = candidate['start_token']\n                end = candidate['end_token']\n\n                if label or (i % sampling_rate == 0):\n                    processed_rows.append({\n                        'text': \" \".join(text[start:end]),\n                        'is_long_answer': label,\n                        'question': question,\n                        'annotation_id': annotations['annotation_id']\n                    })\n\n        train = pd.DataFrame(processed_rows)\n        \n        return train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_test(test_path):\n    with open(test_path) as f:\n        processed_rows = []\n\n        for line in tqdm(f):\n            line = json.loads(line)\n\n            text = line['document_text'].split(' ')\n            question = line['question_text']\n            example_id = line['example_id']\n\n            for candidate in line['long_answer_candidates']:\n                start = candidate['start_token']\n                end = candidate['end_token']\n\n                processed_rows.append({\n                    'text': \" \".join(text[start:end]),\n                    'question': question,\n                    'example_id': example_id,\n                    'sequence': f'{start}:{end}'\n\n                })\n\n        test = pd.DataFrame(processed_rows)\n    \n    return test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"directory = '/kaggle/input/tensorflow2-question-answering/'\ntrain_path = directory + 'simplified-nq-train.jsonl'\ntest_path = directory + 'simplified-nq-test.jsonl'\n\ntrain = build_train(train_path)\ntest = build_test(test_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"def compute_text_and_questions(train, test, tokenizer):\n    train_text = tokenizer.texts_to_sequences(train.text.values)\n    train_questions = tokenizer.texts_to_sequences(train.question.values)\n    test_text = tokenizer.texts_to_sequences(test.text.values)\n    test_questions = tokenizer.texts_to_sequences(test.question.values)\n    \n    train_text = sequence.pad_sequences(train_text, maxlen=300)\n    train_questions = sequence.pad_sequences(train_questions)\n    test_text = sequence.pad_sequences(test_text, maxlen=300)\n    test_questions = sequence.pad_sequences(test_questions)\n    \n    return train_text, train_questions, test_text, test_questions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = text.Tokenizer(lower=False, num_words=80000)\n\nfor text in tqdm([train.text, test.text, train.question, test.question]):\n    tokenizer.fit_on_texts(text.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_target = train.is_long_answer.astype(int).values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_text, train_questions, test_text, test_questions = compute_text_and_questions(train, test, tokenizer)\ndel train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_embedding_matrix(tokenizer, path):\n    embedding_matrix = np.zeros((tokenizer.num_words + 1, 300))\n    ft_model = fasttext.load_model(path)\n\n    for word, i in tokenizer.word_index.items():\n        if i >= tokenizer.num_words - 1:\n            break\n        embedding_matrix[i] = ft_model.get_word_vector(word)\n    \n    return embedding_matrix","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(embedding_matrix):\n    embedding = Embedding(\n        *embedding_matrix.shape, \n        weights=[embedding_matrix], \n        trainable=False, \n        mask_zero=True\n    )\n    \n    q_in = Input(shape=(None,))\n    q = embedding(q_in)\n    q = SpatialDropout1D(0.2)(q)\n    q = Bidirectional(LSTM(100, return_sequences=True))(q)\n    q = GlobalMaxPooling1D()(q)\n    \n    \n    t_in = Input(shape=(None,))\n    t = embedding(t_in)\n    t = SpatialDropout1D(0.2)(t)\n    t = Bidirectional(LSTM(150, return_sequences=True))(t)\n    t = GlobalMaxPooling1D()(t)\n    \n    hidden = concatenate([q, t])\n    hidden = Dense(300, activation='relu')(hidden)\n    hidden = Dropout(0.5)(hidden)\n    hidden = Dense(300, activation='relu')(hidden)\n    hidden = Dropout(0.5)(hidden)\n    \n    out1 = Dense(1, activation='sigmoid')(hidden)\n    \n    model = Model(inputs=[t_in, q_in], outputs=out1)\n    model.compile(loss='binary_crossentropy', optimizer='adam')\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/kaggle/input/fasttext-crawl-300d-2m-with-subword/crawl-300d-2m-subword/crawl-300d-2M-subword.bin'\nembedding_matrix = build_embedding_matrix(tokenizer, path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = build_model(embedding_matrix)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = model.fit(\n    [train_text, train_questions], \n    train_target,\n    epochs=2,\n    validation_split=0.2,\n    batch_size=1024\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# saving\nwith open('tokenizer.pickle', 'wb') as handle:\n    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('model.h5')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}