{"cells":[{"metadata":{},"cell_type":"markdown","source":"I initially found natural language models tricky to get started with especially once I moved on from starter tasks like sentiment analysis. I wanted to share this notebook to show how some common and slightly more complex language tasks could be approached. In this notebook I have used Googles question answering dataset to show how to match questions to answers and how to extract shorter answers from longer ones.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import json\nfrom tqdm.notebook import tqdm\n\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport tensorflow as tf\nfrom tensorflow.keras.layers import LSTM, Bidirectional, GlobalMaxPooling1D, SpatialDropout1D, Dense, Dropout, Input, concatenate, Conv1D, Activation, Flatten\n\nfrom nltk.corpus import stopwords\nimport re","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To make it easier to experiement with hyper-parameter tweaking experiments I have added them to this notebook as globals here.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# data to load\nNUM_OF_TRAIN_QUESTIONS = 1000\nNUM_OF_VAL_QUESTIONS = 1050\nSAMPLE_RATE = 15\nTRAIN_PATH = '../input/tensorflow2-question-answering/simplified-nq-train.jsonl'\n\n# TOKENIZATION\nFILTERS = '!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n'\nLOWER_CASE = True\nMAX_LEN = 300\n\n# long answer model parameters\nEPOCHS = 40\nBATCH_SIZE = 64\nEMBED_SIZE = 100\nCLASS_WEIGHTS = {0: 0.5, 1: 5.}\n\n# short answer model parameters\nSHORT_EPOCHS = 80\nSHORT_BATCH_SIZE = 32\nSHORT_EMBED_SIZE = 200","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Matching questions to long answers\n\nThe first task I'll approach in this notebook is how to find answers that are relevant to the question asked. So if someone typed the following question into Google search:\n\n> \"how i.met your mother who is the mother\"\n\nHow would we find the paragraph in wikipedia that contains the answer? For example, if we had the two following potential answers from a wikipedia page how would a machine recognise that the first paragraph is more likely to contain the answer than the second one:\n\n> \"Tracy McConnell, better known as 'The Mother', is the title character from the CBS television sitcom How I Met Your Mother. The show, narrated by Future Ted, tells the story of how Ted Mosby met The Mother. Tracy McConnell appears in 8 episodes from 'Lucky Penny' to 'The Time Travelers' as an unseen character; she was first seen fully in 'Something New' and was promoted to a main character in season 9. The Mother is played by Cristin Milioti.\"\n\n> \"In 'Bass Player Wanted', the Mother picks up a hitchhiking Marshall, carrying his son Marvin, on her way to Farhampton Inn. On their way, it is revealed that the Mother is a bass player in the band, that is scheduled to play at the wedding reception. But the band's leader, Darren, forced her to quit. The Mother ultimately decides to confront Darren and retake the band. She ends up alone at the bar, and while practicing a speech to give Darren, Darren walks up to her furious the groom's best man punched him for no reason. Amused by this, the Mother laughs, and Darren quits the band in anger.\"\n\nBefore I get into how to model this I will do the usual pre-processing to get the data ready.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Load data\n\nThe data for this task comes in a rather complex form. The raw data is in a json file with each line of json containing the following:\n\n- **question_text:** A string representing a question asked on Google.\n- **document_text:** A string containing the html of the wikipedia page relevant to the question.\n- **annotations/long_answer:** A json object containing the start and end index of the correct answer substring (substring of the document_text) \n- **annotations/short_answers:** An array of the possible short answers. Each short answer is represented as the start and end index of the substring that is within document_text).\n- **long_answer_candidates:** An array of the possible long answers to the questions. These are basically the paragraphs within the wikipedia page.\n\nI've got a cell of code hidden below that reads a number of rows from the file (specified in the hyper-parameters at the top of this notebook) and puts them into a pandas dataframe. The code is a little long so I have hidden the cell so you can just see the dataframe that I will be working with for the rest of the task.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_line_of_data(file):\n    line = file.readline()\n    line = json.loads(line)\n    \n    return line\n\n\ndef get_question_and_document(line):\n    question = line['question_text']\n    text = line['document_text'].split(' ')\n    annotations = line['annotations'][0]\n    \n    return question, text, annotations\n                \n                \ndef get_long_candidate(i, annotations, candidate):\n    # check if this candidate is the correct answer\n    if i == annotations['long_answer']['candidate_index']:\n        label = True\n    else:\n        label = False\n\n    # get place where long answer starts and ends in the document text\n    long_start = candidate['start_token']\n    long_end = candidate['end_token']\n    \n    return label, long_start, long_end\n\n\ndef form_data_row(question, label, text, long_start, long_end):\n    row = {\n        'question': question,\n        'long_answer': ' '.join(text[long_start:long_end]),\n        'is_long_answer': label,\n    }\n    \n    return row\n\n\ndef load_data(file_path, questions_start, questions_end):\n    rows = []\n    \n    with open(file_path) as file:\n\n        for i in tqdm(range(questions_start, questions_end)):\n            line = get_line_of_data(file)\n            question, text, annotations = get_question_and_document(line)\n\n            for i, candidate in enumerate(line['long_answer_candidates']):\n                label, long_start, long_end = get_long_candidate(i, annotations, candidate)\n\n                if label == True or (i % SAMPLE_RATE == 0):\n                    rows.append(\n                        form_data_row(question, label, text, long_start, long_end)\n                    )\n        \n    return pd.DataFrame(rows)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = load_data(TRAIN_PATH, 0, NUM_OF_TRAIN_QUESTIONS)\nval_df = load_data(TRAIN_PATH, NUM_OF_TRAIN_QUESTIONS, NUM_OF_VAL_QUESTIONS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So this gives me a dataset with two features (question and long_answer) and a binary label (is_long_answer) to predict. This simplifies the challenge into a binary classification task where the model is given a question and a possible answer and returns a probability telling us whether this is the right answer or not.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Clean and tokenize the text\n\nWith the raw data loaded I'll clean up the data to make it easier for the model to train on. There are so many things that can be done here to improve a model. To keep this notebook short I have only removed html tags and stopwords (e.g. \"a\", \"the\" and \"are\"). There are however many more that I could have used to improve this model which can be found in notebooks like this [one](https://www.kaggle.com/raenish/cheatsheet-text-helper-functions).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def remove_stopwords(sentence):\n    words = sentence.split()\n    words = [word for word in words if word not in stopwords.words('english')]\n    \n    return ' '.join(words)\n\n\ndef remove_html(sentence):\n    html = re.compile(r'<.*?>')\n    return html.sub(r'', sentence)\n\n\ndef clean_df(df):\n    df['long_answer'] = df['long_answer'].apply(lambda x : remove_stopwords(x))\n    df['long_answer'] = df['long_answer'].apply(lambda x : remove_html(x))\n\n    df['question'] = df['question'].apply(lambda x : remove_stopwords(x))\n    df['question'] = df['question'].apply(lambda x : remove_html(x))\n    \n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = clean_df(train_df)\nval_df = clean_df(val_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that the data is clean it can be converted to tokens. This assigns a numeric index to each unique word in the dataset so that an array of integers (representing a sentence of words) can be fed into the model instead of a string. I've included an example below of how a sentence may look like after tokenisation.\n\n> ['how', 'i', 'met', 'your', 'mother', 'who', 'is', 'the', 'mother']\n\n> [2018, 44, 1275, 932, 312, 3293, 1164, 1, 312]\n\nIt's worth noting that I also pad sentences so that they are all the same length (which I set to an arbitary 300 words). If a sentence is less than this maximum length they get padded with 0 tokens to get them to 300 words. The model needs the sentences to be the same length but is also taught to ignore the 0 tokens. This is actually a good hyper-parameter to tweak when trying to improve the model.\n\nI'll use the words in loaded questions and answers then to define a tensorflow keras tokenizer. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def define_tokenizer(df_series):\n    sentences = pd.concat(df_series)\n    \n    tokenizer = tf.keras.preprocessing.text.Tokenizer(\n        filters=FILTERS, \n        lower=LOWER_CASE\n    )\n    tokenizer.fit_on_texts(sentences)\n    \n    return tokenizer\n\n    \ndef encode(sentences, tokenizer):\n    encoded_sentences = tokenizer.texts_to_sequences(sentences)\n    \n    encoded_sentences = tf.keras.preprocessing.sequence.pad_sequences(\n        encoded_sentences, \n        padding='post',\n        maxlen=MAX_LEN\n    )\n    \n    return encoded_sentences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = define_tokenizer([\n    train_df.long_answer, \n    train_df.question,\n    val_df.long_answer, \n    val_df.question\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer.word_index['tracy']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the tokenizer defined I can encode the loaded questions and answers. It is easier to input the sentences into the model as numpy arrays so I have outputted the encoded sentences as numpy arrays of shape (m, 300) where m is the number of sentences in the array and 300 represents the maximum length of all the sentences.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_long_answers = encode(train_df['long_answer'].values, tokenizer)\ntrain_questions = encode(train_df['question'].values, tokenizer)\n\nval_long_answers = encode(val_df['long_answer'].values, tokenizer)\nval_questions = encode(val_df['question'].values, tokenizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at what a sentence that is ready to be inputted into the model looks like.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_long_answers[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The final piece of pre-processing to be done is to create an array of binary labels for the model. The labels were boolean values in the model so they have simply been converted to 0-1 integers as the model prefers them that way.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = train_df.is_long_answer.astype(int).values\nval_labels = val_df.is_long_answer.astype(int).values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Import GloVe embedding\n\nWith the data loaded and pre-processed I can now define the model. Natural language models rely on an embedding layer to provide meaning to the words in a sentence. I won't go into all the detail here but I like to think about embeddings is as a grid of the words the model understands alongside different meanings:\n\n|             | tracy | excited | mother |\n|-------------|-------|---------|--------|\n| male/female | 0.8   | 0.5     | 0.95   |\n| happy/sad   | 0.5   | 0.9     | 0.6    |","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In the above example you can see some sample words in the header with some meanings in the index column. The numbers represent how far that word is one meaning or another. So the word 'mother' for example is a higher number in the first meaning category telling the model that it is more female than masculine while its neutral number in the second category tells the model that it doesn't hold much in the way of happy or sad sentiment. Of course those meaning categories are black box categories in the real embeddings and the human eye would never be able to understand what they really mean. However it is these numbers that help the model understand words as we do.\n\nIt usually takes a very long time to train a good embedding. As the meaning of words rarely change from one document to another (assuming they are in the same language) it is a good idea to load a pre-trained embedding. I'll use the popular GloVe embeddings for this challenge. GloVe can be loaded with either 50, 100 or 200 dimensions (what I referred to as menaing categories just above). More dimensions tend to mean better accuracy but also longer training times. I have opted for 100 dimensions here.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_dict = {}\n\nwith open('../input/glove-global-vectors-for-word-representation/glove.6B.' + str(EMBED_SIZE) + 'd.txt','r') as f:\n    for line in f:\n        values = line.split()\n        word = values[0]\n        vectors = np.asarray(values[1:],'float32')\n        embedding_dict[word] = vectors\n        \nf.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_words = len(tokenizer.word_index) + 1\nembedding_matrix = np.zeros((num_words, EMBED_SIZE))\n\nfor word, i in tokenizer.word_index.items():\n    if i > num_words:\n        continue\n    \n    emb_vec = embedding_dict.get(word)\n    \n    if emb_vec is not None:\n        embedding_matrix[i] = emb_vec","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define model\n\nFor this challenge I'll define one of the simpler models that can be used. For bleeding edge results BERT is probably the best model to use. However this works for now and is something that helped me understand the fundamentals of how these types of models work.\n\nThe model works by inserting one question and one potential answer as a pair. The model then outputs a probability telling us the what extent the potential answer is in fact, the correct answer for this question. We could run this model against all potential answers for a question and use the highest probability  to determine which answer is correct.\n\nWith this in mind, the model I'll use has three parts to it:\n\n- **Question encoding:** This part of the model encodes the question. That is it uses the embeddings discussed earlier and a bidirectional LSTM layer to determine the meaning of the sentence. This outputs an encoded array representing the sentence in a way that the model can understand its meaning.\n- **Answer encoding:** This is the same the question encoding part except it does this for a potential answer.\n- **Classification:** This takes the sentence encodings from the two encoding parts of the model, compares them using dense layers and outputs a probability regarding how close the potential answer is to the true answer to this question.\n\n![Screenshot%202020-07-15%20at%2009.41.21.png](attachment:Screenshot%202020-07-15%20at%2009.41.21.png)","attachments":{"Screenshot%202020-07-15%20at%2009.41.21.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAATsAAAEjCAYAAABNWaYxAAAK1mlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk8kWx+f70kNCC4QiJfSOdAJICT0UQTqISkhCEkoMCUFFbMjiCq4FFREsK7oiouDqCshaEFEsLIK9bxARUJ6LBRsq7ws8wu6+8947754zmd+5mbllMnPOPwCQQ1kiUTasCkCOME8cE+JPS0pOoeGeASwgAhgYA3MWWyJiREdHAMSm57/a+zsAks837eSx/v37/2rqHK6EDQCUinA6R8LOQbgNGS/YInEeAKgjiN9kSZ5Izj0Ia4iRAhF+JmfeFH+Uc/oko0mTa+JiAhCmAYAnsVhiHgAkW8RPy2fzkDgkeQ8OQo5AiHAhwj5sPouD8GmEbXNyFst5CGFLZL0IADJyOoCe/qeYvL/ET1fEZ7F4Cp7qa9LwgQKJKJu17P88mv9tOdnS6RzmyCDxxaEx8nzI+d3LWhyuYGH63KhpFnCmapIzXxoaP81sSUDKNHNYgeGKvdlzI6Y5QxDMVMTJY8ZNM1cSFDvN4sUxilwZ4gDGNLPEk3mJCMukWfEKP5/LVMQv4MclTnO+IGHuNEuyYsNn1gQo/GJpjKJ+rjDEfyZvsKL3HMmf+hUwFXvz+HGhit5ZM/VzhYyZmJIkRW0cbmDQzJp4xXpRnr8ilyg7WrGemx2i8EvyYxV785DLObM3WnGGmayw6GkGAhAJWIBNU5kmAPK4S/PkjQQsFi0TC3j8PBoDeW1cGlPItrelOTk4OQAgf7tT1+EtdfJNQtSrM77cNgA8ShEnb8bHMgHg1HMAKO9nfCZvkKu0GYAzPWypOH/Kh5Z/YJBfTwVoAB1gAEyAJbADTsANeAE/EATCQBSIA8lgIVIrH+QAMVgCCsEaUALKwGawHVSBvWA/OASOguOgGZwG58ElcA30gNvgIZCBAfASjIL3YByCIBxEhiiQDmQImUE2kBNEh3ygICgCioGSoTSIBwkhKVQIrYXKoHKoCtoH1UE/Q6eg89AVqBe6D/VBw9Ab6DOMgkmwBqwPm8OzYTrMgMPhOHgBzINz4QK4GN4IV8I18BG4CT4PX4NvwzL4JTyGAiglFBVlhLJD0VEBqChUCioDJUatRJWiKlA1qAZUK6oTdRMlQ42gPqGxaAqahrZDe6FD0fFoNjoXvRK9AV2FPoRuQnegb6L70KPobxgyRg9jg/HEMDFJGB5mCaYEU4E5iDmJuYi5jRnAvMdisVSsBdYdG4pNxmZil2M3YHdjG7Ft2F5sP3YMh8Pp4Gxw3rgoHAuXhyvB7cQdwZ3D3cAN4D7ilfCGeCd8MD4FL8QX4Svwh/Fn8Tfwg/hxgirBjOBJiCJwCMsImwgHCK2E64QBwjhRjWhB9CbGETOJa4iVxAbiReIj4lslJSVjJQ+leUoCpdVKlUrHlC4r9Sl9IqmTrEkBpFSSlLSRVEtqI90nvSWTyeZkP3IKOY+8kVxHvkB+Qv6oTFG2V2Yqc5RXKVcrNynfUH6lQlAxU2GoLFQpUKlQOaFyXWVElaBqrhqgylJdqVqtekr1ruqYGkXNUS1KLUdtg9phtStqQ+o4dXP1IHWOerH6fvUL6v0UFMWEEkBhU9ZSDlAuUgY0sBoWGkyNTI0yjaMa3RqjmuqaLpoJmks1qzXPaMqoKKo5lUnNpm6iHqfeoX7W0tdiaHG11ms1aN3Q+qA9S9tPm6tdqt2ofVv7sw5NJ0gnS2eLTrPOY120rrXuPN0lunt0L+qOzNKY5TWLPat01vFZD/RgPWu9GL3levv1uvTG9A30Q/RF+jv1L+iPGFAN/AwyDbYZnDUYNqQY+hgKDLcZnjN8QdOkMWjZtEpaB23USM8o1EhqtM+o22jc2MI43rjIuNH4sQnRhG6SYbLNpN1k1NTQNNK00LTe9IEZwYxuxjfbYdZp9sHcwjzRfJ15s/mQhbYF06LAot7ikSXZ0tcy17LG8pYV1opulWW126rHGrZ2teZbV1tft4Ft3GwENrttem0xth62Qtsa27t2JDuGXb5dvV2fPdU+wr7Ivtn+1WzT2Smzt8zunP3NwdUh2+GAw0NHdccwxyLHVsc3TtZObKdqp1vOZOdg51XOLc6vXWxcuC57XO65UlwjXde5trt+dXN3E7s1uA27m7qnue9yv0vXoEfTN9Ave2A8/D1WeZz2+OTp5pnnedzzDy87ryyvw15DcyzmcOccmNPvbezN8t7nLfOh+aT5/Ogj8zXyZfnW+D71M/Hj+B30G2RYMTIZRxiv/B38xf4n/T8EeAasCGgLRAWGBJYGdgepB8UHVQU9CTYO5gXXB4+GuIYsD2kLxYSGh24JvcvUZ7KZdczRMPewFWEd4aTw2PCq8KcR1hHiiNZIODIscmvko7lmc4Vzm6NAFDNqa9TjaIvo3Ohf52HnRc+rnvc8xjGmMKYzlhK7KPZw7Ps4/7hNcQ/jLeOl8e0JKgmpCXUJHxIDE8sTZUmzk1YkXUvWTRYkt6TgUhJSDqaMzQ+av33+QKpraknqnQUWC5YuuLJQd2H2wjOLVBaxFp1Iw6Qlph1O+8KKYtWwxtKZ6bvSR9kB7B3slxw/zjbOMNebW84dzPDOKM8Y4nnztvKG+b78Cv6IIEBQJXidGZq5N/NDVlRWbdZEdmJ2Yw4+Jy3nlFBdmCXsWGyweOniXpGNqEQky/XM3Z47Kg4XH5RAkgWSljwNRCR1SS2l30n78n3yq/M/LklYcmKp2lLh0q5l1svWLxssCC74aTl6OXt5e6FR4ZrCvhWMFftWQivTV7avMllVvGpgdcjqQ2uIa7LW/FbkUFRe9G5t4trWYv3i1cX934V8V1+iXCIuubvOa93e79HfC77vXu+8fuf6b6Wc0qtlDmUVZV82sDdc/cHxh8ofJjZmbOze5LZpz2bsZuHmO1t8txwqVysvKO/fGrm1aRttW+m2d9sXbb9S4VKxdwdxh3SHrDKismWn6c7NO79U8atuV/tXN+7S27V+14fdnN039vjtadirv7ds7+cfBT/e2xeyr6nGvKZiP3Z//v7nBxIOdP5E/6nuoO7BsoNfa4W1skMxhzrq3OvqDusd3lQP10vrh4+kHuk5Gni0pcGuYV8jtbHsGDgmPfbi57Sf7xwPP95+gn6i4RezX3adpJwsbYKaljWNNvObZS3JLb2nwk61t3q1nvzV/tfa00anq89ontl0lni2+OzEuYJzY22itpHzvPP97YvaH15IunCrY15H98Xwi5cvBV+60MnoPHfZ+/LpK55XTl2lX22+5natqcu16+Rvrr+d7Hbrbrrufr2lx6OntXdO79kbvjfO3wy8eekW89a123Nv996Jv3Pvbupd2T3OvaH72fdfP8h/MP5w9SPMo9LHqo8rnug9qfnd6vdGmZvsTF9gX9fT2KcP+9n9L59Jnn0ZKH5Ofl4xaDhYN+Q0dHo4eLjnxfwXAy9FL8dHSv6h9o9dryxf/fKH3x9do0mjA6/FryfebHir87b2ncu79rHosSfvc96Pfyj9qPPx0Cf6p87PiZ8Hx5d8wX2p/Gr1tfVb+LdHEzkTEyKWmDUpBVDIgDMyAHhTi2jjZEQ7ILqcOH9KW08aNPV/YJLAf+Ip/T1pbgDU+gEQvxqACESj7EGGGcIkZJZLojg/ADs7K8a/TJLh7DQVi4QoS8zHiYm3+gDgWgH4Kp6YGN89MfH1AFLsfQDacqc0vdywiJY/hpFTl8FK8Heb0vt/6vHvM5BX4AL+Pv8TZuQX02Ir5hoAAABWZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAOShgAHAAAAEgAAAESgAgAEAAAAAQAAATugAwAEAAAAAQAAASMAAAAAQVNDSUkAAABTY3JlZW5zaG90iExFMAAAAdZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MzE1PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjI5MTwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpjtM7aAABAAElEQVR4Ae19B5wUVbb+6TDd05NhyDkKimRBQFBQdhVFXbOu65o3mZ5Pd1efYXXddVf/+lxd3dVnzjnrmhMiggqSQUUyDBlmmNC5//e7PaepbrpnhmE6VPU5v191hVt1w3dOf/fcULdsESUkIggIAoKAxRGwW7x8UjxBQBAQBDQCQnZiCIKAIJAXCAjZ5YWapZCCgCAgZCc2IAgIAnmBgCnJbvPmzRQIBLKqoNraWlq4cGFW8yCJWwcBjBPCpsPhsHUKlWMlsZlpNPZf//oX/ec//6GGhgay2WzUo0cP+vOf/0y9evVKK6zBYJA+//xzat++PQ0fPlyndckll9Dy5cvpjjvuoJEjR6Y1fYnc+gjcf//99OKLL9KAAQPogQcesH6Bs1BC03h2TzzxBL388su65jvyyCNp0KBBtG7dOvrd735HNTU1aYVu9+7d9Je//IVuu+22WDqnnnoqjR8/ng444IDYNTkQBFqLwDvvvKMfXbFiBe3cubO10chzTSDguElJE+E5EQT3/ve//73Oy9NPP03HHHMMHXfccbR27VqCcYCMJkyYQM888wy99957dOihh5Ldbqdly5bR448/Ti6Xi7p166af/+677+ipp56iN998U5PkgQceGCvjDz/8QE8++SR99NFHVFRURF27dqU1a9bQQw89RKtXrya/30+bNm3S8cOrQ7pDhw7V9yKSTz75RMf96aefas+zT58+Ou4vvviCXnjhBSotLSXk/91336V27dpRly5ddLj85DcCixYtojfeeEPbLJqzDoeDRo0apUGBjT/44IOE6x9//DE9//zzFAqFqG/fvvp+3LRt2zZ66aWXtGeI/0rPnj31Pf/4xz9o48aNdNBBB+m44D3iHI4C5J///CctWbIk1jJJ9d+A7YKMETdaV0gbrRyzidMMGYZCICCIjh07xrJ87rnnEoiFw1955RVdK1566aXaYBYvXqyV5Ha76ZBDDtFNUSO3z549m7755hv661//Snj2vvvu0ySFBGBYAwcOpHPOOYdmzJih04SyP/jgA7r88ss1YSHdY489liorK3UceIYFz0ybNo2uvvpq/fyHH36o88LhM2fOpJtvvpkmTpzIl2Sfpwg899xzuuRXXnkl3Xnnnbqr5qKLLtLXVq1ape2GPT9c/Oqrr2j+/Pn0xz/+UVfGF1xwgb7X6XTSl19+SXfffbcmPtgcSBKtkPXr1+trBQUFdMIJJ+jz1157Tds4nkc3Tar/Bu5DPznnYcOGDbq5rRM10Y8pmrEgLUhi31j37t31ddRWLRE0RSH33HOPrgmLi4sJhFdVVaU9MoQ99thjWqkVFRXacwRJomaDdO7cWXuO8BSNgvyB6BDfww8/THfddRfB8GAcMDKWMWPG0Pvvv0/nn3++vgQPTyS/EUAFCvKCvaDFAq+purqali5dGgdMeXk5vfrqq9rLQ8CsWbN0+KOPPqr3IEe0akaPHq27ehYsWKA9OvQ3o1WCMAjSgyc5Z84cfc6VbVP/DX2j+kG3DbxB/CfMKKYgOx6hgvtulLq6OuNpk8dojkLxMCo0GeDSo5aDoEnKfW8YeAAZ3nrrrbqGhVfYnLDhHH744YSm67Bhw2LEzGGIY8qUKdrjRJ8jBE0UkfxGABUe7Hvs2LG6WTp9+nQNyLPPPhsHDLpLysrKqF+/fgSbxGwAyIgRI/QelfT1119Pxx9/vK5k0a0zadIkHYYK/bPPPtPP4QKIDy0aCGyyuf+GvlH9oJWCJjEqdTOKKZqxIA8IaiujMJG0pO8LgxkQHlnleGA4uHbLLbcQ+jhgFBjxxYaajGs8vj/ZHv14EPSVsHBzOxkho09GRBAAAhh0g4CQQHSwRT7nY33B8IOZCCw/+9nPtLeGPjs0YbF16NCBHnnkEZo8eTLde++9uomKpie6VdD8hVcIx8Hj8RBaR+hSgaT6b3BaZt+bguyGDBmicd61a5fuO0BzEsJ9ZIMHD9bnhYWFer99+3atRAwosKBGhMCbw+AECAf9GWjCIj645xjEeOutt7Snd8UVV2jDQVwY7GhKEDfygtryjDPO0LcyMWOqCgxNRBBIRAD9YKiEYV88gIZ7tm7dSj6fT3tgJSUliY/FnaO/D5UtZivg/4GBPNgbBtnQN4fmL1ouEJAdumfYaxw3bpy+3tR/A/niprK+2cQ/piA7EBQ6UVFbYVAC/QwgKSgRBIcBCQg8MQw0XHfddXqU1NgnBqWhFoMhYNABLj5qOMSDES6MpKJpAMOBp4imBWpQeH6oAWGQW7Zs0YMYF198cZzKMTKMvM2bN4+uuuoqgjeHdGCoIGp4iSKCQCICmFcHOeqoo+iaa66JBWNg4W9/+5v2+mDvTQn6i+HNrVy5ktCNws3bTp066cfQh4eKGP8h2CIG+ZjscD+kuf+GvskCP027LDlUwLPPPpvOOussTT4gJq6t0HTkfrWTTjqJevfurWtLEAxGSSHs9mPgoH///to7hNsP8jrvvPP0fRjBgiHgORAXhtZhgCAseIFHH3209gRBpsamKeJGX8rtt9+u+zLQTMAUFniLmByKPsJUwvlKFS7XrY0ABqsgsFujoPmJyhV9aZjeBEllK9deey2hZYOZAf/+97/19KhTTjmF2GtDnxwEpAcBscHbg/DgBI6b+m8gHNJcCyd6V+7+muoNCoYRTcsdO3YQFI0JmCA4EBQLrmGeHJMgX+c9mreYiIy+jUSpr6/XfRcgsEQByaGvI1kY34u0kS7SFxEEMoUA+tvQjE1m0/uSh6b+G/sSTy7ea0qyYyAxjI63Gk477bTYREkOk70gIAgIAkYETE12xoLIsSAgCAgCTSFgmj67pgohYYKAICAINIeAkF1zCEm4ICAIWAIBIbsENWIm+ty5cxOuyqkgYB4EMEh2ww03mCfDGcqpkF0C0KvVe4QY6RURBMyKAEZU+X1ys5YhHfkWsksHqhKnICAI5BwCQnY5pxLJkCAgCKQDASG7dKAqcQoCgkDOISBkl3MqkQwJAoJAOhAQsksHqhKnICAI5BwCQnY5pxLJkCAgCKQDASG7dKAqcQoCgkDOISBkl3MqkQwJAoJAOhAQsksHqhKnICAI5BwCQnY5pxLJkCAgCKQDASG7dKAqcQoCgkDOISBkl3MqkQwJAoJAOhAQsksHqhKnICAI5BwCQnY5pxLJkCAgCKQDAVmWXaF63333xT7AjeWd8HlGfDAHXxXDV82MX2FKhxIkTkFgfxHAJ0DxQXcs74SPQuHLeV27dtXRwp7vueee/U3C9M+n/s6f6YvW8gIMGDCA3njjjdjX2PEkPqCNz9chTEQQyHUE8K1jfAMZXxhj+fHHH7UN8/dh+Xq+7qUZqzR/2GGHxREdjAFEN3z4cP3B7Hw1Dim3uRCYPn26/hi2Mdfw6o488kjjpbw9FrJTqseHsEeNGhVnBGjG4sPYIoKAWRCYNm2abpEY8xsOh6UbphEQIbtGIE488UTdV8eGgo8Ow+MTEQTMggCasp06dYplF33OUmHH4CAhu0YsMAiBWpBl/Pjx2uPjc9kLAmZA4Be/+AW53W6dVafTSccdd5wZsp2RPLbZAAU69M0uILgZM2ZoY0E/h9nLhH5HkX1DwOw6nzBhAt1+++260Oiv69+/v9hxowm0iuwSDaK5830zt+zdfeaZZ9LMmTPJ5XIRiM/o6WUvV/ueMpMc9MLHHEviOV/Px31TdpsYZhZ80Nc8cuRI+vbbbwn2bHYbBu5tZcf7RHZGA8Bx4sYZM4thJOazd+/euumKJi367MwqTGjYGzdjefge47V8OWY7Nu6NtgwcOMyMmGBUFmQ3btw4sWODAltEdqx43qO2wDH2vPG50VD4fkN6OX+IDt2xY8dSQ0NDzufVmEEjeTHB2e12TXbYG48T7zXGY+VjtkfseWP7Ndo0hzEW/Byf5/p+xIgRNGbMGCorKzOtHbMN8z6ZDe+rHTf7BgUrGnujYWCWNrwf7Ku8EVrntdPOAFGtcojC6l6zSqChjgo8xWbNvs63Ww07tSsg6uCK0OAS0m+CYGSON6PhsMHw3tQFT5F5ow3jGBvsNnFbVmujbX6i6qCNvCHz2jBgqNuxhYrb7xmZTQFNzl62q1ZJqXLFYMe9PWHq5LbF7DeZHaMgzdlwk2THRsIkx8YRCASozh+kD7Y7aE61k6qCKkciOYmAxx6m4UV+mloZon7FNsIIHW9G0kPmmzOWnCxgM5liG8beaMeoqGHHK2vD9N72AlpYX0C+iExOaAbOrAX3KAjQIaUBOrIyTGXuPTYM4uNWC+y3KRtOSXYwDt5gJGwcePdu1o4IvbKtiHaFHVkrvCS87wgc4PbTtA4BGlFh1zPtCwoKdG3JxoIYmzKWfU8xu08kEh0qa7bjHQ0Benmzk2bVesjcPlx2Mc506m5bmA4v89FPOyhvr8ip7RiVN0iPyS6VDSclOzYSkBwbCEiuweej56uc9GmtuZt5mVZQLqVXYg/R5V13U/+yAj3qDMKDsTDhpTKUXCpDS/NirKxhx/DkfMqG1+wO0L83ldDWYIu6rFuanNyXQQRGFDbQxd19VOwp1HbMhAc7hiSz473Izkh0IDsYCIhuW52P7t9QSCv80QmLGSyXJNXGCBRSiH7VpZaGVTj0WyOJHl4yQ2njLKQ9umRE5/V66esdYXpsa4k0WdOugfQncKC7gX6tCK/c49ZzY2HHILtUhJe0kwKGwk1X7dGpkcnXNtuF6NKvv4yk4CUHPbvFQ9X1Xu3poGkHfXNFl5FMpDERLgfbMXt0VbV+elZ1v0jfXBrBz2DUy3ween+rGkxSlRh0DDuGzln/iVmJIzu+KdFI5qg+upn1pYnPyrmJEdgadtGTmwq1oaBCQzOPCY/twMTF0wbPFTb+CPhDPLnZQzVhabqaWa+JeX9ndzktrg7HKm3YMdsv7/mZOLLDRSY67sit9fro1eoyvl/2FkJgvr+ElteEdDeFFb07Jjv006HC/j7gsZD2pChAAINLr+8sSlppJyKUlOx4UAJGsqDaRtVSGybiZpnzD6s9ulaE92MF7w6VNVfYKA83YWfUFllGZ1KQeATWhzy0sDrSbKUdIzujkXCNiObN0gaZQxcPrbXOVgaLqM7n16TATQDYgpmFyY5bJ7u8AVqn/hAi1kVgcYNLkx1X2sxnRluOkR3DgED27EB2PwSkRmRsrLj3qcGKFXW2GNmhojO7MNmxZ/d9vUONP8sKMGbXa1P5XxkojPPsktlxHNkxG7Jn5wsEqSYinl1TIFshrMpnjyM7tgMzlo3zjj2T3Ra/EJ0Zdbkved4WKaQGf3RENlULZS+yA9Ex2W33q/4PWRNtXzA35b01YXvsPWfoHkQB4b1ZCmXML8rBLZSdIXnTxyw6bG0+w8pzrwlEKzijDRvjiyM7BMBg2FD8QfM3aYyFlePkCNSFbJoYUtWIyZ/K3atsw2zHwbC5+yBzF+ncyll9IFrBQe/JCC8l2fEDuVUcyU06EAA5GIkO59jMKJx37NmGzVoWM+KfzTyzzlPpO0Z2uIFv4mM+z2YBJO30I8D6TlYbpj/19KWAclmtTOlDy/wxc+XGOme75pLFyI4vMMEl3sjhsrcmAqxv3pu5lFwG3pu5LJL3liPQnL73IjtEjYeMe30iP5ZFoDkjsWzBpWCWQsDIW3xsLGBSssMNyW42PijHgkCuI8AkLrac65pqu/w1peuUZNd2yUtMgoAgIAhkHwEhu+zrQHIgCAgCGUBAyC4DIEsSgoAgkH0EhOyyrwPJgSAgCGQAASG7DIAsSQgCgkD2ETAl2YV2baFIUH2kNosSaail4OrFWcyBJG0lBPTIMWxaveokkh4ETLVGtffdBykw9z0if4NeoMBe2Z08Z15Pjo4904NOY6yRUJCCS2eRrbQdOfsM1Vfrn7yRwhu+I8+5t5Kz3/C0pi+RWx8B3/sPU2DWq2Tv2p+Kf3OP9QuchRKaxrPzffoMBb58TU0ADJNz6BHk6HYARbatp/r/u5Ii9TVphS7SsJu8L91G3lf/N5aOa/zPyDHoUJWPAbFrciAItBaBwLwP9KPhqh8pXLuztdHIc00gYArPDk1W/ydP62IUX/YA2cs76uOGF/9OwcWfk++Dx6jwxMvJN+MFCu+sosLpl5JNfTQ3tH45+ZUnWDBkEjkHjNLPBDd8TzCsSM02cvYfSa5xJ8TgCW1coe8nXx0VjJhKDhUe3rqW/J89q+8J795JDa/freOPKO/SVlxOEb+XbIXR7+gGFs2g4PJZqikSIefBE3W6eDCw7EsKfv+VjjOw4GNNzq5Djydn32GxtOUgfxEIrllC5K1VrRU72VRl7v/ydSr8yXkakNDWdeSf9Qo5DxhLIMLQxh+i9jxsirZx3BRWthyY9z6Flf06DpxABQdP0s96/3O/avX0IteEk6Ln7z1E9vZdyTXmuFi4zV1E7qN+qc9T/Tf8X7xCoW3r9H8I/6fCn5xPDuWBmk1MQXahdcuiuJa0ixEdLrgmn63JjsP9c94gUrViZNqvtSEE1y6joDICcrq1ogJLvyDv87fGdBRSBBT88VsqOvtP5J/9Ovne+b/Y+n3BRZ+pJsUAch1xJgWWfKHXubWF1OKAiqzo2N9S8NsPCOkWjPop2csqqeGl2wnPsASXzqTgqHnkOfEK1QT+goILP4nmpfGGhmWzqPDM66hAGadIfiPgn/mSBqDw+EvJ98Y9mriY7MJbVmu70XbcCFPoh2/IuXoheU76bwptWUv19/02GuJwUvC7OeR7+19UcvWTFFigbE59kgZkF9q2QTeTIw71cXRFdvp8zpvaxkF2Tf43vnqTIqqfnPMQHn20KcnOFM3Y0NqlWpmJnpC9fTd9PbSjqtEMmt55FSFBii68g4p//xSR8shAeKEdm8j32fM6rPjSB6j0+leV11ZB4cbarOTqJ3SYraITld74OtkKXPqcf4Iqf5roVHxFl/yLPOffRgTDQ227fQPfRo4Bo6nkxjfIdeQ5+lrg2w9jYXKQnwig1RJU5AV7cY38Cdk79yGqr6YgV/AMS1E5Ff/xWSr67b36SnD5HL33fazsWIlr6nnaNu2qNRJRgxzB1YvI2XMwkepvDm1ZQ4H5UVvTFbbyJEM/fK2fcx44Xu+b+m/oG9QPum08F92hWjzRVhJfN8veFJ4d+ukgkXAoDteIam62VFADQvEwKv/Xb+vHbKqWw5IHYdW0Rd9baMVcqn/wv8l50GFUqLw9Z/cD9H0RX32TyQS/bzQc9ZyjU299r6PvcB0fh+FiwcGHa4+zQPU5+j9+UjWR1zUZrwRaH4GAaiGg6eoYMJbUp+zJOfoY8qvmp//zl8j58xtiADh6DyF7kfqkKbYCt272ItDRdyiFln2hunme0i0N1yHTyHnOLWRTK4zDGwsp0oMNBpfMjD4X8Gnii+zeoeN2Kpts7r/BmUBXkV05AWYVU5Cdo/fBGt9wwlQPuPMQe7vOet/UT3j7+mgwRlZVEzImMJxwkDxn3UDet+7TRhGc957yyt7TNVnRz2+M3ZrqILJrsw6yV/aI3WIr66CPkxKlXZYJjwGV5wfoPoGgT3f3X08ljPzjixk4x3Fy2fNNDbfq+0UljsG7kGrCYoPtFasWhlP13fneeUDPJIjs2EhO1eUSWrVQ9SsrrxCOg8tDjspuqk+58f+Q4r+RPA/mu2oOsut5oEY2UreLwqq2sqvmJCTQ2EfmaPTAbIq44KnpWkspkYJ+fR9+HJ376mP0WZRe+6L2sPTcpp2byFbeidCZi87b0uteUgMb31H9I3/QhhOu2a5r3FhESQ7siFvlJfTjPKKJp+g7UKNCHGqqSnj7Rn0sP4KAEQE9X1TNKMDABGwPAhrD4Jmt0QPjwS8dmOTHp/r7wjs3U/Hlqr+5rprqH7+OQGyBRZ9GByJU8xdTpCCukT+lgDoPzHxRnzvUoAekqf8G8uVTrRAriCnIzuZUnaqqE9X/0RNUe8+vVKf+eK1grcSCQipUAxIQp+pTCKiasuGZm8nZawj50URo1BKUZkMfnzKEunsuIqcaGEBnbliRXcl/P64GOmbopgEMB54i+j1UW0D1zynPT9WAMMhI9VbyqkEM99TzGmON7gpUh61P5S20cj7VP3YtRbz12uCosIQciqj13MC4J+REEFCVtZpXB3ENm0yFJ18Vg8SvBhZ8r9yhvLXXyTXl57HryQ7C6C9WdtyweZXufol4o107trKO+nZn/xG6PxmVvKPXgWQrqYiRHbprIM39N/RNFvgxxQAFcHYffgYVTDxNkw+IiWsrewfVdAQhKcF0Drsaasf8u4BqhtpL2+vr6L+AFJ3/d7J36af7MuD2g7zcU36hR1OLLlSDF2q0F81XkCqeLTzpKrJ5SqL9bCOnqr6ViCZTY18h4kZfStEv/xId8FDNhHDVCsJgBiaH2lQfIUt0SVRVc/OFxnzxqezzC4HAgo90gZ1oihoEU0dQuWLaU6S+tjGEq23DjerQrUjS3n2Q7q/zqaklaM0UqDmgBYOiXhv65CCYZgXRHqTy7iA8OIHjpv4bCIfYVJ7MLOr/G12WGOu246MrPp+P6urqaNeuXbRiex09GD4o58qHpiUmXjY8fZOeamJTBFdy6b9j+UQY5g9pryx2dc9BJOCnSEONIrlov9qeENWVgcEI1XehO4ONAepY15qqr8OGTuIUoieEKvK1q/TNIsNCVXRqu93Url07Ki0tpcLCQiooKFD95Wrel4kIGaYMO8ZX4b1eL9XU1NDOnTvppV1ltMgRbSaaRSf7mk/076GbJ5lN70tcTf039iWebNx7sX0pDagspoqKCiouLia3200ONd8WdgzZ43ZkI3etTBPz2rCVXPmofquhoHHSJEdnVx5aU4KpI7aCvYkOzzRFUs31n+jnm0m7qXxJmCDQWgTQguBBsdbGgeea+m/sT7y58KwpyY6BQ1+e57Q/8qnsBQFBQBBIiYC5G+EpiyUBgoAgIAjEIyBkF4+HnAkCgoBFEch5stNz4dQrNXitJnHLtE6C6kXrhmf+TJj1DpE17TKtAROnpwZOEu1Xn6uBhUyLX72mCDvGi/+QkHqHPKwGN6wuOd9nF1Cvdvne3jPSalSI5+yb1GoQY4yX0nocVi9TY05TRPUVFqj3GGVNu7TCbanI69XcT37jJ7FgpTe/nXgpred4LRJ27DhoIoXU2zz1D19NNrU2ZImamGxlyXmya5wZoyYCle+1SKYdb0lkUbCmHWaqy5p2WVSCWZKOzvDS8zz13FDOd5ZfHUReMMEeE/KtLjlPdqwAvAidbOS1JWvFYdUHrBFGylV3qsnBzsHjY/PHMK8uqNz6oKrtsNIJJiYbySu4Yh75v3lHvzJmbxc/V8u4pl1YvVjd7Lpj6lU3/2fPUXj3dm1gmNSMRQ54PTEuq+yti4Br3ImqVTA1roCo0L1qaSesPUeeUvV+9ufk6DFI2eIJelI7bsY9eDsouHKBWmyiFxVgAr1hniiaon6slVi9Rb+i6J5wsrZZTsg/+w29ugpetYx7X1u9lkZq4nwk4NW3+j5/UU2630zO4UdSAEumuYrINfFU/Q5tLC71fwgun63exlArdw8ep71El1rAAHnOZTEN2aUCsbm14kBW9U/dqN9+QBxw3/HaWMkVDyoF+6n+X5foNyo4/qBaCqfw1D9SwdDD9b3o22CJX3NFxWVY0w7vM2JJJ17zC88Y1x3DROi6ey7WE5Y5TMerXncTstNI5O0P3szRtmNAALaD1wxLrnpcX61/4L/0mzk4wbJkWAOPu3H88z8m36t3xp7WzypCKr7iIV2pe1U3UOCrt3R4og1jErxeiky93+0aO13fB1sOoIJvFLxVVPI/L6mJ+h7yvnlvfFhj/zVW+8l1ssv5AQoGPKRWZqi56fjYVv/crRyk96nWivOqlYVhTK6fXEDF171M9m4D9XureNnf9/ETmujwuk2JWivMPf13Oi7v6//Qe//X/9F716TTqOSG19QqsNF3CfXFVD+quZ103bGPlNHizQy1ICjW0vP8vPnVVFIlIdfNi0CDsi2jHetWQ2NxIuptFc+v7qKSa56Pvo+tSAdkhFVJ8AoiXmcsVhPpPeq1Rwhe0Nde4Wv/q88L1co9CMerihH1zrcfnzFQElALxyLuwp//iYr/Ry2CYfAI9Q1JftzHXKzWXnydeMWhgHpFE28Q+ee+q+8uPP0aKsG6j3hd0yRiGrLDQpuOPgfv2dRLzUYxrhWH61grDoaAWgqKdh12MtldheQ5/Vpyn3C5eqmwSC93g3vdR5yhXwHTy1Wrl/5JufZYXYVXLnGp92cxgZlfnMYzqYTXHXN06Ru37lh4ww/6Eax8jDc88qGPJBVG+Xzd3q7LHhtW9swL0AITm6dMr6GI97G5KyW8ebVaYGKBhgzrIKIZio8+FZ7xP1Qw4RS9TDsqcxBcgWpSIrwATVgl8PD04rGKpJAu3pfFG0KOXs2/AupUA3B4K4PtNKwWGghtWqkdB52W+tQB3rZw9huh0zLDj2masVgMs0gtY96sGDt81XQViH5PtvEdT6xo4lKrlEAitdHhdiwewBL9roT6eln9bk16WGTA+DI/39eyvS12G/eJoL9FJH8RcB9+5l59dkk/n2iwY3zwCaJf4m+ErqBxxZKA6t/TYWqZMhaHIjZIxNegVwfCMX+3Bcf7JI3vlepnGhextZVW7lMUuXKzeTy7ViCml0/HqiOqZuMvkGFAo+6J6ymgaj17+6hRBFQ/HiSiFkIIq85ZeIL2rv2iLjq8PNWU0NLYiRs92bdfNjZeuRhpiQgCLUGAK+Pgim9jt9e/8Df93ROHWsUHgo9Fsehl3tUJlnSCkwB7DlWt5GDdVx072YcDHkUOb1q156lGh2LPhdw9Mo1nF1Lr5kPBRkEt2Zw4lbuNj93U3ftb7ZJjxArr8NtVn0TBpNMptOZPhKVxwupLZKFVi7SbjnX8sdoH1s3zq9Gpun/+Wo+eBhZ+Glsfr7l0E8MLxkxXaS0hr0oL3xcIG4wv8V45ty4CWJk40Pj9By6l56Sr+TDpHnM6fVh2/bvZejIwKm587Mmu1krEt5Mx4IYFO2vvvkgtcdZbrXKsKm/Yr/pCHr6yh8U5w6oJirUg9ecHVDytEZ2W8uoiajZB7V3qC2N9hqmP+nzU6v9Ea/KwP8/kvGcXW2JIfYQkpNbRN25hNaWEhdeI4z2UDXGrLzY5VL8Clr/BGndqZX7dZ4fRo4KBh5D7pxequ2z6K2VhlQZWFi5Sk5UhrsNOjXp3yjPEyKu9PPlKKbE86qf2NF31aeMPRnd1P4jqX0EZbGqKgUgeIdBojyAdow3jWC+RngoKrGKsVvjxnKG6cFSXCmYTaKJTHl3RL27WTxWdd6u204j68BQIEd02HtWnp/uN1R3uEy5TzxZSRH38CR+GSjpA0Zi/WDYaz2P/p8YAD9JU/ef6a2OwYxN9k8KU69nFFLIPB1jvC3PhHKoDN5lgiWwHRqmMfRSNN+pXadRKr3al5P0RvGbmVLPWCZ9kXLWAvC/8XY92FV1w2/5Eu9/Pynp2+w1hxiLQnwlQdojBtkRBHx36hlMtcQYbx6K0re+DVqarHAzM00O3DOb51d55rh4ELLrk33r+X2KeMnluyfXsWgMgFJyK6BBfU2Ft8UUlTAb1vqamtLz7oDK4Sr0KLdLFcvMigkBLEYCXl0owDw5bKmnKxlM9k3i9QX13GSuB45OPYUyKV60eLJ5rhoG3nG/GJoJt1nNnv+FqHt8letY5BkEwfI8mtlO9GSIiCJgFASzf7lDdP9FPMdr0t5CLfnmLKbJvmgEKU6DZTCZdY45VX3w6tpm7JFgQyF0E0ETmvsLczWXynIlnlxwXuSoICAIWQ0DIzmIKleIIAoJAcgSE7JLjIlcFAUHAYggI2VlMoVIcQUAQSI6AkF1yXOSqICAIWAwBITuLKVSKIwgIAskRELJLjotcFQQEAYshkJLs4t/3tFippTh5gQBsmLe8KLAUUus7FQxJyY6JrtgmyxClAs5K1z0W17OHglZSl5QlBQLMV8xfibftRXZ8I/Yee5js6oMwItZGoJz8MQ8IemcbMGupuQy8L45EF3E1a3kk380jAJ4CX7Ht8t74ZIzs2DAQyMfYV4aiq6QaH5JjayHQweZTi73YY4ZildLBflGuTvYGqxRJypECAfAUdM12bOQwfiRGdnyBDcSBRf/U1j+yg4Nkb0EEPGEfdbOrJXsaDcVoLGYsLhu5sTw9FNl5Ql4zFkfy3EIEwFNGnSd7LI7sEg0FZDeAqpM9J9csgkC/wGYqcKrlr5SuzU50rBKjHaNcKF+f0HYOlr0FERhA6pvQSs9MeGwDxqLGkR0C+GY8WFBQQJ2dAerl32J8Ro4tgoA9HKKxtEnrGbo2Eh6KCIMxkxjzCztGeWDH2EZHqsgRkQE3M+mzpXnt69ukeCoY0zVzWOLzcWTHbIibmexcLheNC6xOfE7OLYDAYN86alegFhBVOoa+mexQNCNxmKmoRhvWXp0icZSvU0GYBnnXmakoktcWIjA+tIbcbnecHbMdGKOIIzsEGGtEGElhYSF1c4fpsN2LjM/JsckRqPTvoImR9Vq/0DM8OxAe9G9WomOVIP/GCht/BNjxYeF1VOmTPmjGyQr7w3YvpE4u9V0ZA9lB99gSJXaFmRB7dv/ZSDweDw2z76ARNUsTn5dzEyJQ7t9FR9ctpJJCF0G3IAIQHvTOdmDCYsWyzGVgzw52jHKWetx0dP1CQvlFzI/ASMVHw+w7qahIfWBI6ZcrbaMdwxZYYmTHFxDItSIeRiTFxcV6G6P6PSZum0OOkJ9vl73JEOhRu46Or/mKOhS7tE6hX5CBsRnLZGGyounsct5hw0x2IHO2Y5R7uip/j91rzFg8ybNCAPwzadtsOkTxUUlJiSY7rrCbap3stSx7ItmF1Mecg8EgYR9RnwE8sKaGKrd9Sl8VDaKq0t4CvkkQKPTX0fDqRTSYdlJZWRmVlpZqQzHWiCAI6N8KwnaM5jlsF+XEPhwO6+2o3UvpO28VzS8fSl7X/n01zgp4maUM3WrX0pi65dSp0KHtOJHsUMGlsuM4soOBgNCwx0NgSdT6MBBc3xNWS1PrFtP62pW0qrAbVZX0IF+BGEwuGkzH2o3Uo2EDDVBTTMpUM664uFwTHcgO7j/XiEYjMTvhcf7Zu0MLhUkONgxB2JC63dRTVdwrCjrROk832lbSPRdVmPd5cgfqqEvteurn3UCYM4mWJkiON9ixsXXC+uc9AxhHdnwRN2ED2THJIQzXmARRY7obGqiHbwX5Ny2l+oiD6mwu8tKeKNmwOF4z7JfP/pS6DxxCpZUdzZDdWB5ZsVpH6tWZooiPStWm9aQqrMLyKLlxlwQMxGgkqWrDWAImPAAWKBdX2lwEtmNg43LVU4l3h2qxVFFg+2yqtbmVHbsppD5OzfbLe34+1/e7t2+lDT8socHjJud6VvfKH9sxAgrVO83FET8VqXe3UWG5VWXt8VRosjPaMSps6BJ6xvPGOIwJ7GGmxqu4EcqFkUAQAYQjAdlFjSTan+fz+cjv91OZauqiuRsOB0xrJCjnoh2bqVOkN3V3l+PUVMI6gu7sdvTDFe8hu8Z+KzTnsKEmxAZdWsmrY4WxwbMd83XsmQBRdvxRvF4vwY4DgYC246gXGIyr6I3P5/rxhkgtrVZ2PMBtvr511hv2sEuHo1hzkCY7VNpKX6ikYcM4hg0jDDwVtfsob3E8Rl3tRXYIxI1GwsM5b+zZIQEmOhgJNu4TwbNmqw0ZFPwB0KdVWZn6Y8R8b67tWUfYs55gBGwoTHBMcggzEl2ulact8gMsjIRnxIbJju0YlbWxf9qsdrx7925diZnRhqFztmPoDfYJPWFLtGOc43pLiA7xJiU7TtBIeIjQaChICMwKkmMjMfaLmJnsysvLTU92bCjQGxsL75nkYEhsWKxz7K0iKBvsEHvgwWVlbIAHiJ8rayY6Yx+1Ge14586dWudWIDvoCvaayo4RzhvsFjpOJSnJjh9kwktmKGwc7NEx2eFZMxoJ8g1Q0fHZrl07nJpOjHriPzVILXHj+3hvuoK2MMMon1GAA/85cAzCg/3yxjYM+zWrDcN+YcdmtmHojPVktGOUy6hDhLGOeW/Ut/G4SbLDjYgASseeE4dBIEEkbDQOHEPYSHivL5rkB+VCfwAMxkxiVDSOWV+8Z91xGPYQ3puprK3JK5cTe7Zn6JrtF3tc573Rdo3HrUk708/AflE2s9kwcDLqCedMZmy/xvNk9+NaKmmW7PAgZ4AjAZAwACSMPW8IN5thcJl4jzKhaQODMauwvrA3biiPMcys5WttvrnsjEMyG2b75X1r08rmc7Bf2LGZbZh1xHujHSfqsaVYt4jsODJOBIbAiSMs0TASz/l5M+xRLpA5vFazCusJ+TceJzs3axn3J9+pMDHarfF4f9LKxrOwX5TRKjYMDFPpbF/wbdU/OlnCRuNIDN+XDGX7XuQdtSI2s4uZ9ZAJ7BPxwTnbcWJYJvLTVmnAdpF/K9gwMGkrXbSK7JIppa0ylCzubFyzWnmygaEZ07SS3q1UlrawJfO7L22BgsQhCAgClkdAyM7yKpYCCgKCABAQshM7EAQEgbxAQMguL9QshRQEBAEhO7EBQUAQyAsEhOzyQs1SSEFAEBCyExsQBASBvEBAyC4v1CyFFAQEASE7sQFBQBDICwTa7A0KK6H1448/ttkrKlbCpS3K0qCW8jf7C+ptgUM644D9iuyNgJBdAiYDBgygmTNn6i0hSE7bAAH8EfHOZt++fdsgNokiFQKwY5F4BGzqxefo55bir8uZIJAWBK688kod71133ZWW+CVSQSAVAtJnlwoZuS4ICAKWQkDIzlLqlMIIAoJAKgSE7FIhI9cFAUHAUggI2VlKnVIYQUAQSIWAkF0qZOS6ICAIWAoBITtLqVMKIwgIAqkQELJLhYxcFwQEAUshIGRnKXVKYQQBQSAVAjKpOBUycr3NELjmmmtowYIFOr5gMKj3+MwfPghz7LHH0qWXXtpmaUlEgkAqBMSzS4WMXG8zBPr3708+n4/8fj+Fw2G94RjXjjjiiDZLRyISBJpCQDy7ptCRsDZD4KijjoqLC28pwrt7//33467LiSCQLgTEs0sXshJvHALdu3ePfYCaA8SrYyRknwkEhOwygbKkQeeff37cslnor7vooosEGUEgYwgI2WUM6vxOaMqUKZrseJGd0tJS6ty5c36DIqXPKAKynl1G4W59YkwSrY8h+0+OGjWKvvnmG52RqVOn7tWszX4O9y0H8E5FzIOADFDkoK4Sia258xwsQtIsLV68mLCeHUjCzAMTRpIzHqPQiedJgZCLWUFAPLuswJ48USOp4di44Qk+T/507l8dPHgwFRYWUnl5OfF8u9zP9d45ZELDPtkxP8FhfC777CIgZJdd/GOpM9Fhj7lovDceM9nxvXjYeByLLIcPJk2aRAMHDiR8i8JMYiQuJjnsscQ8Nj427lk3uCaSfQSkGZtlHfAfAntsILdQKBTbwoEaimx7h2j3PLL5N5MtsFXdF30LIctZb1XyW6tt1LHc3F8CsNldFCnoTBG3mk5TcTjZ2k0mh8MRtzEBMtHxvlWgyUNtgoCQXZvA2LpIjETHJIfmnd4aqsi+6kZy7XixdZHLUxlDIOyooGDl6RTufhk5i3vqydKYMA0CZE8PmRHCy5hKkiYkZJcUlvRfZKIDyTHRBQIBCvhqiTY8QAUb7yR7WB2LmAYB9U4IBdqfQtT7anKWDoqRHjd1URAhvOypU/rssoB9ItHBk8O7on5vNTm+v4TcNW9lIVeS5P4iYKOg8sSfJ2/ETg0D7iK3u1BHCS8PAtKD7oXwNBwZ/xGyyzjk0QRh9PDomOi8DbvJtfzn5Kr7Iks5kmTbCoHCnc9S/apy8va5KRYlr/IiRBeDJOMH8gZFhiEHyTHRYSACTVev10uOFVcI0WVYF+lMrmjb/RTa/KZe2QU6hq5RubH+05m2xJ0cASG75Lik9SqTHbw6LHMU3PEleapfSWuaEnnmESje8Afy1W3VXRRGsst8TiRFICBkl0E74Fode/bqQHaeDTdlMBeSVKYQcIZ3knPjXXHendEGMpUPSSeKgJBdhi3B6NVhUCJYvYTcvugqvhnOiiSXAQSKq58nX0NNnHeXgWQliSQICNklASWdl9ir44EJZ/W76UxO4s4yAo5ILUWq5+i+Weic++2ynK28TF7ILkNqNzZfjKOwzrq5GcqBJJMtBFy7P4nz7NgWspWffE1XyC7Dmjc2YzFK5wxWZTgHklymEXAENsZ5dqjsILAFkcwhIGSXOaxj0w5g7BigQLPGFdmUwRxIUtlAwB7apnUtI7LZQH9PmkJ2e7DIyBF7dkx4UrlnBPasJmKPeGPz7MSry54qhOwyiD331TDhoaYXsT4CrG8QHduA9UudeyUUssuCTtjgsRfJDwSMRCd6z47OhewyjDsbOhNehpOX5LKEQKK+2Q6ylJ28TFbILgtqZ8MXg88C+JJk3iIgZJe3qpeCZxoBqdwyjXh8ekJ28XjImSAgCFgUASE7iypWiiUICALxCAjZxeMhZ4KAIGBRBITsLKpYKZYgIAjEIyBkF4+HnAkCgoBFERCys6hipViCgCAQj4CQXTweciYICAIWRUDIzqKKlWIJAoJAPAJCdvF4yJkgIAhYFAEhO4sqVoolCAgC8QgI2cXjIWeCgCBgUQSE7CyqWCmWICAIxCMgZBePh5wJAoKARRFwWrRceV8srAu6cVcRdSz1kssZ/cALQPl2dXt6e34P+unQjTS2/7a04bRsYzl1rainiqKATqOmwUlPfdGffthUqvITodF9t9Hph65JS/qJaaclEYnUdAgI2ZlOZU1n2Bew082vDqelG8opHIk67n077qbrTlhIncp99L0im69WdqCelXVpI7uVW0roj8+Npu7t6ujf589RS5ET/e6x8VTTUBDLfNWuwjYhu+21blq6vpx6d6ylXpX1lJh2LEE5yHsEhOwsZAIglWueH0U/binTXtXQnrto7qpKWrW1lK55YTQ9cvGsjJS2e7t6Gj9ga4xMv6sq00RXUeSjf503h2w2ou21rjbJy5wfO9D9Hw2inxy8kS776XJFsPFpt0kiEoklEBCys4Qao4X4RhEbiK7M46f7z5+tSQUEeOo9R9C23YXKqytLWtqPlnQhPOtWzV2QxpAe1bH74Dm9v6gr/bi5jMYP3EITD9hC7oJos7jB76AZyztrT/Gg7rvoqIM2UUWxn3xBB5UUBsirvMyFa9vREzP76/gcytF8elY/OmPcKnp9bi/q33k3HTt8gw6r8znphdm9ac32Ejqk73aaOmQjFbqi6fiDdnr1m560XDWN+3Ss03ns1q6BZv3QkT5QeYMsWNueXpjTm44ZtjGWtg5QP5uUF/nx0q7q+TLq1aFeeZSrFUbR5vVLX/WiLTUemnzgJnrr2x5U5ArSyWPWEuIXsRYCQnYW0ieIBTKs505NdDiGF/XH6UsUCZZSYUEQl+Lkb28MpS9XdFT3RVRz06ZJ4ZzDfqTTVH/a2u1FdOnj4/T9TntYkxq8qMd+/YUixhBd8OBhBJIqcIToa9U0fvzzAXTr6fOoTBHdB4u7KWKqpcMHbaIfNpfqOHbUuQjEOm3YBh2+YnOU7NCfd9FDExQ5Rs1x3upKeujTAfT07z5XeQ7ThSqsut5FdluY5q7uQC9/3Zvu/eUcWrCmHa1UXitk6243fba8C41T/ZCc9nEjNtCGnUV02eNjKRiONunnr62k/8zvTvdfMJs6lXlV/2VP5WW66d2F3XU8+Hl/cXd6/tLPyOOSr7/FQLHAgYzGWkCJXAR4dZAx/bbzJb3HQMRZ41fpPi1jQDhs00SEgYQXL/uMHrrwCx38+rxeev+M8sIgv5y4gl75r09pRK/tqh+QaPH6CkUsnTXRDVEe3ctXfEbnTlqhSe+L7zvpZ/jn1LFr6ZKpy/Xp5AM30wuXzeCg2P6RzwZqohvVZ7smmVPGrNH9ja8oUsOABrxTNItfu/JT+l1jXPAQfzv1e/r1kd/reKYOqaL7zp0Ti5MP/v7mwZroThi1lp69ZAZNUN4piO/u9w7kW/T+osk/0CtXfEIH99ilzz//Lr4ccTfLiSkREM/OlGpLnulQ46BrWHloLRG7PUL/d8GXugn44CcDac22Eu3h7fZGzWJoj52qqdiJQHpoQh4zfCPdfMoC7S1u2OHRSSzZUEF/eHY0HTF4Ez39W+WJqabn2m3FLUk+dg/ihpw2drX2pkDMIOCuFQ00uFuNbpLDc7xNEdd65alB1imvsyXC9/1y4ko9Kg1SQ5lAokY5SjWbnY6Iqii2aTJHPydRlfEWOTY5AkJ2JlegMfsDVB/Yso0V9O2a9nSU8nRY5q5qrwYpSlSf295TTS5/YqwikGI1RcRH/TrV8iN6f9zIDRQI2Qme3lcrO+qtssRH9yoPqnv7Brr55Pn0yIwBtLyqXG8PKw/ttjPn6r6/uIiaOfEGHPoOkBsEU2V+OjSaf4zgnvfAYdobAwFWlvj1PS35CQRt2kNEk5un31QURZ8PqnIlE4eqAESsiUByjVuzrJYv1fBeO3UZF66tiJUVTdVbVb/cEzMHUG2jx8aBmI/GRPfEb76gm05eQOibY3n5615UtcujPKsv6aGLZlE3RTbo30IT9ssfOui+vqumLaXnLvmMDh+8WRPSM7P68uMt3qPvDIJBEsiWmkK68aXhqm+uF72sBhDQ7Jw0aDM9oPrZfnNUtEmsbzT8oKmbKAVqPh/6EzFgsqXarYMXrYv2a3aVAYhEuCx/Lp6dhVSMUczeHWp1c/Q3j4yjQaoJ+O3qdso7c1AHNbl4UNcamq869Vl6VdbqZmt1g0tN3zhAeYXl+l6EY77eMtVExZy81dtKdV9XrRqMgGCi8o7GTv3ZKzrQSYes0yOeCKssbbnnhfshJ4xeR8vfKtdTSJaqNDGyClLF1JlhvXbRq3N7EwYt0NTGIASk3h/NSx9VXshsNcjSu0Mdjey9Q5/zz08OrqL/LOhBlz1xKB06YBt98X1HHYRBEpH8QkA8OwvpG31wt6tmJDrZN1UX0idLu9Cuejcd0KWa/t9Zc+NKilHaYneIMPLqcoQ1IWxU/WE8YovR2yunLdHPggTRRA2oKSAnjFqnB0COVlM8pg1bTyDKR1VTdoUacR3acwf9akp0wACJ2Sjqbtm5CzGZ+6Xuw3QW9NeFGkeDMWo75cAqPSI8WhH4of23KnJz0Jvf9ozlb6canYUc1L1aT16u9RXQc7P3eJWc9oVH/KCJukGN9H66LEqUJ41eQ9NHrtfP8w9nUY+G8EXZWwoBm/pwb5IGgKXKmBOFAcyhUIj8fj/V19dTdXU17dixgw7YciQ5Kb6vrC0yDK1uVoTXscxHLemHQjMPb1gkk2DIpkkN/XXJpKlnk92f6hryjCZsx1IfgbiNgv63OkVomMeXTHaoScqYI8dz8xLvCanmPLxFbjInhqfzvNY2iDZ0eYbatWtH5eXl5PF4qKCggBwOh/KsYzSbzixI3AoBacZa1AzwH+pSEe0La0kRUxEdnsUoZSqiQ3hTzyK8pYI8dy5Pnmf0v1U4kxMd4m/fzMAFCD8bRNfSsst96UdAmrHpx1hSEAQEgRxAQMguB5QgWRAEBIH0IyBkl36MJQVBQBDIAQSE7HJACZIFQUAQSD8CQnbpx1hSEAQEgRxAQMguB5QgWRAEBIH0IyBkl36MJQVBQBDIAQSE7HJACZIFQUAQSD8CQnbpx1hSEAQEgRxAQMguB5QgWRAEBIH0IyBkl36MJQVBQBDIAQSE7HJACZIFQUAQSD8CQnbpx1hSEAQEgRxAQMguB5QgWRAEBIH0IyBkl36MJQVBQBDIAQSE7HJACZIFQUAQSD8CQnbpx1hSEAQEgRxAQMguB5QgWcgPBGQJ9uzqWcguC/jD6HkL2fbtg9JZyK4kuZ8IBG2l8q2J/cSwLR4XsmsLFPchDq7dmey81HkfnpZbzYhAwNY+Vrmx3tkOzFges+ZZyC4LmmODt9vt1GDrlYUcSJKZRMBr7xtHdplMW9Lag4CQ3R4s0n5kJDkQHT6lV10wMe3pSgLZRMBGNa7Dta7504ni1WVHH0J2GcadCQ+Gj63OPYF8kYoM50KSyxQC1ZGBFHT1IafTqb6Fa9cb20Cm8iDpRBEQssugJbCRg+Rg/PhQssvlonW20zKYC0kqkwhsdJyp9Qx9Q+8gPPHsMqmBPWkJ2e3BIq1HbODYw+CNZLej8GfUEK5Ma/oSeeYRqA73p1rPFF2hoWJj744rPbaJzOcsP1MUssug3pnomOzg1RUWFpLbU0FLHDdTKOLKYG4kqXQi4I20p+XOm7R+oWPo2kh26Uxb4k6OgJBdclzSdpUJD00a/AHcbjd5PB4Kew6iRaGrhPDShnzmIvaGy2lB5CZyFPWkoqIirWPoWpqxmdNBspSE7JKhkuZriZ4d/hAlJSXUUHwkzQn8lepCHdKcA4k+XQhsD/SnOaH/JSoeonWKigyeXWIzNl3pS7ypEbBFlKQOlpC2RgBwYwuFQuT3+8nn81FdXR3V1tZSTU2N3hp2b6Hekaeon/t9stvCbZ0FiS8NCPjDhfS972Ta7PwZFZe2p7KyMr2hEmPvDoQH74777NKQDYmyCQSE7JoAJx1BXLeEw2EKBoMUCATI6/VSfX29JjyQHjYQoN27Ur1f8TF1c31N5c6qdGRH4txPBLb5+9Gm4FjabJtCDk93Ki4u1h4dSA6b0bPjZiySlMGJ/QS+FY8L2bUCtP19hL07I+HBw2toaNAkB+LjDdexUaiWXJFtVBDZpTzDkM4CE+f+5ieTz89ZsJHqGwI0ZVzvTCbbJmmxRxYhJ/mpHfns3fSgA/pd0VQFsYHs4MlhwznC2KPjaSdCdG2ijn2OxLnPT8gDbYIADJ777jhCvoY/B4/UgujQ3A0EylTTt7Nu/jJZ8nNm2kfcX1M4WEeh0glmynYsr6wjt5o+VGyYK8kDTUx6OIcOhehi0GX9QMguCyrAHwaExX8cTEngYzR1jGQXJbqAbvKinw/eIDYzenWAGh4PylFZab55hdARBJUUV1Q8X5LJjffQIU81EY9Ow5b1HyG7LKkgkfBwnkh4+OOgX483I9EZyc54nKXitDhZNO3QT9m+ffsWP5PtG5nkkI9EPXHlxKSHPTbun+P7jXFkuzz5mr6QXRY1jz8Ae3jIBv9BsOc/j9GbM5KdMdtmIzs0zSsqzPM+sJGojOTFHh70xRtfY28OejI+b9SbHGcWASG7zOK9V2rGPwKOmfzwZ8EfCOfcbMUxExvv94owxy9wPxZGKs0orC/ssTGp8Z6vG+8zYzmtmGchuxzRKv85kB0cg8zwB0pFcGYlO27ioYluRknUE59jz8esQzOWz8p5FrLLMe0a/zCJfxqzEpwRYhA4NpCemSVRT4m6MnPZrJp3c1ucVbWiytWSP5NZyQ9lS1a+XFenGfOc65hmMn9CdplEu43TMuOfj/PM+zaGRKITBFIiIAsBpIRGAgQBQcBKCAjZWUmbUhZBQBBIiYCQXUpoJEAQEASshICQnZW0KWURBASBlAgI2aWERgIEAUHASggI2VlJm1IWQUAQSImAkF1KaCRAEBAErISAkJ2VtCllEQQEgZQIyErFKaGRgLZCYP78+bRw4UId3aJFi/SKzGPHjtXn/fr1o4kTJ7ZVUhKPIJASAXmDIiU0EtCWCDz++ONx0S1btky/Mvb73/8+7rqcCALpQkCaselCVuKNITBixAj9pa3YhcYDrOZ79NFHJ16Wc0EgLQgI2aUFVok0EYHJkyfr1U74OlY+mTDBnN+h4DLI3lwICNmZS1+mze0ZZ5wRR3bw6o4//njTlkcybj4EhOzMpzNT5rhLly7UsWPHvpeOMgAABq5JREFU2ErL+BYFmrcigkCmEBCyyxTSkg6ddNJJ+vuqaMJOVs1aEUEgkwgI2WUS7TxPC4MR+NgOyO60007LczSk+JlGQMgu04jncXr4yM6oUaN0cxbNWhFBIJMIyDy7TKItadHUqVOptrZWkBAEMo6AvEGRccglQZCdWT+lKNozLwJCdubVneRcEBAE9gEB6bPbB7DkVkFAEDAvAkJ25tWd5FwQEAT2AQEhu30AS24VBAQB8yIgZGde3bUq5/iwdiAQiG2hUKhV8RgfWr16NV1//fX01ltvGS+3+Pjhhx/Wz3u93qTPrF+/ntasWaPDvvnmG33vl19+qc+NYUkflouCQCMCMvUkz0zhjTfeoHvuuSeu1E6nU89/u/baa5OuThJ3c5KTjRs3EsgHRDp9+vQkdzR96aOPPqLNmzcTyK6wsHCvmy+88EICKb///vu0fPlynVbv3r1p/PjxZAwLh8P0+eefU/v27Wn48OF7xSMX8hsBIbs80z8ICVJeXq4Jrq6ujubNm0dfffUVXX755fTYY4/p8Fz6wdsWIEK8eZEoxrDq6mr6y1/+Qp07d6Znnnkm8VY5z3MEhOzy1ACGDh2qm4Mofn19PZ1wwgm0bt062rZtG3388ce0du1aGjNmDL399tv0q1/9igYMGEBLliyhDz74QHth8JxOP/30OAICId133326yTls2DA6++yz9QKdSMPv99MLL7yg48DqxMceeyx1794dQTGZM2eO9sywSADiHjhwoA6DV2ez2WL3GQ84DE3pJ598Ugft2LGD7rjjDjr88MNpxowZNG7cuNhqyM8++yxt2LCBzj33XP0mhzEuObY2AkJ21tZvi0pXVFREbrdbe08gu9dee00T2jvvvKOfBzmsWrWK/v73v8figyeIPjomGARg+XVskLlz5+omJzwtENJZZ51Fu3bt0uSIZ5977jlCX12fPn30/fi5/fbbY8effPIJYXVjEOKrr75KwWCQLrnkklg4H3AYVlABsUHQJwlSxnLvKAM8VxzjvVykCbniiiv0Xn7yB4G92wX5U/a8LilIDcQ0c+ZMuuGGGzTRtWvXjgYPHhzDBX1i//znP2n06NF022236eu33HKLbiKiqVhVVUUvvfRS7H68FQHyufvuuzWpff311zrsu+++0/15IByQ0JVXXqmvP/roo7FncXDRRRdpTxL3obn94YcfxoU3dXLYYYdpzxH3IG/vvfee9uhA4ugP3Llzp/YaES+8UqynJ5JfCAjZ5Ze+Y6VFR/9VV11Ff/rTn2jWrFmEQYo//OEPsXAcXH311XTQQQfpZh9IAiSC1YWxRzMTAi+N5eCDD9YDHNiDNOGNLV26VMfxxBNPaO/q5ptv1p4jnuERVn5+2rRpeoDi/PPP15dAxPsraMpC3n333Rh5nnjiifsbrTxvQgSkGWtCpbVFlnv27Kn76RwOh24qgqCSjYQirU2bNukkQXIsXbt21Yfo70smuBdEh760Hj166CWdQH7dunWjDh06JHskdo1XRIH3ub9yyimnaG8SI76YpoJBDniOIvmHgJBd/ulclxhTN04++eQWlb5///76vu+//z52PwYTIEOGDIlda2hoiB0vXrxYH8MTxKAAiG7KlCl6UAQe3QUXXBC7lw9AnBUVFYS5dBAMcuyvYJADI8/oc4RgsCLZqO7+piPP5z4CQna5r6Os5xCeGQYKMFBxzjnn6EEFzKvDCOkxxxwT8/wWLFhAaKbCE9y6dStVVlZq8ho5cqTuT0OTF6O13BeX6BVeeumldOihhxIGJyCTJ0/W+5b+lJWVaSLbsmWLTufiiy8ml8ull5V6+eWXdTTShG0pmta7T/rsrKfTJkvEUzh439TNRg/ozjvvJDR9MYEYfXwYwb3pppsI00g4LnxjApN64QGiDxB9fhB8EBseHsjtlVdeIUwtgWCKiFGKi4v1xGGMpsKrmzRpUiyY0+ALxnM+RpMcqyGjfxHpYA4hBMvBQzAowR/n1hfkJ68QkCWe8krd+19YNFUxnw4jt8kEbzHAqzP27/F9IDGsZZfqWdyHUVOMoIJMWysgOUx3gacHmT17Nl133XV0xBFH0I033tjaaOU5kyMgZGdyBUr2m0YAr8ChDxDeHt4OQZNcJD8RkD67/NR73pQab31g/h/eoRWiyxu1Jy2oeHZJYZGLgoAgYDUEZIDCahqV8ggCgkBSBITsksIiFwUBQcBqCAjZWU2jUh5BQBBIioCQXVJY5KIgIAhYDQEhO6tpVMojCAgCSREQsksKi1wUBAQBqyHw/wGi9ZZel5bVrQAAAABJRU5ErkJggg=="}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding = tf.keras.layers.Embedding(\n    len(tokenizer.word_index) + 1,\n    EMBED_SIZE,\n    embeddings_initializer = tf.keras.initializers.Constant(embedding_matrix),\n    trainable = False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# question encoding\nquestion_input = Input(shape=(None,))\nquestion_x = embedding(question_input)\nquestion_x = SpatialDropout1D(0.2)(question_x)\nquestion_x = Bidirectional(LSTM(100, return_sequences=True))(question_x)\nquestion_x = GlobalMaxPooling1D()(question_x)\n\n# answer encoding\nanswer_input = Input(shape=(None,))\nanswer_x = embedding(answer_input)\nanswer_x = SpatialDropout1D(0.2)(answer_x)\nanswer_x = Bidirectional(LSTM(150, return_sequences=True))(answer_x)\nanswer_x = GlobalMaxPooling1D()(answer_x)\n\n# classification\ncombined_x = concatenate([question_x, answer_x])\ncombined_x = Dense(300, activation='relu')(combined_x)\ncombined_x = Dropout(0.5)(combined_x)\ncombined_x = Dense(300, activation='relu')(combined_x)\ncombined_x = Dropout(0.5)(combined_x)\noutput = Dense(1, activation='sigmoid')(combined_x)\n\n# combine model parts into one\nmodel = tf.keras.models.Model(inputs=[answer_input, question_input], outputs=output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While this model has quite a few moing parts, at its heart it is just a binary classification model. This means I can make use of the easy and well tested loss function that is binary cross-entropy. I'll also use adam to optimise the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    loss='binary_crossentropy', \n    optimizer='adam',\n    metrics=['BinaryAccuracy', 'Recall', 'Precision']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll also define a few callbacks. The first helps the model to avoid plateauing as it approaches zero loss. I've also added early stopping in case the model reaches the optimum before it has finished its final epoch.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    tf.keras.callbacks.ReduceLROnPlateau(monitor='loss', patience=2, verbose=1),\n    tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5, verbose=1),\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally the model can begin training. One extra hyper-parameter to mention here is the class weights. This dataset contains far more wrong answers (label 0) than correct answers (label 1). While I want the model to know when it sees a wrong answer, I'd like it to focus more on the correct answers that it sees. As such I have weighted the correct answers far more heavily than the wrong answers.\n\nOn GPU 40 epochs should take around 8 minutes. On CPU it will take about an hour and a half.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(\n    x = [train_long_answers, train_questions], \n    y = train_labels,\n    validation_data = (\n        [val_long_answers, val_questions], \n        val_labels\n    ),\n    epochs = EPOCHS,\n    callbacks = callbacks,\n    class_weight = CLASS_WEIGHTS,\n    batch_size = BATCH_SIZE,\n    shuffle = True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate\n\nTo evaluate the model there are a lot of the usual quantitative methods. A good start is the loss curve across both the training and validation datasets. If they are both trending down the model is learning something.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15, 5))\n\nax[0].set_title('Training Loss')\nax[0].plot(history.history['loss'])\n\nax[1].set_title('Validation Loss')\nax[1].plot(history.history['val_loss'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The same can be done with the other metrics, accuracy, recall and precision. This time I'm hoping the curves trend upwards.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize=(15, 10))\n\nax[0,0].set_title('Training Accuracy')\nax[0,0].plot(history.history['binary_accuracy'])\n\nax[0,1].set_title('Validation Accuracy')\nax[0,1].plot(history.history['val_binary_accuracy'])\n\nax[1,0].set_title('Training Recall')\nax[1,0].plot(history.history['recall'])\n\nax[1,1].set_title('Validation Recall')\nax[1,1].plot(history.history['val_recall'])\n\nax[2,0].set_title('Training Precision')\nax[2,0].plot(history.history['precision'])\n\nax[2,1].set_title('Validation Precision')\nax[2,1].plot(history.history['val_precision'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll also print out the number of epochs the model trained for to see if early stopping took place.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Epochs: {0}'.format(\n    len(history.history['loss'])\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The final metric I'll use is F1 score. This is a combination of precision and recall and the metric the competition (now closed) used to measure the quality of models. It's thus a good single metric that encompasses the quality of the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"recall = history.history['recall'][-1]\nprecision = history.history['precision'][-1]\n\nprint('Train F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"recall = history.history['val_recall'][-1]\nprecision = history.history['val_precision'][-1]\n\nprint('Validation F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Quantitative methods are a great way to see the whole models quality in one number. However I always find it invaluable to look at some of the results the model produces. This helps me validate the metrics and see exactly what users would see if this model were to be deployed to an application. To this end I have defined a function that takes a question, the right answer (positive) and a wrong answer (negative) and returns the probability for each of the two answers. I'm expecting to see a high probability for the right answer and low probability for the wrong answer. The greater the difference between the numbers, the better.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_question(question, positive, negative):\n    sentences = [question, positive, negative]\n    \n    for i in range(3):\n        sentences[i] = remove_stopwords(sentences[i])\n        sentences[i] = remove_html(sentences[i])\n    \n    sentences = encode(sentences, tokenizer)\n    \n    predictions = model.predict(\n        [np.expand_dims(sentences[1], axis=0), np.expand_dims(sentences[0], axis=0)]\n    )\n\n    print('Positive: {0:.2f}'.format(predictions[0][0]))\n\n    predictions = model.predict(\n        [np.expand_dims(sentences[2], axis=0), np.expand_dims(sentences[0], axis=0)]\n    )\n\n    print('Negative: {0:.2f}'.format(predictions[0][0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's a sample question with two answers from the training dataset. Let's see how the model does with them.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'which is the most common use of opt-in e-mail marketing'\n\npositive = \"<P> A common example of permission marketing is a newsletter sent to an advertising firm 's customers . Such newsletters inform customers of upcoming events or promotions , or new products . In this type of advertising , a company that wants to send a newsletter to their customers may ask them at the point of purchase if they would like to receive the newsletter . </P>\"\n\nnegative = '<P> Email marketing has evolved rapidly alongside the technological growth of the 21st century . Prior to this growth , when emails were novelties to the majority of customers , email marketing was not as effective . In 1978 , Gary Thuerk of Digital Equipment Corporation ( DEC ) sent out the first mass email to approximately 400 potential clients via the Advanced Research Projects Agency Network ( ARPANET ) . This email resulted in $13 million worth of sales in DEC products , and highlighted the potential of marketing through mass emails . However , as email marketing developed as an effective means of direct communication , users began blocking out content from emails with filters and blocking programs . In order to effectively communicate a message through email , marketers had to develop a way of pushing content through to the end user , without being cut out by automatic filters and spam removing software . This resulted in the birth of triggered marketing emails , which are sent to specific users based on their tracked online browsing patterns . </P>'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_question(question, positive, negative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'how i.met your mother who is the mother'\n\npositive = \"<P> Tracy McConnell , better known as `` The Mother '' , is the title character from the CBS television sitcom How I Met Your Mother . The show , narrated by Future Ted , tells the story of how Ted Mosby met The Mother . Tracy McConnell appears in 8 episodes from `` Lucky Penny '' to `` The Time Travelers '' as an unseen character ; she was first seen fully in `` Something New '' and was promoted to a main character in season 9 . The Mother is played by Cristin Milioti . </P>\"\n\nnegative = \"<P> In `` Bass Player Wanted '' , the Mother picks up a hitchhiking Marshall , carrying his son Marvin , on her way to Farhampton Inn . On their way , it is revealed that the Mother is a bass player in the band , that is scheduled to play at the wedding reception . But the band 's leader , Darren , forced her to quit . The Mother ultimately decides to confront Darren and retake the band . She ends up alone at the bar , and while practicing a speech to give Darren , Darren walks up to her furious the groom 's best man punched him for `` no reason . '' Amused by this , the Mother laughs , and Darren quits the band in anger . </P>\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_question(question, positive, negative)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I've always been impressed when I ask Google the same question but slightly differently and it still understands it as the same question. Let's see if this model can do this.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'how i met your mother who is the mother'\ntest_question(question, positive, negative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'who is tracy mcconnell'\ntest_question(question, positive, negative)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not bad. Even when the question is phrased in a very different way i.e. ask who Tracy is rather than who the mother is, the model still knows I am asking about the mother in \"How I met your mother\" and returns the right answer.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Text extraction\n\nWith the correct long answer matched to a question, the next challenge is to extract the short answer from a long answer. So if the correct answer is a paragraph about who the mother in \"How I met your mother\" is:\n\n> \"Tracy McConnell, better known as 'The Mother', is the title character from the CBS television sitcom How I Met Your Mother. The show, narrated by Future Ted, tells the story of how Ted Mosby met The Mother. Tracy McConnell appears in 8 episodes from 'Lucky Penny' to 'The Time Travelers' as an unseen character; she was first seen fully in 'Something New' and was promoted to a main character in season 9. The Mother is played by Cristin Milioti.\"\n\nThe correct short answer would be the mothers name:\n\n> \"Tracy McConnell\"\n\nThe approach to this challenge isn't too different from the question, answer matching model above. The pre-processing is much the same. So first I'll load the dataset filtering out the wrong long answers and extracting the start and end index of the short answer within the long answer. The start and end index are the indexes of the words in the long answer that make up the short answer. So if we use the same example answer above, the start and end index would 0 and 1 as the short answer begins on the first word of the long answer and ends of the second word.\n\nI've hidden the code for transforming the json file into a pandas dataset again.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_short_answer(annotations, long_start, long_end):\n    if len(annotations['short_answers']) > 0:\n        short_start = annotations['short_answers'][0]['start_token']\n        short_end = annotations['short_answers'][0]['end_token']\n        \n        short_start = short_start - long_start\n        short_end = short_end - long_start\n        \n        return short_start, short_end\n    else:\n        return 0, 0\n    \n\ndef form_short_data_row(question, text, long_start, long_end, short_start, short_end):\n    long_answer = ' '.join(text[long_start:long_end])\n    short_answer = ' '.join(long_answer.split(' ')[short_start:short_end])\n    \n    row = {\n        'question': question,\n        'long_answer': long_answer,\n        'short_answer': short_answer,\n        'short_start': short_start,\n        'short_end': short_end\n    }\n    \n    return row\n\n\ndef load_short_data(file_path, questions_start, questions_end):\n    rows = []\n    \n    with open(file_path) as file:\n\n        for i in tqdm(range(questions_start, questions_end)):\n            line = get_line_of_data(file)\n            question, text, annotations = get_question_and_document(line)\n\n            for i, candidate in enumerate(line['long_answer_candidates']):\n                label, long_start, long_end = get_long_candidate(i, annotations, candidate)\n\n                if label == True:\n                    short_start, short_end = get_short_answer(annotations, long_start, long_end)\n                    \n                    rows.append(\n                        form_short_data_row(question, text, long_start, long_end, short_start, short_end)\n                    )\n        \n    return pd.DataFrame(rows)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_short_df = load_short_data(TRAIN_PATH, 0, NUM_OF_TRAIN_QUESTIONS)\nval_short_df = load_short_data(TRAIN_PATH, NUM_OF_TRAIN_QUESTIONS, NUM_OF_VAL_QUESTIONS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_short_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Usually I would clean the text here but it can get a little confusing when trying to extract text if sentences have parts removed from them. As such they have been maintained. However I have kept the tokeniser used in the first challenge of this notebook which I use here to encode the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_long_answers = encode(train_short_df['long_answer'].values, tokenizer)\ntrain_questions = encode(train_short_df['question'].values, tokenizer)\n\nval_long_answers = encode(val_short_df['long_answer'].values, tokenizer)\nval_questions = encode(val_short_df['question'].values, tokenizer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The labels look a little different this time. The model will be outputting two arrays (for for the start index and another for the end index) 300 elements in length (to represent the 300 words in a padded sentence). All but one of the elements in the arrays will be a zero value. The one value of the arrays that is a value of one represents the  models predicted start or end of a short answer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def form_short_labels(df, sentence_length):\n    start_labels = np.zeros((len(df), sentence_length))\n    end_labels = np.zeros((len(df), sentence_length))\n\n    for i in range(len(df)):\n        start = df.loc[i].short_start\n        end = df.loc[i].short_end\n\n        if start < 300 and end < 300:\n            start_labels[i, start] = 1\n            end_labels[i, end] = 1\n        else:\n            continue\n    \n    return start_labels, end_labels\n\n\ntrain_start_labels, train_end_labels = form_short_labels(train_short_df, MAX_LEN)\nval_start_labels, val_end_labels = form_short_labels(val_short_df, MAX_LEN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see an example. Here the correct short answer is \"a newsletter sent to an advertising firm's customers\". If you are counting the words in the sentence to see if they match the label, be sure to include the html tag in the count.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_short_df.loc[0].long_answer)\nprint('Start index: {0}'.format(train_start_labels[0]))\nprint('End index: {0}'.format(train_end_labels[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the data ready to go I'll begin defining the model. The below code imports the glove embeddings again and the code is much the same as last time I used it in this notebook. However this model requires more dimensions (200) to be effective meaning I couldn't just re-use the embeddings from the question, answer model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# load from file\nembedding_dict = {}\n\nwith open('../input/glove-global-vectors-for-word-representation/glove.6B.' + str(SHORT_EMBED_SIZE) + 'd.txt','r') as f:\n    for line in f:\n        values = line.split()\n        word = values[0]\n        vectors = np.asarray(values[1:],'float32')\n        embedding_dict[word] = vectors\n        \nf.close()\n\n# write to matrix\nnum_words = len(tokenizer.word_index) + 1\nembedding_matrix = np.zeros((num_words, SHORT_EMBED_SIZE))\n\nfor word, i in tokenizer.word_index.items():\n    if i > num_words:\n        continue\n    \n    emb_vec = embedding_dict.get(word)\n    \n    if emb_vec is not None:\n        embedding_matrix[i] = emb_vec\n        \n# load as tensorflow embedding\nembedding = tf.keras.layers.Embedding(\n    len(tokenizer.word_index) + 1,\n    SHORT_EMBED_SIZE,\n    embeddings_initializer = tf.keras.initializers.Constant(embedding_matrix),\n    trainable = False\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model itself looks similar to last time. There are three parts where the first two encode the question and the answer. They feed into the classification part of the model that then predicts the start and the end indexes of the short answer. The two encoding parts are much the same as the encoding parts in the last model. Only the classification part has changed which uses convolutional layers instead of dense ones and outputs two arrays instead of one.\n\n![Screenshot%202020-07-15%20at%2009.42.19.png](attachment:Screenshot%202020-07-15%20at%2009.42.19.png)","attachments":{"Screenshot%202020-07-15%20at%2009.42.19.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAUUAAAEyCAYAAABtU8IkAAAK1mlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk8kWx+f70kNCC4QiJfSOdAJICT0UQTqISkhCEkoMCUFFbMjiCq4FFREsK7oiouDqCshaEFEsLIK9bxARUJ6LBRsq7ws8wu6+8947754zmd+5mbllMnPOPwCQQ1kiUTasCkCOME8cE+JPS0pOoeGeASwgAhgYA3MWWyJiREdHAMSm57/a+zsAks837eSx/v37/2rqHK6EDQCUinA6R8LOQbgNGS/YInEeAKgjiN9kSZ5Izj0Ia4iRAhF+JmfeFH+Uc/oko0mTa+JiAhCmAYAnsVhiHgAkW8RPy2fzkDgkeQ8OQo5AiHAhwj5sPouD8GmEbXNyFst5CGFLZL0IADJyOoCe/qeYvL/ET1fEZ7F4Cp7qa9LwgQKJKJu17P88mv9tOdnS6RzmyCDxxaEx8nzI+d3LWhyuYGH63KhpFnCmapIzXxoaP81sSUDKNHNYgeGKvdlzI6Y5QxDMVMTJY8ZNM1cSFDvN4sUxilwZ4gDGNLPEk3mJCMukWfEKP5/LVMQv4MclTnO+IGHuNEuyYsNn1gQo/GJpjKJ+rjDEfyZvsKL3HMmf+hUwFXvz+HGhit5ZM/VzhYyZmJIkRW0cbmDQzJp4xXpRnr8ilyg7WrGemx2i8EvyYxV785DLObM3WnGGmayw6GkGAhAJWIBNU5kmAPK4S/PkjQQsFi0TC3j8PBoDeW1cGlPItrelOTk4OQAgf7tT1+EtdfJNQtSrM77cNgA8ShEnb8bHMgHg1HMAKO9nfCZvkKu0GYAzPWypOH/Kh5Z/YJBfTwVoAB1gAEyAJbADTsANeAE/EATCQBSIA8lgIVIrH+QAMVgCCsEaUALKwGawHVSBvWA/OASOguOgGZwG58ElcA30gNvgIZCBAfASjIL3YByCIBxEhiiQDmQImUE2kBNEh3ygICgCioGSoTSIBwkhKVQIrYXKoHKoCtoH1UE/Q6eg89AVqBe6D/VBw9Ab6DOMgkmwBqwPm8OzYTrMgMPhOHgBzINz4QK4GN4IV8I18BG4CT4PX4NvwzL4JTyGAiglFBVlhLJD0VEBqChUCioDJUatRJWiKlA1qAZUK6oTdRMlQ42gPqGxaAqahrZDe6FD0fFoNjoXvRK9AV2FPoRuQnegb6L70KPobxgyRg9jg/HEMDFJGB5mCaYEU4E5iDmJuYi5jRnAvMdisVSsBdYdG4pNxmZil2M3YHdjG7Ft2F5sP3YMh8Pp4Gxw3rgoHAuXhyvB7cQdwZ3D3cAN4D7ilfCGeCd8MD4FL8QX4Svwh/Fn8Tfwg/hxgirBjOBJiCJwCMsImwgHCK2E64QBwjhRjWhB9CbGETOJa4iVxAbiReIj4lslJSVjJQ+leUoCpdVKlUrHlC4r9Sl9IqmTrEkBpFSSlLSRVEtqI90nvSWTyeZkP3IKOY+8kVxHvkB+Qv6oTFG2V2Yqc5RXKVcrNynfUH6lQlAxU2GoLFQpUKlQOaFyXWVElaBqrhqgylJdqVqtekr1ruqYGkXNUS1KLUdtg9phtStqQ+o4dXP1IHWOerH6fvUL6v0UFMWEEkBhU9ZSDlAuUgY0sBoWGkyNTI0yjaMa3RqjmuqaLpoJmks1qzXPaMqoKKo5lUnNpm6iHqfeoX7W0tdiaHG11ms1aN3Q+qA9S9tPm6tdqt2ofVv7sw5NJ0gnS2eLTrPOY120rrXuPN0lunt0L+qOzNKY5TWLPat01vFZD/RgPWu9GL3levv1uvTG9A30Q/RF+jv1L+iPGFAN/AwyDbYZnDUYNqQY+hgKDLcZnjN8QdOkMWjZtEpaB23USM8o1EhqtM+o22jc2MI43rjIuNH4sQnRhG6SYbLNpN1k1NTQNNK00LTe9IEZwYxuxjfbYdZp9sHcwjzRfJ15s/mQhbYF06LAot7ikSXZ0tcy17LG8pYV1opulWW126rHGrZ2teZbV1tft4Ft3GwENrttem0xth62Qtsa27t2JDuGXb5dvV2fPdU+wr7Ivtn+1WzT2Smzt8zunP3NwdUh2+GAw0NHdccwxyLHVsc3TtZObKdqp1vOZOdg51XOLc6vXWxcuC57XO65UlwjXde5trt+dXN3E7s1uA27m7qnue9yv0vXoEfTN9Ave2A8/D1WeZz2+OTp5pnnedzzDy87ryyvw15DcyzmcOccmNPvbezN8t7nLfOh+aT5/Ogj8zXyZfnW+D71M/Hj+B30G2RYMTIZRxiv/B38xf4n/T8EeAasCGgLRAWGBJYGdgepB8UHVQU9CTYO5gXXB4+GuIYsD2kLxYSGh24JvcvUZ7KZdczRMPewFWEd4aTw2PCq8KcR1hHiiNZIODIscmvko7lmc4Vzm6NAFDNqa9TjaIvo3Ohf52HnRc+rnvc8xjGmMKYzlhK7KPZw7Ps4/7hNcQ/jLeOl8e0JKgmpCXUJHxIDE8sTZUmzk1YkXUvWTRYkt6TgUhJSDqaMzQ+av33+QKpraknqnQUWC5YuuLJQd2H2wjOLVBaxFp1Iw6Qlph1O+8KKYtWwxtKZ6bvSR9kB7B3slxw/zjbOMNebW84dzPDOKM8Y4nnztvKG+b78Cv6IIEBQJXidGZq5N/NDVlRWbdZEdmJ2Yw4+Jy3nlFBdmCXsWGyweOniXpGNqEQky/XM3Z47Kg4XH5RAkgWSljwNRCR1SS2l30n78n3yq/M/LklYcmKp2lLh0q5l1svWLxssCC74aTl6OXt5e6FR4ZrCvhWMFftWQivTV7avMllVvGpgdcjqQ2uIa7LW/FbkUFRe9G5t4trWYv3i1cX934V8V1+iXCIuubvOa93e79HfC77vXu+8fuf6b6Wc0qtlDmUVZV82sDdc/cHxh8ofJjZmbOze5LZpz2bsZuHmO1t8txwqVysvKO/fGrm1aRttW+m2d9sXbb9S4VKxdwdxh3SHrDKismWn6c7NO79U8atuV/tXN+7S27V+14fdnN039vjtadirv7ds7+cfBT/e2xeyr6nGvKZiP3Z//v7nBxIOdP5E/6nuoO7BsoNfa4W1skMxhzrq3OvqDusd3lQP10vrh4+kHuk5Gni0pcGuYV8jtbHsGDgmPfbi57Sf7xwPP95+gn6i4RezX3adpJwsbYKaljWNNvObZS3JLb2nwk61t3q1nvzV/tfa00anq89ontl0lni2+OzEuYJzY22itpHzvPP97YvaH15IunCrY15H98Xwi5cvBV+60MnoPHfZ+/LpK55XTl2lX22+5natqcu16+Rvrr+d7Hbrbrrufr2lx6OntXdO79kbvjfO3wy8eekW89a123Nv996Jv3Pvbupd2T3OvaH72fdfP8h/MP5w9SPMo9LHqo8rnug9qfnd6vdGmZvsTF9gX9fT2KcP+9n9L59Jnn0ZKH5Ofl4xaDhYN+Q0dHo4eLjnxfwXAy9FL8dHSv6h9o9dryxf/fKH3x9do0mjA6/FryfebHir87b2ncu79rHosSfvc96Pfyj9qPPx0Cf6p87PiZ8Hx5d8wX2p/Gr1tfVb+LdHEzkTEyKWmDUpBVDIgDMyAHhTi2jjZEQ7ILqcOH9KW08aNPV/YJLAf+Ip/T1pbgDU+gEQvxqACESj7EGGGcIkZJZLojg/ADs7K8a/TJLh7DQVi4QoS8zHiYm3+gDgWgH4Kp6YGN89MfH1AFLsfQDacqc0vdywiJY/hpFTl8FK8Heb0vt/6vHvM5BX4AL+Pv8TZuQX02Ir5hoAAABWZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAOShgAHAAAAEgAAAESgAgAEAAAAAQAAAUWgAwAEAAAAAQAAATIAAAAAQVNDSUkAAABTY3JlZW5zaG90IEkpbgAAAdZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MzI1PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjMwNjwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgoJd/cLAABAAElEQVR4Ae19B5hURdb26TTTk2CAIeewKkoQMCGYFcGMaw4oq6trwPCb111X3aTrqqtr/Iyw5ogRs2IgSMYAApLTEAeYmZ6eTn+9NZzmTtM99PR0uPfOOc/TXTdWnXrr3LdOhVvXEVFCIoKAICAICAIaAafgIAgIAoKAILALASHFXVjIliAgCAgCJKQoRiAICAKCgAEBIUUDGLIpCAgCgoCQotiAICAICAIGBIQUDWDIpiAgCAgCQopiA4KAICAIGBCwJSlu376d8MulBINBmjNnDgUCgVyqIWlbFIFwOKxtR6YRZ78AbUWKn3/+OZ111lk0evRo/cP2F198kRVUZ86cSZMnT46m9eKLL9KNN95ITzzxRPSYbAgCySJw2WWX0ciRI+nZZ59N9ha5Lk0I2IYU58+fT//85z9p8+bNNHToUP3D9j/+8Q/CuUzL3XffTfixHHHEEXTQQQfRiBEj+JCEgkBSCMBuly1bpq/94IMPkrpHLkofAq47laQvutzFdP3111NlZSVdd911dMUVV9DRRx9NXbp0oW+++Ybmzp1Lv/3tb/X266+/Tp06daLS0lKqqqqihx9+mNavX099+/bVypeXl9Obb75JL730Ei1fvpz23Xdf8ng8+pzP56NPPvmExo8fTxs3bqTOnTtTfn4+3X///bRw4UJCUwfHET90Wbt2LXXt2pU6duyo7//pp590vO+88w5t2rRJx+1wOPR9iAPXr1q1ip555hm93bt3b8rLy8sdqJJyThCAff3888/kdDqppqaGDj74YCorK9O6fPfdd/Taa69RSUkJoTXy0UcfUatWrahDhw76PJrd06dP1x4m7K1Nmzb6/OLFi+m5554jl8uln4utW7fSo48+SgUFBdo+Yfdo1eCZgN0hXRAydJk3bx5169aNWrRoodOAra5YsUIf/9///kfHHnsswY7tIm47ZARktGHDBp0VkCHLMcccQ/fdd58+h2u++uor/Rs4cCD16NFD9ztOmjRJG8npp5+uyfHiiy+O9gPOmjWLPvzwQ3rjjTc0cZ199tnaaECS06ZNo6eeekrH/+mnnxKMEYJteKqo6RF3y5YtadCgQfr4Pffcw6rR999/T++//z7BqKAbrjUKziPtV155xXhYtpsBAiA6yDXXXEP/+c9/6OWXX6a77rpLH/v666/ps88+q2cv3377rT4/fPhwuvXWWwl263a7Cf3aqOAvuOACOuqoo/Q9sEuQLCp32BzI7b///S99+eWXeh8kivvQfF+zZo1OE3+49umnn9bPDezSKLBfO4ktms8oWBQMCrSoqKhe+cAjxLmVK1fWOx5v569//asmxDFjxmhC2m+//TRxwrNDfyVq0f79++va+fe//732IFFzf/zxx9F0sT1s2LB60SP9e++9Vx9DGvBC27dvT+vWrdOEyxejtn3ssccI6aEGh9eJGl2k+SAArwytjL322otOOOEE7S1OmTJFE5URhQMPPFAT29ixY/VhECnsE4SISht2CM8P27BROAFo1fz666/6eu5rX7BgAYVCIX0fToA84YGCEGHr8BbxPMCG4VkaBWnDXuF92klsQYrspaFwY8Xv98ceSrjPBrN69Wp64IEHqLa2Vl8LwwFBQn744QcaN24ceb1emjhxot7WJxr4W7RokTYqEOGhhx6qCRGDQBB4hCxonuy9995UXFysHwocX7p0KZ+WsBkgwC2Dk08+WZPNkCFDdCsEnp1RQF4gI24ZodKHQ4DKFDMeLrzwQt1dhJYIPDxIv3799DnYFGwd14LsYIO//PKLJk2QJ4gVgngefPBB7U1iH84HC2wUHijs1W5iC1Ls2bOn7tOoqKjQtSUXEmpcTM2BB4Y+kYaEp0DgGtSs6IuEoaF25X4YeHtIC/09aHKceuqp2pgaihfn0GcJASmycD9jdXU1H6oX2q32rZc52YmLACrhGTNm6HOPPPIInXTSSVGCeuutt+LeE2snIMDBgwdrm4PHd84559Dbb7+t78XgHwS2CzK86aab9LOB7iF4mSBNCHdFLVmyRD8H6CrCc4BfcxBbkCJIr127drq80N/Cgn4SCDqpcQ03rbds2aKPsyeIHXRqc6Fj8AX9Jvih3xD9NOi3AVH+8Y9/pPfee0/X0Oh7ef7553VcDf2h4xoCj5EFneEQ9kD5uITNFwH024Gs0App27at7g7CoB1sF32BGJVuSJargUH0P4JM0Xy++eab9eUYYIEcdthhOsRsDNg79nv16qUHInHi8MMP1+eRJuT888+PPgcg2wkTJujjdv+zxUALCglkhZFnEBqaAzAk1HB8DiGaGugjQZMC/XnotDbKqFGjdJP42muvpeOOO073q8yePZvuuOMO3c+DgRF4kWj6YqQYwqOCIGUY7t/+9jc677zzjNHqgRyMVKOfBs2aHqqJMnXqVK0j5qKJCAJAgD26W265JUpQOI6pXpgD++qrr2I3oWBEGjaKyhxNWyZRNHUh6J5BvzuOY7YFiBF2zt1GTIrwLtG3CRKETaOZjeY7RpnhINhdbOEpopDg+oO8UMOCDEE6qHVRoNxU3X///Ql9NJhuAG8PNTIEBAq58sordcHzebyRgrmGGDg58cQTCf08aKI/+eST2utDfOhfhJx55pnayOCdYioEC8eNaQyYngMyRcd5YWEhYTYUauqGBIYrYn8EMEUL07EwaoxRZKNgZgQE3l8igZ2B8G677Tbd3QNCAzl2795dz9Xl+zBAA+H5s8cff7zex7085QY2f9VVV+l44DggXTxDl156qb7W7n8ORRz2Gk9XJYb+kW3btun5g5ijCEPD1ASuMTHfEJ3IbASxhQxI0K8C749JzXgN5nQZ+wf5HJrTqIXjneNrkDZIF3PLRASBTCEAO4Tn2NR5rogHzw13LWVKXzPFa0tSNAL8wgsvaPI75ZRTjIdlWxAQBASBuAjYnhTj5loOCgKCgCCQAAHpsEoAjBwWBASB5omAkGLzLHfJtSAgCCRAQEgxATCJDmNEmt8QSHSNHBcEzIQApvNgEQmR5BAQUkwOp+hVeBcZk2RFBAGrIMDTfayib671FFLMdQlI+oKAIGAqBIQUTVUcoowgIAjkGgEhxVyXgKQvCAgCpkJASNFUxSHKCAKCQK4REFLMdQlI+oKAIGAqBIQUTVUcoowgIAjkGgEhxVyXgKQvCAgCpkJASNFUxSHKCAKCQK4REFLMdQlI+oKAIGAqBIQUTVUcoowgIAjkGgEhxVyXgKQvCAgCpkJASNFUxSHKCAKCQK4REFLMdQlI+oKAIGAqBGTl7SSKA995wWchIfi6GX78EXJ8j4U//pNEVHKJIJAVBIw2u2DBAv3BtaFDh+q0xWYbLgLbfOK04Ww27WyHDh3ohhtuqBfJ+PHj9df7Yj9nWu8i2REEcogAbNQo+O64y+XazZaN18i2+ga8gLBnBECK+MRj7IcP8flRfCtaRBAwGwL4/G5paeluaoVCIf3J3t1OyIEoAkKKUSga3hg9enT0O9F8Jb4xDcIUEQTMiMCRRx6pWzNG3QYPHhz91K/xuGzvQkBIcRcWDW6h39Dv90evwXdwL7zwwui+bAgCZkPgzDPPrEeKsNlTTz3VbGqaTh8hxSSLBB8ER5OEBQQ5bNgw3pVQEDAdAmjFoDXD3T4Ihw8fbjo9zaaQkGIjSmTEiBGUl5en7wBBgihFBAEzI4BuH4/HQw6Hg3j02cz6mkG3rEzJ4ZrKDBluqg4nnHAChcNh+vOf/2wbTxEPjEh9BOxis5WVlXTaaacRms4PPfQQ9enTp35GLbqXSZvN2JQco1HxdmxoxfI45JBDaNq0aYQQI3lWFDYoDo15iHfMeN7O22yfyCNvx4ZWy39BQQH179+fVq5cST179hSbTaIA006KRiPCtvEHD8t4Pgn9THfJyJEjye12U21trel0S0YhJj2EmFKE0PjjOPg63rdzaLTJePZqPG9FHI455hgqLy+3tM2yPWbDZtPWfDYaDrZBgMYfvCreZ8NjA+N7ed/s4caNG3UHttn1ZP3YoLDPBAjjwg+TeXmbQ76Gr+d47Bay3bE9sn0iNNor9vkaxoDv5X2zh1a3Wdgm7NJos7BdPpZOm00LKbKBIGSDglHhFwwGaaMvRB9vdtMin4t2BJ1UGXZSkKQfKxcPkpsiVOoOUxt3iPoXh+jglmEq8zq19wsj4x8bIXSEwdlN2GaZ8Iz2Cptd5wvTzO0u+qHSRVtDLtqm7FZsNjdW4HFEqKUzRG09Ydq/JEQHKpttlV9ns2i1GSt2tlUOU9G4yaRoNC4mxEAgQPitqAzRh5vcNKvaS2EhwVTKJ+P3gO4GFtbQyWUB6lns0iOVGK00GhqUaIqRZTwTjUwANss/2CxIED/Y7OIdYXpvk4d+9OU3Mla5PFsIYMrMYGWzp7QLUteiOpuNJUfokqrNNokUYwkRhoW+Nr/6fbHJQRO3FFJNRGb9ZMtYmprOSa2q6eS2QfKqkUpMPYKhcZO6KUbWVL3Seb/RZrklAzL0qXmnb6x302fbC9OZnMSVQQTALKNbV9HItmE9uo7KPB02mzIpGo2La1tMaMbvubUemlYlxpVBe8hY1P28NXRJJz+1KqwjRhiaXYgx1mZBhqjEt1b76fE1XlrkF+8wY4aVwYgHF/jo0i4BKvKmpzJvkhsHI8OPPcSamhp6dZ1LCDGDBpDpqH+s8dJL69yEsgRpoGy53y3TaWcjftgrV+IgRJ/PR68pD1EIMRvoZyaN2b4CmrjeocsSZQqbZW5KJcWUSBEJQhByEwQe4uTNDvqiUt7ySKUgzHTPDF8RvbfBGSVGIyly2ZtJ32R0Yb2RF/xA+LDZSRudNLW6KJko5BoTI/DJjhLNPyhTkCJ4icucw2TVT4kUETkSMta48Cym7qh7BS7ZxOU68yIwdYeXqmv8tvIWYbP4ccsG+fu2Urp5zGuFjdPsq2355FM8xN6isTJvTExpI8Wft4dpebCgMWnLtSZGYEskj76vcGhvKrbmNbHaDaoGQuSWDR4c5G9L2NPgPXLSOgisC3tp3jbKPilybQsWZgODyzqzymsd9ETTpBD4wefVBoamJsqba17YgJUk1ma56Ty/WgZWrFSOyeg6vzqvXkWeis02yVNkUkStuyAg/TLJFJqVrlkcLKxnYFYjw1isoT8eEm4+LwqJzcZiZPV98BA3n439io3JV0qkyMYFA0Otu60mSFUkzZDGAG+Fa/3kovKaujKGgXGtawXdY3Vkm0U+YLM6Xyp/IvZCADwEPuLWDcodv8ZIyqSIhGBg+G1Wr0CJ2BOBzQFntJxTMTCzoMK6s81uqrXfq4tmwTrXelQEKWqzqVTkKbMZjAwJwsj8wXCucZD0M4SAem1dlzHKOhUDy5BajYqWCZHzAJv1hYUUGwWihS6u3kmKqdpro0mRDYxDGBgSF7EnAihblDGXN3KJbSsK50Fs1oqll7zOXPlxeXOYbAyNJkWOGAkxE2NbxJ4IcBkjhFi1rFnv2PzYs9Sad66ayk0pkyLDzgrwvoT2Q4AJhUPk0Lht5hwb9cQ2fkzwZtZbdEsdAS5nY9k3JraUSJET48Qbk6Bcay0EuIy5zK2lfWJt7ZafxDltfme4bGPDZJFIiRQROSeYbEJynSCQawTYZjnMtT6SfuYQ4DLmsDEppUyKjUlErhUEBAFBwCoICClapaRET0FAEMgKAkKKWYFZEhEEBAGrICCkaJWSEj0FAUEgKwjYkhQj1dsprH65lEhIfQxp6TyKBAO5VEPStioCal4obCeVgQKrZtkservNokg69AjM/4r8nzxLkR2bdXSOkjaUN+ISyhtwRDqibzCO4JLZFPGr19H3O0xfV/v1q1T71UvkOfhk8p7whwbvlZOCQCwCVU+Mo3D5cvIcdhZ5j70o9rTsZxAB23iKweU/ku+tf2tCdO19MOEHcqx56z7CuUyL7/V7qOa1e6LJuPcbTq7fHEDugUdHj8mGIJAMAuHtmzUh4trArI+TuUWuSSMCtvEUa96+n9SnCSn/pKso78ATNETwHGvevI9wrvj65yjw83cUXDyT8oaeRq523SlSU0U1Hz9NrvY9KO+QU/U9oYoNFJzzKQVXLSB3x96Ud/g55MivW1E84vdR4MfJFFw4nVzd9yPP/seQs7Al+d77r/ISffrL1r53HtLxI25HSWsidZwltHIB1c77giLbNpCrR3/KP/R0Up/K002kmncfJlfbbkQFJRT86Rtyddmb8g4+hRwF8s0bxq+5hLVT3tJZjTic5KjeRqHVv2h7wMHAgqkUXPS9sr1jKQBbUt1Eeao14u45oA4e1ewOKBsPzPucnC3K9HWuDj0ptHYJ1c74gDx9DyX3XgdSuHIr+T+fQJ7+R5K710AKK7v3T36Z3MouPaoiD9fWUHDuZyqtGeRULS7PsN+Sq6yzTsP3zsPkbN2RKKxWyVr+AxVe+Fdtx3UKWP/fFqSIfpfwto2alNz9dzWVPQOOJN/E/ygS2qiJB2QT/FERTvf+mhTD1TsoOPsTCrXprEkxvLWcqv57OTlCdf2Atb/OoYA6X3TjC6qkI1T5gGrGKLKLuDwUUoZZ++lz5B3zdwoq43RE6t4NxrZ7r4MovGGFjttZ2EIbXe3cL8ivyJklBMOdOYmKrn1akzn0UIt7REWfV15C8Q3jo8dko3kgEJjzmc6o98QryP/+o+T/9g0qPOd2fSyIin3+l9q2GA3fginkVedBeNUv3EEhZbfkcquFPNS6glMnkkdV7J7+h+t7wuUrNCkGlD3C5sIbVyr7vF9V9l/rfUdxa3Kr+6ofH0eRLWt1EmqhJKqdo56DKx/Vz01wdn3vFc+fndYcskXzGQULLxF9iE5v/dWUnUUt9bnwxlVsQwlD3+v3akLMO/I8Kv7TW+Ts2lfXxIEZ7xO8ThAiPMQWd0ykvOPG1pHjL9Oo5I53iHami21P36H10oDR1Ex8QB/znvtnKlJeq6O0HUW2rqdaZbQsEYeDCi57kIpvfZUor4Ai2zfpGp3PS2h/BILK86KaSnJ2+g15Bh9P8BaDC6cRBu6M4uozhIrveJfyjr5QHwaRonUCQkSlDTssuPyhqI2iZUSefAqtX6qvD6oWDwReaEStgqSJVO2DPNEfDkKErRfd/ibhecDz5Z/0f/oe/kPasFeHy16L9dqCFNUb/nXlpNz53SRYu9uhRAfYYMKb11DNu/8l2jlyrJsviiAhoRU/UdXTN2gDK771laQGUUJrF9eRtiJCzz6HkBMhms6IT3mMLI4C5VV23ks3mV2d+ujD6GwXaT4I1CqvEOI5YJQmG3fvQcp2VJN47uf1QPD0O1yf9+xsGaHSd6BiVpUpWjqVD11KIeVBFl/7lPbwcLOz2776XGj9sjpyxLWK7EJLZqrm9WJt0yBPJsiI+lSo/71HKKScDkg9x8JbTPlHnKPtVZ+00Z8tSNGp+gThZUWqKiisakuWiK9Se3o452zblQ/HDxWxcrMZNXNQGVR4k/IuVe1KTpfuTylQfSdIK7xqIdV++ARV3XM2Bdcsih+f4WikolzvOVu2ix51teqgt9EXGVdUmiLNC4FIQH3IfcksnWn/pCdpx9/PoCCawkoC09/V4W5/MXaim7i99id0BcHjq3rgYqqdVnevRw3+QfzKdkGG3tOu1c+NH60V9dyANCHoitLh+l/1cxBS/Yr6OfA0j08Y26JP0QHSa9mWIhgkUf0teQedpAsV/SQQdBTjGsqva1pHVCezFqMXqQY8dMEH/FR4yX3kUoMskNBm1Yxo00l1cE9RNeps8p5+Izlatdc1aPCHyVT75YvkvuAufW2iP1eHXvoUOrtZMOADcXWr80D5uITNF4GA6rcDWZHHS44WbTQQ6KsLq6YsWgwYlW5IQqofOzD9PeVljtSDHxiIqZn4IPm/fIHyDjmF3KrP0a9aQKEVP+pmuafvMKpt35PCy+braD37DtMhBlFCqusm7/CzKV81nSHhLeuU3ddV5PqAjf9sQYooH+9vb6LqZ28m/wePqxHmWZoEA2owBEaFcxA0NYKzPqJaNcoWVv156LQ2imfwCG1U1c/crKfSoF8ltHQuec+6VffXYGAkoLxIjBqHt6zXt4JwISBleKnol8xTc8uM4lQDOY7WnXQ/DZo1zrbd1ajedFJK6tFB47Wy3XwRqFWEBvGOvl7Nd63z6rBf/do/KfTTt1T73ZvYTSgO1aSFjdaqAbqwatqGd2zR1+pmtdrCoB/63TFVDbMbMPMBU8Zqd/YzuneSomf4GXpU2a/m2WJghvK8ejQ7b8BRyilQXUc2F1s0n1FGbuX6F5x5q6phy/TIcPCX6XW1ruo3cSrPTl+jpi04VR9NJKCmG8z8kBx5O7/7Cy9SSf7xvye3KnjaeT64bF7dXMN9hlLekJHkPuAE3UT3f/KM7oNxqvjyd07M9gwdXdcprrzTkJrOE5WdcRde/A9ylHVRxLiOQmpwxpFfSAVn/5EwXaJBUR3tIvZHIKw8s8im1XrUGKPIRuHpYsZ+ReVPauEQFaxTeZf5qiWD7h68OBBQo8RONc2r4Pw7d16tou8zWG9jOhkkb2eoBykVaUI8an5t/qjLyKHiCf78rZ6a41ReYl4zmUSuvHX468kLVi3Gd3P9fj9VVVVRRUUF/bjZTy9EVM1jEtH9imr+lk/N/dNNAzU9ofimF6Nz/tCPF1Gd0ag54wkgwTQeh/L+dLM75iLMZXSpwZJYwQghaud45/hanbYiXWdxKz5k6nA0LaGD2riotLSUiotV53p+Prndbo1LPGzMlhldlqo8YbM1NTXaZrdu3UrfbwnTRKobzDKbzunQB01th5rz6mhiP2BdPMUqnp0ORDqUy3AcFzh+oX5t8rXNFhUVRW3WiS6yJMQ2zWdjXvW0HDUSV3TxP8k/+RVyFCrjMEyCxmRsNfnFeEu9bTzsmDKTSBKRnkORb6JzHJdOe+dkcD4moSCQbgTgNaZD0hVPOnTJVhy2JEUjeJg2ICIICAKCQLIIJOdPJhubXCcICAKCgMUREFK0eAGK+oKAIJBeBIQU04unxCYICAIWR8AWpIh3N/WCnFiU0/hTx7MtWHXH99Ld+l1VWWg22+hbJz3YRj1b3Wm30VdWs5UVNZsE9lrz0VM6RbFZNb0vW9hnMp3Kf52rX1OKTQMz8Iuveyb2cEb39cIR6o0ZGFdATbaVhWYzCrclI8c0l6r7x8TV3a0WgSg49Zq45zJxMKLWC9BzeotKiUb+Xr8a2Nxt1hakyLUrXmPCkkksTqwMkkPBQrMh9W60LDSbw0IwcdJYAcf45gpUde91QE41Fpu1iafIVpR/2nW7LR0WUquHYNFOvcbhul/1myj4ZADeXOElj7DkEuYzYg1EN1bLVq/7OdWrTSyBH9RacwunqBdAI+TuNzz6yQGcx2KcterdUrybioVj8e4qz4Y3LjSLScR7WkgWCwLoV6vWLdFLR7nU64HBlT+pN23Uq4ExS6KxbhJaFwFMri4485bdMpBpmw1tWkO137ym385yq0VmjSI2azNSNBYub4c3LNeLZ2JBTRYs1+VePp8KRv8//YGrqgfG6lf7cD6kVilB/wqWBQMR+d74F2HhBxb92tPg2aqJc60+VG8xzp0rmvC1IfWiPdLVC80qwtzTQrJVj12pXwPE/Vi+KbAzovwjVPeAkCLDavswkzaLD7pVPfIHVXnXLbdnXLoOwIrN2owUK9VSXmqVhehDU/z/notuk/psQNG4J9TCrZvVqsJX608K4KRfDYzgXWcs2llw1m2qT+UVwpp2+GHZdk2IipCwck5ErdTtm3C7Jrewemkeq+pg0QiHel2wYOy9KrYIVT12ddTgdiW+awvLmBX+/gFyqQUiKtWyTryQLN6XxnvRIL9C9SYOXu6vNKwCvisG2bINAupTA9vvPDmaHYdaELnkphei+5mwWSxFBkJ0qfVB8U40PtHhV6/DNiTNzWZ3dcA1hIpFzmFRiIhhfTmHWoKJBasI63ed8b4z3uNUqxtD8N0UCJZJwit4eUeer1e0wfJJ+D4FBKuH6JWL1bar50DtTeIcBlN2ne+mt2HYxEuT6SP1/3ghWRzFQrL4xgWa3li8FuJWi0/wsmV6OTRFuiL2RAB9iq4e/aKZwzdVjJIJmw2qb7xAPOpNL7z6iu+x7IkUm5vN2ooUvef9Jcm+t13eJKn1EyFYBQSCF+jzhhyvtwNqmTGIs00XHeIPq/BAIv5q1SezTW87VN9fSmIgcMQHwTJjIs0DAfQp4v385CQ9NsvLibnK9rDociKlmoHN2mKeYqLyS+a4o7StvowXfcVXzaom/El/LMipFuCEhH6drUO9jW9oKMGgih7tVtvoh2FxxPskAp9sIHSq5jQkpJYri0q4zhON7suGIKAQaJLNqoFESJBteqdToA828s+uNmsrT7HmrQdU+3NXljB669zDeoX43GnNqnvU4rSPqab0T2pR2Xm6n496DNArGOMzkFhotvr529RCs8o7RHNW9fehTwbr1qEJhC/7VY+/XTGamjyuOrJTEc+gY9UnJ8fXpaUWy9UfrlIELWJfBCK+HXoBWWMOPar7hPbwIaim2KzD7dGfNqh5X9m7quC528aoQ7LbdrVZe3iKO9dJw+KtWKGYf7WGEWfjAIyx0DE9x6NWytaTWNXHgcJqVWLMK8w//CzdB1k45m968APeYFhNlcGSYkV/eFhN56lbU9B7khoxVoMnIE584CruKLE636AoYsX6it5Tr6sjWdW/iC8U4qtsIvZFAAMebKscBpUN75L4dtMUm3V12UdNK1MfvVJp86wK2O9uEu+Y8SIb26wtF5k1ll2y23oxUuWZYXADy7THCj4ejgEap1oxezdRr0ph5WR8pa8pgk8d6GXi8REtlZYeTVdhsfrMZK4WdJVFZptSopm9tyk2G1HNZnzYranrJZrRZmWR2TTZnV5YdudnC+JF2eBK2YpEm0qIWJHb98rf1RLwKi61hHxIjUij/tafutxTrR1PYTlmewSaYrOodJu6mrZdbXZ3l8j2pmTODGI6UNHv/qX6QHtTWH1BEFN7POqrhPnHXmxOhUWrZo+AXW1216hEsy/i3AOAz50WXaYGi0QEAYsgYEebFU/RIsYnagoCgkB2EBBSzA7OkoogIAhYBAEhRYsUlKgpCAgC2UFASDE7OEsqgoAgYBEEhBQtUlCipiAgCGQHASHF7OAsqQgCgoBFEEiZFPkNCw4tkl9Rs5EI2Kl8OS8cNhIKudxCCKCMjb/GqJ4SKbJRcVjslNVcGgO6la4todqocVlJ70S68oNS5BCbTYSR1Y83lY9SIkUGLWpgJAbGmNgtLHaGoqTI5Y08coVo9vwa9eRthMVCimYvupT0wzeSWqiydarXZVHOXOaNiSxlUuQEkXi+y0Etw3WLpDYmcbnW3AgUhP1UokixKQZmlhzyw4G84FeqWjeFKn8i9kKgbXg7eVy7EyKXfzK5bTQpMhkihHG51Npv+PUKbU4mPbnGQgj0VGXqVmXLRGIsewtlI+oxcD4QIl89xGatVIxJ6dorvCXKSShnttmkbt55UaNJEfcxISJREKJbLew60LGpMenKtRZAoA9V6LJF+bKBWUDthCqy3SI/Ho+H9o2IzSYEy6In+jq2apsFL8FmU7HblEgReMHAmBDz8vKorSdM+/hXWxRKUTsWgZ7+9bSXp1qTB0iEjQzlbkVhQmSbBSl29/gJ+RSxBwIDapZTmSdSz2ZTsdeUSJENDCyMBwakmJ+fT8PDq+yBbjPPRV7IT4eFV+gyRbmCQEAmKPdUjMwMcLLuTIqwWa/XS8NDyylf+hbNUERN0qE4UEkH0zpdpihbttmseIpsXAhhYEicDay0wENnbvuWvCFfkzIoN+cOAVckRCOq51Fbr5MKCtQnX5WBGT3F3GnW9JTxgLDNguxBisjnsZXzCPkWsSYC7nCARvnmU0uvR5cpyhY2y4TInJVs7lLyFBE5kyIShxJ4gAoLC6lToYtGbJ9FBQEZjU62EMxynUd5iCO2fU+98oNUVFQUNTBUfKkamBnyxg8FQuQD+WGbRT575QdoRMX3BA9ZxFoIFAaq6IQdM6lTgUPbLFfk7CmizBsrTSJFGBg3n6EMDKy4uJi6FRCdun0qdayWD7k3tkBydX3rmk10csUU6u0N6TJEWbKBsaeYioHlKj/x0mVSjLXZkpIS6uUN0onbphFwELEGAu185XTK9mnU3RuO2iy8f7Ru0CLgiryxuUlp5W1+OBAamyOhUIjC6qNL+KCOw7GDRlb+QIt8q2lRQTfaWFj3sfnGKijXZxaBktpttHfVMuofLteGBYLAD16/0cBQ1vzLrEaZiZ1tFg8KbBYPDmw11mZPrZpFP/ja0y9FPWlHXsvMKCOxNgmBlv6ttF/lYtrHUUHFJcXaXo02a/QSU7HZlEiRc4QEYWAgQTRHEEL4OGrkftVV1KdqDlVsc9JGZzFVOzxUo35qzrm+lu/ROxb4q1yzVGtZ3LmXBbTdpSKTgpvCetJyccRPZeFKKvE4NfkVFpZGPX32EmP7ZnbFZt0t4GBs4bD9sc3igRpSvZX2rVhLlcEIbVI2W+nIp2pnvthsloudbVZVX1QQrqFiNSDWNlJFRcpm67rr6mwW9so2a+wD5/sbq3bKpGhMEMTIguP4sQcJb8Pn81FRbS21rfVRMLijXu3MRsn3mz2cuXKOVvGAdrvybHadoR+XC3tKdU3IEu0xwcBifyBErnG5GWIscyvkOVZH1h/5gSB/LEyUOBbtb/T7qXUgoGzWJzbLQGUxjLVZjxpI8XhKdPmAV+qIsVCH2DfaLN/LZd4YtVMmRSTCCbKRccJsYGBtNrBaRYoBbWDB3QzMSsSIgoC0bt2as2v60GggKBtUWCBFEACXEcrJaFixhGj6TDZCQeDBNsrY8D7yDRz8ihDFZhsBapov5XLhsjLaLGwVdsv2yvuwaVyHsuT7U1GrSaTICbLi2Odt9hShPMjQSIjGfhwmRA45TrOG6GuDtGnTxqwq1tML5QFhI2FSNBoZkyNCGBZ+uI6Ni+/XEVn8DzjA1thOOW/IK+cdD5nRZoPBoO5/5P5ytlUOzQ6JnWyWbRS8gm3ej2ezqZZLk0kx1sj44TM+dCBBGBaTIYwr1sBSzUC277OipwiMuFyY7BByGRlDPs/X873ZxjmT6SWyWcYED5zYbCZLYM9xo4wgCNkmYafGH4gQ+3yebZbv3XMq8a9oMimy4hw9K8aKQmkmQSZCDnGPVWpbzh+8CEiLFi34kCVCNhSEKBsOuZw45PJDpvgeS2SwkUoa88Z5Bgax9sq2yiGSEZttJNgpXs5lhDCbNpsWUuQ8Q3kYDBsZZwbHjD9cz4bFIcdh9hC1E4Q9RrPry/qhLCDG0FhOvB17Pe/bNUS+YYOcf4R4AI32yjYaG1oFE7HZxpVUWkkRScOojML7bFA4Z9yOt2+832zb8CQg6MuwinAZsL7G/UTbfG1zCI0YIL+8b7RT4zauid3HMbOK1WyW8TfiaTyWaNt4fVO2006KRmVilWdDMh43Xm+FbdadDc0KOifSkfOS6HxzPG7EBNtis+ayAmP5ZEqzjJJirNLZyFBsmune5zxwmO74JT5zIWCHcuY8cGguhM2nTcrvPpsvK6KRICAICAJNR0BIsekYSgyCgCBgIwSEFG1UmJIVQUAQaDoCQopNx1BiEAQEARshIKRoo8KUrAgCgkDTERBSbDqGEoMgIAjYCAEhRRsVpmRFEBAEmo6AkGLTMZQYBAFBwEYICCnaqDAlK4KAINB0BLL6RkvT1TVHDOXl5TRv3jxzKCNaCAJ7QAD22r59+z1cJacZASFFRiLJsEOHDjRhwgT6+OOPk7xDLhMEco/AzTffnHslLKKBQ73wXve1KYsoLGraE4Hrr79eZ+zBBx+0ZwYlV5ZBQPoULVNUoqggIAhkAwEhxWygLGkIAoKAZRAQUrRMUYmigoAgkA0EhBSzgbKkIQgIApZBQEjRMkUligoCgkA2EBBSzAbKkoYgIAhYBgEhRcsUlSgqCAgC2UBASDEbKEsagoAgYBkEZPK2ZYrKfoqOHTuWVq5cGTdjw4cPp7vuuivuOTkoCGQSAfEUM4muxN0gAmPGjEl4/owzzkh4Tk4IAplEQEgxk+hK3A0icNRRR0U/PG+80Ov1Uv/+/Y2HZFsQyBoCQopZg1oSiofAoEGDoh+c5/NHH300b0ooCGQdASHFrEMuCRoRiNeEvuCCC4yXyLYgkFUEhBSzCrckFosAmskejyd6uKysTNb+i6IhG7lAQNZTzAXqaUzTDiu/HXfccTRp0iTdjD722GN3a06nEa6sReVwOLKWliSUXgRkSk568cx4bEYSTLTNShjP8zEzhlgZGs1oEMlHH31kRhXj6hSP+PgYh3xj7D4fl9B8CIinaL4yiasRE5wxxHbsDzfzNXEjMuHB1q1bU4sWLai0tJQCgYAJNWxYJSY8hIl+XCZ8bcMxytlcIiCkmEv0k0ibHyYj+YXDYYr9Gc/zPUlEb5pLMOLcs2dP8vl8ptEpWUWMROh0Oin2h/M4ZhQhRyMa5tqW5rO5yqOeNkxuCJkEQ6EQ8S8YDBJtn0HOis8p4ltJjsB6orC/XhyykyUEnIUUyetIVNCLwm1OJmdRH3K5XOR2u3WIbRAjEyi0EmLMUtk0MhkhxUYClq3LmRCNZAgSxC9QU0HOtU+QZ+N4cgUVEYqYDoGgdz8KtBtLkQ5jKC8vT5MjEySTI5QWYjRd0ZGQovnKJNonyISoiVD1tQV8m8ix5klyl/8fucIVJtRcVIpFIOjpSsGO48jR8Xzy5Jfo6UfwGvFjQuQw9l7Zzw0CQoq5wT1hquwhIkQzGYRYW1tLfr+fHCvvo8Ly+9W98gHGhACa8ESEPFTV9d/k7Hgu5efnRz1H9hiFFM1VaDLQYq7y0NpwHyJIEaOxNTU1FFn9FJWU/9uE2opKe0LAQQEqXnU97XDkUU3706MeIprTPAAjxLgnFLN3vv6QWPbSlZTiIBDrJYIQ4SGGNn1KxetlGa04kFnoUJiKVl5HgYo5ukzRAkD3CJe5hTJie1WFFE1WxPG8xPz1D5OD1EiziKURcJKf8tY+pD1/VHhoCTAxCjmap2iFFM1TFloTJkV4EvASw1u+IW/NDJNpKeqkikBh1SQK7vhF9xOLt5gqipm9T0gxs/gmHTvIkH/cl4gBFs/Wt5OOQy60BgKeTS/pCg/eoniK5iszIUWTlQkeEh519vtrqMj3pck0FHWaioC36puop4iy5qYzh02NX+5vGgJCik3DL61346FgUtReRPVycoc3pzUNiSz3COSHllGgtlpPt0J5s7eYe81EAyAgpGgiO+DmMx4S9DdRzUoTaSeqpAsBDJpFatboMjZ6iumKX+JpGgJCik3DL+13G71FR2BD2uOXCE2CQG15dPRZPEWTlMlONYQUTVIeRi8RDwk8iHBE5tabpHjSrkYo7IiSIspexDwICCmapyyiHe7sOSAUsScCXAkyIfK+PXNrrVwJKZqgvPjBgCq8LQ+JCQomgypwxSflnEGQU4xaSDFF4DJ5mzwomUTXPHFzBWgejUQTICCkaDI74AeFQ5OpJ+qkCQEp3zQBmYFohBQzAKpEKQgkiwCTI4fJ3ifXZQ4BIcXMYSsxCwINIiBE2CA8OTsppJgz6CVhQUAQMCMCQopmLBXRSRAQBHKGgJBizqCXhAUBQcCMCAgpmrFURCdBQBDIGQJCijmDXhIWBAQBMyIgpGjGUhGdBAFBIGcICCnmDHpJWBAQBMyIgJCiGUtFdBIEBIGcISCkmDPoJWFBQBAwIwJCimYsFdFJEBAEcoaAkGLOoJeEBQFBwIwIyNLOZiyVDOqERZ7XVhRS25IaynPvWsR2zvLW9MHcLjSi/1o6qPemjGmwYG1L6lhaTaWFAZ3Gdp+bXviuNy1eX6L0idCQnpvorINXZCT92LQzkohEankEhBQtX4TJZcAfcNJdbw+kn9e0VJ85qGsg9Gy7g24/ZT61a+mnRYqUvl9aRl3bVGWMFJduKKZbXhlCnVtV0eNjp6sFdYmufH4obfd5oplYV+FNCylursynn1e3pO5tK6lbm2qKTTuaoGwIAjEICCnGAGLHXZDPra8Opl83tNBeWv+uFTRrWRtatrGEbn1tCD37+ylZyXbnVtU0tM/GKOn+sq6FJsTSQj89dvF0cjiINlfmpUWX6b+W0ROf703H9VtL40YsVERcP+20JCKR2BIBIUVbFmv9TM1UBAhCbFFQS0+MnabJB0R5xsNH0KYdXuUltqh/w869z3/qQLg3XzWzQS77ddkWvQ6e2Cc/dKRfy1vQ0N9soOF7baB8T11z3Ffroq8Xttee576dK+iYfddTaVEt+YMuKvYGqEZ5rfNXtqIJ3/bW8bmU4/rilF509iHL6J1Z3ah3+x10wsA1+lyV302vTetOKzYX0wE9N9Ox+60lb15dOrVBJ709systVE3yHm2rtI6dWvloyuK29KnSDTJvZWt6bXp3GjlgbTRtfUL9rVde6Rc/d1T3t6BuZdXKQ12uMKpr1r/xfTfasL2Ajuy7nt6f04UK84J0+oErCfGL2BsBIUV7l6/OHQgIMqDrVk2I2IZXdstJPymyLCGvR31jOkb++W5/mrqkrbouopq5Dk0eFw77lc5U/X0rNxfS1eMP0Xe4nWFNfvDKnr/8O0WgIfrdU8MIZOZxhWiGapKP/6YP/eOs2dRCEeKnP3ZSBFZJh++9nhaXl+g4tlTlEQh41IA1+vyS8jpSRH/jpU8fqki0zkxnL29DT3/Vh1688hulc5guUee2VeeR0xGmWcvL6M0Z3emRMdNp3opWtFR5wZCNO/Jp8sIOdIjqJ+W0T9x/Da3ZWkjjxh9EwXBdV8LclW3ow7md6YnfTaN2LWpU/2pX5bXm00fzO+t48PfJj53p1asnU0FeKHpMNuyHgIw+269Md8sRvETIgb021zuHAZVzhy7TfW7GE2H1+U0QFgZEXh83mZ6+5Dt9+p3Z3XT4kvLqIGOGL6G3rvuK9u+2WfVTEv24ulQRUHtNiPspD/HNayfTRYct0eT43aJ2+h7+O+OglXTVsQv17pF9y+m1cV/zqWj47OTfaEIc3GOzJqPfHrhC94e+pcgPAzPwdtEcn3j9V3TlzrjgcV5x7CK6/OhFOp5j91tHj140PRonb9zzXj9NiKcMXkkvX/U1Haq8XRDkQx/35Ut0eOmRi+mta7+kfl0q9P43v9TPR72LZccWCIinaItibDgToZ2DzGHl8SUjTmeE/u93U3XT86kvf0MrNhVrj3FHTZ259O+yVTVR2xHIEU3XkQPX0l2/nae9zzVbCnQSP60ppZtfHkJH7LOeXrxCeXaqybtyU1EyyUevQdyQMw9arr0zEDiIumOpj/bptF13BcATvVcR3Grl+UFWKS82GeHrxgxfqkfhQX7IE8jWKMeo5rrbFVEVyiZN+uiHJVpnvES2bYaAkKLNCjRedvqoProFa0tpzorWdIzynFhmLWutBluKVZ/g7lNwrplwkCKaIjV1xk+92lXyLTo8cdAaCoScBM/x+6Vt9a9NsZ8eUR5Z59Y+uuv0ufTs131o4bqW+veM8vjuPWeW7pusF9EedmoCLn0FSBCCKUQj+tfpjxHri58cpr07EGWb4lp9TTJ/gaBDe5xo6vO0pNLCuvuDKl/xxKUqCpHmgUB8C2geeW82uRzYbavO6/yVpdE8o4n8D9VvOOHbPlS50wPkk5jPx4Q44Q/f0Z2nzyP0HbK8OaMbrasoUJ7aVHr60inUSZES+t/QdJ66uEz3Rd4w6md65arJdPg+5Zq4XprSk29POkTfHgSDPZAN2710xxsDVd9hN3pTDYSguXvY3uX0pOoH/MMxdU1xfaHhD03sWPGo+ZDo78TAz4Zt+fr0D6vq+l07ykBKLFzNbl88xWZQ5Bi17V5WqZvBf3j2ENpbNT3nLG+lvD0XlalJ3Ht33E5z1eAES7c2lbq5vM2Xp6a17KW8zJb6WpzHfMcFqmmMOY3LN5XovrhKNagCwYTwLTsHJ6YtKaPRB6zSI7w416YkeU8O10NOGbKKFr7fUk+t+VmliZFkkC+mFA3oVkFvz+pOGHxBEx+DKZDq2jpdeqj8QqapwaLuZVU0qPsWvc9/x/VbRx/O60LjJhxMB/fZRN8taqtPYbBHpHkjIJ5iMyh/9BH+SzVfMViwfpuXvvy5A1VU59NeHbbRfefOqocARqWL8kOEkeY8V1gTx1rVX8cj1Bitvn7UT/pekCWaxgE1NeaUwav0QM7xaurLqAGrCYT6nGpCL1EDNv27bqHLjqob+EBiDqpz35zcxRnPnVPXYZoP+hNDO0e/MUp9VN91egR8iCL6g3tvVCToovfmdI3qt1WNRkP27bxNTxKv9HvolWm7vFRO+5IjFmtC96mR7a8W1BHq6CEr6KRBq/X9/Mcq6lEdPiihrRFwqM8sxmlg2DrPpsscigC/YDBINTU1VFVVRVu3bqVw+UTqWX17WvVFaZcrYmzbwk/J9JOheYk3XuJJMOTQ5If+xHjS0L3xrk90DDqj6dy2xE8geKOgf7BKER/mQcaTLWoyOOYY8tzG2GtCqhsB3ic31WPPZ2p/UcnTVNBuGJWWllJxcTHl5eWR2+1W+RM/JVOYJxuvNJ+TRcom18ET7FBa11eXTJYSESLuxahsIkLE+YbuxflkBTq3bxlfZ/QPlrrjEyLib72HARhUDNkmxGTzLdflBgGplnKDu6QqCAgCJkVASNGkBSNqCQKCQG4QEFLMDe6SqiAgCJgUASFFkxaMqCUICAK5QUBIMTe4S6qCgCBgUgSEFE1aMKKWICAI5AYBIcXc4C6pCgKCgEkREFI0acGIWoKAIJAbBIQUc4O7pCoICAImRUBI0aQFI2oJAoJAbhAQUswN7pKqICAImBQBIUWTFoyoJQgIArlBQEgxN7hLqoKAIGBSBIQUTVowopYgIAjkBgEhxdzgLqkKAoKASREQUjRpwYha9kfAgYUiRUyHgJCiyYqEHxQOTaaeqJNmBLicOUxz9BJdCggIKaYAWrZuCTnrPmKfrfQknewhEHbt+lBY9lKVlJJBQEgxGZSyfA28BvxCzrpPe2Y5eUkuCwgEVdlyOYuXmAXAG5GEkGIjwMrUpfxQGB8SfMCo1tOTglSYqWQl3hwhUE2dyeEq0B+p4rLPkSqSbBwEhBTjgJLLQ3hIQIj4uVwu2koH5FIdSTsDCGxxDtNly+VsrAwzkJxE2UgEhBQbCVimLucHgx8UfO4SvwrPkZlKUuLNEQLb8w7XZcufNBVvMUcFkSBZIcUEwOTiMBMjPEQ8MB6Ph7Z7j6HKSLdcqCNpZgCBzZFB5M8fqMsW5cyVoBBjBsBOMUohxRSBy8RteDC42QxSxAfS8/PzaalrXCaSkzizjEAg7KUVnqt1maJsUemBGLkyzLI6klwCBIQUEwCTq8N4QPCg4IHBg+P1eslfcAAtCF6eK5Uk3TQgEIk46Ee6jSIFfXSZomxR8bG3mIYkJIo0ISCkmCYgmxoNewtGTxFeYkFBARUWFtKWgtNpQe2YpiYj9+cIgfnB66im4FBdlihTlC0qPpQ3lz1Ckdwj4M69CqKBEQE8GHhQ8MDgwQkGgxQIBHS4KXwuza4uo/55j5PH4TPeJtsmRaAm1ILmBm+mYNEB1LK4mIqKiup5ikyKJlW/WaolpGiyYmdS5CY0ms+hUIjC4TBFIhHaQcfRt9X70m8cT1OXvGkm017UYQTQXF7qP5pWOMdSfnFbalFSQiXqB68fZcr9iUyK4iUycrkPHepBi+ReDdGAEUBx4AcShJdYW1tLPp+PqqurqaqqiiorK/UP+xH/eioOL6CWjsXUyvUrtfKsoHyXeJCMZTZDX6iYtgZ6UkW4t/r9hqpdfcmZX6Y9Q3iHxcpLxI9JEX2K3J8IQhRSzGZpNZyWkGLD+GT9LNdRCOEhMjH6/f4oOYIQQZT4gTTxw3W4ngkVinNcWc9ECglOmjRJ3zVq1KgU7s7NLUxk8PbwA8nxVCruD+Y+YZAh9yXyIIt4ibkptz2lKs3nPSGU5fN40EBmCPHQ4CGD8L6xWY2HDITIfY7GZjYTIodZzkajk+N8tm7dutH35uIGJkSEXDbIA5MizxwAOaK5jB+OxTabc6G7pNkwAkKKDeOTk7N4yEBm7ElACX7wQIp4uECGIEUQIn4gxHikmJMMpJAoyAJSVlaWwt25u4XLBWXFlRiIkQkQIW/juLHJDK1xv4i5EBBSNFd5RLVhYkSIB4kfPmzzQ4cmMzebmRB5QAYRWcVLhK7IE8QqniJ0ZULjsgEpony4jJAnJkIcY+I03od4RMyFgJCiucqjnjZ4eEBs/NBxiAcM5GckQiZDXM9kyGG9SE26w6TYsmVLk2q4u1pGcsM2lw+TI5MgQj5vvGf3GOWIGRAQUjRDKTSgAz9EuATbRpIEOTIJGsMGojPtKeQFgpFaK4qR9IzbIEQIlyOHVsxjc9FZSNEiJW18mHibPcHY0CJZqqcm5wn9b1YVzkNsiPzwMavmrTnpLaRosdKO93DFO2axbEVJgz1Gq+kfT187lEu8fNn9mJCihUvYjg+dHfNkYRNrlqrLghDNstgl04KAIJAIASHFRMjIcUFAEGiWCAgpNstil0wLAoJAIgSEFBMhI8cFAUGgWSIgpNgsi10yLQgIAokQEFJMhIwcFwQEgWaJgEzJyXGxL1myRK+TmGM1cp481oqEzJs3L+e6mFWB3r176zUZzaqfXfSS9RRzWJIgxMsvv5wGDBiQQy3MkfTWrVv1gglWevc5m8iVl5fT8ccfTxdddFE2k22WaYmnmMNixyrakAcffDCHWkjSVkBg/PjxVlDTFjpKn6ItilEyIQgIAulCQEgxXUhKPIKAIGALBIQUbVGMkglBQBBIFwJCiulCUuIRBAQBWyAgAy1ZLkYMrvz66686VQ6N01AGDhyYZY0kObMisH79esKoM4RDtpX27dtThw4dzKq6pfWSKTlZLj6Q4llnnaXXD8SqzPiMANYQxMenevbsSY899liWNZLkzIrAxx9/TA888ADhi4CwEwhsBnM6b775Zj1Fx6y6W1kvaT5nufTwQfQDDzyQampq9AfuEcLIsY7gqaeemmVtJDkzIzBs2DD9YTLYB3/nG9v48iHOiWQGASHFzODaYKzHHXecNmzjRfjYvRi6ERHZRgU6fPjw3YA4+OCD5c2W3VBJ3wEhxfRhmXRMMHR8s9koOIaHQEQQMCKACtTr9UYPwUvEMZHMISCkmDlsG4z5yCOP1H2JuAhGL4beIFzN9iQqS+5PBAioTON5j80WoAxkXEgxA6AmE+WJJ54Y/VgTvsYnhp4Mas3zGlSY6HPGgBwqU5HMIiCkmFl8E8bep08fKisr08aOF/1FBIFECJxyyimET7+CGFGZimQWASHFzOLbYOynnXaa/pi9GHqDMDX7k6hA0Wxu27YtYVskswjI5O3M4ttg7Icddhi9/fbbYugNoiQngcB5550nQGQJAZm8nSWgEyWDNRWl9k+EjhxnBPB2C0TeYmFEMhcKKWYOW4lZEBAELIiA9ClasNBEZUFAEMgcAkKKmcNWYhYEBAELIiCkaMFCE5UFAUEgcwi47lSSuegzF/PatWupsLBQrxqSuVR2jzkUChF+eMsg3g8TbBPJihUr6N///rdeAGLvvfdOdFlajmPVHcxrw6+xgtVZnn/+eercubOeS9nY+812faIyg55YdaYxgjL/85//TL/88ote2MN4L84Fg8G4doFzSCtReVjFNhYvXqy/KVRbW2vbAUJLTcnBF9+wlNKsWbMICyhAsNwWjrVo0YI2bdpEP/zwA/Xq1Yu6d+9utNdGbc+cOVMT1xFHHLHbfWeffTZBj0QC0hs0aFDc0+vWraOpU6dqUj35hmWjgQAACxtJREFU5JPjXpOOg99//z3ddttttM8++9Cjjz7a6ChnzJih9UT+M03ejVYuhRtGjx6tyzP21o4dO9ILL7wQe7jBfRAsyrC0tJSuuOKKetf+/e9/p6+++qreMeMO5qWOGzfOeCi6bRXbWLVqlc4/JpOPHDkyqr+dNixFin/7299o7ty5hAU2DzjgAJozZw4tW7aMLrvsMnrllVdoypQp9NBDD9GoUaPoxhtvTLmc7r77bv0Qff7557vFcdJJJ9Hq1av18W+//VavgzhkyBBNyjhohikTeG1QZBcC8NIgeJUSCyqw9OjRgzfTEmKVI/YEUTmjkoa3vddee+n4YbO5FrGNPZeAZUgRhg1ChDz11FNUVFSkmyknnHACbdy4kT744AP68MMP9fnZs2fTiy++SOeff77eR3Nw+vTperFOXN+/f399HDX+d999p5tBX3zxBWFJpp9//lmvXYcL4PWdccYZZHx4Lr74Yn0v/i644AJCDX/llVfWu+bLL78kECZ0xruq8TxO3L98+XJ644039CtcV111lX639ZtvviHcj4cLr/8ddNBBuJRefvllwlw1vAeLCd/oOjjnnHP0Q6cvaOAvmXsRJzBChYM1+2Ilnl7IO3DG64qMy9NPP00VFRV04YUX6rhi48nlPhZmhd0YBSRx//336/IrKSnRnl7fvn3p9NNPj65ahErwpZde0i2EY445xnh7ve2jjz6a8IMgTtjjiBEjtJ3whT/99BN9+umneiVtrLKOBYfjNeHj2Qaa7JMmTdL2DoKFBwxJpnw5/dgQ9o/nALYGvbZt26bj3X///aOXouX03nvvafvs1KlT9DhvJNLr4Ycf1k4DPGrYK55DVBZ4JsxQQbD+saFlSBGGA4PGAwuyuuaaa6hVq1aaDGHYjzzyCGEiNGTDhg0ELw+k+Je//EUTFEgG133yySd0ySWX6DcEQFwfffSRNjTch/5AGAl7FjCSoUOH1iM8XNeQoAkFgmX5+uuv43qumzdvJhAhFpm9/PLLddroBgC5s6Aphnxi8dmJEydqz+P999/n0/qhw37sMmTRC3Zu7OleGO8777wTe1t0P5FeeCd32rRpmiwwAR0LoeIBBUnecMMN0fvNvAGbANEYBd0PIDS0PrZv305jx46N2gTOpSqwp3vuuSd6O+JC+f3vf/+LHsNGPNtApWTs/gfuICvY257Kt17kMTuwz88++6weBngu7rrrLu1ZgzD/9Kc/xdy1a7chvVA5Tp48WfezotK877779I1cge6KxVxbjetlzrHuIDiQGwoSHtz1119PqHndbjddd911mkCgIvo6nn32WW3IqMVQu8HIUdtDXn/9dR3yH7yjf/3rX9rjQ23G3gS2G7Pw648//qgJEfc/88wzukMauuGh4yY30kR/KIgQhIh+JngL6MAGIWJNxVdffTXaFwiv2CjwSqHXgAED9GF4lclKonvxUABXPGDwCEBqLA3phXvw8EBg8P/5z3/0No7hnNkElQs8bf6hecsCffEpCFQOqGTQ+kDfMfBFJbnffvvpc6mSPcj33nvv1cn99a9/1bYIu4O3jdYCSzzbwDl0HUFQgeF62BiIEfezJCpfPt9QiNXg4TCgAoDAWYC8++67Ojz33HP1MbyaapSG9Lrpppt0dwXiveWWW7RTMmbMmHr2ZYzLLNuWIkX03YHQYNz4bsX8+fM1MYIA4wm8S3Sko3MbniQKEMa/Y8eOepfjvVLEbSSDehckuYPmJ+Twww+nHqq/CsTFgy58Dufx8SE8cL179452vLMHgnyBCNGchf7wvjDSxwLCB9HCg4Xwx6/4fENhvHtB1vC+MehwyCGH6GYOdy8grj3pBbLAg4Jvz4DkjzrqKD3A05AeuTrXr18/XSYoF/yMi7dioA6DSqiUuA9w6dKlujkNfdFVgnMg1FRk0aJFmhRAhIceeqjuWkBlCGGMsR3PNjAyjVFtlDtI6oknnoj2jS5cuBC3aYlXvnxuTyHKDS0lbv6vXLlS38IfyoJ3h/5Y2DbLnvRC5QLHBYIBmjZt2uhuFb7frKFlms/V1dW6T61ly5baI7z66qu19/fggw/qfq1EL8xfeumlukDQ1M70O8b8fmrXrl2j5Y2VTSDx+unw0OEeDM6wJ7llyxZCkwTCgwLx7m1o6o++uYE/472sc7t27eLesSe9MAo5ePDgqM7YNqvAE+ZWAOvIXSW8j9CID5qykG7duukw1T/GGaTIgooIAtuOFaNtgFAgIEa2DeyjAsWxWDHqH3tuT/vGe4ENPFekA0KOlWT0Mn6dEpWOGVsQsfnaPaexV5hkH0Zy7bXXaqNGbQkvCiPBjz/+uPZQuICgLpoqEDStcRyEyE0U1KbxDEnf0MQ/TAVCfyL6ejB1B8I1LYwDXh8EDwYGX1577TU9So4+JUwtgsBweBoN+rMg8GIyJfBkYajcH4t0eLoTtvekFx5o4xcI4ZHD2zB6YYjHqoKBLnjtKFPYmxGbxuQJrQIIPEYWbj3A22aJZxuwKwgqSR7wgI2j6YyuIQxuZULwjKGCxzOElg2eI7QGWPakF65DlxcENobZIfBsMVXMzGIZUkTTB80XNNO4uYv+LhQSajEUEBssOorxMGNqDgoDHb7oi0GfHyY1Q/jaeIUDrwlTfdDcRlpc+PGuNR7DuohoymP0G31P8PDWrFmj9Ybh48GCYA4l+hShJyahozmEQSE0m2E0aO5jBNTYx2hMJ53b8AyQPzTDL7roIt10hPGyoBJpSC/0JQJTXAfPAv1HGHm9/fbbOQrThBjkYO8bSnXp0oXQLGxIMNoMUsRUL1Rw6LJJRZAWpufAHjAy30N1r2AQA/YJ7DA4CIlnGxi9Nd6L7gqUEUgR/c+ZFExjwuAZbAPpGqepgZAb0gvPASpbED3ycOedd9Idd9yhdUa+zSqW6lOER7XvvvtqA8LACR5kkA0eQjzcIE4YH4hzwoQJ2qvESDOaeOhAR1OQPRgQKhcMh1xIZ555pvZE0ckOb3NPwvfDo8OADZpomD6ENGAQTz75ZNzmB67FvfBiYeDIB7oHMC3ozTff1Hm69dZb6yXPadU7GLOT6JpEx0HgwAX4wNPlJj+iRZ4S6YVBLAx6gWhA5BgpxzbiQN7NIvB4ICASjIbyDzbUkOA+VE7obwPh86yCRDga4+JrOMQ54AjPCxUhdME0FRBFvErXaBvAGd1E8DbLy8u1vYBEQejop2MxpsXHYsNkrsE9fB2mfUFnVPAYfDHaBq5LpBemNyG/EIxeg1ChP7ojeMBTnzThn2WXDoNxwKPjwjNiC+BBTEyAOIfrQVDJCprYiKcx9xjjRnMDfTEw/MYKSB0PYSabzfF0AkZ4yOL1H+H6XOkVT9dsH0PLAvk3klCqOqAbBS0cNEcbKxh0Q7dKUwcFG5suWluo8GL7ZDmeXOnF6acztCwpphMEiUsQEAQEAUbAUs1nVlpCQUAQEAQyhYCQYqaQlXgFAUHAkggIKVqy2ERpQUAQyBQCQoqZQlbiFQQEAUsiIKRoyWITpQUBQSBTCAgpZgpZiVcQEAQsiYCQoiWLTZQWBASBTCEgpJgpZCVeQUAQsCQCQoqWLDZRWhAQBDKFgJBippCVeAUBQcCSCAgpWrLYRGlBQBDIFAJCiplCVuIVBAQBSyIgpGjJYhOlBQFBIFMICClmClmJVxAQBCyJgJCiJYtNlBYEBIFMISCkmClkJV5BQBCwJAJCipYsNlFaEBAEMoWAkGKmkJV4BQFBwJIICClasthEaUFAEMgUAkKKmUJW4hUEBAFLIiCkaMliE6UFAUEgUwj8f7UxpFK/ru6OAAAAAElFTkSuQmCC"}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# encode question\nquestion_input = Input(shape=(None,))\nquestion_x = embedding(question_input)\nquestion_x = SpatialDropout1D(0.2)(question_x)\nquestion_x = Bidirectional(LSTM(200, return_sequences=True))(question_x)\nquestion_x = Bidirectional(LSTM(100, return_sequences=True))(question_x)\n\n# encode answer\nanswer_input = Input(shape=(None,))\nanswer_x = embedding(answer_input)\nanswer_x = SpatialDropout1D(0.2)(answer_x)\nanswer_x = Bidirectional(LSTM(250, return_sequences=True))(answer_x)\nanswer_x = Bidirectional(LSTM(150, return_sequences=True))(answer_x)\n\n# merge the encodings\ncombined_x = concatenate([question_x, answer_x])\n\n# predict start index\nstart_x = Dropout(0.1)(combined_x) \nstart_x = Conv1D(1,1)(start_x)\nstart_x = Flatten()(start_x)\nstart_x = Activation('softmax', name='start_token_out')(start_x)\n\n# predict end index\nend_x = Dropout(0.1)(combined_x) \nend_x = Conv1D(1,1)(end_x)\nend_x = Flatten()(end_x)\nend_x = Activation('softmax', name='end_token_out')(end_x)\n\n# merge the parts into one model\nshort_model = tf.keras.models.Model(inputs=[answer_input, question_input], outputs=[start_x, end_x])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model will be optimised with adam and use categorical cross entropy as a loss function. Like last time, accuracy, recall and precision will be used to measure the models quality.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"short_model.compile(\n    loss='categorical_crossentropy', \n    optimizer='adam',\n    metrics=['categorical_accuracy', 'Recall', 'Precision']\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I've increased the patience of the reduce learning plateau call","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    tf.keras.callbacks.ReduceLROnPlateau(monitor='loss', patience=3, verbose=1),\n    tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5, verbose=1),\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And once again, set off the training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = short_model.fit(\n    x = [train_long_answers, train_questions], \n    y = [train_start_labels, train_end_labels],\n    validation_data = (\n        [val_long_answers, val_questions], \n        [val_start_labels, val_end_labels]\n    ),\n    epochs = SHORT_EPOCHS,\n    callbacks = callbacks,\n    batch_size = SHORT_BATCH_SIZE,\n    shuffle = True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Evaluate\n\nAs this model has two outputs there are more metrics to display this time. The only one that is the same is the loss for the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Epoch: {0}'.format(len(history.history['loss'])))\nprint('Loss: {0}'.format(history.history['loss'][-1]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15, 5))\n\nax[0].set_title('Training Loss')\nax[0].plot(history.history['loss'])\n\nax[1].set_title('Validation Loss')\nax[1].plot(history.history['val_loss'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This model has accuracy, recall and precision metrics for both the start and end index outputs. This is further split by training and validation. Here are the metrics for the start token output of the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize=(15, 10))\n\nfig.suptitle('Start Token')\n\nax[0,0].set_title('Training Accuracy')\nax[0,0].plot(history.history['start_token_out_categorical_accuracy'])\n\nax[0,1].set_title('Validation Accuracy')\nax[0,1].plot(history.history['val_start_token_out_categorical_accuracy'])\n\nax[1,0].set_title('Training Recall')\nax[1,0].plot(history.history['start_token_out_recall'])\n\nax[1,1].set_title('Validation Recall')\nax[1,1].plot(history.history['val_start_token_out_recall'])\n\nax[2,0].set_title('Training Precision')\nax[2,0].plot(history.history['start_token_out_precision'])\n\nax[2,1].set_title('Validation Precision')\nax[2,1].plot(history.history['val_start_token_out_precision'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And here are the metrics of the end token output.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 2, figsize=(15, 10))\n\nfig.suptitle('End Token')\n\nax[0,0].set_title('Training Accuracy')\nax[0,0].plot(history.history['end_token_out_categorical_accuracy'])\n\nax[0,1].set_title('Validation Accuracy')\nax[0,1].plot(history.history['val_end_token_out_categorical_accuracy'])\n\nax[1,0].set_title('Training Recall')\nax[1,0].plot(history.history['end_token_out_recall_1'])\n\nax[1,1].set_title('Validation Recall')\nax[1,1].plot(history.history['val_end_token_out_recall_1'])\n\nax[2,0].set_title('Training Precision')\nax[2,0].plot(history.history['end_token_out_precision_1'])\n\nax[2,1].set_title('Validation Precision')\nax[2,1].plot(history.history['val_end_token_out_precision_1'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally here's a list of all the final metrics for this model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = history.history['start_token_out_categorical_accuracy'][-1]\nrecall = history.history['start_token_out_recall'][-1]\nprecision = history.history['start_token_out_precision'][-1]\n\n\nprint('Training')\nprint('Start token accuracy: {0}'.format(accuracy))\nprint('Start token recall: {0}'.format(recall))\nprint('Start token precision: {0}'.format(precision))\nprint('Start token F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))\n\naccuracy = history.history['end_token_out_categorical_accuracy'][-1]\nrecall = history.history['end_token_out_recall_1'][-1]\nprecision = history.history['end_token_out_precision_1'][-1]\n\nprint('End token accuracy: {0}'.format(accuracy))\nprint('End token recall: {0}'.format(recall))\nprint('End token precision: {0}'.format(precision))\nprint('End token F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy = history.history['val_start_token_out_categorical_accuracy'][-1]\nrecall = history.history['val_start_token_out_recall'][-1]\nprecision = history.history['val_start_token_out_precision'][-1]\n\n\nprint('Validation')\nprint('Start token accuracy: {0}'.format(accuracy))\nprint('Start token recall: {0}'.format(recall))\nprint('Start token precision: {0}'.format(precision))\nprint('Start token F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))\n\naccuracy = history.history['val_end_token_out_categorical_accuracy'][-1]\nrecall = history.history['val_end_token_out_recall_1'][-1]\nprecision = history.history['val_end_token_out_precision_1'][-1]\n\nprint('End token accuracy: {0}'.format(accuracy))\nprint('End token recall: {0}'.format(recall))\nprint('End token precision: {0}'.format(precision))\nprint('End token F1 score: {0:.4f}'.format(\n    2 * (precision * recall) / (precision + recall)\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Like last time, let's finsih off with a look at how the model performs on some examples.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_short_answer(question, long_answer):\n    sentences = [long_answer, question]\n    \n    sentences = encode(sentences, tokenizer)\n    \n    predictions = short_model.predict(\n        [np.expand_dims(sentences[0], axis=0), np.expand_dims(sentences[1], axis=0)]\n    )\n    \n    predictions = np.array(predictions)\n    \n    pred_start = np.argmax(predictions[0,0])\n    pred_end = np.argmax(predictions[1,0])\n    pred_string = ' '.join(long_answer.split(' ')[pred_start:pred_end])\n\n    return pred_start, pred_end, pred_string","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'which is the most common use of opt-in e-mail marketing'\nlong_answer = \"<P> A common example of permission marketing is a newsletter sent to an advertising firm 's customers . Such newsletters inform customers of upcoming events or promotions , or new products . In this type of advertising , a company that wants to send a newsletter to their customers may ask them at the point of purchase if they would like to receive the newsletter . </P>\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start, end, short_answer = test_short_answer(question, long_answer)\n\nprint('Start token: ' + str(start))\nprint('End token: ' + str(end))\nprint('Answer: ' + short_answer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question = 'how i.met your mother who is the mother'\nlong_answer = \"<P> Tracy McConnell , better known as `` The Mother '' , is the title character from the CBS television sitcom How I Met Your Mother . The show , narrated by Future Ted , tells the story of how Ted Mosby met The Mother . Tracy McConnell appears in 8 episodes from `` Lucky Penny '' to `` The Time Travelers '' as an unseen character ; she was first seen fully in `` Something New '' and was promoted to a main character in season 9 . The Mother is played by Cristin Milioti . </P>\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start, end, short_answer = test_short_answer(question, long_answer)\n\nprint('Start token: ' + str(start))\nprint('End token: ' + str(end))\nprint('Answer: ' + short_answer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Final thoughts\n\nIt took me a few days to work out how these models work but it turns out that it's not difficult to get language models like these working to some extent. If these models were to be made competition ready though I would endeavour to use the BERT model (or an equivalent such as roberta) and apply far more pre-processing to the text before it goes into the model. I would also increase how much data the model is trained on as it is likely to be overfit with so few examples fed into it. In the meantime I hope this helps anyone looking to get started with natural language tasks such as these.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}