{"cells":[{"metadata":{},"cell_type":"markdown","source":"Since I completed the Convolutional Neural Networks course on [Coursera](https://www.coursera.org/learn/convolutional-neural-networks?) I've been looking for a challenge where I can practice a model with triplet loss. While the size and complexity of this dataset may not be ideal I thought this would be a good opportunity to do so.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport tensorflow as tf\nimport tensorflow_addons as tfa\n\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nfrom scipy.spatial import distance\nfrom tqdm.notebook import tqdm\n\nfrom kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As there are many large images to work with this challenge will need a bit more processing power than usual to get through even a fraction of the training dataset. The below piece of code was taken from the tutorials in the [Petal to the Metal getting started competition](https://www.kaggle.com/c/tpu-getting-started). It requests a TPU for the notebook. If there isn't one (if the notebook was not configured to use TPU) it will get a CPU or GPU instead.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The TPUs in Google Cloud (the ones Kaggle are using in the background) require the data to be close to the location of the TPU. Thus a Google Cloud bucket needs setting up to put the data in. Kaggle provides a handy library to get the path to the nearby bucket for us. \n\nTo use tfrecords I needed to create a number of public datasets including the records. The paths to the GCS buckets containing them are found here.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"PT1_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-tfrecords')\nprint(PT1_GCS_DS_PATH)\n\nPT2_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt2')\nprint(PT2_GCS_DS_PATH)\n\nPT3_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt3')\nprint(PT3_GCS_DS_PATH)\n\nPT4_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt4')\nprint(PT4_GCS_DS_PATH)\n\nPT5_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt5')\nprint(PT5_GCS_DS_PATH)\n\nPT6_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt6')\nprint(PT6_GCS_DS_PATH)\n\nPT7_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt7')\nprint(PT7_GCS_DS_PATH)\n\nPT8_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-triplet-loss-tfrecords-pt8')\nprint(PT8_GCS_DS_PATH)\n\nPT9_GCS_DS_PATH = KaggleDatasets().get_gcs_path('google-landmarks-2020-tripley-loss-tfrecords-pt9')\nprint(PT9_GCS_DS_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Hyper-parameters\n\nWith the TPU setup I'll define some hyper-parameters as global variables. Sticking them up at the top of the notebook makes them quick to adjust when experimenting with the model.\n\nYou'll notice that the batch size has been multiplied by the strategy.num_replicas_in_sync. It's worth mentioning here that when using a TPU the notebook will effectively replicate the model for as many TPU chips there are available (at the time of writing, Kaggle make eight available). The batch size thus needs multiplying by the number of TPU chips available.\n\nThis all means that we effectively get eight models training at the same time which is one of the reasons why TPUs are so fast. The strategy that was set is a set of instructions that manage how the model is replicated, distributed across the eight chips and then merged back into one model at the end.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 64 * strategy.num_replicas_in_sync\nEPOCHS = 20\nSTEPS_PER_EPOCH = 1451645 // BATCH_SIZE\nRATE = 0.0001\n\nIMAGE_SIZE = 128\nEMBED_SIZE = 2048","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load and pre-process data\n\nTo get the most out of the TPU I have converted a sample of the dataset into tfrecords and published it in this [dataset](https://www.kaggle.com/mattbast/google-landmarks-2020-tfrecords). If you're interested in how I created this dataset and what it looks like you can find the notebook I used to create it [here](https://www.kaggle.com/mattbast/google-landmarks-2020-create-a-tfrecord-dataset/notebook).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Pipeline\n\nTPUs and tensorflow work best when the data is fed into the model with a tf dataset pipeline. For those new to the API, tf data is a preset series of functions that converts the dataset into tensors (tensorflows favoured data type), transforms it (e.g. resize the images) and applies a bunch of other useful functions for training such as shuffling and batching.\n\nI'll start by getting a list of the tfrecord filenames.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = tf.io.gfile.glob([\n    PT1_GCS_DS_PATH + '/*.tfrec', \n    PT2_GCS_DS_PATH + '/*.tfrec', \n    PT3_GCS_DS_PATH + '/*.tfrec',\n    PT4_GCS_DS_PATH + '/*.tfrec', \n    PT5_GCS_DS_PATH + '/*.tfrec', \n    PT6_GCS_DS_PATH + '/*.tfrec',\n    PT7_GCS_DS_PATH + '/*.tfrec', \n    PT8_GCS_DS_PATH + '/*.tfrec',\n    PT9_GCS_DS_PATH + '/*.tfrec',\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then I'll begin the pipeline by telling it to expect tfrecords.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = tf.data.TFRecordDataset(\n    filenames,\n    num_parallel_reads = tf.data.experimental.AUTOTUNE\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"TPUs work a little faster if a batch of data is not necessarily inserted in the order the batch started in. Setting the experimental deterministic option to false gives us a little bit of extra speed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ignore_order = tf.data.Options()\nignore_order.experimental_deterministic = False \ntrain_data = train_data.with_options(ignore_order)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next define a function to load the three images in a triplet example. These functions get an example from a tfrecord file and decode the three images inside it. I've also added some random augmentations to the images so to add further variety to the dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_triplet(example):\n    tfrec_format = {\n        \"anchor_img\": tf.io.FixedLenFeature([], tf.string),\n        \"positive_img\": tf.io.FixedLenFeature([], tf.string),\n        \"negative_img\": tf.io.FixedLenFeature([], tf.string),\n    }\n    \n    example = tf.io.parse_single_example(example, tfrec_format)\n        \n    x = {\n        'anchor_input': decode_image(example['anchor_img']),\n        'positive_input': decode_image(example['positive_img']),\n        'negative_input': decode_image(example['negative_img']),\n    }\n    \n    return x, [0, 0, 0]\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.\n    image = tf.image.resize(image, (IMAGE_SIZE, IMAGE_SIZE), method='nearest')\n    \n    image = augment(image)\n    \n    return image\n\n\ndef augment(image):\n    rand_aug = np.random.choice([0, 1, 2, 3])\n    \n    if rand_aug == 0:\n        image = tf.image.random_brightness(image, max_delta=0.4)\n    elif rand_aug == 1:\n        image = tf.image.random_contrast(image, lower=0.2, upper=0.5)\n    elif rand_aug == 2:\n        image = tf.image.random_hue(image, max_delta=0.2)\n    else:\n        image = tf.image.random_saturation(image, lower=0.2, upper=0.5)\n    \n    rand_aug = np.random.choice([0, 1, 2, 3])\n    \n    if rand_aug == 0:\n        image = tf.image.random_flip_left_right(image)\n    elif rand_aug == 1:\n        image = tf.image.random_flip_up_down(image)\n    elif rand_aug == 2:\n        rand_rot = np.random.randn() * 45\n        image = tfa.image.rotate(image, rand_rot)\n    else:\n        image = tfa.image.transform(image, [1.0, 1.0, -50, 0.0, 1.0, 0.0, 0.0, 0.0])\n\n    image = tf.image.random_crop(image, size=[100, 100, 3])\n    image = tf.image.resize(image, (IMAGE_SIZE, IMAGE_SIZE))\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.map(\n    get_triplet, \n    num_parallel_calls=tf.data.experimental.AUTOTUNE\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally add some helper functions. \n\n* Shuffle helps to prevent overfit \n* Batching ensures the right amount of data gets put into the model per step\n* Repeat enables us to train for more than one epoch\n* Prefetch gets the next batch of data while the model is training on the previous batch","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = train_data.repeat()\ntrain_data = train_data.shuffle(1024)\ntrain_data = train_data.batch(BATCH_SIZE)\ntrain_data = train_data.prefetch(tf.data.experimental.AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at a small part of a batch of triplet sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(5, 3, figsize=(15, 15))\n\nfor images, landmark_id in train_data.take(1):\n    anchors = images['anchor_input']\n    positives = images['positive_input']\n    negatives = images['negative_input']\n    \n    for i in range(5):\n        axes[i, 0].set_title('Anchor')\n        axes[i, 0].imshow(anchors[i])\n\n        axes[i, 1].set_title('Positive')\n        axes[i, 1].imshow(positives[i])\n\n        axes[i, 2].set_title('Negative')\n        axes[i, 2].imshow(negatives[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model\n\nNow the model can be defined. To enable triplet loss the model has three inputs (one each for the anchor, positive and negative images) and three outputs (again one each for the anchor, positive and negative embeddings). The three inputs use the same middle embedding layers to ensure they are embedded the same way. This embedding centre is made up of a pre-trained backbone (that does not get trained), one GEM pooling layer (generalised mean pooling which you can find out more about in this paper [link](https://arxiv.org/pdf/1711.02512.pdf)), one dense layer to create the embeddings and a normalisation layer to help the embeddings generalise. I've kept this model small as training will already take a long time thanks to the size of the dataset.\n\n![Screenshot%202020-07-31%20at%2009.35.49.png](attachment:Screenshot%202020-07-31%20at%2009.35.49.png)\n\nNotice that the strategy variable has made an appearance again. This replicates whatever model is defined for each of the TPU chips and works out how to merge the model again once the training ends.","attachments":{"Screenshot%202020-07-31%20at%2009.35.49.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAcAAAADnCAYAAACJ+4ZJAAAYWWlDQ1BJQ0MgUHJvZmlsZQAAWIWVeQVUVF3397mTzDAM3d0l3c3Q3Q0CAwzdHSopEiqCgCCgggqCChYhYiGIKCKogIFIGJQKKigC8r+Evs//eb/1fes7a517f3efffbZcWrPAMCZRo6MDEXQARAWHhttZ6zH5+Lqxod9ByCAAABwAYjsGxNJsrGxgL/An/f/LsvDMDdcnkltyvrv9v9rofejxPgCANnA2McvxjcMxtcAQGX4RkbHAoBRhemCCbGRm9gDxkzRsIIwjtzEAds4axP7bOOyLR4HO30YXwCAikAmRwcAQGyD6XzxvgGwHOILuI0h3C8oHGadh7G2byDZDwDOXTDPrrCwiE3sAmMxn3/ICfhfMn3+yiSTA/7ibVu2CpVBUExkKDnp/9Md/+8SFhr3ZwwRuBICo03sNm2G/fYiJMJ8ExNgPB/uY2UNYwYY/wzy2+KHMQIfGGfiuM2P4PKN0Yd9BlhgLOtHNjAHmxEHCKPwUCuLHbqPf5CRKYzhGYJIDIo1ddjpm0OJMbTfkVkZHWFn/Qf7R+uTdvo2kqO3xt3k74oLcSTtyH8RSDH9I/97cqCDM4zxACDx8UFOVjAmwpgpJsTefJsHKZAcqG/1hyc6zm5TfyEYq1LCjfW25SM9/KON7Hb4I8Ni/tiLzA4MMrXawWWxgQ4m2/5BNviSt/Rng3ELJZzk+EcOJcbF4o8tfhQDw23bkf2UcMcde5FjkbF6djt9FyJDbXb4UVSUUONNugCMOWLi7Xf6ojRj4Qm5LR9lERlr47CtJ8o7mGxms60PKh5YAH1gAPhAHFx9QAQIBkH9863z8Nd2ixEgg2gQAChAaofyp4fzVks4/LQHyeATjCgg5m8/va1WCoiH6et/qdtPKeC/1Rq/1SMEfIBxGDAHofB33Fav8L+jOYF3MCXov0b3hXUNhetm23/TSDDFYocS90cuH+0fTowhxgBjgjHCiKM4UNooDZQF/NSFqzxKFaX2R9v/8KM/oAfRk+gh9Dj6pVdQRvS/dLEE47B8ox2Lff5pMUoElqmE0kNpwdJhySgWFAeQQinC45BQOvDISjBVf0fvTdv5/g92/rXgHz7f4cPJ4hA4VpwuTuzfPYkSRKW/UjY9+k//bOvq89er+n9b/j2+/j/87Ae/zf/NicxBXkX2IO8ie5EdyFbAh7yNbEP2IW9u4r9z6N3WHPozmt2WPiGwnKD/Go+8M+amJ2Nkz8vOyK7ttIFYSmLs5gLTj4hMig4KCIzlI8E7P4XPNNxXehefvKy8LACb58j2NvXNbut8gFie/IdGmQZAE14buIH/0IKPAtDQDQBr3n9oIu4AsMP78uWnvnHR8ds01OYDDe8GtPCKYgc8QBCIwRbJA2WgAXSBITAD1sABuAJP2M+B8HyOBglgD0gH2SAfHAEl4Dg4CU6Dc+AiuAJaQQe4C+6DR2AADIHX8Px5D+bAAlgGqxAEYSEaiBFih3ghYUgSkodUIW3IELKA7CBXyBsKgMKhOGgPlAnlQ0XQcagaqocuQ9ehu1AvNAi9hCagGegr9AuBRBAQTAhuhAhCBqGKICHMEQ6I3YgARBQiGZGFOIwoQ9QgLiBaEHcRjxBDiHHEHGIJCZDUSBYkP1IKqYrUR1oj3ZD+yGjkPmQeshRZg2xEtsORfoYcR84jV1AYFCOKDyUFz2ETlCPKFxWF2oc6iDqOOodqQXWhnqEmUAuo32gaNBdaEq2ONkW7oAPQCehsdCm6Ft2M7oZX03v0MgaDYcGIYlTg1eiKCcakYA5iqjBNmDuYQcwUZgmLxbJjJbFaWGssGRuLzcaWYy9gb2OfYt9jf1JRU/FSyVMZUblRhVNlUJVSNVDdonpK9ZFqFUeHE8ap46xxfrgkXAHuDK4d9wT3HreKp8eL4rXwDvhgfDq+DN+I78aP4r9RU1MLUKtR21IHUadRl1Ffon5APUG9QmAgSBD0CR6EOMJhQh3hDuEl4RsNDY0IjS6NG00szWGaepp7NGM0P4mMRGmiKdGPmEqsILYQnxI/0+JohWlJtJ60ybSltFdpn9DO0+HoROj06ch0++gq6K7TjdAt0TPSy9Fb04fRH6RvoO+ln2bAMogwGDL4MWQxnGa4xzDFiGQUZNRn9GXMZDzD2M34ngnDJMpkyhTMlM90kamfaYGZgVmR2Yk5kbmC+SbzOAuSRYTFlCWUpYDlCsswyy9WblYSK4U1l7WR9SnrDzZONl02ClseWxPbENsvdj52Q/YQ9kL2VvY3HCgOCQ5bjgSOExzdHPOcTJwanL6ceZxXOF9xIbgkuOy4UrhOc/VxLXHzcBtzR3KXc9/jnudh4dHlCeYp5rnFM8PLyKvNG8RbzHubd5aPmY/EF8pXxtfFt8DPxW/CH8dfzd/PvyogKuAokCHQJPBGEC+oKugvWCzYKbggxCtkKbRH6LzQK2GcsKpwoPAx4R7hHyKiIs4iB0RaRaZF2URNRZNFz4uOitGI6YhFidWIPRfHiKuKh4hXiQ9IICSUJAIlKiSeSCIklSWDJKskB3ehd6ntCt9Vs2tEiiBFkoqXOi81Ic0ibSGdId0q/VlGSMZNplCmR+a3rJJsqOwZ2ddyDHJmchly7XJf5SXkfeUr5J8r0CgYKaQqtCksKkoqUhRPKL5QYlSyVDqg1Km0rqyiHK3cqDyjIqTirVKpMqLKpGqjelD1gRpaTU8tVa1DbUVdWT1W/Yr6Fw0pjRCNBo1pTVFNiuYZzSktAS2yVrXWuDaftrf2Ke1xHX4dsk6NzqSuoK6fbq3uR5I4KZh0gfRZT1YvWq9Z74e+uv5e/TsGSANjgzyDfkMGQ0fD44ZjRgJGAUbnjRaMlYxTjO+YoE3MTQpNRky5TX1N600XzFTM9pp1mRPM7c2Pm09aSFhEW7RbIizNLI9ajloJW4VbtVoDa1Pro9ZvbERtomxu2GJsbWwrbD/YydntseuxZ7T3sm+wX3bQcyhweO0o5hjn2OlE6+ThVO/0w9nAuch53EXGZa/LI1cO1yDXNjesm5NbrduSu6F7ift7DyWPbI/h3aK7E3f3enJ4hnre9KL1Intd9UZ7O3s3eK+Rrck15CUfU59KnwVffd9jvnN+un7FfjMULUoR5aO/ln+R/3SAVsDRgJlAncDSwPkg/aDjQYvBJsEng3+EWIfUhWyEOoc2hVGFeYddD2cIDwnviuCJSIwYjJSMzI4cj1KPKolaiDaPro2BYnbHtMUywRf2vjixuP1xE/Ha8RXxPxOcEq4m0ieGJ/YlSSTlJn1MNko+m4JK8U3p3MO/J33PxF7S3up90D6ffZ2pgqlZqe/TjNPOpePTQ9IfZ8hmFGV8z3TObM/izkrLmtpvvP98NjE7OnvkgMaBkzmonKCc/lyF3PLc33l+eQ/zZfNL89cO+h58eEjuUNmhjcP+h/sLlAtOHMEcCT8yXKhTeK6Ivii5aOqo5dGWYr7ivOLvJV4lvaWKpSeP4Y/FHRsvsyhrKxcqP1K+djzw+FCFXkVTJVdlbuWPKr+qpyd0TzSe5D6Zf/LXqaBTL6qNq1tqRGpKT2NOx5/+cMbpTM9Z1bP1tRy1+bXrdeF14+fsznXVq9TXN3A1FJxHnI87P3PB48LARYOLbY1SjdVNLE35l8CluEuzl70vD18xv9J5VfVq4zXha5XNjM15LVBLUstCa2DreJtr2+B1s+ud7RrtzTekb9R18HdU3GS+WXALfyvr1sbt5NtLdyLvzN8NuDvV6dX5+p7Lveddtl393ebdD+4b3b/XQ+q5/UDrQUeveu/1h6oPWx8pP2rpU+prfqz0uLlfub/licqTtgG1gfZBzcFbT3We3n1m8Oz+c9Pnj4ashgaHHYdfjHiMjL/wezH9MvTl4qv4V6uv00bRo3lv6N6UjnGN1bwVf9s0rjx+c8Jgom/SfvL1lO/U3LuYd2vvsz7QfCj9yPuxflp+umPGaGZg1n32/Vzk3Op89if6T5WfxT5f+6L7pW/BZeH9YvTixteD39i/1X1X/N65ZLM0thy2vPoj7yf7z3Mrqis9v5x/fVxNWMOula2Lr7f/Nv89uhG2sRFJjiZvXQWQcEX4+wPwtQ4AGlcAGAfgNMF9O8/bKUj48rGZyzpBhggSUhXFhsZjqLCyVK64TPxtAoaGTGylw9OHMjxkUmKuZAVsIez9nMpcR7jneHX5CvgHBfFCasKuIiGiYWIe4noS3BKLkvd3lUuFSGvJ0Mi8lW2SS5O3VeBX+KR4XWm/sq0Kl8p71Ua1RHWSBl7jmWallp/2Lu2vOq26e0h6egS9t/q3DBoMq4wKjfeZkE11zNjMFs37LBotq6yqrTtspuzQ9uwOHI50TkinNedVV+CGcyd60OxG7V7ynPQa8L5DvupT61vul0dJ8g8IcAjUC1IMlgjhD2UPow1Hhn+PmIwciLoRfSbmcGxqXHZ8cyIqiZJ8Zw/YK7JPPdU0zT09LuNwZklWyn7F/VPZBQdscoRzqfNAPuIg/SGxw9oFVkecC92K3I66FDuVOJTaHrMqMy83Pq5XoV2pVqVwQuqkxCnZavOazNPjZ01rL9TN1dM3CJ+Xu6Bx0aDRssn5ktflwCuRVxOu7WvOaNnfmtOWf72gveRGZUftzWu3um+P3Bm/O9zZdM+/i63rQXfp/YQe/we7e50f2j4y7zN+bNLv8CRq4NTgy2fUz2WG9IdNRwxfqL4UfkV8tfJ6evTFm7tjp99mjgdMOE5aTVm+s35v/cHso9o06/T4TN6s4uz43Ln55E8mn6k+138x/jK1cHox8avnN+vvlkvBy50/D/xqXTfY2NiJvxwShZxBjaOnMAtUSJwyPpC6kjBOlKBNoLvPwM6YxPScRZ41g+0NhxJnNtcADwevC18hf4fAqOCS0LLwrMhj0dNi0eLaElQSzyVP7gqWUpL6LX1f5rCssxyv3Ef5RoV4RS0lSKlbOU/FWpVRdVitXN1dg1tjFJ4FHtrs2iM6x3TdSSKkVb0h/csGBw0pRprG9MYfTDpMS8zizSkWPpaBVhHWYTY+ttZ2GvYSDpyORCeE07LzR5dh13tuje4VHnm7kz2DvFy8DcgyPmy+kO+s3xCly785oDawNCgrOCLENVQ3TDScBp4JE5FjUd9j+GO94srj7ya8SJxKmk9e2UO9l2efWCpfGibtbXpzRkFmdJbnfsdslwNBOZm5VXkX85sPthy6dvhywcUj9YVni04drSguKSkozT2WUZZUHnE8oCKoMq3q9knxU+dqRE8XnXl2dqWOeI6jXrBBAp4HKhe1Gw2aLC+5Xg69kn319LVbzYMtY63Tbd/akTdYOyRvatzSva1yh/8u4u5kZ8+95q667or7R3r2P0jujX4Y+yi3r6Of5cnegTdPOZ7pPHcY8h9OGzn74snL768ZRqXeWIxFvj02fmPi6eTY1OS7uQ9oOPrpM4Nz9POyn5Q+i3yh/fJz4cPiyNeH365/r15KXXb6Ifpj+WfHSvIvjVXCmsH6zE78paE5RBXSEyWOxqIXMTPYWapJ3CI1niBMQyK60abTXaAfZNhgEmY2ZAlm3c92kv0aRzfnA6773Dd4qnkT+fT4fvGfETAXmBPMERIV6hT2FF4RKRaVFX0oFiCOFa+TMJH4KJm9S2xXt5SvNJCuktGUeSEbB99umuQt5KcVMhV5FNuU7JTmlfer8Kq0wreWabVUdRb18xokjaeavpqftVK0sdoVOoo6w7rJJB5Sm5613kv9QP0NgxpDGyOc0T3jPSaKJrOmNWYe5mzmwxYllvZWtFa91pk2GjbfbZvsQuxF7d85VDvudmJ3eu5c4GLisuHa7BbqLuT+xqN0t9XuZc9iL2Gva94k71fkRB8BnxfwPhJIMfZXCVALNA0iB4eFkEN1wujCRsPPRoRFKkWuRd2LzouxiWWOfR13Mt4vQSThQ+KJJMOk0eTQFKaUZ3tu7L21ryv1Xtr19PqM0szMrIj97tmGByRy0DnPc8vz3PKF8lcPjh96fPh6wakj+wrdi9SPchxdKR4uuVJ67NihsqLy6uNXK+5XvqiaPbF6iqaar0bhtMkZj7MRtfvqcs8drE9rIJ9XuUC88PXip8aVS4TLPFfkr9pcS2m+1vKzTe16ZHv5jUsdbTdv3Oq9vXTXuPN6l333Uk9pr8LD532H+r0HTJ+SnusNh74kjs5N9s8ufV/ZjP/2732bBaMMwNF0OEPNBsBRB4DCLjjPHILzTjwANjQAOKgBhIg/QBD6AKQ+8ff8gODTBgOoAT1gA7xAFMgCdTgXtgZuwB/EwNllATgBGsEt8ARMgO9w5sgFyUHGkBeUABVCF6AH0AcEBiGGsEDEIKrgPG8DzuvikdeRv1HGqKOoSbQCOgf9FqOOKceswhnWQyoVqjocJ64QT43PpcZTHyFwEOpoFGk6iFrEdlpV2ht0JnSv6WMZ6BguMhowDjI5MA0yWzM/ZfFi+clazqbFNsa+l4OTo53TkwvH1cEdz6PI8433Cl80vxL/mkCPYKlQoLCmCFFkXPSqWI64jwRJUmQXcdeq1GfpdzJDss1yKfJy8mMKOYpKil+U2pSLVJJU/dQs1GU1WDWJWtLaFbqSpEN6vfpfDKmMmI3ZTbhMhcwUza0soizLrLqsv9oK2jnbH3bocUI5G7hku/a5s3j47G7wfOeNIdP7YHyWfN/7jVJmA2gDzYNKgj+GaoYVh3+ONItqiCHERsW9SjBKbEuWSqndy7evIo0lvTATn5W+f+lAcM5cXv7BsMPNhfRHOYo/ldaXeR1nqRioOnTS+NRSTcEZprM5tcvnQuq/nj9y0bCJ/tLilQ/Xplvm2j62T3Us3ma9q3/Ps9u7x75X55HMY/EnyoPhz36OoF7hRk++ZZy49Z44vWeO9Knpy+pX5e9Gy/gfh34+XJn+9X715dq19SO/fTZkt/aPzfhjAQEwAHbADySAAtACJsABeIMwkAJyQTmoB9fBI/AGLEBoiAOS3Yp+ElQMXYL6oU8IWoQCwg2RibiCeI/kRXohzyDnUcqoLNQQWhydjh6FY1+BBdhA7BCVIVUbTgbXgBfHX6BWpL5NsCFM0SQSccQSWn7aS3D++po+gYGFoZXRifET015mPHMZixTLQ9YINla2O+xBHEwcdzgjuIS4RrnLeVx42Xhf8lXx+wnICgLB50LnhbNEPEQV4VxuVrxP4ip8ihVIZUrvkYmV9ZXTlSfI9yvkKZorsSotKr9U6VFtUatRP6iRrBmvlavdpvODpKDnp59vUGvYYnTD+IbJTdNeswkLhKWElZP1fptW23l7IQcvxyqnMRcB12C3Fg/sbmfP417d3oPkTp963xy/IIqdv0mAa2BG0J0QmlCfsI4IjsjkqDcxerH18bQJkYmPkvlT4vcM7FNKPZPOmVGchd+fkj2fQ86dzE8+JFuAOPKm6HJxfKnisa/llyviqtRP/DpVWyN/uurMx1rRusBzlxpYz1de1Gr8dKn8itrV/mZyy2pbTbttB7hZf9vizmLnyS6f++oP+B+iHj1+HP8EM5D3lPCsZshrxPJl6Ou6Nx/HeSdt3qV/uDXDOnfks8jC42/FywdXTFfl106sv/u9uBN/FMABOnj18wNJoAz0gQ3whGO/F1751eAaeADG4HVPgEQgXWg3lAJVQDehCQQOjjoZUYIYQDIjKcibKC5UGmoW7Yp+jNHH3MRqYe9SWVC9wcXgafGXqJ0ISEIrTRRRjviTtpuunD6OwZXRlMmM2ZbFjFWFTZxdicOLM4krltuHx4HXis+S31LAQtBSyE7YSyRG9JBYg/gDiZldNFIq0v4yx2WH5TkU/BSblFZVbFQfq+dqumqjdY7orumZ62fCEWw16jC+ZdJvumpubtFiJW19wVbarsVB33HYOcwV73bBw8mT3pvax8vPnfIuQCMwP+hDiF1oX7hlxNMo9+jp2JR4noSxpPspd/ZWpTqm/cqoznLK5j2wkHsz/+Ah/wLjQvaiR8X+JcvHMsvpj9dUKlc9PulfDdVUnlE9O1QXV8/Z8OBCaqPxJZkrRtdSW2raCtpdO1hvjtyuuOt6D9t19r5iz41ew4cjfYn9MgPIwYVn00ODI4UvRV9Vvf79xnAs7+2jCdpJx6lT72Y+yH0MmT4182B2dh79ieuz7BeDBedF8le/bzbfBb4vLR1a5lpu+KH24/iPlZ/OP1tWWFaiV1pWVn/p/sr61btKXLVfPbY6sEa1pruWuHZ5bWadf911vWj94fr6b7nffr+P/X70+/eG3AZlo2yjbzP+Mf4K8lvHB0TQAwA9trHxTQQAbBEA64UbG6s1Gxvrp+FkYxSAO6Hb/yFtnTV0AFQubKJHbb/S/v1fzv8AeEDZYozmsIQAAABWZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAOShgAHAAAAEgAAAESgAgAEAAAAAQAAAcCgAwAEAAAAAQAAAOcAAAAAQVNDSUkAAABTY3JlZW5zaG90YCrG4wAAAdZpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+NDQ4PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjIzMTwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgq8/B2FAAA1y0lEQVR4Ae2dB5gcxdGGay/qlCVACPhJEiCCwMjkDCaIYLLBYGMbAyYZsAGbYDImGDDJJJMzJuecESAyApEkcpKQkBDKl2//eVvqZW5vd+/2pL2b3fnqee528nS/VdPVXd0znUgGYhIREAEREAERiBmBspjlV9kVAREQAREQAUdADlCGIAIiIAIiEEsCcoCxVLsyLQIiIAIiIAcoGxABERABEYglATnAWKpdmRYBERABEZADlA2IgAiIgAjEkoAcYCzVrkyLgAiIgAjIAcoGREAEREAEYklADjCWalemRUAEREAE5ABlAyIgAiIgArEkIAcYS7Ur0yIgAiIgAnKAsgEREAEREIFYEpADjKXalWkREAEREAE5QNmACIiACIhALAnIAcZS7cq0CIiACIiAHKBsQAREQAREIJYE5ABjqXZlWgREQAREQA5QNiACIiACIhBLAnKAsVS7Mi0CIiACIiAHKBsQAREQARGIJYHIO8CZM2fahx9+aHPmzFlgBXGNAw880G655ZYFvlb6BT7//HP75JNP0jdrvZsIoAvshr/x48dbS0vLQknJc889Z+PGjct4ralTp9r9999vjY2NGffnu3Hy5Mku/QvrevneP87HT5gwoQ37ZDLptrGvq0Q2VVjSFYW9/IJf/dxzz7VXXnnFtt56azvuuOMW6IINDQ322Wef2cCBAxfoOplOPvLII62urs6eeOKJTLu1rYsJHHrooa2cXllZma211lqGnhZffPFOpQYnesYZZ7jzb7vtNhszZow988wztvfee9tSSy1lN910kz3yyCM2ePBgW3/99Tt1j/BJ//nPf+zVV1+1K6+80lZYYYXwLi0XmMA///lPV6Hdfvvt7eijj3Z3owJ9+OGH20orrWRXXHFFQVIgmyoI1qwXjXQLkJrva6+95hL//PPPGzUwiQh0lEB1dbVRkJ1wwgm2xhpr2BtvvOGWO3p++nE40XPOOcfOOusst+utt96yxx57zL755hu3fsABB9g//vEPW2edddJP1XqREnj00UeN6E5XiWyqq0jPu0+kW4C0pqh1L7nkkjZx4kQbNWqUbbbZZi7l++67rzU1NVmPHj3syy+/tGWWWcaOPfZYGzZsmM2dO9dOP/10V0OvrKy0LbbYwmgRePniiy9st912s1mzZrlWwUknnWS9evWyH3/80U4++WQX4iovL7eNN97YXZNr7Lfffi4MixOmpffggw/6y7X5feGFF1xBSVo++ugja25uduleZJFF7IEHHjCuvcsuu9if/vQnd+69995rN954o7v+csst58K06667rtvHfWgBcN9tt93WHn/8cZd2CtvZs2fbiSeeaB988IGRxu22287VUNskKKYbYLLhhhu63GMD8MNWEHj+61//spdeesmIDAwdOtROO+0017rDrnB0RB6wP1qO2Fbv3r2dQ6UFgJ4IdyKnnHKKoY/a2lq7/fbbXWsQJ4md3Hnnnalj3nzzTRd+J1356i1fm3r//fedo/7+++8Nu8Pef/3rX7u0yKYchg7/Q1e0+DPJeeedZ1TOsaHhw4e7CAFlCV03xxxzjH366aeu9U45RVieCAHlAedRnrFM5YxW5j333JPTptAb96L1ueyyyxrlxjXXXGNHHXWUbbXVVu6amdKSKd1sk02ZRboFiIITiYRTLArzhQnL06ZNs++++86o5VPj/uqrr+zaa69ll5166qmutk8hRWFFLe6OO+5w+/hHXH3NNdd0jpVWwdNPP+32HXzwwS7Gj0ESJqW/5+6773b7fvjhB3ce911ttdXctmz/CJXU19fbe++9Z2uvvbZVVVW5a1Fg4sDJEwUlDhcHedlll7l8/OpXv3L5OPvss92lKUAvvvhidy3S9NBDD7ll0oAccsgh7h44S9LL9enzkrQlMGnSJFdh6tmzp9uJ80PvcFtllVVc4eS5U+F49tlnnV1ttNFGNnr0aLv66qvdeVQ6pkyZYuutt54NGTLEbSPcufrqq9uMGTOcftAblR9shsoJzpZrUDAOGDCgU3rLx6YoVAn1cv899tjD5fuqq65y67KptraRawtlC32xPK/pcuGFF7oK6dJLL+3Kk7Fjx6YcJTaEwxs0aJBzjpQFsEeuv/56e+qpp1zZtM0229jbb7/tnFp7NsV+yhUqwQi/rGN/udLiDs7wTzYVYQdIIYNTo3CiT4UaD4MPqFl5wfnhPCi4qGFR2CAYFIUNRnjBBRc44/RhKvbj/GjpUQgiL7/8srsXjpGC7Pzzz3eGXFFRkaqRcRwhsFtvvdXdj/X2hFYBfUb77LOPO/S3v/2tHX/88cYvwn15QGgt0N+DgZNP8vjtt9/aiy++6I6j9UlaMXIvPJS0iimEGdjz97//3e0iJCeZRwBnRauH1o/XAawQar+0xG6++WbHfuWVV3aVCdjTakJoOR100EGu79m3JN2O4B8VGx/qpOXN+WHZfffd3SqFFGEtWpIUdguqt47YlO+rpKWw+eab2wYbbODS8uSTT8qmwkrqwDIVCcoZKtdUKMJCBYoygrEJhx12mNXU1Di74hiiB9gXLcfrrrvOll9++dSptNZ45ok8UcHi+nT1tGdTRKQogygX6B4iNEvkgshErrSkbpxlIc42FdkQqG/tUSCdeeaZqdoTrULCnwjG4AUHSI2GVhU1blp/XnBoCPsQDBOhNo5QM/POEwfohZod4VJfc6Mll88ACh4OpF+/fu4Xpxxep/bWt29f17rgPrQMvRCGo+BEcNiITzfL1DYRHgIcpJd33nnHL8b+F56LLrqoCznvvPPONnLkSFeZwqYoQMIDS9A7FazXX3/dhcupvd93333uD537fr+OQsUhom9afoTHENLg9dNZvXXEprATHDytDJyhP4eWoWyqoxqcdxxOjcolFVn+vBDu9uXC/vvv7ze7MDjPLtEAHyFgp9eBP5AQO8ewnePT9/vjwr90neAksVEqM5RzO+ywg7tntrSEz8+27O+dq5wqVZuKrAP0oylp5fDn5eGHH045QL8t/Nu/f3+3Gh6qzLUwFlpY2cQ7tnCHN/dF8TjXQgl9fzg/RhJSE8PZE35DCHtSiNGPsOqqq7ZKwmKLLebWKcQZ5IHwIPXp08ct6585B0SEIF2odCCERb18/PHHbhGeRAvoV1liiSVcBIBKF7V1Wov5yC9+8QsXtsYZcS101hV6I6yOzRP+J7JAP6cvvGVT+Whw3rH0H2MD4e4FHCMVLJwHkSaWqWz4bbTKaO1nEipTOD+6N+g33HPPPd16pmPTtxFZwAFecskl7l5U6iif/H3T05J+fmfXS9WmfmpCdZZMAc5jKDCtOQYf4AD8H7VqWnG815VNMAQKsenTp7tQJWEgXqXwDjXbeRQMPhSBcdGBTQttxIgR2U5ZKNt9KxQDo5+STmyE/PsBPxg1+fBhTvbjEDF8nONdd93l+jwZou2dJ8dIshOgH5cQKZWHiy66yLWoabEtF0QO6O9j0AuOw7cSqUCli3dmVMrCFSd/HKFXhNYmA3CQrtCbd/CE7Bh0ge0gsimHoVP/GOhE2RIWdIluqWS8++67bjAUTg2hXx7edHkwiIaIghff0iJMSlmDnmidI+3ZFC1Ayinu622J83Klhf0LKqVqU5F0gD78yUjJsOy0005u1e8Ph0DDxzEClBo3cXEGsfCOFn1+Xrwh+1+uQ0uPkVkUggwmIVSEw+VaSLZ7+Wtm+vXXz7TPbyMsRhyf0Bid46QbwcmTFgbGEN4gH/7BoTAmHEtol20M8rn88svdMn0+kvYJMOKT/lbes2NwEYNhYIhQ2FEQMXCEyhPRAVqAXrwt0DJARxRkOJp0YWSyr+B4281Hb95+/P24vt+Wfq/wOvbO+2sMlrr00ktThSrvwMqmwqQ6vkwo3Xe9+LMoG1ZccUX37FKJYgzBXnvt5Xb/8Y9/dM8jrTUGwOC0vDB4jf5lBtYQHaC1yDPNuIf2bIpr+P7oHXfc0V/SlVPZ0pI6KFjw9iObmk8lAF+yEtTwk/zlK0HBkQz6bfI9bYGOJ51BLbDNNYLCORm0el0+gtplMgirJYNWXqvjgtc5kkFts9U2rXSMQPDKTBJ+mQSmHeEaFFzJIPyV6RI5txVab9gw+UsX2VQ6kQVbDyqo7hkNXyUYoZ4MnFuSsgQdbLnllslgRG74kGQQBm21Hl7prE1lSkv4ugu6XGo2lQBIuIag5egQINzLMHZqbQzfJ2RHbZF3CSUi0BkCsqnOUMv/HN/tQoSGQVcUs0cccYQbCJX/1aJ9RjHblBxgtG3LvQ5B+JMBG0Et0oVI/KitiCddyYsoAV6xkU0VXjlUVAmx8yGPXXfd1f7v//6v8DftpjsUq03JAXaTwei2IiACIiAC3UsgkoNguhdJ7rsztDnbbAC5z9ReEWhNgIFPDIeXiMCCEODTj3zRSpI/ATnAPJnxLhCjBiUisKAE+EoIoSOJCCwIAV6j4JugkvwJyAHmz0xniIAIiIAIlAABOcASUKKyIAIiIAIikD8BOcD8mekMERABERCBEiAgB1gCSlQWREAEREAE8icgB5g/M50hAiIgAiJQAgTkAEtAicqCCIiACIhA/gTkAPNnpjNEQAREQARKgIAcYAkoUVkQAREQARHIn4AcYP7MdIYIiIAIiEAJEJADLAElKgsiIAIiIAL5E5ADzJ+ZzhABERABESgBAnKAJaBEZUEEREAERCB/AnKA+TPTGSIgAiIgAiVAQA6wBJSoLIiACIiACORPQA4wf2Y6QwREQAREoAQIyAGWgBKVBREQAREQgfwJyAHmz0xniIAIiIAIlAABOcASUKKyIAIiIAIikD8BOcD8mekMERABERCBEiAgB1gCSlQWREAEREAE8icgB5g/M50hAiIgAiJQAgQqSiAPXZ6FCRMm2L333tvl99UNS4vA999/b88//7yNHz++tDKm3HQpgenTp1tDQ0OX3rNUbiYHmKcmN910U5s0aZJNmzYtzzN1uAi0JjBx4kQbO3asVVdXt96hNRHIg0AymbRjjz02jzN0qCeQCOAl/Yp+RUAEuo7AyJEjbbvttrO//vWvXXdT3UkERCBFQH2AKRRaEAEREAERiBMBOcA4aVt5FQEREAERSBGQA0yh0IIIiIAIiECcCMgBxknbyqsIiIAIiECKgBxgCoUWREAEREAE4kRADjBO2lZeRUAEREAEUgTkAFMotCACIiACIhAnAnKAcdK28ioCIiACIpAiIAeYQqEFERABERCBOBGQA4yTtpVXERABERCBFAE5wBQKLYiACIiACMSJgBxgnLStvIqACIiACKQI6GPYKRRaEIHCE7j55pvthhtuaHOjsrIye/zxx628vLzNPm0QAREoDAG1AAvDVVcVgYwEhg4dai0tLW32VVRUyPm1oaINIlBYAnKAheWrq4tAKwIbbrhhG0fHjGRbbbVVq+O0IgIiUHgCcoCFZ6w7iEArAsOHD7f0aTiPPvroVsdoRQREoPAE5AALz1h3EIFWBE455ZTUOo6wf//+qXUtiIAIdB0BOcCuY607iYAjMGDAAKuqqkq1An/zm9+IjAiIQDcQqOiGe+qWIrDQCKSHEhfahQt8oc0228yeeuopSyQStvvuu6ecYYFvu9AvT/olIlCsBPQaRLFqLsbp9k4v/BteLhY0I0eOtMGDB9uNN95YLEl2DpvE4vj8n0+8nKEnod9iISAHWCyaUjpTrSScHX+8TuD//HoxOcLJkye7/r/q6urIa9c7O//Le4v+z2/zDtD/Rj5TSmDsCcgBxt4Eog8g7NRYbm5uTv01NTXZjIakfTLH7IcGs5lNZi3BMZKFT6BHecIGViZt8eqErdAr6V7n4P1F/rwz5BeRE1z4/HXFhU9AfYALn6muuBAJeOfnW3o4v8bGRvc3dkbSnvyhwj6s67EQ76hLdZTA8Jp6++0StbZEzwqrrKx0f+hLTrCjBHVcdxNQC7C7NaD7ZyUQdn44Plp7DQ0N9uH0Rrt3SrV9Wl+V9Vzt6BoCldZiOw+stW0GJa0mCOXiCH2L0IdGuyYluosI5E9ADjB/ZjqjiwjgAGn5+VZfXV2djZ7abDdP7W0NSb3B00Vq6NBthveotf2WbLBFe9e4VzzkBDuETQd1MwGVIt2sAN0+MwGcn3eAhDxxfuN+rLfrp/SR88uMrFu3vl9XY1dPqLLZc+a6VjqVFt+C79aE6eYikIOAHGAOONrVPQR8wUnrj7BnfX29TZ5Va1dN7hME3PTeWfdopf27flRfY/dNLrPa2tpWTtDrs/0r6AgR6FoCcoBdy1t36yCB9NDnw1MrbUaysoNn67DuIvDs7N42cVa9c4BUXtCjRASiSkAOMKqaiXm6aDUQRmPQy/dzGmx0Xb+YEymO7DdbMK/htCoXsvYOEF2qFVgc+otbKuUA46bxiOfXF5bh8Oe7MxX2jLjaWiVvXGMvF7am71Z9ga3QaCViBOQAI6YQJcdSg1/8aw963aG4rIJQ9eTaJveuJg5QYdDi0l+cUisHGCdtF0leaQX6FiAh0Onq+ysSzf2UzMl1lnKAvlX/014tiUA0CMgBRkMPSkWIgHeAtB5oBU5v0eCXEJ6iWJzaWOZ0R0UGfSL+tygyoETGgoAcYCzUXHyZDDvB4ku9UpxMtv5QuZyfbCKKBPQt0ChqJcZpoqD0f7Qe1H9UnMYwT3eJlC6LMxdKdakTUAuw1DVcpPkLO0G1HopPiejMhz+lv+LTX1xSLAcYF00rnyLQDQS88/O/3ZAE3VIEshKQA8yKRjtEQAQWhICc3oLQ07ldQUAOsCso6x4iIAIiIAKRIyAHGDmVKEEiIAIiIAJdQUAOsCso6x4iIAIiIAKRIyAHGDmVKEEiIAIiIAJdQUAOsCso6x6RJdD8w0Rr+uajNn8tc2fmleb6F++yOVccbi1zpud1XvrBzZO+cNdpGPN0+i63Pueqo6zu0f+65cb3Rlnzt+MzHqeNIiAC7RPQi/DtM9IRJUyg7q5zrOW7T9vksHrkAVa14a5ttmfb0Bw40ZZJn5s11Jv1ynZU+9uTs39013HXynB4y4TA4TXUWvCSndXdfY4l+g+y3kden+FIbRIBEWiPgBxge4S0PxYEeux2tFl1z1Rey5dcMbUcyYWyMqv53T8t0XeRSCZPiRKBYiAgB1gMWlIaC06gYsW1LdGzb6v71D10qTW++6yVL72KNX0x1hIVlVa9xT7W9Mkb1vTl+1bWZ6BVbb2fVa2xWeq8uqeus6Zxr1mirNyqtzvQqtYa6fYRZq27/yJr+WGCJap7WfX2B1vVz7Zw+xpG32f1z94cLCeCe62cuhYLTV9/aLX/O8OStbOsYtUNW+2rDVqv5UuuYD3/cKbVPnCxNb33gpUPGWFNH79uZUFeqrfe1ypHbO0+R1Z/3wXW+P6oII/9rHzZVa1p/OtWs/dJVjF0RKtrakUE4kRAfYBx0rbympVA3XO3Wd1TN7i/hjcedce1zJoWzOlTb01ffWAVK29gyaYGq3/yWmue/JVVrraxtcz6weofubzVNVsCx1ix0rrBsfVW99AlLlTZMvMHq73+OEv+OCk4bxOz5iarCxwSYcxk3Ryrf+Ka4PhGK19pbWue+Emr69XecZbZ3BlWsfwa1jL5y1b7rG62tcyc6rYl56e1OQiRVg7fxFqCc+oevdLtaxr/mnPkVlZhZYsuZU3vv+jylWwI5iySiECMCagFGGPlK+s/EWh6/aHUCv1qVetsn1rvdehlVh44jrnXHWPNgTOkxVU+eHlLBg6Q9Zag385Lzb5nW/mgZazuiWutcfS91vT5O9Yy5Wvn9CrW3dFdNzFwSWscdbs1ffqWc6Kc22OboCUZ9Dk2Bi2zuttOc5dz1w2uXRY4P+6ZDKaHmn36Tv5WGX977v9vKx842JJzZlgz957+vTW9+5w7tudBF1n5YksHabsmSNt9Gc/XRhGIEwE5wDhpW3nNSqD3sf9rEwL1ByfK5z8mPXrP29Rj/igXvx603rwkKqvdIi025wA/eTPl5HCyYUdLyywZtOKQsmVXc7+pewVrzZ+/67ZVBCFYJFFe7n5z/SNM66RmXlqTQQu2+dtxlkwE4dXA+SGJiir3q38iEHcCcoBxtwDlf+ESCObBQ3wos3yZwHl9/ZE1B9sqN9jFKtfa1u0nDMoAlsag386CkGTy+6CVuNRK8/bN/18+3yk2T/mm1fZ8V8qWGGrJIFRKKLZMg2byxafjS5iAHGAJK1dZ6ziB+hduDwan/DQKtGLl9bOenMi6x6z29jOCwSobWcNLd7ujKlZazxIDlrDGVx+wxjcfs0TQekzW11rjaw9az/3Ocf11DcF7fbzbR39e49tPpK5e1m8xs2DQStO4V6zusavcAJrUzjwWKlff3JqD1iYhXAb70MeZKw95XFqHikBRE5ADLGr1KfELiwAOqpUErxkwKtNJED6cv+B+kvPX/M+89XnHJOvnWsNzt7pdVZvsYYnKKqsIRmpW73iY1QdOzu8rH7KmlQ0e6kaWVq77S2t8/WFrCEaCJvou6i/rfnsEo0Vr7znPOVCrqnGhzPABiUTucWyJIB8Vq2xgZUsNM94hbHzrcTdCNLmAL+yH06BlEShWAolgypL057lY86J0lwABJlFtamqy2tpamzFjhk2dOtUumD3EZpf1KJrcMXglEfQPpvrjQiknDJno1b9Nfx59dYw4TX8Vg1N5RBlwU5bmHEOXzbnI4JnGNx62yp9tGcRhq23OZYdactpE63XMrVYWpKUQsq19YZsvmrCBAwda7969raqqysqDPsxEqjJRiLvqmiKQHwG1APPjpaNFoF0CZb0HZD0mWx+cGzwzfwBN+sk4jfSWYfoxudYJodYHIdT6p2+yZEuzJZqDVy6CUGihnF+utGifCESJgBxglLShtIhAAQjwzmIi+GpM45ingtZkS/Aqxg7uvcIC3EqXFIGiIiAHWFTqUmJFoHMEKlb4ufEnEQER+IlA7h70n47TkgiIgAiIgAiUFAE5wJJSpzLTFQQY5MK3PZO1815i74p76h4iIAILn4Ac4MJnqiuWOIHG1x6y2mv+Zo0fvlTiOVX2RKC0CcgBlrZ+lTsREAEREIEsBDQIJgsYbRaBzhDgnbv6By4KWocvB7M9NAdTDw23HjsdEUw/9FrwGsINVr3l760q+CQa0ynV3nKyVayxhdXsdHjwdZZHrP75/wUfsZ5ufAGmZu+T3Qe3Z196iFnwcn0wfNOSjXXW5/g7rX7UHcF7fY8G3xidZhX/N8yqgmmXKtI+o9aZtOscEYgbAbUA46Zx5begBJjXz80hGHz9hRfPmZGB6Y74Ggsvuje+Ne9TZ41jnnTrfJqscdyrVv9wMK1S8BEAPqPWMmOK1d17vksn0xy573jOnjZvXsLgA9kNz9xkVtXDqjfe3ZqCD13X3X1uQfOki4tAqRJQC7BUNat8dQuBymCS24rgI9Zlg4dY83efBR+7ft4aP37DavY6wRIDl3BTI7UELbrmYJsFs0zwzdHam050ae2xy1+tbMBgMwbZBBPhJhsb3PZk8LmzXn+91sqDaZoaxzzttjGpbvmqG1vP5X/mpjzqlszqpiJQ5ATkAItcgUp+9AjU3nehm8QWB+dmfZg/nVLlz0daQxAGbXzxTkvOnWnlw9Z3nwbzE936eQDJEV8WbfpyrMsc0xfh/JCKNTa38sCpNn82xmqv/EvQEqyx6l/+2e3TPxEQgfwIyAHmx0tHi0BOAnX3/Ns5v5r9z7OKZVa12f/+feDsZrhzqtbezjnAhhfvmre+3i/nXYu5+4K+vx77nGZl/Refty2YLqlskSXnLYf+NwezwleuNdJ67Ha0NY593l2v7r7z3awS4bkEQ6doUQREIAsBOcAsYLRZBNoj0PTBS5acMTV1WPmwdVMfs2b6oab3R7mPWDMZLZIIHB1z87UEoVE+Sl0xdITbTl9hwzM3Bv2AlwUDZHYNZpF/N5hp/j3rfdSNbn/4X1MwuIYWZEUQ/qQ1yHXcIBkm5fUT94ZP0LIIiEBWAnKAWdFohwhkJpCcP00SYUj+vFQGIzWrRx4QjO48Zd58gHwUm1khglnfGdjC6M7KoBVY/9Clru/Pn1e96Z7BXH/fWtM7zwQfrb4ymPKozKqZSqm6xpjOKMlsuvOlerO93AzvTcE7iPwxewQhUI6ViIAI5EdA0yHlx0tHF5hAKUyHBKKWoI+vLHBO6VL36JVuMtyaA/5tFUsHs8WHJRgF2hKM9uzItEe8bpGsDe6RY+aJ8KW7elnTIXU1cd2vMwTUAuwMNZ0jAu0QyOT8Zl+0vyV/nGRliy/X1vlxvaC11xHnx6EJ5taLqPMjfRIRKAYCcoDFoCWlsSQIJHr2c+/yVW/9x5LIjzIhAsVOQA6w2DWo9BcNgV4HXlA0aVVCRSAOBPQlmDhoWXkUAREQARFoQ0AOsA0SbRABERABEYgDATnAOGhZeRQBERABEWhDQA6wDRJtEAEREAERiAMBOcA4aFl5FAEREAERaENADrANEm0QAREQARGIAwE5wDhoWXkUgW4gkJj/DdRuuLVuKQIdIiAH2CFMOqirCVB4+r++Vt/Vt9f9FpBAP2tI6Y9LyRkuIFCdXhACcoAFwaqLdpaAd3qcz3JZ8HmwPi11nb2czusmAgPKfnKAcn7dpATdtl0CcoDtItIB3UHAOz8c4PI2bz697kiH7pk/gYHNs2xAeYurvKA/OcD8GeqMriEgB9g1nHWXPAh451cefPC5oqLCViybZZUtwXx3kqIgsGLzFKc3dOcdIDqVIywK9cUqkXKAsVJ3cWQ27AArKyutV1W5rVb/dXEkPuappKKypk2xqqoq5wS9A4w5FmU/ogTkACOqmDgnyztAWhA4wOrqalsrOdlqmmvjjKUo8r5W3WfWt7rCOUB0RyteTrAoVBfLRMoBxlLt0c20D5NRaOIAcX49evSwAT2rbMuZYyyRbIlu4mOesiVqJ9laiSnWs2dPpzPvAL1OY45H2Y8gATnACCol7knyLUBaD4TSampqrFevXrZ8z6RtPH2Mlas/MHImsvjc72ybhnHWu3dv5wCpuMgBRk5NSlAaAc0HmAZEq9EggBMMO8DGxkbjb3jTj9Z/6ov2Sq9VbWqvwdFIbIxTUdHcYMNnfGQ/t8k2YMAA69OnT8oBpg+CiTEmZT2iBOQAI6qYOCfLh8z8KFBCoM3Nze4vmUwGowlnWP9Zb9kXM3vb+D4r2JTeS8UZV7fkvaZhti096wtbvf4bF57u27e/9evXz7UAabHTckd/XpfdkkjdVATaISAH2A4g7e4+AhSetCJwei0tLe7Xtwzd4Jg5c2zZWW/brB/H2NTKfja7rMbqy6utJTi+GGTcHZfZoBGb2MCV1iiG5FqltbiBSH2b59rg5GzXP9trQB8Xnqblxx+haios6McPfpETLAr1xjKRcoCxVHv0M+0LzfBgGLb5dd83WFdXZ33r623xpoaghVjrnGTSisMBjpkywQZOGW8brbRo9BUSpNBVPqrL54emF3WOjtYeg15wfCx750frD11JRCDKBOQAo6ydmKfNO0EKUyTdAVLw1gfOr6GhwZqamlyIlJaiF1qOURbyg9NYbLHFopzMVBgzzN+1wIOBLn6ULr9USvzAF+/8vA4jnUElLrYE5ABjq/riyLgvQH1/EgUryxS0tDYYGBN2fmEHGPUckjec+KBBg6KeVJc+7wDhT2iaP/Tg/9iOfuT8ikKdSmRAQA5QZhB5At4JUrD6QpjClxYHg2Nwev4v6q2+dNg4cUZPFoPA3vP3FRHv9Pj1+8kLyxIRiDoBOcCoa0jpcwR8geoLWV8A4/D8IBkOLDYHSOuJ/rNikEw68PoI7yuGvCiNIgABOUDZQdEQCBey3tH5XzIRXi6WTNFyoiVbTBLWg0+33+bX9SsCxUBADrAYtKQ0tiHgC1z/i/Pzy20OjvAG0uz7zCKczFZJK0bOrTKgFRGYT0AOUKZQEgSKuVAu5rSXhPEoE7EloBd1Yqt6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k1ADjDe+lfuRUAERCC2BOQAY6t6ZVwEREAE4k0gEUwjk4w3gvxyX1tbazfccIPV1dXld6KOFoE0Ao899pgNGTLEhg0blrZHqyLQcQItLS226aab2jrrrNPxk3SkI6DpkPI0hFdffdU+++wzGzlyZJ5n6nARaE3g008/tdVXX92GDh3aeofWRCAPAlOnTrWrrrpKDjAPZv5QOUBPIo/fxRdf3Lbeeus8ztChItCWwOjRo23EiBG23nrrtd2pLSLQQQJfffWVjRo1qoNH67AwAfUBhmloWQREQAREIDYE5ABjo2plVAREQAREIExADjBMQ8siIAIiIAKxISAHGBtVK6MiIAIiIAJhAnKAYRpaFgEREAERiA0BOcDYqFoZFQEREAERCBOQAwzT0LIIiIAIiEBsCMgBxkbVyqgIiIAIiECYgBxgmIaWRUAEREAEYkNADjA2qlZGRUAEREAEwgT0KbQwjRzLfPy6vr7e5s6daw0NDTZjxgwrLy+33r175zhLu0SgLYFZs2YZHzBubGy0OXPmOFvq0aOHVVdXtz1YW0QgCwHKIT7Ojz01NTU5O+LQfv36ZTlDm9MJaDaIdCIZ1imoDj74YJsyZYoruJhAA+dXWVlp5513nvuif4bTtEkE2hD46KOP7Pjjj3d21NzcbIlEwsrKymzJJZe0//73v22O1wYRyEYAO3r//ffdbmypoqLC+D322GPd7BDZztP2nwioBfgTi6xLOLpFF13Uvvzyy1bHVFVVyfm1IqKV9ggw8wPRBCpVYRk8eHB4Vcsi0C6BlVde2d5++23X+uNgIlS9evWy5Zdfvt1zdcA8AuoD7KAl7LTTTlZTU9Pq6M0226zVulZEoD0CVJp+/vOftzqsZ8+etssuu7TaphURaI/ANttsY9hTWKioL7300uFNWs5BQA4wB5zwro022siFrfw2Cq1f/vKXflW/ItBhAjvvvLOrqfsTCF2tueaaflW/ItAhAksssYQxNZsXumWoqEs6TkAOsOOsbPPNN08dzYCF4cOHp9a1IAIdJcD8f/QjI/QBUpOXiEBnCBA58K1AKlJbbbVVZy4T23PkAPNQPS0+Rn1SaGlG+DzA6dA2BHxlirD69ttv32a/NohARwiEHd6wYcM0Kr0j0ELHyAGGYLS3uOqqqxrD1flTodUeLe3PRWDHHXd0fcoDBgywZZddNteh2icCWQlQFv3sZz9zrcBdd90163HakZmAHGBmLlm3MvClf//+ttRSS2U9RjtEoD0CK620knOAiiS0R0r72yOA4+NVmk033bS9Q7U/jYDeA0wDolUREAEREIF4EFALMB56Vi5FQAREQATSCMgBpgHRqgiIgAiIQDwIyAHGQ8/KpQiIgAiIQBoBOcA0IFoVAREQARGIB4Euc4Cffvppm29pdhbxm2++aQceeKDxuzCFr6t/+OGHbf4+//zzvG7Dtx5J34033pjXeZkO/vOf/2yXXHJJpl12wQUX2BFHHOH2TZ061e6///4235jMeGKRbEzXx9dff71QUs53OGEFs0wybtw4e+655zLtyntbeh68feVrU9z4tNNOsxNOOCHvNKSfELab9H2PPvqos11Yt8cp/dworn/yySfGX1iYgQM9/Pjjj+HNBV2WTR1oUbSpLvkY9g8//GAHHXSQM7AHH3yw1WegOmN1EydOtM8++8y+/fZbW3vttTtziYznfPHFF3b44Ye32ce7No888kib7dk2UOiRvj59+mQ7pMPbeXCY8iSTvPPOOwYL5KabbnJp5KPK66+/fqbDi25bJn3wsV+Gex911FFu6HdnMjVmzBhXqUBHRx99tGGTVNC4JnLGGWfYd9995+7D56UWRDLlgevla1OcE/7wMeudlbDdpF8DDnCZPn26TZo0qRWn9GOLYf3QQw91nzBEpxtssIFL8gsvvGDnn3++/e53v7N99923INmQTf2ENco21SUO8M4770zRuO+++2yfffZJrUdxgS/2hx+M9I9gRzHNBxxwgHshdp111oli8hYoTV4fVKT+97//2WOPPea+eMEUVZ2RtdZay/7xj3+YZ/Xwww+7Qj/sAGkdLqjzC6fN58FvKwabSufk016Mv2eeeabde++9qc+GFToPsqnMhKNmU13iAB9//HH30i8hFWpG3gE+88wzribGlwzeffddV1PbYYcdUq2wr776yk4++WSbMGGCm46I797ttddeKbK0yq666iq3zmfKqO0hXIsaHq0jatp/+MMfbI899nCTj/JLK4na7RprrGH/+te/3Dnhf4MGDbINN9wwvCl17jLLLGOTJ092k1CyvOeee9oVV1zhJspdd9117cQTT0ydRwuVUCjhLj5TdM4556Q+VcQ8gs8//7ybXJdvilJDpXXD/F7kmUkuN9lkk9S1WCBk8/e//92FkrketXQvhPRuv/1294I+vMh/Nq60Kkknk/ryke+PP/7YFllkkayhVn+P7voN62O11VazP/3pT/b666+7ORqz6Zq0vvLKK3bttdc6Xny44De/+Y37hN348eMdH9Zfeukl5/w4nq/7XHrppXbZZZe5FuENN9zgbHXEiBFOPxyDvrEpWtyE0c4991wXiUB3RA/Cn6bieC/hPPhthOKwx47YlHeYRBfQ3WuvvWZ8RYaCfcUVV3SXzGZTueyGiXmxvRdffNEWW2yxVg4izImPLPPsrbDCCi6/2Ccf8D711FOd3Ubdpoii/Oc//7G//e1vHn/qd/bs2Y7pBx984Ob43G677VJlEOXVlVde6b7duu222xpl2W677WZUOHlWzzrrLPv+++/d88P2X//6184uaUUjxW5T5B9bp6zBhrGVIUOG5CyLismmCt4HOHbsWMPAtthiCxeCoBbv+z8oAJjDitAOX1jho64U5DgYJnakQMGJ8PFgvnRw9dVXt4rnf/PNN+48PgJ7zz33uD4dHBtGTgjLz+DARKM4U67J/VhGScynlUlI3zXXXJP6w5H6c+lPYPJSnCjXodChIGD91VdfbdV3RCuCYzEcCoibb77Z3e7CCy90DxLTlnAujG677Ta3jwIFx8R2z8mnkYKZkBrX7Nu3r3PKfh/nkDf6H3Nx5fjTTz/dGfAqq6zi2MMbvRSDvPfeey6ZfAU/l67nzp3rKhLYARUunAC68mxgBTP/fVcu+qtf/coGDhzo7IjjcDA4NpwN9sJktnCi8sHvkUce6eyM73oyIzcVHI7LJAvLprg+dkWliTScffbZ7na5bCqX3VAJIyTIx92pXPC8eYGR5+TtHyeBbWKDPLdPPPGEOzzKNsX3e6ngETkgHJcuhxxyiGFXVGDRP2UQlRuepYsvvtgxoLL80EMPueVp06a58gD9owMqMeifyjjrpWJTo0ePdvnH5uh2wNF7eysVmyq4AyRkhey+++6u5sSyL+xZRqhNMYsxLTWEWjlOgUIIo6SWy0AQHAY1fi/+PGL5yKhRo4xWJQrbb7/93KABavRI+J48vA888ECrMKc7aP4/HALp9n/hh4YHAeX7liNOlMEJ3jBefvnl1KUoUHBot956q/uANi0S5Omnn3azNx933HF22GGHudYxhRCFNH8UMBTWtF7C8sYbb6RaH9wvPBVK+Di/7PmEuc6cOdNVMEgbNWIedioXURbyTc2agoU0Iwz+yaVrCiTsgEoVjv6iiy5yfXwUamHhmp4jNsNn7sLCTA1chzQ89dRTbhc1fRwH9+B87A/n6Y8Ln++XF5ZNMTkzFTocHhOfMrCAdGSzKe6fy26effZZl0QqfAywofWTS7BNIhS0fBFsuhhsCgeNkPawoBcquLRqiNYQYUFwlrSKEeyC5x3mXtA1rSGiP1SAfP/ik08+6WyiFGyK7irkmGOOcWUzlS4q4EQhSsWmKrxCC/ELKEZqMnsCTsALhoUBeeGhRmjVINQ6fU109dVXd9uowRGSQmiWI7T8EF9oUbhRS0f8pKMYNvcPjxilRUaNN5tgzBh3WHjIEe8sfFqZFxDx6+TZi88X96eFyINGKMYXwvvvv78/1G2nRo3gnJBwHxTn0JL24S72+/yznEn8/X3a4MoACMRz9fnJdH5UtpFPJvokVEhYeMstt3Qts1y6Js84pltuucX195FP5uEjJJXPaFLC7lSEaOnQ+qFFiEO96667HB4qEPx5odZMxCJdFpZNhW2CdFAgMagnm021ZzdEJmCDfSLeZtLT79f9/n79+rlNxWJTVFR/8YtfGA7fF+xkgIo2QgsdR+eF54QuGwSnj/i8+2UqrVSKKMv8s0hLuT0pBpuiBewjUOkjj+l+yFUWFZNNFdQBEi/HOHAA1JgRlqmxtjfMnMkeEd/6whgJc4YdgDsg7Z+veRGqJFSF02Hute74eLWf840kTpkyxYVh6MuBAQ8MfQsse0bemRJaTRf6nTiXMISX8PX9tvZ+vePzxt3e8VHYH+6DC6cnl67hTQsJB8jDDGsKPkbIhp1I+HqZlnG8RAxo2WO3fhYQKmQIkQ1agQj7Ob6QEta51yEVpmw21Z7d0KdHbZ4KHpWGcMW0o/koFptixC969NzIH/2eCBx8QY8eGcHtW/2UQcwEExYqX1SK+Kg5UQmiVumV5vDx4eVisCl4wIC+P6JNvoIEG8rSXGVRMdlUQR0go64QCiEPEGPilQhq0L4wCRuHX6a5TY2L1iLhGTrkaSERpvADAvyx4V9CONTICRMRj/d9FLQaOio8IL61yTnU+jvz1X5ql1yHtGM4hHMRHiZaEzw4hE/uvvtu5xx5P4uaNQ8pYVbfCnYnBf8YQcWDdtJJJzkjxLlT8OUj9GtRePNwcz868jtT6OVzz0Idm0vX9NMQ9uJhxd4odOg7o4ae7gDhwaAF7DTTjNrYKTaI4PAQQqPYN6P96GOiZc86oVbf4nYHzv+3sGyKFhddAhRO2BUVRSpOuWwql93QN48tEFLGxqm05ivFYlNUBuhq8eFQ8umnOKNcokxabrnl7LrrrrPf//73rmLDYDIqTwwsIyzqxeuYMobBeL6LhW4bpBRsChunGwabpp+T8pcuqDvuuCNnWVRMNlXmFbqwfylsGKRCLd07P+5B7YBCnhaaD9v4Qjz8i7FSo6IPhxAU8BmZiSK8hI9nG+tcn8EzFBSMXqIlQMiB8JcP9/lffx3/67fTL8CAFf9H4ef3pf/6c8O//hgeEq5ByIA8+9GvPIC0ZAmzYFyw8qNbSTv5oDClY97nkesz+hFnTJgNp8lyLvHnpv/+5S9/cdflwcUxR1U8x2zpy6VrWv977723iwDQ74ODw2FmCk/66WSuv/761IjQ8D2Zuw8h1E4BiaA/BkHgUPngAa/60D9MmsLi87CgNsU1uZavFNKHjV7pQ0Zy2VQuu6ECxisaPKvYOc48l6Tbks9fsdgUg+18FwP5pHzByfF88hGAyy+/3C3DheeLvl3KKZ43jkF8RImKERUtxhn4lqQf/VkKNsUoaZwgtkseGeRHWUoDpGRsKlBm5CVwEMkgBJp3OgPnl/c5C/sE0k36M0nwYCWDQS9tdgUtsmSutHO9oOBtc15HNwSGnAz6jpLBw5sMav/JoG8kGVQ2Onp6JI9rj1d7iQ4qAln11N656IPzu1KCUHgSO0mXbDbFcbnsJhgRmwxasemX6/B6KdhU8GpHMmjBtcpzMPLTPaNBn1cyGBHqnpWgHzF1TNDnnwxGHKfWwwulYlOUNdmer2K3Kc0H6Op08fpHzY6+RFqovAqA0FKlr0siAp0hUIo2xYhsIk60egkzM/CDFjKtb0nhCXSFTckBFl6PkbsD78gRVma0FqNl+fiA79OIXGKVoKIgUKo2RT884U/eOWUcAf1bfsRnUSimiBPZFTYlB1jEBqKki4AIiIAIdJ5AwQbBdD5JOlMEREAEREAECk9ADrDwjHUHERABERCBCBKQA4ygUpQkERABERCBwhOQAyw8Y91BBERABEQgggTkACOoFCVJBERABESg8ATkAAvPWHcQAREQARGIIAE5wAgqRUkSAREQAREoPAE5wMIz1h1EQAREQAQiSEAOMIJKUZJEQAREQAQKT0AOsPCMdQcREAEREIEIEpADjKBSlCQREAEREIHCE5ADLDxj3UEEREAERCCCBOQAI6gUJUkEREAERKDwBOQAC89YdxABERABEYggATnACCpFSRIBERABESg8ATnAwjPWHURABERABCJIQA4wgkpRkkRABERABApPQA6w8Ix1BxEQAREQgQgSkAOMoFKUJBEQAREQgcIT+H9H9HJ+5qRnFgAAAABJRU5ErkJggg=="}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class GeMPoolingLayer(tf.keras.layers.Layer):\n    def __init__(self, p=1., eps=1e-6):\n        super().__init__()\n        self.p = p\n        self.eps = eps\n\n    def call(self, inputs: tf.Tensor, **kwargs):\n        inputs = tf.clip_by_value(\n            inputs, \n            clip_value_min=self.eps, \n            clip_value_max=tf.reduce_max(inputs)\n        )\n        inputs = tf.pow(inputs, self.p)\n        inputs = tf.reduce_mean(inputs, axis=[1, 2], keepdims=False)\n        inputs = tf.pow(inputs, 1. / self.p)\n        \n        return inputs\n    \n    def get_config(self):\n        return {\n            'p': self.p,\n            'eps': self.eps\n        }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reg = tf.keras.regularizers\n\nwith strategy.scope():\n    # backbone\n    backbone = tf.keras.applications.Xception(\n        input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),\n        weights='imagenet',\n        include_top=False\n    )\n    \n    backbone.trainable = False\n    \n    # embedding model\n    x_input = tf.keras.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3))\n    x = backbone(x_input)\n    x = GeMPoolingLayer()(x)\n    x = tf.keras.layers.Dense(EMBED_SIZE, activation='softplus', kernel_regularizer=reg.l2(), dtype='float32')(x)\n\n    embedding_model = tf.keras.models.Model(inputs=x_input, outputs=x, name=\"embedding\")\n\n    # anchor encoding\n    anchor_input = tf.keras.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3), name='anchor_input')\n    anchor_x = embedding_model(anchor_input)\n\n    # positive encoding\n    positive_input = tf.keras.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3), name='positive_input')\n    positive_x = embedding_model(positive_input)\n\n    # anchor encoding\n    negative_input = tf.keras.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3), name='negative_input')\n    negative_x = embedding_model(negative_input)\n\n    # construct model\n    model = tf.keras.models.Model(\n        inputs=[anchor_input, positive_input, negative_input], \n        outputs=[anchor_x, positive_x, negative_x]\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now define the triplet loss function. This function splits the model output into its three parts and then compares the anchor embeddings to the positive and negative embeddings. A reduce sum has also been added at the end as the batch size is likely to push more than one triplet set through on a single step.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def triplet_loss(y_true, y_pred, alpha=0.2):     \n    anchors = y_pred[0]\n    positives = y_pred[1]\n    negatives = y_pred[2]\n    \n    pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchors, positives)), axis=-1)\n    neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchors, negatives)), axis=-1)\n\n    basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), alpha)\n    loss = tf.reduce_sum(tf.maximum(basic_loss, 0))\n\n    return loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compile the model with the loss function and the adam optimizer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=RATE),\n    loss = triplet_loss\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add a couple of callbacks to catch the model if successive epochs are not converging.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"callbacks = [\n    tf.keras.callbacks.ReduceLROnPlateau(monitor='loss', patience=2, verbose=1),\n    tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5, verbose=1, restore_best_weights=True),\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally train the model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(\n    train_data,\n    epochs = EPOCHS,\n    steps_per_epoch = STEPS_PER_EPOCH,\n    callbacks = callbacks,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluate\n\nTo evaluate the model, let's begin with a simple loss curve.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.title('Model loss')\nplt.plot(history.history['loss'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we can eyeball some results. Here I'll use the five triplet sets that I displayed earlier in this notebook. I'll pass them through the model and calculate the euclidean distance between the anchor and the positive and negative embeddings.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def distance_test(anchors, positives, negatives):    \n    pos_dist = []\n    neg_dist = []\n    \n    anchor_encodings = embedding_model.predict(anchors)\n    positive_encodings = embedding_model.predict(positives)\n    negative_encodings = embedding_model.predict(negatives)\n    \n    for i in range(len(anchors)):\n        pos_dist.append(\n            distance.euclidean(anchor_encodings[i], positive_encodings[i])\n        )\n        \n        neg_dist.append(\n            distance.euclidean(anchor_encodings[i], negative_encodings[i])\n        )\n    \n    return pos_dist, neg_dist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pos_dist, neg_dist = distance_test(anchors[0:5], positives[0:5], negatives[0:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display each triplet set alongside the distance between the anchor and positive images and the anchor and the negative images. The distances are in the positive and negative image titles.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(5, 3, figsize=(15, 20))\n\nfor i in range(5):\n    axes[i, 0].set_title('Anchor')\n    axes[i, 0].imshow(anchors[i])\n\n    axes[i, 1].set_title('Positive dist: {:.2f}'.format(pos_dist[i]))\n    axes[i, 1].imshow(positives[i])\n\n    axes[i, 2].set_title('Negative dist: {:.2f}'.format(neg_dist[i]))\n    axes[i, 2].imshow(negatives[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This competition uses the embeddings created by the model to cluster an image with index images. To see how this model performs it might be worth taking one example image, getting its embeddings and seeing what images it is most similar to.\n\nBegin by getting a sample of 100 images.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image_ids = pd.read_csv(\n    '../input/landmark-retrieval-2020/train.csv',\n    nrows=100\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next get the pixels for each of the images in the sample and put them in a numpy array.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_image(img_id):    \n    chars = [char for char in img_id]\n    dir_1, dir_2, dir_3 = chars[0], chars[1], chars[2]\n    \n    image = Image.open('../input/landmark-retrieval-2020/train/' + dir_1 + '/' + dir_2 + '/' + dir_3 + '/' + img_id + '.jpg')\n    image = image.resize((IMAGE_SIZE, IMAGE_SIZE))\n    image = np.asarray(image) / 255.0\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images = [get_image(img_id) for img_id in image_ids.id]\nimages = np.array(images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feed the images into the model to get the embeddings for each one.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"embeddings = embedding_model.predict(images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now generate a matrix comparing each image against the others. This gives us a euclidean distance between all the images. The smaller this distance the more similar the model believes the images to be. If we have 100 images in the sample then we will have a matrix of shape (100,100) that contains all the distances.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"distances = distance.cdist(embeddings, embeddings, 'euclidean')\nprint(distances.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now get the indexes of the ten images that are closest to each image in the sample.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"predicted_positions = np.argpartition(distances, 10, axis=1)[:, :10]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can have a look at how an example anchor image from this sample compares against all the other images in the dataset. Let's pick an image in the sample.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"anchor_img = get_image(\n    image_ids.loc[41].id\n)\n\nplt.title('Landmark: {}'.format(image_ids.loc[40].landmark_id))\nplt.imshow(anchor_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then get the ten images closest to it including the euclidean distance. There are only four images containing this landmark so there will be some other landmarks that appear in the top ten. If the model is good though the distance for these other landmarks should be lower than the distance for the images containing the same landmark.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 3, figsize=(12, 12))\n\ntop_ten_indexes = predicted_positions[41]\n\nfor i in range(3):\n    for j in range(3):\n        img_index = top_ten_indexes[j + (i*3)]\n        \n        landmark_id = image_ids.loc[img_index].landmark_id\n        dist = distances[41, img_index]\n\n        ax[i,j].set_title('landmark: {}, dist: {:.2f}'.format(landmark_id, dist))\n        ax[i,j].imshow(\n            get_image(image_ids.loc[img_index].id)\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at an image containing a different landmark.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"anchor_img = get_image(\n    image_ids.loc[80].id\n)\n\nplt.imshow(anchor_img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(3, 3, figsize=(12, 12))\n\ntop_ten_indexes = predicted_positions[80]\n\nfor i in range(3):\n    for j in range(3):\n        img_index = top_ten_indexes[j + (i*3)]\n        \n        landmark_id = image_ids.loc[img_index].landmark_id\n        dist = distances[80, img_index]\n\n        ax[i,j].set_title('landmark: {}, dist: {:.2f}'.format(landmark_id, dist))\n        ax[i,j].imshow(\n            get_image(image_ids.loc[img_index].id)\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save model\n\nFinally save the model. The competition needs the model to accept only one image as input and output only one embedding. I'll only be saving the embedding layers of the model then.\n\nIt's worth noting that at the time of writing models trained with TPUs cannot be saved in the tf format. They instead need to be saved as h5 files.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"embedding_model.save(\n    'embedding_model.h5', \n    save_format='h5',\n    overwrite=True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I would also finish this off with the below functions to save the model into the exact format needed for a submission. However the competition does not allow notebook using TPU or the internet to be entered for submission. As such I have moved the below functions into an inference notebook where the h5 file can be loaded and inserted into the below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# embedding_model = tf.keras.models.load_model(\n#     'embedding_model.h5',\n#     custom_objects={'GeMPoolingLayer': GeMPoolingLayer}\n# )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# class MyModel(tf.keras.Model):\n#     def __init__(self):\n#         super(MyModel, self).__init__()\n#         self.model = embedding_model\n    \n#     @tf.function(input_signature=[\n#       tf.TensorSpec(shape=[None, None, 3], dtype=tf.uint8, name='input_image')\n#     ])\n    \n#     def call(self, input_image):\n#         output_tensors = {}\n        \n#         input_image = tf.cast(input_image, tf.float32) / 255.0\n#         input_image = tf.image.resize(input_image, (IMAGE_SIZE, IMAGE_SIZE)) \n                \n#         extracted_features = self.model(tf.convert_to_tensor([input_image], dtype=tf.float32))[0]\n#         output_tensors['global_descriptor'] = tf.identity(extracted_features, name='global_descriptor')\n#         return output_tensors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# m = MyModel()\n\n# served_function = m.call\n\n# tf.saved_model.save(\n#     m, \n#     export_dir=\"./model\", \n#     signatures={'serving_default': served_function}\n# )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from zipfile import ZipFile\n\n# with ZipFile('submission.zip','w') as output_zip_file:\n#     for filename in os.listdir('./model'):\n#         if os.path.isfile('./model/'+filename):\n#             output_zip_file.write('./model/'+filename, arcname=filename) \n    \n#     for filename in os.listdir('./model/variables'):\n#         if os.path.isfile('./model/variables/'+filename):\n#             output_zip_file.write('./model/variables/'+filename, arcname='variables/'+filename)\n    \n#     for filename in os.listdir('./model/assets'):\n#         if os.path.isfile('./model/assets/'+filename):\n#             output_zip_file.write('./model/assets/'+filename, arcname='assets/'+filename)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}