{"cells":[{"metadata":{"id":"NtIeFBZaG-TF"},"cell_type":"markdown","source":"# Cassava Leaf Disease Classification\n## Pipeline entrenamiento Modelo de Clasificación de imágenes\n\n#### Modelo de Clasificación para la detección de categorias a partir de imágenes."},{"metadata":{"id":"4r5ngLmHsu-g"},"cell_type":"markdown","source":"\n**Referencias:**\n\n- [Chris Deotte - KFold triple estratificado con TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/comments)\n- [The training script is using only TPU and TFRecord](https://www.kaggle.com/ludovick/baseline-tf-tpu-efficientnet-kfold-training)\n- [data augmentation](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n- [data augmentation](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease)\n- [Keras.io: Transfer learning & fine-tuning](https://keras.io/guides/transfer_learning/)\n- [Keras.io - Image classification from scratch](https://keras.io/examples/vision/image_classification_from_scratch/#run-inference-on-new-data)\n- [keras.io - EfficientNet B0 to B7](https://keras.io/api/applications/efficientnet/)\n\n**Autor**:\n- [Jaime Sendra Berenguer](https://www.jaimesendraberenguer.com/)\n\n\n\n \n \n \n \n
\n Linkedin\n \n GitHub\n \n Medium Blog\n \n Kaggle\n
"},{"metadata":{"id":"yz5tr3GZtyfK"},"cell_type":"markdown","source":"## Fase 1: Importar dependencias"},{"metadata":{"id":"oyyZ6mJtt3Cz"},"cell_type":"markdown","source":"### Importación Librerias"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null","execution_count":null,"outputs":[]},{"metadata":{"id":"jwJ3tU_aHApF","outputId":"a0d0e4b1-b1eb-4a58-e703-a76e20891ba9","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport datetime\nfrom functools import partial\nimport random\nimport re\nimport math\n\nimport matplotlib.pyplot as plt\n\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import backend as K\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.layers import Activation, Dropout, Flatten, Dense\n\nfrom tensorflow.keras.optimizers import RMSprop\nimport efficientnet.tfkeras as efn\n# from tensorflow.keras.applications import EfficientNetB0, EfficientNetB1, EfficientNetB2\n# from tensorflow.keras.applications import EfficientNetB3, EfficientNetB4, EfficientNetB5\n# from tensorflow.keras.applications import EfficientNetB6, EfficientNetB7\nEFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\n\nfrom collections import Counter\n\nfrom kaggle_datasets import KaggleDatasets\n\n\n# Load the TensorBoard notebook extension\n%load_ext tensorboard","execution_count":null,"outputs":[]},{"metadata":{"id":"SiqO6l2KuEM0","outputId":"9d3672fb-083c-437d-ea32-1814be375143","trusted":true},"cell_type":"code","source":"print(f'Versión de tensorflow: {tf.__version__}')","execution_count":null,"outputs":[]},{"metadata":{"id":"ilauUu7auGK2","outputId":"c735812f-0d8b-4032-ac1d-94ff73ab6e5b","trusted":true},"cell_type":"code","source":"from tensorflow.python.client import device_lib\ndevice_lib.list_local_devices()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Configuración entorno de desarrollo"},{"metadata":{"id":"9-4TyzO7tMq8"},"cell_type":"markdown","source":"#### RAM"},{"metadata":{"id":"SHJbHJXKtQ8G","outputId":"f1df6c36-6583-4b62-a0f5-7b9249626cf3","trusted":true},"cell_type":"code","source":"from psutil import virtual_memory\nram_gb = virtual_memory().total / 1e9\nprint('Your runtime has {:.1f} gigabytes of available RAM\\n'.format(ram_gb))\n\nif ram_gb < 20:\n print('To enable a high-RAM runtime, select the Runtime > \"Change runtime type\"')\n print('menu, and then select High-RAM in the Runtime shape dropdown. Then, ')\n print('re-execute this cell.')\nelse:\n print('You are using a high-RAM runtime!')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### TPU - GPU - CPU"},{"metadata":{"trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\" # or \"GPU\"","execution_count":null,"outputs":[]},{"metadata":{"id":"rlTciQFsd2Mz","outputId":"25c75753-29b7-4a50-f355-acc973820bec","trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n print(\"connecting to TPU...\")\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\n except ValueError:\n print(\"Could not connect to TPU\")\n tpu = None\n\n if tpu:\n try:\n print(\"initializing TPU ...\")\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"TPU initialized\")\n except _:\n print(\"failed to initialize TPU\")\n else:\n DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n print(\"Using default strategy for CPU and single GPU\")\n strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"id":"9H1uKVXauL0B"},"cell_type":"markdown","source":"#### Configuraciones del proyecto"},{"metadata":{"trusted":true},"cell_type":"code","source":"cfg = {\n \n \"version\": 1,\n\n \"smoothing\":0.00,\n# \"arch_fn\":efn,\n \"folds\": 5,\n \"eff_nets\": [4,4,4,4,4],\n \"name\": [f\"EfficientNetB{i}\" for i in [4,4,4,4,4]],\n \"resize\": [420, 420, 420, 420, 420],\n \n 'unfreeze': False,\n \"weights\": \"imagenet\",\n \"num_class\":5,\n \"seed\":99,\n \"verbose\":1,\n \"lr\": 1e-3,\n \"val_split\": 0.2,\n \"drop_connect_rate\": 0.4,\n \"suffle\": True, \n\n \"crop_size\":380,\n \"rotation\":0.0,\n \"shear\":0.0,\n \"h-zoom\":5.0,\n \"w-zoom\":5.0,\n \"h-shift\":5.0,\n \"w-shift\":5.0,\n\n \"path_models\":\"../input/cassavatpumodelsbaseline/\"\n }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZES = [32]*cfg[\"folds\"]\nEPOCHS = [12]*cfg[\"folds\"]\nIMG_SIZES = cfg[\"resize\"]\nEFF_NETS = cfg[\"eff_nets\"]\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 11\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/cfg[\"folds\"]]*cfg[\"folds\"]\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### SEMILLA ALEATORIA"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.environ['PYTHONHASHSEED']=str(cfg[\"seed\"])\nos.environ['TF_CUDNN_DETERMINISTIC'] = '1' # new flag present in tf 2.0+\nrandom.seed(cfg[\"seed\"])\nnp.random.seed(cfg[\"seed\"])\ntf.random.set_seed(cfg[\"seed\"])","execution_count":null,"outputs":[]},{"metadata":{"id":"vyaBjxM6uTol"},"cell_type":"markdown","source":"## Fase 2: Preprocesado de Datos"},{"metadata":{},"cell_type":"markdown","source":"#### Load data"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path(\"cassava-leaf-disease-classification\")\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/*.tfrec')))\nfiles_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test_tfrecords/*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Utilidades TFRecord"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'target' : tf.io.FixedLenFeature([], tf.int64),\n 'image_name' : tf.io.FixedLenFeature([], tf.string),\n\n } \n example = tf.io.parse_single_example(example, tfrec_format)\n \n label = tf.cast(example['target'], tf.int32)\n target = tf.reshape(tf.one_hot([label], depth=cfg[\"num_class\"], axis=-1), [-1])\n \n return example['image'], target\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n tfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'image_name' : tf.io.FixedLenFeature([], tf.string),\n }\n example = tf.io.parse_single_example(example, tfrec_format)\n \n return example['image'], example['image_name'] if return_image_name else 0\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_image(img, augment=True, dim=256):\n global cfg\n \n img = tf.image.decode_jpeg(img, channels=3)\n img = tf.cast(img, tf.float32) / 255.0\n\n if augment:\n if dim!=cfg['crop_size']: \n img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n img = transform(img, cfg, dim=cfg['crop_size'])\n img = tf.image.random_flip_left_right(img)\n img = tf.image.random_saturation(img, 0.7, 1.3)\n img = tf.image.random_contrast(img, 0.8, 1.2)\n img = tf.image.random_brightness(img, 0.1)\n# elif dim!=cfg['crop_size']:\n# img = tf.image.central_crop(img, cfg['crop_size']/dim)\n\n# img = tf.reshape(img, [cfg['crop_size'],cfg['crop_size'], 3])\n img = tf.image.resize(img, [cfg['crop_size'],cfg['crop_size']])\n\n return img\n","execution_count":null,"outputs":[]},{"metadata":{"id":"euoKdETYubka"},"cell_type":"markdown","source":"#### Generación y preprocesado del dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n labeled=True, return_image_names=True, batch_size=16, dim=256):\n \n ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n ds = ds.cache()\n \n if repeat:\n ds = ds.repeat()\n \n if shuffle: \n ds = ds.shuffle(1024*8)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n \n if labeled: \n ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n else:\n ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n num_parallel_calls=AUTO) \n \n ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, dim=dim), \n imgname_or_label), \n num_parallel_calls=AUTO)\n \n ds = ds.batch(batch_size * REPLICAS)\n ds = ds.prefetch(AUTO)\n return ds\n\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"id":"kYdPgW9zyo2C"},"cell_type":"markdown","source":"#### Data Augmentation"},{"metadata":{"id":"LGo2de7DzDDO"},"cell_type":"markdown","source":"Cuando no se tiene un gran conjunto de datos de imágenes, es una buena práctica introducir artificialmente la diversidad de muestras aplicando transformaciones aleatorias pero realistas a las imágenes de entrenamiento, como volteos horizontales aleatorios o pequeñas rotaciones aleatorias. Esto ayuda a exponer el modelo a diferentes aspectos de los datos de entrenamiento mientras ralentiza el sobreajuste."},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n # returns 3x3 transformmatrix which transforms indicies\n \n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n shear = math.pi * shear / 180.\n\n def get_3x3_mat(lst):\n return tf.reshape(tf.concat([lst],axis=0), [3,3])\n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n \n rotation_matrix = get_3x3_mat([c1, s1, zero, \n -s1, c1, zero, \n zero, zero, one]) \n # SHEAR MATRIX\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear) \n \n shear_matrix = get_3x3_mat([one, s2, zero, \n zero, c2, zero, \n zero, zero, one]) \n # ZOOM MATRIX\n zoom_matrix = get_3x3_mat([one/height_zoom, zero, zero, \n zero, one/width_zoom, zero, \n zero, zero, one]) \n # SHIFT MATRIX\n shift_matrix = get_3x3_mat([one, zero, height_shift, \n zero, one, width_shift, \n zero, zero, one])\n \n return K.dot(K.dot(rotation_matrix, shear_matrix), \n K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image, cfg, dim=128): \n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated, sheared, zoomed, and shifted\n DIM = dim\n ROT_ = cfg[\"rotation\"]\n SHR_ = cfg[\"shear\"]\n HZOOM_ = cfg[\"h-zoom\"]\n WZOOM_ = cfg[\"w-zoom\"]\n HSHIFT_ = cfg[\"h-shift\"]\n WSHIFT_ = cfg[\"w-shift\"]\n \n \n \n XDIM = DIM%2 #fix for size 331\n \n rot = ROT_ * tf.random.normal([1], dtype='float32')\n shr = SHR_ * tf.random.normal([1], dtype='float32') \n h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n # GET TRANSFORMATION MATRIX\n m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n y = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n z = tf.ones([DIM*DIM], dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n idx2 = K.cast(idx2, dtype='int32')\n idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"id":"y1jqZTHxzdzz"},"cell_type":"markdown","source":"## Fase 3: Construccion del modelo"},{"metadata":{"id":"qbp_2nazz5uL"},"cell_type":"markdown","source":"#### Construcción del Modelo"},{"metadata":{"id":"MAVS-rk-0gNy"},"cell_type":"markdown","source":"**Image classification via fine-tuning with EfficientNet**\n\nEfficientNet, presentado por primera vez en Tan y Le, 2019 se encuentra entre los modelos más eficientes (es decir, que requiere menos FLOPS para la inferencia) que alcanza una precisión de _State-of-the-Art_ tanto en imagenet como en tareas de _transfer-learning_ de clasificación de imágenes.\n\nPara los modelos B0 a B7, las formas de entrada son diferentes. Aquí hay una lista de la forma de entrada esperada para cada modelo:\n![EfficientNet-B0-B7.PNG](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA3YAAAGACAYAAADh+fCgAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAFe3SURBVHhe7d3PSxvd////77/iQrCruMriDZZugguDUKVQhULKtVC6ULoIveC60A8VBcVFFi6EgItAFoFAkcDFRUEkIOpCyaISKCVQtFCIIEQQZiE8v+fMnElOkklM2vhjrtf9BgeaycxkJkKefcycc+b/EwAAAABAqBHsAAAAACDkCHYAAAAAEHIEOwAAAAAIOYIdAAAAAIRcPdj9v//3/2g0Go1Go9FoNBqNFoLWimBHo9FoNBqNRqPRaCFrrdqCHQAAeDqozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsHvSSrI6/EyGdPsjL1WztD+D2AcA4Kn5L9Tman7Oq0+qrZ6YhffiUnYXoupzRiW+VRLHLAWA/5LBBLuTjfoPc1OLTkj83Ypk9ipSuzXrog8EOwBAMIJdO6d2LbWrgNhWLchbvxa+2JYzsxgA/kvuN9hZLbbOFbL+EewAAMEIdi2uCjLv7mtDTs2iBkfO0gsSf7kgq3sXZhkA/LcMPNjF1wpyeHDstl31Ixozy4eGpyRTMeujRwQ7AEAwgl2zWmHB7Cso2AHAf9/Ag93b/KVZ6DnbitXfa/rRvr2Wb3s7spiYkmjEe38oMibxdxuy+7X93l71SK87IRGzL93NczaZklyp+fPkpiK76wsSHxut73M6uSOHP837XZyue/t+vl1WH3gs6eSM+bxRib5ekZx7XI58y6/ItNn/yNiMLKaPpRrQ1dT5XnT34Z+ft25Rzm/MCrbbSzncUkE4aj5vbkP2v19I7g9v26BQ1vqdRMYXZLVQbun2SrADgP+iu2pzPTQtF90eM+cFv3bNSc6uib3Wzdb1dK2aXJClbHtdq5bysvqupT7lS221slOw8+tx27EG1TT1/4mz/N8S9/8vYbd63eteC3uu1z/zpktnUv6tqeM/2Jb5cT12Ty3T39tyXr4F1XgAeAD3Huy+Zabq7zUFu3qXiaA2IZulRrir/ZOUkcD1VAjbUiHMd6N+uMeD19P7XD3p3hm0XkhezsnboP1E5mT140z7ctViOgxaHPWdNO5WtrTxDTlt+uG/ln/f+4XSapEZmX5p/t1SiM5VMez0nYzM5eW8XjwJdgDwX9RzsHuxLbtNNcMKS73WzduybHVcb0F2r7zVtN7r04CCXT1sBbQegl1f9dr6rLdzwec5Mleg1gJ4FPca7JwfRVnyC0FkRQ5bctVZdkXShbKcX12L4zhSPbB+XN9/lpq7lg49ZpkuHt8dte611L6XJLe+Irnv7kqKI4cf/XA0J+kv197iq+NG0XqVlXNvaaBGIVFNhbitg4pUvx/L5ms7dI1KfC0vZ98v5DT7d+N47cHYzrEs1e9CzsimPsfvZdlNzdSLwIi5gur6si3P/f1EFyS9V5LTk6Jk/pzwlulmF6LvWZk2y58vf5ZzvaNbR77l/CIzKksH/t4JdgDwX9RzsPObeyduQzbXCvLNXaOPullu1Kl46liqqmY7V5dez5vt40Y9s+rT0PjfklF19Lxy3FTPpq1xGYO5Y+dI7aosmTf+Niuyr/5fUdOtdkct7Ldet4TI2J9ZOaxcyvnJtryt3zFk6AmAxzHwYBfYoglVMOo/i11UJNN2h8oOdhPyId/a1dBwVIj0P2/t2Cz06Dt+3vatBaKZHezmC6bAaUfW+b3ckW/1z7+QXL2QeN0yNKe4Ul9/OmsP0r6W3YX29c+2/e6qKpAVre/JLjhWIWrcBU1I7odZ6CrL1guz/nrJLCPYAcB/UV/Brq2niNJP3bSC3cjrVMfhDY36ZF9gVPQdP78+WRdCBxLsXJeNoQuBY+yCt+u3XjcFO30e1v9HOvZQAoAH8iDBzhv7VWlc8dLcMXZZWXo3JXG/f7rdrB/etq6YKii29dWvZCVur9OhdfuxbRSSlqtt9g95PTB5Dtf8bRrFpxHU2j+v/YffDq5zstuUvKxCVf8+HNlf9tfv0urrE+wA4L+on2C3+I91sdLXT91s64o5KvH3O7JfsffbXM+aA5lduxrvPXaw669eK/b/B+w7eUqncwGAhzLwYDebKXvdH35W5FQFtw+TjUHWjat36sf/TyvM6efdvVyQpdSKvPWv6DX9YOsJSwIGRkcXZNe/Y2Udw8iYCosvg1u6OZc16VhIugS7oG06F6SgH/7uxageHOvfh71+VGIB5+i2vz6b9Ql2APBf1E+wCwwa/dbN6rFsJVovxKqAl/IfZ9S9ngXVxscOdv3Va6XL/wcIdgAe28CDXevkKU3jx/wfQfsq4R9ZayC19cMcFEIcb1xbU8B71d7/353V8hd0/IHvM9h1647x+3fs7PWt7iEdEewA4L/ot4PdL9bNWqXYEvD8C7fBd+U8A7pjZ3cfHUCw669eKwQ7AE/Y/Qc79V69G6X/I2itP5uz+rR3/MFucVOSTX8snv8Dbm/7KmuNg+vdoIKdc7BSP+fOffYbIa5pjF3TmARViLqOsWsZkxeIYAcA/0W/Hex+s25W68+Na9T+86w/c3SXMXaRVD143R3sWoZG2BeLBxDs+q3XBDsAT9nAg539gPL9/IbMus9l81r9R9O6Sqjv2H3Tv/03F7L/MWgWyEvZXV+RjD97pi48VyXZemXWqxcIRxWCRl/52MeCfPtpZsX6WZFDfSxbzT/CrQYV7JoLmJllSx3Dvj3LVn3WT8UuVHoWsZMLd9bPjrNi2sejZ+/U65sZwM6/FCT9/m/ZrR8/wQ4A/ot+O9j1UzdLO7KYyrszQNb0JCy3jpwXGuPf334yF3XtRxmZWTGrLfXMfjxQp2O075aNzG3L6XdV306y8sEe59dU0+zeLBOyVFCfq4eEFEpmNuwOtbDfek2wA/CEDTzYdWojr3fkzL941zYI2zT1o/rhvfkxt4Jd4ypcaxuV2ax1KU/fyauP6QtoLT/CrQYW7BTnJBX8sFTdVLH7196/Dq9zAcet1vvQoWvq+aeFxsPa25p9LAQ7APgv+v1gp/RaN7vV+eiKHFozbnarT03/F1A6HuPPgvX4AKtF5uRth7oY/Mxb/+5d51rYV70m2AF4wu452I1KdHKhfQZLTQ/CfjdhfvyjEnu3Ibtf1a+9P/6u/sPryLd/UrKY8Nc16yeSkj5o6fap3V7KaX5D5ifHGj/w0QmZTaa8/XcxyGDn+nks6eSMRE3B0IPT59cLchY0Lu72QvbX58y6+vvYlkP1BdSnnA4IZXqcQzqZkFj9rqj3fS9lj63vm2AHAP9FAwl2Wi9186okmeUFiY81QmC3mlYrF2S1XuP17NgJWUzbtcnT7Ridr3lZeu0fk6mLqs52rIu313KaXmjURPccCt3v2Pl6rdcEOwBP2GCCHQAAeFDUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAredgR6PRaDQajUaj0Wi0p91acccOAIAnjNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbI8T7GolSb+bkMjwMxmKjEl8p2zecOQsvSCxqFo+PCrRyR05Ne8AAIAGgh0AwDaYYPczL291SOvQ3uYvzYrapeT+UMtfbch+5VLOvxTlsOK479T+ScrI8IQsFSpS/V6W/aOKOM6xrKqgF0tX3HV6VvssHyK/sN3AmfNV57VZ8s4zyOm6WuePvFTN64E52Wj7e3ihOSGLqYKc1cx6LapH2zI/HvXWj07I/HpBvt2YNwEAj66XYFc9ycvquymJqnqof88j4wuyWihL7das4Lu9lm97O7KYMBdddZ14nZT0gV2/O3FUDZtw9796YhYBAB7cQINdfK0ghwfHbe30hxVoKlmJq4LR/uNvAtByUZUIi1OSzfFRie/0GdDcQDgqs7kLs+Cx+MFOtfFtOWstpsZ9B7v5HftvUpRcKilxXegjSfn3yqxrVD/NqYA9KtNrednX6xd2VMh7JiNzeTnvcPwAgIfVvTZfy6EbtqIyu25+y1Xb3Uq4wS2WKlm19kJyiVFVD6ZkMe3X8aJk/ppya8HbT93DnXO0Ic9fxOQ5wQ4AHtVAg13znbkO3KAxJ7mf5nVdSVZ73cdT9GVbngcGMy/Yzb5PSkyd33QmOKD+brCrFRZkaL1kXllMsAsstipkT6v34vYxqUC8pALfc7WvpoBt/sbzhWuzAADwmO6szT9KctpWa0W+ZWZUXUjKv1aPDedrKaAHhyP7yyrwvdiWM7OkzdVnWYzEZDW349YIgh0APB6C3YCcbcc6BDMv2D3fVgU2pa+eqnP/Yd6y/F6wu5Z/36vt+w125jtv2k6tPzI8Je35Uxf43wufAIDBubM2d+LWhVHZDCgZrar5Obdu7Qb98N9eSG5uVGL6QqD5fwDBDgAez8MFO6csu8v+xChWWy/K2aeVxniuevPDX+fAVz2yxwPosQMJWSz4XS/v2O71mAowajs9ecu7bTlsCZpuMZsrSPWmIrvrjeMeGZuRpbw1PqFWkq05sy+71cOS6YqpX9+oY3K7NKr9mnd9nYKd87Ugq/5EM8NRiSVWJFe27pp9L8jiq9bvzjrvbsHuqiDz9rrKeS6h1t8InLTGK/Arst95qCAA4IH8crA70nUhEXiRsZVfEw4DuuGfq5owMq7qhR5/TbADgEf3cMHu1pHa1bXU9lbcgpJR4cR9feOIU9P/LsqS2sdsRoUmvVw1xy0kwQHNLSjDoxJ/vyO7/rix9Ips7fmhp9t2UZndKsjhl7KcHRRkK6GCUaT5TpobYiJTEleBs2l8QmrGDXHTWRMg3fMqS+aNClRvsnJmjr1Wn2jECnaKOxZBHffiP81dGoOCnaNCWUyf419Z2T9Rx3rij3mYUMXTpCv3873vbuhjsf7d1fzw1SHYOdWy5P6ckJHJbTm1gpp7HG/ycm5eN3H3FXS3FQDw0H4t2Dly+PGO7pV1F5J5pWrC+8/S1kvzh677E7LlT2pNsAOARzfQYGffMWq0lrs/HcNBpztsAcvNnSa3+4dZ1C5oOz0WQC1rnVDltixbL57JyMfj+v68u1MB6+ouiX+q83qZlW9mST28demK2biDZ8YstExa0hbs9DGNP5Pna41j8lzL7oJa91XWCl/euTZ1qfSZYNfexmQxU5Jqy1XYrl1C3X0FddMEADy0Xwl2/gXD1ouLQbwLoVZ485n61FQfCXYA8OgGGuyCZ8WsNF/pG0Cw049FuPvOUaftmgeM+9xAE0nVQ2i3cQXee3Zg7SfYKSZgjlgzgLYFqvK2PB+OtRdUpf3Y7g52zbNiHst+PiWLk6MyMrkh+9ZBE+wAIBz6Dnb6LpuqPd0vinr8APg2335xUz/aoG2WZIIdADy6gQa79lAWYADBzp2opMM4sIZO26ng0rE19tke3hp+O9gp3iMFYrJ65JXX1kDlhVD/uIKa/R3eHewCi6256jq0UKiHb/c46IoJAE9eX8GudmzGePfw2JrvnQOgG/gic7LbWgcIdgDw6P4Hg92K7Pvj0Npao4Tdd7BzZxPTy8dT7hi34GBnjUUMaN4YRO0Xg53ifq41IQqTpwBAOPQc7OoTd2XlW338dwf+Xb2Pxbau+vV6puvNHY2ABwAPL5TBTt/tuvvOUft2vXXh9Nx7sNPMc+Ri22U5bO0C6XbFDHqQe5DBBTsppaTr4w5e2mMLAQCPpadgVw91Pdypq3fVPG7M/NzEkfNSc7f+eitsSFzVGr/b/zdrDDkA4GGEMtj5n6cDUWcB25mxbb2ML+gv2JnQE9iFsUuwU8629bPtJmR+YaY52Jlukj0VY6lI5qX6jLVj89rSLdjdHMvqC7XdK2siFjORTNukLe4MaNZsoACAR3VnbVahblOPpe4p1BVkPtot1N2BrpgA8OgGGuyCJ085ltMfVkQYRLBT6o870I8CMJ+zm91wZ3r0dNju04L7TLhIYkNye+YY9/KS1s/Y22qEr/6CndpvVgUzFdA+ZEtyfnUhZ6ULE4y6B7v6ODd1TK13/JyTlMRVEB2ZTEq64H+fBcmsJWX6vf0sPDN9dWRGNvcqUv1ZkdOvZsYzE+zaJ0/ZkLdjaht1zJul5pjrD5qfXjOPeSjsyLw+RtNtFADw+LrX5gvJvFa/8f+n6odf61qbX6fM+Luh1xuyWwxYTze/pnRCsAOARzfQYOeGk4DWFK4GFOy06oH1oHHVIuMLslr01+u8Xa2sH/o9JVEVmtxjjE5I/J0Kel8ahavfYCe3l3K4NWf2OSrR5GcTvO4IdopTSqkgpdYJ6sr581jSyUTjwe76geoJVaiLfnA0biqSW55pPMg8bT7PBLu2ps55NrnT9mB2X62UbXqI+/RyXs4CZhMFADyO7rXZq4GBv/9+82vOHTXcbV1qmItgBwCPbjDBDgAAPChqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDA1nOwo9FoNBqNRqPRaDTa026tuGMHAMATRm0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEAtscJdrWSpN9NSGT4mQxFxiS+UzZvOHKWXpBYVC0fHpXo5I6cmncAAEADwQ4AYBtMsPuZl7c6pHVob/OXZkXtUnJ/qOWvNmS/cinnX4pyWHHcd2r/JGVkeEKWChWpfi/L/lFFHOdYVlXQi6Ur7jo9q32WD5Ff2G7gzPmq89oseecZ5HRdrfNHXqrm9cCcbLT9PbzQnJDFVEHOama9ICqAb70eVetvELAB4InpJdhVT/Ky+m5Koqoe6t//yPiCrBbKUrs1K3TkqLo04W6zemIW2W4qklueMftVNeV1UtJHdq0HADy0gQa7+FpBDg+O29rpDyvQVLISV0WgvVCYALRcVOXE4pRkc3xU4jt9BjQ3EI7KbO7CLHgsfrBTbXxbzjoU0/sOdvM79t+kKLlUUuK6IEeS8u+VWdfifM3KvArUIxGCHQA8Rd1r87UcusEsKrPredk3v/+7Wwm3t0wsVWqutS2cow15/iImz4OCnVuXdU1bkHTB1JS1GffC7OpJt70CAO7TQINd8525DtygMSe5n+Z1XUlWe93HU/RlW54HBjMv2M2+T0pMnd90Jjig/m6wqxUWZGi9ZF5ZTLALvOKqQva0ei/eckzn//ytjlWF6dSxnOXmCHYA8ATdWZt/lOS0rdaKfMvMqN/1pPzbqcfG1WdZjMRkNbfj1vbW+nGe1du31nF9hy8mQy86X8AEANwvgt2AnG2rgtYl2D3fVgU2pa+eqnP/Yd6y/F6wu5Z/36vt+w125jtv3e48/7d8KHh3Oqt5gh0APEV31uZO3LowKpsBJUNuLyQ3NyoxVRccU9ub60eH3jVaxx45AICH8HDBzinL7rI/MYrV1oty9mlF5sejzcvr4a9z4Kse7chiwkzColpkPCGLJpDcud3rMRnR2+nJW95ty2FL0HQDzVxBqjcV2V1vHPfI2Iws5a3xCXoc2pzZl93qYckUQf36Rh3TuNqH3q9519cp2DlfC7LqTzQzHJVYYkVy5WvzrvK9IIuvWr8767y7Bburgszb6wYg2AHA0/TLwe5I14VE4EXGc/WbPzKufvNv1IugYHd73OUirFd3W3uBAAAexsMFu1tHalfXUttbcQtKRoUT9/WNI05N/7soS2ofsxkVmvRy1Rw3PAUHNLf46O6C73dk1x83ll6RrT0/9HTbLiqzWwU5/FKWs4OCbCVUMIo030lzA01kSuIqcDaNT0jpcQTPZDprAqR7XmXJvFGB6k1Wzsyx13RRdFnBTnHHLajjXvzHCmdKULBzVChzu0T+lZX9E3WsJ0XJ/DWlPt8ax+B+vvfdDX0s1r+7mn8ptUOwc6plyf05ISOT23LaZUgEwQ4AnqZfC3aOHH4c9bpMmiV1P3Qtn5Atf6LqoGDnLzsyr5s01zsAwMMaaLCz7xg1WksocINGP10xA5abO01uVxGzqF3QdnrcgFrWOqHKbVm2XjyTkY/H9f15gSZgXbXG/p/qvF5m5ZtZUi9mXbpiNgqd2n5ZFdWWSUvagp0+pvFn8nytcUyea9ldUOu+ysq5WeKfa2AxNcGuvY3JYqYk1TvGQhDsAOBp+pVg518wbL246NecpprXLdgF9QLx611QN00AwL0baLALnhWzIk3jswcQ7PRjEYL3Yeu0XfCAcTdYRVL1AOMFmjnZbU9qAWGnn2CnmIA5YhW/tmBX3pbnw7HGlVNL+7HdHeyaZ8U8lv18ShYnR2VkckP2A87RR7ADgKep72Cn78ip2tN+UdR7tMHIXF7O7Yt9BDsACJWBBrv2UBZgAMHOnajkzrDRaTtVdDq2xj67BZrfDnZK9ZPuEhqT1SOv/LUGOy+E+scV1Ozv8O5gF1iEzRXaoYVCc/i2EOwA4GnqK9jVjs0Y75bwprh38SJzsttal7sFO7piAsCT8z8Y7FZk3x+H1tYa1xjvO9i5M4/p5eMpd4xbcLCzxiIGNG8MovaLwU5xP1d/Jx0urxLsAOBp6jnY1Sfuysq3+vhvn6lRuobc0dw6wuQpAPBkhTLY6btdwfuwtW/XWxdOz70HO808Ry62XZbDwK6YvU4b7Z0rwQ4A/nf0FOzqoa79Tp3HkfNSc1f9eitsSFzVD78r/zd3XPil7M6putHxcQfPZKnYoaAAAO5VKIOd/3k6EHUWsJ0Z29Z90hVPf8FOT4iiCt0bVTjNkoYuwU4529bPtpuQ+YWZ5mBnukl2Lsa2imReqs9YOzavLd2C3c2xrL5Q272yJ2JpRrADgKfpztqsQt2mHkvdUx0J4He7bKkftcKCqguqjjc9LsGRw7WYDEVW5JBcBwCPYqDBLnjylGM5/WH9yg8i2Cn1xx3oRwGYz9nNbrgzPXo6bPdpwX0mXCSxIbk9c4x7eUnrZ+xtNcJXf8FO7TergpkKaB+yJTm/upCz0oUJj92DXX2cmzqm1jt+zklK4nqSlcmkpAv+91mQzFpSpt/bz8Iz01dHZmRzryLVnxU5/WpmPDPBrn3ylA15O6a2Uce8WepchQl2APA0da/NF5J5rX7j/0/VD7/WtbZ6neqgQ7DzH2I+NL5galNRcmv6UUCj8vZTaw0HADyUgQY7N5wEtKZwNaBgp1UPrAeNqxZRRWa16K/XebtaWT/0e0qiKjS5xxidkPg7FfS+NKZ/7jfYye2lHG7NmX2OSjT52QSvO4Kd4pRSEtPHEdSV8+expJOJxoPd9QPVE6pQF1sK8k1FcsszjQeZp83nmWDX1tQ5zyZ32h7M3opgBwBPU/fa7NXAwN9/vwUOH7B0CnaaCnf76wv12qTr79ZBe70FADycwQQ7AADwoKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAABbz8GORqPRaDQajUaj0WhPu7Xijh0AAE8YtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYHifY1UqSfjchkeFnMhQZk/hO2bzhyFl6QWJRtXx4VKKTO3Jq3gEAAA0EOwCAbTDB7mde3uqQ1qG9zV+aFbVLyf2hlr/akP3KpZx/KcphxXHfqf2TlJHhCVkqVKT6vSz7RxVxnGNZVUEvlq646/Ss9lk+RH5hu4Ez56vOa7PknWeQ03W1zh95qZrXA3Oy0fb38EJzQhZTBTmrmfVsP0uSW1+Q+Niot350QubXO6wLAHgUvQS76kleVt9NSVTVQ/17HhlfkNVCWWq3ZoVWt5dymF2R+ckxVY/NNlsl86blpiK55RmzX1VTXiclfWTXegDAQxtosIuvFeTw4Litnf6wAk0lK3FVBFZPzOs6E4CWi9IUf5ySbI6PSnynz4DmBsJRmc1dmAWPxQ92qo1vy1mHYnrfwW5+x/6bFCWXSkpcF+RIUv69MusqtaMNielCntiQ3J5Zv7Ats/ou6nhKTjtnUwDAA+pem6/lcH1C/f5HZXY9L/vm9393K+H2lomlSs21VqsWZWn8mYxMJmUzX5Sz79dS+1mR06/XZgXDrcu6JixIumBqytqMe2F29YQiAQCPZaDBrvnOXAdu0JiT3E/zuq4kq73u4yn6si3PA4OZF+xm3yfdwDSdCQ6ovxvsaoUFGVoPuKpqgl17kFZUyJ5W78XtY7q9kNOTgL+BWXfxn5YCDwB4FHfW5h8lOW2rtSLfMjOqLiTlX7sXhvrt17VqZC4v553u5hnnWb19ax13VB2LydCLzhcwAQD3i2A3IGfbqqB1CXbPt1WBTemrp+rcf5i3LL8X7K7l3/dq+36DnfnOA7dr4607kuplXQDAfbuzNnfi1oVR2bR+zp2DFRkJrM2tOvSu0Tr2yAEAPISHC3ZOWXaX/YlRrLZelLNPKzI/Hm1eXi8wnQNf9WhHFhNmEhbVIuMJWSz4XS/v2O61GT+gJ295ty2HLcWsmp+TobmCVG8qsrveOO6RsRlZylvjE2ol2ZprjEWot3pYMkVQv75Rx6S7uej9mnd9nYKd87Ugq/5EM8NRiSVWJFe27pp9L8jiq9bvzjrvbsHuqiDz9rrd3B673+fjd20FAGi/HOyOdF1INF1kPF0f7e3ioqkFwXXDq7tNvUAAAA/m4YLdrSO1q2up7a24BSWjwon7+sYRp6b/XZQlHRwyKjTp5ao5bngKDmjnKniNDI9K/P2O7PrjxtIrsrXnh55u20Vldqsgh1/KcnZQkK2ECkaR5jtpbrCLTElcBc6m8QkpPY7gmUxnTcBxz6ssmTcqUL3Jypk59tqN93ZTsFMcVVCfq+Nu7dIYFOwcFcpi+hz/ysr+iTrWk6Jk/ppSn2+NY3A/3/vuhj4W699dzb+U2iHYOdWy5P6ckJHJ7Z7GzXlXc2Oy5U9gCgB4VL8W7Bw5/KhCnO4yaZaIVCTzUvcsUfW3nJcl68LndDIrp3aXTVPvV4/M6ybN9Q4A8LAGGuzsO0aNttH8yAI3aPTTFTNgubnTFFPFo3MmCdrusyxG1LLWu063Zdl68UxGPh7X9+cGO7192x0qR/b/VOf1MivfzJJ6MevSFbNR6NT2y6qotkxa0hbs9DGNq0K71jgmz7XsLqh1X2Xl3CzxzzWwmJpg197GZDFTkmovYyH8O43vPwsTYwLA0/Arwc6/YNh8cdGrIfG5OYlP/i2Zg4pUr67l/EteVidVvRpXddy/WOkHu8DulqbeBXXTBADcu4EGu+BZMSvNYWAAwU4/FiF4H7ZO27UMGDfcYBVJ1UOoF+zmZDegX4r3nh1Y+wl2igmYI1bxawt25W153uEOWfuxeefaLdg1z4p5LPv5lCyqgj0yuSH7AedYpwfUz7UUdgDAo+s72P1QtVrVnvaLoqaGBP3Omwup9V4qBDsAeLIGGuzaQ1mAAQQ7d6KS1juBbTptp4pOx9bYZ3t4a/jtYKdUP+kuoTFZPfLKX2uw80Kof1xBzf4O7w52gUXY3BUcWigE34m7NdNlt3RTBQA8vr6CXe3YjPEOmvXSqyHBY+MuZXdO1Qk/rPnBjq6YAPDk/A8GuxXZ98ehtbXGNcb7Dnb+1NL+s+GCg501FjGgeWMQtV8Mdor7ufo7abu8qqeu9kJd5mvbmwCAR9ZzsKtP3JWVb4E9L0qyGek8OdbhmlWfmDwFAJ6sUAY7fbcreB+29u1668Lpufdgp5lnw8W2y3IY2BWz12mjvXMdXLBrhLrcd7MIAPCk9BTs6qGu2/PpzCNzAntvXEhOTw5Wry8td/Bs7uMOnslSkYuBAPAYQhns/M/TgaizgO3M2Lbuk654+gt2ekIUVejeqMJpljR075pytq2fbTch8wszzcHOdJPs5WGx/oxmQ2vH5rWlW7C7OZbVF2q7V/ZELCrUpaZkhFAHAE/anbVZhbpNPZa6hzrizXxszbjscy9AjsrSQWN5rbDg1fGmLvqOHK7FZCiyIofkOgB4FAMNdsGTpxzL6Q/rV34QwU6pP+5APwrAfM5udsOd6dHTYbtPC+4z4SKJDcntmWPcy0taP2NvqxG++gt2ar9ZFcxUUfyQLcn51YWclS5MeOwe7Orj3NQxtd7xc05SEteTrEwmJV3wv8+CZNaSMv3efhaemb46MiObexWp/qzI6Vcz45kJdu2Tp2zI2zG1jTrmzVLj76PPY0TPmLlTbFq/0dT5UbQB4NF1r80XknmtfuP/T9UPv9a1tnqd0i5ld0E/+mdK1THz+1/YltlowAXG+qRaC6Y2FSW3pmvHqLz91FrDAQAPZaDBzg0nAa0pXA0o2GnVA+tB46pFVJFZLfrrdd6uVtYP/Z6SqApN7jFGJyT+TgW9L43pn/sNdnJ7KYdbc2afoxJNfjbB645gpzillMT0cQR15fx5LOlkovFgd/1A9YQq1EW7ICs3FcktzzQeZJ42n2eCXVtT5zyb3Gl7MLvXNbNbC/rbAQAeWvfa7NXA4N9x01przu21nGaTMu1e9FOBbmxGFtPHwY/FUeFuf32hXpt0/d06aK+3AICHM5hgBwAAHhS1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBg6znY0Wg0Go1Go9FoNBrtabdW3LEDAOAJozYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAA2+MEu1pJ0u8mJDL8TIYiYxLfKZs3HDlLL0gsqpYPj0p0ckdOzTsAAKCBYAcAsA0m2P3My1sd0jq0t/lLs6J2Kbk/1PJXG7JfuZTzL0U5rDjuO7V/kjIyPCFLhYpUv5dl/6gijnMsqyroxdIVd52e1T7Lh8gvbDdw5nzVeW2WvPMMcrqu1vkjL1XzemBONtr+Hl5oTshiqiBnNbOepfalIJvJhAnYqkUnZDa5I4c/zQoAgEd3Z22+vZazQkoWE+ZCqmqRcfXbnz6W6q1Zx3ZTkdzyjERV7XTrxOukpI/s+m3pZ10AwIMYaLCLrxXk8OC4rZ3+sAJNJStxVQRWT8zrOhOAlovSFH+ckmyOj0p8p8+A5gbCUZnNXZgFj8UPdqqNb8tZUDFV7jvYze/Yf5Oi5FJJieuCHEnKv1dmXeU8n1DrR2VaBbldf/3CjixOjqp152SXcAcAT0L32nwhuYT6jY/OqCDXqM27ae+3f2Su0Fxv3Fqr69SCpAumTqzNuBdbV09aLkr2sy4A4MEMNNg135nrwA0ac5JrCwglWe11H0/Rl215HhjMvGA3+z4pMXV+05nggPq7wa5WWJCh9ZJ5ZTHBrj1IKypkT6v34vYx3V7KeVB4uyrIvFr3+bbfbRYA8JjurM3Vi8A7c269GI7JlvVzfp6dCajNjqpNMRl60XxRsp91AQAPh2A3IGfbqqB1CXbPt0tymprwzv2Hecvye8HuWv59r7bvN9iZ7zxwuzb9rAsAuG931uZO2upChx4zWlsvm37WBQA8pIcLdk5Zdpf9iVGstl6Us08rMj8ebV5eD3+dA1/1aKd97EDB73p5x3avx2REb6cnb3m33TZ+rJqfk6G5glRvKrK73jjukbEZWcqXpeZfkayVZGvO7Mtu9QBkiqB+faOOaTygC4zSKdg5Xwuy6k80MxyVWGJFcuVr867yvSCLr1q/O+u8uwU7cxeupzDdT3gHANy7Xw12bn2zL7DeHne5sOrV0nrPjn7WBQA8qIcLdreO1K6upba3ogpKQjIqnLivbxxxavrfRVlS+5jNqNCkl6vmuOEpOKCdq8I0Mjwq8ff+WLCi5NIrsrXnh55u20Vldqsgh1/KcnZQkK2ECkaR5jtpbuGLTElcBc7Z9bzs++MTUnocwTOZzpoA6Z5XWTJvVKB6k5Uzc+y1G+/tpmCnOEcb8lwd9+I/VjhTgoKdo0JZTJ/jX1nZP1HHelKUzF9T6vOtcQzu53vf3dDHYv27q/mXUjsEO6daltyfEzIyuS2n3YZEqP1Xv+TlgxtI83JOFxsAeBL6DnY3l3KW/9utK2/z1vhzU8NXj8zrJs01rK91AQAPaqDBzr5j1GgbzY8scINGP10xA5abO00xVTw6Z5Kg7T7LYkQta51Q5bYsWy9UcPl4XN+fd0UzYF21xv6f6rxeZuWbWVIvZl26YjYKndp+WU9E0jxpSVuw08ekwtTztcYxea5ld0Gt+yor52aJf66BxdQEu/Y2JouZUvDMaIp7PPV1p2QxV2rcpQQAPLregp2pD36bTEqu1HxhsR7WArtQmhrmd73sZ10AwIMaaLALnhWzIk0z6g8g2OnHIgTvw9ZpOxWoAqb4d4NMJFUPoX5Xld32pGbeswNrP8FOMQFzxCp+bcGuvC3PWwa3+9qP7e5g1zwr5rHs51PuTJcjkxuyH3COta+NdXezK/J2TK/7t/wbMD4QAPDwegt21/Kt/ttfkMzynEQjuifI50YPDIIdAPwnDDTYtYeyAAMIdu5EJa13Att02k4VnY6tsc/28Nbw28FOqX7SXUJjsnrklb/WYOeFUP+4gpr9Hd4d7AKLsLkrOLRQaA7fQW4rkn6l1n1l3ykEADyW3oJdgK877ozI9SEFflijKyYAhNr/YLBbkX1/HFpba1xjvO9gJ7cX3vLxlDvGLTjYWWMRA5o3BlH7xWCneF0u1XfSw+XV85x+xl1v6wIA7tcvBzv9jDs9Lty/s8bkKQDwnxDKYKfvdgXvw9a+XW9dOD33Huw08xy52HZZDgO7YvY6bbR3rgQ7APjfMbBgp+rU7pz92uI+wuCZLBX9d/pZFwDwkEIZ7PzP04Gos4DtzNi27pOuePoLdnpCFFXo3uQDuil275pytq2fbTch8wszzcHOdJPsbSbKimReqs9YOzavLd2C3c2xrL5Q272yuld2+qx+um0CAO5d19qsf8s7/Z5/2ZaYqgvzhcYkKt5Dy1VtbhpH7cjhWkyGIityaBXNftYFADycgQa74MlTjuX0h/UrP4hgp9Qfd6AfBWA+Zze74c706Omw3acF95lwkcSG5PbMMe7lJa2fsbfVCF/9BTu136wKZiqgfciW5PzqQs5KFyY8dg929cCkjqn1jp9zkpK4nmRlMinpgv99FiSzlpTp9/az8FRB/ahn2pyRzb2KVH9W5PSrKdgm2LVPnrLhToiij3mz5P991LHOjcl0ckMy9c9rTLTS+kgIAMDj6VqbdV0em5HFtax5JJBuRcmlkl5dab1oqIcHzKnf+fEFU2/Uumv68T6j8vZTS13uZ10AwIMZaLBzw0lAawpXAwp2WvXAetC4ahFVZFaL/nqdt6uV9UO/pySqipt7jNEJib9TQe9L4+plv8FObi/lcEvPNqb3OSrR5GcTvO4IdopTSrlXTwO7cv48lnQy0Xiwu36gekIFvaIfHI2biuSWZxoPMk+bzzPBrq2pc55N7rQ/mL2Ud78b+0Hy7oPfUwU541YdADwZXWuzqkmn+Q2Zfzlh6oJuqjao+rFZKAc/vkYFtv31hfrvv66pWwftNdTVz7oAgAcxmGAHAAAeFLUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGDrOdjRaDQajUaj0Wg0Gu1pt1bcsQMA4AmjNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAADb4wS7WknS7yYkMvxMhiJjEt8pmzccOUsvSCyqlg+PSnRyR07NOwAAoIFgBwCwDSbY/czLWx3SOrS3+UuzonYpuT/U8lcbsl+5lPMvRTmsOO47tX+SMjI8IUuFilS/l2X/qCKOcyyrKujF0hV3nZ7VPsuHyC9sN3DmfNV5bZa88wxyuq7W+SMvVfN6YE422v4eXmhOyGKqIGc1s143P9TfV32X93J8AIBfcmdtvr2Ws0JKFhPmQqpqkXH1258+luqtWcfifC/K1rspierfe7XuyNhMx3XlpiK55Rmzrqopr5OSPrJrPQDgoQ002MXXCnJ4cNzWTn9YgaaSlbgqAqsn5nWdCUDLRWmKP05JNsdHJb7TZ0BzA+GozOYuzILH4gc71ca35SyoQCr3Hezmd+y/SVFyqaTEdUGOJOXfK7NukNsLdfyj8vzFGMEOAJ6Q7rVZ/XYn1G98VIezRm3eTXu//SNzhebfc1WbZ9Xy2J9Z2f9yIbWfFTlVoXBar/v+szRdA3Trsq5pC5IumJqyNuNemF096XwBEwBwvwYa7JrvzHXgBo05yf00r+tKstrrPp6iL9vyPDD4eMFu9n1SYur8pjPBAfV3g12tsCBD6yXzymKCXXuQVlQhn1bvxTsck3a2PeEeV27tnoInAOCX3FmbqxeBd9vcejEcky1/FIRymhqVoZc78q1lfWfvb7VuQnI/zALlPDsTUMcdVcdiMvSi8wVMAMD9ItgNyNm2Kmhdgt3z7ZIqnCok6XO3CqTv94Ldtfz7Xm3fb7Az33ngdoqjto1FvOO9tzuKAIBfcmdt7qStLpgaEvQbb3rZbNbLRIfeNVrHHjkAgIfwcMHOKcvusj8xitXWi3L2aUXmx6PNy+vhr3Pgqx7ttI8dKPhdL+/Y7vWYjOjt9OQt77blsCVoVvNzMjRXkOpNRXbXG8etxxws5ctS869I1kqyNWf2Zbd6WDJFUL++Ucc0HtAFRukUnJyvBVn1J5oZjkossSK58rV5V/lekMVXrd+ddd7dgt1VQebtdW3usY6q97zvk2AHAE/LrwY7t761XGB178JFkrLb8iP/LaOXr8ihn+Juj7tchPXqbrdeIACA+/Nwwe7WkdrVtdT2VlRBSUhGhRP39Y0jTk3/uyhLah+zGRWa9HLVHDc8BQe0c1WYRoZHJf5+R3b9cWPpFdna80NPt+2iMrtVkMMvZTk7KMhWQgUjc2fK5xa+yJTEVeCcXc/Lvj8+IaXHETyT6awJkO55lSXzRgWfN1k5M8deu/Hebgp2inO0Ic/VcS/+Y4UzJSg4uXfM9Dn+lZX9E3WsJ0XJ/DWlPt8ax+B+vvfdDX0s1r+7ml+EOwQ7p1qW3J8TMjK5Ladtl10dOVyLNY2rINgBwNPSd7C7uZSz/N9uXfEv2tXp8dRzozIUTcjWwYU4NxdyqOtdZEa2vlhFwtT71SPzuklzvQMAPKyBBjv7jlGjbTQ/ssANGs1XCj3BQSxwubnTFFPFoy2T1AVt91kWI2pZ64Qqt2XZevFMRj4e1/fnXdEMWFetsf+nOq+XWflmltSLWWDwaS10avtlVTxbJi1pC076mMafyfO1xjF5rmV3Qa37KivnZol/roHF1AS79jYmi5lS8PgLPTvpiw05rIdTgh0APDW9BTtTH/w2mZRcqfnCYp0Kd/8u6yEDZt2JDdlv/dH3g11gd0tT74K6aQIA7t1Ag13wrJiV5tm0BhDsdPAI3oet03YqUAVM8e8Gl0iqHkL9riqt3VI07z07sPYT7BQTMEes4tcWnMrb8rxlcLuv/djuDnbNs2Iey34+JYuTozIy2VK43UcbtM9sRrADgKelt2B3Ld/qv/0FySzPSTSie4J8lnP7wt7tpeyvTclIZEo+ZIuyn12RaT0EIbogma9WPSDYAcCTNdBg1x7KAgwg2LkTlbTeCWzTaTtVdDq2xj7bw1vDbwc7pfpJdwmNyeqRV/5ag5MXQv3jCmr2d3h3sAsswuau4NBCwQvfpitOLNV+J5RgBwBPS2/BLsDXHXdG5PqQAvWL707uNb7SfKFPh72Park9VMEPdnTFBIAn538w2KnC5Y9Da2uNOHPfwc57PpxaPp5yx7gFBztrLGJA88Ygar8Y7BT3c/V3ok/drHtnI+ABwKP75WCnn3Gnx4X7d9Zqn2VR/bbPFwK6aPpDFVKmvjB5CgA8WaEMdvpuV/A+bO3b9daF03PvwU4zz5GLbZflsPWOmNsVs9dpowcU7K4qTd017ZbW4/pebngT1ZQu6GYDAI9sYMGul+6V9fpyKbtz1rY293EHz2SpSIUAgMcQymDnf54ORJ0FbGfGtnWfdMXTX7DTE6KoQvcmb01o4usS7BT3AeDDEzK/MNMc7Ew3yZE5tc+ACU6aVSTzUn3G2rF5bekW7G6OZfWF2u6VPRFLMLpiAsDT0rU267rRqXZ82ZaYqgv1O3TOsSyp2jgddKftpui9V++2KeYB56qONz2T1ZtNuenRCACABzXQYBc8ecqxnP6wfuUHEeyU+uMO9KMAzOfsZjfcmR49Hbb7tOA+Ey6S2JDcnjnGvbyk9TP2thrhq79gZ54BpALah2xJzq8u5Kx+V6t7sKuPc1PH1BqcnJOUxPUkK5NJSRf877MgmbWkTL+3n4WnCupHPdPmjGzuVaT6syKnX03BNsGuffKUDXk7prZRx7xZursKE+wA4GnpWpt1XR6bkcW1rHkkkG5FyaWSXl1puWhoPwrotHIpNV3H9rLyYVLViXFV76xZkuuPRhhfMLVJ7XdNPwpoVN5+aq3hAICHMtBg54aTgNYUrgYU7LTqgfWgcdUiqsisFv31Om9XK+uHfk9JVBU39xijExJ/p4Lel8b4gn6DnR5kfrilZxvT+xyVaPKzCUF3BDvFKaXcq6eBwennsaSTicaD3fUD1RMq6BVbukPeVCS3PNN4kHnafJ4Jdm1NnfNscqftweydEOwA4GnpWptVTTrNb8j8ywlTF3RTtUHVj81CWWoBd/PcmpporD8yNiXz6wX5Zoc6nwp3++sL9dqk6+/WQXu9BQA8nMEEOwAA8KCozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAW8/Bjkaj0Wg0Go1Go9FoT7u14o4dAABPGLUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2B4n2NVKkn43IZHhZzIUGZP4Ttm84chZekFiUbV8eFSikztyat4BAAANBDsAgG0wwe5nXt7qkNahvc1fmhW1S8n9oZa/2pD9yqWcfynKYcVx36n9k5SR4QlZKlSk+r0s+0cVcZxjWVVBL5auuOv0rPZZPkR+YbuBM+erzmuz5J1nkNN1tc4feama1wNzstH29/BCc0IWUwU5q5n1LO6xtG2j25zkfpqVAACP6leC3Xl+TtXZ1rqsdKvj6yWzUovbSznMrsj85Ji7T71uZKvDugCAezfQYBdfK8jhwXFbO/1hBZpKVuIqWKyemNd1JgAtF6Up/jgl2RwflfhOnwHNDYSjMpu7MAseix/sVBvflrNbs7jFfQe7+R37b1KUXCopcRV8hyJJ+ffKrGu4x7KwY63vt5Kcd86mAIAH1Hew+6FqdSQmz/+vU7CbktVC6+++al+vzUqWalGWxp/JyGRSNvNFOft+LbWfFTkNWhcA8CAGGuzaCkUQN2gE3fkpyWqv+3iKvmzL88Bg5gW72fdJianzm84EB9TfDXa1wkLwVVUT7NqDtKJC9rR6L950TCaIdrpCCwB4EvoKdrdl2VJB7G0uH1xrO9bmALcXbp0YmcvLeYeLlQCAh0ewG5Cz7ViHYOYFpefbJTlNTXjn/sO8Zfm9YHct/77vEMa6BTvznTdvpwr2mxAHbAD4H9F7sHNUjZkwQaxDrT3StWKjp3HtzsGKjNA1HwCenIcLdk5Zdpf9iVGstl6Us08rMj8ebV5eLxqdA1/1aEcWE2YSFtUi4wlZLPhdL+/Y7rUZE6Anb3m3LYctBaqan5OhuYJUbyqyu9447pGxGVnKl6XmX6WslWRrrjG+oN7qYcm6A3ajjkl3XdH7Ne/6OgU752tBVv2JZoajEkusSK5sdXX5XpDFV63fnXXe3YLdVUHm7XVd5nv71OVvCQB4dL0GO0fVgVjEv6gYXBvdmhdJ9RTsTtdH72foAADgtzxcsLt1pHZ1LbW9FRU0EpJR4cR9feOIU9P/LsqS2sdsRoUmvVw1xw1PwUXIGwA+KvH3O7LrjgMoSi69Ilt7fujptl1UZrcKcvilLGcHBdlKqGBUL3oer8hNSVwFztn1vOybsQa7qRk3xE1nTYB0z6ssmTcqUL3Jypk59tqN93Zr10bnaEOeq+Ne/Kd5HEJQsHOLsT7Hv7Kyf6KO9aQomb+m1OdPqKBmBru5n+99d0Mfi/XvruaPhesQ7JxqWXJ/TsjI5Lac2uPmbo/d7y0y3gjMQ9EJmU3utIVfAMDj6SnY3ajf9Bd2zelQG3MJ90JnbEyFNve3X0+ytSCrBetCpqsimZe6F4paXs7LknWRdDqZldOACbkAAA9joMHOKwatraVrhxs0grpwBBebwOXmTlNMhaXOc3kEbfdZFiNqWeuEKnrswYtnMvLxuL4/N9jp7dsmX3Fk/091Xi+z8s0sqYe3Ll0xG3fw1PbLqnC2TFrSFuzMeIjna41j8lzL7oJa91VWzs0S/1ybu1QaJti1tzFZzJSk2jY+wpFvhazk9hoD5/fzKVmc1MfcHH4BAI/n7mCnu+mPttSRDrW2eiyZrD0BWkEyywn3Al9zrfW2j8/NSXzyb8kcVKR6dS3nX/KyquvEuKr59QubAICHNNBgFzwrZkWaLuANINjpxyIE78PWaTsVqDpN8W91Q/GC3ZzsBvQ18d6zA2s/wU4xAXPEmgG0LdiVt+X5cEy2/Ef8WdqPzTvXbsGueVbMRlgbmdyQ/YBzbGO6kQ4tFJr/ngCAR3FXbXZ7qLQFrU61NpjXy0XVoi9mgV9vggKcueha79ECAHhQAw12PRWKAQQ7d6KSOwd5d9pOFaSOrbHP9vDW8NvBTql+8orl6pEX7VqDnRdC/eMKavZ3aAptl2AXOMbO3BXsNax9y0ypfa3IfvMtRADAI+ham91HGwQ9P7VTre3A8br6N2ZP9rZvnk3Zdym7c6qmtD62CADwIP4Hg50KJv44tLbWKEX3Hez86aKHxlPuGLfgYGeNRQxo3hhEzTvXvoOd4n5uj2Gt/U4hAOCxdKvN3m/73e3uuu3Vl5GUX19Kshl51vEZsYdrar+BtRAAcN9CGez03a7gfdjat+utC6fn3oOdZp4jF9suy2FgV8ygB7kHeZhg5wbjHmdNAwDcr261ufa1uft9o+243SX9oROnP+748a99lkVdS+szJZvH6wT29PAelxNYiwAA9y6Uwc7/PB2IOgvYzoxt6z7piqe/YKcnRFHF7E3emtDE1yXYKWfb+tl2EzK/MNMc7Ew3yd4eAOvNUja0dmxeW7oFO3e2NLXdK2silk6fVSvKkvrungcN+gMAPLg7a3OgDrW2w2//tx1Vm4YXZNea7Mt7jp01O7PPvVg5KksHPVwpBAAM3ECDXfDkKS1XBAcR7JT64w70owDM5+xmN9yZHj0dtvu04M7yFUlsNGZ+3MtLWj9jb6sRvvoLdmq/WV38JuRDtiTnVxdyVrow4bF7sKuPc1PH1HrHzzlJSVxPsjKZlHTB/z4LkllLyvR7+1l4jhx+1LNWzsjmXkWqPyty+tVMbW2CXfvkKRvy1p3WunkMhj636NyKpPPFpnVno/o4Usx2BgBPxOCCnVoWnZHFtax5fJBuBUm/14/Xicr8p9Zul5eyu6AfEzSlap6pFYVtr070dDESAHAfBhrs3HAS0JoKyICCnVY9sB40rlpkfEFWi/56nberlfVDv6ckqkKTe4zRCYm/U0HvS+PZcv0GO7m9lMMtFYrcfY5KNPnZBK87gp3ilFIS08cR1JXz57Gkk4nGg931A9UTKugV/eBo3FQktzzTeJB52nyeCXZtrdOz6are58XrzzIyD35PHwc8GgEA8FgGF+z0Y242ZP6l9fxSXWtUXdwtNz9zte72Wk6zSZk2tWJkTAVD6gQAPKrBBDsAAPCgqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwEewAAAghajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AAFvPwY5Go9FoNBqNRqPRaE+7teKOHQAATxi1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgeJ9jVSpJ+NyGR4WcyFBmT+E7ZvOHIWXpBYlG1fHhUopM7cmreAQAADQQ7AIBtMMHuZ17e6pDWob3NX5oVtUvJ/aGWv9qQ/cqlnH8pymHFcd+p/ZOUkeEJWSpUpPq9LPtHFXGcY1lVQS+Wrrjr9Kz2WT5EfmG7gTPnq85rs+SdZ5DTdbXOH3mpmtcDc7LR9vfwQnNCFlMFOauZ9QLUygXZTCZM0FYtsiC7Az9AAMCvuDvYlWS17ffftNZ6c3st3/Z2ZDFhLrrqOvE6KekDu343ON+LsvVuSqKqzur9jYzNyGL6WKq3ZgUAwIMbaLCLrxXk8OC4rZ3+sAJNJStxVTBWT8zrOhOAlovSFH+ckmyOj0p8p8+A5gbCUZnNXZgFj8UPdqqNb8tZh6J338Fufsf+mxQll0pKXBfkSFL+vTLr1jnyLTOnQnZUZpezsnuigvbVtQrhJTnvnE0BAA+o12DX/PtvWunCqrUXkkuMqnowpcKZX8eLkvlrStWBUXn7qSXcqTo+qy+c/pmV/S8XUvtZkdNCSqbVspH3n6XL9UIAwD0aaLBrvjPXgRs05iT307yu8wpQT/t4ir5sy/PAYOYFu9n3SYmp85vOBAfU3w12tcKCDK2XzCuLCXbtQVpRxXlavRdvOSbnaEOeD0/I6tG1WQIAeGp6rc2Bv/8tnK+lgB4cjuwvq8D3YlvOzBLtNKWWvdyRby0XKp29v1W9SUjuh1kAAHhQBLsBOduOdQhmXrB7vl1SxXDCO/eAovd7we5a/n2vtu832PnddJq2u5DMK3W8ahk35wDg6bqzNv/Iy2xgve1dNT/n1q1GN3xTb4LqlemRsxlQigAA9+/hgp1Tlt1lf2IUq60X5ezTisyPR5uX14tR58BXPbLHAzyTyHhCFgt+18s7tns9JiN6Oz15y7ttOWwpfG4xmytI9aYiu+uN49bjCJbyZan5VyprJdmaM/uyWz0sma6Y+vWNOqZxtQ+9X/Our1Owc74WZNWfaGY4KrHEiuTK1p207wVZfNX63Vnn3S3YXRVk3l5Xc/+WQV1lAQBPyZ212VxI/Z2x0ee5hNrHhhxad+fOszNuN/7W/X7L6OUrcshVQQB4FA8X7G4dqV1dS21vRRWJhGRUOHFf3zji1PS/i7Kk9jGbUaFJL1fNcQtJcEA7V8FL9/2Pv9+RXTMeIJdeka09P/R02y4qs1sFOfxSlrODgmwlVDCKNN9Jc4NdZEriKnDOrudl34xL2E3NuCFuOmsCpHteZcm8UYHqTVbOzLHXbry3m4Kd4nVzHJXFf5q7OQYFO0cV5Zg+x7+ysn+ijvXEH/MwoYKXqZzu53vf3dDHYv27q/mFtUOwc6plyf05ISOT23JqFWGnqP8+Sfm3eimHW41AGxlfkNW9xx6vCADw3VmbVb1xLwhaF07dC6A9T3Li9eAYah03d3shublRGYomZOvgQpybCznUtTEyI1tfSHUA8FgGGuz8wtHcNpofWWCuIPbeFTNgubnTFOvaXTBou8+yGFHLWidUuS3L1otnMvLxuL4/r/tJwLp6zMGf6rxeZuWbWVIPb126Yjbu4JkxCy2TlrQFO31M48/k+VrjmDzXsrug1n2VlXOzxD/X5i6Vhgl27W1MFjOltuLud7t5Ozfmht+z7yok/qzIYXpBIipkvs0T7gDgKbizNuseJ9nGhUl74qyRubyc3xHuvAuhE7LlP5HIpsLdv8t6eIGpKRMbst9eAAEAD2igwS54VsxK85W+AQQ7/ViE4H3YOm2nAlXAlF1usIqk6iG0fVxBg/eeHVj7CXaKCZgj1gygbcGuvC3Ph2OBBbX92O4Odq2zou3nU7I4OSojk83F2Nt3cIA724rRzQYAnog7a3MHXm8QVRcKnSfI8nuMBF7Mu72U/bUpGYlMyYdsUfazKzKte3dEFyTzlQIBAI9loMGuPZQFGECwcycqab0T2KbTdqr4dGyNfbaHt4bfDnZK9ZO+EhqT1SOvCLYGOy+E+scV1Ozv8O5gFzhmztwVHFoo1MO3d24rsh9Um0spdcxT0mFiTwDAA/rVYCdSkcxL9dvf+ngh33dV0/XjDFRNaX/f8SYCG1d1wi54Oux9VMtbhjUAAB7O/2CwU8XIH4fW1hol7L6DnTtGQS8fT7lj3IKDnTUWMaB5YxC1Xwx2ivu5VpDTgVOfmz1Qvs6dYa3zvgAAD+fXg52pS3PtE3np33k31H0sBo/Dq32WRVUHAu/2+cMaUgG1CABw70IZ7LzwEbQPW/t2vXXh9Nx7sNPMc+Ri22U5DOyK2evslIMLdvUuoF/Ma5s7EL+37w8AcL9+Pdh1CGB+qFs/bsz83MrU++Ca0qXeAQDuXSiDnf95OhB1FrCdGdvWfdIVT3/BTk+IoorZm7w1oYmve6E729aDzydkfmGmOdiZbpK9DHCvd6tZOzavLd2C3c2xrKri3jwRizcLWvvnmklfmtYFADyWO2tzh9qhZ6fWQwGaLuD9KMh89I5QpznHsqTq6HRQn/yboveeP2s0AOBBDTTYBU+eciynP6wYNYhgp9Qfd6AfBWA+Zze74c706Omw3Sc9u+MziSQ2JLdnjnEvL2n9jL2tRvjqL9ip/ern+qiA9iFbkvOrCzkrXZjweMcVTH+cmzqm1jt+zknKm71sMinpgv99FiSzlpTp93YXGkcOP+qZNmdkc68i1Z8VOf1qusmYYNc+ecqGvB1T26hj3iw1x1yntO1+biSxXX+URNtjFgAAj6p7bVa1Z25M3i7vNGqdnhVzPeHOcBxPWRc4a8fuM1aHXm/IbrFRJ5qaX1MU+7FBp5VLqemat5eVD5Oqpoyr2lh/3A8A4CENNNi54SSgNYWrAQU7rXpgPWhcNfdZa0V/vc7b1cr6od9TElXhxT3G6ITE36mg96VRuPoNdnrg+OHWnNnnqESTn03wurtrilNKuTOUBXbl/Hks6WSi8WB3/UD1hAp6RT84GjcVyS3PNB5knjafZ4JdW1PnPJvcaXswe933YvOD0dX3s2s/GB0A8Kjuqs3VI1UjE1at07/lun4ctNTFO2q421pqmFt/E36NeCYjY1Myv16Qb4Q6AHg0gwl2AADgQVGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQC2noMdjUaj0Wg0Go1Go9GedmvFHTsAAJ4wajMAwEawAwAghKjNAAAbwQ4AgBCiNgMAbAQ7AABCiNoMALAR7AAACCFqMwDARrADACCEqM0AABvBDgCAEKI2AwBsBDsAAEKI2gwAsBHsAAAIIWozAMBGsAMAIISozQAAG8EOAIAQojYDAGwEOwAAQojaDACwPU6wq5Uk/W5CIsPPZCgyJvGdsnnDkbP0gsSiavnwqEQnd+TUvAMAABoIdgAA22CC3c+8vNUhrUN7m780K2qXkvtDLX+1IfuVSzn/UpTDiuO+U/snKSPDE7JUqEj1e1n2jyriOMeyqoJeLF1x1+lZ7bN8iPzCdgNnzled12bJO88gp+tqnT/yUjWvB+Zko+3v4YXmhCymCnJWM+u5/GPt1uYk99OsDgB4ND0Hu9trOSukZDFhLqiqNrJQaK83NxXJLc9IVNVOt068Tkr6yK7fln7WBQA8iIEGu/haQQ4Pjtva6Q8r0FSyEldFYPXEvK4zoWK5KE3xxynJ5vioxHf6DGhuIByV2dyFWfBYrLA0vi1nt2Zxi/sOdvM79t+kKLlUUuK6IEeS8u+VWVd98+cle73mtvtxQu1Lrd8UBgEAj6GnYKcCWGZuVIaiCVnKFuS0cim1qws5K10E1FpdpxYkXdC/+apOrM24F1tXT1ouSvazLgDgwQw02DXfmevADRpBd31KstrrPp6iL9vyPDCYecFu9n1SYur8pjPBAfV3g12tsCBD6yXzymKCXXuQVlTInlbvxTscU7MLybx6JiMfj5v/MwAAeBR3BztHDtdiKoBtyOEdF+TOszMBtdlRtUlt/6L5omQ/6wIAHg7BbkDOtlVB6xLsnm+X5DSl73ipc/9h3rL8XrC7ln/fq+37DXbmOw/croWj9vN8OCZbX8wCAMCjurM2f9cX72I93EXr0GNGa+tl08+6AICH9HDBzinL7rI/MYrV1oty9mlF5sejzcvr4a9z4Kse7TSNGYiMJ2Sx4He9vGO712MyorfTk7e825bDlqBZzc/J0FxBqjcV2V1vHPfI2Iws5ctS869I1kqyNWf2Zbd6WDJFUL++Ucc0rvah92ve9XUKds7Xgqz6E80MRyWWWJFc+dq8q3wvyOKr1u/OOu9uwe6qIPP2uh1dy+5C8PEBAB7HXbXZrWORDTm96w7a7XGXC6teLa337OhnXQDAg3q4YHfrSO3qWmp7KypoJCSjwon7+sYRp6b/XZQltY/ZjApNerlqjluMggPauSpYI8OjEn+/I7vuGLCi5NIrsrXnh55u20Vldqsgh1/KcnZQkK2ECkaR5jtpXkGckrgKnLPredn3x5ml9DiCZzKdNQHSPa+yZN6o4PMmK2fm2Gs33ttNwU5xjvSdr1FZ/McKZ0pQsNN3yWL6HP/Kyv6JOtaTomT+mlKfb41jcD/f++6GPhbr313Nv5TaIdg51bLk/pyQkcltOb3rYq571XdUlg7uWhEA8FC612ZH9pdVXXj/Wao/j2XLvkD4bkP2v5vVNFPDV4/M6ybNNayvdQEAD2qgwc6+Y9RoG82PLHCDRj9dMQOWmztNMVU8OkeNoO0+y2JELWudUOW2LFsvmsePucFOb982+Yoqln+q83qZlW9mSb2YdemK2Sh0utiOtkxaEhDs9DGNP5Pna61j2szds1dZOTdL/HMNLKYm2LW3MVnMlKR651gIxk0AwFPUvTb7dWlO3o4lZKtQlvOra6lWjiW90HIx0w9rgV0ozX78rpf9rAsAeFADDXbBs2JWpGnM9gCCnX4sQvA+bJ22C57V0Q1WkVQ9hHrBbk52A/oeeu/ZgbWfYKeYgDliFb+2YFfe9sa0+Y/4s7Qf293BrnlWzGPZz6dkcXJURiY3ZD/gHOtMiK7foQQAPAk9BbuW3iiu1ouZBDsA+E8YaLBrD2UBBhDs3IlKWu8Etum0nSo6HVtjn+3hreG3g51S/aS7hMZk9cgrf63Bzguh/nEFNfs7vDvYBRZhc1dwaKHQHL4t7uxnkRXZr3ctBQA8BT0Fuw4h6zQ12uh54oc1umICQKj9DwY7FVL8cWhtrVH+7jvYye2Ft3w85Y5xCw521ljEgOaNQdR+Mdgp7ufq7ySo8jvHshR5Js+DbhsCAB7VXcFud079vq8dm9fNznOJRh1j8hQA+E8IZbDTd7uC92Fr3663Lpyeew92mnmOXGy7LIeBXTF7nTbaO9dBBzv32XjDM5KxB9kDAJ6Eu2qzezGzw/jowzW75pgQGHR3z32EwTNZKvrv9LMuAOAhhTLY+Z+nA1FnAduZsW3dJ13x9BfszOxjb/LWhCa+7l1Tzrb1s+0mZH5hpjnYmW6SI3Nqn3dOWlKRzEv1GUFXZrsFu5tjWX2htntlT8Ri3Kp9vlLvdemmCQB4PHfWZjOj8dt8yxjpm6LbG8MeO+1dyFO1uWk8nnnAeWRFDq2i2c+6AICHM9BgFzx5yrGc/rB+5QcR7JT64w70owDM5+xmN9yZHj0dtvu04E75HElsSG7PHONeXtL6GXtbjfDVX7AzY9FUQPuQLcn51YWclS5MeOwe7Orj3NQxtd7xc05SEteTrEwmJV3wv8+CZNaSMv3efhaeKqgf9UybM7K5V5Hqz4qcfjWPUzDBrn3ylA15O6a2Uce8WWqvws7BijcG8M4H2wIAHsOdtVnVhtMt/Ygc84gft95l5cOk+u0fVzXMHjuthwfM6eULpt4UJbemH++jguGnlrrcz7oAgAcz0GDnhpOA1hSuBhTstOqB9aBx1SKqyKwW/fU6b1cr64d+T0lUhSb3GKMTEn+ngt6XxrPl+g12cnsph1tzZp+jEk1+rndx6RrsFKeUkpg+jqCunD+PJZ1MNB7srh+onlBBr+gHR+OmIrnlmcZzitLm80ywa2vqnGeTO20PZvcEPVIBAPCU3B3sPOd7GzI/Hq3/9s+vF+QsqCuGCmz76wv1eqNr6tZBew119bMuAOBBDCbYAQCAB0VtBgDYCHYAAIQQtRkAYCPYAQAQQtRmAICNYAcAQAhRmwEANoIdAAAhRG0GANgIdgAAhBC1GQBgI9gBABBC1GYAgI1gBwBACFGbAQA2gh0AACFEbQYA2Ah2AACEELUZAGAj2AEAEELUZgCAjWAHAEAIUZsBADaCHQAAIURtBgDYeg52NBqNRqPRaDQajUZ72q0VwY5Go9FoNBqNRqPRQtZa1YMdAAAAACCcCHYAAAAAEHIEOwAAAAAIOYIdAAAAAIQcwQ4AAAAAQo5gBwAAAAAhR7ADAAAAgFAT+f8B/E4bvK1JgSMAAAAASUVORK5CYII=)\n"},{"metadata":{"id":"VumMFRvhHQpZ","trusted":true},"cell_type":"code","source":"def build_model(dim=128, ef=0, weights=None):\n model_input = tf.keras.Input(shape=(dim, dim, 3), name='inputs')\n\n model = EFNS[ef](include_top=False, weights=weights, input_shape=(dim, dim, 3), pooling=None)(model_input)\n MODEL = f'EfficientNetB{ef}'\n print(f'Seleccionado {MODEL}')\n\n # Freeze the pretrained weights\n if weights is not None:\n model.trainable = False\n\n # Rebuild top\n x = tf.keras.layers.GlobalAveragePooling2D(name=\"avg_pool\")(model)\n x = tf.keras.layers.BatchNormalization()(x)\n\n top_dropout_rate = 0.2\n x = tf.keras.layers.Dropout(top_dropout_rate, name=\"top_dropout\")(x)\n outputs = tf.keras.layers.Dense(cfg[\"num_class\"], activation=\"softmax\", name=\"pred\")(x)\n model_out = tf.keras.Model(model_input, outputs, name=f\"{MODEL}\")\n \n if cfg['unfreeze']:\n for layer in model.layers[-20:]:\n if not isinstance(layer, tf.keras.layers.BatchNormalization):\n layer.trainable = True\n \n return model_out \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def compile_new_model(cfg, dim=128, ef=0, weights=None): \n with strategy.scope():\n model = build_model(dim=dim, ef=ef, weights=weights)\n\n losses = tf.keras.losses.CategoricalCrossentropy(label_smoothing = cfg['smoothing'])\n model.compile(\n optimizer = tf.keras.optimizers.Adam(lr=cfg['lr']),\n loss = losses,\n metrics = tf.keras.metrics.CategoricalAccuracy()\n ) \n return model","execution_count":null,"outputs":[]},{"metadata":{"id":"Zzw2NR7XKtdg"},"cell_type":"markdown","source":"### Consejos para ajustar EfficientNet\n**Sobre descongelar capas:**\n\n- Las BathcNormalization deben mantenerse congeladas ([más detalles](https://keras.io/guides/transfer_learning/)). Si también se convierten en entrenables, la primera época después de la descongelación reducirá significativamente la precisión.\n- En algunos casos, puede ser beneficioso abrir solo una parte de las capas en lugar de descongelarlas todas. Esto hará que el ajuste sea mucho más rápido cuando se vaya a modelos más grandes como el B7.\n- Cada bloque debe estar encendido o apagado. Esto se debe a que la arquitectura incluye un atajo desde la primera capa hasta la última capa para cada bloque. No respetar los bloques también perjudica significativamente el rendimiento final.\n\n**Algunos otros consejos para utilizar EfficientNet:**\n\n- Las variantes más grandes de EfficientNet no garantizan un rendimiento mejorado, especialmente para tareas con menos datos o menos clases. En tal caso, la variante más grande de EfficientNet elegida, más difícil es ajustar los hiperparámetros.\n- EMA (Exponential Moving Average) es muy útil para entrenar a EfficientNet desde cero, pero no tanto para el aprendizaje por transferencia.\n-No utilice la configuración RMSprop como en el documento original para el aprendizaje por transferencia. El impulso y la tasa de aprendizaje son demasiado altos para transferir el aprendizaje. Corromperá fácilmente el peso preentrenado y hará estallar la pérdida. Una comprobación rápida es ver si la pérdida (como entropía cruzada categórica) se vuelve significativamente mayor que log (NUM_CLASSES) después de la misma época. Si es así, la tasa / impulso de aprendizaje inicial es demasiado alta.\n- Un tamaño de lote más pequeño beneficia la precisión de la validación, posiblemente debido a que proporciona una regularización eficaz."},{"metadata":{"id":"pQQCTwiLC2t0"},"cell_type":"markdown","source":"## Fase 4: Proceso de entrenamiento y validación"},{"metadata":{"id":"k0NjMJHyDCdP"},"cell_type":"markdown","source":"#### Definición de Callbacks"},{"metadata":{},"cell_type":"markdown","source":"**Train Schedule**"},{"metadata":{},"cell_type":"markdown","source":"- La tasa de aprendizaje comienza cerca de cero, luego aumenta hasta un máximo y luego decae con el tiempo. \n- Tenga en cuenta que la tasa de aprendizaje máxima es mayor con tamaños de lotes más grandes. Esta es una buena práctica a seguir."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8): \n lr_start = 0.000005 \n lr_max = 0.00000125 * REPLICAS * batch_size \n lr_min = 0.000001 \n lr_ramp_ep = 5 \n lr_sus_ep = 0 \n lr_decay = 0.8\n\n def lrfn(epoch):\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n\n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n\n else:\n lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n\n return lr\n\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback","execution_count":null,"outputs":[]},{"metadata":{"id":"HiX8axykC79j"},"cell_type":"markdown","source":"### Monitorización de las métricas en Tensorboard"},{"metadata":{"id":"Adct9R5RC_H_","trusted":true},"cell_type":"code","source":"# %tensorboard --logdir \"logs/\"","execution_count":null,"outputs":[]},{"metadata":{"id":"hiBDS0pKDNgH"},"cell_type":"markdown","source":"### Proceso de entrenamiento - Base Model"},{"metadata":{"id":"tIXewNsAEvAC"},"cell_type":"markdown","source":"#### Entrenamiento"},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\n%time\nVERBOSE = 1\nDISPLAY_PLOT = True\n\nskf = KFold(n_splits=cfg[\"folds\"],\n shuffle=cfg[\"suffle\"],\n random_state=cfg[\"seed\"])\n\noof_pred = []; oof_tar = []; oof_val = []; oof_names = []; oof_folds = [] \npreds = np.zeros((count_data_items(files_test),cfg[\"num_class\"]))\n\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))): # 15: Archivos tfrecord\n \n # DISPLAY FOLD INFO\n if DEVICE=='TPU':\n if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n print('#'*25); print('#### FOLD',fold+1)\n print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n \n # CREATE TRAIN AND VALIDATION SUBSETS\n files_train = tf.io.gfile.glob([GCS_PATH + '/train_tfrecords/*.tfrec'%x for x in idxT])\n \n np.random.shuffle(files_train); print('#'*25)\n files_valid = tf.io.gfile.glob([GCS_PATH + '/train_tfrecords/*.tfrec'%x for x in idxV])\n \n files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test_tfrecords/*.tfrec')))\n \n # BUILD MODEL\n K.clear_session()\n with strategy.scope():\n model = compile_new_model(cfg, dim=cfg[\"crop_size\"], ef=EFF_NETS[fold], weights=cfg[\"weights\"])\n \n # SAVE BEST MODEL EACH FOLD\n save_name = f'{cfg[\"name\"][0]}_v{cfg[\"version\"]}_fold-{fold}.h5'\n sv = tf.keras.callbacks.ModelCheckpoint(\n save_name, \n monitor='val_loss', \n verbose=0, \n save_best_only=True,\n save_weights_only=True, \n mode='min', \n save_freq='epoch')\n \n # TRAIN\n print('Training...')\n history = model.fit(\n get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold]), \n epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n repeat=False,dim=IMG_SIZES[fold]), #class_weight = {0:1,1:2},\n verbose=VERBOSE\n )\n \n print('Loading best model...')\n model.load_weights(save_name)\n \n # PREDICT OOF USING TTA\n print('Predicting OOF with TTA...')\n ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n ct_valid = count_data_items(files_valid) \n STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n pred = model.predict(ds_valid,\n steps=STEPS,\n verbose=VERBOSE)[:TTA*ct_valid,] \n oof_pred.append( np.mean(pred.reshape((ct_valid, TTA, cfg[\"num_class\"]),order='F'),axis=1) ) \n #oof_pred.append(model.predict(get_dataset(files_valid,dim=IMG_SIZES[fold]),verbose=1))\n \n # GET OOF TARGETS AND NAMES\n ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n labeled=True, return_image_names=True)\n oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n oof_folds.append( np.ones_like(oof_tar[-1],dtype='int8')*fold )\n ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n labeled=False, return_image_names=True)\n oof_names.append( np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n \n # PREDICT TEST USING TTA\n print('Predicting Test with TTA...')\n ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n ct_test = count_data_items(files_test)\n STEPS = TTA * ct_test/BATCH_SIZES[fold]/4/REPLICAS\n pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n# preds[:,0] += np.mean(pred.reshape((ct_test, TTA, cfg[\"num_class\"]),order='F'),axis=1) * WGTS[fold]\n preds[:] += np.mean(pred.reshape((ct_test, TTA, cfg[\"num_class\"]),order='F'),axis=1) * WGTS[fold]\n \n # REPORT RESULTS\n# acc = accuracy_score(oof_tar[-1],oof_pred[-1])\n acc = accuracy_score(np.argmax(oof_tar[-1], axis=1), np.argmax(oof_pred[-1], axis=1))\n oof_val.append(np.max( history.history['val_categorical_accuracy'] ))\n print('#### FOLD %i OOF ACC without TTA = %.3f, with TTA = %.3f'%(fold+1,oof_val[-1],acc))\n \n # PLOT TRAINING\n if DISPLAY_PLOT:\n plt.figure(figsize=(15,5))\n plt.plot(np.arange(EPOCHS[fold]),history.history['categorical_accuracy'],'-o',label='Train ACC',color='#ff7f0e')\n plt.plot(np.arange(EPOCHS[fold]),history.history['val_categorical_accuracy'],'-o',label='Val ACC',color='#1f77b4')\n x = np.argmax( history.history['val_categorical_accuracy'] ); y = np.max( history.history['val_categorical_accuracy'] )\n xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max acc\\n%.2f'%y,size=14)\n plt.ylabel('ACC',size=14); plt.xlabel('Epoch',size=14)\n plt.legend(loc=2)\n plt2 = plt.gca().twinx()\n plt2.plot(np.arange(EPOCHS[fold]),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n plt2.plot(np.arange(EPOCHS[fold]),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n plt.ylabel('Loss',size=14)\n plt.title('FOLD %i - Image Size %i, EfficientNet B%i'%\n (fold+1,IMG_SIZES[fold],EFF_NETS[fold]),size=18)\n plt.legend(loc=3)\n plt.show() \n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Calculate OOF ACC"},{"metadata":{},"cell_type":"markdown","source":"Las predicciones OOF (out of fold) se guardan en el disco. Si desea ensamblar varios modelos, use el OOF para determinar cuáles son los mejores pesos para combinar sus modelos. Elija pesos que maximicen la puntuación de CV OOF cuando se utilizan para mezclar OOF. Luego, use esos mismos pesos para combinar sus predicciones de prueba."},{"metadata":{"id":"hW3GKx7XFtKp"},"cell_type":"markdown","source":"#### Predicciones sobre el conjunto de validación"},{"metadata":{"id":"rDX2i7eawvZz","trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF ACC\n_oof_tar = np.argmax(oof_tar, axis=2)\n_oof_pred = np.argmax(oof_pred, axis=2)\n\noof = np.concatenate(_oof_pred)\ntrue = np.concatenate(_oof_tar)\n\nnames = np.concatenate(oof_names)\nfolds = np.concatenate(oof_folds)\n\nacc = accuracy_score(true, oof)\nprint('Overall OOF ACC with TTA = %.3f'%acc)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(\n image_name = names, target=true, pred = oof, fold=folds[:, 0]))\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fase 5: Submit to Kaggle"},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, augment=False, repeat=False, dim=IMG_SIZES[fold],\n labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(dict(image_name=image_names, label=np.argmax(preds, axis=1)))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}