{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1. Project Topic and Goal\n\nThis project is about developing an algorithm to identify metastatic cancer in small image patches extracted from larger digital pathology scans. The dataset used for this competition is a modified version of the PatchCamelyon (PCam) benchmark dataset, which focuses on the detection of metastasis in cancer patients.\n\nThe primary problem that this project aims to solve is the automated identification of metastatic cancer in digital pathology images. This will be done by creating deep learning models that can accurately classify whether a given image patch contains evidence of cancer metastasis or not. The dataset provided for this competition simulates this task by presenting binary image classification challenges similar to well-known benchmark datasets like CIFAR-10 and MNIST.\n\nThe submissions are evaluated on the area under the ROC curve between the predicted probability and the observed target.","metadata":{}},{"cell_type":"markdown","source":"# 2. Data\n\nThe dataset consists of many small pathology images for classification. Each image has an image id, and the train_labels.csv file offers the true labels for images in the train folder. The objective is to predict labels for images in the test folder. Positive labels indicate the presence of tumor tissue in the central 32x32px section of a patch (with tumor tissue in the surrounding area not affecting the label). This outer region supports fully-convolutional models that maintain consistency when applied to whole-slide images, without zero-padding.\n\nWhile the original PCam dataset included duplicate images due to random sampling, the version used in this competition has no duplicates. The split ratio of train and test data remains consistent with the PCam benchmark.\n\nTrain and test data combined exhibit a size of rougly 7.76 GB which is quite large. There are 220,000 training train images and 57,000 test images, each represented in a .tif format. TIFF (Tagged Image File Format) is a versatile image file format used for various purposes, including photography or medical imaging. Even compressed it maintains high image quality which leads to larger file sizes that can impact the required storage size compared to formats like JPEG or PNG.\n\nFor evaluation we are required to create a submission file consisting of a 2-column table. First, there will be the ID of the test images and second the predicted label (1 = positive, 0 = negative).","metadata":{}},{"cell_type":"markdown","source":"# 3. Import Python Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nimport cv2\nimport gc\n\nfrom PIL import Image\ntrain_on_gpu = True\n\nfrom sklearn.utils import resample\nfrom sklearn.utils import shuffle\nfrom sklearn.metrics import roc_auc_score\n\nimport torchvision.transforms as transforms\nfrom torch.utils.data.sampler import SubsetRandomSampler\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import TensorDataset, DataLoader, Dataset\nimport torchvision\nimport torch.optim as optim\nimport torchvision.models as models\n\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import EarlyStopping","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:03.604638Z","iopub.execute_input":"2023-08-17T07:05:03.605381Z","iopub.status.idle":"2023-08-17T07:05:18.455279Z","shell.execute_reply.started":"2023-08-17T07:05:03.605336Z","shell.execute_reply":"2023-08-17T07:05:18.454151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# helper method for clearing GPU memory\ndef clear_memory():\n    gc.collect()\n    torch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:18.457669Z","iopub.execute_input":"2023-08-17T07:05:18.458358Z","iopub.status.idle":"2023-08-17T07:05:18.463426Z","shell.execute_reply.started":"2023-08-17T07:05:18.458326Z","shell.execute_reply":"2023-08-17T07:05:18.462487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. EDA and Data Preprocessing\n\n## 4.1 Data Inspection\n\nFirst let's look at our training data by loading it into a dataframe. We also load the sample submission file which we will use later on.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(\"../input/histopathologic-cancer-detection/train_labels.csv\")\ndf_sample_sub = pd.read_csv(\"../input/histopathologic-cancer-detection/sample_submission.csv\")\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:18.465147Z","iopub.execute_input":"2023-08-17T07:05:18.465878Z","iopub.status.idle":"2023-08-17T07:05:18.936493Z","shell.execute_reply.started":"2023-08-17T07:05:18.465825Z","shell.execute_reply":"2023-08-17T07:05:18.935155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first column contains the ID of an image that is also the name of the corresponding image TIFF file. The second column shows the label where a 1 indicates that the center 32x32px region of a patch contains at least one pixel of tumor tissue. ","metadata":{}},{"cell_type":"code","source":"folder_train = \"../input/histopathologic-cancer-detection/train/\"\nfolder_test = \"../input/histopathologic-cancer-detection/test/\"\n\nprint(\"Number of training images: {}\".format(len(os.listdir(folder_train))))\nprint(\"Number of test images: {}\".format(len(os.listdir(folder_test))))","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:18.938308Z","iopub.execute_input":"2023-08-17T07:05:18.939371Z","iopub.status.idle":"2023-08-17T07:05:26.869983Z","shell.execute_reply.started":"2023-08-17T07:05:18.939327Z","shell.execute_reply":"2023-08-17T07:05:26.868898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As already mentioned we are working with a really large dataset. To get an intuition what the images look like, we will print a few of them choosen randomly with the according labels.","metadata":{}},{"cell_type":"code","source":"# load the images\nimg_train = os.listdir(folder_train)\nimg_test = os.listdir(folder_test)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:26.873840Z","iopub.execute_input":"2023-08-17T07:05:26.874186Z","iopub.status.idle":"2023-08-17T07:05:27.095044Z","shell.execute_reply.started":"2023-08-17T07:05:26.874155Z","shell.execute_reply":"2023-08-17T07:05:27.093912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the first 10 images\nfig = plt.figure(figsize=(25, 4))\nfor i in range(10):\n    ax = fig.add_subplot(1, 10, i + 1, xticks=[], yticks=[])\n    im = Image.open(folder_train + img_train[i])\n    plt.imshow(im)\n    label = df_train.loc[df_train['id'] == img_train[i].split('.')[0], 'label'].values[0]\n    ax.set_title(f'#{i+1} - Label: {label}')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:27.096627Z","iopub.execute_input":"2023-08-17T07:05:27.097009Z","iopub.status.idle":"2023-08-17T07:05:28.831512Z","shell.execute_reply.started":"2023-08-17T07:05:27.096972Z","shell.execute_reply":"2023-08-17T07:05:28.830327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, I do not have the necessary domain knowledge to explain why certain pictures are labeled as positive and why some are not. But what we can see is that even positive ones among themselves differ quite alot looking at their color and structures. Interestingly, picture 5 and 9 look really similar on the first glance but they are not labeled the same. This indicates that there can be really small image features deciding whether it is a positive or negative one and our model has to recognize those tiny differences.","metadata":{}},{"cell_type":"markdown","source":"Next, let's investigate the images in more detail by looking at their pixel values. For this purpose, we will plot the distribution of the grayscale pixel values for the first 10 positive and negative images. ","metadata":{}},{"cell_type":"code","source":"# sample first 10 images for both labels\ncancer_samples = df_train[df_train['label'] == 1].head(10)\nno_cancer_samples = df_train[df_train['label'] == 0].head(10)\n\n# plot histograms of image pixel values for cancer and no cancer images\nplt.figure(figsize=(12, 6))\nfor i in range(10):\n    plt.subplot(4, 5, i + 1)\n    cancer_img = cv2.imread(folder_train + cancer_samples.iloc[i]['id'] + '.tif', cv2.IMREAD_GRAYSCALE)\n    plt.hist(cancer_img.ravel(), bins=128, color='red', alpha=0.7)\n    plt.title(f'Cancer Sample #{i+1}')\n    \n    plt.subplot(4, 5, i + 11)\n    no_cancer_img = cv2.imread(folder_train + no_cancer_samples.iloc[i]['id'] + '.tif', cv2.IMREAD_GRAYSCALE)\n    plt.hist(no_cancer_img.ravel(), bins=128, color='blue', alpha=0.7)\n    plt.title(f'No Cancer Sample #{i+1}')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:28.833042Z","iopub.execute_input":"2023-08-17T07:05:28.834656Z","iopub.status.idle":"2023-08-17T07:05:37.766504Z","shell.execute_reply.started":"2023-08-17T07:05:28.834589Z","shell.execute_reply":"2023-08-17T07:05:37.765366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The upper two rows show the first 10 images labeled as positive and the the lower two rows show the first 10 images labeled as negative. Since we are only looking at the grayscale values we can only talk about images being darker (higher pixel values) or lighter (smaller pixel values). A plot like the one on the left in the third row (No Cancer Sample # 1) signals that the image mostly consists of one color with little contrast. The second plot in the first row however (Cancer Sample # 2) suggests that the image has higher contrast and different colors. We can verify this by looking back at our corresponding printed images that were labeled as positive respectively negative (#2 and #4).\n\nWith this information we could suggest the positive images typically have a higher contrast and exhibit more different colors. We count 8 positive images with high contrast and only 5 negative images with high contrast. But overall the number of samples is too small to draw a statistically relevant conclusion.  ","metadata":{}},{"cell_type":"markdown","source":"Now, we will have a look at the label distribution to see if we are working with a balanced dataset.","metadata":{}},{"cell_type":"code","source":"# plot histogram of label distribution\ndef plot_label_dist(df):\n    plt.figure(figsize=(6, 4))\n    plt.hist(df['label'], bins=2, edgecolor='black', alpha=0.7)\n    plt.xticks(np.arange(2), ['No Cancer', 'Cancer'])\n    plt.xlabel('Labels')\n    plt.ylabel('Count')\n    plt.title('Label Distribution')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:37.767908Z","iopub.execute_input":"2023-08-17T07:05:37.768944Z","iopub.status.idle":"2023-08-17T07:05:37.776018Z","shell.execute_reply.started":"2023-08-17T07:05:37.768903Z","shell.execute_reply":"2023-08-17T07:05:37.774677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_label_dist(df_train)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:37.777985Z","iopub.execute_input":"2023-08-17T07:05:37.778392Z","iopub.status.idle":"2023-08-17T07:05:38.051258Z","shell.execute_reply.started":"2023-08-17T07:05:37.778356Z","shell.execute_reply":"2023-08-17T07:05:38.050179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate the imbalance ratios\ndef calc_imbalance(df_train):\n    \n    df_train_cancer = df_train[df_train['label'] == 1]\n    df_train_no_cancer = df_train[df_train['label'] == 0]\n    \n    cancer = len(df_train_cancer)\n    no_cancer = len(df_train_no_cancer)\n\n    imbalance_ratio = no_cancer / cancer\n    cancer_ratio = cancer / (cancer + no_cancer)\n\n    print(\"Imbalance ratio:\", round(imbalance_ratio, 3))\n    print(\"Ratio of cancer:\", round(cancer_ratio, 3))","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.052956Z","iopub.execute_input":"2023-08-17T07:05:38.053648Z","iopub.status.idle":"2023-08-17T07:05:38.061235Z","shell.execute_reply.started":"2023-08-17T07:05:38.053606Z","shell.execute_reply":"2023-08-17T07:05:38.060277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"calc_imbalance(df_train)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.063127Z","iopub.execute_input":"2023-08-17T07:05:38.063557Z","iopub.status.idle":"2023-08-17T07:05:38.091691Z","shell.execute_reply.started":"2023-08-17T07:05:38.063492Z","shell.execute_reply":"2023-08-17T07:05:38.090518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, there is a remarkable imbalance present in the data that we should take care of. If we would not deal with this problem we would actually train our model to be biased and to predict 'No Cancer' most of the time. We would obtain a model that cannot predict 'Cancer' in a reliable way. In fact, there are multiple ways to deal with imbalanced data and prevent biased model performance. A few of them are:\n\n* Resampling (Oversampling / Undersampling)\n* Class Weighting during model building\n* Focus other evaluation metrics\n\nSince the evaluation metric is given by the competition we cannot change that. For this project we will use the resampling approach.","metadata":{}},{"cell_type":"markdown","source":"## 4.2 Balancing the Dataset\n\nWhen balancing the dataset using resampling we have two choices. We could either perform oversampling by increasing the number of instances in the minority class or undersampling by reducing the number of instances in the majority class. Both options have their pros and cons. One thing to consider here is that undersampling typically reduces the risk of overfitting and focuses more on learning the minority class. All this is can be achieved at the cost of some lost information in the majority class. Now, our minority class represents the images where cancer is present and since the main interesent lies in detecting those positive cases we will apply undersampling on our data instead of oversampling.\n\nAn additional advantage that comes for free doing so, is the fact that we are safer when it comes to memory issues by dropping some of our samples. We will even reduce the training size down to 50,000 samples per label. Ultimately, to avoid potential ordering in the image files that could introduce bias into our model we randomly shuffle our final dataset.","metadata":{}},{"cell_type":"code","source":"# sample positive and negative images\nsample_size = 50000\ndf_train_neg = df_train[df_train['label'] == 0].sample(sample_size, random_state=42)\ndf_train_pos = df_train[df_train['label'] == 1].sample(sample_size, random_state=42)\n\n# create a new shuffeled training dataset\ndf_train_sample = shuffle(pd.concat([df_train_pos, df_train_neg], axis=0).reset_index(drop=True))","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.093621Z","iopub.execute_input":"2023-08-17T07:05:38.094021Z","iopub.status.idle":"2023-08-17T07:05:38.146598Z","shell.execute_reply.started":"2023-08-17T07:05:38.093976Z","shell.execute_reply":"2023-08-17T07:05:38.145363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_label_dist(df_train_sample)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.148102Z","iopub.execute_input":"2023-08-17T07:05:38.148523Z","iopub.status.idle":"2023-08-17T07:05:38.411315Z","shell.execute_reply.started":"2023-08-17T07:05:38.148480Z","shell.execute_reply":"2023-08-17T07:05:38.410265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"calc_imbalance(df_train_sample)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.418203Z","iopub.execute_input":"2023-08-17T07:05:38.418564Z","iopub.status.idle":"2023-08-17T07:05:38.437217Z","shell.execute_reply.started":"2023-08-17T07:05:38.418534Z","shell.execute_reply":"2023-08-17T07:05:38.436014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the resulting dataset is now balanced and contains a total of 100,000 images that we will use for model building. But before we can actually build our neural networks, we have to prepare our data appropriately since we will work with the PyTorch library.","metadata":{}},{"cell_type":"markdown","source":"## 4.3 Plan of Analysis\n\nBased on our EDA so far, it makes sense to use the undersampled (balanced) dataset during model building. Before we can actually start with model building there are a few necessary steps to perform.\n\n* Data augmentation\n* Normalization\n* Splitting into training and validation set\n\nData augmentation is a technique used to artificially increase the size of a training dataset by creating new examples through various transformations. These transformations can include rotations, flips, cropping, resizing, and changes in color intensity. This helps to improve the generalization and robustness of a model by exposing it to a broader range of variations present in real-world data.\n\nNormalization is another preprocessing step oftentimes performed in image-data related problems that involves scaling the pixel values to values on a smaller range (originally 0 to 255). This helps in stabilizing neural network training and can lead to faster and more accurate convergence of the underlying optimization algorithms.\n\nSplitting the data into training and validation set helps us estimate how well the trained model will generalize to previously unseen data. It can also avoid overfitting by stopping the training process early if the validation performance starts to get worse. ","metadata":{}},{"cell_type":"markdown","source":"### 4.3.1 Data augmentation and normalization\n\nAfter doing some research on the PyTorch library it becomes clear, that the data augmentation and normalization can be done using built-in features of this library. In order to avoid duplicated code (since we have to do similar things for the test data later on) we will create a custom class that wraps our dataframe in such a way that is can processed by PyTorch. We have to inherit from the 'Dataset' class and and are forced to implement the 'len' and 'getitem' methods. This is a quite similar to the Decorator pattern commonly used in software architectures.\n\nAn example can be seen here: https://pytorch.org/tutorials/beginner/basics/data_tutorial.html","metadata":{}},{"cell_type":"code","source":"# wrapper class for PyTorch dataset\nclass PyTorchData(Dataset):\n    \n    # set the necessary super class properties\n    def __init__(self, df, folder = './', transform=None):\n        super().__init__()\n        self.df = df.values\n        self.data_dir = folder\n        self.transform = transform\n\n    # returns the length of the dataset\n    def __len__(self):\n        return len(self.df)\n    \n    # returns the image with the given index and applies a transformation (if specified)\n    def __getitem__(self, index):\n        img_name,label = self.df[index]\n        img_path = os.path.join(self.data_dir, img_name+'.tif')\n        image = cv2.imread(img_path)\n        if self.transform is not None:\n            image = self.transform(image)\n        return image, label","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.438678Z","iopub.execute_input":"2023-08-17T07:05:38.439160Z","iopub.status.idle":"2023-08-17T07:05:38.448682Z","shell.execute_reply.started":"2023-08-17T07:05:38.439120Z","shell.execute_reply":"2023-08-17T07:05:38.447353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we can define the transformations peformed during data augmentation. We will define some simple flips and rotations using the 'transforms' object from PyTorch library. After definition is done we can create a new object of our custom class by passing the transform object into the constructor.","metadata":{}},{"cell_type":"code","source":"transform_train = transforms.Compose([\n    transforms.ToPILImage(),\n    transforms.RandomHorizontalFlip(), \n    transforms.RandomVerticalFlip(),\n    transforms.RandomRotation(20),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])\n])\n\ntrain_torch = PyTorchData(df_train_sample, folder_train, transform_train)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.450404Z","iopub.execute_input":"2023-08-17T07:05:38.451132Z","iopub.status.idle":"2023-08-17T07:05:38.472699Z","shell.execute_reply.started":"2023-08-17T07:05:38.451095Z","shell.execute_reply":"2023-08-17T07:05:38.471518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.3.2 Data splitting\n\nNow we will split our data into training and validation set. First, we determine the indices of the training and validation samples while using 15% of our data for validation. Then we pass them to another PyTorch object with the ability of random subset sampling. Finally we pass the previously created dataset, a batch size and the sampler to a so-called 'DataLoader'.\n\nThe 'DataLoader' allows us to access features and labels of our dataset on a per-sample basis. During model training, it's common to work with samples in minibatches, shuffle data per epoch to mitigate overfitting, and leverage Python's multiprocessing to accelerate data retrieval. The 'DataLoader' simplifies this process for us.","metadata":{}},{"cell_type":"code","source":"batch_size = 128\n\n# set training and validation indices \nindices = list(range(len(train_torch)))\nsplit = int(np.floor(0.15 * len(train_torch)))\ntrain_idx, valid_idx = indices[split:], indices[:split]\n\n# random samplers\ntrain_sampler = SubsetRandomSampler(train_idx)\nvalid_sampler = SubsetRandomSampler(valid_idx)\n\n# prepare data loaders\ntrain_loader = DataLoader(train_torch, batch_size=batch_size, sampler=train_sampler)\nvalid_loader = DataLoader(train_torch, batch_size=batch_size, sampler=valid_sampler)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.474653Z","iopub.execute_input":"2023-08-17T07:05:38.475066Z","iopub.status.idle":"2023-08-17T07:05:38.486917Z","shell.execute_reply.started":"2023-08-17T07:05:38.475024Z","shell.execute_reply":"2023-08-17T07:05:38.485610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.3.3 Preparing test data\n\nAfter setting up our training data, we perform similar steps to prepare our test data. Of course this time we do not need to split the data or apply any transformations like rotation or flipping. By using the sample submission CSV file we get access to all the image ID's while still having a dataframe format. ","metadata":{}},{"cell_type":"code","source":"transform_test = transforms.Compose([\n    transforms.ToPILImage(),\n    transforms.ToTensor(),\n    transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])\n])\n\ntest_torch = PyTorchData(df_sample_sub, folder_test, transform_test)\ntest_loader = DataLoader(test_torch, batch_size=batch_size, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.488870Z","iopub.execute_input":"2023-08-17T07:05:38.489614Z","iopub.status.idle":"2023-08-17T07:05:38.504488Z","shell.execute_reply.started":"2023-08-17T07:05:38.489573Z","shell.execute_reply":"2023-08-17T07:05:38.503272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Model Building and Hyperparameter Tuning\n\n## 5.1 Basic CNN model\n\nFirst, we will check whether we can run our neural networks on a GPU. This will help to reduce computing time.","metadata":{}},{"cell_type":"code","source":"clear_memory()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.506559Z","iopub.execute_input":"2023-08-17T07:05:38.507236Z","iopub.status.idle":"2023-08-17T07:05:38.958840Z","shell.execute_reply.started":"2023-08-17T07:05:38.507192Z","shell.execute_reply":"2023-08-17T07:05:38.957532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\nprint(\"Device: \", device)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:38.960574Z","iopub.execute_input":"2023-08-17T07:05:38.960935Z","iopub.status.idle":"2023-08-17T07:05:39.004798Z","shell.execute_reply.started":"2023-08-17T07:05:38.960899Z","shell.execute_reply":"2023-08-17T07:05:39.003548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we define our network structure. The first layer captures the 32 pixels as required by the competition. The following architecture consists of several convolutional blocks with increasing numbers of channels, followed by a fully connected section for classification. Batch normalization and ReLU activation functions are used to enhance training stability and convergence. Max-pooling is used to reduce spatial dimensions, and dropout is applied to reduce overfitting.","metadata":{}},{"cell_type":"code","source":"class CNN(nn.Module):\n    def __init__(self):\n        super(CNN,self).__init__()\n        \n        self.conv1 = nn.Sequential(\n                        nn.Conv2d(3, 32, 3, stride=1, padding=1),\n                        nn.BatchNorm2d(32),\n                        nn.ReLU(inplace=True),\n                        nn.MaxPool2d(2,2))\n        \n        self.conv2 = nn.Sequential(\n                        nn.Conv2d(32, 64, 3, stride=1, padding=1),\n                        nn.BatchNorm2d(64),\n                        nn.ReLU(inplace=True),\n                        nn.MaxPool2d(2,2))\n        \n        self.conv3 = nn.Sequential(\n                        nn.Conv2d(64, 128, 3, stride=1, padding=1),\n                        nn.BatchNorm2d(128),\n                        nn.ReLU(inplace=True),\n                        nn.MaxPool2d(2,2))\n        \n        self.conv4 = nn.Sequential(\n                        nn.Conv2d(128, 256, 3, stride=1, padding=1),\n                        nn.BatchNorm2d(256),\n                        nn.ReLU(inplace=True),\n                        nn.MaxPool2d(2,2))\n        \n        self.conv5 = nn.Sequential(\n                        nn.Conv2d(256, 512, 3, stride=1, padding=1),\n                        nn.BatchNorm2d(512),\n                        nn.ReLU(inplace=True),\n                        nn.MaxPool2d(2,2))\n        \n        \n        self.fc=nn.Sequential(\n                nn.Linear(512*3*3, 256),\n                nn.ReLU(inplace=True),\n                nn.BatchNorm1d(256),\n                nn.Dropout(0.4),\n                nn.Linear(256, 1))\n        \n    def forward(self,x):\n        x=self.conv1(x)\n        x=self.conv2(x)\n        x=self.conv3(x)\n        x=self.conv4(x)\n        x=self.conv5(x)\n        x=x.view(x.shape[0],-1)\n        x=self.fc(x)\n        return x","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:39.006462Z","iopub.execute_input":"2023-08-17T07:05:39.006864Z","iopub.status.idle":"2023-08-17T07:05:39.024838Z","shell.execute_reply.started":"2023-08-17T07:05:39.006824Z","shell.execute_reply":"2023-08-17T07:05:39.022116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create CNN\nmodel = CNN().to(device)\nprint(model)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:39.026613Z","iopub.execute_input":"2023-08-17T07:05:39.027435Z","iopub.status.idle":"2023-08-17T07:05:44.582119Z","shell.execute_reply.started":"2023-08-17T07:05:39.027394Z","shell.execute_reply":"2023-08-17T07:05:44.580995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We perform a manual hyperparameter tuning on the learning rate. In order to stay within computation time limits, we will only look at the three different values listed below. For the same reason we will only iterate over 10 epochs. We use a binary cross-entropy loss function.","metadata":{}},{"cell_type":"code","source":"# hyperparameters to tune\nlearning_rates = [0.001, 0.0005, 0.0002]","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:44.583752Z","iopub.execute_input":"2023-08-17T07:05:44.584121Z","iopub.status.idle":"2023-08-17T07:05:44.588934Z","shell.execute_reply.started":"2023-08-17T07:05:44.584084Z","shell.execute_reply":"2023-08-17T07:05:44.587709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save final results\nresults = []\nbest_model_idx = None\nbest_auc = 0.0\nbest_model_cnn = None\n\n# iterate all values for the hyperparameter\nfor idx, lr in enumerate(learning_rates):\n    \n    # use GPU if available\n    model = CNN().to(device)\n    \n    # define optimizer and loss function\n    optimizer = optim.Adam(model.parameters(), lr=lr)\n    criterion = nn.BCEWithLogitsLoss()\n\n    # save results over epochs \n    train_losses = []\n    valid_losses = []\n    valid_aucs = []   \n    n_epochs = 10\n\n    # iterate all epochs\n    for epoch in range(1, n_epochs+1):\n        \n        valid_aucs_epoch = [] \n        model.train()\n        train_loss = 0.0\n        \n        # process the training images\n        for data, target in train_loader:\n            optimizer.zero_grad()\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            loss = criterion(output.view(-1), target.float())\n            loss.backward()\n            optimizer.step()\n            train_loss += loss.item() * data.size(0)\n        \n        model.eval()\n        valid_loss = 0.0\n        \n        # process the validation images\n        for data, target in valid_loader:\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            loss = criterion(output.view(-1), target.float())\n            valid_loss += loss.item() * data.size(0)\n            y_actual = target.data.cpu().numpy()\n            y_pred = torch.sigmoid(output).detach().cpu().numpy()\n            valid_aucs_epoch.append(roc_auc_score(y_actual, y_pred))\n        \n        # determine values for current epoch\n        train_loss /= len(train_loader.sampler)\n        valid_loss /= len(valid_loader.sampler)\n        valid_auc = np.mean(valid_aucs_epoch)\n        \n        train_losses.append(train_loss)\n        valid_losses.append(valid_loss)\n        valid_aucs.append(valid_auc)\n\n        print('Learning Rate: {:.6f} | Epoch: {} | Training Loss: {:.6f} | Validation Loss: {:.6f} | Validation AUC: {:.4f}'.format(lr, epoch, train_loss, valid_loss, valid_auc))\n    \n    # save results for current value of hyperparameter\n    results.append({'learning_rate': lr, 'train_losses': train_losses, 'valid_losses': valid_losses, 'valid_aucs': valid_aucs})\n    \n    # save best model\n    avg_auc = np.mean(valid_aucs)\n    if avg_auc > best_auc:\n        best_auc = avg_auc\n        best_model_idx = idx\n        best_model_cnn = model\n    \n# print best hyperparameter\nprint(\"Best learning rate according to hyperparameter search: \", learning_rates[best_model_idx])","metadata":{"execution":{"iopub.status.busy":"2023-08-17T07:05:44.590867Z","iopub.execute_input":"2023-08-17T07:05:44.591529Z","iopub.status.idle":"2023-08-17T09:03:22.943529Z","shell.execute_reply.started":"2023-08-17T07:05:44.591481Z","shell.execute_reply":"2023-08-17T09:03:22.942320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's plot the results.","metadata":{}},{"cell_type":"code","source":"# plot training and validation loss over epochs for best model\ndef plot_losses(train_losses, valid_losses, title):\n    plt.figure(figsize=(10, 5))\n    plt.plot(train_losses, label=\"Train Loss\")\n    plt.plot(valid_losses, label=\"Valid Loss\")\n    plt.xlabel('Epochs')\n    plt.ylabel('Loss')\n    plt.title(title)\n    plt.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:22.945111Z","iopub.execute_input":"2023-08-17T09:03:22.946577Z","iopub.status.idle":"2023-08-17T09:03:22.954426Z","shell.execute_reply.started":"2023-08-17T09:03:22.946534Z","shell.execute_reply":"2023-08-17T09:03:22.953268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_result = results[best_model_idx]\nplot_losses(best_result['train_losses'], best_result['valid_losses'], 'Training and Validation Loss over Epochs (best model)')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:22.958317Z","iopub.execute_input":"2023-08-17T09:03:22.958647Z","iopub.status.idle":"2023-08-17T09:03:23.308341Z","shell.execute_reply.started":"2023-08-17T09:03:22.958617Z","shell.execute_reply":"2023-08-17T09:03:23.307227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot validation auc over epochs for best model\ndef plot_aucs(aucs, title):\n    plt.figure(figsize=(10, 5))\n    plt.plot(aucs, label=\"Validation AUC\")\n    plt.xlabel('Epochs')\n    plt.ylabel('Validation AUC')\n    plt.title(title)\n    plt.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:23.310172Z","iopub.execute_input":"2023-08-17T09:03:23.310947Z","iopub.status.idle":"2023-08-17T09:03:23.318352Z","shell.execute_reply.started":"2023-08-17T09:03:23.310905Z","shell.execute_reply":"2023-08-17T09:03:23.317123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_result = results[best_model_idx]\nplot_aucs(best_result['valid_aucs'], 'Validation AUC over Epochs (best model)')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:23.319847Z","iopub.execute_input":"2023-08-17T09:03:23.320556Z","iopub.status.idle":"2023-08-17T09:03:23.660248Z","shell.execute_reply.started":"2023-08-17T09:03:23.320517Z","shell.execute_reply":"2023-08-17T09:03:23.659261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results seem to be reasonable and pretty good. We will discuss the results in more detail in the next section. But before, we will try out another model architecture.","metadata":{}},{"cell_type":"markdown","source":"## 5.2 Dense (Pretrained) Network\n\nFor our second model, we do not perform a hyperparameter search again since the whole notebook would need too much GPU time. Therefore, we reuse the previously found best learning rate of our first CNN architecture. For the same reason we also perform only 10 epochs as with our first model. And the same as before, we use a binary cross-entropy loss function.\n\nThe main difference will be that we now build our architecture upon a given pretrained dense network architecture where each layer is connected to every other layer in a feed-forward fashion. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several advantages compared to standard CNNs. They mitigate the vanishing-gradient problem, and reduce the number of parameters to name just a few.","metadata":{}},{"cell_type":"code","source":"clear_memory()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:23.661828Z","iopub.execute_input":"2023-08-17T09:03:23.662951Z","iopub.status.idle":"2023-08-17T09:03:24.015052Z","shell.execute_reply.started":"2023-08-17T09:03:23.662910Z","shell.execute_reply":"2023-08-17T09:03:24.013756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DenseNetModified(nn.Module):\n    def __init__(self):\n        super(DenseNetModified, self).__init__()\n        \n        # use a pretrained dense net architecture\n        self.densenet = models.densenet121(pretrained=True)\n        \n        num_features = self.densenet.classifier.in_features\n        self.densenet.classifier = nn.Sequential(\n            nn.Linear(num_features, 512),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.5),\n            nn.Linear(512, 1)\n        )\n        \n    def forward(self, x):\n        return self.densenet(x)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:24.016881Z","iopub.execute_input":"2023-08-17T09:03:24.017337Z","iopub.status.idle":"2023-08-17T09:03:24.027505Z","shell.execute_reply.started":"2023-08-17T09:03:24.017268Z","shell.execute_reply":"2023-08-17T09:03:24.026436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# reuse best learning rate from first CNN\nlr = learning_rates[best_model_idx]\nmodel_dense = DenseNetModified().to(device)\n\n# set optimizer and loss function\noptimizer = optim.Adam(model_dense.parameters(), lr=lr)\ncriterion = nn.BCEWithLogitsLoss()\n\n# store final results\ntrain_losses_dense = []\nvalid_losses_dense = []\nvalid_aucs_dense = []\nn_epochs = 10\n\n# iterate all epochs\nfor epoch in range(1, n_epochs+1):\n\n    valid_aucs_epoch = [] \n    model_dense.train()\n    train_loss = 0.0\n \n    # process training images\n    for data, target in train_loader:\n        optimizer.zero_grad()\n        data, target = data.to(device), target.to(device)\n        output = model_dense(data)\n        loss = criterion(output.view(-1), target.float())\n        loss.backward()\n        optimizer.step()\n        train_loss += loss.item() * data.size(0)\n\n    model_dense.eval()\n    valid_loss = 0.0\n\n    # process validation images\n    for data, target in valid_loader:\n        data, target = data.to(device), target.to(device)\n        output = model_dense(data)\n        loss = criterion(output.view(-1), target.float())\n        valid_loss += loss.item() * data.size(0)\n        y_actual = target.data.cpu().numpy()\n        y_pred = torch.sigmoid(output).detach().cpu().numpy()\n        valid_aucs_epoch.append(roc_auc_score(y_actual, y_pred))\n\n    # store final results\n    train_loss /= len(train_loader.sampler)\n    valid_loss /= len(valid_loader.sampler)\n    valid_auc = np.mean(valid_aucs_epoch)\n\n    train_losses_dense.append(train_loss)\n    valid_losses_dense.append(valid_loss)\n    valid_aucs_dense.append(valid_auc)\n\n    print('Learning Rate: {:.6f} | Epoch: {} | Training Loss: {:.6f} | Validation Loss: {:.6f} | Validation AUC: {:.4f}'.format(lr, epoch, train_loss, valid_loss, valid_auc))","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:03:24.029151Z","iopub.execute_input":"2023-08-17T09:03:24.029710Z","iopub.status.idle":"2023-08-17T09:50:29.695586Z","shell.execute_reply.started":"2023-08-17T09:03:24.029673Z","shell.execute_reply":"2023-08-17T09:50:29.694497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_losses(train_losses_dense, valid_losses_dense, 'Training and Validation Loss over Epochs')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:29.697301Z","iopub.execute_input":"2023-08-17T09:50:29.697739Z","iopub.status.idle":"2023-08-17T09:50:30.007135Z","shell.execute_reply.started":"2023-08-17T09:50:29.697698Z","shell.execute_reply":"2023-08-17T09:50:30.006199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_aucs(valid_aucs_dense, 'Validation AUC over Epochs')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:30.008735Z","iopub.execute_input":"2023-08-17T09:50:30.009393Z","iopub.status.idle":"2023-08-17T09:50:30.309122Z","shell.execute_reply.started":"2023-08-17T09:50:30.009357Z","shell.execute_reply":"2023-08-17T09:50:30.308229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again the results seem to be quite good. Let's dive into the last section where we discuss the results in detail and compare our two model architectures.","metadata":{}},{"cell_type":"markdown","source":"# 6. Results and Analysis\n\n## 6.1 Model summary and comparison\n\nAs we have already seen both of our models seem to work quite well on the training data. Now, it's time to compare them and select the better one for predicting on the test data. First, lets have a closer look at the training and validation losses.\n\nAs a reminder, we did a hyperparameter tuning on the learning rate for the first model where we tried out three different values. The best one selected was the following.","metadata":{}},{"cell_type":"code","source":"print(\"Best learning rate according to hyperparameter search on Classic CNN model: \", learning_rates[best_model_idx])","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:30.310455Z","iopub.execute_input":"2023-08-17T09:50:30.310954Z","iopub.status.idle":"2023-08-17T09:50:30.316850Z","shell.execute_reply.started":"2023-08-17T09:50:30.310914Z","shell.execute_reply":"2023-08-17T09:50:30.315794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to stay within computation time limits we reused this value for the second dense net model. Now, let's compare the losses of our two models.","metadata":{}},{"cell_type":"code","source":"plot_losses(best_result['train_losses'], best_result['valid_losses'], 'Training and Validation Loss over Epochs - Classic CNN')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:30.318509Z","iopub.execute_input":"2023-08-17T09:50:30.319169Z","iopub.status.idle":"2023-08-17T09:50:30.647303Z","shell.execute_reply.started":"2023-08-17T09:50:30.319132Z","shell.execute_reply":"2023-08-17T09:50:30.646227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_losses(train_losses_dense, valid_losses_dense, 'Training and Validation Loss over Epochs - Dense Net')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:30.648867Z","iopub.execute_input":"2023-08-17T09:50:30.649266Z","iopub.status.idle":"2023-08-17T09:50:30.961103Z","shell.execute_reply.started":"2023-08-17T09:50:30.649227Z","shell.execute_reply":"2023-08-17T09:50:30.960203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first plot shows the classic CNN architecture. We observe a decreasing training loss as the epoch number increases which is what we would expect. The validation loss looks more unsteady but overall also decreases when reaching the higher epoch numbers. As already mentioned, we could potentially improve here if we would allow for more epochs. But since there is time limit on Kaggle for using the GPU we restrict ourselves a little bit for this project.\n\nThe second architecture clearly shows better results. Already after the first epoch there is remarkable difference in training loss, which means the dense network architecture works much better. The same can be said about higher epochs. We constantly get smaller loss values. The reason for the better performance is probably the fact that the connected layer structure of the network can capture more complex patterns in the dataset as compared to the classic CNN structure. Also the fact that the dense network is pretrained gives the model a headstart and helps to optimize faster.\n\nLet's see if our dense model also outperforms when looking at the AUC scores.","metadata":{}},{"cell_type":"code","source":"plot_aucs(best_result['valid_aucs'], 'Validation AUC over Epochs - Classic CNN')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:30.962432Z","iopub.execute_input":"2023-08-17T09:50:30.965285Z","iopub.status.idle":"2023-08-17T09:50:31.572718Z","shell.execute_reply.started":"2023-08-17T09:50:30.965256Z","shell.execute_reply":"2023-08-17T09:50:31.569827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_aucs(valid_aucs_dense, 'Validation AUC over Epochs - Dense Net')","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:31.574339Z","iopub.execute_input":"2023-08-17T09:50:31.574729Z","iopub.status.idle":"2023-08-17T09:50:31.884895Z","shell.execute_reply.started":"2023-08-17T09:50:31.574692Z","shell.execute_reply":"2023-08-17T09:50:31.883247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now looking at the AUC scores we can basically conclude the same. The first model shows good results and an increasing score over the epochs. Similar to the losses the dense structure starts out at better values than the classic CNN. Even if the increase over epochs is not as significant as with the first model, the absolute values are still slightly higher.\n\nSummarizing the final results everything seems to be pretty easy. But of course, there were some problems during model building and training, e.g. computing time limits, understanding model architectures, and debugging the training process. We will talk about these topics later on in the Learnings and Takeaways section.","metadata":{}},{"cell_type":"markdown","source":"## 6.2 Predicting on test data\n\nWe have seen that the dense model performed the best. We will use it now to predict on the test data and create our submission file.","metadata":{}},{"cell_type":"code","source":"clear_memory()","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:31.886479Z","iopub.execute_input":"2023-08-17T09:50:31.886862Z","iopub.status.idle":"2023-08-17T09:50:31.891983Z","shell.execute_reply.started":"2023-08-17T09:50:31.886826Z","shell.execute_reply":"2023-08-17T09:50:31.890758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# turn of gradients\nmodel_dense.eval()\npreds = []\n\n# iterate all test images\nfor batch_i, (data, target) in enumerate(test_loader):\n    data, target = data.to(device), target.to(device)\n    output = model_dense(data)\n\n    pr = output.detach().cpu().numpy()\n    for i in pr:\n        preds.append(i)\n\n# add predicted labels to submission file    \ndf_sample_sub['label'] = preds","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:31.893704Z","iopub.execute_input":"2023-08-17T09:50:31.894111Z","iopub.status.idle":"2023-08-17T09:50:31.902130Z","shell.execute_reply.started":"2023-08-17T09:50:31.894078Z","shell.execute_reply":"2023-08-17T09:50:31.901061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# convert probabilities to float\nfor i in range(len(df_sample_sub)):\n    df_sample_sub.label[i] = np.float(df_sample_sub.label[i]) ","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:31.909809Z","iopub.execute_input":"2023-08-17T09:50:31.910106Z","iopub.status.idle":"2023-08-17T09:50:31.914318Z","shell.execute_reply.started":"2023-08-17T09:50:31.910080Z","shell.execute_reply":"2023-08-17T09:50:31.913201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create submission file\ndf_sample_sub.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-08-17T09:50:31.915820Z","iopub.execute_input":"2023-08-17T09:50:31.916396Z","iopub.status.idle":"2023-08-17T09:50:31.924599Z","shell.execute_reply.started":"2023-08-17T09:50:31.916362Z","shell.execute_reply":"2023-08-17T09:50:31.923505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![image.png](attachment:405d7b16-2310-41c7-950a-9c282d5800b0.png)","metadata":{},"attachments":{"405d7b16-2310-41c7-950a-9c282d5800b0.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAABLcAAAECCAYAAAAMxzVvAAAgAElEQVR4Aezdj09c56Hn//1LKtCX7xd9NZKlpZuVvK2rqEV1vewXXyGhjWxxv2IvUriOYq+jei2nLqW1uKT6oukVjnNr6lrm4pW9eJ1F9CaOuw6kdaGpXOZ2HcjGgfba0Jp0UlPPJc4dN6Sfr55zznPmOYczMMAAg/MeyWV+nPM8z3k9Z6Y9nz7Pc/6VeCCAAAIIIIAAAggggAACCCCAAAIIILBDBf7VDm03zUYAAQQQQAABBBBAAAEEEEAAAQQQQECEW5wECCCAAAIIIIAAAggggAACCCCAAAI7VoBwa8d2HQ1HAAEEEEAAAQQQQAABBBBAAAEEECDc4hxAAAEEEEAAAQQQQAABBBBAAAEEENixAoRbO7braDgCCCCAAAIIIIAAAggggMB2CHz06GPN/va+3nn3jt7+5a/0k7FfaOSnP9ebPxnnXxkNjKmxNcbG2pgb++1+0P8bO883o18Jt7b7W0H9CCCAAAIIIIAAAggggAACO0Lg/vzvNfGrSf3s57/U/74zo/sfZLW4+JE++eQT/fnPf94Rx7CTGmlMja0xNtbG3NibPjB9sdUP+r884pvRr4Rb5ekbSkEAAQQQQAABBBBAAAEEEHhCBbIf/sEbPfSrd97Vh39YeEKPcucclukD0xdmRJfpm81+0P+bLeyXv5F+Jdzamj6iFgQQQAABBBBAAAEEEEAAgR0o8N77v9bbt/6RUKsC+86EIaZvTB9t1oP+3yzZ4uWup18Jt4p78gkCCCCAAAIIIIAAAggggMBnVCD/+LE3/c1MhWPKYeWeBKZvTB+ZqYqmz8r1oP/LJbm+ctbar4Rb63NmLwQQQAABBBBAAAEEEEAAgSdUwAQbZsrbP92be0KP8Mk7LNNXps/KEXDR/5VzfpTar4RbldNntAQBBBBAAAEEEEAAAQQQQKACBMwoIIKtCuiINTbB9Jnpu40+6P+NCpZ3/1L6lXCrvOaUhgACCCCAAAIIIIAAAgggsIMFzBpLZpobj50pYPpuI2tw0f+V2e+r9SvhVmX2G61CAAEEEEAAAQQQQAABBBDYYgHvrni3/pE1trbYvZzVmbWazCLz67mLIv1fzp4ob1mr9SvhVnm9KQ0BBBBAAAEEEEAAAQQQQGCHCpg1m8yd2njsbAHvbnu//NWaD4L+XzPZlu6wUr8Sbm1pV1AZAggggAACCCCAAAIIIIBAJQrcn/+9fvXOu5XYNNq0DgHTl6ZPS33Q/6VKbe92xfqVcGt7+4XaEUAAAQQQQAABBBBAAAEEKkDALCLOqK0K6IgyNcH05VoWl6f/ywS/ycUU69eEcCun6ZvjGrP/7uQ2uWmbWPzCTOE4bmY0m9/EumJF5+44hjdntIMVY0e20su8ZjPOce/kc2elw+QzBBBAAAEEEEAAAQQQeKIEPnr0sX7281+W9ZgmumtUVd2tsaUNFruUV24hF/2X28KLW9P8R7H6F3LKb/S4NshSyu6mT03frvYoZ//nZkbV90KznkqZ/jf/6tT0wjmNzOSk+UG1mvdOja/WpMjn+Vy5/POavTGgnnRaPelhTW/WaWTP2UeRwyjbi6R+TQi3MuryOiDoiO5M2Rqw5QXd6g5OJnMsbbo8v3Ut8H/I7MncrYmtq3oba8rqcqs95hpV7eRzZxsVqRoBBBBAAAEEEEAAAQS2VmD2t/fLfofEsoVbNhBxr9PN87oW9WW2ZhhF9PrWXvPVqenkoKbXFGAUBkRMb8HSZuYOe6ZvV3uUp//zmnylWbWmb1K71XrSBEhp9ZxsC4KulFo7O9VgPl9TuBW7zg7PgzodOJtRbg0h4+zF5kJGsueMJleDWe/nNovZpEwgqV8Jt9bbWavsF/3yE26twsXHCCCAAAIIIIAAAggggMC2Cbzz7h3d/yBb1vrLH24164g34iatjrbdfoiyRYM47PXtnrZOL7DpeqFF9XVByFWf1kTJI4AKQU3XrbJyJxZm+tT07WqPcvT/7JU2r09q2wY0GQ/uclPqf66uECytM9xqeiEemNWo9Uqp523BvuOmpDWEYqv5Lft8k8OtpH4l3FrWC+V5w375/WGIhFvlUaUUBBBAAAEEEEAAAQQQQKD8AuYueYuLH5W14PKHW+51ZU5Dz/nhUuurhXAjNzWsvlNHdaDxkDrSAxqzs5fujuq0CcYuZoIlc3KauOgHJUNTQTI1P6o+s80Px1Uo0Sex17eRIGVpTpfbUl5g03RxrmC3lNXYxbQ62ht14IVu9Q1PBaOLTJ2dat3jt9sLat4o7Fe07YWS1/zM9Knp29UeG+7/hes6YqYh1ndrLDaYLm+nlGZH1fVvg0BwneGWGwjmhg/5YVnbcKG/cjMauditIy2Naj95TkNTtjFzupaO2Xvngj0PBjQxN6Oh7kNq6Bz1z5GiZQWaTj83tHeq70pGWRuYueHW/Lj6Tx6S2aZ/PH5mrdYzyZ8n9evmhVt2jqXpyLXMB3b3W8s83rydg+pExhbUG7a3yrTEcP+ccmsaVulj2zmwdt6x/fIvC7eceuy2yd3lvBsxcY7P2aTYU9sub4722nb1irRfxKS22s8K3VtIgr3j3qQhiMWOlfcRQAABBBBAAAEEEEAAgfUI/GTsF/rkk0/Ws2t0n0cZnX7hnCYfSYVwK6/Js0d1+tY6LshM6eG0RDfcksZOBeFWMHIn+6o/cshMiWto3Ktd3nXwXp2ekpQfVYf3OlgDbGk8XI5ozytmAyn32lEvLKlNuI6z17eRcMvsdKvbH0FmA5Z8Rj31frt21TeqYbcfftUevq6cYteLpj1BXSu23Wvd+v7D9Knp29UeG+1/035zDdzuBI1+nbFln7w+WP+0RDfc0niwDFProB9uPcqoy7H31/xKqfWKCRAT2uHtZ/ukRe3PPe2HZeb9Fcsyo76mdNqpq6HeH5VWa88Dm8Uc61RHuPaYOS+eVtd6vwdOJyb1a/nDrey4t3ia/0UKUsnqGtXublPX8IyKfp1zU7p8sjn4Ahb28xZfOzmoSRs4OgfkPTX7nWgMhmT6+9XuPqr+O3nvi+aHS+b95HArf2dYXeGQzkK9u5qOqi8hVcxe8U/aMLR6NBMZXmhPNvvlD7dbymnibEv0+Oqa1XFlBZMillV1zTpydryQisZNlnKavNKpJjtM1H6Bqmu0q6lTl8P0trBjpL2tg5o1qXKD/0MUt8vePKMDkbLr1NQ9quyS/WIEjgk/ioUaeYYAAggggAACCCCAAAIIVIbAyE9/rj//+c8bbExes5eCgKnpjE4fM9dFR9XT66/DZKarrWsB7zDcOqS+4MZvIxdPqN67zntap2+bZmc19kMzEuucRoLBULOXWvywKm3W0c7p2mHTnkb1z0iaOqM99jrxwKBmZcI4//qvY3T5Vbu9XlwWboVt84O3/NSwv87UlSn/2j8M0do05A3aKVwz2mvn1du+/m4xfWr6drXHRvvf92lR/93lNYWDQuwIrrUOAHJCwfYfBjdwuzGg40G4tKfXDycne004ZcMscxOAIOxKdWrM69Jke7tudm3boKZzeeUf5bVqWZm0n8E0D3jnjjm/Ji6Y829AEya7seFWdZv6Z/LSUk5jp4LwrAw5QVK/ljXcyt8+o6ZIKlcIi2zIVH/sumbtUDXb73PDao+EJcv3q0q16XJhxKK/p+0s+6V0/6baNHRl5QXlZ1+zPwgJ9XllpdT0SvClDNoaD7dOv7LXTzeDuu0X1H75bbjVf8FZuM1tZ3VKDb2ZZaFf/s6AWlexrG1IayI+ymxpTkPuXN5IXfY4nRM+OK5Ie1vP6PRhG2yZfQrBoJ1HbPvT/VufHlY/C8rbs5q/CCCAAAIIIIAAAgggsEMENhpuuIeZdM1kgoNl18HuTis9DwMkez1n/5oFxaPXq/m7Uxq7Mai+dLeO2MEKQZgwfaHRu3Y10xj94KtNRw6bwOGoruVm1N9oyjXPlzfGXi8WDbdS6cJN1PJZTd8a15CZ9njSDvCw15RJAYtf30ptX96i0t5JCkGS9txo//s+zYnhVlJ9a3uvYOZef5vnu1r8UYLSlE570z2fVmtnsC5XOq0jTaZPU+rx7hNYKMfmFiZYtOFW4b0Syspd13Evr0jpqeaj6jo7qJHbc4U7aNpwy44qMwec9N7aIMKtk/q1fOHWo3F1BXNnPfBUo46kBzVyY1A9h92RVSm5c4JNwmfnCof7nR3W2M1xDfXaL4L/5a09ZoYy2kdeY51uAGPuFrFX7SfT6gpuu1mbcj+3X6Zg/7ngFpw2/KlrUcfFYY0ND6ijxVnorXqvejKF5DoabqVUGwRQtbsbvQX17Alhv/z+yedvZ0avdaTdxf/sj1IwVNQeWtyyuk4HgjstRNtWI394p91RCufdeseVUsPhcxoy6f5wbLRV6kTkRyvS3lQqHAlnhpI+ZYJFM1f77oCarFdQvncipzvV7g1DLHh4x12GRLZwZDxDAAEEEEAAAQQQQAABBDZHYKPT0uKtcgOuDQVbpuAw3ApGbl3q9Edd7el2FnLPaeRkMPCibq8aWo7qeFtspIwdrXVqWENmFFfjgKaDaYVdw8M6Yq7x3DDCOSh7vRgPt/Kjnf5gD2/aoaSZgWDAS0pPNZh1n44G15D2ejwpTCmh7U5b1vI0afpa0v4b7f/i0xKTalvrewUzf+TWoDqCIKswxc9OO/TdGxob5f7rW1O4VUpZkpkF1+PeWMDM2Gs445+TSUFW0ntrpQi2T+rXsoVbkVtKJoyyMsMz/aCnRlXmlpN29FYuo8tmcTEPv2XZPGQ7j9jf15ljnLvuf/ls2BJfuM2EV5GRT/bLZDRiwVh9d2wEVL4wZM6U/9xwGKpFwy2zYFynRuwifU7H2C+/PebaF4Yj0wjdHzuzTW3neDh6K2JZnVLHjUKk57XdDufzjt0OQzWV5zRxqVPtwYl8ID4izM7JDcxsEGf2XNZec4cHt1pJk68EP47e/im1XnKH0uU10R0dxWbnTzssPEUAAQQQQAABBBBAAAEEKk5gYwuKZ9RnrsFOXC8s7C3JXPPtio3Yyr52wrv29cOGEhnCcMteDxcGiDRdMHMMzazEYbWa67TGc5oOrrXttMTwumwpoy5zjbxnr+pTNfLW2grW4tqzx7/Wa7DlxZpmrxcj4db8qDq8qXEpHXnNv3icSPsDTI7fCAaIONMSvQETSSOFSml7rD2lvkxaeDxp3431v6SFYbUbfzfrSKpoXe8Vwi17DR8OamkaCPo7q6E2M3imUX3OzSFz2azM+tv+GtrLy0keuVVCWd5a4nOavBOEBgvj6vFG/tXIa2NSkJX03ro85N38IX6jgDKFWwUkE9TYBeki7QwXsPPBvXm+kQ0SXiyZBNcdfWW/zFK4eFoQ1LQPx5IYE8Z4c07t6Cgn3LJf6mBf+0WMtGBmQA3B51XVnRoJvpvRcMsNliJ7x8KiYF5zZBM77DNon/kSeJ/P6fIB2+YgTY/s5yb3/naJ3vF9TPR1ozMckWX6yX4xzKb2x8oP4w5pKH7rUsXalfSltV9o68bIrYRe4C0EEEAAAQQQQAABBBCoNIF33r2j+x+s905uwUiXYO2qlY7NXk92ja+0VeyzZeGWv2aWt+ZW6qiumWs3O/gj1abTN8Y1duNcYemf8Losr5GT9lozFVwP2rW4zPtP+4vPx6o3L+31opmx5A1MCRYQN9ePu54bDqdc2gER9Z2D3mysy6fsLC57PV6or769Wz3mbokltT2hUSW8ZfrU9O1qj431v1+6HcBi1laLDxRRbkr9LxzVkDs+ZLVGhZ8X8pbwGj5c0L0QLJpRdLXmWrz+kPqGxzV09pC/LluJa26FZXv3H1ilrKkzQdnN6hk2M8XSwYi9YGpmUpCV9F54jGt7ktSvZQq37LA1/4sSnvCRoXD2bg3+Ni6cdxhLOU3fGla/mbbXbr4w0e390KUQbtkfBf/9pPDI3BAiGCLphS32y7Q8HPLu4hBpa6MaGnY7QVBh32i9hdAr3hX2yx9vt7tddBtbVtQyTNndHZVRjzsqzQ4BdbbJzWSCOc7+qLj6hDXN3D6ItMUMT3XK8p/G2nVyNBxpVtg0FtiFP6KFLXiGAAIIIIAAAggggAACCFSawOxv7+t/3wlGQa25ccG10r9t8Zah6UkX1jyKP+8IpgpuONySCar8gSB7TvmzgGZfPSr/DnlmyZ5DGroUrEHtXJfZOyImDuBIdWvCzrCKGUSuF73razP97ZC6hqeUc/cxd4tsssv8pNSQHlSfty5z4Zo6PzNYCN6Cu+uV0vZYk0p6afrU9O1qj431vy09r8lX/JsHmBvj2aWFek62Bf2SUlM6E84Ks3ut/jch3HIDqD3dGgvW4p4djt5YztxQbihc5D6pnKT3/BatXJY0+1q0LrMM02l7U76kICvpvdUPPnGLpH7dlHDLD3RsIpz81x3OaBZ2b3DDGjvyZ9nfYuFW4YsSOXKLt0q4VUp7bRAUDbcK7YnU6yTbXtnhHQSiW0V/IOwxxEIk54eosHfhBPTKd+dFz13Xcbtw4DK/aF/YYzLlRtrilhdWuo52JbY9LJAnCCCAAAIIIIAAAggggEBFCHz06GP97Oe/XHdbpi9G14te6RpzV4udSrbu6orvuJRXLldYM7r4hpv8ySM7Fa54PeaufJHHJrTd9Knp29UeG+1/t/zczKj6gnXAvfMgtVtNL5zTyMzy2WbufuV87t2hMX7zuXVWsFpZ+Vxuy8+5pH7dlHCr4ZS/ILxZFL7Yv4m54ES2w9lsEFPXrCOnBrxF0Cfv5jTtrtVVXQiToiHTxkZuhbfTXKG908E0vWi9hfbEz5NIWOS0290uuk0ZRm6FQxNtiFWnphe61W+GCd6eU24muoj+hsItRm65XclzBBBAAAEEEEAAAQQQ2OECE7+a1Id/WLY+yw4/qs9u801fmj4t9UH/lyq1vdsV69cyhVvR9Zhq1zBix87J9RJNZzid5QoXwfPCLydMii2O3nRx+eTViW53vS47MkpSuKidHwK5o8hsvcX+ri/cSgjfTBDl3l1yLWtuxe5aGK65Ze9+4Vk9ra7xWBI+N6gDNkRcac2txJFb0T6uCuftOlLhXPAgXFvDeeCUwlMEEEAAAQQQQAABBBBAYMsF7s//Xr96590tr5cKN0fA9KXp01If9H+pUtu7XbF+LVO4JUXu8Jdwt0TdNguO1aneW9uqTXZB+cjopfgCfEsz6m+yo5DMXyfcsovO2bAmdVRDzvp/+Tvn/LtF2M+rnXBr1bslmptNHFVtandwF8dujQQjCNcXbtUofvtXc/dIb7G3oH1uIBixrE6p9YoT3C3londyNIv+3Q5Orsg0zBZddnYzW0xfaC7csXLN4Vb8bok1auqdKqy7tZTVyEn3boo1Sl4vbHu/CNSOAAIIIIAAAggggAACCBQTMHdgY/RWMZ2d877pw/jd9EppPf1fitL2bbNSv64eboXhkBsy2edOYPRoVB3uulmpRh1JD2rk5rD6Tx2NrqkV3q5SioZFdTrQO6rp+ay3uPzpFrsQna3PCbeU10R3LEzx5rJ2q6M9aTF6p62mL2YG1OQeW12LOi4Oa+zGoPpORudL1zpT8KLtddsT7eBIaGcDrN2Naj/ZqXbnzhL+XOzmMOzzSnk0ri53VFd1Sg2Hu9WT7taRcHE+36TWXUw+NnJqV8sZjcxklTWLy/dGj8nUu7ZpicsX4jdl7Ko/pI5TR9W02x0lF/QXI7eiJwWvEEAAAQQQQAABBBBAoKIFsh/+QW/f+kf9+c9/ruh20rjiAqbvTB+avlzrg/5fq9jWbb9av5Yv3DKr9d8+E9z+0YZRCX/NXRvcEUUL13XEDcXcwKl6r7q6TzijjWJh0qOMuuoT6jBlpNo0dCW4O4RXZizcklnd/4R/+8pIndHyahvSmnAWYltfuNWt/tioKT/UMnXFRmYF50b+zoBai7r4bYy3TTK3VU0ImYLjq+/u1nHnWNccbhmzK9ERZ4XjqFF9elj93p0wAkPCra37plMTAggggAACCCCAAAIIlEXgvfd/vYE7J5alCRSyAQFzJz3Th+t90P/rldvc/Vbr17KGW96hZMe9OwPsckIULwBJ7VZr97Amk24QcHdYHbERSVV1Lf5tJCNT7WLhlqnw0Ywun2yWW19twwldvpOXIvsuD7fM7vk7w+pq2x2ZIui11yxsf3ZcWfe2poqPNEtoT9Cf0ZFb5paqOU2ctbcADcIfc4w3nbmUwb7hn9zUsmNbqW3efktzGop5eLch7TXHEr3j4XrCLVNHdvyMWiMjtcyIO1N+7C6OhFthV/IEAQQQQAABBBBAAAEEdo6AWVz8n+65ozJ2Tts/yy01fbaWReSLWdH/xWS25/1S+jUh3CpTY81tPBdy4b98LCRKrOWR3T62EHrixrE3bX3rve2p3d9r8zrqjzWn6MuwnjXUEe5jfErcz9mnJPuiDS7+gXfLz4XVb/FavAQ+QQABBBBAAAEEEEAAAQQqTyD/+LG3ZhMBV+X1TbEWmb4ya2aZvtvog/7fqGD59i+1Xzcv3CrfsVASAggggAACCCCAAAIIIIAAAlsqYAIOM4LHTIdiDa4tpV9TZaZvTB+ZvipHsGUrp/+txPb8XWu/Em5tTz9RKwIIIIAAAggggAACCCCAwA4QMGswmQXKuYti5XWWd/e8W/+4oTW2Vjsq+n81ofJ/vp5+Jdwqfz9QIgIIIIAAAggggAACCCCAwBMk4N1F75e/0q/eeZeQqwL61YQfpi/MNMT13BVxrYdA/69VbH3bb6RfCbfWZ85eCCCAAAIIIIAAAggggAACnzGB+/O/96a//eznv/Smwt3/IKvFxY/0ySefMHVxE84FMzXN2BpjY22mHxp7MwXR9MVWP+j/8ohvRr8SbpWnbygFAQQQQAABBBBAAAEEEEDgMyLw0aOPNfvb+3rn3Tve6KGfjP1CIz/9ud78yTj/ymhgTI2tGaFlrI25sd/uB/2/sfN8M/qVcGu7vxXUjwACCCCAAAIIIIAAAggggAACCCCwbgHCrXXTsSMCCCCAAAIIIIAAAggggAACCCCAwHYLEG5tdw9QPwIIIIAAAggggAACCCCAAAIIIIDAugUIt9ZNx44IIIAAAggggAACCCCAAAIIIIAAAtstQLi13T1A/QgggAACCCCAAAIIIIAAAggggAAC6xYg3Fo3HTsigAACCCCAAAIIIIAAAggggAACCGy3AOHWdvcA9SOAAAIIIIAAAggggAACCCCAAAIIrFuAcGvddOyIAAIIIIAAAggggAACCCCAAAIIILDdAoRb290D1I8AAggggAACCCCAAAIIIIAAAgggsG4Bwq1107EjAggggAACCCCAAAIIIIAAAggggMB2CxBubXcPUD8CCCCAAAIIIIAAAggggAACCCCAwLoFCLfWTceOCCCAAAIIIIAAAggggAACCCCAAALbLUC4td09QP0IIIAAAggggAACCCCAAAIIIIAAAusWINxaNx07IoAAAggggAACCCCAAAIIIIAAAghstwDh1nb3APUjgAACCCCAAAIIIIAAAggggAACCKxbgHBr3XTsiAACCCCAAAIIIIAAAggggAACCCCw3QKEW9vdA9SPAAIIIIAAAggggAACCCCAAAIIILBuAcKtddOxIwIIIIAAAggggAACCCCAAAIIIIDAdgsQbm13D1A/AggggAACCCCAAAIIIIAAAggggMC6BQi31k3HjggggAACCCCAAAIIIIAAAggggAAC2y1AuLXdPUD9CCCAAAIIIIAAAggggAACCCCAAALrFiDcWjcdOyKAAAIIIIAAAggggAACCCCAAAIIbLcA4dZ29wD1I4AAAggggAACCCCAAAIIIIAAAgisW4Bwa9107IgAAggggAACCCCAAAIIIIAAAgggsN0ChFvb3QPUjwACCCCAAAIIIIAAAggggAACCCCwbgHCrXXTsSMCCCCAAAIIIIAAAggggAACCCCAwHYLrDvc+vTTT5X754/0++wD3Zu7r5nfzOr9mbv8w4BzgHOAc4BzgHOAc4BzgHOAc4BzgHOAc4BzgHOAc4BzYNVzwGRJJlMy2ZLJmEzWtJ7HmsOt/OPHXqUmyLo/n9XD3D/rX/4lr6WlpfXUzz4IIIAAAggggAACCCCAAAIIIIAAAp9BAZMlmUzJZEsmYzJZkwm6TPa0lseawq0P//BHzfzmnh4sPCTMWosy2yKAAAIIIIAAAggggAACCCCAAAIIrChgwi6TOZnsyWRQpT5KCrdMYmaGiX2Q/QOhVqmybIcAAggggAACCCCAAAIIIIAAAgggsGYBE3KZDMpkUaWM4lo13DLDw8wcSDNEjAcCCCCAAAIIIIAAAggggAACCCCAAAJbIWCyKJNJmWxqpceK4ZZJx0wh/7z40Upl8BkCCCCAAAIIIIAAAggggAACCCCAAAJlFzCZlMmmVhrBtWK4ZYZ/MWKr7P1CgQgggAACCCCAAAIIIIAAAggggAACJQqYbMpkVMUeRcMts3CXmd/IAwEEEEAAAQQQQAABBBBAAAEEEEAAge0UMBlVsUXmE8MtfzriPRaP385eo24EEEAAAQQQQAABBBBAAAEEEEAAAU/ALDJv7qKYND0xMdz6ffaBd+tF/BBAAAEEEEAAAQQQQAABBBBAAAEEEKgEgQcLD2Uyq/hjWbj16aef6v2Zu4zaikvxGgEEEEAAAQQQQAABBBBAAAEEEEBg2wTM6C2TWZnsyn0sC7dy//yR7s9n3W14jgACCCCAAAIIIIAAAggggAACCCCAwLYLmMzKZFfuY1m4ZYZ3cYdEl4jnCCCAAAIIIIAAAggggAACCCCAAAKVIGAyq/jUxGXhlrm14r/8S74S2ksbEEAAAQQQQAABBBBAAAEEEEAAAQQQCAVMZmWyK/exLNya+c0s6225QjxHAAEEEEAAAQQQQAABBBBAAAEEEKgIAf+uibORtiwLt8zCXDwQQAABBBBAAAEEEEAAAQQQQAABBJb5NGwAACAASURBVBCoRIF4dkW4VYm9RJsQQAABBBBAAAEEEEAAAQQQQAABBBIFCLcSWXgTAQQQQAABBBBAAAEEEEAAAQQQQGAnCBBu7YReoo0IIIAAAggggAACCCCAAAIIIIAAAokChFuJLLyJAAIIIIAAAggggAACCCCAAAIIILATBAi3dkIv0UYEEEAAAQQQQAABBBBAAAEEEEAAgUQBwq1EFt5EAAEEEEAAAQQQQAABBBBAAAEEENgJAoRbO6GXaCMCCCCAAAIIIIAAAggggAACCCCAQKIA4VYiC28igAACCCCAAAIIIIAAAggggAACCOwEAcKtndBLtBEBBBBAAAEEEEAAAQQQQAABBBBAIFGAcCuRhTcRQAABBBBAAAEEEEAAAQQQQAABBHaCAOHWTugl2ogAAggggAACCCCAAAIIIIAAAgggkChAuJXIwpsIIIAAAggggAACCCCAAAIIIIAAAjtBgHBrJ/QSbUQAAQQQQAABBBBAAAEEEEAAAQQQSBTY/nBrKa/cQk75Jad9Rd7LzmQ0dnNKs4+cbc1Tb/t89M1HOeWc7fILOa8eU1cuF9s2uqdW3DbvlBOUGWm7KevRnCZvjmtiJhs9Liladqwtpt5IWbHjyueW151bWOFYlvKavT3um8U2W2tdkXbZ43B8LaFn57y/rB5JOa8fM5pesHs5f/O+3djtuRUtwj1WsjZe7nEnnVdhQTxBAAEEEEAAAQQQQAABBBBAAIGdKLD94Zayutxao/pXpkK/6QvNqmoa0LQNvLKj6mpIqapurxoadqu2uk7tr86F2+tWt6qquzVReEcT3TWq6s4E72TUVV2jXfWNamhs1FOpGu16bliztnxnP2nlbbNX2lSV2u2VY8oy//psNcpr+uIh7apO6amGRtXX1ai2Ia2JMOyJlu3tf+K6sl79/me1h68rZ9sTOa6srp3w62vYnXLacC5y3HZXPcqoqz44ZmOWatTpjE16/Lq6boVbK2ro90nt7kJ9tQ1nNGF3D4yq9pzRpGt4d0BN1cvdw3qW5nS5LSW/3L3aZfpx2D9605L8rW7VV9ep3rjuTilSZ8TC23oVa/8YquqdNs4PqrW6TZfnnePmKQIIIIAAAggggAACCCCAAAII7GiBCgi3JE2dUb0NHRau60hqr3rCICana4dTqj05qlwQpPghSLP67wb2y4IPJYZbYciyMKz26qd1upCnOZ0YC35i23rhVutgEEg5u5mn3nE06/TtIAUyYU5rjfaEwV2s7Mju/mdV1SkdeS2ItxKOy+yyYhuCMr1tDgxqNng9femQGk6NBsFZQjsidfnBUOuVIHhaGvfCwdDPhlvVKXXcDBMvTfY+raqVwq2gjjHbjze71dA6oGmvjX6dBy4FoeXSjC63N6prtIhFYN13JzhAG5x1jstvURBuVTvBKeFWgMUfBBBAAAEEEEAAAQQQQAABBJ4cgcoIt5TXRPfTqj05rBHv72gQUJjhPKPqWBZEzenygRq1vhqEL5Fgxu+cpJFbYTjzaFTHq1PqCUdcuR0aC35i2/qh0YAmw2mOhXBn+kKjqtyRV6ZYM40xnBvnl91xw5leGO4e1PvKGdWn2jRkRhclHJcpspRwK3+zU7XVe3X84rims2ElwYHGjtG8G6krCJouTHlTObM3u1Vfd0LXwmmE/v5HDh9V1XPDfmCWH1dH6pDan1th5JY3siulpvSwJu/G25TXWGdKVfUn1H9zRsuaHGmflGg93q2qVDoYyRYEdL1nCmEp4VbQ//xBAAEEEEAAAQQQQAABBBBA4MkRqJBwS5I3YqtGVTbYscaJgUQQXNiRRbHgw+yaFG7ZaYnedMG2wZKmJca39YIlMzop/FeYDhmt0x6A+9cPhQr71igcHRWMhjIBnJmWWWvaZ8Ka2HRLU1op4Za3XWZQXe3+NMzahhO6Fs7kLC3cCqcl1tep/tigpmPTK7tGTaDlj6DLDR9Sbee4RhKmg4ahosn67o6q74UWb8pmVV2L+uwoN9PgpawmrnSr3Zt6mlLDieuFPor1caJ1ZJvCOZIf7VRtfbcm7jAt0T0beY4AAggggAACCCCAAAIIIIDAkyBQOeGWDaROjUddlzLqStWoY9QZ6bM0pdN7apZN37PT3UwBY6eWjyBq/+G4xl5Lq6m6UeF0tmhtsmtuFdt2pWDJ+6zRTrPzC87dGdfY7WwwEi0hVArrdz7zpjOm1NrWtv5wy11QfymnsVNPq6ptOLK+V5dLHQnSCsFQcBQaeq5GDRdmgtYW2mqmIu7pHVR/kz9FMRo6FbbzdoyMYpNmL5r1y+xIK3+x/XDh+ty4uvYUH52XaD18SN46YF5l7jHkveOvb2tTg53+GhwJfxBAAAEEEEAAAQQQQAABBBBAYGcLVF64FS4CX4CdfGWvquo7NTKXl/JZTV5oU22qU2M278pd1/FUSq2XZrw77OXNCJ1USsffsEuzuyFLMAUyPn0wrG7lbb1Q5YA7LdG5w6E3+iylVjOdb8kfpdRRnypMnwxGZ0WmJcamLIajnObMMZgRYoWRYbaJKwVsdpvJV55WlVnPyhgt5TRiwq3wmHO6dizljQ7zRmM9mvEXej9mF7N3gyFDntHppiJrh9lF5IPF5VcKt3KvHVXVnm6NmOmNS2bx/TYnjPIDy9aLfh9qYdQLt4quP5ZoXaOmi3Z4WvQY7AL7VYRb9hThLwIIIIAAAggggAACCCCAAAJPhMCOCLfMdLWx3hbtCqYC7mrq1JBdTD7ohvztczpQZ6cL1unA2Uy4AL0djRUGR14w8rS6xm065valG27Z6ZKFbb1gKZyS6NcXlmum3d0+p1ZzN0OzTWq3Ws9OFdYPCxdit+2sUVW4OH2sXkmzV9pUu85wS4+m1Ndm7izp1xWdlih5n7fUhdMrd7Wc04TNAoM7WBamT6b0VNs5TcanJXp3W8x5o7psqLRSuKWlOV072Rz2Y3xaYsSueuVpiabHTIh53NxFM7R2+zwWbpntvbsxcrdE92znOQIIIIAAAggggAACCCCAAAI7XaCiwq2yYD5KCqzKUvLOLGQpr/xKJObz4O6FW3mA+TAoS6g1vz1tSmgJbyGAAAIIIIAAAggggAACCCCAQIULPHnhVoWD0zwEEEAAAQQQQAABBBBAAAEEEEAAgfIJEG6Vz5KSEEAAAQQQQAABBBBAAAEEEEAAAQS2WIBwa4vBqQ4BBBBAAAEEEEAAAQQQQAABBBBAoHwChFvls6QkBBBAAAEEEEAAAQQQQAABBBBAAIEtFiDc2mJwqkMAAQQQQAABBBBAAAEEEEAAAQQQKJ8A4Vb5LCkJAQQQQAABBBBAAAEEEEAAAQQQQGCLBQi3thic6hBAAAEEEEAAAQQQQAABBBBAAAEEyidAuFU+S0pCAAEEEEAAAQQQQAABBBBAAAEEENhiAcKtLQanOgQQQAABBBBAAAEEEEAAAQQQQACB8gkQbpXPkpIQQAABBBBAAAEEEEAAAQQQQAABBLZYgHBri8GpDgEEEEAAAQQQQAABBBBAAAEEEECgfAKEW+WzpCQEEEAAAQQQQAABBBBAAAEEEEAAgS0WINzaYnCqQwABBBBAAAEEEEAAAQQQQAABBBAonwDhVvksKQkBBBBAAAEEEEAAAQQQQAABBBBAYIsFCLe2GJzqEEAAAQQQQAABBBBAAAEEEEAAAQTKJ0C4VT5LSkIAAQQQQAABBBBAAAEEEEAAAQQQ2GIBwq0tBqc6BBBAAAEEEEAAAQQQQAABBBBAAIHyCRBulc+SkhBAAAEEEEAAAQQQQAABBBBAAAEEtliAcGuLwakOAQQQQAABBBBAAAEEEEAAAQQQQKB8AoRb5bOkJAQQQAABBBBAAAEEEEAAAQQQQACBLRYg3NpicKpDAAEEEEAAAQQQQAABBBBAAAEEECifAOFW+SwpCQEEEEAAAQQQQAABBBBAAAEEEEBgiwV2ZriVv6/JX/yD+i+d1DN/16IvpL+kzwX/vnD6r/TMhe+pf2RC03/cYk2qQwABBBBAAAEEEEAAAQQQQAABBBDYUoEdFW7l70+of+CvI2GWDbWK/f3C2e/o8uTilqJSGQIIIIAAAggggAACCCCAAAIIIIDA1gjsjHDrT/d17b/+lT7vjNAqFmYVe//zf3dGY3/YGtSdVks+l1N+qfytzi/ky18oJSKAAAIIIIAAAggggAACCCCAAAKOQOWHW/d/opNnCtMO/fDq3+svLnxPl382qensovKfFo4ov/hA05P/0xvh9eW/je/XolMTO2kUV04TF9PqSQ9oIlc4xnI+y77aptrqGu06Na7yRVF5TaT3qqo6pQOX5srZXMpCAAEEEEAAAQQQQAABBBD4LAnkZjRysVPtjY1qaGxU+8kBjcxs0gXyVrrOj6v/5CHvmBraO9V/M1ta7fMZXU4f1QHj0XJUPVcyyq42WCU/paG0yRbS6nkj4Rp9KafpGwPqaDfGLTqSHtTEfEJzlnKaHE7rSEvl9UVlh1v3/0HPRwKqej3Tf1lj9x8nKCe8lb+vsaH/HJvGWK+TYzsl4MrqcmuNqqrbdDnpxEo45LW+lXvjhHZVp9Rwdmqtu664/eTZRtVW1+n4G0/Aj86KR8qHCCCAAAIIIIAAAggggAACmyGQv5VWQ8pcE8f/pdR6JSGk2YxGbEKZ+Vvdql92TDWq786sOOikqEd9tyYeFW/o5Ctm8Elg2J2Jbrg0p2vHnM/tdqlG9dxyhsAszelyW6pQjt2ueq86Rrf/ur9yw60HP9FJN9j627/W//ezuyt2dLSHCq/yv/6H2OivFvVOlhiQFYrZhmebH25tw0FRJQIIIIAAAggggAACCCCAAAKrCMyor94EMim1XphSLhidlLs9oFYv8DqkoYVViqjEj/Pj6vDav1ddN4NQaGFcXcGxdow6gZLb/qUpnfa2eVpHXg2CvaWcxrr9YKq2s8hsrKkzXpBWmwqCqVi4lXvtqDebq7ZtQJPGcymv6VePao8Jr+rPaTpoQ274kBds1bYNatb2xXgQ0u05o8nVRo+5x7IJzysz3Pr0vobO14d3QPzc3/61+t7d4Gir3IR63emNf/sdjTzYBFFbpDvE0BvWN6xJJ8ycOOsP4zv+mjv0MKM+b6jlOU145Tjh1lxWY2cLQw/7IkMWnf0ezWio2x/aeODkoKZNepubKrz3wjmNOVVmXzvhD4M866S3S1mNOcM+D7yQ1tCU03jTtvUcX6xcM5x0LDIizTmOpTmNBMfR0N69vH7r7P6NDZFMbHesDZFt8hmdbjb90qlrkR/JOV32hme2qX/GrZDnCCCAAAIIIIAAAggggAACmyIwP6hWb3RQd3B9XKhl+jV/it21u4X3oteoh9RxcXz5dL3Y9eCK16S5jPpfaFFDo70+j18HH1JXfEqgV35a/ePORbfTRPPUhEneKKqTo5HBO/nRTv/9w9cVu/r2S5gZUIPxODCoWbfM/Kg6PKdOjcRzsaUZ9TeZgLBNly91++VHwq0Z9Teaz5vV71oqp4kL0SWSbIbR50QH0pwuHwjKj1zbuw3cmucVGW7lxr7jLB7fopNj98ujEZvm+Pn/9vPIyVSeSiTNDQZJcp3q24/qSPNuLwmtqk9rIjjZJrrNCVCj1ivuSZ9RV+TLa8OtFh05vFdVdXvVUF/nn5AmvQ6HYdr9DunI4ZRq6+r8+qprVHs4ra6mGtXu3q1dXtkmfS2kqtkrbdET3BlquKv+kI680KynglS5JxM0fj3HZ4KjBj8prt3dqIaGwKR6r7rCoY7R46iq2x3Ubaz26vSKMyezGnousEntVkPj3uB4namRqx5bXmOdfhuPvOb8nNgfEZNGl+0koSAEEEAAAQQQQAABBBBAAIGiAksZdXnXotGRW0nb5zNnwumLu+ob1bA7uPZsOFcYUbSea1Ln+rxQR53qzaCU4No8MpLpjRPB9fryQM62O8wCXnWzAJN6XdcRpz67ffg3k/av81sHFd3TXkc/veyaefaSf73fatbCvpUQbmWH/QDRBGbBulveulxni6y5FTYmeGJHoaU6NRYP1uLbbvLrygu3Pn1ffacLC8H/xdXJjQVQnz7QxM8mw+Qzf+t7TnD2V7r8T+UXtoGRu5j69KVDaj9pRkD5PR6e0CWFWzWKfGHs0L/wBLInc41aLwbDE3OjwVBH571H4+raY4KilHqCtNW2tcqmtzYdd9PgmUG1t3eqZ3jK6wu7z1qOb/Zis/cl33NqPBxOGs4zDocwOsfxwxm/380wy1NPe/vWpiMRcaTj8jc7/S960xlN2rnGdwf8L6oNpUo4NvOFNwvsVzlpuW17wwWGbUXQeYEAAggggAACCCCAAAIIbKLA7GsnnLWp6lTvLaA+rmlnLIIZPeSPTkrpyHAQ+zjXkXbggr2uK/WatL5z1Bn5Zeto1unbNsXJayKYEtg+HDRo/ro6Whp1oLfY2ll2AEuNum7F4ez1cJE1t8Pwq02XneXGcje7/SmE1bEy54f9QTdNA5o2UwaTwi0bmB07p/5l62m5A1HibTWvC8df/8qKI1GSdi77exUXbplRW/4dEb+kz53+jkY+2MAxf/pAY/+txSvv8/0/CQKuBxo6VwjPPrcJo7fMHQjNqKza5rSGbs8pZ8MW51DWFm6lYie+HfpnQ6qkL4H90kS/GGOn/BFj9otkg6ow3LLJbarZC7NmF+wXt9D4tR+fbW9jbFqffd8mzEnHIWk8IWEuNMd7Zo8rPj85n8spt5BT3nyZSzg2hf/vgB3SaR3jwzRjDeAlAggggAACCCCAAAIIIIBA+QXM3QHNkjXBaCx/YfS9Ov5akPDMDeqAGaDQOBCuD+U1YinvXQv61+P22rPUa9KWSIBkZmd5dbQPanrBv8Y015m52+fUZOo+NV7icdvry1gQ5e1d5HrYKTlcGD61W60n0+p6oVm7Um1q9W5E55aZ07XDZvSaMwMqKdyy71XXKAzz3DW39nSHs8+cZnjB1vSFNm9giDsQJ7rN1r6qsHBrUSP9heDpL4Y2MGrLCbZsWPbM8PveaKD8hDt663sa+1OZ0ZfmNHTYTrvzw6Ta3c3quBKMRpK0tnArGlCZ1tr9/ZAq6UtgvzTRfaP7ScvCLUmzrx51pgPWqCq1W012/S5T+ZqPL6l9vnm0PUW2s184O7psWXfZY3W/zMs28t5Y9dgkTfaakWIpddzMSwvDajc/Vk0D0XnNycXzLgIIIIAAAggggAACCCCAwGYJPJrTxEU7misYgGCvF5dN13MbUeRas6Rra2fUk7k2TPq3Yt1uO/IaOemXER+YoaXxYJkiO9DC3c8+z2v6SqeavKDPLIPUrZG7WV07bMosBHdm/S4zI6k+7Ywgs07udbUduVUdr9NeY9sBNbZ+8zevyVeag2BrwF/n2/14m55XVriV/7lOpW241aK+O+tUSQi2/uK//kSznwblfTqpXudOjL2311nPars9mtPkzWH1nzoazP9NyQ6JtKFOZM2t8GS283PtCVU4Sf0qTQrrfyE2I9zy6ljKa/b2uIYuduuIXSvLmarnbVPy8U3ptDcdMpZ+K6uhttWOw/khcb+EEXvHo5TAfLVju33GG9ZZ250JF/uL9FOkbl4ggAACCCCAAAIIIIAAAgiUWyA/P6Wxm+MauxOZg+hVY2fueNdpU/7127KF1iMN2uA1qa3jhWHNuiO37PPc8hlPkeqdF9MXGv0le+JT+exaz/ERaHZfOxItXpedfZTq1oR3x0KbI9TIW3/Mu2ldYY0wby3vxhO6ZmZw2tlN1TaDsJUVQjg768v/pDKDLdO2ygq37vy99tlw6/SZIsPfLHaRv6sFW95ujzV2yYZoX9L/+z/LtGB90KTZjAmFzmnEuVvA7KUWP+ENAprwhD5VuF1nftzOlbUnVuGkbHLXezILuntp8VFd877nSSm03XdtI7fycxmNDQ+ob9RZos4OwfRO+LxKOb5oeFdYqN1byM52nR0VVb3ScRQJtxZmNDlT+JGz86dr3TtOPLLrjnVrbEla/dhsw4IfvtRRHXnOhG879Baz9nD4iwACCCCAAAIIIIAAAgjsNAEbKNV3a8Jd6sfcKCyYhueNfrKLmps7/jnLJE9f8Nd99teK3uA1qa0j1akRty1zo7o8HA3gcjNTsTXBYvB3zvnriO3p1lhYVj5ca7r+bHAQ3qCMqcK6X7YN5s6Hzppbs1eC6YGdNlvI6tqJRjXYUMv+tTenMzeqs+GWchryrnmfVte4E9Blh3XEW8zfXqubY8hrIt3oj9hqdta6jh3edr2srHDrV2cK622d+4fYHQBKICop2PLLmb3+V2FdXx5+v4TCS98kDFqau3X5xrjGhs+pvd6EJMFUN1PU1JlgYbyUnmo+qiPte7Wrqdmfrxumpjagelr19XVqOmFuxdmpA3X+aKfCEMPyhVu6O+C3IdWsriujGrs5rL72vf4aYsGXpZTji4Zb0TtIHjg5oMtX0oFJjQqLzyUdR0K4ZZNpZ2F8PcqoKzBuONytnnRhxFnoVMKx2V624aM35PS54fCGBPZz/iKAAAIIIIAAAggggAACCGyigHO3ezPayNygrefUUTUF18NV9elwQIwNeKrqWtSRTqvnZIt2mQEhqTYN2UEnZpCIF9jUac3XpGb5HhsiNRxV3/C4Rq50B9fmznW+vUFZKq2JojSFhdhtezta6vzBME54Fi4h1DYcZiM2sDNLB7We7NaR5mA5JPc4i9WbNC3RRFaZdJBNGJdY5uCMLpv+oR9smWvkyIiwIDzrK37/t2ItKuv7FRVuZUf+Sxg4fS5cAL5wvLO/+J+a/KjwOvJsDcGW2W+1uiJlr/XFUlZjvW3RdavqmtUx7MSrymv6YmFtq9qGE7o2Z+fYxkdutenynYx6gumBJiRrOHFds96QQ9O4pFDIBmNrG7nl2YyfUWtksb46NZ0cLtRXwvEtC7fMl+bOoI6Hx+Cv5dV61r8Do0+cdBwJ4ZZm1GfKSbXo8l2nc7LjOm1/FLyRbXU60DteSLpNv692bLY4G4RV14RTSe1H/EUAAQQQQAABBBBAAAEEENgCgaU5jXTHrq2r69T0woAmChN5vFFF01dOBMsB+YNBzDX25TvOaKSNXJN6h2rWu4rWYcKprhvOdf6dc14bag8Mrrxm81JWI93NfgAXrOG1q6lT15yicjdOeJ/vcmZ7mdFT8TbsajmjMRvgrdQlRcIts0tualAdTUHA5oWCjTp+ZUq5MHMorPuduOZY/E6NK7Vjkz7bMeHW7Jv/RZ83UxbPnNFEPOAqIdh6/PhPWlh4qLnffaDf3J3T9K/venMy/9ed9/T67WnN//GhzDblfuTNHNz4nNhYJfno9y32afSld/e/NWwf3XuNrx45dxossmspx7dsV6/cDR7EUt6/A+Kyws0v1urtVgnHllQ07yGAAAIIIIAAAggggAACCGy9gHftuZArfh0YNMnbLpzuV6SdG7wmXfE6eC0X+HYdrWLtXaEsrw0bvKxeprNBl2XlbeEbFRVuqci0xPzkD/RluxaXF3D9vSZtDrVKsPWnP32i+Q8+1K//aU7ZDx/oo48+1iefLOnPf/6z9888f/ToY334hwVvG7Ot2YcHAggggAACCCCAAAIIIIAAAggggEDlC1RWuOUuKJ8+E5mjOjsSjNyyIdcZE3A90Nh/aylMZUx/Se5dER/mFjX963ta+GPOC7JW6w4TeJltzT5mXx4IIIAAAggggAACCCCAAAIIIIAAApUtUFnhVv7nOmXDq/R/VN+dKN6ygCvc1r/zoRtsPVh4qLuzv9O/5B9HCynhldnH7GvK4IEAAggggAACCCCAAAIIIIAAAgggULkClRVuaVEj/X5Q9bn0l/T5ocllcsUCLjfYMqOuTDhlphzGH/nsjCZujmss+Dcxk02cs2v2NWUwgisuyGsEEEAAAQQQQAABBBBAAAEEEECgcgQqLNyScmPfcaYZfkcjkTsg+HDh4vLByC032DLrZZlphfERW2b1/8id+oI7Evgr/e9ddicAU5Mpw5TFGlyVc8LSEgQQQAABBBBAAAEEEEAAAQQQQMAVqLhwS/lJ9Z52Rm9dXT56yxyADbjcYMu8bxaEN+tmFR55TV885N9i09ym80pGs/buhUt5zd4aVFeLf8vL2rYBTcfuUmDKMmXyQAABBBBAAAEEEEAAAQQQQAABBBCoPIHKC7ckZX/qLh5fr5NjyYu7T//i55r9tID6+PGfvDsemoXh7WP2Sptqq2tkgqtJm3nlMupPp9V/y76R1/QFf7uq1kHNOrMZTVnmToumbB4IIIAAAggggAACCCCAAAIIIIAAApUlUJHhlj69r6Hz9c70xBb1vbc63MIfH+rDPywUNly4riOpGlXVd2vM5ljm0/lBtVbX6MClucK2ymuie6/MNMX2YXdjKfvhAy2wuLxjxVMEEEAAAQQQQAABBBBAAAEEEECgMgQqM9wyNg9+opN/W5ie+Ll0i06O3V9R7be/+0CPHn0cbjN7sVlV1Sl1jObD97wnieGWpEej6jBhWOOApp09PvroY8397gPnHZ4igAACCCCAAAIIIIAAAggggAACCFSCQOWGW0bn/j/o+UjA9SV94dxlTfwxme43d+ecOyRmNdRWo6rqTo050wy9PYuFW5LGTpl92nR5vlCHuXOiKZsHAggggAACCCCAAAIIIIAAAggggEBlCVR2uGWs7v9EJ8+4I7jM83p9+ex31P+z9zWbexyKmjsbFtbbyqjL3BHxwKBmwy2CJyuEW9krbd7UxK7xwk6mTFP2Wh6L772ul48eVOvBgzr84nlNOGHZSuW884ODunR7pS1K+Gz6olq/+i299dDf9vHioh67Ad/SYy0WBriVUKCzSfZ1vdT5uh44b9mn5Wj7g9e/5ZkZt9aDz+rbPRfLYrc4+i19ufm87trGlvR3SpcOfktvZUvaOLJRWc0jJa/yYmleE+df1GHP7+t6+fVpLbp9X2R34/7S60m9zfrWFwAAIABJREFUWmSHxLcXNd65TwfPB+MeE86zZS6J5SS/ueL5tTitH/d83T93jvbox+8lr9OXXHLSu9O69Mw+vXRjg+V8HPvuJVXFewgggAACCCCAAAIIILBcIHtdxxsb1RD8O/BCWpczpV2cTZxtVF9meZHeO6bcE9eVVU4jJ1KqPztTZMMd+PbdUXW1G7MWHbmYUa7YteCjGQ11H/Jt289oLJJXZNTnuHv+Zx3M7Lj6Xmjx9m3vHl52U77ZG91qN/u3HFXfeGn9VQ7pyg+3zFH+6b6u/de/0ufT8ZAr+tocTEnhVnbYW3OrKtWs05nY+lrlCLemvq+DzV0av7eoxx8v6sHURR3b96LeKiE7eOd7X9HLtzbYtUuLujttz855/ejQ8/qRfWmKvtWrL37POTnXUt38kI4dGkoOt8rQ9gdXn9cX/+ZNLT5c9P7dH+tVeznsluZ19zdrDSoyevnpmF1JVmU2L6lOs9EDvXV8n471T+nBx4/1+OE9jf9Nsw5+f2rVEoz7savuSbLqLskbzE/rbhCqLj/PElySS0l8t/h3Y14/fv6gXnp92gtxH08P6ti+QribWFgJby7+ZloPiv2XQQn7m02Kt7nEAtgMAQQQQAABBBBAAIHPqoA3KKVTIws55cy/u+Pqaa5bvuxQgs9Ed426il1Xm3JbB+XFLvMzmo5GAgml7ZC3zJrjdW3qv51TPj+nkc69qn8l6VrQzHKrU+uFKWUf5ZW9PaDWuqO6Zpcvz4+qY09aY9bd/H0UGCxN6XR9m/puZ5XP5zV7o1MNbcO+paTca0fVcGxY07m88tmMTjftVU8mtkzUJnHujHArOPj8/Qn1D/y1vlAk5Ppfd94rbVqipPydQR2vN1MQa1TfOapscBFbjmmJSUHBg/fe1t0g3Lp/vVdv/abQo+5r/2J4UXdfP6++3vP60a1C4LCYGdRb04t63/ssGNH0cFo//kGv+n7wut63oYLu6a3eN3Vfi3rnUo+O7d+vY9/tVd/1e9Jv3lTfi6364l++qL7eQb1j95nP6EemnN6LGo+HQPNv62pvr/ouvqn7760WbiW0/fG0ftw7pPfd0WLTr6vvR9MqjLvzPbxwKxa8vf/9/fr29UIwtfibN/32/GBI7xR4giAhoX5T9MOMrl7KyCvFPL8+LTO67lJvry5dzRQJMVYIt9zyIuVv3Nzv5weauBj0WeFUWeWZaW+v3nG3WryndzLznrMp92rGcXRe23P2wdsX1dfbq6s37jkjvu7prUsZPQjOg0vBaLAHY8G2Tswf1rHsPEtwMe00QewNv5xl/bBkz/Xz+lHmgSaKhaeP5/X+2LTft8GxT/xNkZA41vdXTdudetwRX4XvZfT4ve9ByGi/axa98Nrs/+2//IraX+xVnz33JBU7f1e0sMWbv4v3NHHV/D7E+8l4zvvnjfke31vU/evOd3wla7d8niOAAAIIIIAAAgggUAkCXrjVrQm3Lbe6VXv4ukweNftGWtecqTnuaz/cymt6+Jx60ud0+ZYzgsgJt3K3BtR/y0m35jO6fDatnvSARmac9902SMreGlRfOq2ei6OatcGP2WYpq4krps60+m/MREdOFSs7l1H/GzPKjg+oJ33dn3m2lNP0DfM6rb4rmTCvMFVk7zrH4rTLzEJruOCMQjMhVapbE/H/w94cf2yd8dlLLWq9EpTr+DjF+0/DUW/2kxn17e3UiJdfzam/6YRGHI/8XEZjKzjaUsrxd0eFW+EB5+9r8hf/oP5LJ/XM37WEYddrmZ/KLP5uH/6C8jXFk92lrEZONaq2OqXWizPK2wXlmwY07ZwAZpF6s1h9qY/HP+3S19p6NfHbeHTjlxAfzeG+Ns/bn+1S/41pPbiX0dVvHgxH1JgA4uCzPbp66568EU37n9XhzvN65949vfOD5/Xlr78eXODbkOOx7mde18sHD+rlH72tiekH0oNpTfR/XV/8+kVNjGV033D9dkjHnvmWV+6D6df18jMH1ZcJ2u6NQvM/uz81pJefbVbzCiO3ktv+WBN/s0/pnxY83vle9LW1TQq3zLY23Hp8q1etbb0an57X/VuDeumZ5/Wj3xZck+s3d8h0QjnzvPmgXvpBRvfnp/Xjv2lW8/cyy4I2aYVwyy3PVB++3ri518/PvKi+14M+szir/r2nq63NeunqlBNMFXayAZZ9x31tnje3vaiXL/km473PqjU0yejl/Qf17TNv6u69Kf3o+H61m21fn9b96deVbt6v/mAqbVjmsvMswcWEr987qPbeN3V3/p7eOf91NT5vRwX6o9C8z+5Na+IH31Jrc5HAyh6Q/Tv/ptLNXZoo/BTYT/x+an7WP857b+vlv9yv9me/pX7znbp1Xse++nX9OAihC99Lc/zNOvzd1/12mu9a+B2w3zVbReH14vTbuvT1r+gb/W9rIggYo+fveX1j//P6cRDQvv99Y/G27j+c1/v9z+trf3Nz+Tn5cUZ9zc16yfTTw3ua+F5rYWTe0pT67Wemn777otr/0o48jFlf+pZanx/Sfed3zh4BfxFAAAEEEEAAAQQQqAiBpHBr6oz2BOFWfHSW+9o8b2rp1OkbM8rezaj/WKNarwTraDvhjQmEwlBnblCtDSfUf2tO2ZlhddQnjzryRycNanI+541cqm8aCAKpOV1ua9TxixnNzs9oyIycSmfk5z5u2aPqaW5U161gRJNpT32jjqSHNXZzRjnlNdHdqKb0qKbn5zRx9pCesqOj7pxTfXWz+p1Qz/aVGagTHa2W1eXWFl2OLx9u6jsQXb7JhFtV3cHsrlvdqnrhnIbOdqvrVFr9N5PDNK/ehWG115/zb8hnZsi1Dmr67qj6T3Wq49RAbLqjbenm/N2Z4VYRi4WFh8p+6Mz9M8PyzN0P93RrzEkP47tnZ0yiak6gvd7dFY8MRzvvwz8saOGPdohTfO/k1w/ePq9vt+7X1/Yf1Dd6kkYYFfYrXET705ha7ZpFZpOPbyr91V69sySZ4ODwoB2qZKZ4HdTVcARY4aLaD2XsCJ6EqWCRaYkmeGoOwwmvVe+dV+vxN7Woxxo/tS86TfL291cMt4q13UxRCy/WzTHtSw4fzDH6wdvbmhh7W+OXvqWDzb16xxspY8KbVl11lj/zgsQgBDCOResPw6cgiGobVHjvzYdv6tv7v6/3C10SPFtPuGV23Yh5vJ+XNUqP72X0zr1w6FB0g2DtqYP79ungsy/qkjMCKwyegj3c1577N990wpR7unrQBiPGoaeQ+Jvz51QheImXE05vjJxnCS73BtUeBrLm80WNf3O/+s3I2d9cVGsYIJnPHujHX18l3DJ9/PRX9MX/8Kz6fur8DgTH6/0x2zh9b9re2l84oeLfRX+KcOz4NaX+/S8Ga9q53ztTQ/S1W55kzt9CeGa2fnzjW2r2po0+8KZWXnrPaWyx4OlxISTWbwd1OHB6PNqlL7ujHk3YtT/oQ2Pd6pzzWh44OzXzFAEEEEAAAQQQQACB7Rfwwi1nWuL8lPrb6nTkNX9ElRtmmca6r83zhh8WGcWUGG7lNdZZp46bzhS67JTG7iwfvTWRTun4G877wf9uz9/s1K7OcT/MMg1aymoyCKuWlX13QE02FIuHTeaz54a90Wl+J+Q1cuxpnQ5mGOYfOW0Me8kEWdEb48VNwk1lRljtVccNO1JrVB1mVlsQbuWnhtVzdlBjM1llZ0wQV1cIBguFSEt+mBcJ6fbsVdPhc5q4m9PsrXNq3922PFxzyyjj8ycq3Hr8+E/69T/NOetuSbNX2lRrFpZvPbdsobOCY17TF/ztatsGNetcVJo1vEyZpux1PR4/0N0bZt2owgiN6AVvdF0e77O33ZoKQYkbIiwPUNyLavd5Yf+w1EjoYD7fp8Zn/MXvvYXcn9kfjExJ2NcNicIC/ScrtV3ehXaXxj+WTCB18EzS3F8/2PGnTPaq77vPq/k/dGk8zCnc4woqd9qzYv3OdoVRVvYAEsr1PjLv24DHbru8Xu8dt/xVw62VzH2DMCCKVasglPjyd992gqhlG3lvmBDs6jebdbDXH5UWPX+i9ZjPCsGpX17hPI35RM6f5eWEbY9tt+ycNZ/va3ZuIHBQB/cFAdayfaPfk+QjDt59MKWrzzujD92NI/0UbbvZrHDM7vPY8Uf6N/5Z9LVbnh987dNBb7H/4PvWtC9c/+7x7fM6HISSL5spwEXyyweZIV06FSye32S/q8uPJeKd4Bk/H1wmniOAAAIIIIAAAgggsO0CXrhVp3q7uLlZoNwZReSGWaat7mvvuXOTOMkJfxLDLefz1Q587rqON9TpqYZD6jg7qIkgI4qMAouUkVS2uQFeMOXSaY+3mxk5Vbc3XEjfLOheXxcflRWpwCy8pJGTKfVEltY2a2s1qt/J+MK9suM63bbbm8X2VNs5TbzRHYZb4Tb2iRmdteeMJu1r83dpTkPP7Y6GXl5/2SmK/sa54UOqTUca5ZZS1udPVLhlZOY/+FALf3RSVOU1ffGQdpmAK9WsriuZwpzYpbxmbw2qq6XOW3tr13MDmnR3lbyyTJlreSTdEe7+4LM6NuSnNNELXvci2n9up+D5dZoRNP4okejFaDx4ci+q3efx7eILypvPnw2n9kWPM+EzZ6RIdNuV2262ff/MfqVHHywfKeYUZI6xsNi9PzXt29dtumWOyxlBZPYz7QmmshnXYnaRQCsWcMRH2xSaY+rbrHCrmHlSSFFo0YrPzN0JF51RPWbjpbf1shn5Z8Y+xRaNd1+b5/4IIluDGbUXjKKKjUaKLxQfL2dN4dZ3I0murdy/6UHss4nvFhm5lXDcZkTUl3sTfkRjfe+23VTufjcLz93vk9nK/U7FP4u+LpRh9jOfxc7fwhGHzx6bddL6X1RzZKRV8PF753Xw0Hm9Px/0s3M8D4aio9C8drYF568Jt2Ke7m9SWDlPEEAAAQQQQAABBBCoFAEvLImtueW0zQ2zzNvua/PcjvDyd5lRf+NRXTPX+06YVAikTBCUMIXPqW/Z00dZTd/oVlOqU2N5Kftqmw5cis8BNHuZcCtW9tK4ulJpfz0xpz1eHSbcOhVJ5pZVnfTG5CtPq33YCTTM4u97gmN2d1jKK5eLjv4yDqFXPqfIx6atNojzyvGnTbZemCmMUjPv567rSDwEG18hNHPbVIbnT1y49ac/faLpX9/Tv+SjF/m5qUEdb0h5IZZZRD7yL9Wo41emoou9SV4ZpixT5loeJsRpd6cWLt3T1UOFdYne6d2nY5eCqVAP39bL+wsX7eZi+IuHBsO1cB6/3aPmYOpW9ELcvcA2rXMvqt3nCdOdMr36sjMF7f6l1kh7H986r/Sgv9i7+exguPaSdPd8q74YmS5WkFmp7d5W3lSz53Xs4EUlTBH2NjHHWAi3/Olp7ft79Y63fpI/lSoMT7w1m5rVHliuWL8TAkSCLq9W16twPL5pkXArO6Rj+3s0EcxWvX/pecdlY+bRfnbbs8rzB6/rG1/t0rgzg3ZxzDl/hp5X83ff9tfj8s7Jr0TWc/uiOR47Wui3gzpmpmp6oxhjPrFRQG573eeKnWfe1MLnDyqcehdMTw3buzSvH/f0asLMvPU+K6ynpodvKv3VwvckImGmlbrHvbSoie/u1zdet6Gos7V7HiQEfm4YVXgeO/5YuNX31efDqbKet7Oov/muv3TDogZTAcPX0v2hHvV5C/Lf01vf/H7Bf/FNvRSEkk7rl4V+D15/sbD+lzm25i69FcxcfnCjR61fDc7fuOdiRi83R6f4RurhBQIIIIAAAggggAAC2y2wWriVNmtnB2FSblxdewqjm0y4VdVamJWVH+/WHjvVzwmTCuGWZNbsru8O1siSNH2hWU22/NAip4mznbpsR0MtzaivsU1DZvSWmU5Y360JuyTSTGHqYbxsM8MsnMLotMerJj+ujrpOjdicyoyS6k7761d5Ux2nIgvMh02L1R+pw93PC72c6YLGrqEzXAjerCm259R4mI9EylFek680q8FxCutXTtcO71XXzaDh3rTFwjTSwnab8+yJC7cM08Pcou7O/s65c2IBL5+d0cTNcY0F/yZmsso70xDtlp98suSVYcpa82NxWleP++ttmWl+jfv269uXnDsD/vZNvfTMPn153z41fn1IP3JGpHgX1IND+rY3TbBZjc98S28FC6ZHgoPIBbZpoXsB7j6XFn/ao9Z9+9XuLWZkRvNM6+qz+/S1/V3+ukFL83qr86Aam8xUqWid5u5rPz6+3/+sqVUv3RjSyyuEWy8XabtvaNYc+koYRiW5mmOMhFt6rHe+Z9oehIGLU7r07H5/GmXTfrV2vhkGgSvZRQKtWMARtXNbZRy/oi9G/tmwa1Hvn39WX3vaOB5U39XzOua4bMQ82s9ue1Z//mC0R+1mapuZ+ta0T1/7yx6N2yXkPLt9+qI5757p1Y/OPx8Jt46dH1Jfqz9drnH/s7p025770fOp5JFb8fPMrKoVOxe9qXj79/tTE/fv1+HzU+F0y8e3vq/24LODz57XW/3PR9d/czjc4zbrjR0+n0lcVD9yHpQl3JLuX/+WWr+6T1/bt1/fGBqM3rFyetCbath4yqxhZwCc8/eZ/Wp89rzeD5i9cvb70zQPNrXq5aR1w8z38evB+f/MQX27tydcc8sU//i9IaWfNX34rNJX39SlQ/Z8lVzrg/sP6qXrdv0+B5KnCCCAAAIIIIAAAghUisAq4Za86YEp1dbV6annhnXZWVDdG8V1aVhHGhrV0LhXTzWc0DU7qMoJk9xwy0y1u3aiUU/VN6qhYbeeem4gcWmj/O1zOrB7tz9tsH6vjlwqjGCafe2EGnb7Uwqf2n1I/XeCEVLxslvOadKGYE57LH2kjt27deDslD9KygRY1bFRYHYncwdJW79pv1tHbL/8nQG11/lTPp/a3aK+285ILnPjve5mPbW7UQ31ddrllpNJ+8s+RQYMOWt9PZpSX8tuPdVgplLWFdrttHGznj6R4ZbBerDw0Aun4iO4SoE0+5hwzJSxocfjRS0+XNTjhPBs1XITplqtus9GNzB1Jt1hzpT78RqOo2jbTbhVuNjeUHM/XlR8Bl5YXtH6wy0q58lK5hto5eOHi8X7cpVyHz+MjnpcZfOyfJw0ldcreI19WbScsrSyjIWY34YkZnO8pfgXO/8jvzXz+pGdlug03Zwb6/pNcsrgKQIIIIAAAggggAACO0IgYQrequ020/Js8LTCxvmFXOJAGZk6F5ywyC2jxLLtLvlckTrsBkl/Tf0ltN/smo/MP4wVZsopchixLZe/zOeTbZZvWbZ3nthwywiZUVdmWqFZg8ssDL/aw2xjtjX7rGvE1moVfIY/X5y6qR/1PquvOXfZ+wxzcOgIlF/gwev69r5n1fd6Ru9PZfTjM8+qMViPrvyVUSICCCCAAAIIIIAAAgggUDkCT3S4ZZjNellmQXhzx8MP/7CgR48+9qYrmiDL/DPTDz/66GNlP3zgbWO2XesaW5XTnZXbksXptzWRuZc8Vaxym03LENhZAg/vaeLqefX19urq61N6EBnJtbMOhdYigAACCCCAAAIIIIAAAqUKPPHhloV4/PhPWvjjQ/32dx/oN3fnvNFZZoSWeT73uw+0sPBQZhseCCCAAAIIIIAAAggggAACCCCAAAI7R+AzE27tnC6hpQgggAACCCCAAAIIIIAAAggggAACpQoQbpUqxXYIIIAAAggggAACCCCAAAIIIIAAAhUnQLhVcV1CgxBAAAEEEEAAAQQQQAABBBBAAAEEShUg3CpViu0QQAABBBBAAAEEEEAAAQQQQAABBCpOgHCr4rqEBiGAAAIIIIAAAggggAACCCCAAAIIlCpAuFWqFNshgAACCCCAAAIIIIAAAggggAACCFScAOFWxXUJDUIAAQQQQAABBBBAAAEEEEAAAQQQKFWAcKtUKbZDAAEEEEAAAQQQQAABBBBAAAEEEKg4AcKtiusSGoQAAggggAACCCCAAAIIIIAAAgggUKoA4VapUmyHAAIIIIAAAggggAACCCCAAAIIIFBxAoRbFdclNAgBBBBAAAEEEEAAAQQQQAABBBBAoFQBwq1SpdgOAQQQQAABBBBAAAEEEEAAAQQQQKDiBAi3Kq5LaBACCCCAAAIIIIAAAggggAACCCCAQKkChFulSrEdAggggAACCCCAAAIIIIAAAggggEDFCRBuVVyX0CAEEEAAAQQQQAABBBBAAAEEEEAAgVIFCLdKlWI7BBBAAAEEEEAAAQQQQAABBBBAAIGKEyDcqrguoUEIIIAAAggggAACCCCAAAIIIIAAAqUKEG6VKsV2CCCAAAIIIIAAAggggAACCCCAAAIVJ0C4VXFdQoMQQAABBBBAAAEEEEAAAQQQQAABBEoVINwqVYrtEEAAAQQQQAABBBBAAAEEEEAAAQQqToBwq+K6hAYhgAACCCCAAAIIIIAAAggggAACCJQqQLhVqhTbIYAAAggggAACCCCAAAIIIIAAAghUnADhVsV1CQ1CAAEEEEAAAQQQQAABBBBAAAEEEChVgHCrVCm2QwABBBBAAAEEEEAAAQQQQAABBBCoOAHCrYrrEhqEAAIIIIAAAggggAACCCCAAAIIIFCqAOFWqVJshwACCCCAAAIIIIAAAggggAACCCBQcQI7Mtya+fWvdaH/73XkPx/Vv2/4f/Sv6/6N/s//6//W/1FTyz8MOAc+g+eA+f6b3wHze2B+F8zvg/md4IEAAggggAACCCCAAAIIIPDkC+yocOu/X31V//GZA/p3X/iSXvzGN/Xqq/9D7777rh4+zOnTTz998nuLI0QAgUQB8/03vwPm98D8LpjfB/M7YX4vzO8GDwQQQAABBBBAAAEEEEAAgSdXYEeEW29cv+6NyGj9T226cePNJ7c3ODIEECirgPm9ML8bZkSX+R3hgQACCCCAAAIIIIAAAggg8OQJVHy4dfKbHdr7tQZCrSfv3OOIENgyARNymd8R83vCAwEEEEAAAQQQQAABBBBA4MkSqNhw6/e//703pchML2LK4ZN10nE0CGyHgPkdMb8nZqqi+X3hgQACCCCAAAIIIIAAAggg8GQIVGS4ZS48zTSil19+5clQ5igQQKBiBMzvivl9IeCqmC6hIQgggAACCCCAAAIIIIDAhgQqMtwyIysItjbUr+yMAAIrCJjfF/M7wwMBBBBAAAEEEEAAAQQQQGDnC1RcuGXWxDFTh3gggAACmylgfmdYg2szhSkbAQQQQAABBBBAAAEEENgagYoKt8zdzMyiz6yxtTWdTy0IfJYFzO+M+b3hLoqf5bOAY0cAAQQQQAABBBBAAIEnQaCiwi2zDo65qxkPBBBAYCsEzO+N+d3hgQACCCCAAAIIIIAAAgggsHMFKibc+u9XX1Xrf2rbuZK0HAEEdqSA+d0xvz88EEAAAQQQQAABBBBAAAEEdqZAxYRbZnFnRm3tzJOIViOwkwXM7w6Ly+/kHqTtCCCAAAIIIIAAAggg8FkXqIhwa+bXv9a/+8KXytcX84Nqra7RgUtz5SvTlpTPKbeQU+6RfYO/ZRd4FBjny15yocClvN+PuQ1WUq5yCi3b2LMKaM/sa5060H5Okyt8R/K3z6m9pVPXNuEruh5A8/tjfod4IIAAAggggAACCCCAAAII7DyBigi3LvT/fXnvkLiJ4Vb2SpuqqmtU1Z2pqN7O3RnX2M1xTcxtMKypgKOa6K7xjFuvZDevNbe6/X5sHdSGailXOeU60m1vT1ZDbab/UuoJviJJ5+ZEOuX38asb0i+Xmvf7Y36HeCCAAAIIIIAAAggggAACCOw8gYoIt47856N69dX/UT69z2C4tSWBUPl6aMWStuRYyhUClaucFUXW8GEltCc3o4nbhdAqsT+Xspq8NaPc0hqObRM3Nb8/5neIBwIIIIAAAggggAACCCCAwM4TqIhwy9yt7N133y2f3haGW7NvpNWTTqv/Vi5of04TF/33rt01b9nXA5pYyGrsYqfaW46qZ3hGZoxVdnxAHe2Naj85oDGbB+Qy6k+bMq5rOjeloe5DamjvVP9Nu0GUyrThSJM/2mlPW6d6LmZkW6PcjEYudutIi6njnC7fSi4jf2dYp02dF5x9l2Z0zWvHOY3NB3XOZ3T5bKfaG1t05NSARmbCmqS71z0Lt/7crQH/vTf8+WfW69pMVmNnj+pAy4Cmo4ejxDAkVm/YHrvvkrFNe5YHXuhW3/BUNDjxPjftPqSOKxnlxhNGbi3lNH1jQF0vtHjekTJsn1zMaPbOsLraG9VxIyc5YdLs/Lj6Tx7SgRfSupyJOcfLvpJRNhbs5KaG1XfqqA6YNqYHCubuOTQ3458PnaN+H5dyXNbI6Z/s/Lj6Xmjx2jo05fSh2baEtmZvDqjn5CE1mLaeHdSEPT/CtqZlzv9i56Y9DwrfG0mxPk48t96YU/7uaODcrf7xmLM91jX+Nb8/3DVxjWhsjgACCCCAAAIIIIAAAghUiEBFhFv/uu7f6OHD2AX2RoC2MNxaHsRkdbnVD5q6bpmDsK9bdOTwXn8qnJnWWJ1S67Gj2uM997evqj+jSRN4BO2vqm5WUxBaeVMhq1M6cHFmmYxtg79NjarsVLuFUXXU+2XX7t6rp1L+8/rujBesRQrKj6vD+/yortmumDrjt68xCKBmBtQUlLGrfq922eO4Eiyc5AQ9NnKIT+O0bW1/7lBg0a2JSEO0PNyaG1SrV29KTzXYeveq6/9v745e4jrTOI7vn2JYWykDwsr2whZLLwZcGPBCkF0ELyxzIQnETqEEd9MgiMyNeBHbQIKBpClEbGtFCFnBTS7CeFPiRYmFbkxJR2g2s62bk61kCspTnnPe98w7Z47OaEY903wPJBnHM+c85/O+OTA/3vc9X5spmOU1mTLX2Znuk0x3MOWt4+yyCYA2ZS4bvGeN0mnTFtZKynI/b97r6q0cIzsvRbdNBk/LSE/g6E+btNfcl5Vh875n2X81AAAQYUlEQVRtq9FbFtKTO+fNsVPdkjb1taXzct+sS1VayEqHeqa6JdNXucaZ9eo+NHLm3cp0yp1GrsvBDWsdCNsxqLVLLtyz01nr1/rgkuNka01lZdEPuGx/bxft/7a9rbvtm/b9cOppo33rTE5GTR8MjtkrU2u2dudaD/hS7z96H2JDAAEEEEAAAQQQQAABBBBoPYFEhFuvvf6G7O7uHk5ve02mMl0ybAMWPUpcuLW5LOfS78rkPRs4HO50e4U14Zf0MMwKvtxXwq126b+45o8mKn5m1u3qyUtBy/HummDpXfHDjDDcqtTr3cubIGxc7sR8l68JC0SDhSDQCcMsb1Um/QCmV658F73+shTGg/0v3A1O8PBanx+k9H+m4ZUNLVIV68c3/IX721JjQSBmw5MwMBLZy0uDncKTspS3ay+m+lo8WTyjYZITZtmwq/+GFEWkvL4UjA77fD0I7XZWZdIP3rKyWBIp3xsPgiPrveNJYcIJiZTimyDI67BhVhh2pYLgJ2yTlAzf3BBvuyxlLd1es1Nf8aYJqno+kQd67K/zwfltmOWcP31Zw8qSFK7qSL1ZuWNywuLNId++Y1oXrrL27aL1PfQCt4auy23msNasXPdHFYqEfdEGmHVrXZMpP1waCI9hR+cFo7AqtQbhbiXgqvwfib5nP9NA30plZW6j7I8uu2PbsAnr3+n9R+9DbAgggAACCCCAAAIIIIAAAq0n0PrhlgY2/qgd54txNNzygy0TkKyeXLhlv+yHgUgYAtkv9yYQC4MUd1TTusz4wZQJwCJ9rToQ0l/a/fvkujPYq3Y/50Am2OjwwwJbkwkxvGUZ9QMjN1yz+5i6bXgSXtfe4ZYbdDgV+C+ragzPOyCj/hRJDYHGzSipILzyP1QuycOvV2VRp4SeHzKjyrIy90TEhnQ9l/xhUMHp1qaDwMnU+uBSEHb50zrteT4YqARMsW3ihFuD837QFtRyVy444Zo9duaa0xAxVuXH61JYmZcr03kZzZiRZlVtYQPT4BIaua5gT/N3zDkl9A1G7NWv1ZPbHwa1dXQPyOjErMytrEsxzCgjfcIPWZ2RbqaU+DY+WN8SO7WUcKuqmfkBAQQQQAABBBBAAAEEEHjVBBIRbr30tMTtteqAyw23NpdkpKs5wZZ2jr1GIlXCmuiX++jPTiAShkCRfWKDlE2ZGwxCgjAkc3prVVjgv78Wjl7SgMdutfvZ3+haSyYQ0xFHNvQwo6PsaLi2U27gFqk7Jjyp7+Wc37ysqjG06JJ0X59kqv6MyW2d/xhOadNpi7q2WE76Tbik126fzFcVLm3ckIzuY9rAnrOjO3qOPslcXgtHA1Zff1xb6kVU29tjV/pI9HPOVECdEjmUk3NZM7Jsn3Crkeuq0o1pH3FGuflWcU+qjH5ue0MWp3MymO4KpkiqY6pPZvzpgZE+0Ui4FbbxwfpWGBI3IdxiWmJVT+EHBBBAAAEEEEAAAQQQQKClBBIRbjVlQXk34MrnZfBUu/SP518u2Cotyzk/TJmWghmZ8uBiEDoM3gzmj9ngInPVjMrZ2ZArfW4IVftlP/xSXjfcqkz9kq0lGXFGA0V7ma2jEqB4cvtsUMe5FTusZlOumzW87NTD6HGC6+uTmYvjfnBROV5lJFg4pTFcp8uMDrMhSN+sPDSLpT+8GkxtbDMBRG2d0QqiU9bsNLgxuWPWpxIpi/fEE2/Lk/JOJbwKrzMS2Oh6Vv76TGeWwoX2w9DNtIHdJ2xHLcsrSWnLE0/PGxvAOCFVarxS33ezQXB2Ki+FHRHvVi44/4d3w7XOip8Fo8Lazt+VcmkpmN7puNlpiYFbTB/SoLWB66rSte2TGg/7s52O2Zaa9tc+q1vrTtl3L36zYdYz86Qw7bZxba1xbV793gH6Vvh/xrFvQrjFgvJVPYUfEEAAAQQQQAABBBBAAIGWEkhEuDX6fk4WFr56ebgw4ApCHbvg9OShpyJWwqDOoXGZmjgtaT9gqoROYRhwqkv6P8jJcHev9JiFxYMRVrVf9hsPt1LSkTktk840tXCR9IiWnU7WMTAWPi2xvGrW6UoNyOTn83LlbJ9Z+8ksXB85hv+jWXsqsDsti1uVnXQtKf/99Gm58vm8TA4FI3fCmuxor1Pt0tmfk9Fst6R73BFI0eCqcmz3VXXwodMKzfTAgbzMrdyVuYmB4DrMqDJ77enxeSncW5W5CXOdp4JpibK1bBYhT0nmbF6mJnKS6UpVTUuUbbvuWa+MXF6SwtKsjJjprv5i63XDrZR0ZnIyOT0ug/5IwXbpsQv3b9u1zlLSPzEvc5dzkvHXreoN1lizbqmszKysSmFl1oSy7bJfuNXQdbmwYbiVkk7tJ06taTtls16tOrrPuPRPL0nh3pJMDQTTFKvXZqtMobTt4/bNaBvX7Vu29iMKt/T+o/chNgQQQAABBBBAAAEEEEAAgdYTSES4de36p/L3f3zUHL2qgKtXDh9smXI2l+VCvzP9qmtIZlbtswB1Kt+m3B6zYUqXDF5elztmatfLh1t5mbs1ZoKQdukcmpUH4eilCNfWqkzadZrMKBzdo3hrXPpN2KLBVOfQJ1Jwyo8cxVmrq13anJFOwX5leXA5Gz51UZ/4mBmbl4dOTcVovSv5IBA79MgtNfbkfuS8b2Zn5b5dPm17TWbCNkpJZnpervhPrDThli46/81sGDq1dZ2WxdX5YLSUG5Y8Xoq09YBcWDIrvNcLt4ZvyO2rdq2vlLx59kaVi9TpR8WFXMVV67vpusUEpKbhGrou28hhQDQri5crtWbGloMnQtr96tRacy2pbhm+uColf7ReTK0xfTMabulovH37Vlj7vITd177XhJFbev/R+xAbAggggAACCCCAAAIIIIBA6wkkItzaePRI3nr7nebp+QFXE4Itt6KyJ55np/e5vziC19EgRaeCOQHSvmeMefqg7l/W6XUNlW+fTtguI0s2PYqc0UxN0ymBx7qZ8+55HdvBNMX9aop7OmPN/ts65bEhrJqPita430f12HvtoJ/d63e1Z6p6p6HrsmGQDfReplY9u/6fMFNDq4rZ64c9+mbV7qaNj7tv6f1H70NsCCCAAAIIIIAAAggggAACrSeQiHBL2f76t0FZWflX6wkeRcXRcOsozhFzzNJCTjJ2kfD0tNzfL6SJ+TxvJVwgGm4lvNzjKk/vO3r/YUMAAQQQQAABBBBAAAEEEGhNgcSEW198uSDD72VbU7HZVYcL2c/6i3w3+/Dxxyv7C4P7TyMcqTd1Mf4IvJtwgbXZ4GmTY8uVqX0JL/k4ytP7jt5/2BBAAAEEEEAAAQQQQAABBFpTIDHhlvLpUxMZvdWaHYmqEWhFAb3f6H2HDQEEEEAAAQQQQAABBBBAoHUFEhVu/XN5WXr/kpHd3d3WFaVyBBBoCQG9z+j9Ru87bAgggAACCCCAAAIIIIAAAq0rkKhwSxnPf3SheU9ObN12oXIEEDhiAX1Cot5v2BBAAAEEEEAAAQQQQAABBFpbIHHhlnLq4s4ff3yptWWpHgEEEiug9xcWkU9s81AYAggggAACCCCAAAIIIHAggUSGW0+fPvXXwSHgOlBbsjMCCDQgoPcVXWdL7zNsCCCAAAIIIIAAAggggAACrS+QyHBLWfWLp46s0KlDrMHV+h2NK0DgpAX0PqL3E72vEGyddGtwfgQQQAABBBBAAAEEEECgeQKJDbfsJeqaOLroM09RtCL8iwACBxXQ+4feR1hj66By7I8AAggggAACCCCAAAIIJF8g8eGWEurTzHQa0fB7WUKu5PcpKkQgMQIaaul9Q+8fPBUxMc1CIQgggAACCCCAAAIIIIBAUwVaItyyV/zFlwv+lKK33n7Hn160sPCVfPvtt/LsmcfURYvEvwi8ggI65VDvA3o/0PuCTj/U+4ROQdT7BhsCCCCAAAIIIIAAAggggMDvV6Clwi3bDBuPHsm165/K6Ps5f0TGn7r+LK+9/ob8sb2DPxjQB17BPqD///U+oCO09L6g9we9T7AhgAACCCCAAAIIIIAAAgj8/gVaMtz6/TcLV4gAAggggAACCCCAAAIIIIAAAggg0IgA4VYjSuyDAAIIIIAAAggggAACCCCAAAIIIJBIAcKtRDYLRSGAAAIIIIAAAggggAACCCCAAAIINCJAuNWIEvsggAACCCCAAAIIIIAAAggggAACCCRSgHArkc1CUQgggAACCCCAAAIIIIAAAggggAACjQgQbjWixD4IIIAAAggggAACCCCAAAIIIIAAAokUINxKZLNQFAIIIIAAAggggAACCCCAAAIIIIBAIwKEW40osQ8CCCCAAAIIIIAAAggggAACCCCAQCIFCLcS2SwUhQACCCCAAAIIIIAAAggggAACCCDQiADhViNK7IMAAggggAACCCCAAAIIIIAAAgggkEiBuuHWxvdF2dnZSWTxFIUAAggggAACCCCAAAIIIIAAAggg8OoKaGal2ZW7/cH9QV//sPmjvHhRjr7NzwgggAACCCCAAAIIIIAAAggggAACCJyogGZWml25W0249bT0szzznrv78BoBBBBAAAEEEEAAAQQQQAABBBBAAIETF9DMSrMrd6sJt7znv8iPT0ruPrxGAAEEEEAAAQQQQAABBBBAAAEEEEDgxAU0s9Lsyt1qwq3d3V3RhblYd8tl4jUCCCCAAAIIIIAAAggggAACCCCAwEkKaFalmZVmV+5WE27pL3V4189bz9z9eI0AAggggAACCCCAAAIIIIAAAggggMCJCWhWFZ2SqMXEhlvlX3+Vje9/YPTWiTUXJ0YAAQQQQAABBBBAAAEEEEAAAQQQsALBUxJ/EM2soltsuKU7/fen/8l/Sj9F9+dnBBBAAAEEEEAAAQQQQAABBBBAAAEEjlVAMyrNquK2PcMt3VkfrciTE+PYeA8BBBBAAAEEEEAAAQQQQAABBBBA4DgENJvSjGqvbd9wK5ieWJTn/69ehX6vg/E+AggggAACCCCAAAIIIIAAAggggAACzRLQTGrj+2LsdER7jn3DLd3pxYuyfxBGcFky/kUAAQQQQAABBBBAAAEEEEAAAQQQOGoBzaI02NJsar+tbrilH9YRXDr8S+c36gJebAgggAACCCCAAAIIIIAAAggggAACCByFgGZPmkFpFhW3gHz0nA2FW/ZDunCXPkVRH71IyGVV+BcBBBBAAAEEEEAAAQQQQAABBBBA4GUFNGvSzEmzp70Wj487x4HCLT2AJmZPSz/Lvzcey49PSv6C8zo8jLArjpf3EEAAAQQQQAABBBBAAAEEEEAAAQTiBDRL0kxJpx9qxqRZk2ZOjYzWco934HDLfnh3d1e857/4J9VhYjoHUovgDwb0AfoAfYA+QB+gD9AH6AP0AfoAfYA+QB+gD9AH6AP1+oBmSZopaaClGZNmTYfZDh1uHeZkfAYBBBBAAAEEEEAAAQQQQAABBBBAAIFmChBuNVOTYyGAAAIIIIAAAggggAACCCCAAAIIHKsA4daxcnMyBBBAAAEEEEAAAQQQQAABBBBAAIFmChBuNVOTYyGAAAIIIIAAAggggAACCCCAAAIIHKsA4daxcnMyBBBAAAEEEEAAAQQQQAABBBBAAIFmCvwGG4GsTV6BanYAAAAASUVORK5CYII="}}},{"cell_type":"markdown","source":"Submitting the resulting CSV file, we get a final leaderboard score shown in the screenshot above which is a solid result for this project! Unfortunately, I could not make a screenshot of my exact leaderboard position since the button that makes me jump to that position did not work properly. ","metadata":{}},{"cell_type":"markdown","source":"# 7. Conclusion\n\n## 7.1 Result Summary\n\nAs we have already seen in the previous section, the dense network performed better than the CNN network because of its ability to capture more complex structures in our images. However, both models did a good job on the training data. Looking at the performance of the dense network on the test data we saw that we obtain a really good final score.","metadata":{}},{"cell_type":"markdown","source":"## 7.2 Learnings and Takeaways\n\nWorking with neural networks requires a different style of work compared to simpler supervised learning techniques like regression analysis. Getting results from model training might last several hours. Therefore, it is really important to work clean and precise so that you avoid having many training iterations. One aspect that might be underestimated but helped me a lot during this project is the usage of print statements in the code. This way you can debug your code better, inspect intermediate results and find errors more easily. Besides that aspect, image data in particalur requires quite different preprocessing steps compared to e.g. textual data. Techniques like image augmentation can really improve your model if used correctly.\n\nAnother problem I encountered during my work was the memory limitations of the GPU on Kaggle. On way to solve this issue was to reduce the batch size from 256 down to 128 which leads to less RAM consumptions during training and evaluating. Additionaly, I had to call the garbage collector from time to time in order to free unused memory.","metadata":{}},{"cell_type":"markdown","source":"## 7.3 What didn't work\n\nOverall, everything worked quite well. However, it would have been great to work with more epochs in order to get more precise results. The limited GPU memory really was a problem in this project since image data requires so much disk space. It took quite a few iterations to find suitable values for the number of included images, the batch size or number of epochs while still getting meaningful results. ","metadata":{}},{"cell_type":"markdown","source":"## 7.4 Possible improvements\n\nOne possible improvement would be to include the whole image set instead of just a smaller amount due to memory issues. Also using more epochs would possibly enhance model performance. It might be also interesting to see how much better the model would perform with a balanced image set compared to the original unbalanced one.\n\nIf we had enough computing power, we could also expand our hyperparameter tuning by including more parameters and a wider value range.","metadata":{}}]}