{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Incredible TPUs - finetune EffNetB0-B6 at once\n\n> ### **Updated on 2020-06-27**\n> * Fix: generating of submission files for single models\n> * Change: reverted TTA repeats to 25. Local tests showed 20-25 repeats are enough for used augmentation pipeline.\n\n> ### **Updated on 2020-06-22**\n> * Fix: description of used dataset\n> * Change: more code cleaning\n> * Change: increased TTA repeats to 50 and using batchsize 256 * 8 replicas (=2048 total batchsize !!)\n\n\n### Following notebook uses:\n* Tensorflow and TPUs\n* 224x224 sized pretrained EfficientNetB0 to B6 combined into one huge model -> Total params: 116,336,013 (!!) \n* The scaled down dataset [Melanoma TFRecords 256x256](https://www.kaggle.com/cdeotte/melanoma-256x256) from Chris Deotte - Thank you :-)\n* Multiple image augmentations, including rotation, shear, color, contrast, random cropping, ....\n* TTA (Test time augmentation) with 50 repeats, done in one predict()-call with repeated test dataset\n* BCE (Binary Crossentropy) with a label smoothing factor of 0.05\n\nInternally all pretrained CNNs are combined into a single Model, with a single input (=image) and multiple outputs, one \"normal\" output with BCE-loss for each \"sub\"-network.\n\n![image.png](attachment:image.png)\n\n### There is **only one model.fit()** and **only one model.predict()** call. Is this a \"single model\"-model?!  ;-)\n\nI do not know if this is a feasable approach to use TPUs in a efficient way. The preparation time in the first epoch is huge >15min. It seams you gain nothing compared to sequentially training of the models, but I did not measure and compare it.","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAmwAAAFVCAYAAABSCLPdAAAgAElEQVR4Aey9eXRVVZY/zj+/rlqruqutWnbp0q4vSlUhKCrgCKhNCRJwIKDIHEBlkFnmACEkzBAhA2RgRmYyMEcRQxjyhoQgCpYWXdXarlUEdXWt1qpua6ldun/rcx77ct7Nm9+7yX3v7qx11jl3n7332Xeffc7+3HPfe2nzww8/kBTxgcSAxIDEgMSAxIDEgMSAfWOgDcmfeEA8IB4QD4gHxAPiAfGArT0ggM3W0yPGiQfEA+IB8YB4QDwgHiASwCZRIB4QD4gHxAPiAfGAeMDmHhDAZvMJEvPEA+IB8YB4QDwgHhAPCGCTGBAPiAfEA+IB8YB4QDxgcw8IYLP5BIl54gHxgHhAPCAeEA+IBwSwSQyIB8QD4gHxgHhAPCAesLkHBLDZfILEPPGAeEA8IB4QD4gHxAMC2CQGxAPiAfGAeEA8IB4QD9jcAwLYbD5BYp54QDwgHhAPiAfEA+IBAWwSA+IB8YB4QDwgHhAPiAds7gEBbDafIDFPPCAeEA+IB8QD4gHxgAA2iQHxgHhAPCAeEA+IB8QDNveAADabT5CYJx4QD4gHxAPiAfGAeEAAm8SAeEA8IB4QD4gHxAPiAZt7QACbzSdIzBMPiAfEA+IB8YB4QDwggE1iQDwgHhAPiAfEA+IB8YDNPSCAzeYTJOaJB8QD4gHxgHhAPCAeEMAmMSAeEA+IB8QD4gHxgHjA5h4QwGbzCRLzxAPiAfGAeEA8IB4QDwhgkxgQD4gHxAPiAfGAeEA8YHMPCGCz+QSJeeIB8YB4QDwgHhAPiAcEsEkMiAfEA+IB8YB4QDwgHrC5BwSw2XyCxDzxgHhAPCAeEA+IB8QDAtgkBsQD4gHxgHhAPCAeEA/Y3AMC2Gw+QXYyr6amhqSIDyQGJAacFgN22ofFFud6QACbc+c+6jvfs2cPeb1eKeIDiQGJAcfEAPY9+RMP2MEDAtjsMAtJYgM2rsuXL0sRH0gMSAw4JgYEsCVJgnKAmQLYHDDJibpFAWwCVgWwSww4LQYEsCUqg4ieeD0ggC1eDzpIXgCbJGunJWu5X4l5AWwOSnI2v1UBbDafIDuZJ4BNkpcAGIkBp8WAADY7ZSFn2yKAzdnzH9Xd2w2wbdmyhR566CFbfJYGdqxevdoWtjgtocr9Coi0MgYEsEWVJoTZQg8IYLPQuamm2m6Abc2aNdSmTZuQIOnpp5+mMWPGhORJxGYPO2bMmGH5OImwVXQIwJEYiDwGBLClWiZL3vsRwJa8c9filmPjunTpkm0KTrQAlELZ9Mtf/pLS09ND8oSSj7QPdrz22muWjVNVVaVOE/H7V5HaJHz2iVWZi+SdCwFsLZ5qZMAgHhDAFsQxQm7ugWQEbLt376ajR49aDnKsBmw7duxQ4LS6utryexFwkbzgQuYu8XMngK15LhBK63hAAFvr+D0pR40EsD344IPEZc6cOQa4wMkQ07nWkwt4UVatWmXw4VUmeEBnGfSzHJ+wmXXrp1DQodsBPZWVlYRXpaxz8+bNhk7oNutjO3hc1CyLGvKhABtshg7wsRzG1/Whrd8n+GAn0zt06KDGuPfee5UOs81mXXKd+MQtPnWmTwWwJWW6SkmjBbCl5LRac1PhABtAxqBBg2jr1q2q/Pa3v1WgA8ADQGPChAl+feDnJDh48GD6xS9+QRkZGYoHIOef//mfFTiZNGmSokH+X//1Xw0gw4ANuvUxcc16u3fvTtDN1wBWAD+LFy82dLZr106BNPAArJlthQ4dtAFsderUyRjz0UcfDQnY8KoU99K/f39DBq9qdSCJ9j333GP0Z2ZmKjvhO5wQ4hq2wy+419OnTxv3xPcmtTMBhcy7tfMugM2afCJao/eAALbofeZYiVCADUACgOLtt982gERFRQW53W6aOHGiAjg6yAAIYQCChANQBRDEry/Pnz+vrgH6WCfkIZOfn6/GYMAGAMNJC23w8AlUIMAG4Ae7IMM6N2zYoK5zc3Ob2bpy5UoFuADmAKB+9KMfUUlJiTEmXrtizGCfYQMdYBT+YDtHjBihABpfox8gkq9hH/wB34Emr0StTcrsd6nFz+YYEMDm2JRnuxsXwGa7KbGvQaEAG07GADDMmx2ucXL0yiuvNOsDP+TAA8AGcKXLm8EW+gCM5s6dq/gYsOkyAHoAP8xj1gF5yOkyus6+ffsaJ3v8+pJfR+LzYwCL4Mc4Zh2hABtO1HR+8AL4gQYQCJ0Yh8dEjVM5Ph0UwCZAQo8fabdcPAhgs29OcpplAticNuNx3G8owBYIcHFSAVgJBGZ0MBVIXu9nXQA2rCsQYAOfPp5ZB+QDATbWCX6c6vErVr0GSAs2pm4X28o1dAcCbJABD4Mxft2pj8knjswjXzpouUTN8ye1s30ugC2OpCGiCfWAALaEujO1lYUCbPicGU6EAiW3Bx54gHByZe4Df1ZWlqInCrDhtSVOrpYuXar0RgvY8BMgOBE028rXmzZtUqdhGIdpqOMBbIcPH1by/FpW18ttAWzOBg0cB1K3fBwIYEvtvJZMdyeALZlmq5VtDQXY+HNcOCXipIIP6uN1H38GjD9Xhn58yB6vLvlzbfEANv3LC9OmTfPTGy1gw+kWAJ9+H7gH/tIBPlsGcIZx+D7xJYR4ABtO7nCqhy8/sE7U7D+0GdSFmgNdVtotn9jF56npcwFsrZx4ZHjDAwLYDFdII5wHQoEFgA6AHHyLkz+HhdMqABwUfMsRgIT7cIoFkMdJLlbABtAH3awX17reaAFboPt44okn/L4wgPvEODwmviyAz+Pxa1W+J67DvRIFH159ArSxTtQAtfAd+hnU8efcACJZv9SpCRRkXu0xrwLYwmUG6W8pDwhgaylPp8A4oQAbg4pAn79CH4CH3qeDKvQDsOjfogQN1/wZLlyjQIf+rVHoMetm3kA6IM+nesyn6wQN4CiUreZ+jA9b2S7WyzXo5vsFDWMwD2rcqz6u2U69n4GcLi9teyR4mYfUmgcBbCmQvFLkFgSwpchEtsRthANskqhSK1HJfMp8SgxcIgFsLZFdZIxIPCCALRIvCY/ygAA2SeCSwCUGnBYDAtgkAdrFAwLY7DITSWCHADZJ1k5L1nK/EvMC2JIgOTnERAFsDpnoRNymADZJXgJgJAacFgMC2BKRPURHIjwggC0RXnSIDmxc77//vhTxgcSAxIBjYkAAm0MSXBLcpgC2JJgku5gogE3AqgB2iQGnxYAANrtkILFDAJvEQMQeEMAmydppyVruV2JeAFvEKUIYLfaAADaLHZxK6gWwSfISACMx4LQYEMCWSlksue9FAFtyz1+LWi+ATZK105K13K/EvAC2Fk0zMlgIDwhgC+Ec6fL3gAA2SV4CYCQGnBYDAtj884BctZ4HBLC1nu+TbmQBbJKsnZas5X4l5gWwJV2qSlmDBbCl7NQm/sYEsEnyEgAjMeC0GBDAlvhcIhpj84AAttj85kgpAWySrJ2WrOV+JeYFsDky3dnypgWw2XJa7GmUADZJXgJgJAacFgMC2OyZj5xolQA2J856jPcsgE2StdOStdyvxLwAthgThogl3AMC2BLu0tRVKIBNkpcAGIkBp8WAALbUzWnJdmcC2JJtxlrRXgFskqydlqzlfiXmBbC1YtKRof08IIDNzx1yEcoDAtgkeQmAkRhwWgwIYAuVFaSvJT0ggK0lvZ3kYwlgk2TttGQt9ysxL4AtyRNXCpkvgC2FJtPqW8HG9d5770kRH0gMSAw4JgYEsFmdWUR/pB4QwBapp4SPBLAJWBXALjHgtBgQwCbJzy4eEMBml5lIAjsEsEmybqlknbn/dyTFWT5oqdiKdhwnA7Zlxz8lKdb7INL0L4AtUk8Jn5ywyWuwFnsNNmLzh7S59qoUh/gA8x0tkGopficDNlmH1u9B8HGkfwLYIvWU8AlgE8DWYkkVm9ihhs+lOMQHAtjsmWBkHVq/Bwlgs2fsJ71V8kpUXom21KkGNrGDDZ9LcYgPBLDZMz3IOrR+DxLAZs/YT3qrBLAJYGtJwFZV/zlJcYYPBLDZMz1gXmQNWrsGBbDZM/aT3ioBbALYWhKwVXg/I6vKY32ep3u6dleFx+Br9DFNauvmQPetADZ7pgfMiz5PydbGml68ocLW9yCAzZ6xn/RWCWATwNaSgK3c8xnFUrLXVxhgjEEY6um5pUof6ltuv4NyiqtUwRijp+fSrzp2VtfLN1cT63h91+mwNkAW+mOxNRqZjcfeN+4LbV0W95QIG3Qgy74L5AMzn25LLG0BbPZMD5iXWObTLMPriWMKdf8RkwLq5lhmXp2P1xr3cR0oRmFDmzZtaPoS37o322SXawFs9oz9pLdKAJsAtpYEbAfcn1EsZcxrS9VGvXhDFeml5OAFpa9773RC0XUDrA3ImGrQkDSw2a/be86g6fx6u8/A0fSzm28Ny6fLhGtjfIAinW/FlreUTbBr2Kvz/fpGTMpSfTp/qHbertMK4JUdfd9Pzz/86MfUe0CG4Te072zfiXQ+2AX/sW8fejxN6Qo1Xrg+AWz2TA+Yl3BzF0l/oDX5jz+9iUZPy/XTD3B2y+1tjdhCjGFdMt9DT/RVD1Yce1xvP/kHPz1skwJsuaUB+5intWsBbPaM/aS3SgCbALaWBGz7XdcoljJ8og+8BJO9/5Ge9NTA0X66ceIGOZbZdOwyZa+vol21/2nQuM9cF+xz0bJN1WH5zHKhrmELbNJ5ckoOK1D24ONp9PObb6WyI+8Z/eHuWdeDNusqKvcaOkBHgtP9UHzwgqLNy9ul+BYVVhBAHRIl60QbcuhjWrS1ALbWTQ/ffPd9QAMwL9HOZSD+QPHZOz1DgS/mR/wAxE3LKfYbEzGINQY+rF0UlglXIy6n5ZRGzB9OnxX9AtgChp4Q4/WAADYBbC0J2Pa5rlEshZODWbb0yHt0d5fuKingRAztHk89r2qAkFtua6va2ODX7KxVbciwnqzCCkWDHAp40Ad+6GE+1KOm5QbkZf7nhk8iXZ8ujzZsgU08FuxgkDVj+RZ1opc26GVjzED3HMwG0O9o30mBrN/c01WNAVtgGwM2vheMC9rUnFLVn54xVY29s/YTY2y04U/0sVy0tQC2eHfn2OUv/uEryt76e6r/6L+bKcG8RDuXgfgDxefTQ8arhxLmf6LfYMLDiB5b3Mf1fY/0JBS+Dlfrscu8vKa41tc4eJiOmtc46Pp6Qpv1xVsLYGsWdkJIhAcEsAlga0nAtrfuGsVShl0/YTPL7jj1CWUVVamn+gceS1PtJRurVQ3A0Ss9Q7WLqi7Q4mLfaVbBAa+yYWFhBd1+R3uavLhY8aDu0r236sN4OA3j8TKm+j4Ph7FQRr+2TAGkVW/UGvw4SUCCYh7IQw46YBNsgU3cD9vZJtRDJsxXwFPXieQUiQ1r97mUTeDn+yk9elnJgob7YT0T5ucrwMb9Dz7elzrc/6jRz3yg9R4wuhmd+8PVAtgSsUPHpuPL//mOTjR8QUt2XKGiqo/pT1/8zVCEeQk3d5H0B1qTiO9+g8cZ+rEmBr0yx7gOpPe+h3sSSqC+QDTE85ScUsVffNj3wMbrHGsL7V/f05XQB/lnh08i3htUf/+Rio71r69H2MlrL9C40dAEsBnhJo1EekAAmwC2lgRsu881USxl6KsLFcjo2KU7cenx1EBDFzb83gNGGdcYA5+bgRyPt2jDIaVj3X6PonV68HG1kW+v+URdo84tO67akIM8y2JjHzs3z7je/NYVBRJfeGmWwf+zm2+hldtrDJ6+L46ldh3uN67NOqGbbUJdcvQyQceT/UcaOpGcIrWBdfH9sRx0/OK2tobf0EYC435OmHzNdSCfcl8ktQC2RO7U0ev62zd/pz9e/V8qP91EC7d8SJVnmwg0zEsk8xeOJ9CaRKzxGoI8ricsKAg5HuIMwI7XNeqRU3OCykDn5MUlqn985jq1ZgorGw1+tLGOhozPVLRftruLXnh5ttHPazRj2hLFx/cJOaxrvo6nFsAWfbyKRAQeEMAmgK0lANuFi+/R4MJLtOtsU0yFk8OCwkriklN23NB178M9qVf6KOMa4wCYQI7HZECzdp9H0fB68qVZq4x+5kMNOcijvXLHKZV42v6mE3Xs0s0oSDI8ps7PekDDGPo162Qa24QaNIBC6MWYkEdyAj0SG1gX3x+PAR0Agew31ACgzwybqHTDdyjMz3Ugn3JfJPXzeRdpwmqvLcvIxTU0q/gDR5RXVl+kp+d5aEjuecrZ9hENK/tds7mOZD7NPByfelwhzn59d1cVr+BH7AEYmWX1a8RZu46d/eIzb09dUBkF2LJLVP+jvdLp0Sf7N+MF7YHH+yr64HGZ1x9Yuhl2YfyCika1PrGmR0zJaaZDtzHatgC2CMCHsETvAQFsAtisBmzu+os0s6CenlncQDvPNMVUhkzwgZdg8vc+1JOeTB/lpxvgCHIsk7Xed8L2+l6PomHjn5RdYvQzH2rIQR5tlpu4qJjmF1T6lTW76xSPzs96zDYH4mHdqCFXVn2F2nXoTJ279VI2wMZIbWBdfH9sB3RgbL5GnT56Bv3kpzdRUdVFerzfEPrXO+/y6wcPaANfmt2MrusJ1R5a8gFVnrhgy7K2rIre/4+vHFEO1F6l4UsbaVrRZTp36c80fNOHMc+pPt/m+EZffrnvdGvgmJlqDJx09X1xXMjxsHZRdN2h2vq6DbTuIYu9gHVuOHxJrdmuPdIID13deg80xsJ6HjltqXqA6di5Gy3ffsroC2VDuD4BbNFjEZGIwAMAbBcvXpQiPrAkBs6536VJeV7qO9tFfea6aceZppjK4OuALZg8b9x6PwAX5Ji24Dpgy9vrUTQkk2dHTDH6mQ815CCP9so3zqiTgpmrdgbkNfOzHrPNuk7mYZtQM21C1nr15P/Ik+lqXNAjsYF18f2xPiQ43Q+gT8wuUbrBO3rmKjUe86MurLqoaDNC3LPOH6iNpGXXvQX7Xqr//fmrb2lr9afqdejhumv0ybWv6dvvvlevRAPNV7Q0c3yzPNYNABOu+wwaq8AQ9wWqsXZRAvUFoiGeEb/ow8PGnXfd30wWNPPaXr27jl5dVKzietm2U4ZMafUVyiyoVA9KA8bMNOiBxo6UJoAt1VdXK92fADYBq1Yl1XfOvUvjVnkofb6b+sxyUe9ZLtpcc5W2n26Kurx4HbAFk+30UE/6bfooP71IHJBjmQVFvhO2NXs9ivbU9WRSUHVRXaPG0zf4IQd5tDe9/TF16ZFGt93RXl2zvn5DJxI2fjM/95ttHpuZr5IF96Nmm1AzveT4FTXeT/7pJgWqQI/EhhXXgWV2abWhC7JIcLofQMO93Nmhs9JbdOiS4hk2OceQ6z3wZfX5HtjCdkVbC2BrpU2diC59/BfK3Pghban+lD789K/016//zzAG8xLtXAbiN8c3eF5dVKJifEruZjXG2gO+EzdeV6xn7rpy4njD2kXhvnA14hnjgA9ACx874GvQ2AaMra9p9M1YuVPFOtaKvn7Rh/U24KXZEdsBmWBFAJsRbtJIpAcEsAlgswKwHXnnXXp5hYfGr/KoemiOh56a7aal+/5I22qboi6vzPN9s7FD526kl2GTcpQuBdj6j/LTC8A1aPxCgza/0AfYVu/xKFre/kb6t+dG0q/u7qp0oh4zK0/1QQ7ybOuKnXUKROljD52USxuOXQnIDznoQHJhHQUHLykQxDpAZ5tQMx/quQWVShangEwPZ0PZCR+wxGsfjLFk6yklCx3/cltbP78BgOZuqTF0j88qpptuvtXgwevQhcXHjX62IZpaAFsid+rodP3XX76lxitf0hdffkM/mEQxL9HMYzDeQGsScYZY4nUBWcTRXfc9asQWYrNH2mBCPKP/sb5DFFjidcH1nLXlAe3EmpqQVaL6EPMYT49vtOfmVxr9L4zNNMa+rW17Gj51KUEO8c9rBWNiTWBPCHa/0dAFsJmCTi4T4wEBbALYrABso5Z6aGZRPb1Ve4FeyHLTsm31lDbfSxMKLtGW2qaoS/7BSzQnv7JZWb6zTunK2VJD3Gb9SBRr9jcaY60/dkXJl5742KChX9cLHsiDDnnWhRr6dV7YxP2B+Fk386CGTtaBa7aJx2Ve2Ai+aGyArG4j69TH5LHNvuLxuN88LtsVTS2ALTF7dCxafviB6O/fm6GaTxPmJZp5DMYbbE3q64tlsT45tlDjmvv0mNV59PXFvKjBo/eZYxf9Or/ZTl4XrIvHNK8JXUe0bQFssUStyIT1gAA2AWyJBmyN716kjRXnCa9ET5x5l9Jmuaim7l16NreRXljcQJtPNUlxgA8EsIXdfluFAfMia9DaPUgAW6uEduoPKoBNAFuiAdu7Fy9S/XmfX3M21dOgRW4CbdDa9+mZeR7aVHNVigN8IIDNnvkD8yJr0No9SACbPWM/6a0SwCaALdGATdf34iIP5W6qV98WHF72O8re8wcqq7kqxQE+EMBmz/SAeZE1aO0eJIDNnrGf9FYJYBPApgOsRLZrXe9S2myXejUKvdjEit/+E5W+c1WKA3wggM2e6QHzImvQ2j1IAJs9Yz/prRLAJoAtkSBN17VqewMNXOh7HcqAreTkVZLiDB8IYLNnesC8yBq0dg0KYLNn7Ce9VQLYBLDpICuRbfyUR2ax1/jxVGxiG05eleIQHwhgs2d6kHVo/R4kgM2esZ/0VglgE8CWSJDGus6636W0OS568/S7foBt4q5/JynO8IEANnumB8yLrEFr16AANnvGftJbJYBNABuDrETWa3edp/T5Hrrw7g3/Hjn7Pklxlg8SGVOJ1IV9z6l/H137mqRY74NI46tNpIzCJx4QwHYDUCQyIThdV8YSL80s9H071Om+kPu33xpzMmCTrGcvDwhgs9d82NoaAWz2SybJnuBd3nep7xw3HT7ZaLwOTfZ7EvtTa50IYLN1WnKUcQLYHDXd8d2sALbUSkR2ABb5u8/Tc5luarwgvrXDfIgNzeNQAFt8eUOkE+cBAWyJ82XKa8LG9e6770oRHyQsBkYv99DUdd6E6ZP4lPWZ6BgQwJbyqS1pblAAW9JMVesbKoBNkmEik6H3/LvUb66L9h8/L4BNHgJsGwMC2Fo/94gFPg8IYJNIiNgDAtgEsCUSsG3Y30DPZrqp/rz4NZF+FV2JjScBbBGnCGG02AMC2Cx2cCqpF8CW2ETg9MT6ykoPTc6T16FOjwO7378AtlTKYsl9LwLYknv+WtR6AWwC2BKVXM9feJeenuOiHUcabfsqLFH3KnqSe90IYGvRNCODhfCAALYQzpEufw8IYEvuxGMn4LClsoGenusmT4P41E7zIrY0j0cBbP55QK5azwMC2FrP90k3sgC25pu5JLjYfDJulYfGr5bXoRI/scVPS/pNAFvSpaqUNVgAW8pObeJvTACb/ZNLSyayeMZ6Zq6LtlXJt0Pj8aHItsx6dDJgW3b8U5JivQ8izdYC2CL1lPCRALaWSRCpnoh3HjlPfee4yF0f3J+Z+z8gKc7ygV3j3smADf+YfHPtVSkW+kD++buAK0s8IIAtOMCwa7Kxo11T1nppzDJPyC8bSKJwVpLEfNsxVmGT0wHboYbPSYp1PhDAZglcEaUC2ASwJSKpPjvPTZvKQ78OxSYmScK6JGE33wpgs2d+wbwcbPhcioU+EMBmz9hPeqsEsAlgixewlVc3UtpsF+GfvofSJYnCWUlSAJs90wPmpar+cykW+kAAmz1jP+mtEsAWGmSEAiDS5/PdrMJ6GrU09OtQ+EoShbOSpAA2e6YHzEuF97OkLfd07U6LN1TY2n4BbPaM/aS3SgCbALZ4gWf/+W4q2N0Q8nSNAZuVieKxPs8TNnMUHoev0cc0qVsmWQtgs2d6wLyUez6Lu2SvrzDWG6+z/iMmBdQ7PbfUj1fnGz0916+Pdb2+63RAXW3atKHpS0oD9iXivhKhQwCbPWM/6a0SwCaALR7AdvDEBeoz20Vn3RciAmyxboaBkgM2diQC6ER9y+13UE5xlSqgIRH8qmNndb18czWxjmCJQLeNk4hOs6K98dj7RrJCWx+Dk5xOi6WtA9lwyRCJFDxmW2IZVwCbPdMD5uWA+7O4y5jXlhLA0+INVUb5x5/eRKOn5frpRkzdcntbgwf8AzKmGnwPPdFXrVNdD9rbT/7BTw/brABbbmnAPuZp7VoAmz1jP+mtAmC7cOGCFPFBTDEwb72XhuZ4IpKNJ1EESg7Y1EsOXlAbd/fe6YSib9QAa0gMTAMAwma/bu85g8Z95rrPwNH0s5tvDctnlgt1jfEBnnSeFVveUjbBrmGvzvfrGzEpS/Xp/KHaebtOK7BVdvR9Pz3/8KMfU+8BGUbCRPvO9p1I50MbQA0JF7YUVdT76Qg1brA+zLdd9xbse6n+98133we8RczLfte1uMvwib741HX1Ts9Q4ItpiworVExNyyn2G6/44AUq2OdStPsf6UkoLBOuRnxOyymNmD+cPiv6BbAFDD0hxusBAWwCVuNJqgMXumnltvqIEnM8iSJQctA3Wmz4Tw0c7beJ48QNcsy36dhlyl5fRbtq/9OgcZ+5RjJZtqk6LJ9ZLtQ1bIFNOk9OyWEFkB58PI1+fvOtVHbkPaM/3D3retBmXUXlXkMH6Ehwuh+QLEGbl7fL4Hv4354m2PDCmBk+wGbSYR4rkmsBbPHuzrHLX/zDV5S99fdU/9F/N1OCednnuhZ34fjUdT09ZLyKcaY90W+wiqudtZ8EHe++R3oSCsuEqxG7U3NK/fjv7tKd9FJ65L2g/Wt21hp9o6blGnJohxs70n4BbM3CTgiJ8IAANgFssQK2oyd93w6trWuMGLBFuuGZ+QIlB/BgY8ZGjZMhnIih3eOp51WNk6Vbbmur2ngix0aNfn0zzyqsUDTe7HkzBz/06Hbomzv4mRc84H9u+CTS9enyaMMW2MRjwQ4GWTOWb1H2pw162Rgz0D0HswH0O9p3UmDrN/d0VWPAFtjGgLCQUo8AACAASURBVI3vBeOak97STdWUv8+lEiH6Csu9hh0sF20tgC0RO3RsOr78n+/oRMMXtGTHFSqq+pj+9MXfDEWYl7111+Iuw66fsOm6sAb7DR5n6Ma6HPTKHONa5+X2fQ/3JBS+DlcjPqfklCr+4sO+9d8rPYOyiqpUQfvX93Ql9EHXs8Mn0QOPpd3o7z9S0RcWVqg1x3Kwc9UbtRHbEcpOAWxGuEkjkR4QwCaALVbAtqDES0Oy3RGBNYwRT6IIlBywYe449YnaiPH6kzflJRurFQ3JgzfyoqoLtLjYd5pVcMBrbNi339GeJi8uVvyou3TvrfowHk7DeFPOmOr7PBxv7qNfW6YAEm/w4EdywokC80AectABm2ALbOJ+2M42oR4yYb7SoetEcorEhrX7XASbwM/3U3r0spIFDfaxngnz8xUf9zMdNRIh+NlHel+0bQFsidypo9f1t2/+Tn+8+r9UfrqJFm75kCrPNhFomJfd55riLkNfXahipWOX7sQFsZNbdtzQjesJCwqM60DjAqxh7bAO1COn5gSVgc7Ji0tU//jMdfSzm2+hwspGgx9t0IaMz1S0X7a7i154ebbRv3J7jWpnTFui+NgmyG1+64rBx/RYagFs0cerSETgAQFsAthiBWyDstyUu8kbFWCLZfODTKDk0OOpgcbmik2/94BRxjVk8EFnyPGYizYcUglm3X6PonV68HEF8rbXfKKuUXOygRzkWRZAa+zcPOMaGztA4gsvzVI08CNJcDKAXN8Xx1K7DvcbMmad4GGbUJccvax0PNl/pKETySlSG1gX3x/LQccvbmtrJES0AW65X6+RCMFv1qHzRNp+Pu8ijV/tsWUZubiGZhV/4IjyyuqL9PQ8Dw3JPU852z6iYWW/o11nm+IuiGfEyoLCSqMgdn99d1daueOU0o9+AKNQ4937cE9q17GzoQP68vbUBZWBzsnZJar/0V7p9OiT/ZvxgvbA430VffC4zOvx382wC/YUVDSqE++OXbrRiCk5zXSEsjlcnwC2CMCHsETvAQFsAthiAWxvnr5AabNc9PaZyF6HeusbaVhp7IkiUHLIKTtubLLY9HuljzKusaECmECON1cGNGv3eRQNrydfmrXK6Gc+1JCDPNpIPkgSbX/TibC5c8GpAI+p87Me0DCGfs06mcY2oQYNoBB6MSbkMS7okdjAuvj+eAzoQCLVEysA6DPDJhq2MS8SIfjNOrg/mnpoyQdU/uZ5W5a1ZVX0/n985YhyoPYqDV/aSNOKLtO5S3+m4Zs+pJ1nmuIuQyb44lPXlV/uO90aOGam0o+HmL4vjgs51r0P9SQUXU+oNuJzUnaJ4ofck+mjmsmCxjo3HL5E8wsqqWuPNLWGu/UeaPCDPnLaUnXy3bFzN1q+/ZTRF8qGcH0C2KLHIiIRgQcEsAlgiwWwLd5YTy9kuakxgm/XHqtppBkFXno+772YN8NAyUHfNANt3ABHkGO+rPW+E7bX93oUTd/4mYdryEEe1yw3cVGx2vixyXNZs7tO8ej8ug6MoV+zTqaxbtSglVVfoXYdOlPnbr2U7SzPfKFsYB6+Px4DOnQ/gJ4+egb95Kc3UVHVRcM+0JEIwW/WwbqiqZG0YomtlpDBvpfqf3/+6lvaWv2peh16uO4afXLta/r2u+/VK9EdZ5oo3jL4OmAz60GMAzCB3mfQWAWGzDz6NdYuik4L1UZ8TswuUfyP9xtCd951fzNZ0J4dMcWPvnp3Hb26qFg9RC3bdsroK62+QpkFlWrdDRgz06CHsiFcnwC2VF9drXR/AtgEsMWSIAdnu2lRWejXoQ2NF6h4XwO9tNxDfee4KH3luzFvhsGSA2+cDNj4GjUSB+SYtuA6YMvb61E0PP2bN3XmhRzkcb3yjTMKxMxctdPQxXxc6/w6DclFv2adTGObUDNtQtZ6lVQeeTJdjQt6JDawLr4/1gcbdD+AjoQHupk3GJ11RVMLYGulTZ2ILn38F8rc+CFtqf6UPvz0r/TXr//PMAbzsv10U9zlxeuATdf16qISFbtTcjcr/WsP+E7ccKql881dV07DJucoWqeHehKK3h+qjbjFOOAB0MIpNl+DxjZg7IKqi6SPPWPlThX3K944Q/2GTiQANx7rJ/90Ew14abZxzfRYagFsRrhJI5EeEMAmgC1awFZzrpHS5rioujb469A3TzXSnPVewhcTRi/10KBFbnpuSWPMm2Gg5KBvpNjwf5s+yk8/wBHkmG9Bke+Ebc1ej6I9df3pH5s6ePTNHXKQB33T2x9Tlx5pdNsd7Q1doOsbvs7P45ltHpuZr5IL96Nmm1AzveT4FTUeEgiSE+iR2IAkBP7s0mpDF2RB0/0AGu7lzg6dlV4eFzWSHfjZR3pftG0BbIncqaPT9V9/+ZYar3xJX3z5Df1gEsW8bKttiru8Ms/35ZUOnbsRl3+5rS2NzyqmDceuGPoXFh+nu+571OABb4+0wbRiZ53ieazvEEKssw6u56wtN3To9iI+J2SVqL6yEx+r8TAuy6E9N7/S6H9hbKbRd1vb9jR86lKCXO6WGvWKlOWwxvP2NwYcUx8/krYANlPQyWViPCCATQBbtIBt5fZ6wu+vnW8M7LvtBxvo1TUeyt9dT7uPnFevTldtr6dnc87HvBkGSg7YaIdNylE6FWDrP8pPPwDXoPELDdr8Qh9gW73Ho2jYnP/tuZH0q7u7qg0d9ZhZeaoPcpDnzRnJBRs6b+6oh07KNRKTmR9yoCG5sI6Cg5fUlwpYB+hsE2rmQz23oFLJ4hSQ6eFsQBKCjfisHcZYsvWUkoUOPaGhD3xIWKz70V4DlQwSGmxmnwRLmiwXqhbAlpg9OhYtP/xA9PfvzVDNpwnzsqW2Ke6Sf/ASzcmvbFZKT3zcTHfOlho/PlyzDct31vn1sU7oZx69Rr/eh/FYhmud32zn+mNXDL3Mjxp26HLxtAWwxRK1IhPWAwLYAoOOaEGMk/jxnw0yi4O/Dt1z9DyhnHU30opt9TS/2Et7jp2nZxY1xLwhmjdd3mh5k0UC4DZvtHiyX7O/0RgTGzXk9ISCftaFmjdz0CHPulCbE4ueNALxs25dB3TyeKCzTTwu83ISisYGs42sUx+Txw7kK+7Ta/0e2bZIawFsYbffVmHAvGw+1STFQh8IYGuV0E79QQWwCWCLBmyec/tehx4/Ffx1qMt7gRrOX6Dac400LMdDx2vO0/GaRuo73ytJwsIkYbckLIDNnvkD87Kp5qoUC30ggM2esZ/0VglgE8AWDWBb+0Y9pc8P/jpU14XXoPgcG16d1py7QGmZHkkSFiYJuyVhAWz2TA+Yl7Kaq1Is9IEANnvGftJbJYBNAJsOssK1M5Z4aGZ+8NehLH+6rpEGZ3vo8Mnz6qcd6s9foEH570uSsDBJ2C0JC2CzZ3rAvJS+c1WKhT4QwGbP2E96qwSwCWBjkBWuxutQ/DzHwRMNYX9fK393A80q9KpXo6wXP9gpicI5iVIAmz3TA+al5ORVKRb6QACbPWM/6a0CYGtsbJQiPggbA0V7vfRcppvqG0LHC4Dd8FwPHXr7vJ9OSRTOSpKYb7vuLdj3nPqHedlw8qoUC30ggM2pq8vi+xbAFhp82DXhtIZdL61w05S13rBJGD/nMeV1TzNgJ4nCWUlSAJvFm3eM6jEvE3f9uxQLfSCALcbgFLHQHhDAJoAtEvDnrm+kfnNdtPdYQ0jA5vKeJ/zsx/4AfJIonJUkBbCF3ntbq/eja1+TFOt9EOn8tomUUfjEAwLYBLBFAtiK93vp2Uw3eepD+6tkfz1Nft0TkO/Q6YskxVk+iCS2WoPHya9EJevZywMC2Ow1H7a2RgBbaADSGsnEjmOOXemmCas9YU/XRi4JfLpmx3sSm5wb+wLYbJ2WHGWcADZHTXd8NyuAzblJK1LAUn++kZ6e46Jth/y/RGCWx+nahNXegKdrZl65lrhrzRgQwBZf3hDpxHlAAFvifJnymgSwSeIMlzg3V9TT03Pd5PIG9xVAHX6j7Y1DoT/jFm4s6Q/uY/FN4nwjgC3lU1vS3KAAtqSZqtY3VABb4pJAqibUCWs8NHZV6Neh26oa1OmaOwSoS1X/yH0l3xoSwNb6uUcs8HlAAJtEQsQeEMCWfMmmpQHCM/NctLG8Pujn13C69tJyDwG0tbRtMp7EbywxIIAt4hQhjBZ7QACbxQ5OJfUC2CThhUp4eMWJ/26AH8MNxgegBsDm8gTnCSYrdPFZa8SAALZUymLJfS8C2JJ7/lrUegFskjBDJUz8RMeYZe6gYO38+UYat8pDZSFO4ELplz6Jv9aIAQFsLZpmZLAQHhDAFsI50uXvAQFskjBDJcxnM11UvC/469A9R8/Tyys8VCena0FBbSj/Sl/rrD8BbP55QK5azwMC2FrP90k3sgC21kkYyZCoDxw/T31mu+hskNehOF17dY2H1u8NDuiS4T7FRuetAQFsSZeqUtZgAWwpO7WJvzEBbM5LVpEClJmFXspYGvzbofuOnaeRS9x01hX699kiHU/4JBZbKgYEsCU+l4jG2DwggC02vzlSSgCbJMlgSbL/fDfl7wp+ejZ1rYfyd4f/Z/DB9AtdYq+1YkAAmyPTnS1vWgCbLafFnkZh4zp//rwU8YFfDBw+4XsderoucGxUvNlAGUvddCZIf6CYmrfvA5LiLB8EigM70JwM2JYd/5SkWO+DSDO+ALZIPSV8JIAtMCCxQ1JpTRvmrffQ0MVuPxCn2zNtnYdW7/AG7dd5uT1i84e0ufaqFIf4APPNc2+32smATdah9XsQfBzpnwC2SD0lfALY5GQtYFIduNBFy7cGBmQH32qgYTluqjkbHdjFJnao4XMpDvGBADZ7JhhZh9bvQQLY7Bn7SW+VnLBFBzrsdlJghT2HTzRQ2mwX1ZxtCAjm5hZ5adX2wGAulD3YxA42fC7FIT4QwGbP9CDr0Po9SACbPWM/6a0SwCaAzQyyskq8NCgr8OvQIydjO13DGNjEquo/l+IQHwhgs2d6kHVo/R4kgM2esZ/0VglgE8BmBmwvLnLT4o2BT9DmrfdSziZPwJM3sx7zNTaxCu9nlpXH+jxP93TtrgqPw9foY5rU1s2B7lsBbPZMD1avQz0GrGhjTS/eUGHr9SyAzZ6xn/RWCWATwKYDq+OnzlPaLBedqG3+OvTNUw00ONtNJ04379N1BGtjEyv3fBZTyV5fYYAxBmGop+eWKn2ob7n9DsoprlIF44yenku/6thZXS/fXE2s4/Vdp8PaAFnoj9XeSOU2HnvfuC+0dTncUyJs0IEs+87sAx6L+3H/ui2xtAWw2TM9xLMO9Tjg9cQxg7r/iEkB48YcXzofrzVdD9rmGOWx27RpQ9OX+NY90+xWC2CzZ+wnvVUC2ASw6QArZ5OXXshyU0OAL2Nkl3kpZ6M3YJ+uI1gbm9gB92cxlTGvLSVs1Is3VPmVkoMXlL7uvdMJRdcPsDYgY6pBQ9KAjnV7zxk0nV9v9xk4mn52861h+XSZcG2MD/Ck863Y8payCXYNe3W+X9+ISVmqT+cP1c7bdVoBvLKj7/vp+Ycf/Zh6D8gw/Ib2ne07EfNBrsP9jxj9L81YTj//l1sVGA41Xrg+AWz2TA/xrEN9zgOtyX/86U00elquX/wBnN1ye1sjvrCGsS6Z76En+qoHK/Pa3n7yD356eGwF2HJLA/YxT2vXAtjsGftJb5UANgFsOsAasthNmcXNX3m+daqB8Ko01tM1jIFNbL/rWkxl+EQfeAkmf/8jPempgaP9dOPEDXIss+nYZcpeX0W7av/ToHGfuS7Y56Jlm6rD8pnlQl3DFtik8+SUHFag7MHH0+jnN99KZUfeM/rD3bOuB23WVVTuNXSAjgSn+6H44AVFm5e3S/Fte/vf/e4V1wC7AMDmMaK5FsDWuunhm+++D2hAPOtQn/9A8dk7PUPFDvMtKqwggLhpOcV+sYQYxBoDH9YuCsuEqxHP03JKI+YPp8+KfgFsAUNPiPF6QACbADYGbCfPNFDaHBcdrWn+ynNxmZfmF8d+usaAbZ/rGsVSODmYZUuPvEd3d+mukgJOxNDu8dTzqsbJ0i23tVVtbPBrdtaqNmRYT1ZhhaJBDgU86AM/9DAf6lHTcgPyMv9zwyeRrk+XRxu2wCYeC3YwyJqxfIs60Usb9LIxZqB7DmYD6He076SA2G/u6arGgC2wjQEb3wvGBW1qTqkxFvdx3fOZIXTfIz2D9jNfqFoAW7y7c+zyF//wFWVv/T3Vf/TfzZRgXkLNW6R9geLz6SHj1UMJ63ii32DCw8jO2k+Cjok4iybWAsUurymu9TUOW5iOmtc46Pp6QpvtjrcWwNYs7ISQCA8IYBPAxoBtxTYv4ffX6hv8fVJzpoGGZLvp2DvNgRzLRlJjE9tbdy2mMuz6CZtZfsepTyirqEo91T/wWJpqL9lYrWoAuF7pGapdVHWBFhf7TrMKDniVDQsLK+j2O9rT5MXFigd1l+69VR/Gw2kYj5cx1fd5OIyFMvq1ZQogrXqj1uDHSQISFPNAHnLQAZtgC2ziftjONqEeMmG+Ap66TiSnSGxYu8+lbAI/30/p0ctKFjTcD+uZMD9fATbuZ7pew87+GVMNGb0v0rYAtkTs0LHp+PJ/vqMTDV/Qkh1XqKjqY/rTF38zFMWzDvW5D7QmETf9Bo8z4gZrYtArc4xrXZ7b9z3ck1D4OlyNeJ6SU6r4iw/7Hth4nWNtof3re7oS+qDr2eGTiPcG1d9/pKJj/evrEXby2gtnQ7h+AWxGuEkjkR4QwOYPTiIBHqnKMyzHQ3PWN38dCiCH0zUzkIvWD9jEdp9riqkMfXWhAhkdu3QnLj2eGmjowobfe8Ao4xrj4HMzkOMxF204pHSs2+9RtE4PPq428u01n6hr1Lllx1UbcpBnWWzsY+fmGdeb37qiQOILL80y+H928y20cnuNwdP3xbHUrsP9xrVZJ3SzTahLjl4m6Hiy/0hDJ5JTpDawLr4/loOOX9zW1vAb2khg3G+uJy8uUX5iX5j7I70WwJbInTp6XX/75u/0x6v/S+Wnm2jhlg+p8mwTgRbPOtTnPtCaRKzpcYPrCQsKgsYa9GHtAtjxukY9cmpOUBnoRIxCdnzmOrVmCisbDX60sY6GjM9UtF+2u4teeHm20c9rNGPaEsXH9wQ5rGu+jqcWwBZ9vIpEBB4QwCaADcDrTF0D9Z3jIvwXAx2I4XQN3wzF76/p9GjaDQ3nlfyw0t/RrrNNMRVODgsKK4lLTtlxQ9e9D/ekXumjjGuMA2ACOR6TAc3afR5Fw+vJl2atMvqZDzXkII/2yh2nFIBp+5tO1LFLN6MgyfCYOj/rAQ1j6Nesk2lsE2rQAAqhF2NCHskJ9EhsYF18fzwGdAAEst9QA4A+M2yiYRvzYhzYOGD0DNr2zsfN+pkvkvr5vIs0fpXHlmXk4hqaVfyBI8orqy/S0/M8NCT3POVs+4iGlcW+DvV55/jU4wpx9uu7u6p4BS9iD8BIlzO3sXbbdezsF595e+qCyijAll2i+h/tlU6PPtm/GS9oDzzeV9EHj8u8/sDSzbALNhRUNKr1iTU9YkpOMx1mO6O5FsAWAfgQlug9IIBNABvA1+tv1FN6ppu89f7+WLXNSzMLvc3o4QAb/mn8riMNtHKblya97lH/KH5YyQe080xTTGXIBB94CSZ/70M96cn0UX66ATwgxzJZ630nbK/v9SgaNv5J2SVGP/Ohhhzk0Wa5iYuKaX5BpV9Zs7tO8ej8rMdscyAe1o0acmXVV6hdh87UuVsvZQNsjNQG1sX3x3ZAB8bma9Tpo2fQT356ExVVXTToaAOUdu2RRhsOXzLoulw07aElH9CB4w22LGvLquj9//jKEeVA7VUavrSRphVdpnOX/kzDN30Y99wiDszxDVp+ue90a+CYmWoMnHT1fXFcyPGwdlEijS193QZa99CDvYB1IpaxbhHXiO9uvQcaY4E+ctpS9QDTsXM3Wr79lNEXqT2B+ASwRY9FRCICDwhg8wco4YBIqvaPzPXQjHz/H8s942pQ/wD+4InIT9e2H6wn/OP4iWs8lFXqpenrPOrfXA1Y4KIRmz6kHWeaYiqDrwO2YPK8cev9AFyQY9qC64Atb69H0ZBMnh0xxehnPtSQgzzaK984o04KZq7aGZDXzM96zDbrOpmHbULNtAlZ69WT/yNPpqtxQY/EBtbF98f6kOB0P4A+Mdv32lPn/dXdXVVSW727zrCFdcRSI2nZdb1g30v1vz9/9S1trf5UvQ49XHeNPrn2NX373ffqlWgs82mWMcc392PdADDhus+gsQoMcV+gGmsXJVBfIBriGfGLvsf7DaE777q/mSxo5rWNuH51UbFaW8u2nTJkSquvUGZBpXpQGjBmpkEPNHakNAFsqb66Wun+BLAJYHN5z6vXoVVv+gOz/F31NCvK07VtVfW0sbyeKt5soC2V9QrwTc/30LR1HpUotp9uoljKi9cBWzDZTg/1pN+mj/LTjcQBOZZZUOQ7YVuz16NoT11PJgVVF9U1ajx9gx9ykEd709sfU5ceaXTbHe3VNevrN3QiYeM383O/2eaxmfkqWXA/arYJNdNLjl9R4/3kn25SgA30SGxYcR1YZpdWG7ogiwSn+wE03MudHTorvbjGfeN61a46P1n0xVoEsLXSpk5Elz7+C2Vu/JC2VH9KH376V/rr1/9nGIN5iXVOdTlzfKPv1UUlKsan5G5WY6w94Dtx43XF8nPXldOwyTmKB2sXhfvC1YhnjAM+AC187ICvQWMbMLa+ptE3Y+VOtR6wVvT1iz6stwEvzY7YDsgEKwLYjHCTRiI9AMDW0NAgxcE+KNrrpWcz3eTx3ogDfKYNv8mGV1rRxMfpugY6526goycbaMJqD23Y66U5RR7K3+VVgG1bbRPFUl6Z5/tmY4fO3UgvwyblKH0KsPUf5acbgGvQ+IUGbX6hD7Ct3uNRtLz9jfRvz40knCxBJ+oxs/JUH+Qgz7au2FmnQJQ+9tBJubTh2JWA/JCDDiQX1lFw8JL6kDPrAJ1tQs18qOcWVCpZnAIyPZwNZSd8wBKvfTDGkq2nlCx0/Mttbf38BgCau6VG9c9ZW66SHhIW28Y1+nj8aGskrWhipyV5U/2E7b/+8i01XvmSvvjyG/rBlDAwL9HOZSD+QGsScTY+q9hYF5BbWHyc7rrvUb/Y6pE2mBDP6H+s7xAFljjmuA4We1hTE7JKlCxiHuPp8Y323PxKo/+FsZnG2Le1bU/Dpy4lyCH+ea1gTKwJ7AmB7jVamgA2U9DJZWI8IIAtOkDSkkmlpcYas9xNU9Z6/JJr4R4vTX7dQ25P9P45erLeAGtnXb5/Fn/k7QYF2LbUNlEsJf/gJZqTX9msLN9Zp/TlbKkhbrN+JIo1+xuN8dYfu6LkS098bNDQr+sFD+RBhzzrQg39Oi9s4v5A/KybeVBDJ+vANdvE4zIvbARfNDZAVreRdepj8ti6r4L5Frz6PbJtkdYC2BKzR8ei5YcfiP7+vRmq+TRhXiKdw1B8weJGX18sj/XJsYca19ynx6zOEyz2zHHJa0WXZd2ozXbyukCfLqOvCV0+lrYAtliiVmTCekAAW/SApKWAVEuMU+dpoH5zXbTnqNcAbOfc9TRyiYf2Has3aJHa8maNP1h7+3Q9vZDlUsAPm9jmU01SHOIDAWxht99WYZB1aP0eJICtVUI79QcVwOZswLbxgJeenusil3aSVrwvttM1gLNJa9y0dqeXcLIGkHfwrXpaUOw7vcMmtqnmqhSH+EAAmz3zh6xD6/cgAWz2jP2kt0oAm7MB27iVbvU7WXyChtO1EblueuNQdKdrAGuT89yU94aX8Dk2Q5+rgU7U+nRhEyuruSrFIT4QwGbP9CDr0Po9SACbPWM/6a0SwHYDXDDIcErtqW+gp+e4aGvVDXC2qdyrAFydO3K/6GCt9lxwOWxipe9cleIQHwhgs2d6kHVo/R4kgM2esZ/0VglgCw4wUh24AZzhdSi+1Yl7BYAbs8xDOzQAF84Hp+vqacpa38laKLAGPdjESk5eleIQHwhgs2d6kHVo/R4kgM2esZ/0Vglgcy5gm7jGTWOWuY3Xl1sr62nsSg9FeroGsDazwENLNnkoHFhjwLbh5FWS4gwfCGCzZ3rAvMgatHYNCmCzZ+wnvVUC2JwL2J6d56LS/b7XoThde2mZR/3obbhTNfTfAGteeudMZD7EJjZx179LcYgPBLDZMz3IOrR+DxLAZs/YT3qrBLBFBjYiATHJxLPjYL367wZnr39BYNfhenXaxt/uDHUvsYA16DtYe0GKw3wQKo5asw/7nlP/Prr2NUmx3geRxlebSBmFTzwggM2ZgG3K6x7jdWh9fQONX+2m0gM3fostWDLFD+nOXY/XoJGfrAXTJXRnxp4d5t3JgE2ynr08IIDNXvNha2sEsDkzaT6b6aL8nT6AhtO1jKVu47fTgiVUgLWFJR4F2CJ9DRpMl9CdGXd2mXcBbLZOS44yTgCbo6Y7vpsVwOa8xLnvaD31me2i2rO+e8fpWv7u0KdrDNbmbfDQm6du/AyIXRKw2OG8OI5nzgWwxZc3RDpxHhDAljhfprwmAWzOS3T4ZmfGEt9/HyivrqcRS9x0OsTvpwlYc16MxAOGkkFWAFvKp7akuUEBbEkzVa1vKDau+vp6KQ7yQf/5bsrb4VFzPm2dh9bt8rUDxYG3vp6WbvbS7CIP4f+EBuIRmvgl2WJAAFvr5x6xwOcBAWwSCRF7QACbs5Ltobfqqc8sF9WcqacDx700dLGLas4G9oHXW08rtnjotXVuOvZOYJ5kS9Rir8wjYkAAW8QpQhgt9oAANosdnErqBbA5K4HNK/LQkMVudVL2Wr6bVm4LfLqmg7Ujb3vlZM1BJ7BOALUC2FIpiyX3vQhgS+75a1HrBbA5C7ANXOiiZVs9dOiEl4bluOjk6cD3zydrAtYCn0n+hgAAIABJREFU+8cJoCaV71EAW4umGRkshAcEsIVwjnT5e0AAm3MSMl5rps1yK5A2t8hDy7cGPl0r3O2lyXluErDmnNhIZXAW6N4EsPnnAblqPQ8IYGs93yfdyALYnJOUs8s8NCjLpU7XBi9y04na5vcOsDYpz02Vb8pr0ECJXmjNYyYZfSKALelSVcoaLIAtZac28TcmgC01ElAkSXPQIhcBtOHHbxeXeQjfANXlGKyVV3sJn2HT+6Qt/kilGBDAlvhcIhpj80DMgG3Z8U9JivU+iG1arZFqLcA2d+9lkmK9DzjJ+l6Humj3EXwz1E1vnfIHIBv2+k7WrARrMt/Wz7fdfMzxZ7daAJs1+US0Ru+BmAEb/sP85tqrUiz0AXxsp7/WAmwSa9avM/iYE+WyLV56fqGLsks9lFXqf7q2tdJLr6xwk5VgDXbInFs/53bav/X44zi0S+1kwCaHMtYfysDHkf7FBdgONXxOUqzzATYxO/21JmCTOLMuzuBbPWEOyXHTtLVuenGRm46evPH5NIC18avctPOwx/LXoLBH5tzaObeTf/X4swtQYzucDNgwL3YC9qloC3wc6V9cgO1gw+ckxTofRDORkU54PHytCdgkzqyLM/iWE+aJWi+lzXHRrEI3ZW7wkOf659N0sOb2+L8i5cSWyBr2yJxbO+d28i/HXyJjKFG6nA7Y7ATsU9GWaPJ8XICtqv5zkmKdD6KZyHiAWKSyrQnYJM6sizP4lhMmfr4jfYFLfXaNf6pj1+EbJ2stAdaQaGGPzLm1c24n/3L8JQpkJVKP0wGbnYB9KtoSTZ6PC7BVeD8jKdb5IJqJjBR0xcPXmoDN6jgb81ouPdbnecfGMyfMETluGpLtptmFvtO1fce8NHaFm3Yc9FJLgTUGbFbOOeb6nq7dVeFx+NrJccC+aOma4y+RQCtRupwO2OwE7FPRlmjyfFyArdzzGVlZ+o+YRChWjmFn3dFMZDxALFLZ1gRsVs9Tn+dH0y233+HoWDt9rp76znZR/0yX+m01gLVxK920pcJLLrf1r0H1BIvYj3XOs9dXGGCMQRjq6bmlSidqzHVOcZUqGGf09Fz6VcfO6nr55mpiHa/vOh3WDshCf6z2Riq38dj7xn2hrcvhnhJhgw5k2XdmH/BY3J+IPVoAW6S7cMvyYV5aGrwncjzE6OINFba+h2jyfFyA7YD7M7Ky3P9IT0Kxcgw7645mIltiGbcmYLN6nvoM9AE2q8exq37EWt4OL/Wb46YZ+R71kx4M1upaGKzxCVusvhrz2lJq06YNLd5Q5VdKDl5Qe0n33umEousHWBuQMdWgAZRAx7q95wyazq+3ETs/u/nWsHy6TLg2xgd40vlWbHlL2QS7hr06369vxKQs1afzh2rn7TqtAF7Z0ff99PzDj35MvQdkGH5D+872nYj5UHe4/xGjf1pOCf3jT29SYDjUeOH6EH86YLdT2+knbPqDQaxtfgBikI86GNAP9UDAD0e6HrTNDxVsJ9bK9CW+BzWm2a2OJs/HBdj2u66RlYUBm5Vj2Fl3NBOZ7IDN7a0P+s1D+MHqeXrqOmCzehy76oePR+a6qd8cFxXv9xgna1aCNfzgLuY9UHKOZ86HT/SBl2C+xr6C+db7ceIGOaZtOnaZstdX0a7a/zRo3GeuC/a5aNmm6rB8ZrlQ17AFNuk8OSWHFSh78PE0+vnNt1LZkfeM/nD3rOtBm3UVlXsNHaAjwel+KD54QdHm5e1SfPCHfq+4BtgFADaPEc015jtQHNiB5nTAFg5sR9If6CEKQH/0tFy/BwaAuFtub2s8EOChCw9SzPfQE31VvJkfxraf/IOfHrZJAbbc0oB9zNPaNWI/0r+4ANs+1zWystz3SE9C4TGeGz6JpuWU0qhpuXR3l+6qoI1+9DEtq7DCkEGfzg+eNTtr/fpZTq9ZL+R7PPW8oRs8bI/VdTQTGemEx8Nn5Qnbhn1emrveE/B/UsIPifa1HhOIHQZsPA7mHLHG16hB47hADCEWSo+85xcbuOY+9ENG1wEa+vWY4nhFHxfogRyuzXbgGnRdb7zt4Zt+p16HjlziprEr3bRhr4esBGtIxG8c8tJr+W46cPzGT4dwgo5nzhm8mH3Cc4VEgRMx+JDnASdLt9zW1vA3zyHPA3Rhnnh+UIMHdMyHeZ71+NJ5mR8xp+vT5dGGLbCJx4MdDLJmLN+i7E8b9LIRA4HuOZgNoN/RvpMCYr+5p6sag2OQARv7DuOCNtW0Frgf9S/bdaCezwwxbNH7Im1jvnnu7VY7AbB98933AVMD5iUa4B2Ml+NT7++dnqHAF9MWFVao09ppOcV+Y+KhAQ9F4Iv2EAexi/XJY9ixho8j/YsLsO2tu0ZWlvse7kkoPAba2MgGvTKHsoqqaPRry4yNN2NarqL1Ss+g2+9oT8WH31NyGVNzDX7IoB+bFet8dvgkeuCxNCWLfmzkuF67z6V4sHl265Vu9EMemyjLW1lHM5GRTng8fNi4vF78K6LElzdrvLRmu4cm5rlp1XYP1Zy9MQb8kEg/I/lgnicvLlbz2j9jqrrGiQaPg/awiVnGNeig9R4wWtEWF/tOO359T1cjNnDSgOsu3XsbNCRdxCDrxQaC+Bs773XFw/GKmGJ7oOfhf3taySAW8QqK5VEjHkHXafG2B6y+QE/NctGgRW4q2uOhM+du+N+K+YbOd057qWCXR/3z+NxNHsJPivBY8cw55g1+Nvtkx6lPlM/hX17zSzZWKxriAXOBPaCo6gLx/BYc8Co9Cwsr1L7Cc4Qa84wxMJ4eO5hvjAFdKNinMOer3qg1+AEan+g32OCBPMcJbIItsIl1wHa2CfWQCfNVctN16vccygbsbbAJ/Hw/pUcvK9tA0+N+wvx8xcf9Zp9iLSHGX1u2Rcmb+yO9xnzz3NutTnXAdvEPX1H21t9T/Uf/3Sw9YF4iBd2h+Biw6TxPDxmv1g3TsB5weryz9pOgY5oPcVg2WI14Nj9s8EMQ1/pDGfQwHTU/lIGuPwChHWzMaOnwcaR/cQG23eeayMrCgI3HwDU22sLKRjXu5reu0M9uvoWe7D+SSo5eVrTcsuNqg8GmB7nX97oMflzPXr1T9W84dFH1Y+N8adYq1UZ/3xfH0oOP91XXK3ecUpsRdLANGBtBsKCg3KBxX6LraCYy0gmPh89KwIZNuuaMl/Yd89CSTW6auMZDGw94yOXxqp94SKRvO9z/sIqZ7TWfqDnEnCKucBTP46A99NWFxjXooPUeMErRFm04pOJgcnaxwTNyaq6iIQZZD/RyPIGG2BkwegYhdnHN8Tpq+lJie6AHSRD905duVm2OV9SIWdB5jETUaZke6jPbB9ZOtwBY46Rce9ZL5dUeWrXNQ6+u9tD6PR46WxffnGPe4OeOXbobpcdTAw1/YR/heWTfmeeb53fdfo+S6/Tg4ypGeI5Q8zxjPD12ALTGzs0zxsNcA8C98NIsRQM/9q2V22sMHuw77Trcb1ybdcJOtgk19jve+9DH98z3E84G1sX3x3Lw2y9ua2v4DW3EMPejxt7HvkW/Hs86XzRtAWzx7MzxyX75P9/RiYYvaMmOK1RU9TH96Yu/GQoxL5GC7lB8gR6iEKP9Bo8z9GNfw2FMKD2MCULx6H2I5yk5pUonDnEAwvjBDA9DaOMhmw94zAc4vfqPVLJ4YNMfoGAnPyzp48XSjibPxwXYdp1tIivLvQ/3JBQeA+1e6aOMa9CxYWCzYp61+zxqs8YTH9OeGTaROnbppkrb3/heBYAP/e06dqbnRkwxeO+672F6ot8QdQ0g59v4fbKsA7TJ2SWGDI+T6Pr5vIs0q/gD25SRi2vUZ5vwYXSrCl7HvbjIRWmzXarGj7fidV0ifQswNH5+vp9OxBViiccxxxXooHH8ccLjOEI/YgKxwTpQg/9f77zLoJljh+MV+liO9aw/eJE2vel7KBmfuU71o0aiBp35E1H3nuVSJ2wZS6yb22AxgznH/yzFnA/KctP0dW4avjH2OWfwsqCwkrjklB03/BXJPmKeX8QM9oNAvsZ4HDt4yMMcY5/h/QI1khHHjs7P+kDDGPo162Qa28SxAlAIvRiT7xm8kdjAuvT4hSxsxwMw+w01EhX2ULaj+Mglv3680dD7mS+aWgCbgZFapfG3b/5Of7z6v1R+uokWbvmQKs82EWiYl2iAdzBejk8G+qgRa/zQAzlcT1hQEHI8ADbEvK5n5NScoDLQOXlxiernvZMPfDAm2thPh4zPVDy/bHcXvfDybEMfP1RlTFui+Pj+IMcP3UyLtW4xwLbzTBNZWe59qCeh8BhoP5k+yrgGHZvakAkLDdrre32AbVJ2iaI9M3SiSpjzCyoJZeQ03zfIwAd56MSG1LFzN1Xuuu9RWrO7TvVBLyaTZfV6w+FLxphsX6LroSUf0Pv/8ZVtytqyKtp71GNp2XPEQ4tKPfTsPDdNXesm/LQEAjqRvsUi5vhgvYgrxBJfm+MKdNA4/rLW+07YOI7QD53QzTpQm/Wax+Z4hT6WYz2su8+gsYQHCfQ/+mQ64Zp5E1X3meOhtNluyt9l7fwGi5+lmz2UPt9FE1a7aftBD43YFPucY92a50H3UyT7iHl+zfOm68N4HDssN3FRcbN9Q99XmJ/1mG3WdTIP6+ZYKau+Qu06dKbO3XqpPZDvmflC2cA8HGM8BnRgbL5GnT56Bv3kpzdRUdVFPzrzYE9F//LtpwL2M1+oGg+nwQB9a9PxoGqnB2crbXll9UV6ep6HhuSep5xtH9GwstgfnHTAzoBNfxDAg8Gv7+6qHjDAi9gDMNLlzG08bOGQRdeTt6cuqAx08uHKo73S6dEn+zfjBe2Bx/sq+uBxmWot4yELDz48fkFFo3qgAn3ElByDzv3x1C0G2HacaSIrCzZWFB6DN1q+Ro2Nb/CEhQZP3nXANjG7RNF+8k830aBxmUb/guuJFnyFVRcJ/a8uKqbMgkpVlm6tMXjHZuarSQKfPmZLtaOZyJZ4LLP6lWjVmx6aU+SmGQVu2njAS8ffwY+1+gBbIn0OED5i6hK/OWVgxeOY4wp00MCHth5HLIOYwwbB16jNetHPsYl+jlfoYznWgz7QZqzaqeJwXn65ildcM2+i6vTlFyhtlovGrXLTiVM3PkvGry6tqo+d9FBWiUeBc7wOxf8urXPHN+fYD8zzoPspkn3EPL+ImWdHTAnod4yH2MAYK984o8aeGWKOdH62y2xzIB62SY+VCVnrVWw88mS6cc+R2MC6OMbYDvgNY/M1anM86n1osy7dLjNPuGs8nAYD861Nx4OqnR6crbTlQO1VGr60kaYVXaZzl/5Mw+N4cNIBuvmBBH355b7TrYFjZiqgjzXW98VxIUE/YwJdd6g24pkfznndm/mxR6MPdBzE4GCma480dUrerfdAwx4+8OEDnngeUHQbosnzcb0S3X66iawsnR7qSSg8Btq/TR9lXIOOjfLFCQsN2prrgO3VRSWKhiB4ZsQUo59fiYJv09sfq03uV3d3pQ6duxll2OQcxV906JI6Yes98GVDHmOCl22yso5mIpMdsJXu99CkPDfl7fDQoRNeOue6ARrgh0T6+eEn+6u4YZ2IFSxCxBLTzDz9hk5UiZHjb0GR74QNccQy0IMNgq9Rg1/Xi36OTfRzvEIfy7Ee1l1y3Pda9LY72qt4xDXzJqoeWvyBeiX58nK3+hJAS4C23YfxuTW3+vxa5Zte0j87F8+cYz8wz4Pup0j2EfP8PjVorIqRgqqLyveosZlDL8bjOcae0qVHGmGu9DERP8u2nWrGzzxmm/lhkftRs016rCAWMB4ePPmeI7FhxXVgmV1a7WcndOj7KcbFvdzZobPaLxGbvD+ybfc/2kv1xxOXmG+rHgri1ZvqXzpA7vjzV9/S1upP1evQw3XX6JNrX9O3332v3m6EA9uR9JsfSFgG64YfgvHmAPsw9wWqGbAF6gtEQzzzA/Lj/YbQnXfd30w/aOaHsdW769RBDj6mgHXLukurr6iDHZxsDxgz06Bzfyx1NHk+LsC2rbaJrCwM2HgMtdH2H+U3JiZ80PiFBm31Ht8r0QlZJYo2ZckWuunmWw0w9m/PjVQbG/igFxvRcxkzaG5+pSrjs4oVP8svLD6uXqnqgA48bJOVdTQTmeyADf+3cv8xj/rigXmDhR8S6ecVO+vUvPOc9hk0TsUAYonHAQ/APvMMnZSrZH57Pf7mF/oAG8cR5BAz2CBYB2rw63rRz7GFfo5X6GM51qPrfuqFsUo3auZLZA0fj1nmpucXuGj5Ft83N60GbW/VeGnPES+dPH0DnPPcxzPnr8zzfbOR547rYZNylO8i2UfM85u3v5Gwd/DDHeoxs/KUPuw/+hwjdgCieFzUiJ8Nx64E5Mc8QoceOwUHfQ+LrAM8bJMeK6DPLahUsohXjolwNpSd8AFLPMBijCVbTylZ6PiX29r62Y57yd1So/rhByQ4tgs1PkbC/Tx+tDXmm+febnWqA7ZLH/+FMjd+SFuqP6UPP/0r/fXr/zPSCeaFgXk8tfmBBLoA/gGIpuRuVmOsPeA7ceMHIR5v7rpy4yGBMQH3hauxpvgBGW/RMB5f6zZgbP0hDH0zVvq+oIiHG/2BC314QBrw0uyE+AY+jvQvLsC2pbaJrCw5W2oIhcdAe/nOOuMadACqNfsbDVrpiY9pTn4l5R+8pGh8DRoKeFGDPnut7xUT80If6DfdfIvaQHlcjMHyLMt9VtbRTGSkEx4Pn5WvRPHq0xPk50Lgh0T7GbHEc4qYQgxgnvVx9HlHvx5/649dMeKIZcADnXyNGrp1veg3xxto0MdyrAexyLSXr4OQBcXHDBr3JaKGj3G6+fxCF63e7qbV232g7WRtczCVqITqwZx7AuuPZ87Zfzy/XPPeoc8j+w5zpO8jgeaX9w7Wx3MGuj7H0ImxmA+1PueB+Fk324Najz9cs008LvPyHheNDZDVbWSd+phsP/uNxzPzwJ/cF2uN+U5UXCVaT6oDtv/6y7fUeOVL+uLLb+gHU4LAvEQLvgPxB3qIwoMBDj/4QQZyiC08AOgPBD3SBhMeQND/WN8hCizp/WjPWVse0E79ARkPKRhPfyBBG4c10I3+F8ZmGmPf1rY9DZ+6VNHxQMIPNxgPDzF4eAl0r9HS4ONI/+ICbJtPNVEyl/HXT0Rytpwy7gMg7v/70Y9p2oqdBq217jGaiYx0wuPhsxKwhdpk4YfWmgO7jNs9bTDd0aEzlbz1sSW+gI/xebL+8100PMdF+F28lVs96hubp856WjyZypwn994a7brBfIfaA1qzL9UB2w8/EP39ezNU82UKzEusIFyXC/YQpT+UMr/+MI2HBv2BQH/I4AcK8wMR60Ft7uOHG11W5zfbyQ8yrIvlzA8xuo5o2/BxpH9xAbZNNVcpmcvrlZfoiWdH0v/7dSe66/5uqgBVD52ylAqP/L7V7y2aiYx0wuPha03AlsxxFq/teeW+L8cMnpRjWUwi1lxuL41Z7qYRuS4q3utRP2ybu9FDM/Ld1NKgDfbE6zeRT579GfPdmqAs1NipDthC5QTMS7TgW/ije9iCjyP9iwuwldVcpWQvK/eep5lrK/xK/pHf2+K+opnISCc8Hr7WBGzJHmfx2L/+zY9VfOZVvm9ZXHLCxKtQfBFg9FK3+tHit2u91BqgDfbE4zORTa69meMvFHBqrT6nAzZ58LH2wSeaPB8XYCt95ypJsc4H0UxkPEAsUtnWBGwSZ9bFGXzLCfPoSQ8NyvL9vMfmCt+rUB204b8QtETihD0y59bOuZ38y/HXErEV7RhOB2zy8GPtw080eT4uwFZy8ipJsc4H0UxkpKArHr7WBGwSZ9bFGXzLCROvRXG6tnyLW31rFP8aDAkOoG1+sYcWFHuoTvvJlWiTX6T8sEfm3No5t5N/Of4ijY+W5HM6YLMTsE9FW6LJ83EBtg0nr5IU63wQzUTGA8QilW1NwCZxZl2cwbd6wly2Gf9twgfYtlbe+MIBvogwb72bFrYAaIM9MufWzrmd/KvHX0uCsUjGcjpgsxOwT0VbosnzcQG2ibv+naRY54NoJjJS0BUPX2sCNokz6+IMvtUT5sG3vDQsx00b9nnUvwvSf3qj+p2WAW2wR+bc2jm3k3/1+IsERLUkj9MBm52AfSraEk2ejxmwfXTta5JivQ/iAViJlm0twFZZ00hSrPcBJ0H8a6hRS92E/0SAH9PdeejGKRt4GLThX0rpYI7lE1HLfFs/33bzcSLixgodTgdsdgL2qWhLiwC2RIMB0Wd/D7QWYLNiExadob88gH/GvmST75/Bj1/lbgbMANqmr8Nn3awDbTJHoedI/NMy/nEyYJNDGesPZeDjSP9iPmGLdADhSx0PCGBrmQRhh0RcUe2lkbkuOnXWq36XDf+A22zX4be99JqAtmZ+MftJrpN73TgZsKVO9kqNOxHAlhrz2CJ3IYAtuRNPNMAB3wTFfzw4fMJD63Z6aMrr7oDA5PAJAW3R+FV4k28NCWBrkfQig0TgAQFsEThJWHwewMbl8eAVmBQn+CCrxE2rtrmp5oxHnbLtPxZ43g+95aHpa13qf5E6wS9yj4HjIFX9IoBNMqBdPCCAzS4zkQR2CGBzVqI6cMxDo5a4FEBfsdVNU9b62oESc9Wbvv+QkL8Ln3dzlp/kflN7vgWwJUFycoiJAtgcMtGJuE0BbKmdmMzA41ydR70WPfK2h06c8tDQxS6qrA4MyNweDwHgTVqDLyoE5jHrl2tnxVOyzrcAtkRkD9GRCA8IYEuEFx2iQwCb8xLs/GI3rX3DB8Dw3w/mFAYHY263gLZkBSVid/C1LYDNIQkuCW5TAFsSTJJdTBTAFnxTT9WEt+uQh15a7nsVilO2wYvcVPVmeNA2YbWLyg4E50tVf8l9pd4aEcBmlwwkdghgkxiI2AMC2FIvGYUDGGfOeWhwtouOn/Tde3apW/17qlByOGnbddhNY1e6aHO5gLZQvpI++68pAWwRpwhhtNgDAtgsdnAqqRfAZv/kYgUAyNzgpnU7fcCr+h0PDcl2GwAu2Hh1Lg+9cdBN41YJaAvmI6Enx3oSwJZKWSy570UAW3LPX4taL4AtORJMooEAXou+ssL3WhRfLsApG/45fLhxBLQ5M17CxUWy9Qtga9E0I4OF8IAAthDOkS5/Dwhgc2YCxmvRQVkuevuU7/6PnfRd82vSUAmYQRtej+K/JYTi1fvm7r1EUpzlA33+7dR2MmBbdvxTkmK9D/wzbfArAWzBfSM9Jg8IYIsccNgp4STCllkFbira7TtVw2fU5q130+IITtkwNkDb5nIPjVvpjhi04R8ib669KsUhPsB8JyJOrdDhZMAm69D6PUj++bsJaMhlYjwggM25gG1rhZvwzU9OiPiXVfgsG5+6MT1YfbbOQ5uiAG3YxA41fC7FIT4QwJaYPTrRWmQdWr8HCWBLdNSKPuUBAWzOBWy1Zz304qIbr0Vdbg/NXe8m/DZbMJBmpuugbd+x0HLYxA42fC7FIT4QwGbPJCPr0Po9SACbPWM/6a0SwOZcwAbwNSPfTRv23gBa+K8HOogzA7RA1wBtGw94aOxKNx1664YuMy82sar6z6U4xAcC2OyZHmQdWr8HCWCzZ+wnvVUC2JwN2DYecNPU12+ALHw27bV8N63efoNmBl6BrgHa8Hm4Cav8Qduxtz20rdKnC5tYhfczy8pjfZ6ne7p2V4XH4Wv0MU1q6+ZA960ANnumB6vXoR4DVrSxphdvqLD1ehbAZs/YT3qrBLA5G7DVnMG3Q930zukbfig/7lb/Y7TmTHSgDa9YzaDt4JseGr/K9zk5bGLlns9iKtnrKwwwxiAM9fTcUqUP9S2330E5xVWqYJzR03PpVx07q+vlm6uJdby+63RYGyAL/bHaG6ncxmPvG/eFti6He0qEDTqQZd+F8kEkPLqdwdoC2OyZHuJZh/pc83rieEHdf8Qkvxhmfo5l5tX5eK1xH9fBYrRNmzY0fYlv3bN+u9UC2OwZ+0lvlQC2G0Al0MmRE2hT17qpbP8NcIZTtunr3PT69f83Go0PdNCGnwjBz4ekz3cRwB82sQPuz2IqY15bStioF2+o8islBy8ofd17pxOKrh9gbUDGVIOGpAEd6/aeM2g6v97uM3A0/ezmW8Py6TLh2hgf4EnnW7HlLWUT7Br26ny/vhGTslSfzh+qnbfrtAJ4ZUff99PzDz/6MfUekGH4De0723ciMx90j57mA7mwJ7f0iJ+eUGMH6hPAZs/0EM861Oc50Jr8x5/epGJI5wM4u+X2tkb8YQ1jXSLWwPfQE33Vg5V5bW8/+YeA8YfYxFrSx7BbWwCbPWM/6a0SwCaADWBt2tobgA0Abe8RfGPUB7SiAWzgBWjDP5eftMZNb9X4fvrjjYMeBdj2u65RLGX4RB94CSZ7/yM96amBo/1048QNciyz6dhlyl5fRbtq/9OgcZ+5LtjnomWbqsPymeVCXcMW2KTz5JQcVqDswcfT6Oc330plR94z+sPds64HbdZVVO41dICOBKf7ofjgBUWbl7fLj2/Nzlplw/y1e1Q/9JnHiOZaAFvrpodvvvs+oAGYl2jmMRhvoPjsnZ6hwBfLLCqsIIC4aTnFfmMiBrHGwIe1i8Iy4WrE87Sc0oj5w+mzol8AW8DQE2K8HhDAJoANr0WfX+ii2rM3QNu5Og9NynNT4fXfaYsUtB1526N+FuTUGQ+t3u6hyXlu9c3T3E2+E7Z9rmsUS+HkYJYtPfIe3d2lu0oKOBFDu8dTz6saJ0u33NZWtbHBA5CgHzKsJ6uwQtFARwEP+sAPPcyHetS03IC8zP/c8Emk69Pl0YYtsInHgh0MsmYs36JO9NIGvWyMGeieg9kA+h3tOymg9Zt7uqoxYAtsY8DG94JxQZuaU2qMhb4u3XsTEm5hudcAbCwTSy2ALd7dOXb5i3/4irK3/p7qP/rvZkowL7FtXuSIAAAgAElEQVTMp1kmUHw+PWS8eihh3if6DSY8jOys/STomPc90pNQWCZcHSh2eU1xra9x6GM6al7joOvrCe1wY0faL4CtWdgJIREeEMAmgA1gbOIaF229/uUABme7D3to5BK3eq3JtHB13g43jV/lppkFblq62Xdy90KWi8Ysd6kTtr111yiWMuz6CZtZdsepTyirqEo91T/wWJpqL9lYrWoAuF7pGapdVHWBFhf7TrMKDniVDQsLK+j2O9rT5MXFigc1QAvGwHg4DePxMqb6XhViLJTRry1TAGnVG7UGP04SkKCYB/KQgw7YBFtgE/fDdrYJ9ZAJ8xXw1HUiOUViw9p9LmUT+Pl+So9eVrKg4X5Yz4T5+QqQcT/oAG+wF36CfyADm1gmlloAWyJ26Nh0fPk/39GJhi9oyY4rVFT1Mf3pi78ZijAvscynWSbQmkR89xs8ztCPNTHolTnGtVkHru97uKcqgfoC0RCbU3JKlc7iw74HNl7nWFto//qeroQ+yD87fBLx3qD6+49UdKx/fT3CTl57gcaNhiaAzQg3aSTSAwBsbrdbisN9ULTbpUCWHgtnz+Fbny5avyfy+Dh+0k37jvp+KmTJJpf6Yd4+s1zUb46LhpX9jnafa4qpDH11oQIRHbt0Jy49nhpo6MKm33vAKOMa4+BzM5DjMRdtOKR0rNvvUbRODz6uNvLtNZ+oa9S5ZcdVG3KQZ1ls7GPn5hnXm9+6okDiCy/NMvh/dvMttHJ7jcHT98Wx1K7D/ca1WSd0s02oS45eJuh4sv9IQyeSU6Q2sC6+P5aDjl/c1tbwG9pIYNyPGjS+P8hDBvp0nmjbSFp6PNmpjX0v1f/+9s3f6Y9X/5fKTzfRwi0fUuXZJgIN8xLtXAbiD7QmETe8hiCD6wkLCkKOh7ULYMfrGvXIqTlBZaBz8uIS1T8+c51aM4WVjQY/2lhHQ8ZnKtov291FL7w82+jnNZoxbYni43uDHNY1X8dTC2BL9dXVSvcngC1yMGKnhJNoW06cchNOwvBaVNe9rcJNw3NcBPCm08O1oaf6HTfhd93eOOj7Md6hJR/QrrNNMRVODgsKK4lLTtlxQ9e9D/ekXumjjGuMAxACOR6TAc3afR5Fw+vJl2atMvqZDzXkII/2yh2nVOJp+5tO1LFLN6MgyfCYOj/rAQ1j6Nesk2lsE2rQAJqgF2NCHskJ9EhsYF18fzwGdAAEst9QA4A+M2yi0j1iSg517ZFGm968oq4hDxm2ifVEWz+fd5Hw/2btWEYurqFZxR84oryy+iI9Pc9DQ3LPU862j9SDU7RzGYif41OPK8TZr+/uquIVMogjAKNA8kzD2m3XsbNffObtqQsqA52Ts0tU/6O90unRJ/s34wXtgcf7KvrgcZnXH1i6GXZh7IKKRrU+saaxBtieRNQC2FoJ0KT6sALYogMi4YBKsva73G7jtah+D6fPutX/C9243xUVYNN1uFxuOlnrpuGbPqSdZ5piKkMm+MBLMPl7H+pJT6aP8tMNcAQ5lsla7zthe32vR9Gw8U/KLjH6mQ815CCPNstNXFRM8wsq/cqa3XWKR+dnPWabA/GwbtSQK6u+Qu06dKbO3XopG2BjpDawLr4/tgM6MDZfo04fPYN+8tObqKjqIt151/0KwHXs3I1QfnV3VwOgPjN0op+criNcGwB9z2G3Lcvasip6/z++ckQ5UHuVhi9tpGlFl+ncpT/HtQ71OTfHN/ryy32nWwPHzFRxg5Ouvi+OCxlD/3973x5cVZXl7V8zU+V0lz1lj10434fa3QiKCnTbCqhNKwr4ICDvNyhvAQUEAiQhCYRXgIRAbkIIr0TeCSAk8sqL3JMojYPGFif2DA5fTYJSbY2PGbvUVtdXv32zjvve3Pcj99ycRdWufc7aa629zjpr7/Vb+yQBaxdN1+3vWl+33tY9ZLEXsM7tJ5rUmkVRgqKr78Dh5lxYzxMXrDbjP2tPtTnmz4ZAYwLYOjtyitPzCWATwMbgKqekgZbktfcH/s/RqWsaqN5oP8aywfTYxPbWtYbVRrcBNl/yvHHr4wBckGPaijbAln2gUdGQTJ6dMM8cZz70kIM8rtftq1MAZtH6Eq+8nvysx9NmXSfzsE3omTYrZZuq/B96PEnNC3owNrAufj7WhwSn+wH0OWkOpRu8aQUVlJxbZrbZqflqDIls9a4q0y7WF2yP9x1MXMSDxw6fRD/9/BvaVXlNfQ494bxOH13/ir759nv1STTYd+iPzzO+mRfrBoAJ90+NnK7AEI9567F20byNeaMhnhG/GHt0yBhVcHjyoQjxXNsbXnMSYhun3mt2V5vzFVQ2q9hHoTRs6iKT7qkzlHsBbHECNJ19WgFskYGQeCSbWM2JT5gjU9p//sQp2wtZDbTjcPinbLAZm9ie2taw2qg2wOZLvueDA+gPSZPddCNxQI5lVuS5Ttg2HmhUtCfbkklu+WV1jx7VN/ghB3lcF529Sr37D6Iud3RT96xvyNg5auP35OdxT5unJ+eoZMHj6Nkm9Ex3VDSr+W7+yS0KOIEejA1r24BlWkGlqQuySHC6H0DDs9zZvZfSy/NyD/9ARreJx0LpBbDFL3s0Xf2CkndcoeLKa3Tl2pf05Vd/M42JZB3q798zvjE2O9WhYnxexk4Vg5sPu07ceF2x/NItR2jcS+mKB2sXjccC9YhNzAM+FBoAYHwPGtuAufU1jbGF60pUbGOt6OsXY1hvw6a9GrQdkPHVBLCZ4SYX0fSAADYBbAwA8elyxjrXz5wxjXvHQdcpG3iYFmqPTWx3TWtY7cVlrt9s7N6rL+lt3Nx0pU8BtqGT3XQDcI2cudKkLd/qAmwb9jcqWvahS/T75yaqT4DQiU+BUxdnqzHIQZ7tXVviVCBKn3vs3AzafqrZKz/koAPJhXXkHmtSP+TMOkBnm9AzH/qluWVKFqeATA9kQ+EZF7DEZx/MkbmrWslCx8+7dHXzGwBoRnGVqZvnQA//wG5Pm3SeYK4FsEVzpw5N11+++IYuNX9GNz77mn7wEI1kHerv3duaRJzNTMk31wX4V+ZX0N33P+wWf/0HjSbEM8YfGTxGgSVeF9wv2XzEa3wiNmelONQYYh7z6fGN66U5Zeb4iOnJ5txdunaj8fNXE+QQ/7xWMCfWBPYE/RnDvRbA5hF0chsdDwhgCx+AhApYEoF/894GSt7W3ifVdQ00ZbWhfoEg3OfAJlZc0xpWyznWREtyytq1rBKn0pdeXEV8zXMgUWw8dMmcb9upZiVfcOaqScO4rhc8kAcd8qwLPfTrvLCJx73xs27mQQ+drAP3bBPPy7ywEXyh2ABZ3UbWqc/Jc3v6iudFz3OzvD4WyrUAtujs0eFo+eEHou++94RqLk2RrEP9/ftak/r6Yn6sT4499LjnMT1mdR59fTEvevDoYxyvuqzO72mnHte6jL81oesL5loAWzhRKzIBPSCArT04CReQdAY5X59F8WzbXjPU/wsa7ikbNrGd1a3SbOIDAWwBt9+4MMg6jP0eJIAtLqHd+ScVwCaATQeaAGP4ebXSE+39UlXbQBMywj9lwyZWVNUizSY+EMBmzfwh6zD2e5AANmvGfsJbJYCtPTDRAYwdrzfsbqCUAu+/YLBpn6H+/Ec4fsEmVljVIs0mPhDAZs30IOsw9nuQADZrxn7CWyWATQCbJ/h6/WwDjU41vP4Zj/O1DTQx06ADJ70DOk9d+j02sYLzLdJs4gMBbNZMD7IOY78HCWCzZuwnvFUC2ASw6aAK106jgaZmNdD+172DMvx/ofM2eR/z1KXfYxNznGuRZhMfCGCzZnqQdRj7PUgAmzVjP+GtEsAmgE0HVXy9fncDpfr4LHq2poHGrgr9lA2b2PZzLdJs4gMBbNZMD7IOY78HCWCzZuwnvFUC2ASwMUjT++OnDRqTZpCv3wjNKm6ghTmhnbJhE5tT+qE0m/hAAJs104Osw9jvQQLYrBn7CW+VADYBbDpQ4+t6ZwNNyjToSIV3UIb/LH5cukHlbwTvv6PnL5I0e/mA48lqPfY9u/774PpXJC32Pgg2vm4KllH4xAMC2IIHHFZLOrG2J3OHQRk7vAM2zJ2xo4Fe3ep7PNb2iX6J3XBjwM6ATbKetTwggM1a78PS1ghgk6TnK+mVVbp+Vs3XZ1H8kd3RqQ10/LT40JcPhW7N2BDAZum0ZCvjBLDZ6nVH9rAC2KyZUKyQ6C8E+CxqNLh+MSHFIT60wvsSG4KPQwFskeUNkY6eBwSwRc+XnV6TALbgN3k7JsRUh0Frd/n2UcU5nMI10Kmzvnns6Dd5ZmvHgwC2Tp/aEuYBBbAlzKuKv6HYuAwDvw0oTXzQPgYOvm7Q5Mz2dPaV0zBo+XaDUhy+eZhXevGRVWJAAFv8c49Y4PKAADaJhKA9IIBNkqi/JFp3waDx6QYde8O3n46/YdDIFIMqzvnm8TeHjInfOjoGBLAFnSKEMcYeEMAWYwd3JvUC2CRZBkqWOD1bt8u3n+qdBi3NM2h1kW+eQHPIuPiuI2NAAFtnymKJ/SwC2BL7/XWo9QLYJFEGSpTqs+hq/37CCdzoNINOn/fPF2guGRf/dUQMCGDr0DQjk/nxgAA2P86RIXcPCGCTBBkoQeKzKMDYyTO+fYVTtle2GLRmp2+eQPPIuPiuo2JAAJt7HpC7+HlAAFv8fJ9wMwtgkyQZTJJcts2gzfv8++rwKYNGpRp0vtY/XzDzCY/4MJYxIIAt4VJVpzVYAFunfbXRfzABbJIYg0mMJccMmpbl31cX6l2nbNl7/fMFM5/wiA9jGQMC2KKfS0RjeB4QwBae32wpJYBNEmMwibGmLvBnUejBKRv+03hvp2xL9jeRNHv5IJjYigePnQHbmoprJC32PggWUAhgC9ZTwkcC2ASwBZsw8Vl0S4DPojhlm7vRIG+nbBN2XqGdNS3SbOIDvO9gY6uj+ewM2GQdxn4Pgo+D/SeALVhPCZ8ANvmDwUEn1V1HDZq5PjDALT1m0NhVBuFUTk/E2MSOX/xEmk18IIDNmglG1mHs9yABbNaM/YS3Sk7Y3EGFDjDk2t031bWuXyoI9Kc78Fulc7MNynvNXR6b2LGLn0iziQ8EsFkzPcg6jP0eJIDNmrGf8FYJYHMHFQLS/PtjUY5BW0v988CHe8td/0OCfso2vuh9Kn/rE2k28YEANmumB7wXWYex3YcEsFkz9hPeKgFsgcGHgLgffVR8xKBZG3689+Wb2gsGTV9r0Pb9Ll78EsJzmX+ko29+HLP2yFPP0719+qnG8/A9xpgmfezege5bAWzWTA94L/p7kuvorwcBbNaM/YS3SgBbYPDhC5TYkY7PoiNWGnSmKrDfAO7wH8dfcBpUcdagQcmNdKTx47Ba2rajJhhjEIb+5YwCpQ/9bbffQen55aphnikvZ9Ave/RS91k7K4l1bCqtDWgDZKE/XHuDldtx6l3zuXCty+GZomGDDmTZd54+YLrewwe6PaFeC2CzZnrAewn1XVqJHzGKtWwlmzxtEcBmzdhPeKsEsAUGHnYEZv6eef4mgxwHfPsNP+MGkIbPoTPWGbTjsEGHTho0ZOVbdLjh47Da1FdW00033USrtpe7Ncext5W+fgOTCE3XD7A2bNJ8kwYABB1bDtSbNJ1fv35q+BT62a2/CMinywS6xvwATzrf2uLTyibYNW72crexCXNT1JjO7+86u7RWAbzCk++66fm7v/8HGjhskuk3XN/ZrSfpfJh/xLSFJg/8nHuowU2Pv7m9jQlgs2Z6iBZg4wJIB/lDJ8z1CqS4+GBenY+LIx7j3rOoYFCEWH0501WoMc1qvQA2a8Z+wlslgM038PAHWuw8VnDQIIA2Xz7YvNegRbkGHT9tUOEhg6asNqjosEHPpF2kQ8b1sNr4OS7w4kv+gYcG0JPDp7jpxokb5Fim6NR7lLatnEpr/tOk8Zhnn3vQoDVFlQH5POX83cMW2KTzpDtOKFD220cH0T/d+gsqfP0dczzQM+t6cM268o68aeoAHQlO90P+sbcVbVl2qckHngXpBea9p+5w7gWwWTM94L0cbrgecZvysquIwpri9o8/vYUmL8hw0w1wdtvtXU0e8CZNmm/yPfjYYHUSzjq433P2Qzc9bLOK1YwCr2PME+9eAJs1Yz/hrRLA5ht4+AIkdqefq3F9Fj1b7d13p84atGGPQTPXGbR+t0ETMwyat8mgZ9P/SAeN62E1Bi+e8gWvv0P39O5HSBQ4EcN1/yefVz1Olm7r0lVdA4xsLKlR15BhPSlbjyoa5NDAgzHwQw/zoUciYj6dl/mfGz+XdH26PK5hC2xiHbCDQdbCrGJl/6CRL5hzentmXzaAfke3ngqI/frePmoO2ALbGLDxs2Be0OanF5hzed4zbyS9ALb4poevv/3eqwF4L5G8V5b1Fp8DkyYp8MU8iEGszfnp+W5zbi9/m3IOGop2/0MDCI1lAvWxiNVAc4Y6LoDNa+gJMVIPCGDzDjrsDsoCPT8AmOOgb98BzOE3RRfnGvTcMoOeWWrQ0Ky36YDzelhtXNsJm6f83uqPKCWvXCWJ3zwySF1n7qhUPQDcE0mT1HVe+du0Kt91mpV7+E1lw8qtR+n2O7rRS6vyFQ/63v0GqjHMh9Mwnm/SfNfPw2EutCmvrFEAaf2+GpMfiemxIaPVOHggDznogE2wBTaxDtjONqEfM2u5Sm66TiSnYGzYfNBQNoGfn6fg5HtKFjQ8D+uZtTxHATYeBx088AWDSQBM5g+3F8AW6e4cvvzlP39Oabv+jd764L/bKcF7Cfed6nLe1uTTo2e6rRusB6zLfdUf+Zzz/t8NIDRdt79rxCqKDZ2H45Z7x4l3fI5vaFuzkMf6ZBleq7recK8FsLULOyFEwwMC2HyDjkCgxc7j+Bm2V7b4912906Djbxi0ptigZ5cZNGz9v9Jr9a1htbGzVypQ0aN3P+LW/8nhpi5s+AOHTTbvMQ8+w0CO50zdflzp2HKoUdF6/vZRlUz2VH2k7tFnFFaoa8hBnmUBtKYvzTbvd55uViBxxLTFJv/Pbr2N1u2pMnkGj5pOd3V/wLz31AndbBN6x8n3CDoeHzrR1InkFKwNrIufj+Wg45+7dDX9hmskUR5HvzKvzK3hJFD3r84b7LUAtmjs0OHp+Ox/vqUzF29Q5t5myiu/Sv9146+mIryX/fWtETdek7ourBPEPdNQxIx44VXznul6z4BNp/m7RjzPW+VQOrcfv6wAF9YMxzCuf3VPH8IY9Dw7bq6Kd30c9BW5R1QBxXTYuW5vtV9b/dmljwlgM8NNLqLpAQA2p9MpTXwQUgycrnKq3xatqg0cO2ernbS7zEmjtjZR6YXWsBonhxVby4hbemGFqeu+3w2gJ5Imm/eYB8AEcjwnA5rNBxsVDaBk2uL15jjzoYcc5HGNTRxJouuve1KP3n3NhmTEc+r8rAc0zKHfs06msU3oQQMohF7MCXnMC3owNrAufj6eAzqQxNhv6JFYnxk3x7SNebkfNmWhsh3zMi3UHknLqnsL9r3O/u+vX39H/97yv3SktpVWFl+hsgutBBreS6jv0hs/x6e+JhBr6QWnTP24n7k8x7z3pgdrFzGv65kwL92nDHS+lOZQ4zOTt6giJ/foJZMf1yh8Rs9IVrR/ufNuBRp57rW7qxR90oJMxcd0yBW90WzqYXo4vQC2zr664vR8AtgCAw6rJp142lXvdNLsDU7Cn+4Ixo4L9U4av+N9KqlrDauNmeUCL77k73twAD2eNNlNN8AR5FgmZZvrhG3TgUZFw8Y/N81hjjMfeshBHtcsNyc1n5bnlrm1ja85FY/Oz3o8bfbGw7rRQ66wspnu6t6LevV9QtkAG4O1gXXx87Ed0IG5+R590pSFdPNPb6G88studOZZtuWwAotsF9ND6Z/Pvqz+Fh/+Hp/V2sRVVbQ4/0+2aC9uuExPL2ukMRl/pPTdH9C4wvep5EJrxG1MW0GhFwJ/GDqRfnlPH1q7p1rpR+xNXJDpdy4Atrt69HIrKDbud/qUMdfthVZ6+Ikkevjxoe14QfvNI4MVfdSMZNcJc6++pl14fgA0FFQ9evUlAMRo+IR1CGCLE6Dp7NMKYBPAFgzg8saztdRJC3OCA2yQxya2t641rDa6DbD5kmfApo8DcEGOaSvaAFv2gUZFQxX+7IR55jjzoYcc5HG9bl+dAi+L1pd45fXkZz2eNus6mYdtQs+0WSnbVCJ56PEkNS/owdjAuvj5WB8SnO4H0OekOZRuT16WWbC6WI1jXqaF2o91/IlKjzst2TYXltO7//G5LdrhmhYav/oSLch7j+qbPqXxRVdoX11rxI3jW9eVc8R1ujV86iKlH2ts0KgZfufC2kXT9fi7RjwjfsHD696TH8Ub69x2okkVWX36D1Kn5H0HDjfnQgE2ccFqdeIM4Ja1p9oc89QZyr0Ats6OnOL0fALYBLB5A2PB0N44H/xnUQZse2pbKZw2qg2w+ZLt+eAA+kPSZDfdAFyQY5kVea4Tto0HGhXtyZHT1UadW35Z3aPHZg5+yEEe10Vnr1Lv/oOoyx3d1D3rGzJ2Dq3ZXd2On8c9bZ6enKOAGI+jZ5vQM91R0azmu/kntyjQBHowNqxtA5ZpBZWmLsgiwel+AA3Pcmf3Xkrv7FQHofH88APG8cyYl+mh9khawcRRPHjs8En008+/oV2V19Tn0BPO6/TR9a/om2+/dxVOta20N8LG8e2pB+sGaxF0XmOePPo91i6aTvN3jXhGvILnkSFj6M67H2gnC9ozKMa0Z1xf6qTZqflqDWLd8hjW27KcMrUehk1dZNJ5PJxeAFucAE1nn1YAmwC2cBMmPovOXOekPWXBnbJhE9td0xpWe3GZ6zcbu/fqS3obNzdd6VOAbehkN91IHCNnrjRpy7e6ANuG/Y2Kln3oEv3+OdcnHOjEp5ypi7PVGOQgz/auLXEqAKPPPXZuBm0/1eyVH3LQgeTCOnKPNamfmWEdoLNN6JkP/dLcMiWLEwqmB7Kh8IwLWOJn7TBH5q5qJQsdP+/S1c1vAGMZxVVqfKXjlNs4/AC/YD6eO5xeAFv8skfT1S8oeccVKq68RleufUlffvU30xi1DmtbaXeEjQGbrgdACp8ZX8rYqfRvOuw6cXv4ieFu8y3ZcoTGvZSuaAzYdD3+rrGmZqU6lOyy3DI1H+ZlGbYBc+eUXSZ97lfWlah1lbWvjlBwrcYaafMDCqRh014175keTi+AzQw3uYimBwSwCWALF7BBbtNeg5blBQ/YimtaKZyWc6yJluSUtWtZJU6lL724ivia9a/Mr6CNhy6Z82071azkC85cNWkY1/WCB/KgQ551oYd+nRc28bg3ftbNPOihk3Xgnm3ieZkXNoIvFBsgq9vIOvU5eW7dVzwXj6GH7WxLuL0Atmju1KHp+ssX39Cl5s/oxmdf0w8eongvu2paI27eiigUBjNT8lUhw3Mg/u6+/2G3gqH/oNGqIADPI4PHEMASFzLcL9l8xKuNCrClONQYihTMpxckuF6aU2aOj5iebOru0rUbjZ+/miCHguX/thU3mBNFDIo4tjuSXgCbR9DJbXQ8IIBNAFskgO3UWSeNTDGoti6wH7GJ7axulWYTHwhgi84eHY6WH34g+u57T6jm0oT3UlzdGnHbUt5Er24pa9ccp6+2071qZ5UbH+7Zhqx9Trcx1gn9zKP3GNfHMB/LcK/ze9qZd7LZ1Mv86GGHLhfJtQC2cKJWZAJ6QABbYKARCaDp7LL19U71G4B7g/gsik2sqKpFmk18IIAt4PYbFwa1DqtbqEhazHwggC0uod35JxXAJoAtUlC5aU9wn0WxiRVWtUiziQ8EsFkzf+C97KhqkRZDHwhgs2bsJ7xVAtgEsEUK2E6ecX0Wrav370tsYgXnW6TZxAcC2KyZHvBepHCKbeEogM2asZ/wVglg8w8yIgUzdpDHH8Wdusag10749yU2Mce5Fmk28YEANmumB7wXKZxiWzgKYLNm7Ce8VQLY/IMMOwCuaDzjul0GpTj8+xKb2PZzLdJs4gMBbNZMD3gv+edapMXQBwLYrBn7CW+VADb/ICMaYMYOOo6fdtKoVIP8fRbFJjan9ENpNvGBADZrpge8FymcYls4CmCzZuwnvFUC2ASwRQNQ8mfRA6/79ufhs2+RNHv5IBqxFQsd2Pfs+k8Kp9gXjQLY7Lq6YvzcAth8A4xYJIrOrHN1kZNWFYo/O/M77izPZmfA9sH1r0ha7H0QbOq+KVhG4RMPCGATgBGtJHy0wqCxaQbhb7NFS6foEV/GIgbsDNgk61nLAwLYrPU+LG2NADZJiNFKiHUXnDQp06DDJ8Wn0fKp6IlNLAlgs3RaspVxAths9boje1gBbLFJCHZNtBk7nJQun0XlhNFp7XUlgC2yvCHS0fOAALbo+bLTaxLAZu3EkmjAjz+LJprdYq+91oEAtk6f2hLmAQWwJcyrir+h2Ljq6+uliQ+iEgP4T+DHpRtUViExJevKujEggC3+uUcscHlAAJtEQtAeEMBm3aSSqAl/Zb6TsoqdUQGAieoDsdva60oAW9ApQhhj7AEBbDF2cGdSL4DN2oklERP/gRNOmpghgC0R351dbBbA1pmyWGI/iwC2xH5/HWq9ADYBbNFO0jW19TRePovKCaOFf8zAzoBtTcU1khZ7HwSbyAWwBesp4SMBbALYog3YoA+fRdfvklO2WPhWdEa+Zu0M2PBX+HfWtEiLoQ/kfzoQcBUTDwhgi3zzlwTa3of7yupp6hp3wLZk/7skzV4+sOrasDtgO37xE5IWOx8IYIsJXBGlAtjagw2rJplEsquqpp5GpTrpxBs/+lcqe3udauB9WzVm7Q7Yjl38hKTFzgcC2ARbxcQDAth+BBRWTS6JatfSPCdl7/nxlA2bmFT1savqreZbAWwx2bIjVor3Uv7WJ9Ji6AMBbBGHqSjw5gEBbALYYgUI93p8FsUmJgCt5PYAABDqSURBVFV97Kp6q/lWAJu3HTf+NLyXo29+nLDt3j79aNX2o5a2XwBb/OO8U1oggE0AW6wAG38WrTjrOmXDJiaVvX1ONgSwWTNl4L0cafw44pa27SgBPOlt6IS5XvW+nFHgk2/KyxluY6xvU2mtV1033XQTvZxZ4HUsGs8VDR0C2KwZ+wlvlQA2AWyxAmzQuyjHSbklLsA2rvB9Sittjlll/MhTz5sbP58g8OaPMaZJ3zGnKwLYrJke8F4ON3wccZv6ymoCeFq1vdxs//jTW2jKggw33QBxt93e1eQB/7BJ802+Bx8bTL/s0cttHDx7zv3ZTQ/brABbRoHXMeaJdy+AzZqxn/BWCWATwBZLwFZ0uJ5mrjfUD5+PzG2imVveDasy9lbNA4yhckdFjP622++g9Pxy1UBD5Y5EAFrWzkpiHb4qd72y5qpfp8Xiesepd02QiWt9Dj6V0GnhXOtAlgGsNx/otjBfOPOxjAC2+KaHr7/93qsBeC+HjOsRt/FzUhRg03UNTJqk1hzTUrceJYC4Ben5bvPlH3ubcg8aivbAQwMIjWUC9QBsC9ILguYPpC8W4wLYvIaeECP1gAA2AWzRBmxnq+spvdAg9Oer62lUipMqzzkpad2/0sKC98OqjL1V86jCHcfeVvr6DUwiNL2yBlhDJc80ACBs9lsO1Js0HvPsnxo+hX526y8C8nnK+bvH/ABPOs/a4tPKJtg1bvZyt7EJc10JUef3d51dWqvAX+HJd930/N3f/wMNHDbJPMHA9Z3depLOl5Z3lH59bx+atjDL5IN//c0XaEwAW6S7c/jyl//8OaXt+jd664P/bqcE7+WgcT3ixoBN1/X0mJmqcGLaY0NG028fHUQlNR/5nO/+hwYQGssE6rFW5qcXuPHf07sf6a3g9Xd8jm8sqTHHJi/IMOVwHWjuYMcFsLULOyFEwwMC2ASwRRuw1dbVU6rDSfM3Owk/v7Zwi+uz6NNpFynr0J/Dqow5OfiqhlGhPzl8iptunLhBjmWKTr1HadvKqbTmP00aj3n2qP7XFFUG5POU83cPW2CTzpPuOKEAG5LaP936Cyp8/R1zPNAz63pwzbryjrxp6gAdCU73A043QFuWXWry3ffgYzRi6kLaffZDk+apP9R7AWzR2KHD0/HZ/3xLZy7eoMy9zZRXfpX+68ZfTUV4Lwec1yNu49pO2HRdKHKGjJ5h6sbp2sgXl5j3Oi9f3/+7AYTG94F6xO689ALFn3/iHQW4nkiaRCl55arh+lf39iGMQdez4+fSbx4Z9OP40ImKvnLrUVWUsRzsXL+vJmg7/NkpgM0MN7mIpgcEsAlgizZgg76TZ+ops8hJczY4aTX6jQY9tbSRis79v7CqWAYvnhUuKmlU1kgMSBa47v/k86rHydJtXbqqa3xCQWWNcb36Ttl6VNG4OufqG/zQo8+nV+PgZ17wgP+58XNJ16fL4xq2wCaeC3YwyFqYVazsHzTyBXNOb8/sywbQ7+jWUwExnJRhDtgC2xiw8bNgXtD4lALPAbvW7z1vzs28kfQC2KK5U4eu669ff0f/3vK/dKS2lVYWX6GyC60EGt7La/WtEbexs1eqOOrRux9xQ1xlFFaYunE/a0Wuee9tXoA1rF/WgX7i/HSfMtD50iqHGp+ZvIV+dutttLXsksmPa9DGzExWtP9z19004oVXzfF1e6rU9aQFmYqPbYLcztPNJh/Tw+kFsIUeryIRhAcEsAlgiwVgg843ztVT9p56mphh0DPLDBq0rFFthv4qU19j3qp58O6t/khVzvj8yVV05o5KRQOA48o7r/xtWpXvOs3KPfymWWHffkc3emlVvuJH37vfQDWG+XAaxvZMmu/6eTiuxqe8skYBJK7IwY+kg09AzAN5yEEHbIItsInHYTvbhH7MrOVKh64TySkYGzYfNAg2uZKZ63kKTr6nZEGDfaxn1vIcxcfjC9fuUveYn8EkepxAsEw4/fPZl2n6WsOSbeKqKlqc/ydbtBc3XKanlzXSmIw/UvruDwi//FN6oTXixoBtxdYy4vb40In0q3v60Lq91Uo/Yg/AyN989/1uAN3Vo5epA7qy9zt9yqgYT3Oo8YefSKKHHx/ajhe03zw6WNFHz0imf+7SlXr07mvaBXtyj15ShQroE+alt9Phz+ZAYwLYggAfwhK6BwSwCWCLFWCD3rNV9ZRX6qTnlhn0TNrFsKtXTg56Fd7/yeGmPlTpA4dNNu9RFeM30yDHFXLq9uMKmGw55AKOPX/7qAJ5e6o+Ujzo+XQAcpBnWQCt6UuzzXtU4gCJI6YtVjTwo6rn6h1yg0dNp7u6P2DKeOoED9uE3nHyPaUDSQ9j4EdyCtYG1sXPx3LQ4UpYrpMQXAPc8jgSKnimLV5PK/PKVBs2ZSEBzCLxMl+o/VjHn6j0mNOSbXNhOb37H5/boh2uaaHxqy/Rgrz3qL7pUxpfdIVK6lojbmNmueJT15VzxHW6NXzqIqUfa2LwqBl+57rvwQGEpuvxd41YnZvmUPyQezxpcjtZ0Fjn9hNNtDy3jPr0H0Rdf92T+g4cbvKDPnHBalVI9ejVl7L2VJtj/mwINCaALXQsIhJBeEAAmwC2WAI26MbfY9txyEnPZ78TdhXL4IUrefTphRWmPlTpTyRNNu9RAQOYQI6rYQY0mw82Kho+AwKk8LjeQw7yoAG0IElgs0c1zg0najynzs96QMMc+j3rZBrbhB40gELoxZyQx7ygB2MD6+Ln4zmgAyBQ9x0A6DPj5ijdPE/+602mrbiGnC//sG5/PZJWrGMrXP3Y9zr7v08//4Z2VV5Tn0NPOK/TR9e/om++/V59Et1b10qRttFtgM1TD2IcgAn0p0ZOV2DIk0e/Z8Cm0/xdIy7npDmU/keHjKE7736g3bOA9uyEeW70Da85aXZqvlqTa3ZXm2MFlc2UnFtGd3XvRcOmLjLp/mwINCaArbOvrjg9nwA2AWzhJr1Q5Gpq69WnmECVqa9xb9W8zuut0kbigBzzpWxznbBtOtCoaNj4uVJnHu4hB3ncs9yc1HxVqaMq57bxNafi0fl1HZhDv2edTGPd6EErrGxWiaNX3yeU7SzPfP5sYB5+Pp4DOnQ/gJ40ZSHd/NNbKK/8Ms1o+0TK/NyrnwPS/Mf0YHsBbHHa1Imo6eoXlLzjChVXXqMr176kL7/6m2kM3sue2taI26g2wKbrmp3qUIBoXsZOpX/zYdeJG061dL6lW47QuJfSFa3ngwMITR/3d414xjzgAdBCUcT3oLENmDu3/LI6UWN9C9eVqEJk7b46GjJ2DgG48djNP7mFhk171bxneji9ADYz3OQimh4AYLtw4YI08UHMYwCbWKDK1Ne4r2qe+Rmw8T16gCPIMW1FG2DLPtCoaAAknlU480IO8rhft69ObfKL1peYupiPe51fpyG56Pesk2lsE3qmzUrZppLQQ48nqXlBD8YG1sXPx/pgg+4H0HFCATp4l245rK71U4et5ZeVDdOTc0y7WF+wPd63VfeWzn7C9pcvvqFLzZ/Rjc++ph88Egbey+6a1ojbi8tcPwvZvVdf4vbzLl1pZko+bT/VbOpfmV9Bd9//sMkD3v6DRtPaEqfieWTwGAJYYh3cL9l8xNSh24u4nZXiUGOFZ66q+TAvy+F6aU6ZOT5ierI51qVrNxo/fzVBLqO4Sp2as1zv/oMo+9Alr3Pq8wdzLYDNI+jkNjoeEMAmYLWjkio2sXCqVch4q+Z1XajQ/5A02U0/wBHkmG9FnuuEbeOBRkV7su1zDapw8OjVOOQgD3rR2auEzbzLHd1MXaDrFbrOz/N52gzwg9MAHkfPNqFnuqOiWc2HJIbkBHowNuDUAPxpBZWmLsiCpvsBNDzLnd17Kb2YD+B1wHMTTbmRM5KVXN7xJpPG9gXbC2CLzh4djpYffiD67ntPqObShPdSXNMaccs51kRLcsratYIzV9vpTi+ucuPDPduQVeJ0G2Od0M88eo9xfQzzsQz3Or+nndtONZt6mR897NDlIrkWwBZO1IpMQA8IYBPA1pGALZjq1BuPt2oelfG4uemqIlaAbehkt+oYgGvkzJUmbflWF2DbsL9R0VBN//65ifTLe/qoChz91MXZagxykGdbcBoA0MbVOPqxczPMkwRPfsiBBrDEOnKPNSlgxDpAZ5vQMx/6pbllShZAiumBbMCpAWzEz9phjsxd1UoWOvQTCIyBDycMrBunIPopB34GiE8pmCfUXgBbwO03Lgx4LzurW6XF0AcC2OIS2p1/UgFsAtg6ErCFW7V6VslcGXNVjIqdr3kOgJCNhy6ZVTMqa8jpJwAYZ13oufoGHfKsC73nSYBe5XvjZ926Dujk+UBnm3he5uVTg1Bs8LSRdepz8tyevoIsj6H3nJftCqUXwGbN/IH3UlTVIi2GPhDAZs3YT3irBLAJYOtIwCaVvX1ONgSwWTM94L0UVrVIi6EPBLBZM/YT3ioBbALYOhKwSWVvn5MNAWzWTA94LwXnW6TF0AcC2KwZ+wlvlQA2AWwdCdiksrfPyYYANmumB7wXx7kWaTH0gQA2a8Z+wlslgE0AW0cCNqns7XOyIYDNmukB72X7uRZpMfSBADZrxn7CWyWATQBbRwI2qeztc7IhgM2a6QHvZU7ph9Ji6AMBbNaM/YS3SgCbALaOBGxS2dvnZEMAmzXTwwfXvyJpsfdBsG//pmAZhU88IIBNAFtHAjap7O1zsiGATfKLeCCwBwSwBfaRcLR5QACbALaOAmyHzjSSNHv5oKNiK9R5sO/JP/GAFTwggM0KbyFBbBDAJoAt1GQn/BIziR4DAtgSJEHZwEwBbDZ4ydF6RAFsknwTPfmK/RLDocaAALZoZRDRE6kHBLBF6kEbyQtgk2QXarITfomZRI8BAWw2SnIWf1QBbBZ/QVYyTwCbJN9ET75iv8RwqDEggM1KWcjetghgs/f7D+npBbBJsgs12Qm/xEyix4AAtpDShDDH0AMC2GLo3M6mWgCbJN9ET75iv8RwqDEggK2zZbLEfR4BbIn77jrccgFskuxCTXbCLzGT6DEggK3DU41M6MMDAth8OEbI7T2Ajauurk6a+EBiQGLANjEggK19LhBKfDwggC0+fk/IWQWwCVgVwC4xYLcYEMCWkOmqUxotgK1TvtbYPJQANknWdkvW8rwS8wLYYpNPRGvoHhDAFrrPbCshgE2SlwAYiQG7xYAANtumPMs9uAA2y70S6xokgE2Std2StTyvxLwANuvmJLtZJoDNbm88gucVwCbJSwCMxIDdYkAAWwRJQ0Sj6gEBbFF1Z+dWJoBNkrXdkrU8r8S8ALbOndcS6ekEsCXS24qzrQLYJHkJgJEYsFsMCGCLc+KR6U0PCGAzXSEXgTwggE2Std2StTyvxLwAtkCZQcY7ygMC2DrK051gHmxcZWVl0sQHEgMSA7aJAQFsnSB5dZJHEMDWSV5kRzzGjRs3SJr4QGJAYsBuMdAR+6vMIR4I5AEBbIE8JOPiAfGAeEA8IB4QD4gH4uwBAWxxfgEyvXhAPCAeEA+IB8QD4oFAHhDAFshDMi4eEA+IB8QD4gHxgHggzh4QwBbnFyDTiwfEA+IB8YB4QDwgHgjkAQFsgTwk4+IB8YB4QDwgHhAPiAfi7AEBbHF+ATK9eEA8IB4QD4gHxAPigUAeEMAWyEMyLh4QD4gHxAPiAfGAeCDOHhDAFucXINOLB8QD4gHxgHhAPCAeCOQBAWyBPCTj4gHxgHhAPCAeEA+IB+LsAQFscX4BMr14QDwgHhAPiAfEA+KBQB74/zX7gpsWSaBOAAAAAElFTkSuQmCC"}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\"\n\nCFG = dict(\n    net_count         =   5,\n    batch_size        =  16,\n    \n    read_size         = 256, \n    crop_size         = 250, \n    net_size          = 224, \n    \n    LR_START          =   0.000005,\n    LR_MAX            =   0.000020,\n    LR_MIN            =   0.000001,\n    LR_RAMPUP_EPOCHS  =   5,\n    LR_SUSTAIN_EPOCHS =   0,\n    LR_EXP_DECAY      =   0.8,\n    epochs            =  12,\n    \n    rot               = 180.0,\n    shr               =   2.0,\n    hzoom             =   8.0,\n    wzoom             =   8.0,\n    hshift            =   8.0,\n    wshift            =   8.0,\n\n    optimizer         = 'adam',\n    label_smooth_fac  =   0.05,\n    \n    tta_steps         =  25    \n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os, random, re, math, time\nrandom.seed(a=42)\n\nimport numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\n\nimport PIL\n\nfrom kaggle_datasets import KaggleDatasets\n\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASEPATH = \"../input/siim-isic-melanoma-classification\"\ndf_train = pd.read_csv(os.path.join(BASEPATH, 'train.csv'))\ndf_test  = pd.read_csv(os.path.join(BASEPATH, 'test.csv'))\ndf_sub   = pd.read_csv(os.path.join(BASEPATH, 'sample_submission.csv'))\n\nGCS_PATH    = KaggleDatasets().get_gcs_path('melanoma-256x256')\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, cfg):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    DIM = cfg[\"read_size\"]\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = cfg['rot'] * tf.random.normal([1], dtype='float32')\n    shr = cfg['shr'] * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['hzoom']\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['wzoom']\n    h_shift = cfg['hshift'] * tf.random.normal([1], dtype='float32') \n    w_shift = cfg['wshift'] * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, cfg=None, augment=True):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img, cfg)\n        img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n\n    else:\n        img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n                                   \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, cfg, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ds = ds.batch(cfg['batch_size'] * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test the input pipeline\nBefore calling any neural net I always test the input pipeline.\nHere are images from the train data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_dataset(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n   \n    for idx, data in enumerate(iter(ds)):\n        img, target_or_imgid = data\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n\n    display(mosaic)\n    \nds = get_dataset(files_train, CFG).unbatch().take(12*5)   \nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test of image augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO)\nds = ds.take(1).cache().repeat()\nds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, cfg=CFG, augment=True), target), \n            num_parallel_calls=AUTO)\nds = ds.take(12*5)\nds = ds.prefetch(AUTO)\n\nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Images from the test data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, CFG, labeled=False).unbatch().take(12*5)   \nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(cfg):\n    lr_start   = cfg['LR_START']\n    lr_max     = cfg['LR_MAX'] * strategy.num_replicas_in_sync\n    lr_min     = cfg['LR_MIN']\n    lr_ramp_ep = cfg['LR_RAMPUP_EPOCHS']\n    lr_sus_ep  = cfg['LR_SUSTAIN_EPOCHS']\n    lr_decay   = cfg['LR_EXP_DECAY']\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(cfg):\n    model_input = tf.keras.Input(shape=(cfg['net_size'], cfg['net_size'], 3), name='imgIn')\n\n    dummy = tf.keras.layers.Lambda(lambda x:x)(model_input)\n    \n    outputs = []    \n    for i in range(cfg['net_count']):\n        constructor = getattr(efn, f'EfficientNetB{i}')\n        \n        x = constructor(include_top=False, weights='imagenet', \n                        input_shape=(cfg['net_size'], cfg['net_size'], 3), \n                        pooling='avg')(dummy)\n        \n        #x = tf.keras.layers.Dense(1024, activation='relu')(x)\n        #x = tf.keras.layers.Dropout(0.2)(x)\n        #x = tf.keras.layers.Dense(512, activation='relu')(x)\n        x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n        outputs.append(x)\n        \n    model = tf.keras.Model(model_input, outputs, name='aNetwork')\n    model.summary()\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model=get_model(CFG)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def compile_new_model(cfg):    \n    with strategy.scope():\n        model = get_model(cfg)\n     \n        losses = [tf.keras.losses.BinaryCrossentropy(label_smoothing = cfg['label_smooth_fac'])\n                  for i in range(cfg['net_count'])]\n        \n        model.compile(\n            optimizer = cfg['optimizer'],\n            loss      = losses,\n            metrics   = [tf.keras.metrics.AUC(name='auc')])\n        \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = get_model(CFG)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_train     = get_dataset(files_train, CFG, augment=True, shuffle=True, repeat=True)\nds_train     = ds_train.map(lambda img, label: (img, tuple([label] * CFG['net_count'])))\n\nsteps_train  = count_data_items(files_train) / (CFG['batch_size'] * REPLICAS)\n\nmodel        = compile_new_model(CFG)\nhistory      = model.fit(ds_train, \n                         verbose          = 1,\n                         steps_per_epoch  = steps_train, \n                         epochs           = CFG['epochs'],\n                         callbacks        = [get_lr_callback(CFG)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### predict the test set using augmented images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"CFG['batch_size'] = 256\n\ncnt_test   = count_data_items(files_test)\nsteps      = cnt_test / (CFG['batch_size'] * REPLICAS) * CFG['tta_steps']\nds_testAug = get_dataset(files_test, CFG, augment=True, repeat=True, \n                         labeled=False, return_image_names=False)\n\nprobs = model.predict(ds_testAug, verbose=1, steps=steps)\n\nprobs = np.stack(probs)\nprobs = probs[:,:cnt_test * CFG['tta_steps']]\nprobs = np.stack(np.split(probs, CFG['tta_steps'], axis=1), axis=1)\nprobs = np.mean(probs, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### sort predictions to have the same order as the submission\nThe submission ist sorted by image_name, but the dataset yielded a different order.\nTraverse the test dataset once again and capture the image_names. Then join this list of image_names with the predictions and sort by image_name.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, CFG, augment=False, repeat=False, \n                 labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### write a submission file for each submodel","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(CFG[\"net_count\"]):\n    submission = pd.DataFrame(dict(\n        image_name = image_names,\n        target     = probs[i,:,0]))\n\n    submission = submission.sort_values('image_name') \n    submission.to_csv(f'submission_model_{i}.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### write a submission file using the mean of all submodels","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(dict(\n    image_name = image_names,\n    target     = np.mean(probs[:,:,0], axis=0)))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission_models_blended.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l .","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}