{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 데이터마이닝 특론\n# (Special Topics in Data Mining)\n# Term Project\n\n## Topic: Using deep learning models with Tensor Processing Units and PyTorch to classify big data Images\n
\n\n### Supervisor: Prof. 김재환 (Jae-Hwan Kim)\n Full Professor (Tenure) of Department of Data Science, (National) Korea Maritime and Ocean University, Busan 49112, Republic of Korea\n\n### Student: TRAN DUY THANH\n### ID : 20207144\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"# Goal of the Term Project:\nThis term Project is from the international competition at the link https://www.kaggle.com/competitions/tpu-getting-started \n\nIt’s difficult to fathom just how vast and diverse our natural world is.\n\nThere are over 5,000 species of mammals, 10,000 species of birds, 30,000 species of fish – and astonishingly, over 400,000 different types of flowers.\n\nIn this competition, competitor will use Tensor Processing Units (TPUs) with deep learning models to classify images( over 100 types).\n\n\nWe will use deep learning with Tensor Processing Units and PyTorch to classify Images and do tasks:\n\n* Build library for Data Preprocessing \n* Build function for train and test, we will use an ensemble of pre-trained models: first train only the classifier on 10 epochs, then unfreeze the network and train all together for another 10 epochs. After that, the model makes predictions on the test data \n* Build function for prediction \n* Use 5 models: Dense Convolutional Network(DenseNet), Vision Transformer (ViT), GoogleNet, Residual Network(ResNet), VGG19 for training (https://pytorch.org/vision/0.13/models.html )\n* Comparison Accuracy value and Loss value for models with Visualization\n\n# Dataset\n\nwe're classifying 104 types of flowers based on their images drawn from different public datasets. Some classes are very narrow, containing only a particular sub-type of flower (e.g. pink primroses) while other classes contain many sub-types (e.g. wild roses).\n\nThe dataset contains imperfections - images of flowers in odd places, or as a backdrop to modern machinery - but that's part of the challenge! Build a classifier than can see past all that, to the flowers at the heart of the images.\n\n\nThis competition provides its files in TFRecord format. The TFRecord format is a container format frequently used in Tensorflow to group and shard data data files for optimal training performace. Each file contains the id, label (the class of the sample, for training data) and img (the actual pixels in array form) information for many images. Please see our Getting Started notebook or our Learn exercise for notes on how to load and use them! Additional information is available in the TPU documentation.\n\n\n\n* train/*.tfrec - training samples, including labels.\n* val/*.tfrec - pre-split training samples w/ labels intended to help with checking our model's performance on TPU. The split was stratified across labels.\n* test/*.tfrec - samples without labels - we'll predict what classes of flowers these fall into.\n","metadata":{},"attachments":{"fbc365bb-db3d-40b6-98a6-948f1754f837.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAANsAAAHTCAYAAACup22ZAAAgAElEQVR4nOy9e1hU5733/XlMpujEMLoDXoggCYcSFFCOEcmL+IKJ+FTsdjSSFHwTrG6j0SQmlpRNihpKQtjRWDQhGLAVU7Vm0orZSiK8UXYRgwciIugW8DCKVPCRmehQshL288cMMOBwUlwjeH+ui+ty1roPvzXOd92Hda/v/b/+53/+538QCAT3nIe7HtAf286W/2roctQWp6BwokLcsH1Ipsi6Q1/GtuyDdI2wnQnzeeNZp3tUpz3TF8XibzuwxQseDG4TW8uFQ2zKOGgh6Xu8pZzA4k2fkhhu389qytgyZw17AO9/+5T3ZvU3vxmGag5mfMQ33Z1fEj7wYmuvMxyn54XYBHfGbWLr4BkSt8UykRa0lYf46o+7+aa+ki3xcdho9vCGn02/Kmo4VclpYIzh7gI2Z2bSNuJ+3uXgGLeBq0AgGEB6EJszAU+H4A/wdDgLFi1i19KZvFVYw6aU3TynicW5LWl9Gbv++Cm5JZeNOX3nE7c8lqkOQMPXvPXr9ew3JT28/tf8Incir3yaykx74Cc91Qd3s+lPeVTrAJUb0+f/msWzJvTaZXX2DWFqoKUzl9nzxnK2VINqzrt8Fj8BgJa/v4c6vQRsnmVd7jKcCxN56ZPTMHE5W9+yZ//aDfylWofKPZrlby1iam8NcH0J2zZv5y/llwEV7v87nldiwnFvb/k6WvQ5az5l4t/XkFpwmTlr9rDYr2t+JwKei+WVmBDsTdfdsK8jvg9f1JGbsp3jTsvZmvoMd9E3EFiJHsTWhYecWPBvsSQXZtNSdogTDbE42wOXdxM/M5FvzFqs06fWkv/XQ7yXv4UFD+u4dqoGvelcS30lp+vHoPsJoIUT6c+hzqoxq6iS03/fS/7pv/DlW370r/1sw4mZseFsUn/E4VOb2fOvm5ljW8mWtdmcroHp73+Kvw00GBo4faoSRpaw6aXtbCszZT9VyeGvTvPxf61n5mjLNbQc24D6xY843em6S9jzyTN8uGczc0w92bYW3Tl7Dan7vgZg6k+W8ldy+tTXbPvjIjR73sJfCbTHt4cPfv01+ZeBcN0dfSMC6zOsX6md3JgKwEFOXwRo4cTu7Vxzc2Ni/DZOnjvH+XP5rPUDDAdJ3l0GDvPJqf0Li01FTH//MOdrt7DAAbj8NdtKbJjoEM7ab85xvvYcZ3JisQGqsz4iv9tZECO7fjuHX8zp/LfFJBgbv2Uk/soG+JoPPilD+9WnbKoB3F7njX/t0i6U7EYbvYeT585xck8y05WAYS+p2ystV/xTJVtWGYXi/KvNlJ47x/nvtrHYDbjxNQnpX7ffXNrIP/gDb2zLp/RwPnGPt+W3YXryHtP39g3vRdhATTa//+vlLvF9TXVEKl9+c5yifw9BDBkHJ/0T223Y4P/6Hr7ck8+XSSHY6PU0XLiGXmU829LSS3an2Xy4Zw9fHt7CQgc9+huXqa6HMabT//yp5+z6mkpOn+r819Cex4bpy5OZDmizVvGr9/bSghOL313ExK7dU2Usr8Qau622PrG8EmtsT7UFJ6i2VPHZg/zlMsAEfv3iM8Zun20Ib/x7rPG69+ZzuIvapq9J5ZWn3bB3cMP5Wlv++Tw3ywn0evR6W/7fefMBOLGvBG2n+BaRljSfiS62OLs53WFrL7A2fe9GAtxq4RoAE3Bq6141lLAp8Q02FTbQm7Zup4Xqz98j4d3tnLjR78ws3HKcN/w7H7Mxv+07zOeVFRv4JuMy2svAs6tYHGjhpxrshrOZAJ3dQ4CDcOrybS0UAAa9SQwTcR5rVvfItrK/R2cA8ybI/XGz1rQ9/3Zefmr77eWX1NAAHWPiLvEJBif9EtvpfdmcBnB6loDHAWrY8sJCPqgB26dfJ23ls0x1c+LcJ978Kqv38vRfreUXv9lNi3ICC955nbin/XH+8Stem5HY/dS+GTYqW2y7GVMBYCgh/69mfdGv8vj/L89mQdcnA9db0EH7pIPulvGWgpNtL61IA9f0gNL0Ueo4M7xP4ogl5/jrBNx23AZb6P5ZomBQ0kM3sgXdDT36G3r09TV8k7GQFzYYxxLTV843dsUaKimpAfDjjXeWMSfQDfvRDVQf7aHUHzvav+rKPGNrOP8t3vtVOBNdbLG5WsnhAbgwgNN/WsuWy2Azez05q92AgyRbGE9xKpu/lJjiMpTwl2zTWG3aBNwtFfxzf+YAcJCvDraNr1o4XPiV8Z9+0/DvabrQLH/JGRtsR9tiO9oWm8tfseWP2WzZV2m5RRUManpo2bYTH3B7F8d93mbWtk0w2Lvh7wTfXC7jg7fX0uBni77MbFavHSfcwoGDcDjlOX6xPZZ1Xy7D3fMZYC/sfo/XHgrHGS3f7Nzb5+7oluc82NL14JK/cP4tP6jZzu/SawA/ElfOZvrYR1m4eTHb9q7ig389ztpwszbLyYZvFk/lGzcnuFxJ9Q0AN16ZG2K5ZbN9hsWr3diTXsM3iTMJ3u7GmJbLnK7RA24s/u38ji5gL/m3LJ7O8WejCeAEu/5ahh4bpn8QK1q2IUjfJkiU9kx8Npa12w5z4P1nzMYPE1i8+S2mjgb937ezKSOb407v8l581wLs+eW/pzLHwcY4nV15hHOXwfbZVXwcMwEbQyV7cj5i018bmfPuW0y/26v66TK73nuPE4Dzkt+ywA1QhvPKO+HY0MK232/ghPnDdfdFfPzHWGxqTEIb7cfCTZ/2+OB+4st7OPD+fCYqW2iorOR0jR4bh3De+CyXREvjwq75l/yFL9+fz0QaOPHXbLb8tQz9aD8Wvr+Hj7vOlgqGBP9roBYit9zQ06K0xba335lBT4uNLTbmY5oWPXqDDbaj5Z1na/h8McG/OQjhqZTmzMf+pxb0evodR8sNPS0P2WDb68Xfm/yCwUH/ZiN7wGZ0b5MJJpQW0tn0QaRy8JBNzxMu3dDna79H+QWDg7t8ziYQCPrKgHUjByUtevQGjC3afdG0CoYyD7bYBAIZEd1IgUAmhNgEApkQYhMIZEKITSCQCSE2gUAmhNgEApkQYhMIZEKITSCQCSE2gUAmhNgEApkQYhMIZMLyKzYGPQapyzHFCJRKheVSpGYMBlCqRnQ5dlshndN0g3SzkbqaehjrhqPdCBSdbgkSkq6ZTiUPt0Up1hEL7nMsiE3i+JYEMk51ORy4hD8u9bstta50O+u3FnNR8iPh0yV4tZVSmsOyrRVdUndOcxuGCjSpWeytB9XoEXBTjw4HZi/7DWqfNpFWsPONLAq7ZFW6RrP2rSjsRVstuE+xILbrNDVAxLKPifO//WwHesozf0/GubHEzfAmZ1/ns003GrsVqEVaa9m7bjNF42JJTw7FXgEgUXdgM2kZGxiVnEjEuI7kneKTtOSnprLhC29S5/Xo/iEQWA0L7cBV6uodcHTsLesP4BHHxvTXCHv89u5l4+V63J3GWshnGcM3GjREkrC8TWgAChxnLGehj5ade29zEepA4UxAgAN1F2oZwH07BIIB5faWTaenARWOdQVo/vMSBpuxBD4TideYroKywzfCrpti9eh0oBp/laLt+7jYosTRL5II/+7SQ/XZWlymxOF4m/wVBKz4+HYXLXNa9dTU1OP4uGu7haNAcL9xu9huGjAoain6LwfC/DwZVVnA+sQDhC37PXH+vU9uGDFw658Kyg8fxnHKZDw4zf4tb5M3aQnvLvWzIIh6GurAfYpDnwMv2pZA+Y626iRUTy8naa7oQgruX24X27gokj6O6vgcFkpgYTpv/mUfEf5qeu1dAuBARPIfiGj/HMqU0EOsT/iCIq0fM7vRhPSj+acycn9tNhHiEE16SlS7a7H7U/OIatuKTVfD/v/MIkOVSMKsvgtWIJCTPs3dqSZ649JYy6W72a1otCeTxjRS+d+WvH4dcHSCqvO1Zsf8iPv0Y/746cf8ccntkyyOnkH4Bpv+ZsSQsHIaur9pON7/DQcEAlm4TWzSmQJyv6joPNFgMKBDhUrVx1JbzlKUnUdVJ3EaMLSA/WjLGx65T/KmqbiA8ttmOJo5XlKG8ueuPW8A+Ph43FsbabqDDToEAjm4fTpilMTFfTloSkwtUIuW/NwCfgiZavS9b63neHYWe081d1+qjQrp/H4ydxxFJwGtEnV/20meFETgRMtZFCFxLBpXRkZKDuVaU9kt9VRt30DmaWfUz3h2U5mEpNNSnpVHkZ03XqIXKbhPseiuZTi5k7TMQ1yUABQ4+qh5fcU04wPjlqNkrsihKmw1G2NdjRlOZPHiR3R+YK2rQJOexd5641oPxWhvFr22nCnjutZmRmsj5X/aTGZJPYZW4yHluFCWLl+Ab/tsaJexHMAwBY5u04hbqsarr62vQCAzPVrZSTf1YGOLoptVWn2ipRnDTw93v9TLcs1IumZQ3mXdAsF9hPCNFAhkQqwkFAhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCasKrbqzxYyydWDJ7r+xe9G+9M9rlxqpPzzHHKzd1J+7R7XNRAY9BjuctMQw7mjHdbu9wtSMwadhZhaJQzas1TVNiJ13Zq9jZZm6ioruHjtLq9JphisaNJaRqrrc91vcug3m1eetrC31CNOhEQ8y1Q3yxt03I5Ew6nvMDgF4TK67ZieI/+RwA4ieT7YlfFP+eFo0+8LkJWqzJfROCWT9Is73czAeM3ZxLDxzWn3xaaR3e3HzpX9pKXkUW1jy6iHmmn6XkXEikRifDr2B2zYl0ri3+oZpRoBBj1No6eRkBiDez8vTM4YLOypfZ9QtpdN3ezsu+m97aw9sIeFbpbPd+Y65btyqJsbRFy72AzcaoLAuWqm9Lhv+FDClilvfswUa4cB9Lgfe+tZdqbnIT2byJZfGm+2hhNZ/DYjB48PlxOgBLQaNvwNYpL/YNpnvZnyj/6d9Z/78dFCsw1YWrm979Z+TKYYzLC+2EJSKfpsPn3dM7Th88UE/+Ygl68BfRDbkWwNJXowHMgh90Iw6ggo+vwQx9qOlbkRsWgajroK8j+/jtccJeWfHaDosWjSY72N3c09X1Dw3VVw9CZyXrTZJh8AEg0leewtquAiYwkMm8vsELPtjHvMr+VIdhmqmd405Gso1AWzatU0VEg0nCig8EApVTgTMff527bLMlwopnDPIYoaFHhNDkc9NwhVL4OCuoM5FDKDuHBnMF2v7/POVO/QUHgeXCaHM3tOkNme5oDuLEV7Cyg5qUM1KZio2ZG4mG9ecu0oez8/yLE68JqmJtpDi+bEY6jnevfQepr2Y1/ijfK7LHLMT12o4PhNP+KiO34RSv+XWDhxJYdK9ARE2MJPdnjN9CSkfZOWEfhOeRLDF7U04Ik9YCjezJsFziQlRuOoAGimKnsNuYp41i70RCFDDF3pXWz6Snb96Ssu/2jppA1OkbEs8Olrl05+xvp44nimgobxPkzysEMxHFx8PKlrO+b2GI8ANGs5VnKYoqsjCHhqBnFu46Glgp2JWVRNiEYd7YfhUjG5yeuIWP0OM10BJKq2JrG+YizqObMIHN7IsV3rSL7yGmvnud6WH90pNEkJnFyeRtwkBdDIuZJiqmrKcJ8aiTrYDSXQ8GU6iQVK1HNnoR6u4+SfNlAE4GS6qMrt/PbDswTOj+X1Jw2U78rld1kPd7OFcge6M0cpxM8otmYtx0pKOXYVXJ6ahdrnB+qKdpCY2kjq21HGHYvqC0hbuw9lhJqo+T+j6fQ+Ut46y9IPTHf3K/tJWXsA1TNq1IE/o+lULhmHoUoKZmaPYuthP/Z/NnMLUFi4cVRduAR4w+PTiHu887nqk2dQ/nxa+49cGRrPijNrSE6FtYkz0G1bw/rLoaxN9EQhUwxd6V1sthP4ZfhBXnh+Ayc6bVRow/Tkv7B4wv0rNACXYG9c8jQoPIPw9TEe8wqGurZjnbqRPxCw4B3UHsZPugN5FI5fwMZFocYfTnAQXqpU3vzPo0SsCEJx4xB5xSri0l8jzNRF9R1vh2b/VepwRXFAQ/7oaDYuisTYGAThrtzMqzvyiJjUtmWynrEz1xAfZhoLtBxFk9dM9FuJJkGDr/cIMl/bTqMpyobaGnSTo4mbYeyuOK5wJuB71R2Mw+qxm76euFBj3b6BY5ES09F8G8nSEAXV+/ehC3+FhHmmQIKDcGQVOfu0BMxz4PgXeRgiVpPUfn4yj2SspOof/Q6kgycnE6bMoaREj1eI6bd1ZT/7T4EUaHmWwnAii4ySsSxMM+++jcBr0RpWZa8heWUBOESytr2VkyuGzvSpG2njs4w/78BMcEahfRg7AduH+lLCYEHJqEfb/i1RXanFceSPVJce7UjSqEC6dIkGgrCvPEuVRygrRpsV4RCE+iVj/uM19bj7T8a816X09sZraymXdOBoOjFqZMegmyuXqBnuSaSreaZggiZuZ7/po72rG6q/5ZKc2UhUmDe+bs7Yt9+kb9+/zv2X3U2sOOPhYVb3MFcC/GzZf64WQhRUVf7I2JDrlJdeb0/S1KygTn8JCYm6SyPwjTAPVIGvvzftgfYrlrYYvIl+0ZvkzCQSi7zxGn6Vkv9WEOJvR7Wl9Ff2sz7zDIFLf8+U0ZYS3AH3KIY+j9k6BPcRqtVDUWhd0dHUBNI/azh5yvz4Y0Q8afxlG3Q6GP4wlm+WxvyOdl26Ko8qGYWBW82ApY0bv9fRZNO1lVJ03qduQiwbU4MoKjrE4T8fILv+R9znJpIwywHwJuaDNNTm2Yd31/tQMapLDIpHlBhuGAAFTbqH0V05xUnzrZNtPIlwUCFhoEmnYNTwLvk7BdqfWDpQ+seTnlFP9UktBptIope50rBjFRftuozsdUfJTT+A3f+XSJz/iC6lGMdo6y+HsvYPxm6ksUvZt9ZtYGLoTL8mSGx8lqEpjaXFxhabIS00ADscnUAaGU7cAlfLSR4fjzL/PBdbQ3Fv69+3Shi+/xGlyg6XJ0agOXOW+GCzrsWFS1QNcyaku5v72LG43Kjo1PJBLRfPA+PNw/MkbJ4nYfOAcztZnZbLkdDVTFEpUKgU3dwAunKJuisQ0H55Epdq6nH3MP6gHMc1c8s/jrhQS6Xp8XDTc6hGz2zXDgFVnz4D7dNd/YnFhK6C/M+1uMdG4R5s+pJaaymqVDBpidmNy1BGbnIOFyOTSQq9fexlKM4h43Jou7gcTV3KTTs8TRMk9z6GrvT/obZycAqt6Ya+33m8pkeiKNxJvtbUT2/VU565imVZFUgAT04lSlnMzjytKUczF3elsiyjGB3g/nQoo/6u6cgvmfYnfzq043lOV8aEEulRy44dR01bHZv2I9e1JZAoz1zJq1lH27dCNjTquTVMhepRy0V2j5683DzqTOEZSrey7bQrkSF2gB0hkd4c372V8ra6JS35a1eSsq8esCVgZhB1uzehOaHFoNNTV7SZ3JP9jaELKjsU5/PY+cVZ4/W16rm4eyd5o2cR1nZTkLTkp2dxMSyx2y6pMnQ5H71t3oqNwGtRGqmxvQhtAGPoivWn/qv38umGy/R1mkVfabHX3AMOhMwNIj8zgRf3hJL0h1jc+5rVVU3Cr7ezPnUlmmEKaJFQjJvGqgRv43/YMFdmvxlLQ1o6L+4DBRKo/FiRYJoQMc+PAiSwD4xlXWz3g2iwZcqK5VxJzWLZ0lyUD0k8EriERf5ZFACgwHfhEsJSs1i2fAf2w5ppaHVg9orf4NXvW6cDMwMa2bB8JbcUEgZTOW3jDmVoPEk3PmHT6pdpUiigBex91Lw+0/jjUkyK593lGjJ3byK5BewmzGPVCwpezetvHJ1jingtnoYPN7NsifEuoHSNJOm1ae297oavctipBbSpvGj+fIwuD6YtfR99+o4GMAYzrLrNb/VnC1G/XUJ/2xzbp5PRbI3FXbYW1rjHtzTcFmV3K00MegyM6Gbv8DvcI7zHMk3nW0egHNm3QqsyXyZD+ZrxoWv9flKSSglJSSZiTFvXt5sxR6uE4ftmFJbi7/LgWCrezOICVzYmR1kckvYLgx7DQz1853IwgDFYtWVz/9U2Tv7KmhH0lT6MPZS2PUy9G/P3mx7L7MN5M6TGs1ReBvfpY28/OUyBsqf4ujmvO5DOm4ddSVgZjftoBVJjGZq9FTgGRt+90KBf13fPGMAYrN+NFMhAIyXbcqh2i2fpdFOHfZgS1WiVxQe3fUUV8RIr/rGVnMSV1EmgsLHDa/pykub2dT3Qg4VVu5ECwYOEeJ9NIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJCJIbFcq9sFzeGpFG2Zj/MgfCVIMPQYAsu1rOk/eZfcqKX8shIvH4f+vWApGJRYoWUrY1sGLFjhhyxvTtxT/8m75HwB679wJt0nqltHJsHQwSrdyMsbnuMF9vDnFRMGTnBy+0+2WbX15iup03LkQB6Hvm1E8YQ3kc+p8bUDw8k8NAe0oG9kb/ZVJkTEM+Xx7moXDAWsNkFyYsMcXsio5C7t62VjrI8njjagavOfBKMv5Fvr0OhciYyexdQxWnKT15Ffa8rUepadyans13miXhlP1Bgt2SlZVBlAYefKpPG2YGOHu48PYwfkBTDB/YxVJ0hObJgz8C3cPcKS/6SuqBdfyWu1VN/0Q/1SpNEQyHk560J0KIeDYpw3vp6H4bQzXsFBohv5AGDl2UgbVLbQ8hOD0ESod19JxzGuuI/MI+N3WcQ8Mw0vf1ccx/XuwiQYmlhRbIPd6LV3X0mGeRKT/g6BBw9RVLKTvB31GMZFk5posvcWPFBYSWyDXWjQJ19JgIfscJ+hxn2GGlpr0SSkk/tNKKsi7m/bdsHAY5X7q9PrWwat0Mz9J3vzlZSOZbF4VVaH7+L319G1gP1oM6G16DB0t9GeYEgxdB5q24ewMMavH/6TeWwrvMziz86RGNK3PIYTOSRnHqVheIf/ZNtmenWdfCXbNsRrpnr7+6QdvI5SBQYdOIYvISHW9NigtZ7C1FRyL0j4Pp8mWrshzhAQ2/3gP9mLr6TUjMEAikdH3JWblWBwMyTEJhAMBsR9ViCQCSE2gUAmhNgEApkQYhMIZEKITSCQCSE2gUAmhNgEApkQYhMIZEKITSCQCSE2gUAmhNgEApkYEmKr/mwhk1w9eKLrX/xutD9ZO7outDZSVVSBrtXagQjkZggsRB5kvpHndvJqWilhCetRe9xFOYJBhxXENtC+kb2IrUcm9MM3sp7CpLXUzf2YOP87qkzwgCN8I+/CN5KTeWhueBOtLGPH16XYz0lD7YPJSzKPw2fOonvUj8DwSCImmXxJdBXkf16Ly7xovFRajmSXoZo7FalwF/u/1aF6ahrRs0JxVN7h9yC4bxG+kX3Ekm/kLW0ZhflbyfxaYsIz8wh0wugVufptsi+omDw3HrUflGxOJfekyfugWcuxkjLqmgEaOVdSjGbzVk6qgomaH8wjJ7aTnHEIXfehCAYpwjeyj1jyjQTgp4nEJMbg0nbbanUlamUiUY87oxoG4En81TKSj50lbpK3hZL1qMLWEBc2AgDf8RIXk0qp0k1jijBuHVII38i7xUaF0rx/MEyBytUB6VotVReuUvffZ6ms1CON797VZ9TIER0fHBxwoZRbzYAQ25BC+EYONIYyct/OomiYKyFPOuPiEcxUznP8prUDE1gb4Rs5wBhKCigcrWZjUmR7w1S3K9eqMQnuD4RvZD8x9420hHK0CsXNRpraeo2NxeSV9Nf3SzAUsULL5sfCFfeg2Oq9fLrhcj98I6v7WYEDIXODyM9M4MU9Rt9Ii0OqydEsOriBlJcPoxwJhocmE/OUA0fE9OIDzxBYQXI/+EZaoKUZg/QwypFiT1GBkSEhNoFgMDAkFiILBIMBITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmbDym9oDQ7cLkcNTKdoyH2erv8pj3OAepS0KsS75gWUILESWyzfybigj99dZsEzY4D3IWKFlG2jfyN6q28umMsunNr23vR++kQLB3SF8I/voG1m+I4emp+IJc+04pjumIa/Rm5iZniha9Vws3EfhibNcZCyBIVFEhDkj7B8FbQjfyD6ilCrYeajC7IiWks8LuKVyRYGeI+kJpHxjwH1mDHEznbnyRSrr99VbLV7B/YdVZyMHk+Dcp4cyquQwVW0bYmhLKfo/3gT5K4ARBLywmrVJ8YRN8sR9UhSL5ntTfbRMmK0K2hG+kX3FOZiwf0nl2Hfg5Q91335HU8gCAmwAFCicXbG/qaW6tJ66S6eoPlMDkis/WDtuwX2DFVu2wWZn50zA0w4UFR9FQkv5sVuEBZocjlvrKUxZybKEHApOnUKn8iHMX8y6CDojfCP7gX1oKO55pZRXOlBo8Off2mzIzxSguRZMQkYs7qbbl1RcarU4BfcnwjeyP4wOZppbBduyilGETcO97fi/PIbjP3U0fm/6bKglv6Cim0IEDyrCN7Jf2BIQ5k12dj1RT5k9ZHAIJyaslLQ3XmbbSAVSiwMxz/rBsTuoQjBkGQIrSO4j30ipGYMBlKoRvacVPHAMCbEJBIMBsepfIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmbCq2Ko/W8gkVw+e6PoXvxvtT/e4cqmR8s9zyM3eSfm1e1zXQGDQY7hL/wjDuaOUa5sHJp6BoFXCoD1LVW0jktRNGkMjF0+epe5mdwlMSM0YdPruy7kPYrHiQmS5/B4lGk59h8EpCJfRbcf0HPmPBHYQyfPBrox/yg9HWXz17pyqzJfROCWT9AuHOyzBeM3ZxLDxzWnWd/26sp+0lMQIpHsAACAASURBVDwuKm15hGaaDI8xc9lvUPu0vTHRTFX2GtYfkxg1UsEPN/X8zCOG37w2DfvbmgiJquwE0kqaibgTb06ZYrl/HZEHzO/xOuW7cqibG0Rcu9gM3GqCwLlqpjwwpqm2THnzY6ZYOwwAtORn5CH972Q+Mt08DKVZvLlFw6Q/xOIOSCU5rD/lxqoPluClBFrryf/dWnK/8WNVRJcbbeUuMk/Z4T5ae3tVrdzef+t0TL5YrC+2e+z3eCRbQ4keDAdyyL0QjDoCij4/xLG2Y2VuRCyahqOugvzPr+M1R0n5Zwcoeiya9FhvY3dzzxcUfHcVHL2JnBeN7xhzD3GJhpI89hZVGP0iw+YyO8TO7HRP+bUcyS5DNdObhnwNhbpgVq2ahgqJhhMFFB4opQpnIuY+j32X6zJcKKZwzyGKGhR4TQ5HPTcIVS+DgrqDORQyg7hwZzBdr+/zzlTv0FB4HlwmhzN7ThD25penO0vR3gJKTupQTQomanYkLiqz89eOsvfzgxyrA69paqI9tGhOPIZ6rnePrecojyjU/09HK60MDiIkK4+L9eDuAJLSmbBf+hl/3ADDHAiY4sDOczUQ4ddRUGste7edIPClV3DcnU5dp1q07E3YQNP8NcQFm0RxZT9pqRWErF1NmJ2csfRFbPpKdv3pKy7/aOmkDU6RsSzwkcPC+84Y6+OJ45kKGsb7MMnDDsVwcPHxpK7tmNtjPALQrOVYyWGKro4g4KkZxLmNh5YKdiZmUTUhGnW0H4ZLxeQmryNi9TvMdAWQqNqaxPqKsajnzCJweCPHdq0j+cprrJ3nelt+dKfQJCVwcnkacZMUQCPnSoqpqinDfWok6mA3lEDDl+kkFihRz52FeriOk3/aQBGAk+miKrfz2w/PEjg/ltefNFC+K5ffZT3Mu0v9evyB684cpRA/o9iatRwrKeXYVXB5ahZqnx+oK9pBYmojqW9HGbtH9QWkrd2HMkJN1Pyf0XR6HylvnWXpB8sJUAJX9pOy9gCqZ9SoA39G06lcMg5DlRTMzB7F5syURV1ur6dOU6J0Y5XpN6+cFE1cpwR6qr6rxzek8x22IS+X/eOf5z8mKSnZfXs9s1+bQVrKGnJZQ9y4YtJSDuD40pp2ockXS1/EZjuBX4Yf5IXnN3DCYH7CaNqzeML9KzQAl2BvXPI0KDyD8DUZ9HgFQ13bsU7dyB8IWPAOag/jJ92BPArHL2DjolDjDyc4CC9VKm/+51EiVgShuHGIvGIVcemvEWbqovqOt0Oz/yp1uKI4oCF/dDQbF0VibAyCcFdu5tUdeURMUuMIgJ6xM9cQH2YaH7QcRZPXTPRbiSZBg6/3CDJf206jKcqG2hp0k6OJm+EJgOMKZwK+V93BOKweu+nriQs11u0bOBYpMR3Nt5EsDVFQvX8fuvBXSJhnCiQ4CEdWkbNPS8A8B45/kYchYjVJ7ecn80jGSqr+0c8wDGXkZhfjPietw9elCw1fbiK3MZSE6Wa/tyt5fLLPjkUfBqGkG0PccVEkJEFaShKLeZiwl8xaOZlj6VM30sZnGX/egZngBqc7Vu8oGfVo278lqiu1OI78kerSox1JGhVIly7RQBD2lWep8ghlxWizIhyCUL9kzH+8ph53/8mY97qU3t54bS3lkg4cTSdGjTSzUbhyiZrhnkS6mmcKJmjidvabPtq7uqH6Wy7JmY1EhXnj6+aMffud2riJR6FZdvdfdjex4oyHh1ndw1wJ8LNl/7laCFFQVfkjY0OuU156vT1JU7OCOv0lJCTqLo3AN8I8UAW+/t60B9qXWFrrKUzP4tiTS3i36/jHhOFEFil5EJPc4V4G9RR+WoByQYqxlR0I7nEsfR6zdQjuI1Srh6LQuqKjqQmkf9Zw8pT58ceIeNL4yzbodDD8YSzvAmXM72hn1/nwo0pGYeBWM3RSYRvf62iy6dpKKTpvNTUhlo2pQRQVHeLwnw+QXf8j7nMTSZjlAHgT80EaavPsw7u7k6sY1SUGxSNKDDcMgIIm3cPorpzi5A2zBDaeRDiokDDQpFMwaniX/J0C7SWWVj3lWe+T9y+xrFvSTRdYm8f6LY1Ern6diHEdh3WFuexsjSQpEAw6PaDD0Aq3buoxGEagVJriuLLf1HVMIWFccUeXsmvrJkMs/ZogsfFZhqY0lhYb2/vfwfiuscPRCaSR4cQtcLWc5PHxKPPPc7E1tOMu1yph+P5HlCo7XJ4YgebMWeKDPTvyXLhE1TBnQrqbwR87FpcbFZ1aPqjl4nlgvHl4noTN8yRsHnBuJ6vTcjkSupopKgUKlaKbG0BXLlF3BQLaL0/iUk097h7GMYzjuGZu+ccRF2qpND0ebnoO1eiZ7drxw60+fQbap7t6iqWZqqw1ZPxjBqlvh1qe3Lmyn5R3DuKy9PfM9uhsolR3rhbpSi3Jb+zvnGdbAkcql/DHpX6Alr0fGsdoRnGZupSpn1Dkutps3CZHLHfyUFs5OIXWdKO/3lvgNT0SReFO8rWmp5OtesozV7EsqwIJ4MmpRCmL2ZnXNs3bzMVdqSzLKEYHuD8dyqi/azryS1rycwv44elQvLqrdEwokR617NhxFEMrgETd33aS175pgER55kpezWo7D4ZGPbeGqVA9arnI7tGTl5tHnSk8Q+lWtp12JTLEDrAjJNKb47u3Ut5Wt6Qlf+1KUvbVA7YEzAyibvcmNCe0GHR66oo2k3uyL/VK1P1tA+svh7M2McrCsypAd5TcdJNQ/G93K/Na+jF//NT8L5k4B4hY9nH7jxucmZ22vnMrNi6KhAxzockVy/0w9X/P/R4dCJkbRH5mAi/uCSXJ9OykT7iqSfj1dtanrkQzTAEtEopx01iV4G28Ww9zZfabsTSkpfPiPlAggcqPFQmmCRHz/ChAAvvAWNbFevZQqS1TViznSmoWy5bmonxI4pHAJSzyz6IAAAW+C5cQlprFsuU7sB/WTEOrA7NX/Aavft86HZgZ0MiG5Su5pZAwmMqZYhqDKkPjSbrxCZtWv0yTQgEtYO+j5vWZxmZZMSmed5dryNy9ieQWsJswj1UvKHg1r5dq6wvI+VKLhJbElzu3Bm0Pgqt25FB4E8hOoCjbPORo0lOibnsU0i2WvhPzYzLGYlUru/vG77FXjNv0SsNtUXa30sSgx4DZWMFC/n5v89tjmabzrSNQjuxboVWZL5OhfI2PFnpC/X5SkkoJSUkmYkxb17cbv8tWCcP3zSgsxd/lobFUvJnFBa5sTI6yOCR9kLFqy+b+q22c/JU1I+grfRgHKW17mHo35u83PZbZh/NmSI1nqbwM7tPH3n5ymAJlT/F1c153IJ03D7uSsDIa99EKpMYyNHsrcAyMFkKzgPW7kQIZaKRkWw7VbvEsbXs2NEyJarQKxV0sRVdFvMSKf2wlJ3EldRIobOzwmr6cpLl9XQ/0YCEckQUCmRDvswkEMiHEJhDIhBCbQCATQmwCgUwIsQkEMiHEJhDIhBCbQCATQmwCgUwIsQkEMjEklmt1u6A5PJWiLfNxHoSvBAmGHkNguZY1/Sfvkhu1lF9W4uXj0MeXPQWDGSu0bGVsy4AFK/yQxRf1nvpP3iXnC1j/hTPpPv14P0swaLFKN/Lyhud4gT38ecWEgROc3P6TbVZtvflK6rQcOZDHoW8bUTzhTeRzanztwHAyD80BLegb2Zt9lQkR8Ux5vP+XLRg8WG2C5MSGObyQUcld2tfLxlgfTxxtQNXmPwlGX8i31qHRuRIZPYupY7TkJq8jv9aUqfUsO5NT2a/zRL0ynqgxWrJTsqgygMLOlUnjbcHGDncfH8aKF8CGPFadIDmxYc7At3D3CEv+k7qiXnwlr9VSfdMP9UuRRkMg5+WsC9GhHA6Kcd74eh6G0854BQeJbuQDgJVnI21Q2ULLTwxCE6HefSUdx7jiPjKPjN9lEfPMNLz8XXEcZ9d9kYIhjRXFNtiNXnv3lWSYJzHp7xB48BBFJTvJ21GPYVw0qd25OAmGNFYS22AXGvTJVxLgITvcZ6hxn6GG1lo0CenkfhN6++4ngiGPVe6vTq9vGbRCM/ef7M1XUjqWxeJVWR2+i99fR9cC9qPNhNaiw3CnG/gJBhVD56G2fQgLY/z64T+Zx7bCyyz+7ByJIX3LYziRQ3LmURqGd/hP6kq3s35rMXWdfCVjcFcCNFO9/X3SDl5HqQKDDhzDl5AQa3ps0FpPYWoquRckfJ9PE63dEGcIiO1+8J/sxVdSasZgAMWjI+7KzUowuBkSYhMIBgPiPisQyIQQm0AgE0JsAoFMCLEJBDIhxCYQyIQQm0AgE0JsAoFMCLEJBDIhxCYQyIQQm0AgE0JsAoFMDAmxVX+2kEmuHjzR9S9+N9qfrB1dF1obqSqqQNdq7UAEcjMEFiIPMt/Iczt5Na2UsIT1qD3uohzBoMMKYhto38hexNYjE/rhG1lPYdJa6uZ+TJz/HVUmeMARvpF34RvJyTw0N7yJVpax4+tS7OekofbB5CWZx+EzZ9E96kdgeCQRk0y+JLoK8j+vxWVeNF4qLUeyy1DNnYpUuIv93+pQPTWN6FmhOCrv8HsQ3LcI38g+Ysk38pa2jML8rWR+LTHhmXkEOmH0ilz9NtkXVEyeG4/aD0o2p5J70uR90KzlWEkZdc0AjZwrKUazeSsnVcFEzQ/mkRPbSc44hK77UASDFOEb2Ucs+UYC8NNEYhJjcGm7bbW6ErUykajHnVENA/Ak/moZycfOEjfJ20LJelRha4gLGwGA73iJi0mlVOmmMUUYtw4phG/k3WKjQmnePximQOXqgHStlqoLV6n777NUVuqRxnfv6jNq5IiODw4OuFDKrWZAiG1IIXwjBxpDGblvZ1E0zJWQJ51x8QhmKuc5ftPagQmsjfCNHGAMJQUUjlazMSmyvWGq25Vr1ZgE9wfCN7KfmPtGWkI5WoXiZiNNbb3GxmLySvrr+yUYilihZfNj4Yp7UGz1Xj7dcLkfvpHV/azAgZC5QeRnJvDiHqNvpMUh1eRoFh3cQMrLh1GOBMNDk4l5yoEjYnrxgWcIrCC5H3wjLdDSjEF6GOVIsaeowMiQEJtAMBgYEguRBYLBgBCbQCATQmwCgUwIsQkEMiHEJhDIhBCbQCATQmwCgUwIsQkEMiHEJhDIhBCbQCATQmwCgUxY+U3tgaHbhcjhqRRtmY+z1V/lMW5wj9IWhViX/MAyBBYiy+UbeTeUkfvrLFgmbPAeZKzQsg20b2Rv1e1lU5nlU5ve294P30iB4O4QvpF99I0s35FD01PxhLl2HNMd05DX6E3MTE8UrXouFu6j8MRZLjKWwJAoIsKcEfaPgjaEb2QfUUoV7DxUYXZES8nnBdxSuaJAz5H0BFK+MeA+M4a4mc5c+SKV9fvqrRav4P7DqrORg0lw7tNDGVVymKq2DTG0pRT9H2+C/BXACAJeWM3apHjCJnniPimKRfO9qT5aJsxWBe0I38i+4hxM2L+kcuw78PKHum+/oylkAQE2AAoUzq7Y39RSXVpP3aVTVJ+pAcmVH6wdt+C+wYot22Czs3Mm4GkHioqPIqGl/NgtwgJNDset9RSmrGRZQg4Fp06hU/kQ5i9mXQSdEb6R/cA+NBT3vFLKKx0oNPjzb2025GcK0FwLJiEjFnfT7UsqLrVanIL7E+Eb2R9GBzPNrYJtWcUowqbh3nb8Xx7D8Z86Gr83fTbUkl9Q0U0hggcV4RvZL2wJCPMmO7ueqKfMHjI4hBMTVkraGy+zbaQCqcWBmGf94NgdVCEYsgyBFST3kW+k1IzBAErViN7TCh44hoTYBILBgFj1LxDIhBCbQCATQmwCgUwIsQkEMiHEJhDIhBCbQCATQmwCgUwIsQkEMiHEJhDIhBCbQCATQmwCgUxYVWzVny1kkqsHT3T9i9+N9qd7XLnUSPnnOeRm76T82j2uayAw6DHcpX+E4dxRyrXNAxPPQNHSTF3lUaq1zUitlpNIN7VUl56l7qbUfTkGPQZDD+d7REK6VkvVyVoaui2jD2laJQy67q/DiguR5fJ7lGg49R0GpyBcRrcd03PkPxLYQSTPB7sy/ik/HGXx1btzqjJfRuOUTNIvHO6wBOM1ZxPDxjen3ReuXw2FH/L+rlp+UI3gZ1IzTQ+5smjVa0wZZ0rQWs+RDzeQfU5i1EgFP9zU8zOPGH7z2jTs25qJ1kbK/7SZjOJ6pMAl/HGpX/+CMFSgSc0i/8YIRinhlq4Z+7AlJMR6d3xHfUhjOLef7Mw8jusciEtJJsLCf9P964g8YH6P1ynflUPd3CDi2sVm4FYTBM5VM+WBMU21ZcqbHzPF2mG0odWwYZeBmcl/IGIcgMTFHetI+eIoASuCUAB1X2wm+/twUjdHGcUl1aJ5Ox3Nt1NZGqIA3VFy1uVSNS4a9SQNO7vW0crtfbdOxySOZ2+myGkJG1P8jMLRFZO5ehdF07yZ6dy3NHV/W0vyVzBz9jQavzjb7SVbX2z32O/xSLaGEj0YDuSQeyEYdQQUfX6IY23HytyIWDQNR10F+Z9fx2uOkvLPDlD0WDTpsd7G7uaeLyj47io4ehM5LxrfMeYe4hINJXnsLaow+kWGzWV2iJ3Z6Z7yazmSXYZqpjcN+RoKdcGsWjUNFRINJwooPFBKFc5EzH0e+y7XZbhQTOGeQxQ1KPCaHI56bhCqXgYFdQdzKGQGceHOYLpe3+edqd6hofA8uEwOZ/acIOzNL093lqK9BZSc1KGaFEzU7EhcVGbnrx1l7+cHOVYHXtPURHto0Zx4DPVc7x5bT0nhTcyKULzaWjEUuHg6I51upAmwR0IxIZpVIZM7WjGFKxMeB83164ADtMCEhSnET7KlKlPTpQYtexM20DR/DXHBpl7Qlf2kpVYQsnY1YXYAzYwNiWeVm19HrCo3PMY0UtcAxh9l72mkUaGs/UMkjtf3U/VF99fcu9j0lez601dc/tHSSRucImNZ4COHhfedMdbHE8czFTSM92GShx2K4eDi40ld2zG3x3gEoFnLsZLDFF0dQcBTM4hzGw8tFexMzKJqQjTqaD8Ml4rJTV5HxOp3mOkKIFG1NYn1FWNRz5lF4PBGju1aR/KV11g7z/W2/OhOoUlK4OTyNOImKYBGzpUUU1VThvvUSNTBbiiBhi/TSSxQop47C/VwHSf/tIEiACfTRVVu57cfniVwfiyvP2mgfFcuv8t6mHeX+vX4A9edOUohfkaxNWs5VlLKsavg8tQs1D4/UFe0g8TURlLfNrUk9QWkrd2HMkJN1Pyf0XR6HylvnWXpB8sJUAJX9pOy9gCqZ9SoA39G06lcMg5DlRTMzF7EpnDwxLdTV6uZqtIzqH4+zXRjUWA/IajzTcZQxrEzI3APM2UcE8SUMd3V4Mzs12aQlrKGXNYQN66YtJQDOL60xiQ0AFscA4M6Z7tSRsl1VyKf6Hsal/DIHq60g97FZjuBX4Yf5IXnN3DCYH7CaNqzeML9KzQAl2BvXPI0KDyD8DUZ9HgFQ13bsU7dyB8IWPAOag/jJ92BPArHL2DjolDjDyc4CC9VKm/+51EiVgShuHGIvGIVcemvEWbqovqOt0Oz/yp1uKI4oCF/dDQbF0VibAyCcFdu5tUdeURMUuMIgJ6xM9cQH2Z6u7vlKJq8ZqLfSjQJGny9R5D52nYaTVE21NagmxxN3AxPABxXOBPwveoOxmH12E1fT1yosW7fwLFIielovo1kaYiC6v370IW/QsI8UyDBQTiyipx9WgLmOXD8izwMEatJaj8/mUcyVlL1j36EcO0QGen7uGSQUExSsy7W87YkusIPScm/yq2WnxGyIJGYCX0se1wUCUmQlpLEYh4m7CWzVq4L1dsT+OSkRJM0FvWrrzBl9J2l6Yk+dSNtfJbx5x2YCW5wumP1jpJRj7b9W6K6UovjyB+pLj3akaRRgXTpEg0EYV95liqPUFaYf+kOQahfMuY/XlOPu/9kzHtdSm9vvLaWckkHjqYTo0aa2ShcuUTNcE8iXc0zBRM0cTv7TR/tXd1Q/S2X5MxGosK88XVzxr79bm3cxKPQLLv7L7ubWHHGw8Os7mGuBPjZsv9cLYQoqKr8kbEh1ykvvd6epKlZQZ3+EhISdZdG4BthHqgCX39v2gPtSywqTyLnK5H+eYlDe3axfruKhIWdW0Xlk5HEPdqM4VIxmu0b2WmbSIzPwFpP2E+dR9zPoen0PnIzP2HUarOJmn6k6Yk+j9k6BPcRqtVDUWhd0dHUBNI/azh5yvz4Y0Q8afxlG3Q6GP4wlneBMuZ3tLPrfPhRJaMwcKsZOqmwje91NNl0baUUnbeamhDLxtQgiooOcfjPB8iu/xH3uYkkzHIAvIn5IA21efbh3fU+VIzqEoPiESWGGwZAQZPuYXRXTnHyhlkCG08iHFRIGGjSKRg1vEv+ToH2IRYbB7yCHYAgfH1UpK3WUPKMd6fZPMU4b3zH0d6zeHXXQSJ8om4bx97Glf2mrmMKCeOKO7qUFlo3lWsQvq7GOhy3riJtbxlTusxs9iVNT/RrgsTGZxma0lhabGzvfwfju8YORyeQRoYTt8DVcpLHx6PMP8/F1tB2v0haJQzf/4hSZYfLEyPQnDlLfLBZ1+jCJaqGORPS3Qz+2LG43Kjo1PJBLRfPA+PNw/MkbJ4nYfOAcztZnZbLkdDVTFEpUKgU3dwAunKJuisQ0H55Epdq6nH3ME5XOY5r5pZ/HHGhlkrT4+Gm51CNntmuHT/e6tNnoH26q/tY6oq2U9g6zTh+bGP0Yziio+H/AA5ajmw9ANPjmfJ4RxLVY3agv44OehGblr0fGsdoRnGZupSpn1DkapoguVlB/q5aXOZF42V201HZq5BO6zEAyr6k6TGODvr/UFs5OIXWdKO/3lvgNT0SReFO8rWmh5itesozV7EsqwIJ4MmpRCmL2ZmnNeVo5uKuVJZlFKMD3J8OZdTfNR35JS35uQX88HQoXt1VOiaUSI9aduw4iqEVQKLubzvJa980QKI8cyWvZrWdB0OjnlvDVKgetVxk9+jJy82jzhSeoXQr2067EhliB9gREunN8d1bKW+rW9KSv3YlKfvqAVsCZgZRt3sTmhNaDDo9dUWbyT3Zt5rtFTqK/pzL3jOm/5dWPRd37adopD+BTwI4oLh5lOw/abjYVr/uLHv3lqEKDOrw7OwWZ2anre/cio2LIiFjdccEyUgVnNtP5rZiGtqeU18rRlNQj2+gabKpL2n6iPWn/u+536MDIXODyM9M4MU9oST9IbYP/1EmXNUk/Ho761NXohmmgBYJxbhprErwNt6th7ky+81YGtLSeXEfKJBA5ceKBNOEiHl+FCCBfWCsxUmADmyZsmI5V1KzWLY0F+VDEo8ELmGRfxYFACjwXbiEsNQsli3fgf2wZhpaHZi94jd49fvW6cDMgEY2LF/JLYWEwVRO28BfGRpP0o1P2LT6ZZoUCmgBex81r880NsuKSfG8u1xD5u5NJLeA3YR5rHpBwat5vdesCFnC2ptZbNqYgMb0I1Y4hLIiqc1VWkHA0kTiPtpEyhsFxpsbClxC43v5/syw9H10OubMzN8u59b6LBJf3m6sY5gtAdGrWRph2480fcOqVnb3jd9jrxi36ZWG26LsbqWJQY+BESiVljpNd7jNb49lms63jkA5sm+FVmW+TIbyNT5a6An1+0lJKiUkJZmIMW1d324mHVolDN83o7AUf5cHx1LxZhYXuLIxOcrikNQS0k09kqKH79bUNVc8OgLFvVpg2BfPz7v0BbVqy+b+q22c/JU1I+grfRgHKW176FIY8/ebHsvsw3kzpMazVF4G9+ljbz85TIGyp/i6Oa87kM6bh11JWBmN+2gFUmMZmr0VOAZG91loAIqRtj1/t73FNxAoRqDsLei+pOkB63cjBTLQSMm2HKrd4lk63dT1GaZENVp1Vy2FKuIlVvxjKzmJK6mTQGFjh9f05STN7et6oAcL4YgsEMiEeJ9NIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJCJIbFcq9sFzeGpFG2Zj/MgfCVIMPQYAsu1rOk/eZfcqKX8shIvH4c+vuwpGMxYoWUrY1sGLFjhhyy+qPfUf/IuOV/A+i+cSe/LK/6CQY9VupGXNzzHC+zhzysmDJzg5PafbLNq681XUqflyIE8Dn3biOIJbyKfU+NrB4aTeWgOaEHfyN7sq0yI6Pz6v2DoYbUJkhMb5vBCRiV3aV8vG2N9PHG0AVWb/yQYfSHfWodG50pk9CymjtGSm7yO/FpTptaz7ExOZb/OE/XKeKLGaMlOyaLKAAo7VyaNtwUbO9x9fBh7F+9JCQYHVp0gObFhzsC3cPcIS/6TuqJefCWv1VJ90w/1S5HGV/2dl7MuRIdyuMkxyvMwnHbGKzhIdCMfAKw8G2mDyhZafmIQmgj17ivpOMYV95F5ZPwui5hnpuHl74rjOLvuixQMaawotsFu9Nq7ryTDPIlJf4fAg4coKtlJ3o56DOOiSU2M6vCvFzwwWElsg11o0CdfSYCH7HCfocZ9hhpaa9EkpJP7TSir+unMJBj8WOX+6vT6lkErNHP/yd58JaVjWSxeldXhu/j9dXQtYD/aTGgtOu54Dz/BoGLoPNS2D2FhjF8//Cfz2FZ4mcWfnSMxpG95DCdySM48SsPwDv9JXel21m8tpq6Tr2QM7kqAZqq3v0/awesoVWDQgWO42SZ6rfUUpqaSe0HC9/k00doNcYaA2O4H/8lefCVNfoP31PdQcN8zJMQmEAwGxH1WIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRBiEwhkQohNIJAJITaBQCaE2AQCmRgSYqv+bCGTXD14outf/G60P1k7ui60NlJVVIGu1dqBCORmCCxEHmS+ked28mpaKWEJ61F73EU5gkGHFcQ20L6RvYitRyb0wzeynsKkxv37PgAAIABJREFUtdTN/Zg4/zuqTPCAI3wj78I3kpN5aG54E60sY8fXpdjPSUPtg8lLMo/DZ86ie9SPwPBIIiaZfEl0FeR/XovLvGi8VFqOZJehmjsVqXAX+7/VoXpqGtGzQnFU3uH3ILhvEb6RfcSSb+QtbRmF+VvJ/FpiwjPzCHTC6BW5+m2yL6iYPDcetR+UbE4l96TJ+6BZy7GSMuqaARo5V1KMZvNWTqqCiZofzCMntpOccQhd96EIBinCN7KPWPKNBOCnicQkxuDSdttqdSVqZSJRjzujGgbgSfzVMpKPnSVukreFkvWowtYQFzYCAN/xEheTSqnSTWOKMG4dUgjfyLvFRoXSvH8wTIHK1QHpWi1VF65S999nqazUI43v3tVn1MgRHR8cHHChlFvNgBDbkEL4Rg40hjJy386iaJgrIU864+IRzFTOc/ymtQMTWBvhGznAGEoKKBytZmNSZHvDVLcr16oxCe4PhG9kPzH3jbSEcrQKxc1Gmtp6jY3F5JX01/dLMBSxQsvmx8IV96DY6r18uuFyP3wjq/tZgQMhc4PIz0zgxT1G30iLQ6rJ0Sw6uIGUlw+jHAmGhyYT85QDR8T04gPPEFhBcj/4RlqgpRmD9DDKkWJPUYGRISE2gWAwMCQWIgsEgwEhNoFAJoTYBAKZEGITCGRCiE0gkAkhNoFAJoTYBAKZEGITCGRCiE0gkAkhNoFAJoTYBAKZsPKb2gNDtwuRw1Mp2jIfZ6u/ymPc4B6lLQqxLvmBZQgsRJbLN/JuKPu/7Z1/XFRV/v+fD20cnQR0HVqEwGXEh1AIojJGINoHaoVWCzGZStgytfLXFuYSroW2fuDjx4+YifsxSF2hXXGLKCxwk/lsokaLJoIk+BUhJYhiXBlGhx0nfXz/mAEGBBnUZhDP8/Hg8WDOvfec97lzX+ece+6Z1yVrYTosETZ4dzN26Nlut29kb8XtI620+01p//V+H3wjBYJbQ/hGWukbWb5nJ81TFxCq6EjTHsshT+OLauZ4JNdaOKfOR338NOcYzZSgCMJC3RH2j4I2hG+klciMFWQfrLBIqaP4w0IuOymQ0MJXGxNY/w89XjNVxM50p/6jZFLzG+0Wr6D/YdfZyDtJcF6PBDOi+Esq216IUVdC0b98CZwkAYYx+ZlVrFuzgFD/8Xj5R/DCU75UHy0VZquCdoRvpLW4Kwn9RTLHToDPJGj45wmag2KYLAWQIHFX4HypjuqSRhrOn6S66iwYFVyxd9yCfoMde7Y7zc7OnckhLhQdOYqROsqPXSZ0itnh+Foj6vUrWJKwk8KTJ9E6TSB0kph1EXRG+Eb2AefgYLzySig/5YJaP4kX22zIqwrJ+VFJwtb5eJmbL+ORErvFKeifCN/IvjBSyfSxFWSmH0ESOh2vtvRfjML131o0OvNnfQ37Cyt6yERwtyJ8I/uEI5NDfdmxo5GIqRYPGVxmoAotYcPKl8kcLsFocEH16wA4dhNFCAYsA2AFST/yjTS2oteDzGlY7/sK7joGhNgEgjsBsepfILARQmwCgY0QYhMIbIQQm0BgI4TYBAIbIcQmENgIITaBwEYIsQkENkKITSCwEUJsAoGNEGITCGyEXX+pbVe/R6OG8k/yKNPK8J+lwu++n7Gs24G+Bf1gR2S34JCkP3OU6qG++Ln3j4XSxkstGK92SRx6fR2Nl1owSrqpu17DuTMXkIxV4Dr8Jgw5rxnR61q7JEqQOAxDMqjrfj9dn44R4491VGskjBnr3ut3Y8eFyLbyezTSdPIE+vsDGTOyLa2Fr/4ngT2E87RSgcfUAFxt4qt381Ruf5mc+5NY8xuXm8zBVOcdqNjy2vR+4PpVw75VG8m52DnV60mLOuprUG9/l+xTLYyxTKeVyh1rST1mZMRwCVcutTBknIrfvzId576M1WpyWJVcSFOnRBdi1ycR1hbCmQJ2bM/ja23ndPSlZL2RTtFVR0ZIjDRrJfjFvMrysJ6/n/7riHzb/B4vUL53Jw1zAoltF5uey80wZU40D901pqmOPPTa//KQvcNoR0vzxS4XsCXf5pG0oRCCI5ipyaPSYpOxeCepJ8cSv2kxPjLgWiP731xH1j8CiA+zaISv0flGqevn5gs0ucxm4/oInLsJoeHjdST9HWbOmo7mo9OWEfD1jnSOjVvMlpcCTA1XfR7rk7L4asoqHnLqvsb2F9vP7Pf41Y4ciltAf2AnWd8qiQ6Dog8PcqwtrXQsYS9Mx1Vbwf4PL+DzhIzyvxygaNRsNs73NQ83P6LwxPfg6kv43Nn43Wc5ZDHSVJzHvqIKk19k6BxmBcktNt/o+Dq+2lGK00xfmvbnoNYqiY+fjhNGmo4Xoj5QQiXuhM15+rqLQf/tEdSfHKSoSYLPxBlEzwnEqZdWveGLnah5lNgZ7mCur9/T7lTvyUFdC2MmzmDWE4E4W1ZPe5qifYUUl2lx8lcSMSucMZYX049H2ffhFxxrAJ/p0cweV0fO8VFEz/G9ce/Z2Gg6Xz11BEYnwlZvItT9Auo1eZ03ydwJfTLAJDSAQS5MfsiF7DNnISzAVNcPE9igmUvK4kBkgzD5xCQnUzb1TeIfNX0/TQ3fw/2B3QoNwDgimHXvhON6oYDKjyy3tCJzC+TpSQEddXRTEuRSwJmz9NiA9y62llPs3f13vvupu41S7g+fT8wEW1h43xyjJ4zHtaqCJo8J+I+TIxkKYyaMp6Etbewo7gVoreNY8ZcUfT+MyVMfJXasBxgqyF6dTuUDs4meHYD+/BGykt4ibNUfmakAMFK5aw2pFaOJfiKSKUM1HNv7Fkn1r7BuruK649GeJGdNAmVLNxDrLwE0nCk+QuXZUrweDidaORYZ0PTpRlYXyoieE0n0UC1luzdTBHC/uVKn3ifx7dNMeWo+r3rrKd+bxZvp95DyUsANL3Bt1VHUBJjE1lrHseISjn0PY6ZGEj3hCg1Fe1idrCH5jQjTcKyxkA3r8pGFRRPx1BCav8ln/euneWnTUibLgPoC1q87gNNj0URPGULzySy2fgmVRiUzexPbvy7QMNIRfXEOWae04DSesEeDcW0T8rjphPZwqMx/NrGdUlqoPNGIX1BH6+s651VmJyeTmA4pL7hTvCGZPHksKWEdDaH2X1qch+sp/3AnZVoYMe5RwkLcTeIExswI7yECR3zmLOicdLGCY40Kpt+gA+hdbI4P8OSML3jm6c0c11tuMJn2LHqg/woNYIzSlzF5OUjGB+JnNujxUUJDW1qnVugKk2P+SPQ40yftgTzUHjFseSHYdOEoA/FxSua1z44StjwQycWD5B1xInbjK4Sah6h+HnJyCr6nAQWSAznsHzmbLS+EY7qGAvGSbeN3e/II84/GFYAWRs9cy4JQ86SF4Sg5ea3Mfn21WdDg5zuM7a+8j8YcZVPNWbQTZxP76HgAXJe7M1nndBP3YY3IH0klNthUtt+U0RhXbyTnn+G8FCShuiAf7YxlJMw1B6IMxJV4dubXMXmuC19/lIc+bBVr2rdP5N6tK6j8ofeSjforcOkkB0uVTJniQVNhHkmfH0SVtJowt77VounTNLI0wSQ8YnEtDnIhbPVqSE7mdytA5h/b0cuZIkD/759oPv0lx4ICmOKhofCTZPL+MbujsbGWa42o38mhIfSVHoeQYOUwUjphCX/dg4Xg7kx3rN6RMcKh7X8j1afqcB3+E9UlRzt20Ugwnj9PE4E4nzpN5bhglo+0yMIlkOjnTcd/fbYRr0kTsTz/Ml9ffHaVcF5Leys+YrjF7GD9ec4OHU+4wvIgJYEPvk+B+aOzYixOH2eRtF1DRKgvfmPdcW5vsE0v8VBbHN5p0qET7owbZ1H2IAWTAxwpOFMDQRIqT/3E6KALlJdcaN+luVVCQ8t5jBhpOD8MvzDLQCX4TfKlPdAbxCKZsoA/TbHYoAzHPzOeDR8dJXR5INbOLeqPp7M+D1RJHc5m1iHBb/E7nSbo/MICyI5/u72xsY5WKtP/m2xmkzx//A33tPqerUNwf8Jp1UAUWle0NDeD8d9nKTtpmT6KMG/Tla3XamHoPT1cGKbjXeXyzskOMkag53Ir0F0rqNPSLO3aS0k6v2rqgflsSQ6kqOggX/71ADsaf8JrzmoSIl0AX1SbNhBtefjQnkYfTozoEoPkXhn6i3pAQrP2HrT1JymznDGUjifMxQkjepq1EkYM7XJ8p0D7Egt4+Xpj/PNpzhHY4Vx2I+rySM3QEL7q1et7Q/M9Wp48li0JpmFkYjpdercuDBqPvzdsPVMDQTcWjgkjDR9vJvWHYNas6r037NMEiXTCEnJK5mOQOvZ/B+NbRo7r/WAcPoPYGEX3u/zKA9n+Ws5dC+5oVc3PZGROcsZ4DiOn6jQLlBZf3LfnqRzkTlBPEwOjRzPmYkWnng9qOFcLeFiGN57QueMJnQucyWbVhiy+Cl7FQ04SJE4SK3uG8zTUw+T26hk5f7YRr3Gm6SpXt1YuT4olNri73FoYN7aFg2dbmKXoEFD1N1XQPt3VcyxNJdns1web7h/bSm81guOobtug66gvYP0fv2DMS//JrHHXPzds+Giz6R7NLK62IeV2tad5gkRD+Z489EELeOhXHfXXG8D1fqsioOnTjSR9ISc+OZoxVozh+76CRHZnCq35Yl+9t8DnkXAk6mz21xlNCddaKN8ez5L0CowA3g8TITtCdl6d+YhWzu1NZsnWI2gBr5BgRhzO6TjeWMf+rEKuhATj01Oh9wUTPq6GPXuOor8GptYzm7z2lwYYKd++gt+lt20HvaaFy4OccHLoPsueaSEvK48Gc3j6kl1kfqMgPEgOyAkK9+XrD3ZR3la2sY7961awPr8RcGTyzEAaPkgj53gdem0LDUXbyCqzruQR97RQ/Ncs9n9rLvziUbI/qMDr4YAeZwfb0R4la+MBXJ9fS+yk7h/Qu87dwBbLXmyQC2Gr32mfiQQnJNoKduzOo8EAYERbvIvMb9wJ6nGKtAN9STrrP5ezfN3ijlnRXrD/1P/P7vfoQtCcQPZvT+C5T4JZ885864YoAIpoEha+T2ryCnIGScBgROI2nfgEX1NrPUjBrNfm07RhI8/lgwQjOAWwPME8IWJ5PBIwgvOU+bx1w7G9Iw8tX0p9cjpLXspCNtjIvVMW88KkdAoBkOAXt5jQ5HSWLN2D86BWmq65MGv57/Hpc9PpwszJGjYvXcFliRG9OZ+HzPegsuAFrLn4LmmrXqZZIgEDOE+I5tWZpotR4r+AlKU5bP8gjSQDyB+YS/wzEn6Xd4MizUgmPc+ap7axOXkF2deAQRK8QpcSH9n7hV65ZyfqS8COBIp2WFanyzOzruej02cJPi+8SuzWNJKWFmAEJFIXZi7/fffP/TpRSk666aUpqStf7rSl5/tjO1vZ9Ru/x14xvabX2M1Sonb0LegZhkzW3aDpJl/ze8M8zduvDUNm5VKlyu0vs1X2Cn+KGw+NBaxfU0LQ+iTC7msb+vawjMu8rEnSXfxdHhQbj2xjUaGCLUkR1g0HMWLU/gTXLYWyIcZW9IZ7rD6PN4tdezavZzMpe9aeEViLFfdBMscbTL2bju8zN8zTiu0WGDWnOfUdeD0y+vqNgyTIbhRfD9u1Bzby2pcKElbMxmukBKOmlJx9FbhOmW2l0OCmz83tRDKMntqz24n9h5ECG6ChOHMn1WMX8FLbs6hBMpxGOt1Sb+IU9jzLf9jFztUraDCCRCrH55GlrJlj7XqguwvhiCwQ2AjxezaBwEYIsQkENkKITSCwEUJsAoGNEGITCGyEEJtAYCOE2AQCGyHEJhDYCCE2gcBGDIjlWnb1nxQIrGQALNeyp//kLXKxhvLvZPhMcLHaBkBw52KHnq2UzK0QszwAm/ii/qz+k7dIbSGpH7mzcUL3voWCgYVdhpHfbZ7HM3zCX5c/cPsEZ2v/yTartt58JbV1fHUgj4P/1CDx9CV8XjR+ctCX5ZFzoA5aNOzb8T0PhFn+PF8wELHbBMnxzU/wzNZTGOwVQB8ZPWE8rlJwavOfBJMv5OtvkaNVED47kofvqyMr6S3215gPunaa7KRkCrTjiV6xgIj76tixPp1KPUjkCvw9HEEqx2vCBEZb/wMwwR2KXSdIjm9+4vb3cD8T3flPaot68ZX8sYbqSwFEPx9uMgRyX8pbQVpkQ0Hi5ovf+C/hG3d8lD278goGDnaejZTi5AiGq9yBJkK9+0q63qfAa3geW99MR/XYdHwmKXB1k/ecpWBAY0ex3elGr737SjJoPKqNf2TKFwcpKs4mb08jerfZJK/uo+OuYEBgJ7Hd6UIDq3wlAQbL8Xo0Gq9Ho+FaDTkJG8n6R3Dnt60I7grs0r7e/2rGHSs0S//J3nwljcfSWRSf3uG7qLuA1gDOIy2EZtCiN9osfIEdGTgPtZ2DiFMF9MF/Mo9M9Xcs+ssZVgdZd4z++E6Sth+laWiH/6S25H1Sdx2hoZOvpAovGUAr1e//Nxu+uIDMCfRacJ2xmIT55scGZovsrG+N+D29QfR2A5wBILb+4D/Zi6+ksRW9nm5eEyu4mxgQYhMI7gREOysQ2AghNoHARgixCQQ2QohNILARQmwCgY0QYhMIbIQQm0BgI4TYBAIbIcQmENgIITaBwEYIsQkENmJAiK36L3H4K8bh2fVvwQfUXbV3dF24pqGyqALtNXsHIrA1A2Ah8h3mG3kmm99tKCE0IZXocbeQj+COww5iu92+kb2I7YY80AffyEbUa9bRMOd/iZ10U4UJ7nKEb+Qt+EZSlkfORV9my0rZ83kJzk9sIHoCZi/JPL6sOo3WIYApM8IJ8zf7kmgr2P9hDWPmzsbHqY6vdpTiNOdhjOq9FPxTi9PU6cyODMZVdpPnQdBvEb6RVtKdb+TlulLU+3ex/XMjDzw2lyn3Y/KKXPUGO751YuKcBUQHQPG2ZLLKzN4HrXUcKy6loRVAw5niI+Rs20WZk5KIp5Tce/x9krYeRNtzKII7FOEbaSXd+UYCcPVBVKtVjGlrtq4piFixmohfueM0CGA8C74vJenYaWL9fbvJuQWn0LXEhg4DwM/DyLk1JVRqp/OQMG4dUAjfyFtF6oTMcnwwSIKTwgXjjzVUfvs9Df/vNKdOtWD06NnVZ8TwYR0fXFwYQwmXWwEhtgGF8I283ehLyXojnaJBCoK83RkzTsnD1PL1JXsHJrA3wjfyNqMvLkQ9Mpota8LbO6aGvVl2jUnQPxC+kX3E0jeyO2QjnZBc0tDcNmrUHCGvuK++X4KBiB16tgDilv8M2Vbv473N3/XBN7K6jwW4EDQnkP3bE3juE5NvZLe3VBNn88IXm1n/8pfIhoN+8ERUU134Skwv3vUMgBUk/cE3shsMreiN9yAbLt4pKjAxIMQmENwJDIiFyALBnYAQm0BgI4TYBAIbIcQmENgIITaBwEYIsQkENkKITSCwEUJsAoGNEGITCGyEEJtAYCOE2AQCG2HnX2rfHnpciDwjmaKMp3C3+095TC+4R+aIRKxLvmsZAAuRbeUbeSuUkrUwHZYIG7y7GTv0bLfbN7K34vaRVtr9prT/er8PvpECwa0hfCOt9I0s37OT5qkLCFV0pGmP5ZCn8UU1czySay2cU+ejPn6ac4xmSlAEYaHuCPtHQRvCN9JKZMYKsg9WWKTUUfxhIZedFEho4auNCaz/hx6vmSpiZ7pT/1EyqfmNdotX0P+w62zknSQ4r0eCGVH8JZVtL8SoK6HoX74ETpIAw5j8zCrWrVlAqP94vPwjeOEpX6qPlgqzVUE7wjfSWtyVhP4imWMnwGcSNPzzBM1BMUyWAkiQuCtwvlRHdUkjDedPUl11FowKrtg7bkG/wY49251mZ+fO5BAXio4cxUgd5ccuEzrF7HB8rRH1+hUsSdhJ4cmTaJ0mEDpJzLoIOiN8I/uAc3AwXnkllJ9yQa2fxIttNuRVheT8qCRh63y8zM2X8UiJ3eIU9E+Eb2RfGKlk+tgKMtOPIAmdjldb+i9G4fpvLRqd+bO+hv2FFT1kIrhbEb6RfcKRyaG+7NjRSMRUi4cMLjNQhZawYeXLZA6XYDS4oPp1ABy7iSIEA5YBsIKkH/lGGlvR60HmNKz3fQV3HQNCbALBnYBY9S8Q2AghNoHARgixCQQ2QohNILARQmwCgY0QYhMIbIQQm0BgI65bQdJy6bI94hAIBjyiZxMIbIQQm0BgI4TYBAIbYcdfateS+UQUm2pv5lg3lv3tMxZ53+6YbIBBh+7KEBwcbOIt1nMYzTpwcLh9v5DXlHOowZVpfvL+kU8/xI49m46mWuDBOFLeTWflYwDexG1IJ2NDHN6Ad2wKGe+mEPcgQBgr300nJdYbqKelr0v8+wmagkRCEvLR2DWKctJCE8m/jUHUfJrEsjfyqekn+fRH7O+IfK8n/kFKNMUAjij8lSgH1+MMGBR+KIOk1GcBuOGtVKL4YeC1eAMBxXO5lD3Xf/Lpj9hfbDZAV3uYz/6SS27dEJRhUfw2Wom8bfh0VUNJTjbq4sM0jYpk2uxIotqGMLpysjObUMa5UZWRwe46OVFPL0I1Gcr2ZZNdUItjWAyL2vOrR73tMPJnwzB8nkF2rgbnX4cTFR2Bt0PP8WmK97L7UzVVeBP2mzhUQdY1KJrSXLI/LOCQ3pPHH1MRE+GJFNCV7mX3GR9+O8+PtmLr1Wkc+sVvUQVYBHJVQ8nfuo/TtL+K8H/ns+nPh8EvikXPR+CmOUxuZi7qVk+iYn9LpPmA68rU1XKoIJvc3Fqkk8KIej4GpfzmYr/R+WmrV7gxn4w9uTSNiiIq7kmmedxgmK6rIj8nl8K/d/P9XNWYv9vDGNwjCVfFEOklBUMV+e+V4fZsDP4jOrKq+TyNww4q4qz4zgb+BEnVe8Q9nYFm8iJS//A0bhUpLNpSbrLPu1pFxpxwUiqcCVm4lsXTrqBevIi0crO5nv4sh9/NJTU1H0NQDCunw96FK4lfvYlDUiVRz4Xj8Nly4try4wIn3s0l443V5DMN1YoY/C9nEzc/lTJ9d8EZKHtXxZPbm/D5zUIWPeZG1eYoFv2t9xtZ3eeJRK4tR/6bRFJXTKMlO46VH9ebcj13iIwvznayCNSUvcfhc51SUKf0HKem7D1yM1JIO60g6rk4FFVrUSWsJGVXPW5hMcT5XyBjXiK5P3B9mVeryJivIqNpIos3JKEaU0XKCx15Wx97l/PzGx802x8nalt5e900Ze+Ru3Mtm446EjYvnvCR+ayctxZ1cw8n7ocC4h9fRK5hIqoVMYRQyMLHUyjRA+hQr36ctcediXp9E8umtZC9YCW5DYDUDen5FHYX6zryMpSw982jDHGzrnEc8D2bpuoENdFLyY0wzaao/pDBNJ2D2YnZk6i3c4ny8DT3TN7EryhEdeg0y/z82nIgJC6NKE8AJYlnJrJ2ZCKp5vyU954lM6WSevwwmSVX0RKUS9o8T9PhQf44XAxl06cxZM5z6xxc8/+xe5c/Kf+3jGkyU/5Kf0fiZ2ZT8kQiyhs0zjUVBfjHfYYqyA3wZNmWXOqH9GWI3XucVa5RZD4XghRQjnmdkpklTDsSwzQHIEiB7lA4ZWcM8MsugWoqKauNYdlLEXgPBjxeJyNER9uckNWx/5BPxjY3VhctI9Lcmyj9HdD9xyZyZ+1G5WGuycgoMpea4iRIgeGbcApLVhP22PXDiaqP06h6NoP8F73b6/3eqAwqq3XgV8uJAn/i9segdAU8l7Htg3pMoTkQ8mQMiVmFaCKikAOG42pyvSLJ9bDujA/4nk3uPRFF1koWpeZy6Jt6dMhxk5u/9cFS5J6eOGhqKSs+TO6f08j8ey2GTq6xcqQWHuLSwaCQW1wYv3RD+U09He2dFKWvp8XxUpQznqTs6yqLfUwYKk6g9nfgSlkJJcXmv8om0JdR03Djeil8IyjbuJLEPxdQVqvB4CDHrU9e525Mm9Q5zonK/+gcp8yxwx5+MEhxRt5+/cpx84R6TddaAXIf/D0ziV+SSu7BKup1IHeVt898Whu74UwVh6aHo7QYtiHzRzm9jLIqi3JHWMSJHIUX6PTdWf/WcuILiJpuOY0txXvWMqL8HABPJkaUkfJKIpkF5dT8YMDhl27tcUsnhRF1XM2hHwAMlB38mJBnI3G7vqBuGfBiw3shuQcyiLq/lvx3lhEROI34faYhC/py0uZNJWxlOoXHa8AzlLBH/G+xQE+LC9KE9F4p6Fquc37WXayHy/VUHi+hpP2vBcWLITj3MiXv8FgK6j3xKDnB7gQV04JVpJX2xVvaDXkXdySHEc7dxtlnBnuz6KNC3nvSjZrPU1n2+FRCXiug/mrfYtddrAePUXTu8xxwkPckpt7Qoal1oOenLg6EJReS/aoSvt5N4vxQlPPSOm4BpEoi5x0l92A9GMpQfxxCeNANbsa7MOCHkQDIvYmc503kvHioziRmThrqGSkoi/eQ4bGWw/8T0X4zXlN/q88UqqitN4BnxzdaU1mC9MFw5NBpyl8+xhvpVWcil8aguC6f3nHwUBL1nJKo5xIxHFyHMjGL8P0L8ZRf39YarnZNKaGqxgCuFnFWd8RZfxPxdEaOd0QM3hExrLxaS+b8KNLUIaSYh3Y9xW4pLPm4ibhtK6dqlZKOvqiWmuNSvH99M7PSnvhMr+KzKh0qTwuR6HXoBreJ0AFFUBSKoCji1hg49OZUEv8aTv5CUwT+v16EJuUwVa41ZE+PZNmIbgvqlgHfs5WlTiNqSwk688Wma9Kgkzkjl4GDszPSxiaa2i5EzWH27q265TKzN26jpG2Ucz6X1HcMLArzu37HBx8nfnAqqTkdl7auJJWoR9Mou2HDXU/ui1NZ9nHHcZofmkBh6gWkHgr8D6vJr9LBVQP1B1PZlNM1Dym571gZZx8xlKYSMie1I299E00XpcidHXqNvRPjwoiTZ5BmcX7qczaRql9E+INWxvLNXjZtO2zuVR0ImRXD4e0ZHbGfdyEDAAAB7UlEQVRpDpMyR0V2NdCQy6LgZeSeN2+7qqFe0+W24cEQojTZJG78GNWTIVjfr/WHnq2mgMxt9bQcB6ghf2ca9fdUUQMYPs8krRGqagBKyN2WhuOZvi058X8pg6hXFxHyiBxvaQNVBn9W7tiK/2DAL4aUcStRBWSY7j+kMSTODYGmW6tSXJwP2Y9PJVE6BI3Ojai30rpf7TLYE9XbW9H9QYX/RinyIRp00hCWvb0J/xsuMHEjKimZslfmoNzuiZuhinqXKFLfNt244/Ekq+PVLJw3jU1I8X4xg2XzMtnbKY9AEl+zMs4+Ig1YyntRy1kYPA1nbyn1VQb843eTFtB77J2es3c6P+CADrzi2LRjoWnixQo03+STmaVgYlwIbg4gDYknu2Etix6diGGIA7orUiLjM4h/ECCKtW+VEz9vKmkerhiqGnCL3kTqLMtmwJvIBY6kpSpJnNS3VUDXWdnZ7ic2t7Jcy5Nlf8vt24Wh16G72sMyqdu2hKqcTX5xsPsEKwP6uCTquvhMeWV2s2ucOX9T6DquDHbA4RZeBHfbl261cdWATneFIT3k3afY9Tp03Fo9rY6tl7hvFjuKbSDSWWwCgSX2H0YOMJwf9IY77R0GApsgejaBwEYM+NlIgaC/ILz+BQIbIXo2gcBG/H+bAzOGSv1q2QAAAABJRU5ErkJggg=="}}},{"cell_type":"markdown","source":"# Evaluation\n\n\n\nIn \"macro\" F1 a separate F1 score is calculated for each class / label and then averaged.","metadata":{},"attachments":{"b8ca442e-0f5a-49d0-a81d-889f52104e23.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAi4AAADqCAYAAACFrM5UAAAgAElEQVR4nOzde1hVVf748TcIyG1QBEVNMDkqGMrFQANUUMPRsOmLWpqWOZNlTPObshzH1HHqa36d0rJxGnNSa9RpDPHWt7yEjhdKrYSBuHzVuClXj8hw8RyEw4H1+wM9eeIiXhCPfl7P4/PE56zP2muv8zzxYe2197ZSSimEEEIIISyAdUcPQAghhBCiraRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWIy2FS71BvR6fTP/DE2aGvR6mgn/+Pn5NNIKW2nQXgzNj/cGO0Ov13Pbz6Km9bm91crT9rNzx052p5W3qb1Br8dQf/mHGj36mvYbmxBCiHtT2wqX7F1Mmzat6b/3j2P6lVZTwJH1i5g9bRqrv2npF52etF0rWbknDf2tGP11KP9mtfl4b8aF46yetprjF9rYvqqAtKzymy50Tu1qbW5vLX3yWl7alINzX0803Z3akHGKXdOmsSv78k+7pjFt16l2HaMQQoh7j03bmz7Nis+fwLe5j0qOsPyPm+HRXzE1NI20FvtwIvj5TWy67mFauJLjLNrlwabfR2DX0WNpI4O+nPLgGKIebPYbF0IIITrEdRQurTA4EfHmesJ6lHPk/1pvqv1uJ8cII2aYBwD6kiSO7TvG4fN2+D4YyaSHfWnt7/vytP18cTSJQvoQHD6RKH/Xqz9k/6EkTqbocB4WQGREBN4u19+X/vR+Ego8GXfVWLTf7SStyziifJobnZ6C44c5nJrW2NewcUQ86IHdlb4OnoTifL7Y8R8GhcYQ3Oty1tljHD6YxLEqZ8KGRREZ6ml27uWnj3D44DFO4UvEo9F0a31qob6ctEOHScpMQucSRsCISCIG/Nhj2+Zaz6kDCZxIOwdVh9m54ySeV8ZcX07aoS84llwIfYIJeySKq6f/2mNrPrc8bTfHCCP6SkB/iv1f5uAxKhp/9x/bHK4LJuZBD6gv59TRwxw+noShRxjBUVGE9blSEuopOK3Dw8fDYopEIYQQbXdrNuf2DSasR9ualud+xEe5ly935O7k9d/vptx3Ir/9ZSQ9czfw+qenWrikYiB3xyJ+F1dAz2HRRId7cS7udyw/oDX19buZm8lx8WfiLyfizwle/++t5DbbWet9GbRJfJSsNRtHee5HJGmb60xP0gexLDqgRzNmKlPHeHFuy+9Y/XXjOdq6adD0dQUXDzT9NXhcrhTKj6/mpb+kYecfydTRvhgS3+ClTWmmY2oTV/LSupM4D4smOtyDwk/W8sXZ1mY2l52/n8nmXGf8H32Wif5wYunrbM0yXOdc2+LRV4PGwxncPX8csyGXnUt+x85yDWETognzLmfnS7/7sf/WXCPXtbOeuL1ppst4hqxjrP54Lbv/70qknLS9aynv5No43x++xIZTzoTN+C0x/nr2//dqjpy//G0c38Cv5y1id9a1hyWEEMLyXMeKy0kO79jJyasinletHtyI8sJTnPr5VFaEegMQNet1gmtsm/9LuSqJLz724Fef/Iqwy6so/r1d2X20AC0eePQZx28/HIdnr8bKwNv7aZ7+7ncknX0C7wHX19f1/aXuhP/kZaxw8cTDHsCbPk/lMPngSZ4dEYaruzf+mp6Q7sEgf38a1xRyOfxpOVMX/JGoy/Pn798TXtrMkSh/otzSSFiv5+m35v34+UBnPnp+P9rQlsbRh3Evr2dcH4/LqyjePD3jGL9LLeSJAd7XMdd2uA7wZ1DJbuikwd+/8VJReWIcH/X4FZ8+Hna5f398HXTM/uQIEa9H4dHKDJV/c43cfr5EZe4m7UIEEe6Qe+oYT8x4gv2ZOehHBeOkz+Hk1zGEzbUDcsndM4iJG6Lw7wH0eYJ5b2mxvfw9OoU+y5qVOrr99DsXQghxV7iOwsUVz/4aPK+OtGXPZms99vHFd8VKlnd6mqhhvgzq44FrC5d2DLmn2P9IGM9e/bm7P9GPXf5vOyc8exgoLzxFgVZLwdkcks6WM6jh+vu63u2vdj08cdVryU3Tcu58DjmZJ6FO03JCSQ5JBk8iS9NIK70S1KFzSaJcawC0nHL1J/zqotDel+DRsLvlUeDUxwNDeQGnCrRozxaQk1pA+YDLqxrXMddNGSjIOUbEgy+YXVqyG+BPWHISBfoo00rSjeX6EjB6Ecdy9US4azl5PIyApQHwmyRyng3GNyuN3Y/487QdgCfej5xk9Z9Xo5s4Dn9fbzxdry6bnPBs9nKeEEKIu8F1FC490fj7N78590Z5x7BiUwDHko+SFP8Fa78ux//3y/jtiKZ/v+t1/wF7O2xb6MpweitvLEvALjQC/z5eePoFE3xqZ7N3L12rr+uj5difF7E2y5fIMRo8e2kID9Kx9XgrKbpyCuz+w7nsHHRXhZ2DfoWzQ13j5y6uP9l/Yodda0tBNafYuvR/SLANI8LfE6++gwjmFDuvTMB1zHVTenTl4NHtJxtanJxxohz9JWh5Y1Jbcu3Q+EWzKDOHX7nncNjXn3EuvjB6AzmFYHfqMFGBUy8fwong2A9YkZZEUnICaz9JoqDLRBb+4Ql87dtwKkIIISzardmcezNcvQl72Juwh4EZu1kUu5OkoBcI/skvQtfeGrzjcyjEH29T1IBeD05OdaQd3IxH7Kf8NtS0nZb9l2i2cGm9LzucXJr5ZV7fNARA7jHicqN5/f0YU1+G1KTWz9m9D/65OgJWxODfXDFS3gf/1LSfrGRoKTgFZkteV9GnHWZzjxf49KWwHzcUX9CZT0Ab57opV3pqvInLyuXpB36cMUoKOOWqIdz95nOd/IKJfv8URzy+p6dfJE7YofEbRFzWETg+iICHrx6kEx7+EUT7RxD9SwNJH07mgy+D+fNj3gghhLi7deiTc099OpvffXrK9LtV/5//oHPthnNzv0i9w5joupMdhy5vxsVAwZ53mLY+CT1OOLu6Ul7+H1NzfWoCCaktHLjVvsDOw5Pgr49x7KweMFCevJnNX7bQl0s3epaX858rJ1FTwJGDh5u20+nQXSl+XIMZN+kwm3f8eO6cP8LK51dy5ALgGkDkI8eI+/zHz7WH4ohrZXOuU5duuJaX858rx6hKI2H/jzemX9dcN8N72ES6bd9h2gRLvZYjWzZjmBzMtcqFNuU6aRjk9wWbP4XgAY2rM07eg3D+ZAM7XYPxvVIcXTjC8pkrr+qrnPJy6Ol6eUXnfBo7P9lPrjz8Tggh7koduuLi+18Lifzz/zDteVf87bSkGQbxwuuvtnA5yoOoV+ZR/u4iHv0YPNCC99Mse6VxhcF3zG/x/PMiHt3lgS9a7KKeJXJE8ysu1+qLXhE8/XwS//ObaazFFf+ZC4l5ZCv7m+vKPZSpjx/jf6bNxM7HGV2VLy88GQXfXdXGJ4LX3Bfx6/9aS/QfPuWFYU74P7mMif9YS+yjWpz7GigwaIiZ/QIR7gBOBP/ydbR//h+mzbTD016Ha9RCXnhkf/NjuHyM3/ZZzaL/2omHD2g7j+PZMWFQdSNz3YxeUcxboGP14kfZgAeUgGbGPF5vyypHm3Jd8Q8dRHmxL5ore3vcfQnuW44u1P/Hzb/uEfzqNzmsnjeTOE8PDGlaPB77I78d1Vi4GM59z85P07AbEYV337aenBBCCEthpZRSHT0IavTo621xcmrj/Tw1evQ44dTMngaDXk+drRNt7aq1vq5LvQF9DW0/hysMevR1rZz79Y6vLf1dz1w3ewg92Dth1+n25v6kJ/T6OmxvSV9CCCEsxZ1RuAghhBBCtIG8HVoIIYQQFkMKFyGEEEJYDClchBBCCGExpHARQgghhMWQwkUIIYQQFkMKFyGEEEJYDClchBBCCGExpHARQgghhMWQwkUIIYQQFkMKFyGEEEJYDClchBBCCGExpHARQgghhMWQwkUIIYQQFkMKFyGEEEJYDClchBBCCGExpHARQgghhMWQwkUIIYQQFkMKFyGEEEJYDClchBBCCGExpHARQghxV0lLS8Pd3Z0//vGPHZJ/r6mtrWXevHlMnjwZjUZzzfjNsrllPQkhhBB3gNTUVIxGIz4+Ph2Sf6+xtbXlxRdfZODAgcTExFwzfrOkcBFCCHFXmTlzJjNnzuyw/HuNtbU1Wq0Wo9FIRETENeM3fbxb1pMQQggh7knHjx/Hzs6OoKCgNsVvhhQuQggh2lVdXR3Lly8nICCAmJgYEhMTmTJlCj4+PixcuNDUrqqqio0bNzJ+/HiCg4MJDw/nm2++MeurtraW3bt3M3HiRAICAggKCiI+Ph6lFOnp6fj4+NC9e3f27duHUsos12g08tlnnzF69GgCAwOJiIjgxIkTAG3K1+l0fP7550yYMIHhw4czePBg3n33XS5duoTBYOBPf/oTgwcPZsKECWi1Wt5//33CwsIYOHAgjzzyCMXFxRY1p1eUlpbyyiuv8NBDDxESEsKHH37I4sWL0ev1pjZarRZnZ2f69etn1ndL8ZuihBBCiBu0ePFiNWLECDVnzhx18OBBNWnSJBUeHq6mTJmizp8/r5RSKj09XS1ZskR9/fXXqnPnzmrUqFEqIyNDhYaGKjc3N6WUUhcvXlQxMTFq2LBhKiUlRel0OrVjxw715JNPmh1v4cKFysfHRyUkJKjS0lIVEhKievXqpfLy8tTIkSNVcnKyioiIUEOHDlWVlZVmuSkpKcrLy0sdP35clZeXq/Hjx6sZM2YopVSb8mNjY5Wbm5s6fvy4qq2tVceOHVOOjo4qPj5epaenqxdffFF9//33ytnZWQ0fPlytXbtWabVadeDAAdWlSxc1d+7cNs3p8OHDVUZGRqttbsecKqVUTU2NioqKUrNnz1bnzp1TWVlZ6v7771eenp7q3Llzpj48PT2Vl5eXqq+vN+u7pfjNkD0uQgghbsjJkyextrbmgw8+YMiQIRQVFbFp0yYcHBzw8fFh9erVLF26lJSUFMaPH09FRQUGg4G5c+ei0WjIy8sjJiaGhoYGFixYQEJCAvv37ycwMJCGhgYcHBxwdnYGGldENm3axNtvv822bduIioqipqaG7OxswsPD2bVrF/PmzaN79+4UFhbSo0cPOnfubDbeHTt28LOf/Qw/Pz+cnJw4ffo0c+bMAWg1XynFmjVrWL9+vekzgNDQUJydnYmPj2fUqFFMmDABOzs7qqurCQwMNPXt4uKCtbU1Fy5caHEu8/Pz6dGjB/b29pw+fZqamhrq6uqorKzE3d29SfvbMadKKebPn096ejo7d+7EyckJV1dXPDw8MBgMpnnQ6XQUFBSwatUqrK1/vJDTUvxmSeEihBDihqxbt47Ro0eTk5MDwMcff4yrqyvQeEfJ3r17Wbp0KaNGjcLNzY2lS5fi6OjI+PHjsbe35/vvv6dLly4UFhayd+9eXF1dOX36NBkZGWRmZpKbm8vSpUsBKCsrY9WqVWg0GoYNGwaAvb09iYmJ3HfffeTk5ODl5UV6ejp5eXksXry4SeFibW1NZmYm0dHRBAUF8d577/Hzn/8cgBEjRrSYf/78ed5++22GDx9O165dzfqsqqri3LlzjBo1ip49e7J582ZsbGxYvny5qc2ZM2eorq7G39+/2Xk0GAy88MIL/PDDD7z33nv069ePzMxM5s+fj1arJSMjo0nO7ZjTgoIC4uPjGTZsGPb29kDj5bbCwkKeeuoprKysgMYiCiA6OtpsjC3Fb9otW7sRQghxT/Ly8lKOjo6mn+vr61WnTp3UkCFDzNqFhISoX/ziF03yjx07pjp37qwGDx6sXn/9dbV8+XJ15MgRszbffvutcnR0VI888kirY/n5z3+ubGxsVHV1dZPPTp8+rXr37q0ABag+ffqorKysa+YfOnRI2djYqJdfftms7YkTJxSgpk+fboqNHDlSPfjgg2btXnnlFQWof//7362O/V//+pe6//77FaDuu+8+9d///d/XvMTSnnN64MAB1alTJ7VgwQJTrLi4WAFq7969SqnG7zogIEB1797dbKwtxW8FWXERQghxw3Q6Hfn5+axcudIUS0xMpL6+3vRXPDT+dZ+SksKkSZOa9GE0GjEajcyaNYtXX3212ePo9XouXbrE8OHDWxxLUVERX375JcuWLQMgKSmJ4OBglFIcPXqU0tJSkpOTSU5O5tNPP2XLli2UlJTQv3//VvN1Oh1Go5E+ffqYHW/v3r3Y2try2GOPAY2rL99++y3PPPNMk3bBwcH4+fmRkJDAuHHjmh3/Z599htFoxMHBASsrK7Kzs6moqKBbt27Ntm/vOa2qqmryPW7btg1HR0dGjx7NV199RUBAAMXFxaYxVFdX4+joiE6nazZ+K8hdRUIIIW5Ybm4uVlZW+Pr6mmKpqanY2NgwY8YMUywxMRGlFKGhoU36cHJywsnJqclejurqav75z38CYGNjg7W1NV5eXmZt9Ho9a9asAeCtt96iU6dOhIWFkZWVxbp16wAoKSkhNjaW1157jfr6eqKjo1m6dCnOzs6m/R6t5Ts4ONCpUyecnJxMbQ0GA/v27aNv376mQuTo0aMYjUYefPBBU7ukpCROnTrFb37zG/Ly8li/fn2T86+pqSEyMpLTp0+TkJBA3759+fvf/05lZSXh4eEtzn17z+mVQqN3796mzzZs2EBwcDCXLl3irbfe4syZM+j1etzc3Dh79iyffPIJQIvxW0EKFyGEEDdsz549AHTq1MkU27BhAw8//DBhYWGmWGJiIj169KBnz55N+vD29mbo0KHs2bOHiooKysvLSUpK4g9/+AP5+fkAaDQahgwZwuHDh6msrKSyspKkpCRee+01jEYjAFlZWfTv35/u3buzefNmpk6dCoCVlRV1dXWEh4fTtWtXSkpKWLVqFZGRkabVltbyBw8ezKBBg0hJSeHixYsUFxfz5ptvUl1dzWeffWba9/Ldd9/h7OxsVsSdPHkSGxsbAgIC2LFjB5MnT25y/vb29hw8eJB9+/bh7e3N+fPn6d69Ozt37mx2f8vtmlMvLy+6detGVlYWlZWVvPPOO5SUlNCtWzeys7MJCwujoqKCuro6XF1d+f3vf4+fnx9Ai/Fb4pZeeBJCiLvEH/7wB+Xl5aVsbW2VtbW16tGjh7rvvvvUfffdp3r37q26dOmiADV58uSOHmqHmjNnjrK3t1dPP/20iouLU6GhoWry5Mmqrq7OrN306dPVww8/rHQ6XbP9ZGdnq0mTJqmhQ4eqYcOGqdmzZ6sjR46o2tpapZRSDQ0NKjU1VY0bN04NHTpUPfTQQ+r5559XiYmJpjbJyclqwIABKiQkRK1du9YUNxqNKi4uTkVERKiQkBAVGhqqFi1apEpKSszG0FJ+Q0ODSktLU+PHj1dBQUEqLCxMvfrqq6qgoMAsPzY2Vvn7+6uysjJTrLKyUoWFhanBgwerN954Q+n1+lbns6amRj333HMqPz//WlPf7nNaV1en/vGPfyh/f38VEhKi3n33XXX48GHl6+urxo4dq7Kzs1VVVZWaPHmy0mg0at68eaa+W4rfClZK/eQJO0IIIaiqqiIrK4spU6ZgNBqJi4sz2+NQU1PDyy+/jKenJ3/72986cKSNy/8rVqxg69atlJaW4uPjw+uvv05kZKTZSkh7CAwMRClFXFwc+fn59OvXD29v73Y/rrh3yaUiIYRohouLC506daKiooLevXsTEBCAl5eX6d/AgQOJjY3l/vvv79BxGo1GRo4cSa9evUhMTCQ9PR1bW1smTpzIwYMH2/XYZWVlpKen8+STT+Lr68u4ceMYMGCAFC2iXUnhIoQQLSgsLKSyspIHHnjAtDHTYDBw6dIlANPehY6Uk5PD//3f/3Ho0CHc3Nzw8PBg/vz51NTU8Kc//andjltdXc2hQ4cAeP7556mrq2u3YwlxNSlchBCiBbt370YpxaOPPmqKLVu2jL///e8AuLm54ePj01HDAxo3xXbq1MlsdWXo0KEA/Pvf/263465atYqFCxfSvXt3oqOjTe/8EaK9yR4XIYRowX333UdJSQk6nQ5HR0eKiooYM2YMu3fvNrsbpSMppTh48CB9+vQxFVEpKSkMHTqUYcOG8e23316zj4ceeogNGzbc2js/hGgn8gA6IYRoRnZ2NsXFxVhZWZneyVJdXY2np+cdU7RA462+Y8eONYsdOXIEa2trXn755RbzrvfdOELcKaRwEUKIZhw4cACAuXPn8s4776CUIj4+3hS/oqGhgQsXLpjeb7Nq1SoGDx7cat8XLlxg3rx5rb5072qurq5s3ry5zWNftmwZS5YsafaJqnBj78YR4k4hhYsQQjRj3759WFtbm56KamVlhbW1temlfFfU1NTw17/+lby8PI4dO0Zbrr67u7ub9sncShUVFURGRvLSSy+xePHiFtvZ2dmxZ88eDh48yLPPPsuZM2dYuHAhc+bMYdGiRa0e4+r9PkLcjM8///yG8qRwEUKInygtLSU7Oxt3d3c8PT1N8QkTJmBjY/6/TUdHR9544w1SUlL43//939s9VDMxMTHMnTuXmTNn0tDQwCeffMLTTz/dYvvrfTcO3PgvGyFuFSlchBDiJ86fP49Wq6VXr15mj1O/+l01N+PChQvMnz+/zZeKunTpcs1LRa+88gqPP/44Tz31FFZWVtTU1LB79+5mC5eamhrGjx+Pvb09CQkJTJo0iffff5+//OUvhIeHc/LkyRs6LyFuBylchBDispqaGsrKykhLS6OsrIzBgwfT0NBAVVUVLi4ut+w47u7ufPTRR7ekr/r6ej744AO2bt1Kv3792LJlC9B4Li3dJXTl3TjW1tbU1taavRunoaHhloxLiPYihYsQQly2Z88eXnnlFcrLy1FKkZSUxMiRIwkLC2P9+vVYWVl19BCbOHnyJPPmzaO2tpaioiKzz37xi1+0mGdt/eNjvCZPnoyrqytWVlby1Ftxx5PnuAghxC2QkpLC6NGj+eqrrxgyZEhHD+eek5aWxpgxY3jxxRd54403bnu+gNraWhYtWkReXh6pqank5OS0Gr9RsuIihBC3kFxq6RipqakYjcYbfpLxzeYLsLW15cUXX2TgwIHExMRcM36j5JH/QghxE6qrq3nppZd48cUX0el0xMbGMm3atI4e1j1n5syZVFRUMH369A7Jv1V2796NTqfr0DHcKGtra7RaLUajkYiIiGvGb5SsuAghxE1wdHTkz3/+c0cPQ9wlkpKS8PX1xdnZuaOHckOOHz+OnZ0dQUFBbYrfCFlxEUIIccvV1dWxfPlyAgICiImJITExkSlTpuDj48PChQtN7aqqqti4cSPjx48nODiY8PBwvvnmG7O+amtr2b17NxMnTiQgIICgoCDi4+NRSpGeno6Pjw/du3dn3759TR4AaDQa+eyzzxg9ejSBgYFERESYXgjZlnydTsfnn3/OhAkTGD58OIMHD+bdd9/l0qVLGAwG/vSnPzF48GAmTJiAVqvl/fffJywsjIEDB/LII49QXFzcTjPc6HbM8xWlpaW88sorPPTQQ4SEhPDhhx+yePFi9Hq9qY1Wq8XZ2Zl+/fqZ9d1S/IYoIYQQ4josXrxYjRgxQs2ZM0cdPHhQTZo0SYWHh6spU6ao8+fPK6WUSk9PV0uWLFFff/216ty5sxo1apTKyMhQoaGhys3NTSml1MWLF1VMTIwaNmyYSklJUTqdTu3YsUM9+eSTZsdbuHCh8vHxUQkJCaq0tFSFhISoXr16qby8PDVy5EiVnJysIiIi1NChQ1VlZaVZbkpKivLy8lLHjx9X5eXlavz48WrGjBlKKdWm/NjYWOXm5qaOHz+uamtr1bFjx5Sjo6OKj49X6enp6sUXX1Tff/+9cnZ2VsOHD1dr165VWq1WHThwQHXp0kXNnTv3uub29ddfV9nZ2W1ufzvmWSmlampqVFRUlJo9e7Y6d+6cysrKUvfff7/y9PRU586dM/Xh6empvLy8VH19vVnfLcVvhFwqEkII0WYnT57E2tqaDz74gCFDhlBUVMSmTZtwcHDAx8eH1atXs3TpUlJSUhg/fjwVFRUYDAbmzp2LRqMhLy+PmJgYGhoaWLBgAQkJCezfv5/AwEAaGhpwcHAwXSZRSrFp0ybefvtttm3bRlRUFDU1NWRnZxMeHs6uXbuYN28e3bt3p7CwkB49etC5c2ez8e7YsYOf/exn+Pn54eTkxOnTp5kzZw5Aq/lKKdasWcP69etNnwGEhobi7OxMfHw8o0aNYsKECdjZ2VFdXU1gYKCpbxcXF6ytrdv8kMEbdTvmWSnF/PnzSU9PZ+fOnTg5OeHq6oqHhwcGg8E0NzqdjoKCAlatWmV2u31L8RslhYsQQog2W7duHaNHjzbd0vrxxx/j6uoKNN49snfvXpYuXcqoUaNwc3Nj6dKlODo6mp7U+/3339OlSxcKCwvZu3cvrq6unD59moyMDDIzM8nNzWXp0qUAlJWVsWrVKjQaDcOGDQMaH56XmJjIfffdR05ODl5eXqSnp5tecvnTwsXa2prMzEyio6MJCgrivffeM71vasSIES3mnz9/nrfffpvhw4fTtWtXsz6rqqo4d+4co0aNomfPnmzevBkbGxuWL19uanPmzBmqq6vx9/dvh2/hR7djngsKCoiPj2fYsGHY29sDjZfgCgsLTU9qhsYiCiA6OtpsjC3Fb5QULkIIIdrs3XffBaBv3744Ojri7u4ONN4GfubMGR544AHT5wCHDh1i7Nixpl94V/46LyoqoqioiAEDBnD27Fk6d+7MpEmTGDVqlOlYubm5ZGVlERkZSa9evUzxK2/fDg4OBmDFihVYW1szderUJuOdPn0669at46uvvuKrr75ix44dHDp0iP79+7eaf/LkSYqLi5k0aRJ2dnameFJSEjU1NfTp08f0vJ5du3YxZMgQUwEH8M0331BbW8vYsWObncdz586xdOlSqqqqzOJpaWkkJSU1KZa8vb2bfb7M7Zjnf//735w/f54HHnjA9IDCyspKiuzBQPcAACAASURBVIqKiIyMBBq////3//4f3bt3R6PRmPpoKX4zpHARQghxXXQ6Hfn5+axcudIUS0xMpL6+3vQXOzT+JZ+SksKkSZOa9GE0GjEajcyaNYtXX3212ePo9XouXbrE8OHDWxxLUVERX375JcuWLQMaC4vg4GCUUhw9epTS0lKSk5NJTk7m008/ZcuWLZSUlNC/f/9W83U6HUajkT59+pgdb+/evdja2vLYY48Bjasv3377Lc8880yTdsHBwfj5+ZGQkGB6y/gVXbt2Zdq0adTW1prF4+PjCQ8Pp3fv3mbxLl26tDgH7T3PVVVVTb7bbdu24ejoaHroYkBAgKnQg8bHBDg6OqLT6ZqN3wy5q0gIIcR1yc3NxcrKCl9fX1MsNTUVGxsbZsyYYYolJiailCI0NLRJH05OTjg5OZlWbK6orq7mn//8JwA2NjZYW1vj5eVl1kav17NmzRoA3nrrLTp16kRYWBhZWVmsW7cOgJKSEmJjY3nttdeor68nOjqapUuX4uzsbHarcUv5Dg4OdOrUyezFmgaDgX379tG3b19TIXL06FGMRiMPPvigqV1SUhKnTp3iN7/5DXl5eaxfv77J+dvb2zNy5Egefvhhs38DBgwgPDy8STwkJKTF76O95/lKoXF1MbVhwwaCg4O5dOkSb731FmfOnEGv1+Pm5sbZs2f55JNPAFqM3wwpXIQQQlyXPXv2AJi912jDhg08/PDDhIWFmWKJiYn06NHD7A3bV3h7ezN06FD27NlDRUUF5eXlJCUl8Yc//IH8/HwANBoNQ4YM4fDhw1RWVlJZWUlSUhKvvfYaRqMRgKysLPr370/37t3ZvHmz6XKPlZUVdXV1hIeH07VrV0pKSli1ahWRkZGm1ZbW8gcPHsygQYNISUnh4sWLFBcX8+abb1JdXc1nn31mupTz3Xff4ezsbFbEnTx5EhsbGwICAtixYweTJ0++JfPekvaeZy8vL7p160ZWVhaVlZW88847lJSU0K1bN7KzswkLC6OiooK6ujpcXV35/e9/b3rBZ0vxm3LT9yUJIYS4p8yZM0fZ29urp59+WsXFxanQ0FA1efJkVVdXZ9Zu+vTp6uGHH1Y6na7ZfrKzs9WkSZPU0KFD1bBhw9Ts2bPVkSNHVG1trVJKqYaGBpWamqrGjRunhg4dqh566CH1/PPPq8TERFOb5ORkNWDAABUSEqLWrl1rihuNRhUXF6ciIiJUSEiICg0NVYsWLVIlJSVmY2gpv6GhQaWlpanx48eroKAgFRYWpl599VVVUFBglh8bG6v8/f1VWVmZKVZZWanCwsLU4MGD1RtvvKH0en2b53bFihXXdTu0Uu0/z3V1deof//iH8vf3VyEhIerdd99Vhw8fVr6+vmrs2LEqOztbVVVVqcmTJyuNRqPmzZtn6rul+M2QlywKIYS4LoGBgSiliIuLIz8/n379+uHt7S1vlr4FVq5cSUxMzC3byHo3ks25Qggh2qysrIz09HSWLVuGr6+v2SUScfPs7OxwcHDo6GHc0WTFRQghRJtUV1ezZ88epk6dSmlpKT/72c+wtbXt6GGJe4xszhVCCNEmq1atYuHChXTv3p3o6GjTO3+EuJ1kxUUIIYQQFkNWXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFaJfCxaDXY6hvj56FEEIIcS9rh8LlFLumTWNX9q3vWQghhBD3NrlUJIQQQgiLYXPtJuWk7TkGw6Lxd2+M6E/vJ+EHDyIf9cf1SpvPD2MYHkNwj8tp9XpOHYgj4YgWu6AIoh8Jw9P+qm71BRw7uJ+k73Q4DwsmakwYnk6NH2m/20lalzB8/3OM3YnlBD//K4JdW88BPQWndXj4eGB38/MihBBCiDtQG1ZcXLGrimP3/5Vf/tlAzrHVfPThbtIuXA5dSGP3h+XYdf0x68QnG0hzCiby8Sg0F3by6z8f4UoPXDjG6rmrSXPwJ/LxSHzrjvDG3M2k1TR+XJ77EYe3fMQ/sroRPCESjdO1c/THN/DreYvYnXXTcyKEEEKIO1QbVlzA+4EoTu5No3xUBK7kcur4Ezw9bT8nc/VEuDuhzz3JsUlhvHrVUodm0gs88eDlgK8dBZN3k/ZsBBHukJsYR/nkhfzxYY/Gz/396clLbP56HP6XY2m9o9g+M9i0enKtHKfQZ1mzUke3AbdiWoQQQghxJ2rTHhc73wAivz5Jjh7IPcnR0ACih0RxLDMHAwZyMncT7acxu0TTzfmqn+zscDb9oCUnpQ7PTlrS0tJM/85dciapQIvhSjMX56v6a0uOE54+HjghhBBCiLtVm1ZcsNPg/8gi0rJ+RbezhxnkNw4nX4jckEMhdpw6FIX/5LaWDHrKz9ryn+IccqquCjsE8ysPO+puMEf2tQghhBB3v7YVLjgx6MFoVp86gkdmTwa95NRYzPjGkZMIR/0CeN2lrYd0pU9gLjr/FcQEtrXcuJEcIYQQQtxt2nw7tJP3IAbt2Uwcwfi6Azih8XNm8/qddHvQ9/LdRW3hSvCYGA7/Yxen9JdD9VqOvDublYnlN55zPo2dn+wnt6bNAxFCCCGEhWnjigvg7k+YXznnfDVc3h6L64BgPMt1hPl5tJr6U3aBT7Gs6h+sjX0UrYsnhrMGNJNe4IVRLZc/18oxnPuenZ+mYTciCu++1zUcIYQQQlgIK6WU6rjDG9Dr67C1d8KuU3vmCCGEEOJu0MGFixBCCCFE28kj/4UQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBZDChchhBBCWAwpXIQQQghhMaRwEUIIIYTFkMJFCCGEEBbDpqMHIIQQQtytNmzYQGxsLL169aJz584A5OXlUV9fz/3334+NjQ11dXXk5+cza9YsNmzYYMr94osvWLNmDWVlZRgMBtzc3Ex9GI1GPD09mT59OiNHjsTW1rZDzq8jWCmlVEcPQgghhLjb1NfXM3bsWEaMGMHMmTNxcHAAYODAgdjY2JCamoqdnR21tbUsWLCAwMBAFi9ebMqvqKigsLCQadOmce7cOeLi4hgwYAAAdXV1/PWvf+XDDz9k48aNTJ48uUPOsSPIpSIhhBCiHRQUFODi4sKbb77JwIED8fT0JD8/n5qaGsaMGYNGo8HT05P+/fsTFhZGYGCgWX7Xrl1xdHTk4sWL9OzZE39/f7y8vPDy8kKj0RAbG4ter2f16tUddIYdQwoXIYQQoh3k5eXx3HPPmcW2bt0KwFNPPWUWr6iowM3NrUkfWq0WrVZL3759cXd3N/ssIyMDgNLS0ls57DueFC5CCCFEO6itrSUkJMT0c0NDA1u2bMHW1pbHH3/crG2/fv3w9PRs0seJEyeora0lKioKKysrs8/279+PlZUVzzzzTPucwB1K9rgIIYQQt0FOTg5+fn74+fmRnJzcppypU6eydetWMjIy8PPzM8UzMzMZO3YsvXv35sCBA3Tr1q29hn3HkbuKhBBCiNvghx9+wGg0Mnjw4Da1NxqN7NmzB1tbW7Zv38727dsBKCsrIy4ujilTprBkyZJ7qmgBKVyEEEKI2yIzM5P6+vo2Fy6HDh1Cp9Mxffp0Zs2aZYpbW1uzYsUK7Ozs2muodzQpXIQQQojb4MrloaCgoDa1P3DgAFZWVsTExODl5dWeQ7MosjlXCCGEuA2Sk5OxsrIy26vSEqPRyPHjx3FxcUGj0dyG0VkOKVyEEEKIdlZcXExOTg6DBg2iV69e12xfUlJCcXExPXv2pGfPnrdhhJZDChchhBCiHRiNRkpKSsjPz2fdunU0NDQwcuRIioqKKCkpaTGnrKyMjIwMiouLcXNzw8bGBoPBcJtHf+eSwkUIIe4BaWlpuLu788c//rFD8u9FmZmZTJgwgbFjx/LOO+8AsG3bNsaMGcMvfvGLZnOSk5OJiIhg9uzZXLp0ibS0NCIjI/nqq69u59DvaLI5Vwgh7gGpqakYjUZ8fHw6JP9eFBAQQGpq6nXlDB8+3PREXNE8eQCdEEIIISyGXCoSQgghhMWQwkUIITpAXV0dy5cvJyAggJiYGBITE5kyZQo+Pj4sXLjQ1K6qqoqNGzcyfvx4goODCQ8P55tvvjHrq7a2lt27dzNx4kQCAgIICgoiPj4epRTp6en4+PjQvXt39u3bx08X2Y1GI5999hmjR48mMDCQiIgITpw4AdCmfJ1Ox+eff86ECRMYPnw4gwcP5t133+XSpUsYDAb+9Kc/MXjwYCZMmIBWq+X9998nLCyMgQMH8sgjj1BcXNxOMyzuWkoIIcQtt3jxYjVixAg1Z84cdfDgQTVp0iQVHh6upkyZos6fP6/S09PVkiVL1Ndff606d+6sRo0apTIyMlRoaKhyc3NTSil18eJFFRMTo4YNG6ZSUlKUTqdTO3bsUE8++aTZsRYuXKh8fHxUQkKCKi0tVSEhIapXr14qLy9PjRw5UiUnJ6uIiAg1dOhQVVlZaZabkpKivLy81PHjx1V5ebkaP368mjFjhlJKtSk/NjZWubm5qePHj6va2lp17Ngx5ejoqOLj41V6erp68cUX1ffff6+cnZ3V8OHD1dq1a5VWq1UHDhxQXbp0UXPnzm3Hb0HcjWRzrhBC3GInT57E2tqaDz74gCFDhlBUVMSmTZtwcHDAx8eH1atXM3DgQMaPH09FRQUGg4G5c+ei0WjIy8sjJiaGhoYGFixYQEJCAvv37ycwMJCGhgYcHBxwdnYGGldENm3axNtvv822bduIioqipqaG7OxswsPD2bVrF/PmzaN79+4UFhbSo0cPOnfubDbWHTt28LOf/Qw/Pz+cnJw4ffo0c+bMAWg1XynFmjVrWL9+vekzgNDQUJydnYmPj2fUqFFMmDABOzs7qqurCQwMNPXt4uKCtbU1Fy5cuF1fi7hLSOEihBC32Lp16xg9ejQ5OTkAfPzxx7i6ugJga2vL3r17mT17Nm5ubixduhRHR0fGjx+Pvb0933//PV26dKGwsJC9e/fi6urK6dOnycjIIDMzk9zcXJYuXQo0vmxv1apVaDQahg0bBoC9vT2JiYncd9995OTk4OXlRXp6Onl5eSxevLhJ4WJtbU1mZibR0dEEBQXx3nvv8fOf/xyAESNGtJh//vx53n77bYYPH07Xrl3N+qyqquLcuXOMGjWKnj17snnzZmxsbFi+fLmpzZkzZ6iursbf37/Febxw4QLz5s1rc3Hj6urK5s2b29RWWC4pXIQQ4hZ79913Aejbty+Ojo64u7sD0NDQwJkzZ3jggQfo27cv0PgivbFjx2Jvbw9gWrkoKiqiqKiIAQMGcPbsWTp37sykSZMYNWqU6Ti5ublkZWURGRlp9jTWKy/xCw4OBmDFihVYW1szderUJmOdPn0669at46uvvuKrr75ix44dHDp0iP79+7eaf/LkSYqLi5k0aZLZy/6SkpKoqamhT58+DBkyBGhcuRkyZIipeAP45ptvqK2tZezYsS3Oo7u7O3//+99bn+wb8Oijj97yPu8ln3/+eYceXwoXIYRoBzqdjvz8fFauXGmKJSYmUl9fb1odKSsrIyUlhUmTJjXJNxqNGI1GZs2axauvvtrsMfR6PZcuXWL48OEtjqOoqIgvv/ySZcuWAY2FRXBwMEopjh49SmlpKcnJySQnJ/Ppp5+yZcsWSkpK6N+/f6v5Op0Oo9FInz59zI63d+9ebG1teeyxx4DG1Zdvv/2WZ555pkm74OBg/Pz8SEhIYNy4ca3O563U0b94xc2RwkUIIdpBbm4uVlZW+Pr6mmKpqanY2NgwY8YMoLGQUUoRGhraJN/JyQknJyfTas0V1dXV7Nq1i+nTp2NjY4O1tXWTNwfr9Xo2btzIr3/9a9566y06depEWFgYWVlZrFu3juDgYEpKSoiNjaWuro5//etfREdH4+fnx+eff27aQwO0mO/g4ECnTp1wcnIytTUYDOzbt4++ffuaCpGjR49iNBp58MEHTe2SkpI4deoUH3/8MXl5eaxfv77ZwuXChQvMnz+/zZeKunTpIpeK7gFSuAghRDvYs2cPAJ06dTLFNmzYwMMPP0xYWBjQWLj06NGj2ZfoeXt7M3ToUPbs2cNjjz2GUoqcnBy2bNlC9+7dAdBoNAwZMoTDhw8TExMDQFZWFps2bTKtmGRlZdG/f3+6d+/O3//+d9PlHisrK+rq6ggPD6dr166UlJSwatUqIiMjTbmt5Q8ePJhBgwaRkpLCxYsXuXjxImvXrqW6uprPPvvMtO/lu+++w9nZ2ayAO3nyJDY2NgQEBLBjxw4mT57c7By6u7vz0Ucf3cDsi7uZPDlXCCHawQsvvMDGjRt5/PHHmThxIu+99x69e/fm008/xcam8W/GGTNmcP78eXbt2mW2cnFFTk4O8+fP58yZM9jY2ODv78/TTz/NQw89hJ2dHUop0tLSTKsSdnZ2+Pv789RTTzF8+HDs7Oz497//zbRp0+jatSvPPvssv/zlL7Gzs6O+vp7t27ezZs0aqqursbGxYcyYMfzmN78xK6RayldKkZGRwfz589FqtTg4OBAaGsrLL79sdvno17/+NUePHuXQoUN069YNaLx8NGHCBKqqqnj88ceZN28ejo6O7fyNNLVhwwZiY2Pp1auXadNxXl4e9fX13H///djY2FBXV0d+fj6zZs1iw4YNptwvvviCNWvWUFZWhsFgwM3NzdSH0WjE09OT6dOnM3LkSGxtbdv9XJYsWcLGjRspKSmhvr4ed3d303GVUuj1eiorK5k8eTLbtm27Y8+jLaRwEUKIdhAYGIhSiri4OPLz8+nXrx/e3t5mKzCi49TX1zN27FhGjBjBzJkzcXBwAGDgwIHY2NiQmpqKnZ0dtbW1LFiwgMDAQBYvXmzKr6iooLCwkGnTpnHu3Dni4uIYMGAA0Phwwb/+9a98+OGHbNy4scUVpVupqqqKrKwspkyZgtFoJC4uzqyArKmp4eWXX8bT05O//e1vd+x5tIU8OVcIIW6xsrIy0tPTefLJJ/H19WXcuHEMGDBAipY7SEFBAS4uLrz55psMHDgQT09P8vPzqampYcyYMWg0Gjw9Penfvz9hYWEEBgaa5Xft2hVHR0cuXrxIz5498ff3x8vLCy8vLzQaDbGxsej1elavXt3mMe3evRudTndD5+Pi4kKnTp2oqKigd+/eBAQEmMbj5eXFwIEDiY2N5f7772/382hvUrgIIcQtVF1dzaFDhwB4/vnnqaur6+ARiebk5eXx3HPPmcW2bt0KwFNPPWUWr6iowM3NrUkfWq0WrVZL3759m2yivvKG59LS0jaPKSkpCa1W2+b2P1VYWEhlZSUPPPCA6dKjwWDg0qVLAKZ9RT91q8+jvUnhIoQQt9CqVatYuHAh3bt3Jzo62vTeH3Fnqa2tJSQkxPRzQ0MDW7ZswdbWlscff9ysbb9+/fD09GzSx4kTJ6itrSUqKgorKyuzz/bv34+VlVWT28Db0+7du1FKmT2nZtmyZaZn4bi5ueHj49Mk7047j2vqiPcMCCGEEHeS7Oxs1blzZzV06NA25zzxxBMKUBkZGWbxjIwM5eHhoYKCglRZWVmb+3v99ddVdnZ2m9v/VO/evZWVlZXS6/VKKaUKCwvVwIEDVVZWVqt5t/o82pvcDi2EEOKe98MPP2A0Gk1PHb4Wo9HInj17sLW1Zfv27Wzfvh1o3N8UFxfHlClTWLJkielOqvaWnZ1NcXExVlZWpqcvV1dXm/bpWMp5tIUULkIIIe55mZmZ1NfXt7lwOXToEDqdjunTpzNr1ixT3NramhUrVpi9BuF2OHDgAABz587lnXfeQSlFfHy8Kd6SO+082kIKFyGEEPe85ORkAIKCgtrU/sCBA1hZWRETE9PkycXXcu7cOZYuXUpVVZVZPC0tjaSkpCYvrfT29uaNN95otc99+/ZhbW1tegKxlZUV1tbWphdmtsd5dBQpXIQQQtzzkpOTsbKyws/P75ptjUYjx48fx8XFBY1Gc93H6tq1K9OmTaO2ttYsHh8fT3h4OL179zaLd+nSpdX+SktLyc7Oxt3d3WwT8YQJE0wPO2yP8+goUrgIIYS4pxUXF5OTk8OgQYPM3rLdkpKSEoqLi+nZs2ezr2u4Fnt7e0aOHNkknpqaSnh4+HUXEefPn0er1dKrVy+z8TT3NOar3ex5dBS5HVoIIcQ9x2g0UlJSQn5+PuvWraOhoYGRI0dSVFRESUlJizllZWVkZGRQXFyMm5sbNjY2GAyG2zz6RjU1NRQVFZGWlkZZWRlubm40NDQ0uQT1U3faeVwvKVyEEELcczIzM5kwYQJjx47lnXfeAWDbtm2MGTOGX/ziF83mJCcnExERwezZs7l06RJpaWlERkby1Vdf3c6hm+zZs4fw8HBeeOEFlFIkJSUxcuRI5s6di2rlbT532nlcL3lXkRBCCHEHWLlyJTExMRa136QjyIqLEEIIcQews7MzvexRtExWXIQQQghhMWTFRQghhBAWQwoXIYQQQlgMKVyEEEIIYTGkcBFCCCGExZDCRQghhBAWQwoXIYQQQlgMKVyEEEIIYTGkcBFCCCGExZDCRQghhBAWQwoXIYQQQlgMKVyEEEIIYTGkcBFCCCGExZDCRQghhBAWQwoXIYQQQlgMKVyEEEIIYTFsOnoAQgghxJ1qw4YNxMbG0qtXLzp37gxAXl4e9fX13H///djY2FBXV0d+fj6zZs1iw4YNZvlffPEFa9asoaysDIPBgJubm6kfo9GIp6cn06dPZ+TIkdja2t7287NEVkop1dGDEEIIIe409fX1jB07lhEjRjBz5kwcHBwAGDhwIDY2NqSmpmJnZ0dtbS0LFiwgMDCQxYsXm/VRUVFBYWEh06ZN49y5c8TFxTFgwAAA6urq+Otf/8qHH37Ixo0bmTx58m0/R0skl4qEEEKIZhQUFODi4sKbb77JwIED8fT0JD8/n5qaGsaMGYNGo8HT05P+/fsTFhZGYGBgkz66du2Ko6MjFy9epGfPnvj7++Pl5YWXlxcajYbY2Fj0ej2rV6/ugDO0TFK4CCGEEM3Iy8vjueeeM4tt3boVgKeeesosXlFRgZubW7P9aLVatFotffv2xd3d3eyzjIwMAEpLS2/VsO96UrgIIYQQzaitrSUkJMT0c0NDA1u2bMHW1pbHH3/crG2/fv3w9PRstp8TJ05QW1tLVFQUVlZWZp/t378fKysrnnnmmVt/Ancp2eMihBBCtEFOTg5+fn74+fmRnJzc5rypU6eydetWMjIy8PPzM8UzMzMZO3YsvXv35sCBA3Tr1q09hn3XkbuKhBBCiDb44YcfMBqNDB48uM05RqORPXv2YGtry/bt29m+fTsAZWVlxMXFMWXKFJYsWSJFy3WQwkUIIYRog8zMTOrr66+rcDl06BA6nY7p06cza9YsU9za2poVK1ZgZ2fXHkO9q0nhIoQQQrTBlctDQUFBbc45cOAAVlZWxMTE4OXl1V5Du6fI5lwhhBCiDZKTk7GysjLbp9Iao9HI8ePHcXFxQaPRtPPo7h1SuAghhBDXUFxcTE5ODoMGDaJXr15tyikpKaG4uJiePXvSs2fPdh7hvUMKFyGEEKIZRqORkpIS8vPzWbduHQ0NDYwcOZKioiJKSkpazSsrKyMjI4Pi4mLc3NywsbHBYDDcxtHfveR2aCGEEM2qra1l0aJF5OXlkZqaSk5OTqvxu83333/PM888g16vR6vVcvHiRdzc3HBzc8PFxYUTJ040m/ftt9/y7LPPUl5eTnFxMc7Oznh5ebF69WrGjh17m8/i7iOFixBCiGY1NDRw9uxZBg4cSExMjOmpsS3Fhbgd5FKREEKIZllbW6PVajEajURERFwzLsTtIIWLEEKIFh0/fhw7O7smtwC3FBeivUnhIoQQd5C6ujqWL19OQEAAMTExJCYmMmXKFHx8fFi4cKGpXVVVFRs3bmT8+PEEBwcTHh7ON998Y9ZXbW0tu3fvZuLEiQQEBBAUFER8fLzp89LSUl555RUeeughQkJC+PDDD1m8eDF6vd7URqvV4uzsTL9+/cz6bikuRHuTPS5CCHEHycjIID4+nnHjxjF27FiGDx/OmjVreO655/jhhx+4cOECOp2OmTNnUlRUxN/+9jcGDBhAQkIC8fHx/POf/zT1tWjRIrZv385f/vIXgoKCeOSRRygsLKS4uJja2loeffRR+vbty5tvvsnFixeJioqivr6eEydO4OHhAYCXlxdWVlbk5eVhbf3j37otxYVob/LkXCGEuIOkpKQwfvx4KioqMBgMzJ07F41GQ15eHjExMTQ0NLBgwQISEhLYv38/gYGBNDQ04ODggLOzMwBKKTZt2sTbb7/Ntm3biIqKoqamhuzsbMLDw1FKMX/+fNLT09m5cydOTk64urri4eGBwWCgR48eAOh0OgoKCli1apVZcdJSXIjbQQoXIYS4g4waNQo3NzeWLl2K4/9v725j2qzaAI7/W0pBWIDSbtmG7dh42ZwOcFE2QZSNTCFGTbcZEVFZ3MwwM2FR0Ri3ROdmfCFTE9SpmKEfJoEt+DI0E7OFqfsgKBIwKm8btLQgYGGUl1Lo84HsjpWy4Z7Bsz5ev29c5z435/RLr5yec52QEDIzMwkODubnn38mPDwci8XCl19+iU6n47fffqOxsZGmpiba2trYt28fMHWB38GDB4mJiSE5ORmA4OBgampqiIqKorOztNQaBwAACXxJREFUk/LycpKTkwkODgamao9YLBZyc3NRqVTAVBIFcNddd3mNcab43/X29vLUU0/R29s7q7nrdDo+/vjjWX5S4t9KEhchhLiKLFu2DJi6nC8jI0NJLC6sglitVqxWK3FxcZw7d46goCA2b97Mbbfdpryjra2N5uZm0tPTvaq8Xrgc8Mcff6Snp4fVq1cTEBAAwMDAAFarlfT0dGDqyPMTTzzBwoULvcrVzxT3xWAwcPjw4f/2I/Fy9913X9H3/Vt9/vnn/+shXDZJXIQQ4irT19fHTz/9xObNm6e1ud1u3G43eXl5PPnkkz77O51ORkZGWLdunc/2wcFBJiYmlNUYgIqKCkJCQtiwYQOnT58mMTGRrq4uZQzDw8OEhIQwNDTkMz5f/PkLV1wZkrgIIcRVpqamBo/Hwy233DKtLTQ0lNDQUAwGg1d8eHiYyspKcnJy0Gg0qNXqabcRO51OSktLlZWSpUuXKm0lJSXcdNNNjIyM8Morr3DgwAGcTid6vZ5z585RXV3Njh07OHv2rM+4L729vRQWFs76p6Lw8HD5qUhckiQuQghxlampqWHRokU+L+ZbsWIFa9eupaqqinvvvRePx0NraytHjhxh4cKFAMTExLBmzRpOnTqF2WwGoLm5mY8++ojY2FhMJhORkZE0NzezatUqPvjgA2w2G8uXL6elpYWUlBQcDgfj4+PodDqeeeYZCgoKAGaM+2IwGPjwww/n4BMS/2ZyHFoIIa4yDz74ID09PVRWVhIaGjqtvbW1lcLCQs6ePYtGoyEhIYGHHnqI9evXo9Vq8Xg8NDQ0KKsdWq2WhIQEcnNzWbduHWq1mrKyMl599VWCgoJ44IEHWLt2LTt37iQqKopDhw6xaNEitm3bRn19PWazmf3796PVajl//rzP+HwrKSkhPz+fJUuWEBQUBEB7ezsTExNER0ej0WgYHx+no6ODvLw8SkpKvPp/8cUXvP322/T19eFyudDr9cp73G43RqORnJwc0tLSCAwMnPP57N27l9LSUmw2GxMTExgMBuX/ejwenE4nAwMDbNmyhYqKCqXf4cOHeffdd+nv76e/v5/777+f4uJipb29vZ3t27fT1tbG+Pg4kZGRZGdne9UE8jeSuAghhPArExMTZGRkcOutt/Lwww9zzTXXABAfH49Go6G+vh6tVsvY2BjPPvssSUlJPP/8817vcDgcWCwWsrOzsdvtlJWVERcXB0wVASwuLua9996jtLSULVu2zPmcBgcHaW5uZuvWrbjdbsrKyrj22muV9tHRUQoKCjAajRw6dEiJDw0NYbfbefTRR/n+++9ZsGABvb29yqZrl8tFd3c377zzDna7neeeew69Xo9Op5vzOc0VOYAvhBDCr3R2dhIWFsZLL71EfHw8RqORjo4ORkdH2bhxIzExMRiNRmJjY0lJSSEpKWnaOyIiIggJCeH8+fMsXryYhIQETCYTJpOJmJgY8vPzcTqdvPXWW7Me1/HjxxkaGrqsOYWFhREQEIDD4WDp0qUkJiYq4zGZTMTHx5Ofn090dLRXvwULFhAREYFOp6OgoACHw8GePXuUdq1Wi9FopL+/n8cff5zY2Fi/TlpAEhchhBB+pr29fdqG4As3VOfm5nrFHQ4Her3e53u6u7vp7u5m2bJl0zY7NzY2AlPXIsxWbW0t3d3ds37+7ywWCwMDA6xevVr5idDlcjEyMgKARqMhMTFxWj+Hw0FkZCSFhYWEhYVRVFREf3+/1zOdnZ1eR+P9mSQuQggh/MrY2Bg333yz8vfk5CRHjhwhMDCQ++67z+vZ5cuXYzQafb7nhx9+YGxsjE2bNilF9y74+uuvUalUPPLII1d+AjM4fvw4Ho/Hq1bN/v37lVo4er2elStXTut37Ngx1q9fj06nIysrC5fLRWVlpdJut9sZGxsjPDx87icxDyRxEUII4VcyMzO9Tly1t7czODjImjVrpj27bds2r70if/Xdd98BsGnTJq94U1MTx44dIykpacaj3nPhs88+Q6VSkZmZCUwVG/zkk0+U8SUnJ/ss+vfpp59yww03oNFolFNk5eXluFwuYCqxWblypXIlhL+T49BCCCH82u+//47b7VYqA8+G2+2mqqqKwMBAjh49ytGjR4Gp4n9lZWVs3bqVvXv3EhkZOVfD9tLS0kJXVxcqlUqpkjw8PKzs1ZmJ3W6npaVFqbicmZmJXq/n22+/pbOzk5iYGMrLy9m1a9e8zGM+SOIihBDCrzU1NTExMfGPEpeTJ08yNDRETk4OeXl5SlytVvPaa6/N+xHv6upqAHbv3k1RUREej4fy8nIlPpMzZ85gMpmIiooCpor47dmzh4KCAg4cOMDLL7/Mr7/+SkpKypzPYb5I4iKEEMKv1dXVAXDjjTfOuk91dTUqlQqz2TytwvCl2O129u3bx+DgoFe8oaGB2tpaIiIivOIrVqzghRdeuOg7v/rqK9RqNXfccQcAKpUKtVrNnXfeedF+p06dIicnxyu2a9cu3njjDUpLS0lNTSUuLu7/ZmMuSOIihBDCz9XV1aFSqbj++utn9bzb7ebMmTOEhYVd8qJIXyIiIsjOzmZsbMwrXl5eTmpqqtdVCsAlN8X+8ccftLS0YDAYvDYSZ2VlodHM/DXtdDr55ZdfePrpp73iAQEBmM1mDh48yIsvvshjjz0226n5BUlchBBC+K2uri5aW1u57rrrZr2qYLPZ6OrqYvHixT6vVbiU4OBg0tLSpsXr6+tJTU39x8lQT08P3d3dLFmyxGs8vqom/9Wff/6JWq32ufk4KyuL4uJiLBYL99xzzz8az9VOThUJIYTwK263G5vNRkdHB++//z6Tk5OkpaVhtVqx2WwX7dfX10djYyNdXV3o9Xo0Go1y+ma+jY6OYrVaaWhooK+vD71ez+Tk5LSfoP7O5XJhsVj45ptv0Gq19PT0TFv9SU9PJzo6GpPJdFnJ2dVMEhchhBB+pampiaysLDIyMigqKgKgoqKCjRs3XnR1oa6ujttvv53t27czMjJCQ0MD6enpnD59er6G7qWqqorU1FR27tyJx+OhtraWtLQ0du/ezcVu4zlx4gQpKSns2LGDEydOsGHDBk6ePOn1TGBgIG+++SarVq265MqNv5G7ioQQQogr4PXXX8dsNl/Wvhkxe7LiIoQQQlwBWq1WufBRzB1ZcRFCCCGE35AVFyGEEEL4DUlchBBCCOE3JHERQgghhN+QxEUIIYQQfkMSFyGEEEL4jf8AvBSbF9bsfjkAAAAASUVORK5CYII="}}},{"cell_type":"markdown","source":"# Models","metadata":{}},{"cell_type":"markdown","source":"In this term project, I refer to https://www.kaggle.com/code/georgiisirotenko/pytorch-tpu-baseline-flowers-tranlearning-ensemble and use, setup configuration for pretrained models to classify big data images with TPU & PyTorch\n\nYou can change other models in the List Models of PyTorch site (in this term project I keep the models that I referred):\n\nhttps://pytorch.org/vision/0.13/models.html (they provide instructions how to use the models)\n\n","metadata":{}},{"cell_type":"markdown","source":"## 1. Dense Convolutional Network (DenseNet) \nA DenseNet is a type of convolutional neural network that utilises dense connections between layers, through Dense Blocks, where we connect all layers (with matching feature-map sizes) directly with each other. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers. paper: https://arxiv.org/abs/1608.06993v5\n\n(and we can read great blog : https://paperswithcode.com/method/dense-block)","metadata":{}},{"cell_type":"code","source":"\nfrom PIL import Image\nimport requests\nfrom io import BytesIO\n\nresponse = requests.get(\"https://production-media.paperswithcode.com/methods/Screen_Shot_2020-06-20_at_11.33.17_PM_Mt0HOZL.png\")\nimg = Image.open(BytesIO(response.content))\n\ndisplay(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:53.852746Z","iopub.execute_input":"2022-09-08T11:27:53.853074Z","iopub.status.idle":"2022-09-08T11:27:54.215450Z","shell.execute_reply.started":"2022-09-08T11:27:53.852990Z","shell.execute_reply":"2022-09-08T11:27:54.214572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Vision Transformer (ViT)\n\nThe Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over patches of the image. An image is split into fixed-size patches, each of them are then linearly embedded, position embeddings are added, and the resulting sequence of vectors is fed to a standard Transformer encoder. In order to perform classification, the standard approach of adding an extra learnable “classification token” to the sequence is used. Paper: https://arxiv.org/abs/2010.11929v2\n\n\n(and we can read great blog: https://paperswithcode.com/method/vision-transformer)","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nimport requests\nfrom io import BytesIO\n\nresponse = requests.get(\"https://production-media.paperswithcode.com/methods/Screen_Shot_2021-01-26_at_9.43.31_PM_uI4jjMq.png\")\nimg = Image.open(BytesIO(response.content))\n\ndisplay(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:54.216938Z","iopub.execute_input":"2022-09-08T11:27:54.217353Z","iopub.status.idle":"2022-09-08T11:27:54.381693Z","shell.execute_reply.started":"2022-09-08T11:27:54.217297Z","shell.execute_reply":"2022-09-08T11:27:54.380732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. GoogleNet\n\nGoogLeNet is a type of convolutional neural network based on the Inception architecture. It utilises Inception modules, which allow the network to choose between multiple convolutional filter sizes in each block. An Inception network stacks these modules on top of each other, with occasional max-pooling layers with stride 2 to halve the resolution of the grid. Paper: https://arxiv.org/abs/1409.4842v1\n\n\n(and we can read great blog: https://paperswithcode.com/method/googlenet)","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nimport requests\nfrom io import BytesIO\n\nresponse = requests.get(\"https://production-media.paperswithcode.com/methods/Screen_Shot_2020-06-22_at_3.28.59_PM.png\")\nimg = Image.open(BytesIO(response.content))\n\ndisplay(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:54.383113Z","iopub.execute_input":"2022-09-08T11:27:54.383518Z","iopub.status.idle":"2022-09-08T11:27:54.706917Z","shell.execute_reply.started":"2022-09-08T11:27:54.383476Z","shell.execute_reply":"2022-09-08T11:27:54.705956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Residual Network(ResNet)\n\nResidual Networks, or ResNets, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack residual blocks ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks. Paper: https://arxiv.org/abs/1512.03385\n\n\n\n(and we can read great blog : https://paperswithcode.com/method/resnet)","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nimport requests\nfrom io import BytesIO\n\nresponse = requests.get(\"https://production-media.paperswithcode.com/methods/Screen_Shot_2020-09-25_at_10.26.40_AM_SAB79fQ.png\")\nimg = Image.open(BytesIO(response.content))\n\ndisplay(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:54.709036Z","iopub.execute_input":"2022-09-08T11:27:54.709439Z","iopub.status.idle":"2022-09-08T11:27:54.993278Z","shell.execute_reply.started":"2022-09-08T11:27:54.709389Z","shell.execute_reply":"2022-09-08T11:27:54.992194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. VGG19\nVGG19 is a variant of VGG model which in short consists of 19 layers (16 convolution layers, 3 Fully connected layer, 5 MaxPool layers and 1 SoftMax layer). There are other variants of VGG like VGG11, VGG16 and others. VGG19 has 19.6 billion FLOPs.\n\nPaper: https://arxiv.org/pdf/1409.1556v6.pdf\n\n\n\n(and we can read great blog: https://doi.org/10.1007/s12652-021-03488-z)","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nimport requests\nfrom io import BytesIO\n\nresponse = requests.get(\"https://production-media.paperswithcode.com/models/vgg19_KzDZnJv.png\")\nimg = Image.open(BytesIO(response.content))\n\ndisplay(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:54.994798Z","iopub.execute_input":"2022-09-08T11:27:54.995213Z","iopub.status.idle":"2022-09-08T11:27:55.281069Z","shell.execute_reply.started":"2022-09-08T11:27:54.995171Z","shell.execute_reply":"2022-09-08T11:27:55.279616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tensor Processing Units (TPUs)\n\nTPUs are powerful hardware accelerators specialized in deep learning tasks. They were developed (and first used) by Google to process large image databases, such as extracting all the text from Street View. ","metadata":{}},{"cell_type":"markdown","source":"**PyTorch/XLA**\n\nThe PyTorch-TPU project was born out of a collaborative effort between the Facebook PyTorch and Google TPU teams and was officially launched at the 2019 PyTorch Developer Conference. This new integration enables PyTorch users to run and scale up their models on Cloud TPUs. PyTorch / XLA package lets PyTorch connect to Cloud TPUs and use TPU cores as devices.\n\nXLA (accelerated linear algebra) is a compiler-based linear algebra execution engine. It is the backend that powers machine learning frameworks such as TensorFlow and JAX at Google, on a variety of devices including CPUs, GPUs, and TPUs.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"#install timm package\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev\n!pip install timm","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:27:55.282399Z","iopub.execute_input":"2022-09-08T11:27:55.283169Z","iopub.status.idle":"2022-09-08T11:29:08.722863Z","shell.execute_reply.started":"2022-09-08T11:27:55.283124Z","shell.execute_reply":"2022-09-08T11:29:08.720625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Importing necessary Libraries","metadata":{}},{"cell_type":"markdown","source":"os, pandas, numpy, tensorflow, pytorch must be imported","metadata":{}},{"cell_type":"code","source":"#for os, pandas, numpy\nimport os\nimport io\nimport pandas as pd\nfrom scipy import stats\nimport numpy as np\n\nimport timm\nimport random\nimport time\nimport copy\nfrom operator import itemgetter\n\nfrom collections import OrderedDict, namedtuple\nimport joblib\n\nimport logging\nimport sys\n\nfrom PIL import Image\nimport cv2\nimport albumentations\n\nimport IPython.display as display\n\n#for tensorflow\nimport glob\nimport tensorflow as tf\n\n#for learning model\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics, model_selection\n\n#PyTorch libraries\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.optim import lr_scheduler\nimport torch.optim as optim\nimport torch_xla\nimport torch_xla.debug.metrics as met\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp\nimport torch_xla.distributed.parallel_loader as pl\nimport torch_xla.utils.utils as xu\nimport torchvision\nfrom torchvision import datasets, transforms\nfrom torch.utils.data import Dataset, DataLoader, ConcatDataset\nfrom torchvision.utils import make_grid\nimport torchvision.transforms as transforms\n\n\n#for visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings(\"ignore\");","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:08.726925Z","iopub.execute_input":"2022-09-08T11:29:08.727286Z","iopub.status.idle":"2022-09-08T11:29:21.995404Z","shell.execute_reply.started":"2022-09-08T11:29:08.727219Z","shell.execute_reply":"2022-09-08T11:29:21.994310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Dataset Loading","metadata":{}},{"cell_type":"markdown","source":"Get Paths of dataset. We use glob to read files","metadata":{}},{"cell_type":"code","source":"train_files = glob.glob('../input/tpu-getting-started/*/train/*.tfrec')\ntest_files = glob.glob('../input/tpu-getting-started/*/test/*.tfrec')\nval_files = glob.glob('../input/tpu-getting-started/*/val/*.tfrec')","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:21.997352Z","iopub.execute_input":"2022-09-08T11:29:21.997713Z","iopub.status.idle":"2022-09-08T11:29:22.199968Z","shell.execute_reply.started":"2022-09-08T11:29:21.997672Z","shell.execute_reply":"2022-09-08T11:29:22.198972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Review some file name results","metadata":{}},{"cell_type":"markdown","source":"### train_files sample files","metadata":{}},{"cell_type":"code","source":"print(train_files[:10])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.201431Z","iopub.execute_input":"2022-09-08T11:29:22.201861Z","iopub.status.idle":"2022-09-08T11:29:22.207510Z","shell.execute_reply.started":"2022-09-08T11:29:22.201829Z","shell.execute_reply":"2022-09-08T11:29:22.206470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### val_files sample files","metadata":{}},{"cell_type":"code","source":"print(val_files[:10])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.210928Z","iopub.execute_input":"2022-09-08T11:29:22.211776Z","iopub.status.idle":"2022-09-08T11:29:22.220657Z","shell.execute_reply.started":"2022-09-08T11:29:22.211741Z","shell.execute_reply":"2022-09-08T11:29:22.219888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### test_files sample files","metadata":{}},{"cell_type":"code","source":"print(test_files[:10])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.222419Z","iopub.execute_input":"2022-09-08T11:29:22.223005Z","iopub.status.idle":"2022-09-08T11:29:22.232693Z","shell.execute_reply.started":"2022-09-08T11:29:22.222934Z","shell.execute_reply":"2022-09-08T11:29:22.231812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create a dictionary describing the features.","metadata":{}},{"cell_type":"code","source":"train_feature_description = {\n 'class': tf.io.FixedLenFeature([], tf.int64),\n 'id': tf.io.FixedLenFeature([], tf.string),\n 'image': tf.io.FixedLenFeature([], tf.string),\n}","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.233843Z","iopub.execute_input":"2022-09-08T11:29:22.234394Z","iopub.status.idle":"2022-09-08T11:29:22.243659Z","shell.execute_reply.started":"2022-09-08T11:29:22.234327Z","shell.execute_reply":"2022-09-08T11:29:22.242642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_feature_description = {\n 'class': tf.io.FixedLenFeature([], tf.int64),\n 'id': tf.io.FixedLenFeature([], tf.string),\n 'image': tf.io.FixedLenFeature([], tf.string),\n}\n","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.245102Z","iopub.execute_input":"2022-09-08T11:29:22.245512Z","iopub.status.idle":"2022-09-08T11:29:22.255289Z","shell.execute_reply.started":"2022-09-08T11:29:22.245469Z","shell.execute_reply":"2022-09-08T11:29:22.254587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_feature_description = {\n 'id': tf.io.FixedLenFeature([], tf.string),\n 'image': tf.io.FixedLenFeature([], tf.string),\n}","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.256630Z","iopub.execute_input":"2022-09-08T11:29:22.257049Z","iopub.status.idle":"2022-09-08T11:29:22.268668Z","shell.execute_reply.started":"2022-09-08T11:29:22.257006Z","shell.execute_reply":"2022-09-08T11:29:22.267685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create parse image function","metadata":{}},{"cell_type":"code","source":"def train_parse_image_function(example_proto):\n # Parse the input tf.Example proto using the dictionary above.\n return tf.io.parse_single_example(example_proto, train_feature_description)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.270011Z","iopub.execute_input":"2022-09-08T11:29:22.270276Z","iopub.status.idle":"2022-09-08T11:29:22.286173Z","shell.execute_reply.started":"2022-09-08T11:29:22.270245Z","shell.execute_reply":"2022-09-08T11:29:22.285261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def val_parse_image_function(example_proto):\n # Parse the input tf.Example proto using the dictionary above.\n return tf.io.parse_single_example(example_proto, val_feature_description)\n","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.288086Z","iopub.execute_input":"2022-09-08T11:29:22.289020Z","iopub.status.idle":"2022-09-08T11:29:22.299115Z","shell.execute_reply.started":"2022-09-08T11:29:22.288980Z","shell.execute_reply":"2022-09-08T11:29:22.298130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_parse_image_function(example_proto):\n # Parse the input tf.Example proto using the dictionary above.\n return tf.io.parse_single_example(example_proto, test_feature_description)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.300325Z","iopub.execute_input":"2022-09-08T11:29:22.301011Z","iopub.status.idle":"2022-09-08T11:29:22.313137Z","shell.execute_reply.started":"2022-09-08T11:29:22.300970Z","shell.execute_reply":"2022-09-08T11:29:22.312160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Processing for all train image dataset","metadata":{}},{"cell_type":"code","source":"train_ids = []\ntrain_class = []\ntrain_images = []\n\nfor i in train_files:\n train_image_dataset = tf.data.TFRecordDataset(i)\n\n train_image_dataset = train_image_dataset.map(train_parse_image_function)\n\n # [2:-1] is done to remove b' from 1st and 'from last in train id names\n ids = [str(id_features['id'].numpy())[2:-1] for id_features in train_image_dataset] \n train_ids = train_ids + ids\n\n classes = [int(class_features['class'].numpy()) for class_features in train_image_dataset]\n train_class = train_class + classes\n\n images = [image_features['image'].numpy() for image_features in train_image_dataset]\n train_images = train_images + images","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:29:22.314672Z","iopub.execute_input":"2022-09-08T11:29:22.315337Z","iopub.status.idle":"2022-09-08T11:30:59.823699Z","shell.execute_reply.started":"2022-09-08T11:29:22.315304Z","shell.execute_reply":"2022-09-08T11:30:59.822695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Processing for all val image dataset","metadata":{}},{"cell_type":"code","source":"val_ids = []\nval_class = []\nval_images = []\n\nfor i in val_files:\n val_image_dataset = tf.data.TFRecordDataset(i)\n\n val_image_dataset = val_image_dataset.map(val_parse_image_function)\n\n ids = [str(image_features['id'].numpy())[2:-1] for image_features in val_image_dataset]\n val_ids += ids\n\n classes = [int(image_features['class'].numpy()) for image_features in val_image_dataset]\n val_class += classes \n\n images = [image_features['image'].numpy() for image_features in val_image_dataset]\n val_images += images","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:30:59.826737Z","iopub.execute_input":"2022-09-08T11:30:59.827630Z","iopub.status.idle":"2022-09-08T11:31:29.577850Z","shell.execute_reply.started":"2022-09-08T11:30:59.827586Z","shell.execute_reply":"2022-09-08T11:31:29.576792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Processing for all test image dataset","metadata":{}},{"cell_type":"code","source":"test_ids = []\ntest_images = []\nfor i in test_files:\n test_image_dataset = tf.data.TFRecordDataset(i)\n \n test_image_dataset = test_image_dataset.map(test_parse_image_function)\n\n ids = [str(id_features['id'].numpy())[2:-1] for id_features in test_image_dataset]\n test_ids = test_ids + ids\n\n images = [image_features['image'].numpy() for image_features in test_image_dataset]\n test_images = test_images + images","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:31:29.579344Z","iopub.execute_input":"2022-09-08T11:31:29.579636Z","iopub.status.idle":"2022-09-08T11:32:14.223496Z","shell.execute_reply.started":"2022-09-08T11:31:29.579604Z","shell.execute_reply":"2022-09-08T11:32:14.222403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### show some example image data","metadata":{}},{"cell_type":"code","source":"import IPython.display as display\n\nfor i in range(6):\n display.display(display.Image(data=train_images[i]))","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.224976Z","iopub.execute_input":"2022-09-08T11:32:14.225292Z","iopub.status.idle":"2022-09-08T11:32:14.233829Z","shell.execute_reply.started":"2022-09-08T11:32:14.225254Z","shell.execute_reply":"2022-09-08T11:32:14.232956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import IPython.display as display\nfor i in range(6):\n display.display(display.Image(data=val_images[i]))","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.235398Z","iopub.execute_input":"2022-09-08T11:32:14.236098Z","iopub.status.idle":"2022-09-08T11:32:14.249083Z","shell.execute_reply.started":"2022-09-08T11:32:14.236061Z","shell.execute_reply":"2022-09-08T11:32:14.248086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import IPython.display as display\nfor i in range(6):\n display.display(display.Image(data=test_images[i]))","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.250297Z","iopub.execute_input":"2022-09-08T11:32:14.250598Z","iopub.status.idle":"2022-09-08T11:32:14.262495Z","shell.execute_reply.started":"2022-09-08T11:32:14.250538Z","shell.execute_reply":"2022-09-08T11:32:14.261640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Data preparation","metadata":{}},{"cell_type":"markdown","source":"Write TFDataset\nThis class has id, label(class), images","metadata":{}},{"cell_type":"code","source":"class TFDataset():\n def __init__(self, ids, labels, imgs, transforms, is_test=False):\n self.ids = ids\n if not is_test:\n self.labels = labels\n self.imgs = imgs\n self.transforms = transforms\n self.is_test = is_test\n \n def __len__(self):\n return len(self.ids)\n\n def __getitem__(self, index):\n img = self.imgs[index]\n img = Image.open(io.BytesIO(img))\n img = self.transforms(img)\n if self.is_test:\n return img, -1, self.ids[index]\n return img, int(self.labels[index]), self.ids[index]","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.263828Z","iopub.execute_input":"2022-09-08T11:32:14.264079Z","iopub.status.idle":"2022-09-08T11:32:14.274120Z","shell.execute_reply.started":"2022-09-08T11:32:14.264050Z","shell.execute_reply":"2022-09-08T11:32:14.272873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's write augmentation and normalization right away","metadata":{}},{"cell_type":"code","source":"train_transforms = transforms.Compose([\n transforms.RandomResizedCrop(224),\n transforms.RandomHorizontalFlip(),\n transforms.RandomVerticalFlip(),\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n transforms.RandomErasing()\n ])\n\ntest_transforms = transforms.Compose([\n transforms.CenterCrop(224),\n transforms.Resize(224),\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.275724Z","iopub.execute_input":"2022-09-08T11:32:14.276145Z","iopub.status.idle":"2022-09-08T11:32:14.286071Z","shell.execute_reply.started":"2022-09-08T11:32:14.276097Z","shell.execute_reply":"2022-09-08T11:32:14.285134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Declare train_ds and valid_ds TFDataset object","metadata":{}},{"cell_type":"code","source":"train_ds = TFDataset(train_ids, train_class, train_images, train_transforms)\nvalid_ds = TFDataset(val_ids, val_class, val_images, test_transforms)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.288138Z","iopub.execute_input":"2022-09-08T11:32:14.288391Z","iopub.status.idle":"2022-09-08T11:32:14.302192Z","shell.execute_reply.started":"2022-09-08T11:32:14.288363Z","shell.execute_reply":"2022-09-08T11:32:14.301128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create XLA(Accelerated Linear Algebra) device","metadata":{}},{"cell_type":"code","source":"device = xm.xla_device()","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:14.304148Z","iopub.execute_input":"2022-09-08T11:32:14.304484Z","iopub.status.idle":"2022-09-08T11:32:20.322903Z","shell.execute_reply.started":"2022-09-08T11:32:14.304444Z","shell.execute_reply":"2022-09-08T11:32:20.321831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create DataLoader object for train and valid set","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import Dataset, DataLoader, ConcatDataset\n\n#create DistributedSampler for train\ntrain_sampler = torch.utils.data.distributed.DistributedSampler(\n train_ds,\n num_replicas=xm.xrt_world_size(),\n rank=xm.get_ordinal(),\n shuffle=True\n )\n#create DistributedSampler for valid\nvalid_sampler = torch.utils.data.distributed.DistributedSampler(\n valid_ds,\n num_replicas=xm.xrt_world_size(),\n rank=xm.get_ordinal(),\n shuffle=True\n )\n#create DataLoader for train and valid \ntrain_loader = DataLoader(train_ds, 128, sampler=train_sampler, num_workers=4, pin_memory=True)\nval_loader = DataLoader(valid_ds, 128, sampler=valid_sampler, num_workers=4, pin_memory=True)\n\ndataset_sizes = {\n 'train': len(train_ds),\n 'val': len(valid_ds),\n}","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:20.328121Z","iopub.execute_input":"2022-09-08T11:32:20.328414Z","iopub.status.idle":"2022-09-08T11:32:20.340699Z","shell.execute_reply.started":"2022-09-08T11:32:20.328382Z","shell.execute_reply":"2022-09-08T11:32:20.339751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### just display dataset size (train and val size):","metadata":{}},{"cell_type":"code","source":"dataset_sizes","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:20.342242Z","iopub.execute_input":"2022-09-08T11:32:20.342515Z","iopub.status.idle":"2022-09-08T11:32:20.356062Z","shell.execute_reply.started":"2022-09-08T11:32:20.342478Z","shell.execute_reply":"2022-09-08T11:32:20.354931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Although, we have already normalized the data, but this datasets, we have three-channel images (RED, GREEN, BLUE), that is, we need to normalize for each channel separately:","metadata":{}},{"cell_type":"code","source":"#Example for dataset with 3 channels\ntransforms_example = transforms.Compose([\n transforms.CenterCrop(224),\n transforms.ToTensor(),])\n\nexampleset = TFDataset(train_ids, train_class, train_images, transforms_example)\n\nx, y, _ = next(iter(DataLoader(exampleset)))\n\nchannels = ['Red', 'Green', 'Blue']\ncmaps = [plt.cm.Reds_r, plt.cm.Greens_r, plt.cm.Blues_r]\n\nfig, ax = plt.subplots(1, 4, figsize=(15, 10))\n\nfor i, axs in enumerate(fig.axes[:3]):\n axs.imshow(x[0][i,:,:], cmap=cmaps[i])\n axs.set_title(f'{channels[i]} Channel')\n axs.set_xticks([])\n axs.set_yticks([])\n \nax[3].imshow(x[0].permute(1,2,0))\nax[3].set_title('Three Channels')\nax[3].set_xticks([])\nax[3].set_yticks([]);","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:20.357777Z","iopub.execute_input":"2022-09-08T11:32:20.358455Z","iopub.status.idle":"2022-09-08T11:32:21.059510Z","shell.execute_reply.started":"2022-09-08T11:32:20.358351Z","shell.execute_reply":"2022-09-08T11:32:21.058619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code below show how to normalize the data for each channel for the test, training and validation datasets:","metadata":{}},{"cell_type":"code","source":"channels = 3\n\nloaders = {\n 'train':train_loader,\n 'val':val_loader,\n}\n\nfor channel in range(channels):\n for x in ['train', 'val']:\n #number of pixels in the dataset = number of all pixels in one object * number of all objects in the dataset\n num_pxl = dataset_sizes[x]*224*224\n \n #we go through the butches and sum up the pixels of the objects, \n #which then divide the sum by the number of all pixels to calculate the average\n total_sum = 0\n for batch in loaders[x]:\n layer = list(map(itemgetter(channel), batch[0]))\n layer = torch.stack(layer, dim=0)\n total_sum += layer.sum()\n mean = total_sum / num_pxl\n\n #we calculate the standard deviation using the formula that I indicated above\n sum_sqrt = 0\n for batch in loaders[x]: \n layer = list(map(itemgetter(channel), batch[0]))\n sum_sqrt += ((torch.stack(layer, dim=0) - mean).pow(2)).sum()\n std = torch.sqrt(sum_sqrt / num_pxl)\n \n print(f'|channel:{channel+1}| {x} - mean: {mean}, std: {std}')","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:32:21.061036Z","iopub.execute_input":"2022-09-08T11:32:21.061528Z","iopub.status.idle":"2022-09-08T11:47:13.648408Z","shell.execute_reply.started":"2022-09-08T11:32:21.061483Z","shell.execute_reply":"2022-09-08T11:47:13.647436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show the pixel distribution after normalization:","metadata":{}},{"cell_type":"code","source":"x, y, _ = next(iter(exampleset))\n\ndef plotHist(img):\n plt.figure(figsize=(10,5))\n plt.subplot(1,2,1)\n plt.imshow(x.permute(1,2,0))\n plt.axis('off')\n histo = plt.subplot(1,2,2)\n histo.set_ylabel('Count')\n histo.set_xlabel('Pixel Intensity')\n plt.hist(img.numpy().flatten(), bins=10, lw=0, alpha=0.5, color='r')\n#show histogram\nplotHist(x)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:13.651076Z","iopub.execute_input":"2022-09-08T11:47:13.651495Z","iopub.status.idle":"2022-09-08T11:47:14.080387Z","shell.execute_reply.started":"2022-09-08T11:47:13.651446Z","shell.execute_reply":"2022-09-08T11:47:14.079482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Show mean and standard deviation:","metadata":{}},{"cell_type":"code","source":"x.mean(), x.std()","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:14.081727Z","iopub.execute_input":"2022-09-08T11:47:14.081976Z","iopub.status.idle":"2022-09-08T11:47:14.200213Z","shell.execute_reply.started":"2022-09-08T11:47:14.081947Z","shell.execute_reply":"2022-09-08T11:47:14.199599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create norm_out function","metadata":{}},{"cell_type":"code","source":"def norm_out(img):\n \n img = img.permute(1,2,0)\n mean = torch.FloatTensor([0.485, 0.456, 0.406])\n std = torch.FloatTensor([0.229, 0.224, 0.225])\n \n img = img*std + mean\n \n return np.clip(img,0,1)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:14.201456Z","iopub.execute_input":"2022-09-08T11:47:14.202156Z","iopub.status.idle":"2022-09-08T11:47:14.207250Z","shell.execute_reply.started":"2022-09-08T11:47:14.202123Z","shell.execute_reply":"2022-09-08T11:47:14.206515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create show batch function","metadata":{}},{"cell_type":"code","source":"def show_batch(dl):\n \n for images, labels, _ in dl:\n fig, ax = plt.subplots(figsize=(25, 25))\n ax.set_xticks([]); ax.set_yticks([])\n #images = norm_out(images[:60])\n ax.imshow(norm_out(make_grid(images[:100], nrow=10)))#.permute(1, 2, 0))\n ax.set_title('Images without augmentation', fontsize=40)\n break\n \nshow_batch(loaders['val'])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:14.208732Z","iopub.execute_input":"2022-09-08T11:47:14.209223Z","iopub.status.idle":"2022-09-08T11:47:20.198524Z","shell.execute_reply.started":"2022-09-08T11:47:14.209189Z","shell.execute_reply":"2022-09-08T11:47:20.197022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_batch(dl):\n for images, labels, _ in dl:\n fig, ax = plt.subplots(figsize=(25, 25))\n ax.set_xticks([]); ax.set_yticks([])\n ax.imshow(make_grid(images[:100], nrow=10).permute(1, 2, 0))\n ax.set_title('Images with augmentation', fontsize=40)\n break\n \nshow_batch(loaders['train'])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:20.200886Z","iopub.execute_input":"2022-09-08T11:47:20.203903Z","iopub.status.idle":"2022-09-08T11:47:26.300442Z","shell.execute_reply.started":"2022-09-08T11:47:20.203854Z","shell.execute_reply":"2022-09-08T11:47:26.297153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Training and Test\n\n","metadata":{}},{"cell_type":"markdown","source":"We will use an ensemble of pre-trained models:\n\n - first, we train only the classifier on 10 epochs\n - then unfreeze the network and train all together for another 10 epochs. \n - finally, the model makes predictions on the test data","metadata":{}},{"cell_type":"markdown","source":"### Create a function Train for one epoch","metadata":{}},{"cell_type":"code","source":"def train(loader, epoch, model, optimizer, criterion):\n #tracker = xm.RateTracker()\n model.train()\n running_loss = 0.\n running_corrects = 0.\n tot = 0\n for i, (ip, tgt, _) in enumerate(loader):\n ip, tgt = ip.to(device), tgt.to(device) \n output = model(ip)\n loss = criterion(output, tgt)\n tot += ip.shape[0]\n\n # Append outputs\n _, pred = output.max(dim=1)\n running_corrects += torch.sum(pred == tgt.data)\n\n # compute gradient and do SGD step\n optimizer.zero_grad()\n loss.backward()\n #torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n #optimizer.step()\n xm.optimizer_step(optimizer)\n\n running_loss += loss.item()*ip.size(0)\n\n return running_corrects, running_loss","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.302257Z","iopub.execute_input":"2022-09-08T11:47:26.302590Z","iopub.status.idle":"2022-09-08T11:47:26.311035Z","shell.execute_reply.started":"2022-09-08T11:47:26.302552Z","shell.execute_reply":"2022-09-08T11:47:26.309914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a function Test for one epoch","metadata":{}},{"cell_type":"code","source":"def test(loader, model, criterion):\n with torch.no_grad():\n model.eval()\n running_loss = 0.\n running_corrects = 0.\n tot = 0\n for i, (ip, tgt, _) in enumerate(loader):\n ip, tgt = ip.to(device), tgt.to(device)\n output = model(ip)\n loss = criterion(output, tgt)\n tot += ip.shape[0]\n _, pred = output.max(dim=1)\n running_corrects += torch.sum(pred == tgt.data)\n running_loss += loss.item()*ip.size(0)\n\n return running_corrects, running_loss","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.312509Z","iopub.execute_input":"2022-09-08T11:47:26.312840Z","iopub.status.idle":"2022-09-08T11:47:26.327144Z","shell.execute_reply.started":"2022-09-08T11:47:26.312808Z","shell.execute_reply":"2022-09-08T11:47:26.326305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create a function for predictions on a test set:","metadata":{}},{"cell_type":"code","source":"def predict(model, loader, device):\n with torch.no_grad():\n torch.cuda.empty_cache()\n model.eval()\n preds = dict()\n for i, (ip, _, ids) in enumerate(loader):\n ip = ip.to(device)\n output = model(ip)\n _, pred = output.max(dim=1)\n for i, j in zip(ids, pred.cpu().detach()):\n preds[i] = j\n \n return preds","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.328422Z","iopub.execute_input":"2022-09-08T11:47:26.328824Z","iopub.status.idle":"2022-09-08T11:47:26.339044Z","shell.execute_reply.started":"2022-09-08T11:47:26.328793Z","shell.execute_reply":"2022-09-08T11:47:26.338317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to create losses and accuracies to store history of quality of training model to visualization:","metadata":{}},{"cell_type":"code","source":"losses = {'train':[], 'val':[]}\naccuracies = {'train':[], 'val':[]}","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.340381Z","iopub.execute_input":"2022-09-08T11:47:26.340820Z","iopub.status.idle":"2022-09-08T11:47:26.353394Z","shell.execute_reply.started":"2022-09-08T11:47:26.340786Z","shell.execute_reply":"2022-09-08T11:47:26.352254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We create Fit function. In a typical XLA:TPU training scenario we’re training on multiple TPU cores in parallel (a single Cloud TPU device includes 8 TPU cores). So we need to ensure that all the gradients are exchanged between the data parallel replicas by consolidating the gradients and taking an optimizer step. For this we provide the xm.optimizer_step(optimizer) which does the gradient consolidation and step-taking\n\n","metadata":{}},{"cell_type":"code","source":"def fit(seed, epochs, model):\n\n # Train and valid dataloaders\n xm.master_print('Creating a model {}...'.format(seed))\n device = xm.xla_device()\n WRAPPED_MODEL = xmp.MpModelWrapper(model)\n model = WRAPPED_MODEL.to(device)\n model.to(device) \n criterion = nn.CrossEntropyLoss()\n #define learning rate\n lr=0.05\n if seed==1:\n optimizer = torch.optim.Adam(model.head.parameters(), lr=lr* xm.xrt_world_size(), betas=(0.9, 0.999), eps=1e-08, weight_decay=0)\n if seed==2 or seed==3:\n optimizer = torch.optim.Adam(model.fc.parameters(), lr=lr* xm.xrt_world_size(), betas=(0.9, 0.999), eps=1e-08, weight_decay=0)\n if seed==4 or seed==0:\n optimizer = torch.optim.Adam(model.classifier.parameters(), lr=lr* xm.xrt_world_size(), betas=(0.9, 0.999), eps=1e-08, weight_decay=0)\n # scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.7, patience=3, verbose=True)\n\n scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 4, gamma=0.1)\n since = time.time()\n best_model = copy.deepcopy(model.state_dict())\n best_acc = 0.0\n \n for epoch in range(epochs):\n \n #train\n xm.master_print('Epoch: {}/{}'.format(epoch+1, epochs))\n para_loader = pl.ParallelLoader(train_loader, [device])\n running_corrects, running_loss = train(para_loader.per_device_loader(device), epoch, model, optimizer, criterion)\n epoch_loss = running_loss / dataset_sizes['train']\n epoch_acc = running_corrects/dataset_sizes['train']\n losses['train'].append(epoch_loss)\n accuracies['train'].append(epoch_acc)\n xm.master_print('{} - loss:{}, accuracy:{}'.format('train', epoch_loss, epoch_acc))\n\n #val\n para_loader = pl.ParallelLoader(val_loader, [device])\n running_corrects, running_loss = test(para_loader.per_device_loader(device), model, criterion)\n epoch_loss = running_loss / dataset_sizes['val']\n epoch_acc = running_corrects/dataset_sizes['val']\n losses['val'].append(epoch_loss)\n accuracies['val'].append(epoch_acc)\n xm.master_print('{} - loss:{}, accuracy:{}'.format('val', epoch_loss, epoch_acc))\n\n #epoch end\n xm.master_print('Time: {}m {}s'.format((time.time()- since)//60, (time.time()- since)%60))\n xm.master_print('=='*31)\n if epoch_acc > best_acc:\n best_acc = epoch_acc\n best_model = copy.deepcopy(model.state_dict())\n scheduler.step()\n \n time_elapsed = time.time() - since\n xm.master_print('CLASSIFIER TRAINING TIME {}m {}s'.format(time_elapsed//60, time_elapsed%60))\n xm.master_print('=='*31)\n\n\n model.load_state_dict(best_model)\n\n for param in model.parameters():\n param.requires_grad=True\n\n optimizer = torch.optim.Adam(model.parameters(), lr=lr* xm.xrt_world_size(), betas=(0.9, 0.999), eps=1e-08, weight_decay=0) \n# scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.7, patience=3, verbose=True)\n scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 4, gamma=0.1)\n for epoch in range(epochs):\n\n #train\n xm.master_print('Epoch: {}/{}'.format(epoch+1, epochs))\n para_loader = pl.ParallelLoader(train_loader, [device])\n running_corrects, running_loss = train(para_loader.per_device_loader(device), epoch, model, optimizer, criterion)\n epoch_loss = running_loss / dataset_sizes['train']\n epoch_acc = running_corrects/dataset_sizes['train']\n losses['train'].append(epoch_loss)\n accuracies['train'].append(epoch_acc)\n xm.master_print('{} - loss:{}, accuracy:{}'.format('train', epoch_loss, epoch_acc))\n\n #val\n para_loader = pl.ParallelLoader(val_loader, [device])\n running_corrects, running_loss = test(para_loader.per_device_loader(device), model, criterion)\n epoch_loss = running_loss / dataset_sizes['val']\n epoch_acc = running_corrects/dataset_sizes['val']\n losses['val'].append(epoch_loss)\n accuracies['val'].append(epoch_acc)\n xm.master_print('{} - loss:{}, accuracy:{}'.format('val', epoch_loss, epoch_acc))\n\n #epoch end\n xm.master_print('Time: {}m {}s'.format((time.time()- since)//60, (time.time()- since)%60))\n xm.master_print('=='*31)\n if epoch_acc > best_acc:\n best_acc = epoch_acc\n best_model = copy.deepcopy(model.state_dict())\n scheduler.step()\n\n time_elapsed = time.time() - since\n xm.master_print('ALL NET TRAINING TIME {}m {}s'.format(time_elapsed//60, time_elapsed%60))\n xm.master_print('=='*31)\n\n model.load_state_dict(best_model)\n \n predictions = predict(model, testloader, device)\n \n for key in predictions.keys():\n ensemble_df.loc[ensemble_df['id'] == key, 'model_' + str(seed + 1)] = int((predictions[key]).item())\n \n xm.master_print('Prediction Saved! \\n')","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.354887Z","iopub.execute_input":"2022-09-08T11:47:26.355345Z","iopub.status.idle":"2022-09-08T11:47:26.382260Z","shell.execute_reply.started":"2022-09-08T11:47:26.355309Z","shell.execute_reply":"2022-09-08T11:47:26.381126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MODELS","metadata":{}},{"cell_type":"markdown","source":"In this term project, I refer to https://www.kaggle.com/code/georgiisirotenko/pytorch-tpu-baseline-flowers-tranlearning-ensemble and use, setup configuration for pretrained models to classify big data images with TPU & PyTorch\n\nYou can change other models in the List Models of PyTorch site (in this term project I keep the models that I referred):\n\nhttps://pytorch.org/vision/0.13/models.html (they provide instructions how to use the models)\n\n","metadata":{}},{"cell_type":"markdown","source":"## 1. DenseNet","metadata":{}},{"cell_type":"code","source":"densenet121 = torchvision.models.densenet121(pretrained=True)\nfor param in densenet121.parameters():\n param.requires_grad=False\n\ndensenet121.classifier = nn.Linear(in_features=densenet121.classifier.in_features, out_features=104, bias=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:26.383790Z","iopub.execute_input":"2022-09-08T11:47:26.384381Z","iopub.status.idle":"2022-09-08T11:47:29.772570Z","shell.execute_reply.started":"2022-09-08T11:47:26.384338Z","shell.execute_reply":"2022-09-08T11:47:29.771656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. ViT","metadata":{}},{"cell_type":"code","source":"ViT = timm.create_model(\"vit_base_patch16_224\", pretrained=True)\nfor param in ViT.parameters():\n param.requires_grad=False\n\nViT.head = nn.Linear(ViT.head.in_features, 104)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:29.773929Z","iopub.execute_input":"2022-09-08T11:47:29.774264Z","iopub.status.idle":"2022-09-08T11:47:35.261171Z","shell.execute_reply.started":"2022-09-08T11:47:29.774235Z","shell.execute_reply":"2022-09-08T11:47:35.260034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. GoogLeNet","metadata":{}},{"cell_type":"code","source":"googlenet = torchvision.models.googlenet(pretrained=True)\nfor param in googlenet.parameters():\n param.grad_requires = False\n\ngooglenet.fc = nn.Linear(in_features=googlenet.fc.in_features, out_features=104, bias=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:35.262644Z","iopub.execute_input":"2022-09-08T11:47:35.263407Z","iopub.status.idle":"2022-09-08T11:47:40.578600Z","shell.execute_reply.started":"2022-09-08T11:47:35.263370Z","shell.execute_reply":"2022-09-08T11:47:40.577498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. ResNet","metadata":{}},{"cell_type":"code","source":"resnet101 = torchvision.models.resnet101(pretrained=True)\nfor param in resnet101.parameters():\n param.grad_requires = False\n\nresnet101.fc = nn.Linear(in_features=resnet101.fc.in_features, out_features=104, bias=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:40.580003Z","iopub.execute_input":"2022-09-08T11:47:40.580780Z","iopub.status.idle":"2022-09-08T11:47:53.609044Z","shell.execute_reply.started":"2022-09-08T11:47:40.580736Z","shell.execute_reply":"2022-09-08T11:47:53.608243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. VGG19","metadata":{}},{"cell_type":"code","source":"vgg19_bn = torchvision.models.vgg19_bn(pretrained=True)\nfor param in vgg19_bn.parameters():\n param.grad_requires = False\n\nvgg19_bn.classifier[6] = nn.Linear(4096, 104, bias=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:47:53.610343Z","iopub.execute_input":"2022-09-08T11:47:53.610609Z","iopub.status.idle":"2022-09-08T11:48:26.523219Z","shell.execute_reply.started":"2022-09-08T11:47:53.610581Z","shell.execute_reply":"2022-09-08T11:48:26.522173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we have 5 models, now we will process training","metadata":{}},{"cell_type":"markdown","source":"### Because competition's TPU max of 180 minutes. So I can not submit 5 models and 10 epochs (In version 4)\n\n### So I re run term project with 2 models and 5 epoch to test Submission in version 8","metadata":{}},{"cell_type":"code","source":"test_transforms = transforms.Compose([\n transforms.CenterCrop(224),\n transforms.Resize(224),\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n ])\n\ntest_ds = TFDataset(test_ids, [], test_images, test_transforms, True)\ntestloader = DataLoader(test_ds, 128, num_workers=4, pin_memory=True, shuffle=False)\n\n#read sample submission file\nsubmit_df = pd.read_csv('../input/tpu-getting-started/sample_submission.csv')\nensemble_df = submit_df.copy()\n\n#------You open this comments to run all 5 models with 10 epochs----------#\nnum_models = 5\n#We run 10 epochs for each model\nnum_epochs = 10\n\nmodels = [densenet121, ViT, googlenet, resnet101, vgg19_bn]\n\n\n#------change for submission----------#\n#num_models = 2\n#num_epochs = 5\n\n#models = [densenet121, ViT]\n\nfor seed in range(num_models):\n preds = fit(seed=seed, epochs=num_epochs, model=models[seed])","metadata":{"execution":{"iopub.status.busy":"2022-09-08T11:48:26.524953Z","iopub.execute_input":"2022-09-08T11:48:26.525280Z","iopub.status.idle":"2022-09-08T20:11:49.669520Z","shell.execute_reply.started":"2022-09-08T11:48:26.525234Z","shell.execute_reply":"2022-09-08T20:11:49.666533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **5. Accuracy Visualization**","metadata":{}},{"cell_type":"markdown","source":"We visualize the accuracy of training models:","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(15, 15))\nmodelname = ['DenseNet', 'ViT', 'GoogLeNet', 'ResNet101', 'VGG19 with BN']\n\nepochs=10\n\n#fig, ax = plt.subplots(2, 2, figsize=(10, 10))\n#modelname = ['Dense Convolutional Network(DenseNet)', 'Vision Transformer (ViT)'] \n#epochs=5\n\ni=0\n\nfor row in range(num_models):\n step=20\n #step=10\n epoch_list = list(range(1,epochs*2+1))\n\n ax[row][0].plot(epoch_list, accuracies['train'][i:step+i], '-o', label='Train Accuracy')\n ax[row][0].plot(epoch_list, accuracies['val'][i:step+i], '-o', label='Validation Accuracy')\n ax[row][0].plot([epochs for x in range(step)], np.linspace(min(accuracies['train'][i:step+i]).cpu(), max(accuracies['train'][i:step+i]).cpu(), step), color='r', label='Unfreeze net')\n ax[row][0].set_xticks(np.arange(0, epochs*2+1, 5))\n #ax[row][0].set_xticks(np.arange(0, epochs*2+1, 1))\n ax[row][0].set_ylabel('Accuracy Value')\n ax[row][0].set_xlabel('Epoch')\n ax[row][0].set_title('Accuracy {}'.format(modelname[row]))\n ax[row][0].legend(loc=\"best\")\n\n ax[row][1].plot(epoch_list, losses['train'][i:step+i], '-o', label='Train Loss')\n ax[row][1].plot(epoch_list, losses['val'][i:step+i], '-o',label='Validation Loss')\n ax[row][1].plot([epochs for x in range(step)], np.linspace(min(losses['train'][i:step+i]), max(losses['train'][i:step+i]), step), color='r', label='Unfreeze net')\n ax[row][1].set_xticks(np.arange(0, epochs*2+1, 5))\n #ax[row][1].set_xticks(np.arange(0, epochs*2+1, 1))\n ax[row][1].set_ylabel('Loss Value')\n ax[row][1].set_xlabel('Epoch')\n ax[row][1].set_title('Loss {}'.format(modelname[row]))\n ax[row][1].legend(loc=\"best\")\n fig.tight_layout()\n fig.subplots_adjust(top=1.5, wspace=0.3)\n\n i+=step","metadata":{"execution":{"iopub.status.busy":"2022-09-08T20:11:49.684294Z","iopub.execute_input":"2022-09-08T20:11:49.685280Z","iopub.status.idle":"2022-09-08T20:11:54.280908Z","shell.execute_reply.started":"2022-09-08T20:11:49.685183Z","shell.execute_reply":"2022-09-08T20:11:54.279750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Submit competition","metadata":{}},{"cell_type":"code","source":"ensemble_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T20:11:54.282469Z","iopub.execute_input":"2022-09-08T20:11:54.282801Z","iopub.status.idle":"2022-09-08T20:11:54.334985Z","shell.execute_reply.started":"2022-09-08T20:11:54.282763Z","shell.execute_reply":"2022-09-08T20:11:54.333982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate Final prediction\nfinal_pred = ensemble_df.iloc[:,2:].mode(axis=1).iloc[:,0]\nsubmit_df.label = final_pred.astype(int)\nsubmit_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T20:11:54.336493Z","iopub.execute_input":"2022-09-08T20:11:54.336788Z","iopub.status.idle":"2022-09-08T20:11:58.395841Z","shell.execute_reply.started":"2022-09-08T20:11:54.336752Z","shell.execute_reply":"2022-09-08T20:11:58.394821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the submission file\nsubmit_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-09-08T20:11:58.397156Z","iopub.execute_input":"2022-09-08T20:11:58.397497Z","iopub.status.idle":"2022-09-08T20:11:58.422443Z","shell.execute_reply.started":"2022-09-08T20:11:58.397466Z","shell.execute_reply":"2022-09-08T20:11:58.421669Z"},"trusted":true},"execution_count":null,"outputs":[]}]}