{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![image.png](attachment:bdb71b7b-a5e0-48c5-b217-55f947b21720.png)\n\n### What is W&B?\nShort, W&B is a place where you can very easy and quickly log on experiments:\n\n* your datasets\n* your visualisations and findings\n* your machine learning experiments (trying on multiple types of algorithms or hyperparameter tunning and logging the validation metric to see how they compare)\n* your final models to use later\n\nNo more messy local folders with everything in 1 place. No more struggle to remember the best parameters for a model because you forgot to write them down. It's your perfect \"Data Science Git\".","metadata":{},"attachments":{"bdb71b7b-a5e0-48c5-b217-55f947b21720.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA9IAAAEeCAIAAADpRLpvAAAgAElEQVR4nOzdaVwUZ9IA8OJwYIBRFEFOJYqgKIp4HyjeF+JJYsyaW41mTTbRrElMXt3EHJtoEtfERGNMNN7G+8ADEEVFUQRRUfBE5AzH4ADDNcz7YXZZVgR6uuvp7hnq/yG/bLa7ugQcqp+ursdCr9cDIYQQQgghhCVLqRMghBBCCCHE/FHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwR2U3IYQQQgghzFHZTQghhBBCCHNUdhNCCCGEEMIcld2EEEIIIYQwZy3NZXUa0CSAOgYqs6EiCyqzAQBUvQEAlL7QNhSUftIkRuRPmwr5h0GbBgCgSQAAULiBjTso3MAxBBxDpM2OEEIIIeSpLPR6vagXrMyCrPVQcLiJwxRu0G4WuDwvSk7ERORtg9zt/75Ja4RTKLjPBYW7KDkRQgghhHAiYtmt00Dedshab8QpCjfwWkTrlwTUMZCxqumCuy73ueDyPFipmOVECCGEEGIEscpunQZS5/27McBY7nPBbS52QsR0ZK2D7J/5nKj0Bb91VHkTQgghRA5EKbu1qZA6D3Ql/CM4hYL3cryEiOl4sLzplqRGWDmA3zp6VYAQQgghkmM/yUSngTuLBdXcAFBwGDJWIiVETIfAmhsAdCVwZzHoNEgJEUIIIYTwxL7sTp1rXEtuQ/J2QMEhhDjEVBQcElpzG1RmQ+o8hDiEEEIIIQIwLrsLDoH2Nlq0jFW0bNlc6DSQsQotmjaN7tkIIYQQIi2WZTdu5QQAuhLIWocZkMhW1jqhjUlPoHs2QgghhEiKZdldcAi5cgKAvB1QmYUck8hNZRbk7UCOqSuhBW9CCCGESIhl2Z3PpsoxbExIzJg6hknYfIxOcUIIIYQQXpiV3ZVZmF3ddTGqyYh8MLqz0qbRoxJCCCGESIVZ2c2uOFafZhWZyAS7bzHdsxFCCCFEIszK7mqWr6/Ru3FmjOmCNNMfS0IIIYSQhjEru5lWxmWpDIMTaVWwLLvpho0QQgghEmFWdpelsYpMCG/0Y0kIIYQQiTAru+18WUUmhDf6sSSEEEKIRJiV3VYqVpEBwM6PYXAiLRt3hsGZ/lgSQgghhDSMWdltzbK+oeLJjClYlt1MfywJIYQQQhrGrOx2DGEWeRiryEQm2H2L2f1YEkIIIYQ0yppVYIU7KDsz2TGHKiezp+rNZHS3sjPbpXRCCMFQVFSUkpLy8OHDzMzMrKyszMzM3Nzc4uJijUaj0WhKSkrKysocHBzs7e0dHBwcHBxatmzZpk0bNzc3Nzc3V1fXbt26BQYG2tvbS/3nIIQ8iVnZDQBtJ0HGN/hhVb3xYxJZcQxh8pPTdhJ+TEIIESw/P//IkSMXL15MSUm5efNmXl5ek6eUlJSUlJTk5uY2dIC3t3dgYGBQUJDhnx4eHqgpE0L4sNDr9axi6zRwbRLoSjBjuswEr8WYAYk8ZayEvB2YAa0cIOAQvRVACJGP+Pj4I0eOREREXLp0ifW1unTpMnPmzOeff97XlwY6ESIZlmU3ABQcggf/QIvWfCondQyoY6AyG8pS/33fYuUAdn6gcAPHkGbRZoN+z+a9DJxotZsQIr2MjIxNmzZt2bIlNVWCrd+CgoLmzp07b9488S9NCGFcdgNAyvNoHd5mXznpNJCxCgoON32kUyh4LTLzOxDEezZlZ/DfjhNKZkJCmsE9GAAAREREKJVKgUGys7Off/55lHxkrk2bNnv37kUJNXLkSJ1OhxJKKlu3bpVJi0VMTMynn34aHR0tdSLg7Oz8xhtvLFy40NnZWepczNmyZctOn2bwqpLsWVhYKJVK+/+ws7MzvI1geCHB8C/Ozs5dunSxs7OTOllRsS+7dRpIeR4qc4TGMfv2kqx1kLfdiPVdKwdoNwvc5rLMSWoPlnO6CWmcwhX8t5vrLYqFhYXUKYhEo9E4ODgIDHL37l0fHx+UfGSuXbt2OTmCP3UBAMDKyqqmpgYllFTS0tI6d+4sbQ6XL19+//33o6KipE3jCUqlcu7cuUuWLHFzc5M6F/M0bdq0ffv2SZ2FrLVr187Hx6dTp05dunQJDAzs16+fk5OT1EkxxL7sBgBtKqTOE9Qw4BQK3svxEpIZnQbuLgZNAp9zHUPAe5m51pQAgitvKwfwWwdKs91ficpuo1DZzQOV3QJlZWUtWLDgwIEDUiXAxcKFCz///HPhf8XIE6js5sHDw6N37959+/YNCQkZMmSI1OkgYza3uy6lHwQcAiXfTz23OWZec6fO41lzA4A6BlLngU6DmpOceC8Htzk8z1V2hoBDZlxzE0LkTKfTrVy50tfXV+Y1NwCsWbPG19d3586dUidibprPygiizMzMgwcPfvzxx8HBwfb29hMnTly1alVycrLUeeEQpewGACsV+K0Hl5nGnaVwhU4rwd2s3/xInQvaNEERtGmQatZfIvd50GklKFyNO8ttDvitN+fnAIQQGbt27VpAQMB7771XWloqdS6cZGdnz5w587XXXjP1Vn5ZEaOhwKyVlZUdPXp08eLFPXv29Pf3X7Zs2fXr16VOShCxym4AsFKB12IIOAhOoU0frHAFr3ch4LCZT+14sBznfVNtGjww3wcCAOAYAgGHwetdTsW3UygEHAT3eVRzE0Ik8csvv/Tv3//mzZtSJ2K0jRs3jhs3TqMx3yeo4qLVbkQ3b9785JNPAgICunbt+vXXXxcWFkqdER+i9HbXp9OAJgHUMVCRBZVZ/37h0iEIAMDOF9pOahaNAZrLkPYGZkDfn0DVBzOgPGlTIf8QlKUBAJRcAQBQuILCHWzcwTEEVL2bVbXdfD7TqbfbKNTbXZeYvd1arXbevHm///47VsD27dtPnjzZx8fHsANlu3btXF1dW7Zs+eDBg7y8vLy8vNzc3JycnLi4uFOnTpWVlaFcNCAgIDo6um3btijRmjPq7WbH1tZ25syZCxYs6Nu3r9S5GEGispsAQOrcf1eNWFS9wXcdZkAie1R2G4XKbh6o7OauoKBg7NixCQl839WpY8CAAWFhYZMmTerevTv3sw4fPrxixYqLFy8KT8DPz+/06dPt2rUTHqo5o7JbBEOHDv3HP/5hKuN0RWwyIXWpY5BrbgDQJIDmMnJMQgghHGRkZAwaNEh4zT1lypSUlJS4uLgPPvjAqJobAEJDQy9cuHDr1q25c4XOlk1NTR08ePCff/4pMA4hrJ05c2b48OHBwcG3bt2SOpemUdktEXUMk7DCp1wTQggxUmpqav/+/dPSBL0fHxgYGBsbu2/fvq5duwqJ4+fnt27duoyMjDFjxgiJc/fu3dDQ0IqKCiFBCBHH2bNnu3fvvmDBgvz8fKlzaQyV3RJhVHYzCksIIaQBjx49Gj58eHZ2Nu8ISqVy7dq1iYmJiFOKPT09jx8/vnXrViEt2vHx8S+99BJWSoQwpdPpfvzxRx8fn5UrV1ZVVUmdztNR2S0FzWVBmwc1QlcC2lQmkQkhhNSj0WhGjRolpObu1q3btWvX5s+fj5hVrVmzZl26dMnT05N3hJ07d65bR28NEZNRXFz83nvv+fv7JyUlSZ3LU9ArlVJAn2FSVzOZZ0IAAECtVj/4j/v379f+++PHj6VOjac2bdp06NDB29v7iX+2bt0aJX5BQcGDBw/S/6P234uKilDii8zS0tLNza19+/Yd/lfHjh2VSiXKJYqLi1NTU2//x507d27fvi3bL5ednZ2Pj0/nOnx9fZm+Fzh8+PCYGP6PGYcPH37o0CF7e3vElOp78OBBcHDwo0eP+J1ua2t79epVX19f3Kyag6qqKsOHzBP/zMrKovnorNnY2GzatOm5556TOpH/QWW3FLLWQfbPrIJ7vQsus1gFJyaisLDwqeV4SQmbxyzGc3FxqV9ed+rUCatYNFZZWdndu3fT6zD8gszLy5PDh6RCofDy8upQT/v27a2trcXPp6CgoLYEry3H1Wq1mDnY2Nh06tSp8//y8PAQc7zP0qVLP//8c96nT548ef/+/Yj5NOL+/fvBwcGZmZn8Tu/Vq9eVK9hjAJq3e/fuGT5nHj58aPiUNvxP8TP529/+1qpVK5RQlZWVmZmZWVlZ2dnZWVlZcrg//9vf/rZy5UorKyupE/k3KrulwLTsdptj5vt6EgHy8/OfWo5jjft9goWFhZubW/3y+plnnrGxsWFxRXSVlZWGL9ET5TijxSqVSlW/sO7QoYObm5v8h0Xm5+c/sSh++/ZtlAcvLVq08Pb29vX1rVthd+jQQdqvycWLFwcOHMj7d+iECRMOHDgg5l1TQkLCwIEDefe8rlu3TviAFNKkhw8fGj5k6pbjd+7cYXfFzMxMd3d3RsHv3Lnz6NGjzMzMzMzMxMTEs2fP8n7qwltwcPC+ffucnJxEvu5TUdkthbxtkPENq+C02k2M9PjxY6yljlq9e/fevn27aHuUSOL+/fuGKnzTpk1C2gxWrVrVsWNHQ5Hdpk0bxAzlIDc311CC3717NzU1dffu3dzP/e6772qLbHYZ8qPRaLp37/7w4UN+pw8ZMiQ2NhY3JS5++ukn3k3kLi4u9+7dY90PQxqSlZW1Z8+et956Cz1yXl6es7MzetiGZGVlnT17NiYmJjo6OjVVpLfR2rdvHxUVJYd9G6jslgL1dhOZGTZs2JkzZxAD2tjYlJSUSNL/IL5BgwbFxcXxPv3UqVOmstGDQHl5edzbrAcNGnTu3Dmm+QixaNGib77huXri5OR048YNqXaimT179pYtW/id+8UXX7z//vu4+RDuoqKiRo0ahR62sLAQ6+UZY+Xl5UVFRR07dmzPnj2lpaVMr+Xq6hobGyt55U2TTKRgzXL3cqbBiZmaMGECbsCKigo510yIDPtyC4mwd+9erGRk7sSJE9wPHj16NLtMBMrOzuZdcwPAtm3bJNz9cfXq1bxXrH/88UdT37LUpLVo0YJFWIVCwSIsFy4uLs8///ymTZvy8/N37949depUdtfKyckJDg5m2q7DBZXdUlD6gcKVSWSFKyj9mEQmZg297AaAyMhI9JgytGPHDoERjOq7MGlGld0Ct3phasWKFbzPnTJlirR/tDZt2ixevJjfuQ8fPjx06BBuPoQ7Ru8FyuGxpK2t7YwZM/bu3ZuSkhIWFsboKobKW9rNLKnslgijPhDHZvGomqALCAjw8PDAjXny5EncgPIkvGjOycm5ePEiSjIyd/z4cY5H2tvb9+vXj2kyvGVkZKxdu5b36UKWybEsXryYdy/vxo0bcZMh3DGqj2X1gnvXrl0PHDhw4cKFwMBAFvENlff9+/dZBOeCym6JOIUyCUtlN+ELfcE7Pj6eda+e5IR3mBg0hz6T5OTkvLw8jgePHj1aDitwT/X999/zPnfu3LnPPPMMYjL8ODg4vPPOO/zObSZPseRJPlPwWOvfv398fPxHH33E4o+cn58/bdq0yspK9MhcUNktEVUfcAhCjukQRC9TEt7Qy269Xm/2C967du1CeSt9586dwoPIHPelbpBxY3d1dfWGDRv4nWttbb106VLcfHibPXs2v/GLZWVlR44cQc+HcMGot1ueWrRo8emnn8bFxaE/iQWApKQk3q1WAlHZLR137AGo7RchByTNyZgxY9DXFcy+7MZqy05PT09OTkYJJVvm8T7lrl27CgsL+Z07fvz49u3b4+bDm6enJ++ZGBEREbjJEI5YLP3KfCJk3759L1++zKLhZM2aNfv27UMP2yQqu6Wj6gMuM9GiucyklymJEHZ2duhj7Mz7eXROTg7itBbz7jOpqKjgPqbay8tLhoO6DX755Rfe586aJa8dFV566SV+JzaTIUUyxGK1W8IxJhy5urqePXt23Lhx6JFfeukl8bcFpbJbUl6LwXEYQhzHYeAlzeMSYk7Gjx+PGzAtLS0jIwM3pnz88ccfiPsemHfZferUqYqKCo4Hs/j9iqK0tDQ6Opr36VOmTEFMRriJEyfyOzE5OVmr1eImQ7hgsdot25co6rK3tz9w4MCQIUNww2o0mtmzZ+PGbBKV3VLzXg5KYes6ys7gvRwpG9KssRgjaMZ9JriD/65du5aeno4YUFbMY3Tg6dOneZ87ZMgQW1tbxGSEc3R07NixI48Ta2pqEhIS0PMhTWq2ZTcAKBSKo0ePdu/eHTfs2bNnRW6aorJbalYq8N/Of7CJUyj4bwcr2iKHIOjatSuNEeQoJycHfXNv4SPAZYt72W1hYcFiHz4UQpqm0BfqUPTu3ZvfiaLt6U3qap5NJrVUKlVERAR6M/oHH3yAG7BxVHbLg/dy8F5m3B46ClfwXkbr3ATX5MmTcQMeO3YMN6BM7NmzB7HDxMBc+0zy8vJu3LjB8eC+ffs6OjoyzYe3qKgo3ufKcwx5nz48h189fPgQNxPCRXNe7Tbw9PT86quvcGNevXpVzCUPGZTdldlQeg20aVDN8/VwM+E0Cfy3g9ucpotvhSu4zQH/7eA0SZTM5Kq6ELRpUHoNKrOlTsV8oLd3q9XqpKQk3Jhy8Mcff6DHjI+Pz8nJQQ8rOaPmzcl2hklNTc3169d5n+7j44OYDJZu3brxO9GMG6LkjEWJbHJDCefPnz9w4EDcmEuXLtXpdLgxGyL6XU5JIpQkQmkKlGdA1Z+ge/zkAS3agcIZlD5g7w8tB7HaRF2erFTgPg/c54E2FfIPQVkaAIA2DQBA6QsAYOcLbSc104kllTnw+DyU3gDtXajMg6p6+25YtYQWzmDrBfb+4NALHHpJkaXJGzNmjEKhwN1HIDIyktF+Y1IpLCwU0ubbiF27dr311lssIkvIPEYH3rx5s6amhvfpvr6+iMlgadu2Lb8TMzMzcTMhXLAokU2u7LawsFi3bl2PHj0QY967d2/Tpk2vvvoqYsyGiFh25++H3K1Q3tSGnFW5UJULpdchfz8AQOuR0O4vYB8gQoIyovQDr2ZZWz9V6TXI3QJFTT3e1T0G3WMovwvqGAAA22eg3QvQVl6jA+RPoVAMHz7cqJ1NmnTy5EmpNiZgZOfOnegdJgZ79+41s7Jbr9dzbzSyt7cfPHgw03x4u3nzJu9zbW1tZbX/dq3WrVvzO7GgoAA3E8KFpSV+hwKLmKwFBASEhoYePnwYMeYvv/wiTtnN/sutewzZv8DVMZC+oumau76iKLj1Ctx6DYrPMEiOyJke1DFw61W49UrTNXd95fchfQVcHQM5G5/yUIU0DH2eyenTp6urq3FjSgt3hkldZ86c4b0bizwlJCSo1WqOB4eEhMi20zQlJUXqFPA5OTnxO5HKbkmw+NthohvOL1u2DDfg+fPn7927hxvzqRiX3YXH4Ho4ZP0otG+79CrceRfS3oCKR0iZEXmreARp8+HuYigVtnVfdSFkroXr4VCIuXxr3tDbuysqKs6cMZ/b5sLCwpiYGEbB9Xr9nj17GAWXhHmMDgSA3Nxc3ufKdkGRd9ldVFSEmwnhws7ODj2miZbdffr0GTp0KG7M33//HTfgUzH7LKgph3tL4P5HUI13T6y5DDeegyIjPsSJSSqKghvPgeYyWsDqAri/FO59ADXlaDHNV+fOnTt16oQb05y2q9y9ezejDhMDM5tnYh6N3QBQXs7/06OsrAwxE1wWFhZSp0C4EvJ2QUNke0/YJPSdbjZv3owb8KnYfLkrHsKtl/k0BjRJXwH3PoSMbwBEeueUiEpfBRmr4N4S0HPd0M4IRSfh1itQLvZOsKaI9/Z1DTGn6d3sOkwMTp48qdFomF5CNKWlpdw3EndxcenatSvTfIQQ+J6xPPus1Go1v3tIFsuupEksSmQTXe0GgOeeew73fdB79+7FxcUhBnwqBmV32S24+Qpo7+BHrpW3DW6/TSuX5kZXArffhrztDC+hvQ23XoMSM5xnhwu9vduoBl85Y9phYqDT6Q4cOMD0EqKJjo7mXm6i3+zh4r65PYvTGeH9IgGV3VJBb+823bJbpVKht0QePXoUN2B92GV32Q1InQu6YuSw9T2+AKnzmKyJEknUVEDaAtDEM7+QrhjSFkAp//m7zcHw4cNxty7T6/Xm0WeyZ88eEca7mk2fidl0mABAVVWVkNO1Wi1WJoh4l928Jw8SgdDLbtNtMgGAcePG4QaMj2dehKB+ucvvQtpbUCNWE1vZDbj7Aejl+OSOGEdfDfeWQJlYgwL0lXD7bbYPZEycQqFAr4HMo+xm3WFicOzYMXlWacYyahIl+sIVLoGPs7Oz5birV1ZWFr8Tvb29cTMhHNFqd139+/fHDXjx4kXcgPXhld36Krj7oRjr3HUVn4HcLaJekbCQ/QsUnxX1irpiuLcUdOZQ2TCC3mdiBu3dhYWF0dHRIlxIq9WK8KyTtYyMjNu3b3M8uFevXrLdE95AYFuFPPeXSUhI4Heil5cXbiaEIyq76woMDMRt7y4uLub+qcUPXtn9aDWU30WLxl3WT1B6TYLrEiyl1yF7gwTXLb8Lj76V4LomYvLkybgB7927l5GRgRtTZHv37hVtA2Ez6DOJiIjgfrCcRwcaCCy7ea8rM8W77Jbzy6/mjcruuiwtLfv164cbk/WCN1LZXXYT8nbhhDKWvhoefkWDTUxVTSXcXwbAcBxbY/L3Yo4pNC8eHh7ov1mNqsNk6I8//hDtWocOHRI4OkNyRnWYyLyxGwAcHByEnH7/vvG7xbF36dIlficGBQXhZkI4Qq+STbq3GwD8/f1xA16+zLYqQPly6yD9MwD8cZJcld2EvJ2SXZ0IkfMrVKRLmUD6F6Cne7anQ++1Nen27sLCQjHz12g0Jv3l0uv13MtuGxubIUOGMM1HOHt7eyGn815XZufhw4d5eXn8zqWyWyq02v0E9H4n1k9lMcrugqNQdgshjhBZP9M8QdNTUw55OyTOoSIdiky4uGGKRXs3041mmNq/f79oHSYGJt1ncuHChdLSUo4Hh4SE2NjYMM1HOA8PDyGnizAP2Fg7dvD8+A0JCcHNhHBHZfcT0MvuggK8TR6fRnDZra+ArJ8xMhFGp4GCw1InQYyUvxd0MtgWJHcrmGwtyFRISAjudF61Wn3lyhXEgGISZ4ZJXfv27TPduxRzGh1o0L59eyGnq9Vqub3b8Ouvv/I7cdKkSbiZEO6o7H6CwL+Y9fGeqsmR4LK7OA4q5fGmSC7LbVYIOr0ecqVe6jYoS4ES2T3/lQMrKyv0F91MtHFCo9GIP4lFhK152DGqsVv+71MCxqLa6dOnUTJBcfXq1Vu3eD6mRn/fmnCHO7gDTL/s9vT0xA0o+9XugkMYaWCoSAdtqtRJEM5KEuRywwZAj0oagt7ebaJjBLnPMOnVqxfidfft24cYTTTFxcXceypcXV0DAgKY5oOiQ4cOAiMcPiyjz5kNG3jOj+rdu3enTp1wkyHcoVfJpl52ow8elfdqt74CiuXUr5Yvm3sA0iRZVbpFUbTv0lOh79d99uxZee6S3TjuHSbr1q2T5LqyEhUVxf3gUaNGscsEkZ2dnbOzs5AIx44dw0pGoKKiol9++YXfuS+88AJuMsQoFhYWUqcgLy1btsQNWF5eXlbGcNtHYWV3yTXQy2nElYZaBUxHiZymrddooeym1EnIkYeHB+5KZEVFxZkzZxADioB7h4m3t3ffvn2nTp2KdemcnJwLFy5gRRON+XWYGPTu3VvI6cXFxTLpGlq9ejXvbVD/8pe/4CZDjIJedpv6AEFbW1v0PwLT4a0Cy+4kpDSQaG9DFc9xSERUVbkSzw2s7zHzLWFNFPo8E5Nr7967dy/HT+Fp06YBQHh4OO7VEaOJw6gB7WPHjmWXCS7hO1H/9ttvKJkIodVq//Wvf/E7d/bs2QKX/IlAVHbX17p1a9yAAqeFNk7Yl7viEVIaeMofSp0B4UArxYamjZNhSvJAu8Rz3yVnypQphn8qFAqsq+/aJdFOZHzdvn2b+8iOgIAAFxcXpvkgEl52b926NTc3FyUZ3latWlVUVMTv3A8//BA3GWIs9LLbDLpWcF8zbdGiBfp7q3UJK7vLHyClgUeGKZH6ZPhtkmFK8jB48GDcMYKJiYlqtRoxIFMajYbjLDwnJyfDni9KpRLxVdT09PTk5GSsaCIws80p6xo0aJDACNXV1WvWrEFJhp+bN29+/PHH/M4NDw/v0qULbj7EWLTaXR9uKzbTpW4QvNotm0kUtarYTn4hOGT4bZJhSvJgZWWFvuBtVGUmrf3793PsMJk8eXLtb0TcPpM9e/YgRmPNqIndJtTYDQCtWrUSvhP12rVrefdVC1RTU/P888/zO7dFixYrV67EzYfwQKvd9eH+hWrXrh1itPqEld1y2OvkCbJ6xZM0pFp+Pzky/GGWjebc3s19loihw8QgLCwMsc/EhNq7q6uro6OjOR5sY2Njcvsdjhs3TmCEoqKi1atXoyRjrK+//vrq1av8zv3b3/6Gvi8J4YFWu59QU1NTVVWFGLBjx46I0eoTOEAQ84+KQ4b1HKlPhndHMvxhlo3Q0FDcgKay2q3RaDimqlQq674aqFKphNdnta5fv56eLrNXkBtw7tw57nvCDx48WP57wj8BZaTml19+WVxcLDyOUa5cufL+++/zO7dDhw68W1MILvSy29TndqP/VZJ32W3dBikNPNYqqTMgHLRwkjqDemT4wywbzs7OuLvAZGRk3LlzBzEgIwcPHuTYYTJhwoQnlrdx+0y2bzeNXXjNuMPEYNiwYcJbP4uLi7/88kuUfDjSaDTTp0/nd66lpeWuXbtUKvrdSuTo0ZF572gAACAASURBVCPk2R4+Pj64AZ8grOy2kt/fQxmmROqT4bdJhinJSfOcZ8Kvw8Rg8uTJzbDPxKiy27TepzSwsrJCmXj43Xff5eTkCI/D0bx58x484PnW+CeffNKvXz/cfAhv1Nv9hMzMTNyAgYGBuAGfIHC1W36VipWD1BkQDmT4bZLhD7OcNMP2bq1Wy3H+tJWVVVhY2BP/UaVSIU6kvnTpkphVGj9qtTohgeueZY6OjkFBQUzzYWTGjBnCg5SXly9evFh4HC7Wr1/P+2nJiBEjli5dipsPEYJ6u5+AW3ZbWloKnxPaxCUEnd1CfmPzbTylzoBwoJDfpF4Z/jDLycCBA1u1aoUYMDo6Wq/XIwZEx32GSUhIyFM3KMbtM9m5cydiNBaOHTvG/XuK2PsusvDwcCcnhDa5rVu3ivDM59ixYwsWLOB3brt27UxubLzZQ6+STb3sxn3vpVu3bkqlEjFgfcK+3CrMdk8MFmBHU0VNgX0PAJk92JLdD7O8WFhYIM6iBgC1Wn3p0iXEgOiEdJjU/vdm1Wdi9h0mBtbW1q+88gpKqFdeeaWkpAQl1FNdvHhx/PjxOp2O3+m7d+9GucEgcmbqTSaJiYmI0QYOHIgY7amEld32AUhpILFpD9aOUidBOLB2BBuZzaKS2w+z/DSr9m6tVnvkyBGOBzf0sppKpUJ8azA2NrawsBArGgvcv2LAYDyOmBYuXIhSrGRmZrLb9zE1NZX3IwVLS8tDhw4FBwfjpkSEoyaTJ1y4cAExGu7q0lMJ+3IrO4OlLVImGBx6Sp0B4cxBTmWuhQLsfKVOQu7GjRuH+4kv5/buAwcOcOww6d27t5ubW0P/L2KfiV6v575Nvfhu3LiRl5fH8eAuXbqY0J7w9bVv337UqFEoodasWXP+/HmUUHU9ePAgJCSE93awv/zyi0nfF5kxajKpKysrKz8/HyuaUqlEX12qT9iX29IWWuN89OBwYv71Eok6Bh4sh7R5kBQCCX0goQ8khUDaPHiwHNQxUieHpI2cZoc5DgMLE5sfLD5nZ+e+ffsiBjx79mxFRQViQETcC9ypU6c2/v82kz4Tsx8d+IRFixZhhQoPDy8owNwl98GDB8HBwfzewbWwsFi/fv3LL7+MmA9BRKvddcXGxiJGqz8HlgXBX24n2dwQK1xB1VvqJITRaSBjFST0gbuLoeAwaBJA95+2P10JaBKg4DDcXQwJfSBjlcnvqqgaCAp3qZP4D2eeE22bG9yVgOrq6lOnTiEGxKLVao8ePcrx4IYauw1UKhViE3NkZKRGI9O/+M2ksbvW2LFjsSaxZGVlzZgxo6amBiVaUlJScHAwv2HGFhYWv/3225w5c1AyISYB6wdPEgcPHkSM9txzzyFGa4jgslvVWy7Fk9Mk2b2lZ5SsdXBtEuRxG/OUtx2uTYLs9YxzYsnCApwQ9ntDoHAFBxO/YRMLet+bPPtMDh06pNVquRzp7e3drVu3xo9B7DPR6XT79+/HioaooqKC+x2UtbX18OHDmeYjjmXLlmGFiomJ+eijj4THiYyMHDx4MO8NRHbu3Pniiy8KT4OwY+pvQCLS6XSHDx/Giubh4dH4o0sswh8uWICrDP6WWtqDy7NSJ8GXTgN3F0P2z/9d2+Z0VglkrYe7i0142ds5HCyF7veGwPVloA8ybvr27evsjDlpUZ5vVXKfYcJlhPO0adMQt1+WZ59JbGws936hQYMGCd/oUQ7CwsJ69kR7oeiLL74w6p3U+n777bfRo0eXlZXxOFelUsXExOCOvCQsUG93rZiYmMePH2NFe+edd6ytrbGiNQLjy+08FWw7IcQRwu0VsG4tcQ786DSQOo9/x7Y6BlLnmWrl3aINuOHM4eLP9hloK8YNrnmwsLDAHbecnJzM/T08cRg1w6TxDhMD3H1zjh49ynElXkzNrcOk1scff4wY7YUXXrh//z6/cz/99FPeYw09PDzOnz8/bNgwfqcTkybz/RMasX492gN/lUo1d+5crGiNQ7nLsQKvtzHi8KVwhXazpExAiNS5oE0TFEGbBqnzkLIRncvzoGhwCoQYvN4BC7SVyObA7LerPHLkCMe61snJadCgQVyORFxErKys5N53Lprm9j5lrenTpyO+Z1xcXDxlyhRj3zPW6XSvvfba//3f//G76IABAxITE7t3787vdCIyajIxyM3NRXz0t3jxYpVKpJ2qkR4utBwk3buVFuC9HCyYv3zKxIPloL2NEEebBhkrEeKIz9IGvJdJ1pTvFAotOZVNpNb48ePNe4wg9w6TqVOncvxSTJ8+HbHPZM+ePVihUOTl5V29epXjwY6OjrjzcCS3YcMGxL8RycnJr7/+Ovfjy8rKxo0bt3HjRn6Xmz17dmxsLG7nGCEiWLNmTXV1NUooZ2fnJUuWoITiAq+np/37YPsMWjTu3F4HVR8Jriuc5jIUoL0NAHk7QHMZLZqYVH3AzYhfM2hsvaH9+xJc18S1atUKdx8vgf2suLRa7aFDhzgezP39G9x5JocPH+Y4U1wcx48f537wqFGjzGy5rkePHrijP7Zs2cKxjL5///6QIUN437iuXbt28+bN4vSzEizof31M8e9jTk7O6tWrsaJ99dVXNjbiTRDG+/tmaQud/gk3X4IaEfsOVX3B3WT7KzJWIQfM/tlU70Dc50HJVdDEi3dFSyV0+lpemz2ZjgkTJiBu8JGXl5eamurn54cVUIiIiAiOHSZKpdKoDVPCw8OPHTvGN6//odFoIiMjRdjWgaNm29hd65///OeuXbt4701T34IFC/r06dOjR49Gjjl48OCLL75YXFzMI37Hjh337dvXeHxEERER//znP8W5lnAeHh5bt26VOosGmWKVjO6tt94qKTFmBEXDevfuLfKUetTbXNuO0Hk13H5bpMrbIRB8sCtX0ahjcNpL6tIkgDoGHEOQw4qj00q48zaUJIpxLUsldF4jzcMZszBhwgSUYWe1Tp48KZOym3uHSWhoqFEbK4SHh8+dO1en0/HK60l79uwx0bJbhL2Xxefo6Pj5558vWLAAK2BFRcXkyZMvXLjQrl27px7w3nvvrVzJs7Hw2Wef/fXXX+3s7AQkaJysrKzTp0+LdjmBOnWSekREo9AHj5hcHX/06FHuH9SNs7OzE3/rX+zBMQ5B0Hm1GJ3WDoHQ+V9gKd4HBzJGm02a7h6WVnbgsxrsujC/kIUCOq8Bh0DmFzJfvXr1wm0GlUl7d2Vl5YEDBzgezGWGSV0qlQprO3EA2L9/P1YFL1BSUhL3WTQ+Pj5eXl5M85HK/Pnzg4ODEQM+ePBg9OjR9XdHevDgweDBg/nV3K1bt966devOnTvFrLmJzJnWJJPc3FzE0fI//PCDt7c3VjSOGMxrdAgC3zXQ4un36DgcQ6DzahOuuYHK7qexsgPftWxX61u0A1+quRGEhmK+Qh0ZGSmHj37uM0ysrKx4fAUQ55kUFhaeOXMGK5oQ1GFSa+fOna1bY86xvXbt2oQJE2r7+PV6/dq1a7t3786vxSskJOTmzZuzZpns1C8CAM27t1uv10+fPr2goAAl2syZM0VuLzFgMybdoTd0286kfrJQQPsl0GmlLLZZ4U1z2bidcbjTlYA2lUlkcVi1hE4rof37YMHg/YZWw8B/O21IiQK3w6G0tPTChQuIAfnh/rRxxIgRLVu2NDb+1KlTzW+eiVHvU5rT6MD63NzcNm3ahBvz7Nmz4eHhOp3uwYMHI0aMePPNN0tLS40NYmdnt3bt2lOnTjXUssKaCRV28tecv5gff/zxuXPnUEL16NGD9/wfgZjtTmSonzp8hLmLjUMQ+G8BZ9pGq1HVprl1Tl3OM6DrJnAIQgto3Ro6fAQ+q8Da6FKJPNWECRMQK0iQwXaVlZWV+/bt43iwsR0mBm3atBk5ciSPE59KDmV3RUUF90V3CwsL817tBoBJkyYZNf6Pi4MHD44dO7Z79+4xMXweZo4aNSo1NXX+/Pm4WRGpNNuy+6effvrss89QQrVt2/b48eNKpRIlmrEYbwradgoE7AfXV4UuTtt2gs4/gN96sO2IlJmkNAmmGlw0Sh/wWw+dfxC6AaqlPbi+CgGHoC2fOok0xM7ObujQoYgBJS+7jx07xn33R+6jA58wc+ZMfifWl5OTg7Xww1t0dDT30bkDBw40jz3hG7d69erOnTvjxoyKiuKxyO3q6rply5aTJ096enri5mMsObSQmQ30VypN4ruzceNGrFeWW7VqFR0d7erqihKNB8ZlNwBY2oPHAuhxBDz/Bgrj/5ytgsH3R+i2E1r2Z5AckbeW/aHbTvD9CVoZ/66SjRd4vQc9joDHAhoUyAJun8mFCxd4FBaIuL8a369fPzc3nlurzpgxw5z6TIzqMDH7pW4DOzu7ffv22dpK/Jnzyiuv3Lp164UXXpA2DSJ/8l8+/+6771577TWU2wMHB4fo6OiAgADhoXhjX3YbWDlAu79AwGHw/RHc54PjMGjRtsGDlT7gFAbt34fu+8HnW1CZ1ZZmAADWLPcgZRpcEqo+4PMtdN8P7T+AtpNB6dPgkS3aguMwcH8DfH+E7vvA5TmwchAx0eYFdxJcdXX1qVOnEAMahekMk7pUKhVifzPi3sj8NNs94RvXrVu3b775Rqqre3t7R0dHb9y4sVWrVlLlQEyInMvuwsLC559//p133kGJ1rp166ioqKAgvP5VXkTfnkrV979ltL4Cqh+DTgPVj8GyBVi1BCsVWDuKnZL4lL6mGlxCNp7g7Akw/d//s1oNOg3oHkNNFVirwKolWLdk8iImaUC3bt08PDwyMzOxAkZGRuIOSOHuxIkT9Se1NURI2Q0A4eHhERERQiLUSk9PT0hI6N1bmreEMzIybt68yfFge3v7fv36Mc1HVubPnx8dHS3+SOBPPvnk448/FvmiTULviyDNQURExEsvvfTnn3+iROvQoUNUVJQchrJL+pfBwgZaOINtR3AIBLtuYOPVLGpuoNVuDNaOYOMFdt3AIRBsO0ELZ6q5xTdp0iTEaBK2d3PvMOnQoUPXrl2FXAu3z0TCBW9jRwc2t03IN2/e3LNnT9EuN2nSpAcPHsiw5gaAmpoaqVMwH81hgGBmZuaLL744YcIErJq7W7duly5dkkPNDRKX3c2W0o9PmzsXCldQymK3P9Ic4LZ3p6SkcN94BZFRM0yEvxOJ22ci/npqLWrsbpxSqYyIiOD9GgB3Y8eOvXTp0sGDBzt06MD6Wvy4urqK8HXAIvC+mjXzLrvz8/MXLlzo6en5+++/Y8UMCwuLi4vD3eJNiOa1/CAjqj5QcJhJWELEMnLkSIVCUbudh3DHjh1D3IGMo8jISO4dJrxnmNSF2GeSlpaWkpLi7++PEo07vV5v1NOJ5tPYXZebm9vRo0cHDRrEfUiOUUaMGPHZZ58NGDCARXBEEydOzMrKKi4uvnHjRkpKiuGfKSkpjx49kjo1cHZ27lmHv79/c3ssIxM5OTkrV65cu3Yt4l+WNm3a/Otf/5Lbi8X04yURp1AmZbeTNK2xpHmys7MbNmwYYnNIZGSk+GU39w4TNze3/v0RRirNmDFjzpw5WLu779mzR/yyOz4+Xq1WczzYy8vLx6fhN6HNWmBg4LZt21Du1p4QFBQUFRWFHpadVq1aDRo0aNCgQbX/RaPRXL9+3VCC37hx48aNGyIU4l27du3Zs2ePHj169+7dvXt3d3d31lfEhd6xI+0AwdLS0r17927ZsiUqKgrrI9EgLCxsw4YN8lnkrkVlt0RUfcAhCEquYMZ0CKLVbiKyCRMmIJbdRvUtoNDpdNzH8Al8mbKWoc8Ea8F779694nf0GtXYjTv0xuRMmTJl6dKlWDt91Lpy5cqcOXN+/vln3LBiUqlUAwcOHDhwYO1/KSkpqbscfuPGjYcPHwqpC1u3bt2jR4/axeyAgACFQoGRu2Rk1RPC2507d86dO3fy5Mk9e/aUl5fjBvfw8Pj222/Dw2W6tSKV3dJxnwtpbyAHJERc48ePx5ruBAB5eXk3btzo1q0bVsAmnTx5UrQZJnUh9pkkJSWlp6eL3NdLjd1GefXVV7/99tuysjLcsBs2bHjmmWc+/PBD3LAScnBw6NevX92hN6WlpYYq/OOPP+a4Fj5lypQ+ffoYqu327dszS1YaJtrbrVarr1+/HhcXFxcXFxsbm5+fz+IqNjY2ixcv/uijjyQfnN8IKrulo+oDLjMhbwdONJeZtNRNxOfn54c+RlDMspt7h0nLli0Rt3bH7TPZvXv34sWLUUJxUVpaevHiRY4HW1hYjBo1imk+cpafn/+Pf/xj/fr1iK9A1LV06dJ27dq99tprLILLgWH0ZFBQ0BtvcFql8vPz4/6GNGEhKyuroKDgzz//vHXrVkpKys2bN2/cuJGbm8v0om3btl20aNGzzz7bsaPc9zKnsltSXouhLA2h1cRxGHiJ90uXkLqmTp36/fffY0U7efLk22+/jRWtcTqdbv/+/RwPDg0NRRz8h95nImbZfeLECe57wvft29fRsXlMhv1fjx8//vLLL//1r3+x3n517ty5zs7OYWFhTK8ircuXL1dUVHA5cuzYsayTMTPnzp1bvnw5x4P1er1Wqy0vL9f+h+Hf1Wp1QUFBQUGB+JsNu7i4vP/++2+88YZSqRT50vxQ2S01n1WQOhe0t/lHUHYGb65/ZwhBN2HCBMSyOyYmprq6WpxhAlFRUYWFhRwPRuwwMUDsM4mLi8vJyXF1ZTOWtB7anLJJX3311ZdffllUVCTCtWpqaiZPnhwbGztkyBARLicJ7lvYmn3Zjf5K5blz586dO4cbUwSGx2ivv/761KlTW7RoIXU6RqC53VKzUoHfev4TSJxCwX87WDWPLXKILBnGCGJFKy0tjYuLw4rWOO5DrxUKBfoOmrj75nB/MVS4I0eOcD+4uTV279mzx8fHZ8mSJeLU3LXCwsLu3Lkj5hXFFBMTw+UwGxsbxE4wIk+enp4fffRRenr6iRMnnn32WdOquYHKblmwUoH3cvBeBlYOxpzlAN7LaJ2bSE6hUOD+qouMjESM1hCjZpiMHj0a/Qkm7r45opXdd+7cycjI4Hiwvb193YFx5i0pKWnAgAEzZsy4e/eu+FcvKioaNWoUozfVpFVdXX369GkuRw4ZMsTGxsy3K0a8Vzct7du3f++992JjYzMyMj799FMvLy+pM+KJym7ZcJoEAYfAbU7TxbfCFdzmQMAhcMLcmpsQ3nC3qxSn7D516hT3DhMWc5cBAHHE1ZkzZ7j/cYQwaobJ8OHDm8PmI6WlpX/961+DgoK4v2nKQnp6+vjx4xltzSOhS5cucWzsbuajKs1S3759P/nkk8TExPT09K+++soMOqmo7JYTKxW4z4PAGPDfCi4zwSEIHIL+/X8Z/t1lJvhvhYDD4D6PGkuIfOBWpXFxcSK8l8O9w8TCwgK9sdsAsc9Ep9OJM8DBqLK7OTR2x8bGdunS5YcffpB22xGDy5cvP/vss3LIBFF0dDTHI5vDz5uZfXOfytHRMTw8/Ndff83Ly4uPj//4448DAwOlTgqPnhBCBPP19UX8XDpw4ADTbGtqatq0acMxmeDgYHaZIK7PTZgwgV2eBlVVVfb29txTunnzJuuUJPT48ePXX3+d9/erffv2jGZlLly4UOqvDSaOrwe4u7tLnakYJk6cyOJnRj7mzp175coVqb/MDNFqNyEEAW6fCeLOl08VExMj4QyTuhD7TCIjI7lv/cPPuXPnuD+IcHFx6dKlC9N8JJSYmNi9e/cNGzbwONfa2vrvf//7rVu3jh07xmL+zJo1a7766iv0sJIoKyvj2NjdHJa6m4P169cHBQVZWFh07dp1xowZn3zyyYkTJ1h/somJym5CCALTau/mvksOAEyfPp1dJoh9JpWVlYcOHUIJ1RCjRgeiz36Rj59++ikoKOjhw4c8zu3Vq9fly5f/+c9/KpVKT0/Pw4cPs5g3vGTJki1btqCHFV9UVBTHnYbM+Oetebp169aePXuWLVs2duzYli1bBgYGvvHGG5s3b05LS5M6NWGkXm4nhJiDiooKOzs7xI+m3NxcRqka1WHSo0cPRmnUQuwzmTZtGtNUe/fuzT2Z7du3M01GEmVlZbNmzeL9DXrjjTcqKiqeiHno0CEWG3RbWVlFRERI8lVCNGfOHC5/WIVCUVZWJnWyYsBd4DBRnp6es2bN+umnn1JSUqT+hhiNVrsJIQjQxwgePXoUMVpdRg39YNphYoDYZxIREcFukIVarb5yheuWuhYWFuPGjWOUiVQyMjL69++/bds2Hufa2tpu2bLlxx9/rD/kPjQ09Msvv8RI8H/odLqpU6fGx8ejRxYTxyHx48aNM5VNColwjx492rZt2xtvvOHv7+/p6blgwYITJ05UVVVJnRcnVHYTQnCYSp+JUR0mjEYH1oXYZ6LVarF2vqzPsHTK8eBevXqZ2Z7w58+fDwoKunbtGo9z3d3dL168+MILLzR0wN///veFCxcKyO7pysvLx44dm5qaih5ZHElJSVlZWVyOZNoJJissHoyYtMzMzB9//HHs2LGtW7dev359SUmJ1Bk1gcpuQgiOSZMwB8kzeqtSr9dz31nGzc1NhMFVuPvm7N27FyvUE5rznvBbt24dPnw4v81oevfufeXKlR49ejR+2OrVqxGfe9RSq9XDhw/n14YuOe6PvMLCwphmIh/cb32bm9LS0nnz5rm4uLz66quJiYlSp9Mg89/IgBAiDg8PD39//5SUFJRoeXl5165dCwgIQIlW6+zZszk5ORwPfu6553Cv3pDw8HCsVeqDBw9WVlbW72QQzqi2H3PaE/7DDz/84osv+J0bGhq6e/duW1vbJo+0sLDYtWvXyJEjuY+p5ig7O3vs2LHnz59v3bo1bmTWDh8+zOWw0aNHm9mjlUagr3YPHDhQ4E2ypo7Hjx9nZWVlZmaWl5djZWgUrVb766+//vrrrwMHDnzzzTcbecQkFenKbnUMqGOgMhuqNaBNAysHsPMDAHAMAVUQKP0kS4zInDYVNFdAHQMAUJYKuhJQ+oK1ChRu4BgCjiFS59esTZgwAavsBoCTJ0+il91GdZiI0NhtMGPGjDlz5uh0OuGhNBpNVFQU+nZ9169fz8vL43iwjY3N0KFDcROQyksvvbR582Z+506aNOngwYNGnXLgwIHBgwcnJyfzu2JDbt26NXHixOjoaC43ADJRVFQUFxfH5cjm02HCwqBBg5YvX44etqCg4NGjR9nZ2Q8ePMj4j6SkJLVajX6tp4qLi4uLi1u0aNG77767cOFC+bT+i152V2ZB1npQx4Duf/tvdCWgSQCAf/9T4Qbu88CJ5gGROgoOQdZ6qMx+8r9r/zNOqOAwWDmAYwi4zwWFu8jZEQCYMGHCypUrsaJFRka+++67WNEAQK/Xcy+7nZycRKsdDX0mWAvee/bsQS+7m+Ge8OXl5dOmTeP9TRk6dCj3nVBrOTg4HDlypG/fvtyfyXAUFxc3a9asP/74w9LSNJpLOTaDWVhYsGjOIQI5OTk5OTn17Nnzif+elZV19erV5OTkM2fOnDlzhnUrdm5u7pIlS7777rv/+7//e/311+XwuSTuX7+sdZAyCwoOP1lz11eZDQ+Ww7VJ/17UJM2cOgauTYIH/3hKzf0EXQkUHIaUWZC9XpTMyP8YOnQo4hjB6Ojo6upqrGgAcP78ee7VTFhYmJhvLyGWDvv27UNvAG1ujd2PHz8eOXIk75p70KBBERER/Fp9PD09IyIijNoNlKN9+/bNnz8fPSwju3bt4nLYoEGDuM8DJZJzd3cfP378kiVLjhw5otFozp8//9lnn7Fe4MjOzp4/f76fn9/OnTuZXogLscpunQZSZkH2z00X3HVVZsPdxZC3nVlaxBRkrYO7i5suuOvSlUDWekiZBTrz2drKJFhZWY0dOxYrWkVFxblz57CigVw7TAwQ55kUFhbGxGAuWFRUVHDcKdDA1Bu7y8vLx48ff/78eX6n9+zZ89ixY0LuPwMDA3fs2MFiWXr9+vVff/01elh0+fn5HF+qnjFjButkCDsDBw788MMPT58+XVJScvDgwblz5zo4ODC61r1792bOnDl8+PAbN24wugQXopTd2lRImfXfTgBjZayCB/iNR8Q0PFgO2T/zPFebBimzQGuqw7NMlJzHCHJf6lAqlSKPnZbzPJPTp09XVFRwPNjFxaV79+6IVxfftGnTeNfcvr6+J0+eVKlUAnMIDQ395ptvBAZ5qiVLlshhza9xHDtMLC0tZfjOHOHB3t5+0qRJ69at02g0O3fuxN0Foq6YmJju3bu/9dZbZWVljC7ROPZlt04D9zn0BjSu4DBkoDWMEpORsRIKOL3J3qDKbLj/D1rzFtPkyZMRoyGOEYyLi+PeYTJ+/HgWw0Aah9hnwqOruBFGNXab9C45er1+1qxZvHtLVCrV8ePHnZ2dUZJ5++23//KXv6CEqkuv18+ePfvMmTPokRFx7DCZOHEi1lfbVKD3j8lwIuGzzz4bGRmZnJwcHh7O6FWENWvW9OjR4+rVqyyCN4592X1nEf917rrydkDBIYQ4xFQUHIK8HQhxtGlwdzFCHMKNs7NzkyOKuYuPjy8tLUUJZVQlKsIuOfUh9pnk5ORcuHABJRQ0p8bujz76aPt2/m2Nv/32m7e3N2I+v//++7BhwxADGlRVVU2aNInf1j8iKCoqOnXqFJcjX375ZdbJyA36Cyey3X8nICBg165dKSkps2bNYhH/7t27gYGBX331FYvgjWBcdhccghKumwk3LWMVLVs2FzoNZKxCi6ZJoHs2MSH2mej1eqNqvkZw39PbyspKkt035NlnkpeXd/36de7Hm25j9+HDhz///HPep8+fP3/atGmI+Rjs3bsXt5Q3ePz48ejRozMzM9EjC7djxw4uS7COjo64W3QRGfLz89u6dWtycjKj+/klS5b85S9/QRneyhHLshu3cgIAXQnkNgK3YwAAIABJREFUcv3FSUxb7jbj3r5tUtZ6umcTjQzbuy9evMi9w2T48OEtW7YUflEeEPtMOD6mb5JRu+T07NnTxcUF5boiM7xuxft0b29vRq3Ybdq0OXLkCIvBJrm5uSNHjiwoKECPLNCOHZyec86ePbtFixask5Gbmpoa3IAybDKpLyAg4Pjx49u3b3dyckIPvnXr1nHjxmk0IlUILMvugkPIlRMAZP8MlVnIMYncVGbxf42ywZjZtOAtmkGDBrVq1QorGkrZbVSHicgzTOpC7DNJT09H6Vw0qrHbRJe6KyoqpkyZIqSdacuWLex2ovH399+9ezeLZoDU1NTRo0dj9XGhSE9P59h33gw7TFiQbZNJfTNnzrx58yaLRxyRkZFDhw4tKipCj1wfy7I7n02VY9hPh5gxRt/ifGFvZxLOrKysEB8IpqWlZWRkCAxi1NIvi1YBjnD7TDiOg2iEXq8/duwY9+NNtLH7vffeE9LoPGfOnMGDByPmU9/48eNXrFjBInJiYmJYWBjugHwhNmzYwOWwLl26BAUFsU6GyI2zs/PBgwfXrFmDHjkpKSk4ODg/Px898hOYld2VWaC9zSQybaBj9hh9i7Vp9KhENLh9JgLnmVy+fPnhw4ccD+7bt6+bm5uQywmE2GcivL07ISGB+2bOJronfExMjJDf4ra2tp9++iliPg358MMPGT2HiY6OnjVrlhyaDfR6/S+//MLlyDlz5rBORp5MaHGanb/+9a8JCQno2yTduHEjODj4zz//xA37BGZlN7viWG3Erg3EJLH7FtOjErFMnDgRMZrAslvOu+TUh9hncuPGjbQ0QYOkjHqfdejQoTY2NkIuJ77S0lKBcxLefPPNdu3aYeXTuG3btjGaib579+533nmHRWSjREZGZmdzGjf84osvsk5GnprDAEEugoKCLl686OXlhRv21q1boaGhVVVVuGHrYlZ2V7NsTm8+78ZV5UFZCpReh4qHUM11zcm0Mf3mVtBqt0icnZ0RHwEb1edQH8c3tAwkGR1Yl6z6TMy+sfutt97iWOc9lZ2d3fvvv4+YT+OUSuXRo0ddXV1ZBF+9evWXX37JIjJ3Gzdu5HLY7Nmz27ZtyzoZeWo+AwSb5OPjc+7cOfSb3vj4eKbPUpiV3UyLpzIz3XdQewfytsH9pXDzL3B1DCT0geQJcPNFuPUyXJ8GV0dBQh+49QpkfANFkVAjzQZLzDH95jafGzYZQOwzUavVSUlJ/M69cuUK9w6TDh06dO3ald+FECH2mQgpu0tLS43arNHkGrvPnTvHsc5ryIIFC0Su/7y8vI4fP65UKlkE/+CDD7Zs2cIiMhfFxcUcf1z/+te/sk5GtprnJJOGeHl5nThxAv2vw6ZNm7777jvcmLWYld1lGFvkNB9FkZA6F1JmQsY3UHgcym5BdeHTjyy9Bnnb4N77cHUspK+AUgn2WDJh9GMpIpmMETSqw0TICDlEiH0mCQkJ6enp/M6Niori/qadi4tLz549+V1IEnq9/vXXXxcSQaFQLFmyBCsf7nr06LFr1y5Gu/e9/PLLAh8u8bZlyxYuD/cDAwP79esnQj7yRKvdT+jRo4dRzzM5eu+99xITE9HDAsOy286XVWRzoq+E/H1wfRrce9/ofYVqtJC/H269Brdegcdo29GZOfqxFNGAAQPkMEaQ+y45IIPGbgOZ7JtjVGO3yXWY/Pjjj7du3RIS4aWXXpKq1SE0NJTRmHCdTjd16tSzZ8+yCN44ji+2yqEHXULU211fWFjYokWLcGNWV1dPnz6dxWxNZmW3lYpVZACw82MYXDSlVyHlBUj/DCq4PgFvIM41uP1XuPUyaC4jZSYpG3eGwZn+WJL/ZWFhgbjgHRMTw2PGWVJSEvcOEzc3t/79+xt7CUbkMM/EjBu71Wr1hx9+KDCItG/1vf322/PmzWMRuby8PDQ0NDk5mUXwhpw4cSI1tekOw7Zt2zbblykZMfXVboOVK1eif3rfv3//gw8+wI0JDMtua5b1jckXT3rI+hFuvQbl99FCll6H229D/j60gFJRsCy7mf5YknoQy+6KiorY2FhjzzKqw2Ty5Mny+Q2E2Gdy7ty5wsIGmtYalpGRcefOHe7Hjx8/3thLSOiHH34oLi4WEuGZZ54ZMmQIVj78/PDDD4xyKC4uHj169L1791gEf6pVqzjtaT137lzWmcgcrXY3ZOPGjVifmbXWrl1r1McgF8zKbscQZpGHsYosDl0x3H4bsjmNJjWOvgLSP4O7i0H3GD+4mNh9i9n9WJKnGTduHGI0Hn0mpjU6sC7EPhO9Xs9jo/gjR45wP7hbt26mtSf86tWrBUaYPXs2SiZCWFlZHTx4sGPHjiyC5+XlhYSEZGWJMf0pNTWVS0eTpaXlW2+9JUI+xBT5+/ujv2ur0+nQf+SYld0Kd1B2ZhJZ1ZtJWHFUZsGt1+CxEcMBjKaOgbtLoEbL8BKsMfoWKzuzXUon9bRt2xbx5Sdjp3cnJyffvs11066WLVuOGjXK+KQYknaeiVGN3aY1w+SHH34QviPGjBkzUJIRqHXr1kePHnVwcGARPCMjIyQkpKCggEXwur766isuh02bNk20EenNh3we8Qn34Ycf2tra4saMiIg4fRpzLxGWm8O3ncQkrOkuWFb9CbdehfIHzC+kuQRp8xuchSJ/jL7FjH4gSaMQ+0wuX77MfcdEMHKpe+LEiegPKAVC7DM5deqURmPE9Mzq6mozfp/y66+/FhihY8eOAQEBKMkI5+fnt337dkbBb9++PXLkSKN+eIyVl5fHcWqh8HZ8Yt5cXFzefPNN9LArVqxAjMay7HaaBFbYt+BOoaa6YFmthrQ3oSpfpMuVXoc774He6FfQZEHhDk6hyDGtHMCJym4JILb86vX6qKgo7gfv3LmTe3DJd8mpD7HPRKfTGbXgfeHCBe6v8FtbW48YMYJXXhI4ffo074mKtcLCwlCSwRIaGspup5urV6+OGzeuvLycUfzvv/++srKyycPGjRvXq1cvRjmYEOrtbty7776Lvn4fGRl5+TLayAqWZbeVCtxR336wcgAv5Bkx4kn/HMrFez0FAKD0KjyUeMsx/rwWId+zuc81/TdxTVLfvn2dnZ2xonFv7z537hz3DhOFQhEain2nh0GqeSZGLXUHBweb0J7wmzdvFh4kODhYeBBcS5YsEbjLfSPOnz8/efJkHqOEmqTVajnODVy2bBn61U2RmVXJ6Nzd3Vn0vCHunsOy7AYAl1mYHd5ei0y1cvrzD1BHS3Dd/P2Q85sE1xXOSoV5i6XsDC6sfiGRxllYWCC+WMm9vfuPP/7gHnb06NGMtv0TCLHP5Pjx41ot11c+jBodaEKN3eXl5UY9A2lISIgcex1/++23gQMHMgp+4sSJmTNnotd833//PZfOsREjRgwYMAD30iaKVrub9PLLL6PH3Lt3L9YMb8ZlNwD4rcdZtnQKNdUmgapceCT0rXn+sn+ByhzJri6E0yScVhOFK/itR4hD+EJs7757925GRgaXI42qrmQ1w6QulUqF1TZdWVl58OBBLkeq1epLly5xj2xCjd0HDx4U/ruzY8eObdq0QckHV4sWLY4cOeLj48Mo/p49exYsWIAbk+PcQFrqrmV+VTK6yZMnKxQK3JharXbfPpwBzezLbisV+K0TWnk7hYL3cqSERPfwKynnitRoIUPo+0OS8V4utPK2cgCfVab6kMRcjB8/HrHZjsvO1XFxcTk5XO82LSwsZNjYXevZZ5/FCsWxvfvEiRPcf7U7OjoGBQUJSEpUBw4cEB5Ezh3GrVu3PnnypJeXF6P4P/3009KlS7Gi/fDDD7m5uU0eNmDAgKFDh2Jd1NTRaneTlEoli7UArC3o2ZfdAKD0A/9t/LtNXGaacM1dkQ5qzNEzfKhPQ+k1iXPgzXs5uMzkea6yM/hvA6VZ7Glqylq1aoX47JtLe7dRHSaDBw92cnISkBFbiH0mhw8f5vLumlGN3WPHjjWhAWTGzqB8KpnfZnh7e0dERLCL//nnn2PtS89xbuAnn3yCcjnzYH5VMgssXnqOjsZpFRal7AYAhTv4rTd65VLhCp1WgtdiNjmJIvtXqTMAAIDcbVJnIIDXYui0EhSuxp3lFAp+60117o3ZQewz4bIWa1TZLdsOEwPEPhOtVsvlWYG5TuyOj48XPq4bAHx9fYUHYapbt24cR/Lxs2jRot9+E/rW0MaNGx8+fNjkYf369TOhLiYRUNnNBYu3L7RaLY8t2+oTq+wGACsVeC+HgIOcim8rB3CbAwGHTXhKNwBUF0Fh07/kxFAUbaod3gaOIeC/HdzmcOpWcgqFgIPgvZx6S+QDcYygWq1OTExs5ID4+Hguv85ryWTfk0aI2Wdy8+ZNjt3zBiZUEnG55eCiU6dOKHGYeuGFFxYuXMgu/iuvvCKwY+eLL77gcthnn30m5Crmp6amBjegWdbxvr6+LHZWQnlcJmLZbaBwB+/lEHgKvJeB4zBwCPpvIaVwBYcgcJkJ/lshMAbc54mdG7rCCNlMztZBPkJTo5SsVOA+DwJjwH8ruMwEh6D/rn9bOYBDEDiFgvcyCDwF3stpkVtugoKCRBsjaNRSd0BAQIcOHQRnxBZin8m+fft0Ol0jBxg1w6RLly7s2ojRnT17FiXOM888gxKHte+++47pfPFnn302JiaG37mbN2++c+dOk4cNGTJEbnvHSs4sq2QW+vfvjx4zPj5eeBBr4SH4sFKB0yRTnUzCXf4hqTOoQ5MgdQZIlH7gRe3apic0NPTXX3F6rk6ePPn3v/+9of/X1HfJqc/QZ4KyWKvRaKKiohrpDDHjzSlRfmVaWFg4OjoKjyMCS0vL3bt3jxw5Eut+4wmVlZUTJ06MiYnp27evUSfqdDqO72V+++23vFIzZyb0KoW0/Pzw6wSUzxDRV7ubD12JGPvAc1d6HfQVUidBmi/E9u7Y2NiKiqf/MCckJBjVYSLzxu5aiH0mjeybU1FRYdRrQybU2J2RkVFcXCw8DuJDGxEoFIqjR492796dUfyysrIxY8akpaUZddaGDRsePXrU5GFhYWF9+vThm5rZavxpFQ/munzepUsX9JhlZWV3794VGITKbmZKEkFfJXUSdegrocRk55kQ0zd69GisTomKiorY2Nin/l9GdZi4ubnJeRhcXYh9Jrt3727oF+3Zs2cbup+pz9raevjw4SgpiSApKQkljqksdddSqVSRkZHt27dnFF+tVg8fPpz7+wBVVVVchnBbWlpynHPS3NAAQY4YDbDnvvlxQ6jsZkYr9JYIX+l1qTMgzVerVq0GDx6MFa2h9u5t24wY2oO4hMwa4jyTwsLChroOjGrsHjRokL29PUpKIkhOTkaJg/5CmwjatWsXFRXVtm1bRvGzsrKGDRuWl5fH5eC1a9dymdU9a9YsFk0CZoDKbo5cXFxYhDXqjfOnorKbGVl1mBhU5UudAWnWEPtMnlp2JyUlmWWHiYEI80zMuLE7PT0dJU5VlZyeYXLm4+Nz4sQJdrdJ9+/fHzFiRJPbvJeXl3MZwm1lZUWzuhtCZTdHjMruzMxMgRGo7GZGhjVuVYHUGZBmDXGM4JUrV+r/jt+9ezf3CE5OTsOGDcPKRwSIfSZPfes0Ly/v6tWr3IOYUGM3AHBpJuaivLwcJY74evXq1Uhbv3A3btwYNWpUSUlJI8esXr26sLCwyVDz5s0zlXEx4qOym6M2bdqwCJuVlSUwApXdzOga+/SRhk4jdQakWevRo4eHhwdKKL1eX3+EqlFl96RJk0xrJgBin0lOTs6lS5ee+I9GdZg4OjoaO79CWlhld2lpKUocSYwZM2br1q3s4ickJEyYMKGhO5O8vLwVK1Y0GcTBwWH5cpPdl5o9c62SWVCp8PfuKCgQunxJZTcz1fKrcWV4J0CaGcQ+kyfK7mvXrhn1sotJjA58AtN5JkaV3SNHjjStmxassrvx1Vz5mzVrFpfal7fY2NjJkydXVz9lw4oPPviAy1fv008/Na1xMSKj1W7u7Ozs0GMWFRUJjEBlNzPW8tsikcsWj4SwhFh2PzHH2qilbqVSOW7cOKxMRIPYZ7J9+/Yn/otRG7CZVocJYPyyrKXVarFCSWLp0qWvvPIKu/gnTpyYOXPmE++eJicnb9y4sclzO3bsyHRzTVKfad0/G4VF2d3kCwxNorKbGRnWuLRZOpHamDFjsArHjIyMuiNUd+zYwf3c8ePHKxQKlDTEhNhnkp6efv36f0cbJSYmcpxEYYDYpi8C7lMRucjJyUGMJomff/6Z6Ruxe/bsee211+ouo77++utcTly/fj3W54O5Qp+lgz4IXD5Y3CEL//pT2c2MDPcnb+EkdQakubOzs0N8kbF2nklKSopRHSamNcOkLkbzTIyaYeLj42NCe8IDwFN7Hnjjsqu5zFlZWe3bt69r167sLvHbb78tWLDA8O+bNm2q/y5BfWFhYSNHjmSXknkw454QdMJXputTKpUCI1DZzYzSW+oM6mnBam4rIdyxGCNoVIeJlZXV5MmTsXIQGWKfSd32bqMau01rdCBgr+eZQdkNAPb29idOnHB1dWV3iZ9++unjjz/WaDSLFy9u8uAWLVp8//337JIxG7j3kGC+q901NTUs5g4Jb1yhspsZGa5227PaIpgQ7hD7EyIjIw1rP0ZtThkSEtKyZUusHESG2GeSnJxsmGZdWlra0K6fT2Vyjd1Udj+Vp6fn8ePHWfS/1lqxYsWwYcPy85sep/vee++Z1iMUqdDm8Bxx+anjoVWrVgIjUNnNjEMggJzeVLByAIceUidBCPj7+2ONEVSr1QkJCbdv367bptwk0+0wMUDsMzEM8I6JieG+hGZhYWFyq93CnwvXdfHiRcRo0urRo8e+ffuYXiIxMbHJYzw8PJYuXco0DbOB3tttihuvcoH4FnVdnp6eAiNQ2c2MtSMofaROog5Vb7AwvXfIiFlC7PGIjIx86uYvjZg+fTrW1SWB2GdiaO82qrF74MCBJrQnvIGtra2lJdovu4sXL6I/6JfQmDFjNmzYIG0O69evZ7robk6o7OaIRWM3ALRv315gBCq7WXIMkTqDOlR9pM6AkH/Dbe82qsOkT58+bm5uWFeXhEqlwmrziI+Pz8nJMe/GbgNHR0esUNXV1efPn8eKJgevvfaahIvNs2fPRvxAMHvUZMLRvXv3WIT19hb62h6V3Sw5TZQ6g/+wtIU2pjTwi5i34cOHY4WKiooyaktzU9wlpz7EPpNvvvkmNTWV+/Fjx47FurSYcLv5T58+jRhNDlasWPHcc8+Jf10XF5c1a9aIf13TRavdHMXHx7MI27NnT4ERqOxmycYTHIKkTgIAAFqPAmu0xR5CBLKzs5PqtTxTb+w2QPxTfP3119wPbtmy5cCBA7EuLSbcsrv+Hp9m4Pfffx81apTIF924caPwd9SaFSq7OWJRdiuVys6dOwsMQmU3Yx5vSJ0BAAA4m3YzKzE/kjxW7tChg7+/v/jXRefo6CjJLpsjRowQ/6IosN7iNUhKSnr48CFiQDlo0aLF/v37AwICRLvijBkzJk6UzTNhE0FNJhxduXIFPWbfvn2FB6GymzGHIGgt9vrBkxyHgb14n6SEcCHJNoeSPEZnJDw8XPyLmmhjNwCgbw2zfft23IByYG9vf/LkSdxblIa0adNm3bp1IlzIzKBXyWZZdicmJrIY2j148GDhQajsZs/zb2Ap3Tvalkpo/3fJrk5IA3x9fTt16iTyRc2jw8QgPDxc/G20TW5id61u3brhBty8eTNuQJlo167dyZMnRWj8+Oyzz9q0acP6KuaHtsvhgtFLzyiP+2RQdldmQ+k10KZBdaHUqbChcAWvRZJd3fNtaNFOsqsT0jCR+0ycnJwGDBgg5hWZQtw3hyMvLy8fHzkNRTWGn58fbsCUlBSj9hgyIV27dj148KBCwXbg7LfffltQUMD0EmYJvRXbLMtuFg+jrK2tg4ODEeIID2GckkQoSYTSFCjPgKo/Qff4yQNatAOFMyh9wN4fWg4CBcOta8XTdjI8joOiSLGv22oIOM8Q+6KEcDNhwgQxhxjMmDHDwkJOO1gJFh4efuzYMdEuJ0lfEBbh8wfq+/7771F+DcvQ0KFDd+zYMX36dHYdCGlpaSNHjoyNjVWpVIwuYZbolcom3b9//9y5c+hhR4wYYWNjIzyOiKvd+fvhRjikzoHMtaCOgfK7T6m5AaAqF0qvQ/7+/2/vvsOivLIGgJ9hkOLMwCB1RoqCSLEhgg0VUB7NIhYE/RTyGKKRZA0pGzUa3VhS1YWNu/kMAp8mNkiMDWVZowmrxgJRBEVDMSBIUxAFZugi3x+zS1gLZd573zKc3x88EWfOPQqRM3fOPRdKPoOcIChaCw057CVJj8MGMGL3LXWjITD0Y1ZXRKgvpk+fTntHrSvdGB3YFct9JsJt7AYAqVTKfODuUw4dOlRZWUk2Jn8EBwf3acqNFq5fv/7SSy81NjZSXUXHEN+c1r3d7r1799IIu3jxYiJx6Jfd7fVQuRuuz4SST6D5Tp+f/ugnyHsV8pZD3XkKybFILIPhf4cBlmwtJwXHrSDGXQTEXwYGBqwNLDMxMWF/OBptbPaZiEQiof8F0sj/4491eWtj1apVb775JtUlLl26NHv27NbWVqqr6BLc7e4RjXMXAwYMIHWKnXLZ/fAU3FwIFbFM+7YbrsNv70HBG9BSRigzLgywhuGxoG9OfSF9M3D5P35dTY/Q87DW3j179mz2DyCygLV5Jt7e3gQveuQEjYt+du3alZ2dTTwsf3z55Zdz586lusTZs2dDQkJ0b8+VEuzt7t6FCxfu3On7Dm9PQkJCpFIpkVDUyu4nzVC0Fu78GR6TOzOhugq3/gcenSYWkH1GQ8Aljm7DuqEtuO7BmhsJAmujRXRphklXrPWZCHeGSaeAgADizf0dHR1vvMGPyxnoEIlEycnJ06ZNo7pKSkrKK6+8QnUJnUG8Stax3e4//elPNMJGRUWRCkWn7G65C3kR8Ogn8pE7WqBoPZT+FUCwr8+MhsCI72mddJT7gXsiGNpRCY4QaYMHDyY+YuJZBgYGc+bMob0KJ1jrMxF0Y7eGXC6fMGEC8bAZGRm7d+8mHpZXUlJSaP9/evDgwT/+8Y9Ul9ANxA+56lLZvW/fvqtXrxIP6+HhQWRitwaFsrsxD3JfhabfyEfuVJUIt9+BJ+RnobNEzxjs14HrbpJ70noScPgzOEVzOSMcob5joc8kICDA2NiY9ipcYaHPRCKRTJ48mfYqLJg3bx6NsO+8805xcTGNyDwhk8nS0tKUSiXVVXbt2rVu3TqqS+iAtrY2sgGJDwLnSltb2+rVq2lEfv99kpefkC67G29BfiS01xEO+6z6dMh/HTpaqC9Ej2QMuH8LDh+CoS2jOHrGYBMBo5LBQjffRke6jYWxdLraYaLBQp+Jv7+/vj7r02YpWLp0KY0hkg0NDeHh4Tp5218npVKZlpZGe9jftm3bPv/8c6pLCB3xKllndru3bdtWXV1NPKy7uzupGSYaRMvu5kIoeBuesDUMqPEWFH4AHQJ/oWYxD0YeB8etIPXs83NlXmC3GkadhMFRoC/s006o3/L19R04kOJbNCKRaMGCBfTic46FPhMdaOzWUCqVlP4sly5d2rp1K43I/OHi4pKamkp76Of69et37dpFdQnhonH8UTd2uy9evLhp0yYakWNjY8m+VhcRe4He0Qa/vgzNhWSi9d7gKLCJYHtRSpp+A9Uv0HALGvKgpeT5j5GMBMkokI4BE28QU7+/l49aK6D2HNSeBQBozId2NcjGAQAYKEDuB3I/brNDWpg3b96JEycoBffx8blw4QKl4DyxZ8+e5cuX04ufm5vr6upKLz6bvv322yVLllAKfubMGaGPWexRVFTUzp07aa+yf//+l19+mfYqQkT87ZoJEyakp6eTjcmyBw8ejBo16t69e8QjL1iw4MiRI2Rjkiu7S6Oh6lsyofpEpA8uCSAZxcHSVD1pgMbb/97L1zMCfRMQm/T3Le2ak1ARD63d3k8hloLcD+xW4cxyAYmLi6M3DiI6OnrVqlWUgvOESqUyMzOjNAjMysrq/v37NCJzorm5WaFQ1NbW0gg+aNCgrKwse3t7GsH5YO3atdu3b2dnreTkZNqzCwVHrVYT7/Px9PTMzMwkG5NNLS0t/v7+ly9fJh7ZxMQkNzeX+JEGQk0mjblQdYhMqL7qeAx3twt4sMmL6ElA6gEyL5B5gWQkGNr365pbdRVy5kDxlh5qbgBoV0NNCuTMgcp4VjJDBAQFBdELHhpKZ2oQn1DtM6H61WGfkZHRihUrKAV/+PDh3LlzGxoaKMXnUH19/YwZM1iruQFg4cKFOv8+VV/RaAgRepNJeHg4jZobAL744gsax4iJlN3tUPIpAHdd+Y25UPUdZ6sj2qoSoeCNngvurtrVUBEPv4ZBu4paWoiYwYMHjxgxgkbkUaNGOTg40IjMN/TmmehMY3endevWSSQSSsGvX78eEBBQX19PKT4nMjIyxo4dm5aWxuaira2tf/jDH65du8bmojxHfIwJCLns7ujoiIiIIN4EojFr1qxly5bRiEyi7K5JhcY8AnGYqEgQ8DxB1I3izVD6Vy2f21QAv4ZBUz7RhBAVlMYI6vYMk64ozTMRiUQ0Lnfk1qBBg9auXUsvfnp6+tSpU2tqyF0Vx522trYNGzb4+PgUFRWxv7parZ45c2ZhIetnxvgKd7s7tbW1hYaG7t27l0Zwa2vrgwcP0ogMBMrujhaoSCCRCTPtKqhJ4ToJRFppNNMva2sl3NmCe978R2mMYP8puyn1mXh6egr9Tvjneu+996ysrOjFv3HjxpQpU2hcUs2mmzdvent7f/bZZxzeH15TU+Pn51dZ2Zd3O3UX7nZrlJaWTps27ejRozSC6+npHTt2zNzcnEZwIFB2112G1goSmTB2P4nrDBBRNSfJHNIVJ/oeAAATtElEQVRtKoBCKiP0EUH+/v7EjwopFApPz77P5RQsGn0mOnA55XNJJJKNGzdSXSIvL2/s2LEpKYLcD6qrq4uKivLw8Lh+/TrXuUBZWZm/vz+lU7DCQqNEFtzc7pMnT44ePZre9JVPP/100qRJlIIDgbK75iSJNEhoKcF2At3RroLSGGLRVJk8+kZFL0C8wmPh+kZeodFnonuN3Z1Wrlzp7e1NdYm6urq5c+euX79eQJXNkydPYmNjnZycdu7cyeEm91Py8/NnzZqlk2dV+4TGV4TGDjolKpXqtddemzt3Lr3XYBEREbSvSmVWdne0QB2VA6RaeoClla64nwjtapIBK+Kx1YTniLd3BwcHkw3Ic8T7TAwNDadOnUowIK+IRKKkpCQjIyOqq3R0dHz++ecTJkzIycmhuhBzT5482b9/v7u7+8qVK3nYmP7LL78EBga2trZynQiXaPzxhfJXeuzYMRcXl927d9NbIiQkhGp8DWZltzoHOvj0BVMJePYk+l1rBVSSPjDQWokb3jxHdkavsbGxDpeML0J2g3/69Om6cSf8izg5OW3bto2Fha5evTp69OgPPviAhbW0k5CQ4OTktHTp0vx8Ld80trW1tbW1JZvVU86fP9/fXks/hUaTCf/L7osXL06cOHHBggVUW/xnz559+PBhPT2id7c/D8OyO5tQGoQ03Ya2Kq6TQIxRevn0QJBNlv2HpaXlmDFjSEWbO3cujckePEe2z0RXG7u7evvtt319fdlZa+vWrUOHDv3qq6/YWa437t279/HHH9vZ2UVGRhYXF2sdx9LS8vz582fOnDExMSGY3rNSU1ODg4MF1LRDFo2GkJaWFuIxSUlPT58xY8aUKVMyMjKoLjRv3rzjx49TXaITs7K7pYxQGuQ03+U6A8SY5uJ34poK+HL8F70AwT6T/rkrRrbPRIcbu7tKSkpSKBTsrFVcXPzmm29aW1t/9NFHjx49YmfR5zp9+nRoaKhCodi4cWNZGaMf5ebm5ufPnx86dKirqyul4RJdHT9+fPny5bRX4Scavd3NzXwcvnz8+PGXXnpp0qRJLEyLX7hw4dGjR1l7Z49Z2d2s/YtjWniYEuqr2nO0ImMbEr+RGiNoYGDQb6+VXrRoEZE4SqWS0h1GfKNQKFJTUw0NDVlbsaqqatOmTQqFIjQ09Pjx46ydaSsvL4+Li5s/f75UKp01axaRe0ZMTEzOnDnj6uqq+eWMGTPoDTzu9M0336xcuZL2KjwkxGF/fZKVlbV69WobG5vg4OAffviBhRXfeuutQ4cOsdBb0olZdd/Cv73DNt4dBEF9Q/XgIw+/Y1EXkydPNjU1raurYxhnxowZxsbGRFISnNDQ0BUrVjDfFesPHSadPDw8Dh48GBoayuaiLS0tR44cOXLkiFwuX7JkyezZs6dPn078+7a5ufns2bM//vhjampqbm4u2eByufzUqVNjx47t+smwsLDy8vL333+f7FpPiY2NHThwYHR0NNVV+EYny+6qqqqLFy+eOHHi1KlT9+7dY21dPT29mJiYd999l7UVNZiV3TwcDcGrI55IC400p0Dy8DsWdSEWi2fNmnXo0CGGcfpnh4mGps/k1KlTDOP0q7IbAEJCQjZt2rRlyxb2l66trY2NjY2NjQWAgICAmTNn+vj42Nra2tvbaxfw3Llzt27dys/Pv3HjxtmzdHr2AFxdXVNSUpycnJ79rTVr1lRUVOzYsYPS0hoxMTFGRkZLly4dPnw41YX4g9Lxx/r6etpN+V2VlpZmZWVlZ2dfuXIlMzOTk7uQjIyMvv/++6CgIPaXFnV0dGj/7EwvcpkQYrEAHNZznQRiQHUVCt6gFVzqCS7xtIIjEvbu3RsREcEkgkgkqq6upnfHGP/t2bOHefPr/fv3qd7jyE9RUVE7d+7kOovfWVlZ2dvbW1tb29jYKBQKzUe5XK5Wqxv+o7Gx8eHDh0VFRSUlJUVFRey0jPv6+iYnJ5uamnbzmKVLl+7fv5+FZADA3t7eycnJ0dHR0dHRyclJ89+DBg1iZ3V6ysrKiouLS0pK7t69e+fOHU2pSnyV6upqCwsL4mE7Xbt27caNGzdv3szOzs7Kynr48CG9tXpj4sSJ+/btc3Z25mR1ZmX3tUnQwbNB6zavwOC3uE4CMUC17LZaDHZ4YyWvVVdXM6z2fHx8Lly4QCofIVKpVGZmZkz6TMaMGZOdzbNBVWxZuXKlZuMZvcirr76akJDQ48yc9vb2wMDA06dPs5PVs+Ryuab+7vrR3t6ezUbeXiouLtYU1l0/FhUVsbP64cOHtdinUKvV9fX1KpWqvr5erVbX1dXV19drPtPY2FhTU6P5jFpN9AoOZmQy2datW7k9GMCsyUQsg8ccv2p5msiA6wwQM4ZKisHFhK8fR8RZWlqOGzcuM1P7w6/9ucNEg3mfST+ZYfJcmgF/WHm/SGxs7Btv9GpnRCwWJycnz5w58+eff6ad1XPV1tZmZmY++49J53a45uPQoUOHDx8ukUho5/P48eOSkpKSkhLNBnbnx7KyMm4vBGX5VANXFi1aFBMTQ3u6fI+Yld36Zrwruw363buiusaAZtmtj2W3AAQGBjIpu/vJj5DuLVy4kEnZ3d8au5/y1VdfGRgY/O1vf+M6EX5RKpXJycleXn1oLjUyMkpJSRk/frzWt/DQUFhYWFhY+NQnLS0tNYV4WFjY7NmzSa114MCB1NRUTXldUYFn+rnh7OyckJDA2oT+7jF7q8WI4xcNz2Fox3UGiDGpJ63Icj9akRE5TMYIjh492sHBgWAyAsXk3hxDQ8Np06aRzUdwduzYsXv3btpXxwtIQEBATk5On2puDRMTk1OnTlFtHSaiuro6IyMjMTGRbHvV0aNHk5KSLl26hDU3JywsLKKjowsKCnhScwPjsnsooTTIMRrCdQaIMTM6xbGxM92tdETIxIkTuz+q1Y358+eTTUagmNybM23aNDaHWPPWsmXLfvnlF63HieiSLVu2nDlzRusTikOGDElNTWWhi4MIGxsbrlNABAwePHjHjh2lpaWrVq3iOpf/wqzsllHbldSOgQ0M4PtLatQzSnvSFnOohEWkiUQird/kxcbuTgsXLtTuif28w6SrUaNG3bhxIyAggOtEODN9+vTffvtt48aNDON4e3v//PPPcrmcSFZU9cMBPjrG3t4+Nja2rKzsnXfe4eEbVgzLbi8QDSCUCQky/g00RFowUII56WmaYimYY9ktGNrdEu/g4ODh4UE8GYHSus+kP5+nfJapqenp06d37twplUq5zoVVCoUiKSnpp59+eu5kbi2MHTv28uXLCoWCSDR6rK2tCUYTiUQEo6HueXp6fvfdd3fu3OnlqV9OMCu7RQYgG0coExJ0pnO39iyUxkDB6/BrGGR6QaYX/BoGBa9DaQzU0rr7gF+UkSAm+kNOGYljTAQkKChIix9XCxYsoJGMQGnXZ2JlZTVmzBga+QiXSCRauXJlbm6uv78/17mwZNWqVQUFBYsXLyYb1tXVNSMjw8XFhWxYssg2mTCa0Yx6zc/PLzExMTMzc9GiRTwcENkV4+QGzSKRBgn6cjCdwnUSzLSroDIesv2gcDVUJYEqE5oK/v1bTQWgyoSqJChcDdl+UBmv4xcuGihBGUksmrEzWIURi4boMzU1HT9+fF+fhY3dT9GizwQ7TF7E1tY2LS3t66+/NjMz4zoXWvT19SMjIysqKqKjoynt7tvZ2V2+fHny5Mk0ghPB//141MnR0XHt2rWFhYX/+te/lixZwnU6vcK47Jb7Ed6V1JplKIiYzUPkVkUc5MyBinho72m2fLsaKuIhZw5U6vSFi1ZhZFpNxFK8mVKI+tpnYm5uPnXqVErJCJQWfSZYdncvIiLi9u3bUVFR+vpC/nHzDD09vfDw8IKCgri4ONp1p5mZ2blz58LDw6muoh1TU9MBA/jUOoueZ8yYMVu2bMnJySksLNy6daujoyPXGfUB47JbLAMrHrzCEEvB+mWuk9BWuwqKN0NlQs8F9389Sw0V8VC8WZe3vYdsZjpMUCwFlzhsLxGivo4RnD9/PrZRPkWLPhMm0xv7CXNz8y+//DIvL09nzu8uW7bs5s2bBw4cGDqUpelk+vr6Bw4c2L59OzvL9R6OMeEtZ2fnFStWHDx4sLy8PDs7e+PGjSNHjuQ6KW2Q6ICxDgOxltO+iFG8xpdN975qV0H+61CTouXTa1Ig/3Vdrrxd4sFK2/5CY2dwTwRjXvcRohfx8vKytLTs/eN1pgYiq099JiNGjMAxDr3k5OR09OjRCxcu8Llfons2NjYfffTRgwcPdu/e7ebmxn4Ca9asycjIcHd3Z3/pFyH+/Y97AVpTKpWBgYEbNmw4fPhweXl5QUFBfHx8WFiYUinsQcAkym6xDOzeJRBHawNdwZoHO+7ayY/8vYFbO00FULiaUDa8ZLcanKLBoI+bEOZB4BKPg7qFSyQS9X7n1djYGLsjnqtPfSY4w6SvfHx8Ll68mJWVFRERwcNRZS8yfvz4ffv2VVZWfvjhh+bm5txmcuvWre3btxsbG3OYRids7ObQ2LFjly5d+pe//OXMmTMPHjwoLy//xz/+8cknn4SEhAi91O6KUHea+Rx4+APUp5OJ1lcO6wG0vI+NY8Wboek2gTiqTCjeDEM2EwjFT3I/kI2D+4lQldRzK455ECgjseDWAYGBgfv27evNI4OCggwMDGjnI0SaPpNeXhSPL1204+Hh8fXXX8fExCQkJMTGxpaUlHCd0fNNmjQpNDQ0NDSUb3cArVmzJiwsLDo6Oi4urqmpicNMyE4PRF1JJBILCwvL/7CystJ8VCgU9vb2nLzfwgkRsek2bQ/g1kIOuh0GR4FNBNuLEqG6CgVER0sO39UvJperrkLtWWgsgNYKaL33709KPUFfBnI/kPthJ7fOqKurMzMz682/UYmJiUI5xs6+PXv2LF++vMeH6evrq9VqvJ+SoY6OjvT09OTk5OTk5Ly8PK7TAQDw9/cPDg4WxJZhXV1dQkJCTEzMvXv3en40BZ988smGDRsIBrx161Z1dTXBgIJjampqYWFhZ2fHdSJ8Qa7sBoC68/Dbe8Si9YbcF5xiWF2RoPxIUF8jGVA2DobHkQyIENemTJly8eLF7h8jFosfPnxoYmLCTkqCo1KpzMzM2tvbu3+Yv79/WloaOyn1E7dv3z527Ng///nP9PT05uZmNpcePXp0QEBAQECAr6/vwIED2VyaiLS0tJMnT544caKoqIiF5SQSiY+Pj6+vb3h4uIODAwsron6LaNkNANWH4C5bZ5Nl48D57yAS5t4M8a1ujX6y4Y36jdu3b5eXl3f/GIlE4u3tzU4+AnXlypWGhobuH2Nrazts2DB28umHsrKyMjMzr1y5kpmZmZmZSTa4QqFwc3Nzc3Nzd3cfMWKEm5ubzhyNzcnJOX/+fHp6+uXLlwsLC0mFHTZs2Jj/cHNzc3Z2JhUZoe6RLrsBoPoI3P2ccMxnSUaD89+FOr0EAIo3az+9pBvmQbrc4Y0QQjrhzp07d+/eLSkpKS0tLS4uLi0tra2tbWpqamxs7PzY3NwskUhkMplUKpVKpSYmJppfasjl8iFDhmiqbZmsX3TW1dTUXLt2raysrLKysry8vKKioqampr6+Xq1Wq9XqhoYGtfrpkz8ymUyhUCiVShsbGxsbG0dHR09PTw8PD4lEwskfASEKZTcA1F+GwnXwpIfNFe1JPcF5B+gJ742z32X79W1Kdy8ZKGDUSfJhEUIIIYQQM3RurjeZBG57wZBCg5TIAAa/CS67hF1zN+VTqbkBoLUSmvKpREYIIYQQQgzQKbsBwGgIuO0Fy0UkY8q8YMQhsHmVYtrseExz3gvV4AghhBBCSCs061exFOzfB9dvwGQi01CSkTDsCxi+CwxtSWTGNYb343AYHCGEEEIIaYXQdTndkIwE5/+FxjyoPgKPfuzzYG+zALAOB8koOslxBHe7EUIIIYT6Gfplt8ZAV3DYAA4boD4dHqVB3Xloe/DCBxsPB4kbSEaAyeQ+XwkuCPo0T51TDY4QQgghhLTCVtndyWQimEwEWA+Pa6HtATyu/a/fFRvDwBFsp8Q+4+FCDY4QQgghhLTCetn9+8py0Jdztjq3cLcbIYQQQqifEfhIEIEydqF10Y+BDRi7UImMEEIIIYQYwLKbI3I/KmHxZniEEEIIIV7Cspsj5kFCCosQQgghhJjBspsjMi+QehKOKfXE3W6EEEIIIX7Csps7yki+B0QIIYQQQoRg2c0dmRfJnhCrxbjVjRBCCCHEW6KOjg6uc+jf8iNBfY1pEKknuMSTyAYhhBBCCFGBu91cGxYDxs6MIhg7w7AYQtkghBBCCCEqsOzmmlgGLvHad5uYB4FLPIjxihyEEEIIIV7DJhPeqIiDqiRoV/f28WIpKCPBKoxmTgghhBBCiAwsu/mkXQX3E3suvsVSsFoC1mG4yY0QQgghJBRYdvOS6irUnoXGAmhXQdNtAABjZxDLYOBwkPvhxBKEEEIIIcHBshshhBBCCCHq8EglQgghhBBC1GHZjRBCCCGEEHVYdiOEEEIIIUQdlt0IIYQQQghRh2U3QgghhBBC1GHZjRBCCCGEEHVYdiOEEEIIIUQdlt0IIYQQQghRh2U3QgghhBBC1GHZjRBCCCGEEHVYdiOEEEIIIUQdlt0IIYQQQghRh2U3QgghhBBC1GHZjRBCCCGEEHVYdiOEEEIIIUQdlt0IIYQQQghR9/+Eyl8T3kw2YAAAAABJRU5ErkJggg=="}}},{"cell_type":"code","source":"!pip install -qq albumentations==1.0.3\n!pip install wandb --upgrade\n!pip install timm\n!pip install torch==1.10.0","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:17:08.250993Z","iopub.execute_input":"2022-03-27T12:17:08.251999Z","iopub.status.idle":"2022-03-27T12:18:44.351599Z","shell.execute_reply.started":"2022-03-27T12:17:08.251879Z","shell.execute_reply":"2022-03-27T12:18:44.350726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### About this notebook?\n* The code in this notebook is modularized using classes and functions.\n* All the hyperparameters are well placed inside a single class, hence its easy to change them.\n* Use of weights and biases will aloow you to monitor the performance of your model.\n* Used swin transformer.\n* Have used Focal loss which is an improved version of cross entropy loss.\n* Reference materials\n\n#### ***Do upvote this notebook if you have liked the content.***","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport cv2\nimport os\nimport functools\nimport seaborn as sns\nfrom tqdm.auto import tqdm\nimport matplotlib.pyplot as plt\nimport torch\nimport timm\nfrom torch.utils.data import Dataset, DataLoader\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchmetrics import AveragePrecision, Recall, F1\nfrom sklearn.metrics import precision_recall_fscore_support\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nimport albumentations\n\nfrom albumentations.pytorch.transforms import ToTensorV2\n\nimport wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:18:44.355238Z","iopub.execute_input":"2022-03-27T12:18:44.355465Z","iopub.status.idle":"2022-03-27T12:18:56.195575Z","shell.execute_reply.started":"2022-03-27T12:18:44.355440Z","shell.execute_reply":"2022-03-27T12:18:56.194862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.read_csv('../input/sorghum-id-fgvc-9/train_cultivar_mapping.csv')\nsub_csv = pd.read_csv('../input/sorghum-id-fgvc-9/sample_submission.csv')\ndata.shape, sub_csv.shape","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:28.337943Z","iopub.execute_input":"2022-03-27T12:20:28.338219Z","iopub.status.idle":"2022-03-27T12:20:28.407836Z","shell.execute_reply.started":"2022-03-27T12:20:28.338180Z","shell.execute_reply":"2022-03-27T12:20:28.407127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(list(data.cultivar.unique()))","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:29.622101Z","iopub.execute_input":"2022-03-27T12:20:29.622673Z","iopub.status.idle":"2022-03-27T12:20:29.639219Z","shell.execute_reply.started":"2022-03-27T12:20:29.622633Z","shell.execute_reply":"2022-03-27T12:20:29.638572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Class Imbalance","metadata":{}},{"cell_type":"code","source":"class_counts = dict(data.cultivar.value_counts())\nkeys = list(class_counts.keys())\nvalues = list(class_counts.values())\nplt.figure(figsize=(10, 20))\nsns.barplot(values, keys)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:30.754163Z","iopub.execute_input":"2022-03-27T12:20:30.754721Z","iopub.status.idle":"2022-03-27T12:20:32.584924Z","shell.execute_reply.started":"2022-03-27T12:20:30.754680Z","shell.execute_reply":"2022-03-27T12:20:32.584148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configurations","metadata":{}},{"cell_type":"code","source":"#None will be filled later during the process.\nclass Config:\n    #general.\n    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n    proj_name = 'Sorghum Kaggle Competition'\n    submission_file = 'submission.csv'\n    \n    #dataset params.\n    train_folder = '../input/sorghum-id-fgvc-9/train_images'\n    test_folder = '../input/sorghum-id-fgvc-9/test'\n    sub_folder = '../input/sorghum-id-fgvc-9/test'\n    val_percent = 0.2\n    \n    #model params.\n    model_name = 'swin_base_patch4_window7_224'\n    img_dim = 224\n    out_features = len(list(data.cultivar.unique()))\n    in_channels = 3\n    pretrained = True\n    dropout = 0.25\n    \n    #train params.\n    epochs = 2\n    batch_size = 16\n    learning_rate = 2e-5\n    penalty = 2 #loss.\n    transform = None\n    ","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:26:22.941866Z","iopub.execute_input":"2022-03-27T12:26:22.942435Z","iopub.status.idle":"2022-03-27T12:26:22.950540Z","shell.execute_reply.started":"2022-03-27T12:26:22.942396Z","shell.execute_reply":"2022-03-27T12:26:22.949850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset Class","metadata":{}},{"cell_type":"code","source":"class sorghumDataset(Dataset):\n    \n    def __init__(self, images, labels):\n        self.images = images\n        self.labels = labels\n        \n    def __len__(self):\n        return len(self.images)\n    \n    def __getitem__(self, index):\n        return (self.images[index], torch.tensor(self.labels[index]))\n    \nclass sorghumTestDataset(Dataset):\n    \n    def __init__(self, images):\n        self.images = images\n        \n    def __len__(self):\n        return len(self.images)\n    \n    def __getitem__(self, index):\n        return self.images[index]","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:45.112041Z","iopub.execute_input":"2022-03-27T12:20:45.112689Z","iopub.status.idle":"2022-03-27T12:20:45.119246Z","shell.execute_reply.started":"2022-03-27T12:20:45.112646Z","shell.execute_reply":"2022-03-27T12:20:45.118534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Swin Transformer\n\nThe SwinT (Swin Transformer) is a type of Vision Transformer. SwinT is the first transformer-based backbone architecture for visual tasks. The contribution of the paper is that the authors are addressing a couple of problems that are faced while employing Transformers for visual tasks, like the large variations in the scale of objects in the image and also the pixel count in images compared to words in the text. Calculating self-attention with every other vector is computationally inefficient. Hence the authors have proposed a hierarchical transformer with a shifted window mechanism for calculating self-attention. The hierarchical structure will enhance the flexibility of the model. Shifted window mechanism will limit the computation of self-attention to local windows and also allow for cross-window connection.","metadata":{}},{"cell_type":"markdown","source":"## Model Class","metadata":{}},{"cell_type":"code","source":"class sorghumModel(nn.Module):\n    \n    def __init__(self, model_name, out_features, in_channels=3, drop_prob=0.2, pretrained=True):\n        super(sorghumModel, self).__init__()\n        self.out_features = out_features\n        self.in_channels = in_channels\n        \n        self.pre_model = timm.create_model(model_name, pretrained=pretrained, in_chans=in_channels)\n        in_features = self.pre_model.head.in_features\n        self.pre_model.head = nn.Linear(in_features, 1024, bias=True)\n        \n        self.cnn_head = nn.Sequential(\n            nn.Linear(1024, 512, bias=True),\n            nn.ReLU(),\n            nn.Dropout(drop_prob),\n            nn.Linear(512, 64, bias=True),\n            nn.ReLU(),\n            nn.Dropout(drop_prob),\n            nn.Linear(64, self.out_features, bias=True),\n        )\n        \n        self.drop_layer = nn.Dropout(drop_prob)\n        \n    def forward(self, image):\n        image_feats = self.pre_model(image)\n        image_feats = self.drop_layer(image_feats)\n        preds = self.cnn_head(image_feats)\n        return preds","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:50.611257Z","iopub.execute_input":"2022-03-27T12:20:50.611995Z","iopub.status.idle":"2022-03-27T12:20:50.620461Z","shell.execute_reply.started":"2022-03-27T12:20:50.611955Z","shell.execute_reply":"2022-03-27T12:20:50.619706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_transform_object(DIM = 384):\n    return albumentations.Compose(\n        [\n            albumentations.Resize(DIM,DIM),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.1, 0.1),\n                contrast_limit=(-0.1, 0.1), p=0.5\n            ),\n            albumentations.Flip(p=0.5),\n            albumentations.Rotate (limit=90, always_apply=False, p=0.5),\n            albumentations.CenterCrop (height=Config.img_dim, width=Config.img_dim, always_apply=False, p=0.5),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(p=1.0),\n        ]\n    )\n\ndef valid_transform_object(DIM = 384):\n    return albumentations.Compose(\n        [\n            albumentations.Resize(DIM,DIM),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(p=1.0)\n        ]\n    )\n\ndef collate_fn(batch, process):\n    images, labels = [], []\n    for sample in batch:\n        im_name, im_label = sample[0], sample[1]\n        im = cv2.imread(os.path.join(Config.train_folder, im_name), 1)\n        if Config.transform is not None:\n            if process == 'training':\n                im = Config.transform['train_transform'](image=im)['image']\n            else:\n                im = Config.transform['valid_transform'](image=im)['image']\n        images.append(im)\n        labels.append(im_label)\n    images_tensor = torch.stack(images)\n    labels_tensor = torch.stack(labels)\n    return images_tensor, labels_tensor\n\ndef collate_test_fn(batch):\n    images = []\n    for sample in batch:\n        im_name = sample\n        im = cv2.imread(os.path.join(Config.test_folder, im_name), 1)\n        if Config.transform is not None:\n            im = Config.transform['valid_transform'](image=im)['image']\n        images.append(im)\n    images_tensor = torch.stack(images)\n    return images_tensor","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:53.842125Z","iopub.execute_input":"2022-03-27T12:20:53.842401Z","iopub.status.idle":"2022-03-27T12:20:53.857028Z","shell.execute_reply.started":"2022-03-27T12:20:53.842371Z","shell.execute_reply":"2022-03-27T12:20:53.856270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Focal Loss\n> **Focal loss is an extension and improved version of cross entropy loss that focuses more on samples that are hard to classify than the ones that are easy to be classified. This reduces the overconfidence of the model. Just to note overfitting and overconfidence are two different terms.***\n\nFor more understanding on Focal Loss do have a read on [this](https://medium.com/visionwizard/understanding-focal-loss-a-quick-read-b914422913e7) Medium post. Its really good!!\n\nThe code for focal loss have been copied from [this](https://github.com/gokulprasadthekkel/pytorch-multi-class-focal-loss/blob/master/focal_loss.py\n) amazing github repository. Do check it out!!","metadata":{}},{"cell_type":"code","source":"class FocalLoss(nn.modules.loss._WeightedLoss):\n    def __init__(self, weight=None, gamma=Config.penalty,reduction='mean'):\n        super(FocalLoss, self).__init__(weight,reduction=reduction)\n        self.gamma = gamma\n        self.weight = weight #weight parameter will act as the alpha parameter to balance class weights\n\n    def forward(self, input, target):\n\n        ce_loss = F.cross_entropy(input, target,reduction=self.reduction,weight=self.weight)\n        pt = torch.exp(-ce_loss)\n        focal_loss = ((1 - pt) ** self.gamma * ce_loss).mean()\n        return focal_loss","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:20:56.894944Z","iopub.execute_input":"2022-03-27T12:20:56.895208Z","iopub.status.idle":"2022-03-27T12:20:56.901883Z","shell.execute_reply.started":"2022-03-27T12:20:56.895172Z","shell.execute_reply":"2022-03-27T12:20:56.901195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_one_epoch(train_loader, model, epoch, criterion, optimizer):\n    model.train()\n    stream = tqdm(train_loader)\n    total_loss = 0\n    images_done = 0\n    for i, (im_tensors, tar_tensors) in enumerate(stream, start=1):\n        im_tensors = im_tensors.to(Config.device, non_blocking=True)\n        tar_tensors = tar_tensors.float().view(-1, 1).squeeze(1).type(torch.LongTensor).to(Config.device, non_blocking=True)\n\n        output = model(im_tensors)\n        \n        loss = criterion(output, tar_tensors)\n        total_loss += float(loss)*len(im_tensors)\n        images_done += len(im_tensors)\n        \n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n    \n    return total_loss/images_done\n        \ndef validate(val_loader, model, epoch, criterion):\n    model.eval()\n    stream = tqdm(val_loader)\n    final_targets = []\n    final_outputs = []\n    total_loss = 0\n    images_done = 0\n    with torch.no_grad():\n        for i, (im_tensors, tar_tensors) in enumerate(stream, start=1):\n            im_tensors = im_tensors.to(Config.device, non_blocking=True)\n            tar_tensors = tar_tensors.float().view(-1, 1).squeeze(1).type(torch.LongTensor).to(Config.device, non_blocking=True)\n    \n            output = model(im_tensors)\n\n            loss = criterion(output, tar_tensors)\n            total_loss += float(loss)*len(im_tensors)\n            images_done += len(im_tensors)\n            \n            target = (tar_tensors.detach().cpu().numpy()).tolist()\n            output = (output.detach().cpu().numpy()).tolist()\n            \n            final_targets.extend(target)\n            final_outputs.extend(output)\n\n    return total_loss/images_done, torch.tensor(final_targets), torch.tensor(final_outputs)\n\ndef test(model, test_loader, id_class):\n    sub_csv = pd.read_csv('../input/sorghum-id-fgvc-9/sample_submission.csv')\n    sub_csv = sub_csv.loc[:100]\n    \n    model.eval()\n    stream = tqdm(test_loader)\n    final_outputs = []\n    with torch.no_grad():\n        for i, im_tensors in enumerate(stream, start=1):\n            im_tensors = im_tensors.to(Config.device, non_blocking=True)\n            output = model(im_tensors)            \n            output = (output.detach().cpu().numpy()).tolist()\n            final_outputs.extend(output)\n    \n    classes = cvt_classes(torch.tensor(final_outputs))\n    sub_csv['cultivar'] = classes\n    sub_csv['cultivar'] = sub_csv['cultivar'].map(id_class)\n    return sub_csv","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:30:50.681124Z","iopub.execute_input":"2022-03-27T12:30:50.681663Z","iopub.status.idle":"2022-03-27T12:30:50.698644Z","shell.execute_reply.started":"2022-03-27T12:30:50.681627Z","shell.execute_reply":"2022-03-27T12:30:50.697804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def remove_missing_images(data):\n    images = data['image'].values\n    indices = []\n    for i in range(data.shape[0]):\n        im = data.image.iloc[i]\n        if not os.path.exists(os.path.join(Config.train_folder, im)):\n            indices.append(i)\n    data = data.drop(indices, axis=0).reset_index(drop=True)\n    return data\n\ndef setup_training(hyp):\n    data = pd.read_csv('../input/sorghum-id-fgvc-9/train_cultivar_mapping.csv')\n    data = data.loc[:1000]\n    data = remove_missing_images(data)\n    \n    classes = list(data.cultivar.unique())\n    id_class = dict([(k, v) for k, v in enumerate(classes)])\n    class_id = dict([(v, k) for k, v in id_class.items()])\n    data['cultivar'] = data['cultivar'].map(class_id)\n\n    images, labels = data.image.values, data.cultivar.values\n    train_data, val_data, train_labels, val_labels = train_test_split(images, \n                                                                      labels,\n                                                                      test_size=Config.val_percent, \n                                                                      stratify=labels, \n                                                                      shuffle=True)\n    \n    train_dataset = sorghumDataset(train_data, train_labels)\n    val_dataset = sorghumDataset(val_data, val_labels)\n    \n    train_collate = functools.partial(collate_fn, process='training')\n    valid_collate = functools.partial(collate_fn, process='validation')\n    train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=Config.batch_size, collate_fn=train_collate)\n    val_dataloader = DataLoader(val_dataset, shuffle=False, batch_size=Config.batch_size, collate_fn=valid_collate)\n    \n    model = sorghumModel(model_name=hyp.model_name, \n                         out_features=hyp.out_features, \n                         in_channels=hyp.in_channels, \n                         drop_prob=hyp.drop_prob,\n                         pretrained=hyp.pretrained)\n    model.to(Config.device)\n    loss_fn = FocalLoss()\n    optimizer = torch.optim.AdamW(model.parameters(), lr=hyp.lr_rate, weight_decay=1e-6, amsgrad=False)\n    \n    return id_class, model, train_dataloader, val_dataloader, loss_fn, optimizer\n\ndef setup_testing():\n    test_data = pd.read_csv('../input/sorghum-id-fgvc-9/sample_submission.csv')\n    test_data = test_data.loc[:100]\n    test_images = test_data.filename.values\n    test_dataset = sorghumTestDataset(test_images)\n    test_dataloader = DataLoader(test_dataset, shuffle=False, batch_size=Config.batch_size, collate_fn=collate_test_fn)\n    return test_dataloader","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:24:15.534679Z","iopub.execute_input":"2022-03-27T12:24:15.535386Z","iopub.status.idle":"2022-03-27T12:24:15.549952Z","shell.execute_reply.started":"2022-03-27T12:24:15.535347Z","shell.execute_reply":"2022-03-27T12:24:15.549089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nConvert the output probablities of the model to the class\n'''\ndef cvt_classes(outputs):\n    outputs = torch.nn.functional.softmax(outputs)\n    outputs = outputs.detach().cpu().numpy()\n    outputs = outputs.argmax(axis=1)\n    return torch.tensor(outputs).int()\n\ndef train(model, tr_loader, val_loader, criterion, optimizer, params):\n    wandb.watch(model, criterion, log='all', log_freq=10)\n    for epoch in range(params.epochs):\n        train_loss = train_one_epoch(tr_loader, model, epoch, criterion, optimizer)\n        val_loss, targets, predictions = validate(val_loader, model, epoch, criterion)\n        outputs = cvt_classes(predictions)\n        accuracy = accuracy_score(outputs, targets) \n        wandb.log({ 'epoch' : epoch, 'train_loss' : train_loss, 'val_loss' : val_loss, 'accuracy' : accuracy }, step=epoch)\n        torch.save(model.state_dict(), '{}_epoch{}.pth'.format(params.model_name, epoch))\n\ndef model_pipeline(train_parameters):\n    with wandb.init(project=Config.proj_name, config=train_parameters):\n        parameters = wandb.config\n        id_class, model, train_loader, val_loader, criterion, optimizer = setup_training(parameters)\n        train(model, train_loader, val_loader, criterion, optimizer, parameters)\n        test_loader = setup_testing()\n        result = test(model, test_loader, id_class)\n        result.to_csv(Config.submission_file, index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:21:48.357314Z","iopub.execute_input":"2022-03-27T12:21:48.357643Z","iopub.status.idle":"2022-03-27T12:21:48.380045Z","shell.execute_reply.started":"2022-03-27T12:21:48.357605Z","shell.execute_reply":"2022-03-27T12:21:48.378700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**As you finish training you will have a submission.csv and the model saved to the output directory**","metadata":{}},{"cell_type":"markdown","source":"## Train and test the model","metadata":{}},{"cell_type":"code","source":"train_params = {\n    'model_name' : Config.model_name,\n    'out_features' : Config.out_features,\n    'in_channels' : Config.in_channels,\n    'drop_prob' : Config.dropout,\n    'pretrained' : Config.pretrained,\n    'epochs' : Config.epochs,\n    'lr_rate' : Config.learning_rate,\n}\n\nConfig.transform = {\n    'train_transform' : train_transform_object(Config.img_dim),\n    'valid_transform' : valid_transform_object(Config.img_dim)\n}\n\nmodel_pipeline(train_params)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T12:30:54.438313Z","iopub.execute_input":"2022-03-27T12:30:54.439180Z","iopub.status.idle":"2022-03-27T12:33:15.404659Z","shell.execute_reply.started":"2022-03-27T12:30:54.439134Z","shell.execute_reply":"2022-03-27T12:33:15.403959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}