{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n

Quick navigation

\n\n* [1 Preface ](#1)\n* [2. What is TPU ](#2)\n* [3. Setting Up the Environment ](#3)\n* [4. Loading Competition Data ](#4)\n* [5. Defining Augmentations ](#5)\n* [6. Modelling ](#6)\n* [7. Some Pointers to Improve Model's Performance ](#7)\n\n\n\n \n"},{"metadata":{},"cell_type":"markdown","source":"\n

1. Preface and About the Kernel

\n"},{"metadata":{},"cell_type":"markdown","source":"1. This Kernel deals with how to Train Deep Learning Models on TPU.\n2. I was learning how to use TPU after discovering about them while doing Flowers Classification using TPU [here](https://www.kaggle.com/c/tpu-getting-started/) \n3. In this kernel I have Also Demonstrated K Fold Cv and Augmentations.\n4. I have trained / FineTuned Efficient Net B0.\n5. Please Feel free to point out some improvement tips ."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Import Libraries \nimport random, re, math\nimport numpy as np, pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nimport tensorflow as tf, tensorflow.keras.backend as K\nfrom kaggle_datasets import KaggleDatasets\nprint('Tensorflow version ' + tf.__version__)\nfrom sklearn.model_selection import KFold\nfrom functools import partial\nimport keras\nimport pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\n\n## Install Efficient Net \n!pip install -q efficientnet >> /dev/null\nimport efficientnet.tfkeras as efn\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

2. What is TPU ?

\n"},{"metadata":{},"cell_type":"markdown","source":"1. A TPU can be thought as of Multiple GPUS clubbed in One Machine .\n2. TPU can be used for parallelising Training of Machine Learning Models.\n3. A Typical TPU has 8 cores and each core is like a GPU .\n\nTo Know more about TPU you can refer the below video ."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import warnings\nfrom IPython.display import YouTubeVideo\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\n\nfrom IPython.display import HTML\nfrom IPython.display import YouTubeVideo\n\nYouTubeVideo('MXxN4fv01c8', width=800, height=300)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is how a TPU v3 looks like(Image Credits Wikipedia) .\n![tpuv3.jpg](attachment:tpuv3.jpg)","attachments":{"tpuv3.jpg":{"image/jpeg":"/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxMSEhUTExMWFhUXGB0YGRcYGB4YFhoYFhgYFxcYFx8YHSggGBolHRgXITEhJSkrLi4uICAzODMtNyotLisBCgoKDg0OGhAQGzYlICYvLS0uLTAvMi0wLS0tNS0tLy0tLzUtLS0tLS0tLS0vLSstLS0tLi0vLS0tLS0tLS0tLf/AABEIAL4BCgMBIgACEQEDEQH/xAAcAAABBQEBAQAAAAAAAAAAAAAGAgMEBQcBAAj/xABMEAACAQIEAwQFBwcMAAUFAAABAhEAAwQSITEFQVEGEyJhMnGBkbEHFCNCocHRUmJykrLh8BUkMzRDU3OCg6LC0hZjk7PxF0RU0+L/xAAZAQACAwEAAAAAAAAAAAAAAAABAgADBAX/xAAuEQACAgEEAQIFBAIDAQAAAAAAAQIRAwQSITFRE0EiMmFxkRShsfBCgSNS4TP/2gAMAwEAAhEDEQA/ANZIpq88AxGaDAJiTy+2pFMXlk7cqS6GAy725a07W72GyuglgLqnkDpIg6EaA0uz8omFPpLdT1qD+yaJb3DrT+natt+kin7qg3ezGDbfD2x+jK/skVpWTA/mj+GV7cnsyg4t2gfGFUwF0qUGe4x8Eg6KozD1k+ymuH8UxagLcuu1xna3oiOlsrt3hCgkHrNX1jslhrZLWu8tEiCUuHbf601Hv9jwS7Lirym56R8OpiJOULS5JRlHbB191z/v+/QMbi7aspMN2sx+Qu1my6jNtIYhDDFRnkgQdQKXY+UQ65sNoOa3AeusFfI1YWuxpQIQ6O9oMLbMGWA/JgpOaJoWvdisSWNlTbYosz41EHQQzJlJ12mrMcoSct6S8drj8iTUlW1v6hJa+UXCn0kur/lVh/tafsqfY7a4Fv7fL+kjr9pWKyzj/Z29g8purAaYIIMkROxPUb1U2pYxP2zTrFCXS/DBvku3+xvFjjuFf0MTZJ6d4s+6ZqejA6ggjyM/CsE5Hw7c5PKCeccj76awoOYZZDTpl3PPlTT0sE63Cxzt+x9AxXorErHGcSi5lv3VgnTvH0yxIhiatsJ2oxoB/nExGjIhEMARrE86pen8SRYsvlGrRXqBf/FmIW3P0b3BaW8wNtlTI5AAVg/pCRIjrTOH+UdoJfDLCxJW7B100DLr76qhilNNx5p0PKai+TQIr1Bq/KJYEZ7N5ZEgjIwg89G+6rbCdq8PcUOveZTzyHkYOgk1JY5xVtBUk+i8iuxVanHsOf7UD9IFf2gKl2cZbf0biN6mB+BqsYfr1dFeqBOVwilVyoQQRSSKcNJNAIikkUs0k0QCCKSaWaSahBBpJpZpJokEGkU4aTUIERFM3RrUiKZujWkAMrqJ9f2GK7FLVf49etey0CCAK7FKivBaJCNjLbssW3CNIMlcwgHxLHmNJ5VWYVP57cJAnuVkgfnDnkn2ZvZzq7IqqtgfPH8JzdyPFIiMw0jLM/5iPKiQE/lbX6Ox+k//AANASuxhTbgxbQNsTlyCCSdog6bTPOtF+VC3KYcHncIO8Qcu9ZxcsOSEksWIgeZhRM7bAT09VdHDjU8cXdVZiy5HGbVd0IxFgJJ3kZoIjQzA0J8quewWPTvnUjKzW4R13EHM2seGQD4joI1gVIw3ZR+4c3LdxbozlRO+VQUEAEEMc06g6ADeonYjDFcdbR0KmHkEFT/RsRINJKe5NRfC/f7DxjTTku/2DjBYC1fMX7Vu5cyjM48XiBbTMIzeErr5U7f7HYRgR3RUH8l25bbk1aOyWyiBYLkquVYEgFvEV9EQDvUvI35X2msu5+TSooGb3ZC2bfdrexCoDITMGWd9iOvKq2/2F3y4kwYkNb3jaYajeW6/CmLtx/yQfZ+BorJKPTA4J9mc43saTCm+ouRktjJ4WIB0LaEEw3I7TTHZwn5uojZm9+Y+cUcY7Du5BTIrAypZcxQnQsnQlZGvr6yPdjeHI2FWSsm5dC5jDtlczy8RoublGgKNMQjHz9n/APIqbg3WfER7R/2p65w8ef2/eRTYwA5H4fcDVY5L4uE7m9ctkArZZg1u6VcOqsQYR9dl0j8KosTi8VZwNrFLj7kuFJDZXXx7hcwJLLrI8jtVnxKyRhcRMwLL6mY9A9aDuI8KwycOtX1uk3TrkzyC7emoT6pWDr5a71bGXVgrsN7fHL+UlbqsPEQ75CkBMys/dqpVevT11V4T5Q8RpntWjPMZ18/zqpOK8MwCPhBaYNnP0hVsxNsCWdo1Q6HpGvSiYdmbN4ktbNsqYU2SArKVHiggg7kTEmKkXFLnkWV3wTOH9uO8PjsqihS7P3oKqoAJLeHSJA9tPN25sLcNp7d5WBj0VYHmCMrEkEajTaq5OyKo2ZL9weHJldVuIViCGUwCCOVQ8Z2PuO5f5wjMWzeK1G2gGhIiIERyFKtrlz1X7hd1wEtntjg2/tSP0rbgab65Y+2plvj2FaIxFrXaXA+JrOcd2TuLkQ3EiWYlcxIXKPq5RJkQBPMVExfFWwyLhyqXFGR1cMwzLnFwaEkCYK+XnzO2L6Bufua5bvq3osreog/ClEVjeI4vZvEu1iIXLpD6gtDE+Ej0h12qBhuIXVBIxFxY2yuwnbTQ9Ndenqp44G0K8yRuBpJrHcH2qxQIBxNwDTUw8aiSc4Own7KKuF9pb7Yu1aZlZH0bVHKmHjx2gFMhVbbZopJ4nF0NHImGxrldNJqssCY0zcGtPxTbjWq2BDaaj3/HSuxS0QDb1+/U10ioSxqK7FKivZahCLjcL3i5SzrqDKNlbwkHfppqOYqss2yMa06/QjxEanxDeFA+32VdvA1JgeelUwYfPysW83cTML3sZtp7zMV8skefKiQGvlWMWrOn1zr5xt9grP8ACk99aM6Z16aeIaab/wAda0D5VDCWJkgXAY6jWRp+6s9w6xctmfrLpOm4jnp/HSungf8Aw19zBm/+v4NfjXT7P3LVfw7CKcbfukSy27Sgnkrd6WiesD3euZ7gSf3fexqBhc4xGIyQTksHltN8HYVgs3In/Pd4tvAJWRl+qSp3aYkV3+UeouD/AClvhNRbMw36dz/3GpYHlWCWpkpNHShpISinY9/Kic2j1qR8VFcHErZ2uW/1hPxqLeWKpcbj3dmtpoFIDMT1AOnTQjzp8WeU5baEzaWGOO5suHxdjDjVt2LRJdiWYsYGsCSfIUK8Gu3LShe7V1S47ofEp8bZhmgwY0/GleBNFEnqfuFOYbC3LzsucAKqMCNQRczxGn5n21t6OdyySeI3T9W2P482py3xG5zRT6jHwqh7TcLuWLJurfbQqIhY8RjpNDGG4tiPywfWv4U8cbkrQrlXZoPGOJI+HvJkYM1t1AyA6lSAJkn7BQnxbD4EcOtNbKjFeEMJPel4+lDqdgNYkDlG9Jw3aQyQ4kDc+rf76tEt2L8XAqM0R4gG0I89+omg7iMpCcPhsB88wPzMglic4Uk6d2QM86q5JIrQ86WmW0oIJBYASFABUGSNAwzDTegrsRwIJjXuELlyu1uJ+jJdQJkaMFYga7TRi+OOYgJsxEhhqQSM0ctp3qrJKMe2WwhKfyoea4/Jh9h+IpDXLg3Ue4fcaYfFt+Q/+0/FqbOLPMN+pP7IpPVh5GeHIv8AFkbGKzMIhWGu3I6ddqzriuDa7jnt65VIQEW5VEA8MhdAB7Na0bOmfORDEBcxUrpOiyRtJPvqps4Q2sZiGDGLqW33iPTUjTfUT7ashNU2mVyg06aM84pw02Ha2ZbL9aI9JQdtR0GhOxqNbRTlBJBJ9kaT7qO+30mxaO/iYczuB1PlQRavhGGb0YiQORB29tbMcn6drsy5Et/IQ/J/gLV65iFu21fKEyzyk3ASIPOB7qIcRwm1h7yvaQqVhgAXK5pYagI/LzFAfZ7tE+De4yW1cPA8RI9EkgiPX0oks9pbuLZP5qgVmW1mZwyhmYATmSQJYbA1Rm9Tll+NQ4Cjs/xy7exFy0+TKqZhAIaZUaknz5qD95HNDnAOB3bN9rr92AUK5UJPNTp4VA26CiOqUWMKYpu5vTtIca0jAItGRy3O3kefnS4ryilVCDZFeFLIr0VAEXGYRLqlLih1MEqdRoQR7iAaqFhceEBP9ATGcn6+8G5r68h/S5Vc43FJaQ3LjBUWJY7CSANvMiq432OMChjkNkmJ8M5iJjLv7aIUC3yuW/5vZMa97HsyNp99ZvaI7xOsjyO49Wm/269NL+VvTC2T0vj9h+tZkl7M6zvmnXTeDO3l+6ulpreL8mDO0sn4NnKmef8Au/AVD4f/AFq//hWf2sR5mpbOPL/b9wNUOL4mbOJuEKSXs2wsREq17eQunirA+jelzRCxXHXt33XQWldgdASWZmbn5dNtOtFeUEA9RNB/DMGrXjIVwvMiczkyzmec6DyAora5IrjylcmzvOFRil4BPi/HSXuWh4QrZcw/pGgwQsiFJOk601hcE5MJbFteYJJYnTxOTqzeZq+u8PtNc7woM8gzruNiRME7a+qn3U9aEZuLuI8oxlHa0QeH8MC3LZbxEkzO39G9OogXFXwBA7qx8b4+6pdo+O3+kf2HqIDOMv8A+DY/axFdDTNuDbOTrUlNJeCt7cJOCcfnJ0H1hWZo8CD7PX5Rz3rWO0+b5pdywTAkb6Zhm005TzrJXujUczy8662nScP9nKyv4hVtFIbxQYJ3Op08I8zJPsoq7PYEPhpUnMrmGj81DG8Rr99C1tUzrnJUEctTGsRR72DAOGbwx9IeW/gTXWkzqlX1GxOzuCxDh1Uk27nXYEGdZ5jSPsq/tXPE06eJjHPViRoNt6bSwpxFtWUFTZugjSCM+HPIRVGLV5cSVAYqGhWJnwqSCCfIVyNXupeDs6BR5XuErXwK6rmq3jLstpyk5gJHv1jziab7O403LYzGTAMneTMg+oisPJ0a4J946H1VExF09/MFfo41jlcP41KxWiOfzT8DUbiIPfrH9232OvP21u0nys5uu+aJTdtTNi0SdnPOeVANyTIP+XTfXYeyj3tx/VUJ1+k6/mnp99Z9euE6V2dPG4pnFzSptHsOUDobnoZvFvt/l8Xuou4XjMIpC27pA7xGRQXJZgUKyHSZzqBAOoA61Qdm+E/O7jWw4QqhecubZlWNx+VV9b7JPYu237wOEdX0RphGVvqhonzoZpQvmQcSlXCND4XiXcNnBBBG6Mn7W/sqbVRgu0SXb3chLgaCZIAGgnrP2Vb1js09BXUfEYpVJkrpvLARz1mpND2PH0ryF30Ma7Dc/hSMh7E9rMPafu7jPmkQVTMpDQVgqSDoRSf/ABvgQxVrxVgYIZGBB91VWAsAWlEzAoW41jbi3WUOwAbSDG2o21po7X7fv/4LK0aGva/An/7m37ZHxFPp2lwZ2xVn9cVk9viNwaBzqSdQjakyfTU0jjDF8M7NBK3LUNkRWAYXcwlFBIMDedqthjjKSj5EnJxi2bCOMYZtr9k/51/Gqo4m189Vle3kFkiQVygliYmd/Ksv4PjkW3bU3jZKXC9xQmcXkI9HTnHhg9Zrh4EjWhezhRc1VGWAAzOAoYNqQFk6dOtR4lCco5birpOu/wC0LHLvgpY+eOVfQafK3dU4S1DKfphsZ/s7nSsqzlekD7dft3qwucIBJi6vuPupF/htyCcykb7R7q6OGWGENu6zFlWac922g44P2ra/ad3RVyGDDORlylpMvOsECJ1jamMOC5a64hmMxyGkACeQECqnhHDLbBGFvLABZm3kjXzG8R+FWrY9BibFgCS7AR+SuviPmSIHt6Vz5Jbm49fwbouTSUuy34Xh2Qksj6gQQuYbeXuirHvY+pc/9N/+tWF92QLlQtLAGCBlB3bXcDpvUs4f84/Z+FYZ6eMnZ0YauUYqNdFC11J1ketWHxFN3cZa/LUesx8aIhhz1PurxsHr9n76T9LHyOtdLwDVi6jOih1YljoGBPoP0pqzh1TFXVEgCxZA9lzE9KvLd8MXAmUbKZBGsBvDPpCCNRQMO0yrxDELcDCWSxbgSPo3uCW1ESXn/wCNdODC4xaRl1Gf1JJvgue1KThLviy6AySRswMbjfb21jasVcGdQfXsD761TtJxrD91csuWLMkgKBp4iAc2aBDKZE8qzGzgXaWUMQu5CkwNeaggaa69K2YVeNP2tGSbqTX0EMRIkGev2mtD+T0E4d5P9p5fkr11rPsWTm9Jm82Mk6c6MewfFbKK1pmIuO8qI3heRC9F5mnzp7QYuwzvIxvW8hAburoBMESWs9PVS3QhmBj0jMbTMmPbVVe7R2ExNhM8tnNtgAZXvVXIToBGbINNp8qI8dgRq2d1neMsbaxmUmuZnxSnFUdHTZY45NsrJk1VXcYiXGykkI7KegD5Sf1WDD2n2217BHlcufqp/wBBQvfDC/dR5OcSDEZoAHviKz49K7+I05daqWwIsU023/Qb9k1B4njlTEWu8IQslxFzECWz2YCkkanpzp+0j3LWjqMywfASRIg/X3oV+Uq+M+HCkZ1zPHSWTLPtU+6tGkwyVpmfV54TacSf2xxSnDqocZs4OXMA2XK2sAyR9lAFy2TJ3+Pr9VSOIcTe8+ZyuaI0EDTWY9ZJnqTS2WNHBXbQqRI5mDE7Guri/wCOFPs5eRb5WiV2M4jbw+JdrzZFNtlmCdSyEDQE7A0YYztBhXKhbyMdRB01OWB4rbfd+Ge3rSEZgQWO410PM+f41M4imHHdG3lnMNFzAhQqT3s6F8+fUbj2VmyxUpe/Jfje1B3wjAXVxqObThMrAsV8IJU9FAGoHKjKluNaTVKLHyFXeDqPfQ7jrw71xOzfcPfQML1g8/hXTcs8m/j30zx2VKYR4W6BbFBnGroN59frn4GpxNvkx/j202cPZOpj3CpGFEc7KPvADTmLvA4W7r9e18L1XHzOx5fqiufNrI2A9w/GrYPbJS8CT+KLj5AcXJOnSi9ca4trbVvowICiWUaz101M1JPcrzA9wrzcRsJ09tXanP6zXFFOnweimk7ISWrjmNfZp7RvUyxwpV1uHUctz+C0zd42ToiiPWFH2TXsGGuN43UifRG0dDO/urNwjTyxjjHaFLf0dsAtyH1R5ufu+FM9m+Ek4zMrublm4Gus6QDClpSDrtlC6cuVDqJa7kuzxdEBbeQnMD6RzDRYE6HePOpGG45iEa2wuvNvVZ1XUZYIPpAr4deVXrE5OoSr2fHf95/kqeRRXxL+/wBo2bB4nNE3DqxSMoBDLBI0md6njCDr7wKx+320xQZWRbK5QQFW14RmILNBJOYwJM1PX5RccNxZP+mfuag9FkItVA1H5v5j3V7ufzh/HtrLx8pOM/IsfqN/+ynV+UrE/wB1Y9zj/nSvSZUP+ogH2PvG3EZT6yVHPQQDJ08gBuazi9wL5xfxF0XGQjEOBCjQqQwMlgQZP2Cl4rt+10Fb+HRlIGiOV1BnnmkHSQIqb2TvNft3bzek99mMKpAlbeknXaKXZPGrDujMg4vss91s1zFMzRElQTAkgaP1Jp3h/Zq7aDrbvoQ4AOe2xOgZdCr9HbkaJMp8/cPupVtoPL2/upXkclT6+wyik7QFYjsLdEsL1qcrNBD6hdTqVgEwY9sbVU8M4Df+bHHpcVAmYqIljlOUnoOY1/fWn3cSSjCE9FhMMTqDOprOcPwvFLw03Rfiw3iNkayMwQknlJ5DynytWWTXIFBIaPCr6nD468VyvdtMTIDAFlyswACqIA++tVuYqzfCPlJK+JCQRGYETvsQayS8+MuWLVhiO6UqVAZVuajMgOYzIGwI9hons9qLaAXmt3ZKrmQZJWS2V8pfQMdPYBVWR3SteOB15SDK4/l9h/GhrtCT3lk5dJMtqN9I166H2UyPlBw+3dYj3W+e316g8Q7WWMTltoLoYsCM6qBoZ3Vjymk2tEsveGlQoZmK93nXc5SCQfEBoSNIJ216mqPtYcORfzZO+KWzbmM8SJynpAb7a9jOOWrBNt8xYuHACyCJEiZ0PhJqDxRUxpUWroIRoZnDKYukBF9HxHwtqYEnzp8fDt9Cy5VAxxDhRt5Mzg94gfSdA+2vPTmKnXcY19yblwtctoxBCDJ9GC55gyfFMrufVSeL8JawAe8DoGCqQWzQwZk8LDwei2gOhmuW2CtctsHNxku+NtDAtOSCHUuJCkelzq91SaYiu2mdxGGuKoZrRCn60R8OpO1U7QSdY1GnQa6/AR50ZYm8r4csQSpUGFADQrLJBboTtppO80Gu4zNBYDMCB5DNqfzhO3mdRzSTuD+40FUg6+S7Es2IuBnYjuiYJJ2dNdfXWlVlnyWP/O3EnW0/7dutUrOXMDjw7DHkn6i/hTb8Kw35Nv8AUH4VqX8j4f8AuLX/AKa/hVHjOD2M7/QoADyEDbXQbUzmkUqIDHg2H6J+rXP5DsH8j7RV1wjCWmsozKCSNSfVQ3xYMt65lJVA0DTQCOsVIzTI4ND79nrB2K/rMPvpp+zNoggEwY2uGdJiJOm9Vl3GMP7Ue3LT6Y89xcuBgWR7YHMQ5YGQD+bTrsDTo9d7Gj6rXB7m+6njwgqBCKSOsj7jScDxF8qvcuJbztkSQ3ibntOVZIEmpjcWuoSrRoY9KdtNdKaU7bi3dCrhKXkifN3H1Lfsn71FeTGsnK2I3mdPXtFWNrjSNo6r7wD9hp97dq4IPuakobcDeCfDp6LWVB3hwfsZjXMFwm3iMYtpWItsSx0hsiobjZQYOsGPWKsP/C9tGD21BUGSjSyHTbfQTHXpVPw63ctY+zmQ2w18BSD4YZ9ACDGxiOlacUlG3fNFOSLnS+oaYDsngiw0unxG2ZcSrBc26qPMERIIq6XsBgpBi7p+fp9oqzIt2QGZlSWCgnKozOYAGkZiYHU6VO8fX4fhWZ58j/yf5Llhh4QLv8nWD63v1x/0ppvk5wf5V/8AXX/pRZ4+vwpDl+vw/Co8+T/sN6cPAA8V7B4W34jdvBI11Qmes5QABoI1ksBpzd7I8OFlL1pg1zLimt5lA2yWSCZB18W3kaKjlusyFgxQgMvhMH0lkRoeYrKO1F28mOxCWXurNySLbMJJVZJCnc9aeDll4bFdQ6RobpBIg79FpKn1/wC0UAdoDiMMVyYi86nNJaDBUwJIJgHQ66weugq7Xaa+maXZyRCMYBQ+ITEQZB2M6hSCIpVBSjuT4GbqW01EvKtryP1j0PQVn3BOD37+AuEX8tpSzC1EyyKGaSD4R5czr64ScdxJ0+cXAT56a6ax+FRsCuIEWbbsi3XCFZIQliEGaBtqNRrVyxNRfKEWRN9E8LjLVi3iHCi2xTKzKrPAEW2YRO20nXTyq9wfZ97tsJcvKpIEnu810qDmRbjZhMEz69JMGhvjfEsTbRcJcuIyWsrAoPSGUMgLEAkAEch7a1e1g0twxIn0FnpqVUdY1gb+veqZ9J8eSzm2gNvfJ9I0xPvsztP/AJmtVz9mzh71qboeTGlsqeepJYz/APFaTdugDY/Z+NB2Mvd5fZ+SLHkGPT7DSb2+yUkVfF+ANeY3w+k5FTKWJOaBzgat7ImmsKf5NJa45dbrKQ1oAgNbnMrZyNw41HKaLbVn6NLZOUnbqW9MxruD8Kre1HCFuLdzTNuwbyhdAbmV18XURaQcufWjCV/DLojTXKBntLx23ft92hd8jKQxRUQhFuLKwc31xuBsarrBCXGtZSCy3DmMKw+ifSPFocpG/M01xW3YXJ3Du0rLZgBB0hfCo1HPf1nWn8C0XO6UQrhpOcyPA43tvl5bEGdjyrRJKONUVptzdkvDG+cMI7vu+7uaENmyh0zTGkyRHlmmqrjOHFq66pmUeEgScuxgnzmY9ZqywVi62HkXoTu70JkU+EMhcAkz4jBnlGm9Qe0d0C8ZLCVQ6HQwefn0pMnT+40O19i9+S9z889dp/fKGfsrWYrIvk0f+fKJOqv+zOvnpWvVnLWFNQ8Xg1Y+imu5jxesEHSpdcagxEC+I7Kd4xZbptCMoRUWBAjMkyZO8+umV7DnUnFOSeZtoSOhXTQ+dFwrooWEELXYcrOXGXhJJ0C7nc7bzScV2D7wBbmMvOsg5SFKyNpEQY1oyrxp1NroDin2BWH7AJbGVb7EBs65raNkeIzJI0NU+K7CJ36WjibxLqWzSsAiZ0mTNaJjbLOjKrm2xEB1ALL5jNpPrqovKwxlkM+Y923KOvIGpGcotyXbA4JpJ9Iz7tj2MGDw/fLiLj+ILDaDUEzofKgwZ0bwsQRuBM8vYa175VP6j/qp/wAqyW9YgyCD1PI6Tvy99dPT5HLH8TOfqIbZ/CFnB7GLykvbZdssqYcEE6ESBsNTA13qwtXQTDAo4IMMI1GoMHz1FXa2xkXQeivJeg8vvNV2Js22vAXB4RYuEZPSBD2YIyjU6nSDMxHXnOXLZvUfYsMPxEejdAgagkSPDqCPMb1Y2+N22Ei6CP0f3UKMxRu7fxLsGjfQH3wRpSrrlB3gbTmeR9f53nz59aqy2o3HkvwKMp7Z8BYeKL+Wf1T/ANaSeIofrt+of+tVmCvZwDPKpY2rB+sl4Oi9BFe47bxNrNpozEAkoQSdhJKjXkKplwIs8Re4pP09ksd9GttaVojkQUPrBqZd+r+mn/uLUfFX82LtGI+hviDHK5h9a16fM5pmLU4FikkhztCrvhb6iZNto9KJjz099Yw2EEnMSI/d++tl4vBw94EgDu3k6aDKdd6yLFYY6EDw7TBAJ12keR9oNdDAk4S4swZHU1zQprI0mcwBjXXw6n40UdiGR+8zW0YqVysygndtiV32M+dCV11GmsnbTkd/fFFfYJiDdnbTQTvJnYabj3U+f5WhcPaLzj3A7WJFsMqhmuqpdQM+SGLLmidQDz31ohxeMtr4XBJ0aBbZ43g+FSAdD51AvP8A0RE/0q9eYYc/XTOPxBmWGUlVkH1uPurm5sjhC0dHT4lknTG+McbthYUuWOwKMs+1lAqowVgZlR5Oue4QpYk7xCg+r2mo9pgztdbYEhemkbddxPsq64ZZgEn0m1Pl5VVLPtxp+7LY6XdlceaRYJctSGOdiNRNtzB6jwaHWhv5RMIr2reJUmVIUcpV5YHXUEEfbRAxAqt49hGv4dbJIRS6AXN9ZYHSR8as0mocp8+wus0sYQ49zKLjGRFXPC2PeqiLNtozlfHEggyygZfUaMePdmVbDibhLWEMHLuFVREAzrloH+d2vm7Wu5m7nzC6TsvhgD2ZtPMHkK6F+pG1fZgra6I/DrzFkt52CuQsBmAh2AbY8+dGPEOxy3DJvywBGbTLC5iIgA9Br1oKw1pnuoqQGZlCnUAMWGX1aneii7wTHqJe+mX9M8zH5HnQzXu7omNcGgdkcKgweFYIgfuLcsFGYnIATMSZq7oN4b2lTC2bFl7bMVVUJQqRIgaSRp7qMorM3zwXU/cJqQ51pdIuUrEE2F3kEeI+3Xca6CnaStdqBFTXq4DXCahDxqnxjD55ZGVZKNB1zDfQeLL71NWONw4uI1tiwDCCVYq0eRGo9lU963lxlkST4H9JpPPzHToaJCr+VIfzE/4iffWTMoyrrr7+R0j361q3ypn+Z76Z009p1rK7txDEAgyIjfodY9WmvL1V0tJfp8eTn6r5+fBq9txlUfmjmOg/PqPhADjEnX6C55/2lj84/wAfY7bPhXf0R16DzrmEMYxJ/wDx7u/+JY61z2b0cxNlWe8rCQXGn+mm3Sqm/wAHuBSVYXFH9nszL659IdPtBir25bHeXT1YfsJTT+VcrLkkpSS6O5gxxcISrlIoOGXu5ZRPgbYOYYHmu0TvExMcqJHeB5VEfhtt2zsstv7YifXGlKxNZzU6bIX8rq1wpA8LpG8nxqZ+Gn74k8UuD53YP/lXh/usH7qrsXw2fGkZpEq2qtygzty+FRsEHd80kNazIqNqZbLnAaTmAygCdd9TFbtG+0c7XwXEk/8ARd49x3N2RI7t5A5jKZG1ZHiOYCCeskkxvOsRvsOQ851L54Wt3F2fI2h65THPrWcWcCLqs5fKVHolQOU666e7zrt6WSjGVnBzpuSoZt2bbANcLgrGqgEb6zrqY22q77EMM9zrHTz8iOtDd60GGUSADpJHqoi7G24vXEYCQpB0B1DKNzoeexqZ3ww4vYM2Olv/ABrXI83A6+dU/brFFbiqGyygk89Gfb2GatroAVIH9tZ2A/vrfQ0xxR7LkvdSdFCodywa57htrXMzJyjSOnpZqGTc/qVuEZVtrcuABVXwg8huW1+sx1Jp7gOPuEhbkEtLQOS6Fduvnyiod3C5mUtqxOiH0FnYkc4/J9pmKuuH4VUkgSx9Jjqx9Z+6udlg4OmdfFmjljuiSWTUmk4/+gtnpfQcud4rz0515pmo3Errdwoy+HvkObTf52BGumxq7R/MzLrvkX3JfE0mxeBEzafTQz4T0191Y/eTmBFa7fE27o623/ZPSsnvMZAiQd9J5xM/bXa08tqZw8sbaG+EvGIskmALtszy0da0/iWMt5G+lQHTZ1B3H5w+NZdYRc/iGZdfR1mOe40q+wnCrVxUZVUZ/RkgEmCY9LeAT7DU1ELdsOOVItsdhrtxUa2l24uYGVUupAO4KlgdRWnmhvstjFtWrWHyNIJEjLl8TMw+tPPpRJWVKuC9yvkJKbub05TdzeoysTYEA6R4ifXJOvtpykA0qaATs14VwV6ahDzH+NqqcVcPzuyNIKNPhBPP62UkeoEVY4vDJdRrdxQyMIZTsR0NU96wqYyyFEDu3MTPUc56dRUIU3ypf1T/ADL8YrLLmgCkbETO2x384/gxWs/Kgf5g3+InxNY8LpzHXaDrtsTz9tdHStenz5MOoTcuPBsNtfCun1RyHQfmUyloNi0U88Nd2/xMP0ioHA+PLiAwW2wKZBrBnMG18M5fRO+mo1qo4f2xDY6yxtFVhrJ8WYnvXSG2AEFF9hPlWWOOUnKK7XZq3xST89BjctwzqORH7C0y0dRVpjltLL3AgAgZmA0kwBJ8zS1wg5Io9gH3Vzp6XdJuzp49Zsgo0VAuKPrL7xUZrisfSHvFEfcHoBXCh60P0a8jfr3/ANQaZgdBJOkQCdZ8qre2V9MK1u4AR3jMGjUaAEGPwot72bjW8raKGzR4DmJEAz6QjUeYoK7TcMbG402GuG2lmyHWFBkuwDTLDy91a9NhjB/Qx6nO8tOjtnHW766tqRAdd4iCG6/GqDGdmbqBjZYMCAAeYg8yN5E7ip1jsUbRlcUR/prr6/pYq0Th91B4bgY+a5ftV2rTu2/KzK432AGO4PfiBbk8yCNTrPOnOCWsbYuFktCCMviYRqVM6GZ0o8uW8QFLFA0clMnUxsVFVWK48lq69m6y2nTfMNJIDQpUGWgjSpKTkSKrolNdvPABAQFW1XxZkcOusgASo0jrSkBZoEvc68h+H8RQ7e7U23suSzre2RIkaxDFtgBJ039dHXZ6zmwdllOV3tI2fKD4mUEsRsaqa2os5ZHs8CuGDKg7mST/AMdPfU9cC45r9v4VONqktaHlWeWCEnbNMNVkhHbHoj/M2/KX9Un76rO0jraw6IzAlr9oLpEscQtwwCeQBPsq1u21IjPl81IBEGfu18poU+UjEqLOHdSDlxAbTY5VYx05U+PDGL4FyZ5zXxMIO5hX3Mq2/qNZBcuRmA59deXlR+O1S9wt7IIclcuYBh6SzznagDEKCfyfIiPcOlbtKuGYs/sNLeMyDEyNPPQipOHxt1VCredVHIGI/iagFoM1ruJsKFaFUaHZR08gfgabPkUOKsOOLlzY32Jv4cYW2957fe5n8dxlz6O2XU67RRiKy7j1v6JtvtB6c0Fabwv+htR/dr+yKw7t3Jo21wFFM3dxT1MXt6jEQm0sA6RqT65JM+3el0gGuzQCOCvCkg12gQRiLyopd2CqoksxgAdSTtVXfxBOLshXOQo0qGOUnXcAwT76tbqAghgCDuCJB9YNUmJVExdogKqC20kQqgmdyBE7bn2UUQrvlQ/qDf4iffWPrhnYZlXTxa8hlAJmNtCBWr/KXxC02CKrcRmLqcqsCYEyYHIVlmHLCAWKKJPWcwAIInpXQwwn6FxXNmPJKHq1J+w7grNzKzNmymVEH25dN9OR8qb4U6i/a19G4h1GmjqSfsp17rFdShG+mjaDLqdzttNK7O9yuLsm6QUz+KfQGhy5pG2bLPKK0ZIzxw4+tlGNwyT5+hsdriVq9mykMBGukanSOvlUzXrQrjOOJZKjEPPilVQhmKhTLHUBVLERrGgjyQflCsfVs3jrA8KgSdhJauWscn0jo74hYVPX7aSU/jWhnD9tDcByYeNQo7y6qZmOyLCmW8qqn+UO54h82VYIGt2TJMGQADprtPsqKDbpdhcklYXcRvsgGQAkn62igcydfMeuhOzj44lea54D82RTroTnBkEHUERVdc7dF8wv2gFkqDaJzgjTN4mHUxBGusmKb4BifnmLvPBRVsgKB4myIyKAdRJ1JNWxxtJt+BN6ugtbilvqT7z9xph+JW/yT+qf+lV3EMEEjxNqoPiCgiROvgPxqv7scoP6h+BFVUOX649Pqgg+0f8AEUK28bhLeOxhxShi6pkzIbmmQZ1G5DHw/uq2wimdj7j9z1BtcZs4XHYo3lJLpbCkDM2iAldToGkeWlFJu6Cmvcp+CcZwlrCX0a34nZiEInMrABFzbDL+/Wr/ALJju8NZd2Zs9sASwyKqKQBA00CTJky2vOqDhPD7N/OzWwCbhOUhgFVgYyFILjkRGmmmtSL/AAy8pW3Zt4jumWbiIlw2s0QAskE6ATJgn1VY4JycaEt0nYZ3OLYdfSv2h67i/jUe52hwg3xFr2MD8KCsN2UxZVf5tdzCZOVQCTGWc3tq6xPZfFPZW2LD/VGVntqqZAQzIcxMvMnTrSyhtaS/kMXdlnf7UYQad8J8lY8p5L0of7SCzcweZQGLEOjKJbMSoYa6kwXkcgPKu3OwONdpy2lEzBuTGkbqutW3Z/sTisNdF4NZzQQRnfYiNwtHbFc2C2+DNsPgLm4tuQNzlOkamYHSpjcNaGJPohSQxloeADoIjxKdT19Va0vAsV9NN6yBdMnwXGI+jW2de8XNooOoqqHyeT6eJJ0AMWgJChVE+LXRQKeGRIEotmU4i3pP79PPl1omw3aTF3lb+hA9GMhljlZiqySJyqd9NqLx8meH+tfvH1ZB/wATVhguxGGtBlVr0NowzjXQjkogwTqOtDNOMlx2NjTXZV9jOH28XhmfEIHbvCunhGUBGUeCBzoysWQiqi6KoCgSdABAGppnhfC7WGt93aXKsltySSdySdeQqXVDq+Czn3CKaavLuddBsN9OnnT0VygIZzjONcTdycPhnt2zqA1oFpOpJJ01MmmZ44/5a+y0taWa5WlahJcQRU8bf+TMj43c4ph1DYi9dVWkAq4IzRIU5IiYpjhT3rls3Lr4m6TcFvIl1xlkAl2g+fqrVuLcLtYm33d1cyyDoSCCNiCNjUDC9ksLbkqjid/pX19cNrTZM6ni2r4ZfRf1iwxuM7fKM8w+IW2WtjE5TbvkuXBdrlsbKN/MEddaortm2xY+BQWJE7hS5MadAdvXFbJa7L4NdsNa9qz8alW+E4ddrFof6a/hQx5MeOTkrt99EyQnNJOqRgDsW2I9mvu12r1rhl5j4bd1v0bTGvohbSjZQPUAPhS5oy1MW7p/kkcLXF/sYLhuz2KIgYS8fMow0gRuB50/Y7E49iD82YRPpMg39bVuJrlGetlJVX8ghpoxd2ZFgOwWMVlbubaxMzdgtI55CY9lTLfyeYoplZrAEzMsx2jXw7eVajXDVb1MvZIf0V5YCcN7D3LYhrllhmzqDbY5HAjMPGJ5aHpSf/p1mBD4tzm9KEAkzPMnnR5XCKr9WW5yXb+g+xUl4AZPk0w31r14+1B8Fqy4b2Kw1hi1trwYiCRdIJEgwcscwKJaTReab7ZFCK6RU3Oz9pt2un13WJ95Mn30gdnMPzRj67jn/lVyaTVdsYrU4Hh12tL7ZPxNS7VhV0VQPUKeNJoBPTSaVXDQCJr1drlQhw1ylGk0SHK4a7SaIRJpJpZpBoAYk0ilGk1CH//Z"}}},{"metadata":{},"cell_type":"markdown","source":"\n

3. Setting Up TPU Environment

\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"First We have to define the distribution of strategy for Tensorflow . So that TPU can use this and distribute Jobs accordingly. The below code does exactly the same"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect TPU, return appropriate distribution strategy\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see 8 replicas are return which means that there are 8 cores present Each core is kind of GPU.**NOTE: If Your Notebook Outputs 1 it means you have not Enabled TPU , Enabling TPU is just like Enabling GPU just go to Accelerator Tab and select TPU v3-8**"},{"metadata":{},"cell_type":"markdown","source":"When using TPU Datasets needs to be stored on Google Cloud Storage bucket , as TPUs are available on Google Cloud Platform . We need to specify the GCS path to load Data ."},{"metadata":{},"cell_type":"markdown","source":"\n

4. Loading Competition Data

\n"},{"metadata":{},"cell_type":"markdown","source":"### 4.1 Loading Data from Tfrecords Format "},{"metadata":{},"cell_type":"markdown","source":"As Pointed Out Earlier in Some Discussions There was problem in Tf Records Format . So I created A New Dataset in Tfrecords Format , I created a 10 folds Dataset using this [Script](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256) , Stratified based on targets , For verifying Exact Stratification across various Folds , Refer this [Notebook](https://www.kaggle.com/sayedathar11/cassava-leaf-disease-stratified-tfrecords-256x256)"},{"metadata":{},"cell_type":"markdown","source":"When we are using TPU's , Dataset are often serialized into Tfrecords Format , You can know more about Tfrecords [here](https://www.youtube.com/watch?v=KgjaC9VeOi8&list=PLqFaTIg4myu-1c3ygYzakW8-hNzQG59-5) , \n**In a Nutshell Tfrecords allows distributing various jobs accross cores and helps parallelizing tasks and speeding up Data Loading and Augmentation process**"},{"metadata":{},"cell_type":"markdown","source":"**Note : That I have created Image of size 256 * 256 if you want to experiment with different Image sizes , just use the above script which was use for 10 Fold Cv Generation and set Size Parameter there to size you want**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH_ORI = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_NEW = KaggleDatasets().get_gcs_path('tfrecords10foldcv')\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [256, 256]\nCLASSES = ['0', '1', '2', '3', '4']\nEPOCHS = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A Notable Parameter is Batch_Size , Note that Batch size is set to 16 * Number of Replicas , In Case of our TPU Number of Replicas are 8. So Batch Size becomes 128."},{"metadata":{},"cell_type":"markdown","source":"Now we need to Assemble Dataset From Tfrecords so that is in the form that we can feed to our Model ! The Below Code Does the same ! \nPS : **I have hidden the code , you can just read Explanantion of Each Functions Otherwise You Can Just Go through Explanation of Each Functions given below.**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"## Code Credits : https://www.kaggle.com/wuliaokaola/getting-started-tpus-new-tfrecords\ndef decode_image(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image\n\ndef read_tfrecord(example, labeled):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } if labeled else {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"image_name\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n if labeled:\n label = tf.cast(example['target'], tf.int32)\n return image, label\n idnum = example['image_name']\n return image, idnum\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False # disable order, increase speed\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n return dataset\n\n\ndef data_augment(image, label):\n # Thanks to the dataset.prefetch(AUTO) statement in the following function this happens essentially for free on TPU. \n # Data pipeline code is executed on the \"CPU\" part of the TPU while the TPU itself is computing gradients.\n image = tf.image.random_flip_left_right(image)\n return image, label\n\ndef get_training_dataset(dataset, do_aug = True): # trainingfiles changed to dataset\n #dataset = load_dataset(training_files, labeled=True) \n dataset = dataset.map(data_augment, num_parallel_calls=AUTOTUNE) \n if do_aug: dataset = dataset.map(transform ,num_parallel_calls=AUTOTUNE)\n dataset = dataset.repeat()\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\ndef get_validation_dataset(validation_files , ordered=False):\n dataset = load_dataset(validation_files, labeled=True, ordered=ordered) \n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache()\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\n\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH_ORI + '/test_tfrecords/ld_test*.tfrec')\n\ndef get_test_dataset(ordered=False):\n dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTOTUNE)\n return dataset\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 Explanation of Each Functions:\n1. `decode_image` :Takes Image which is in Tfrecords format and decodes it into a Tensor , It also normalizes the tensor by scaling values between 0-1 across all 3 Channels . Note that this function is insantiated by read_tfrecord Function which is explained next.\n\n2. `read_tfrecord`: It takes Tfrecord Example and it returns (image , label) pair if we have training data and (image , id) pair , if we have test data.\n\n3. `load_dataset`: It is used Load Training or Test Dataset appropriately . If it loads training data we also do some image Augmentations on our Training Data so that we have more samples of images .\n\n4. `get_training_dataset`: It is used to fetch training dataset by giving appropriate path of Tfrecords in the directory ! Note that this function insantiates `load_dataset` function which uses it to load training dataset.\n\n5. `get_validation_dataset`: In our Case it loads Each Validation Folds appropriately using `load_dataset` utility.\n\n6. `get_test_dataset`: It is used to get test_dataset.\n\n7. `count_data_items` : Usually in each Tfrecords Files we have multiple items for example if we have a Tf record file named `Id_train00-2140.tfrec` , Then this file has 2140 Data items . This code counts all such data items in all tfrecords either train or validation and return their sum."},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Conclusion For Data Loaders :\n\nAs a concluding point for Data Loaders Functions You can learn more about them in Tfrecords Basic [Kernel](https://www.kaggle.com/ryanholbrook/tfrecords-basics) and getting started kernel [here](https://www.kaggle.com/ryanholbrook/create-your-first-submission) . **I would Emphasize to go over this kernels also so that you can get familiar with workflow and connect dots between these 2 , This will help in future to create DataLoading Pipeline From Scratch**"},{"metadata":{},"cell_type":"markdown","source":"\n

5. Defining Augmentations

\n"},{"metadata":{},"cell_type":"markdown","source":"### 5.1 Code for Augmentations"},{"metadata":{},"cell_type":"markdown","source":"We will use code from Chirs Deotte's [Kernel](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96) for Data Augmentation . As Pointed out in this Kernel using `tensorflow.data.Dataset` for Data Augmentations , does Data Augmentation on TPU / GPU which offers Speed up as compared to traditional method of Doing Data Augmentations on a CPU."},{"metadata":{},"cell_type":"markdown","source":"We will have following augmentations.\n1. Rotate by 20 degrees\n2. Shear by 8\n3. zoom by some factor \n4. height and width shift by 16"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n # returns 3x3 transformmatrix which transforms indicies\n \n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n shear = math.pi * shear / 180.\n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n # SHEAR MATRIX\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] ) \n \n # ZOOM MATRIX\n zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n \n # SHIFT MATRIX\n shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n \n return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform(image,label):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated, sheared, zoomed, and shifted\n DIM = IMAGE_SIZE[0]\n XDIM = DIM%2 #fix for size 331\n \n rot = 20. * tf.random.normal([1],dtype='float32')\n shr = 8. * tf.random.normal([1],dtype='float32') \n h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n h_shift = 16. * tf.random.normal([1],dtype='float32') \n w_shift = 16. * tf.random.normal([1],dtype='float32') \n \n # GET TRANSFORMATION MATRIX\n m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image,tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2 Displaying Some Augmentations Example"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Displaying Image with Augmentations\")\nrow = 3; col = 4;\nfoldno = list(np.arange(10))\ntraining_example = tf.io.gfile.glob(GCS_PATH_NEW + '/Id_train00-2140.tfrec')\n\nall_elements = get_training_dataset(load_dataset(training_example),do_aug=False).unbatch()\none_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\naugmented_element = one_element.repeat().map(transform).batch(row*col)\n\nfor (img,label) in augmented_element:\n plt.figure(figsize=(15,int(15*row/col)))\n for j in range(row*col):\n plt.subplot(row,col,j+1)\n plt.axis('off')\n plt.imshow(img[j,])\n plt.show()\n break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Displaying Image with Augmentations\")\n\nrow = 3; col = 4;\nfoldno = list(np.arange(10))\ntraining_example = tf.io.gfile.glob(GCS_PATH_NEW + '/Id_train01-2140.tfrec')\n\nall_elements = get_training_dataset(load_dataset(training_example),do_aug=False).unbatch()\none_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\naugmented_element = one_element.repeat().map(transform).batch(row*col)\n\nfor (img,label) in augmented_element:\n plt.figure(figsize=(15,int(15*row/col)))\n for j in range(row*col):\n plt.subplot(row,col,j+1)\n plt.axis('off')\n plt.imshow(img[j,])\n plt.show()\n break","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Displaying Image with Augmentations\")\n\nrow = 3; col = 4;\nfoldno = list(np.arange(10))\ntraining_example = tf.io.gfile.glob(GCS_PATH_NEW + '/Id_train09-2139.tfrec')\n\nall_elements = get_training_dataset(load_dataset(training_example),do_aug=False).unbatch()\none_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\naugmented_element = one_element.repeat().map(transform).batch(row*col)\n\nfor (img,label) in augmented_element:\n plt.figure(figsize=(15,int(15*row/col)))\n for j in range(row*col):\n plt.subplot(row,col,j+1)\n plt.axis('off')\n plt.imshow(img[j,])\n plt.show()\n break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

6. Modelling

\n"},{"metadata":{},"cell_type":"markdown","source":"### 6.1 Defining Model Architecture "},{"metadata":{},"cell_type":"markdown","source":"1. For Modelling we Will Use **EfficientNetB0** , You can learn more about EfficientNet [Here](https://www.youtube.com/watch?v=4U2WO8ObGGU&t=379s) .\n\n2. We will use Transfer Learning to Fine tune Layers of EfficientNet !\n\n3. Another thing can be using Various Efficient Architectures like B1, B2 .. ,B6 and selecting what suits the problem best.\n\n4. Also Since we have Power of TPU one can Also Train EfficientNet from Scratch !\n"},{"metadata":{},"cell_type":"markdown","source":"**Note : Since we are using TPU we need to define our model inside a context manager which is `strategy.scope` , What this context manager does is effectively it tells the TPU how to distribute Training on different Cores**"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef build_model():\n with strategy.scope(): \n #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.efn.EfficientNetB0.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n\n base_model = efn.EfficientNetB0(weights='imagenet', include_top=False)\n base_model.trainable = True\n\n model = tf.keras.Sequential([\n base_model,\n tf.keras.layers.GlobalAveragePooling2D(),\n tf.keras.layers.Dense(len(CLASSES), activation='softmax') \n ])\n opt = tf.keras.optimizers.Adam(lr=0.00001)\n model.compile(\n optimizer=opt,\n loss='sparse_categorical_crossentropy', \n metrics=['sparse_categorical_accuracy'])\n return model\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.2 Defining Learning Rate Schedule "},{"metadata":{},"cell_type":"markdown","source":"The core idea is since we are using Transfer Learning , if we use very large learning rate initially , That will cause drastic updates in the weights of pretrained Efficient Net ! We want to utilize our pretrained model hence it is better to start with smaller Learning Rate like 1e-5 and gradually move towards 1e-3 , Refer This [video](https://www.youtube.com/watch?v=DEuvGh4ZwaY&t=2706s) for more on this !"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code Credits : https://www.kaggle.com/cdeotte/how-to-compete-with-gpus-workshop\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n return lr\n \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(25 if EPOCHS<25 else EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.3 Training and Validating Model :"},{"metadata":{},"cell_type":"markdown","source":"For the above model an epoch takes **less than 30s** and within **5 Epochs** we are able to reach an accuracy of **~84 % on Validation Data** ! This is absolutely good as we train the entire model from scratch! Just unhide the Output Cell of the Model if you want to see the results for each Folds ."},{"metadata":{"trusted":true},"cell_type":"code","source":"FOLDS = 10\nfoldno = np.arange(10)\nmodels = []\nhistories = []\nfor fold in foldno:\n training_files = tf.io.gfile.glob([GCS_PATH_NEW + '/Id_train%.2i*.tfrec'%x for x in foldno if x != fold])\n NUM_TRAINING_IMAGES = count_data_items(training_files)\n\n validation_files = tf.io.gfile.glob([GCS_PATH_NEW + '/Id_train%.2i*.tfrec'%x for x in foldno if x == fold])\n train_dataset = load_dataset(training_files) #get_training_dataset(training_files)\n valid_dataset = get_validation_dataset(validation_files) #get_validation_dataset(validation_files)\n NUM_VALIDATION_IMAGES = count_data_items(validation_files)\n STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n VALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE\n model = build_model()\n history = model.fit(get_training_dataset(train_dataset), \n steps_per_epoch=STEPS_PER_EPOCH, \n epochs=EPOCHS,\n validation_data=valid_dataset,\n validation_steps=VALID_STEPS , \n callbacks = [lr_callback])\n \n models.append(model)\n path = f'Cassava_Model_EFFNETB0:_Fold:{fold}.hdf5'\n model.save(path)\n histories.append(history)\n \n print(\" \")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.4 Conclusions:\n\n1. With just 5 Epochs we reach pretty good accuracy of 84-85 % .\n2. Using TPU Speeds up things like crazy for Example Training Such a Model without FineTuning the EfficientNet layer could take minutes for each epochs , **Using TPU we completed Each Epochs within 30 seconds**.\n3. Use of TPUs Make K-Fold Cross Validation very much feasible however we must prepare Data in Tfrecords Format before Hand .\n4. Augmentations become much more faster while using TPUs."},{"metadata":{},"cell_type":"markdown","source":"\n

7.Some Pointers on Improving Models Performance

\n"},{"metadata":{},"cell_type":"markdown","source":"Even though EfficientNet B0 used here performs quite well , it can be improved tremendously well by following some of the listed tips.\n"},{"metadata":{},"cell_type":"markdown","source":"1. Trying out other variations of Efficient Net like B1 , B2 , .. B6.\n2. Trying out some more technqiues of Fine Tuning Like Feature Extraction , Freezing Initial Layers and Tuning Last Few Layers.\n3. Adding Dropouts and Regularization like L1 or L2 Regularization.\n4. Trying out different Learning Rate Schedules and adding some more callbacks like Early Stopping.\n5. Experimenting with some more augmentations other than those define here ! \n6. Ensembling the predictions of various EfficientNet models and Also various CNNs **[This should be done at the end of competition after trying out single best models]**"},{"metadata":{},"cell_type":"markdown","source":"### Teaming UP :\n**If You have Good Experience in Computer Vision and are looking for Team Member to Ace this competition I am available 😊 , I would be Glad to Learn From You and also teach you some of the techniques I have learnt**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}