{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this kernel\n* Use gradient accumulation with TPU\n\n    * With EfficientNetB7, the effective batch size can be up to 2048 images.\n    * With smaller model like Xception, the effective batch size can be even higher.\n    * For most NLP tasks, the effective batch size can even be 16384 or higher (because each input example is just a sequence of token ids, usually having length <= 512)\n    * If you ever use my kernel [Tutorial -- Oversample](https://www.kaggle.com/yihdarshieh/tutorial-oversample), it should benefit from larger batch size, because larger batches are more likely to contain training examples of every labels.\n    * It's kind tricky to implement gradient accumulation. For those who want to know the details, check the documentation for the method `train_step_1_update()`.\n    * To understand the configuration variables related to batch, see the comments inside the method `set_batch_configuration()`.\n\n\n* It uses the optimization from Martin Görner's kernel [Custom Training Loop with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu).\n\n* We also try a (linear) scaled learning rate at the end (larger batch size --> larger learning rate). However, you have to find the best learning rate by yourself. Since the larger batch size implies that the parameter updates in 1 epoch are fewer. Even with a larger larning rate, it might be necessary to increase epochs a bit. \n\n* No submission file is generated, and no confusion matrix is plotted. The purpose of this kernel is to show how to do gradient accumulation with TPU. If you find this helpful, you should make you own kernel by modifying this kernel or copy some code from it.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Idea\n\n\nEach time, TPU receives a set of traning examples of size \n\n        BATCH_SIZE_PER_REPLICA * BATCHES_PER_UPDATE * strategy.num_replicas_in_sync\n\n\nThese examples are distributed to all replicas, so each replica receives training examples of size\n\n        BATCH_SIZE_PER_REPLICA * BATCHES_PER_UPDATE\n    \nEach replica processes this set of example in `BATCHES_PER_UPDATE` times, i.e. it computes the loss and gradients on subsets of size `BATCH_SIZE_PER_REPLICA` and accumulate the gradients.\n\nThe final accumulated gradients on each replica are synchronized accros replicas by summing them (this is done by TPU automatically) and the optimizer use it to update the model's parameters.\n\n![Capture.PNG](attachment:Capture.PNG)\n\n","attachments":{"Capture.PNG":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAApAAAAHACAYAAAAV01g6AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAFutSURBVHhe7d0HnFNlvodxpiTT6R1pAgLSQXqVIiAiYgEUVBAFC9JEKdI7MgMKqChVyjDo6u7qrrp6regqoCIqfRd1cS3YO/1335NkYBgCkyM5zNnk+d7Pf72TZDJnTibvPDlJhgICAAAAbCAgAQAAYAsBCQAAAFsISAAAANhCQAIAAMAWAhIAAAC2EJAAAACwhYAEAACALQQkAAAAbCEgAQAAYAsBCQAAAFsISAAAANhCQAIAAMAWAhIAAAC2nFVA1rtyPsMwDMMwDPM/NOFw1gEJAACA/w0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAlugOyKM/Sts/lPb9FDghBId/kf77iZlvzecHTgurY9IzQ6TqtaUHd/hP+jRTatlA6rtMOuI/6Zz48VPp/W3Sz7m+0R//E/x0W4J8nwAA4H9ClAakiZeP1kotSpgtN5veZkHeMfjp81K/5lJqrP9zrEmrLA1aIf0WuExYmG1b3MFcv/k6d/3Tf9Ibw6VY8/XK3iod9J90dszX2Pa0tGiJtPt0G3/IfP0qZjvipPHvBU6zHJZGVzenx0tTPgic9kcE+T7z8tsu6dFFJjxNvAIAgHwThQFpwuUvA6XEGMnrMVtuNr31A2cOyD3LpDImpKzLVm4j3WqCbkh/6aKyUtwF0iYTVWETJKyO/Sxt2Sh9YuNI6RmZ7Z1c13wN8/2n/ytwWm45AvLedwOnWayANN+zdfrkrYHT/og/EJB75krx5jaoN+XcHokFAAAnicKANKW44hZpwp+k9SYCY/IIyKP/lS4r5A+dK5ZKvwdO9zGRtfk16QcTQ0c/l+4fKy19S9r4qNSvn/TnT/0X+8aE1uKp0qAB0i3DpEdfOvl6ft0jzR0q9b5OmvM3cz25wuqHTdLE0ea63/Z1l8/hr00ILzYRN8FEoAncnT8GzjA+fVYaY05/71vpxYVS/z5mRkov7zNnmvL6yzSpeXH/12hrYnrUbPM5uXeAjYA89p3Ztsnm+3pD+sps64TB5vu/2Xz85smhl9f3qQPSW+ul8eYy/c2+srZr41f+s/5jvqebTbxbR2JLtJBGjJKeyhG/P+yQlqWbr22245G/mn1OYQIA4JQoDMgcnjORkldA7ntQSjCX8baWPj7DYcpDJp4qmRgqbeKqYIzZI2ZuM6F42ERWPY/5OmZKl5NSzGWs8/r92R+DR03UXVnKnGa+hm/MeclJ5r85wmrHrJOPvP32ntTBevrdRFzJsoHtO1/6k4ldywsmkGPM59e60P/9ZV+3t7a06Wfp1jInTrMmtoL0ugnGk9gIyCMfmG2LN2FXSypn/nv8uhOlcSZ6LaF8n77tNuclFjHXU9T//3trSm+a2v6nCWArHnN+/sDn/Z+3Z4VU3nzdGHN9ZU0YW5crc5m0O5xHhgEAQDYCMq+AfPl2f5BccI+vnXTAxNSwwdKgQWbMf4cukL4zJZgdkFbYtB4rvfWWtOsHc71fSI88Iv0r8PTzfx6Sksz1Fb7Bd8BNH5gotOKwWFdpu4m7XVnShSa8ThuQZkPvb2XONwF36z/8Ebojw4SpOb/6aP82ZoeYt7L0oAm4r7dL3a0jjuZzxmw21/GLNNbEnvU6xpkm/n6xNiS3PxCQ1jZ3N7fl599Jj/Xy77di10u/msuE8n3uNlG96s3A0VmzTeOtp9kD22yV89bp/uuoM076wXwP1vd6bL/Us7D5fk2cZn7mv9yqnuZjKzCfs64IAACEGQGZV0C+aELRukx2QP78lFQqRUo28WOdHldTeteckR2Q8U1MHOV++tScv8cE5VPrpGUm8oqYuEkwkWMaSKu7m71nrqf3k/6LBnttYM6APPyN1DXBfO0i0pQnpD+b6Hryfv/X9l4ifWU+Pzsgu630f77F93XM173ZRKe1PeF8DWR2QMY3lLYFvveDL5j9ZL5efCMTjOa0UL5Py4GvpNf/bmJwlQlkc33W+Xe84j8v2GsgDzxjYtyc5m0urTP7wtofy8xtZl2u0cwTlwMAAGFDQOYVkLvm+GMk4WJpX44LHd4s1TARZQXkOzkC0tte+q91WDDg8B7pOiu4zHWklpQqlw1cnxWQ2RGVHXaWPALy0F6phfXmH/M5ickmZLPHRG3Z66RvcwTkZY/5P9+SdaX/c3xP+4YSkOYyY2r4t2NkjsCzwnJYJXO6CcZZgT+/czwgLzLbGii2Q29K55vPjTNfZ6u5rlC+z83pUjmvOc1MWfM1rFDPKyB/XO1/Ct8K2uP7wkyK+dwuD5/+dgUAAH8YAZlXQB4xkdQkwR8yfTJP/BmdUANywzD/U7mtM/xPWR96VSpvLpd9BPIvfc11m/ObmFC1osh6Q8pt1fxfL1hAHvlButLEUUxx6RnrCoLIMyDNF5pSz3xsPYV9ur/BaL6HJZ3921Z/on/bLb+Z77uO9XR1YenJwNPy2QFpvZby6a/9p/33Ef9T9QldpP3muvL8Pn+Wrkkz211CWvu57yr822zOzw7If5nA9JjruHCcr299Dr4olbZi+lLp6xz7HQAAOCYKA9JExoYF0uBBUlfr7xmaTS/Twv96xuXvBy6Tyz8nSKnmclaAnd9GGmAC7apmJhbNabmfws4dkNlv/qhuYvX5v0vDW/rjLjsgv14nFTXXG1PIhOFtUrca5vLm49MGpLnux3v7r6Oyiab7l0krFkv3mu0f87Tv28s7IM2FfEf/zGWqdDXbNFr6KPuQXg77/+SPM+voXt1uZh/dZLahmP/zapjvy3pto+X4ayDN6Wm1zXaPkZpar7k0H19iQtKK8zy/T7Mz+ljvdjdxPOpxE5z3m1BN9Z+fHZDfrfJHabwJVetPKS2x/j6l+bw7Kvsv1+YO6dGVZsznDjHBunqv//MAAEBYRWFAmppZ2NZssdnk3HPV+sBlcjPBtcWESdcL/dGYffnU8lKPGdL35vzDG6ULTGgldvG/DjHbkY+lG00sWUFnBVzNq6V2JsKSrwkEmAm3p0xQFTWfa11nefP5a0woJnml8YHXHv4rw/91LzIh6Tt6942Ubj6/aCDarEk5z0Tky/7Lv2pCyorWK020ZXuqj//rW+8Mt+x/NhBo5nIes31B/5al+T7eNXHatExg+83Em8jragJxb4433mQHZJwJuaubm/9alzVfq7qJuN3Zh2xD+D7fN2F/nvUUtjk/zuyjG8z36DVhOHyD//xj30ujG/m/Nytqh77mP/1nE/4DmgSeyg587VINzdf7wn8+AAAIqygMyLN04AfpPyYKrXca537a+7CJqoPBngs3p+3fK+0xQWOdffSQuVyuI34HzfV+9tWJp2ZzO/CbPx5zOmpO++xT6b8mKE86z4Tf7+a8kzbFfPD7776zjrM+3/pevssRg0GZT/rBbPsn+0z05t4I46TXQJpv4Mtd5r+fnbq9lry+z8M/mf20U/o68IcyD5j/nvR9mG351mzHvm9P/l4sB0xgfvqJ2dc/B04AAABOICBx9oK9iQYAAEQsAhJn7+gn0rV1pLrXS5+ddLgQAABEIAISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEg46usfDuqTL39jHJ5fDxwJ7PH88dNvh4NuFxPe+e6nQ4E9DuQv6/c24+x86/L7u7WN4UBAIqjnN3510h2CcWb2fPZLYI/nj63/+jHodjHhnde3fhPY40D+sn4e5z/+L8ahsfbv3s9/Dextd7K2MRwISASVHZDB7iBMeMbav24JyGDbx4RnrP1LQMItuL87O9b+JSBDEK6NgPsQkM6PtX8JyMgfa/8SkHAL7u/OjrV/CcgQhGsj4D4EpPNj7V8CMvLH2r8EJNyC+7uzY+1fAjIE4doIuA8B6fxY+5eAjPyx9i8BCbfg/u7sWPuXgAxBuDYC7kNAOj/W/iUgI3+s/UtAwi24vzs71v4lIEMQro2A+xCQzo+1fwnIyB9r/xKQcAvu786OtX8JyBCEayPgPgSk82PtXwIy8sfavwQk3IL7u7Nj7V8CMgTh2gi4DwHp/Fj7l4CM/LH2LwEJt+D+7uxY+5eADEG4NgLuQ0A6P9b+JSAjf6z9S0DCLbi/OzvW/iUgQxCujYD7EJDOj7V/CcjIH2v/EpBwC+7vzo61fwnIEIRrI+A+BKTzY+1fAjLyx9q/BCTcgvu7s2PtXwIyBOHaCLgPAen8WPuXgIz8sfYvAQm34P7u7Fj7l4AMQbg2Au5DQDo/1v4lICN/rP1LQMItuL87O9b+JSBDEK6NgPsQkM6PtX8JyMgfa/8SkHAL7u/OjrV/CcgQhGsj4D4EpPNj7V8CMvLH2r8EJNzC+nm8dsxfg/6sMmc/14//GwEZinBtBNzHCsg+o/8S9A7ChGcmLtniioDsOGil5q3fE3QbmbOfkfPfJCDhGgSks0NAhoiAjFwEpPNDQEbHEJBwEwLS2SEgQ0RARi4C0vkhIKNjCEi4CQHp7BCQISIgIxcB6fwQkNExBCTchIB0dgjIEBGQkYuAdH4IyOgYAhJuQkA6OwRkiAjIyEVAOj8EZHQMAQk3ISCdHQIyRARk5CIgnR8CMjqGgISbEJDODgEZIgIychGQzg8BGR1DQMJNCEhnh4AMEQEZuQhI54eAjI4hIOEmBKSzQ0CGiICMXASk80NARscQkHATAtLZISBDREBGLgLS+SEgo2MISLgJAensEJAhIiAjFwHp/BCQ0TEEJNzEqYBMX/m2Jj28QZMWv6mpyz9UepDLhG3WPakuF1RVizEfBj8/92Ru0dTFZtus7cs5izdrTlaQy5/FEJAhyo+APPTjl/ps3z599tl/9eW3v+pI4PRwObJroS6t2VuPfX7M9/Gx/Zv1p7Uv6d8HfB+G2VH9/NlH2vTWZm0zP3D+r+gOjgXkuvf9d+TFb2jyo5s1e12Qy4Rtduruvs1Uqt1CTV8f7PzTzS7NeujvGj79Lxq7ZJvmBb3M2U9kB+RuzVn2ZmCR/qemrdzu2H70zcqH1bRcE106e2fw888wGSs3aOyizY79wiMg4SaOBOS6v6rjeXEqEBOrGDMFCsTIW6yJ2ox8JfT7Vdb7mrL4Xc0Ndl7uyVytFkUSVfvO94Ofn2vm3tNDiQUKmO3KNTHl1XbqjqCf80eHgAzROQ/Iw+9q3IXmh/T4D0CsUit30F1/+VSHAxc5W4ffn6DaSS01f+9R38c/ZvZUcnx9Tf0wzKl6YKceu/5CpXnTVKp0ESXGF1bDO5/Wf/1fNt85E5C7NOba2oo1i4u1yMSY2zDGW0qVOs3T2LXBLh9sdmv2kjc0fU2w83LPDg3rXkGeutM1NdSAXP2MrmpaQV5PmtIKpykurqgq91xtM0BDm0gOyHnL56qWx1qgrV8oMea+6lFKxW66cuaW0ENyzWZNWvKBMoKdl2vmLZ+jmt6yaj3Zxi+D9Vs1ZtgAnV8oTrGlb9Zwhx7MEJBwE0cCMvMJXVw6QRWvf8l3f81Y+aJuvLiSYlMu1Y0rQltb0if2VSFPO127PITL2wzI+eu3a/bK9zRzRfa8qzH9myshrZv6h/L1bAwBGaJzHpCH3tKoqglqPHunjpj/++2r97SsTyXFFeyhVV+F5/hd7oDUkZ/11eff66D/ozA5ou1zmiq5TE8t3fWb7+OvnrtNNbwldd1T37niSKQzAblTd19TQ56Kw3S39bRB1laNv6evSsenqNbtG0MLi3V/0sVlElR14BshhIXdgNyuET2rylOyp/o/tN18vEMTR/RQkbhyajkxxKdKbExEB+SSmaruKa5m4639uEf3LV6nLjVTFXf+XRod0lNGuzWhf2N5yt2qkSGEne2AzNqgG9pWVlr5rmrZuKLiTEAOywxyuTAMAQk3ORcBaZ02d8K1Souvr+4P7PJ9fN+ilbqy22WqUaOBzq93uToOe06zA5dNX/qCht7UWSnxDdTp3ic0fOb/abq1TqzforuH3qlmTVro/Fod1KzvMo23DjZkB+SQtzV+/FR16X6zLhm4XPeu9l9fnrPub+pSyatil67TfcHOP4shIEOUvwEZOOm1O1XBc4FGb8o+BnlAnzyfrluvuFgtWnZUr5HL9M532Ul2TN+/k6UlL36ig1+/q6yMezV6fLoyN+8/fn25A/LoF69r2cKntet4QR7Wf19ZoBF9u6ld607qPWqpNn7jv/7fP35BC0b0UedWzdWq83Uau26bgubB4fc1sU6Cat373okjp0c/1v2tvCrS9y9yw4/eOQlI67TMx9WulFdlrn4m8FTHTk2ZOlMd2l6sKtUb6YLmA9R75rv+uMzarHHTZ6phMa/O675Yw2Y8pbFLA09ZrnlJA6/rq/r1mqhKgyvVZeTzZnE6EZATH1yr3r1vUburJmjg/K3BYzVzndqU8KrSDScWQd9CU9Grwp3Xhv0pzugJSOu03Rp/Q0PzYO8aDfaF2h7NTF+gSzteomrVG6hKo17qPn5DYB/v0axFf9FN3WoqrkRPXTvV/EJJf1NzrPOy3tKQgQPVqFFTnV+nq9oOekLTzM/S8YCctFF3jxynjt0HqdvtT2jK6eLTXM+w0VmalLlLY6+rq3gCElHinATkmlfUv11FJVS5S6Os+9X6zbq2SSVV6zpR/cctU7+rWiotrpIunm494NutSbd3U8kiqYqNSVJqiQoqft6VusGs7WMHNFNyci01vXG++g8Zp5Y1K6vxPdsCAelR0QrVVbRyZ7Xo3E3l0+KV2vx+TQnhYMHscb1UyFNP3e/3x204h4AMUb4H5NFv9frd9ZVQqq/+/L0Vccf01VM3qEJqDfVd+He9/vJ6je9YWoXbPaBdvk84om3TGyqtYVd1qlpFba4brAGdL1CKt5pu/8f3viN/uQPSF6hJXbXiW//1//jSUFVPOk+dxy7V448v0cRe9dRq6gc6cuwbrehZTR2GLtATzz2rteM7qpSnmka9dch3PSf5YY0uT0zWNX+yjj5mO6iXbi0jb5M5gW3NX+cmILdp0pi+Kp1YXR1nmEXBuszaR9WkbAM16zdfg8YtUvem5RVX+Crd8piJhKUPqHG50kqMi5EnrZyKlaqiBrdvUEbWG7q+eUl5y16qbkMW6fqbBqpG2Q7qs3S7LyDjilVXmRLVVbtjPzWqVlyxqRerz0O7c2yXfzIenqyqJ0WPNR9pQMtUxdl5GjzEiaaATF/6hLrUKKgi7R7xvxxg3TPqUr2G6l41SwPvfUS9Ol6ohITmumqRdbts1+3dL1SRtATFxBVUoVIVVKLOPRq1bquGdK8qT5EWaj9ogW68dbgaVm6gTmY98AWkJ1UlK1RWsZo91KpDO5VM8Kpk9yx/eJ52CEhEF+cCMk4x8YnyJiQpPraACsRXUMPbntPswLqZnpnj2YHMLLU1D9bP7//qiSOWvqew2554CjtzrVoV86pivxdPPKDP2qY51v3UF5BxSrtoZuDlT3s0fWhnJXrbq29eT0mb3xe966cq8aK5YV/TrSEgQ5Q/ARmnpDLVVbdubVUtlaJiDW7Qwre/9T/te2S7ZjZKUJVhr+t33yeYxvzkAbVOukD3+I5QWgHZQJ7UxhrzauBzju7To5eYH6YOD+sz04xnDMhjX2hp5xSVuO4p+XrV56B++z1w2YM5YvHQBg2vlKBW938s/7knHP34frXyltTN/8j5xPhhvTO2prw1xmhzuF7QeRacDMjYGI88ZpHxeuIVUyBeRZpO0aiV2Xf6nZqbeWIByJg/SuU9VdUx+80RgUe6VW8y4Zh9mfRhKuupbB7NBiLUzLw1H+m+wBHI2LTW6pkeeAp65WJdVNCraje/ecpRyPS5Q1U2vmLgUXH26SZmLimluKqjNS6kp15Dn4gPyPgCivWY2zkhQXExBRST1lRdJr8duN12m18oOd7w8th81fGmqOGo7NvQegr7InnKDdaIwFHEeUvnqlZSYdUbvvnEbZf5ke9dlP6A9KrUJUs11Xc7+cMwrlg/3XHGMCQgEV2cC0ivyvVcq/GLXtX4+U9pYL9OKhpfVLWHnFirrfV99tINGn//crUtn6AK171w2oDMWDhOlTwV1HZakJelBHkNpG/99jbQ5QtOPTiQc9Lvs9b50mo+/oOg55/tEJAhyp+A9Kr6wBV6/u9rNf2KKkqp0Efr9gUS7ce16pGYpEvS39aWLVv8s2mxriqeqO6rfzAX8Adk0klH+Y5q7/yW8pa8WVbPnTEgD76oQSZeOiz+wh+fQR3Wz/v/o727ntOIWoknPd2e7ejH801AltItL5wckO+Oq6mEC8fp3QgPyPhyAzR4wauasPAFjRw7Xg3KJCi57lSNzxloWR9q+uJXde+MkaqccxEJEpAzh7SXJ7mHBp7yxpogr4HMekmXV0tU2V7P5ljU/JN+350qE19JF8/M+U7eHbq9c2nFVxurewnIkMd/BLKoGg75P01Y9IrGzV2rPpfUUaK3pi6ZdSL0rRe3z3z0NY2fN0f1UxNUd3j2a01PDcj0ydersKeprgxy9DjYayDnjrlSyQmXqv8Z36BFQCK6OBeQJ78Gcv76TerTwDx4rDNNU9bv1PiRN6hKicJKLX2hKlStocIJnjMHZMYIlct58CDnBAnIjHRzeW99E5Bnelr6fQ1qX0pxFe/UKIfeNEdAhih/AjLHU9i/btCICzwqf+uLvtcaHtu/VJ298SpZs7latGhxfFq26qbpb/jfrHJqQB7T/qWdlVD4ej1zII+APPC0ri+UoEtXBnujy2HtfeJOtalYVCWr1lfjJnV1XoonaEDq+8fULSFZV63PGQ8H9eKgUkpos0Cf5j5kmQ/O2Wsgzcwc3kUJcQ3VY6EJg6y3NPiqNiqRVlRFK9ZRhcrllRh3poDcoymDWiq+YK/Aa+tyTrCAfFk9qpuAvOZvp7ymMeOhCTrfPGJuOi7nU9gfqn/zFHkapWtmjsuGY6LrNZBmMjPVpoRHxS57XOnr39ddAy9XucJFVPi8WqpQ5Xylxp05IOeOu0YpnlbqteTUbQ0akOOuVoq3KwEJ5HDOAvLx99T3okQTkFM1ZeWDapCSquoDA3/WJ+t5XVo58aSA9L8Lu436LPXfv+ct9b8spd7wLYHryzF/MCAzFs9UjaRkVb8l51HR8A4BGaJ8D0jzv7vva6qEor30uPVGmQPPa2DJZHVf/Z3v3FP5AzKx7mRtPX6U75DeHlVN3trjtcWcdsaAPLxZY2p4VX3M5hNvfsn205PqUyRNnR7ca67ROLJdMxomBQ/Iwxt1dzWvLrhno/+yliM7fE+/Vxq24cRp+ehcBuTsuy5TYnwDX0CmT75BheNrq8t9H/nO84WIN0dAWu/CNgtVlRxHIO8b3VPJnqa66uHcR6bsBeT8tSvVrLBX5XMsavMzrXd9e32XD/eCE30B+YTalfKoaLfHNdf39FQZNRkdeDo6c4WaFUo8NSDL5DgC6TtCXEHtTnqJgX8IyAhw7Es9c8/lunLGa/opcNLpHdaHi/up203LQ3rN+LGvn9XYHj015aUfzvDsUXRwLiDN2nnVek20/vbrg//QkMFXqpSnoGoMek0Zy9NVKyFVNQZb6/YuTRk/yPxePfkIZPrsW1UqvqLaTN6q9BXvaFbmG+rdIE2eSgM0aPE2zVuzQXfc2EuXjPvgDwbkTo26urriCl2hm1YGOz88Q0CGKP8DUjr6n4fUPilVXZZ+bhaGn/XirRWV2miknt8X+MvfB/Zr6/OvaKfvRZH+gIwvkKaGdz6pPb8c1c8fLlK3kgmqM8H/jugzv4nmoDaONgFUsqvuf+97HTnyvbY9MU53PLhVh75dpe7JaeqyZJ/5Kkf0xQv3qEnB0xyBNNezeeyF8pbspoVbf9TRoz/po0euUNmkBpqyxQ356PBT2OfdpNsetP4lgNc1dvpcNa+YpISa4zXWRKXvl358PXWbZ8Ij610N69tSqTmPQPoeuXpVuP0yzczcoukrd2reisW6qJBHxdver3Gr92jukifVu8vNun6x/000IQfk4x/o9s7lFFe8m667/wPNM19/RL+WSk5qop5nfFrkj01UPIU99BVzO2/QxAV/1U09myjFU1Xtp28zi/1wlYsvp+bjrL8L+ZHG3tlDxeO8OQJyj6bf3k6epIvVZ/EOzVn+ru7L/Ks6lfcqpe5YjVi+Sxkr/6EBV/VVT3Mf+yMBOfexzZq+dKOGX3Wh4kv20+BHN5mfp4/C/kCBgAyReRA9q7FXyV2W6os8K++Anh1QXHHn3aFXQlgyj+zJUIuEJHV4eN8pr0mPNo4EZNazuvR874m/0RyToJSyzdXspif8b5pbv0V3XnORUuOTlFKkhIrWHqhWdQqrYr8cRywzn1XPOsUUa/3t2MR26vPoHqUvWqiWVYootoD192Q9Sq3SS9cvNGux7w02yaozbOvxbciYd5fO8waeycq+zpyz5jG1KOpVqSueDO2Plf/BISBDdM4D8vAmjTa//JvN3XNiETj2lVb1KKSkTo/I+sdjjv2wSQ9cU1OF4hNUsFhRpSak6LyW9+oV30PawBHIWn01qncN88McZ35Yk1Wl50K997Pv2nTkgymql9RaDwSeRz70xnBVTr5MjwUOah77aZPmW6+9jIlRbGyskit11rTXvtWxYz9ow+RWKuFNVvGypVW5/V0a0rGob1uDPkD+ZYseNI+G0sx1xHvi5CnWUANX7zLLojs4E5C7Ne7GxvLk+EPw8akVVaXdOA1ZErjTr/mbrmxQ2uyTwipYqLQqdhygOiWsN8hkh8EujRvUXoXirD9GnqSqN5lHtyY2pt47WFWKmAXMWnziCqp0i2m6e81ODb+8krz1Zmra8YB8VVfUSFS53s8FDYV5Kx7XZQ3OkzcmTnFxsebRakO1GfVakNg8+4nkgJy/YoHqp1h/QNx/W8fEpapIla665J5X/It31gb1b19VCfGpSitcQqWbDlaTyqmqN9J/5NmaeY8uVrMyib4/Oh8XeCp79pxJqlsmxf+H6GMSVaTOnbrdesrL+sPlCSf/qxJz7+1l7v/dNCBYQPoeiMQf377j21mot24N85FIAjJ0xw7+qt9CfQx95Hf9GngDY96O6dCvv7ni2Z385khAhjhzl2/Q+Iff860B89Zt09xTXle+XTMXv6bJK0/cj61nkmYufkUTHn3/pHU4I3N7rjV8t9LX7gj+J9p8Y4J0zTalZ/8ucGgIyBCd84A0jh46oMO5H50ePagDh05eSA79+Ln2/vsTfflzziebc74G8qh++fJjfbL/11yPSI/q0MHDOZ7msD7OnYDH9NvXn2rvp9/o91zbcvC7fdq7z/+Hx48dPqCDZ1zfjumAufzHn3yhn1y2sjkTkKHOTs165BVNXOr/ZwRPXSj+pfSV/9TExf6F6PjpWVs1ddErmrQyx9OmWTs0d93JcTRv3Xaln/ENMdY/w2cWuoc2OvpPLUZ0QIY0Zj8vfc38Ytjqu32t2/mUxd26TR/aoOlrcmzf+m2a/pD18/HRSb8sMjJ35Po52aW5meE/cmx3CEi4SX4GZDQMARmi/AjIs5MzIAMnIaj8DcjoGAIyOoaAhJsQkM4OARmi/72APKqPl1yt+teuVPZf/kFwBKTzQ0BGxxCQcBMC0tkhIEP0vxeQCBUB6fwQkNExBCTchIB0dgjIEBGQkYuAdH4IyOgYAhJuQkA6OwRkiAjIyEVAOj8EZHQMAQk3ISCdHQIyRARk5CIgnR8CMjqGgISbEJDODgEZIgIychGQzg8BGR1DQMJNCEhnh4AMEQEZuQhI54eAjI4hIOEmBKSzQ0CGiICMXASk80NARscQkHATAtLZISBDREBGLgLS+SEgo2MISLgJAensEJAhIiAjFwHp/BCQ0TEEJNyEgHR2CMgQEZCRi4B0fgjI6BgCEm5CQDo7BGSICMjIRUA6PwRkdAwBCTchIJ0dAjJEBGTkIiCdHwIyOoaAhJsQkM4OARkiAjJyEZDODwEZHUNAwk0ISGeHgAwRARm5CEjnh4CMjiEg4SYEpLNDQIaIgIxcBKTzQ0BGxxCQcBMC0tkhIENEQEYuAtL5ISCjYwhIuAkB6ewQkCEiICMXAen8EJDRMQQk3ISAdHYIyBARkJGLgHR+CMjoGAISbkJAOjsEZIgIyMhFQDo/BGR0DAEJNyEgnR0CMkQEZOQiIJ0fAjI6hoCEmxCQzg4BGSICMnIRkM4PARkdQ0DCTazf28F+TpnwjLV/CcgQEJCRywpIFhpnx9q/bghIbmdnx9q/BCTcgvu7s2PtXwIyBOHaCLgPAen8WPuXgIz8sfYvAQm34P7u7Fj7l4AMQbg2Au5DQDo/1v4lICN/rP1LQMItuL87O9b+JSBDEK6NgPsQkM6PtX8JyMgfa/8SkHAL6+eRcXYIyBCEayPgPtkByTg7bglIxtkhIOEW3/50yPfzyDg31j52M2tNCgcCEkHt/PTnoHcMJrzz9Q8HA3s8f3z+ze9Bt4sJ7/zb5UckAEQPAhIAAAC2EJAAAACwhYAEAACALQQkAAAAbCEgAQAAYAsBCQAAAFsISAAAANhCQAIAAMAWAhIAAAC2EJAAAACwhYAEAACALQQkAAAAbCEgAQAAYAsBCQAAAFsISAAAANhCQAIAAMAWAhIAAAC2EJAAAACwhYAEAACALQQkAAAAbCEgAQAAYAsBCQAAAFsISAAAANhCQAIAAMAWAhIAAAC2EJAAAACwhYAEAAC2fPntAd/vbsbZsfazW1nbFw4EJE5r3/7f9PrWbxgH55Mvfgvs7fyze98vQbeNCd98/cPBwN4G8tc7O7/3/e6e//i/GIfG2r/WfnYrAhKOs25fxvnJb8G2iQn/AG5AQDo/1v4lIPPAohjZWGScHzfch7idnR/WSrgFAen8WPuXgMwDi2JkY5FxftxwH+J2dn5YK+EWBKTzY+1fAjIPLIqRjUXG+XHDfYjb2flhrYRbEJDOj7V/Ccg8sChGNhYZ58cN9yFuZ+eHtRJuQUA6P9b+JSDzwKIY2VhknB833Ie4nZ0f1kq4BQHp/Fj7l4DMA4tiZGORcX7ccB/idnZ+WCvhFgSk82PtXwIyDyyKkY1Fxvlxw32I29n5Ya2EWxCQzo+1fwnIPLAoRjYWGefHDfchbmfnh7USbkFAOj/W/iUg88CiGNlYZJwfN9yHuJ2dH9ZKuAUB6fxY+5eAzAOLYmRjkXF+3HAf4nZ2flgr4RYEpPNj7V8CMg8sipGNRcb5ccN9iNvZ+WGthFsQkM6PtX8JyDywKEY2Fhnnxw33IW5n54e1Em5BQDo/1v4lIPPAohjZWGScHzfch7idnR/WSrgFAen8WPuXgMwDi2JkY5FxftxwH+J2dn5YK+EWBKTzY+1fAjIPLIqRzbp9b5v1ctA7CBOeccN9qEW/xUpftyvo9jFnP9NWfMBaCdewwqbbkMygP6tMeMb6vUlA5oFFMbIRkM4PARn5Q0DCTQhI54eADAGLYmQjIJ0fAjLyh4CEmxCQzg8BGQIWxchGQDo/BGTkDwEJNyEgnR8CMgQsipGNgHR+CMjIHwISbkJAOj8EZAhYFCMbAen8EJCRPwQk3ISAdH4IyBCwKEY2AtL5ISAjfwhIuAkB6fwQkCFgUYxsBKTzQ0BG/hCQcBMC0vkhIEPAohjZCEjnh4CM/CEg4SYEpPNDQIaARTGyEZDODwEZ+UNAwk0ISOeHgAwBi2JkIyCdHwIy8oeAhJsQkM4PARkCFsXIRkA6PwRk5A8BCTdxLCAzt2jq4g2atPgNTX70Xc3JCnKZsM1O3d23mUq1W6jp64Odn3t2ataSNzTpYbN9Ocds67RVu4Nc/uyGgAwBi2JkcyogM1ZtNHdc6877pqYu+0DpQS4Ttsl6Xb2bVlXtW15RRrDzTzfrP9DkeX/WsJnPafKqPcEvE4aJ5ICct+YdTQ78QpmydIvmOvkLZf07urFtVVXr93f7P09ZWzRx/iuanhnkvDAMAQk3cSYgd2r45ZUUWyBGMTGxiilQQDEJ5VT1ssWauC7Y5YONibxH39SMkO6HOzSsewV56k7X1FACMvMxNS8cowJmu06eGKV1fExzg33OWQwBGQIWxcjmSEBmvaTLq3lUwFpkzFh3YE+R+mp2x/O6L9jlg03WB5r68ObQLp/1orpXTVS5Ps+FHJDpC+apSYU0xScVU8HUBMUmX6iWozbYC9AQJ3ID8kP1b55sbl//7Wz9QolNqao6fddrRkhHDKzZpumL39bsUMJz/dvqVTdRxbs/YSMg92hWRoZaX1BUsbFV1XH2ziCXOfshIOEmzgTkDt15aTl5ak7QROv+um6LxgztqWJxRdRo1PtBLh9kMleqWaEE1R4ayuVtBqS5r6ev2qKZK947PjMWzNCFyWmqdcfbmhf0c/74EJAhYFGMbM4E5PO6tHKCSl/5V98v+oxVr+qWS2sqPqG1ei0O7Uhf+uzbVMpzka5YFMJTD3YDMuv/1KN6stIaTdOYNdbHG3Vb12qKT+uqG5aG/0hk5AbkVvVrnKCE1ot9j+7nrd2o4f1aKTku9FCbt3SmqnvKqc2UHUHPP2lsB+RuTRjaXcUL1VTDSzqoaDwBiehwTgLSOi1zuZqmeXX+ja/6197MDbrj5lvUqEETVareSg0un6tRKwNrauZGjZk6QbVTzOWvWa1hM/6sccut9X2PZs9bqK6duqp6zaaqdfFI3fTAR76vlx2QEx9cq969b1G7qyZo4PytIcbgLo3tV1+eYr01eFWw889uCMgQ5MeieOjHL/XZvn367LP/6stvf9WRwOnhcmTXQl1as7ce+/yY7+Nj+zfrT2tf0r8P+D50wCF9u3eH/vPj0cDH7nEuAtI6LX3mYJWMr3LiF/i6tzXs1tvUuFFTs9C0UN2uUzV8mX+hyVj+kobdfqX5hV9DbUY9ruEz/6Gpgac80h9epSsuvVw1L2ys6i0H6drZ72re8YB8RpNnpuuyHgPVsd8Dusu3OAW2KcdkzBup8zyV1X7miZjIWDxZVeNTVX9kiI+kbUy0BKR12rxHp+mC+BQ1HLXN//HyZ3R9796qU7uhKtXqoObXr9bkwNNd81Zt0Khxg1Uhvrjq35xlfqE8o4m+lxLs1rTpM9S+bQdVrdHc/GxM0pBHzLYfD8j1mnrfAl3ec6A6XJeu4UtO933t1uSpj2jYIzuUnj5C5TwEJKLDOQnI9R/q3mE9VDS5gbpn+O9Xs4ebB2w1++qK4cs1aPhIXVg0XkU6r9Ecc17GohmqW6akEmJi5C1UXsVKVVOzu941p8/ShQVTVKb1GPUbOU9XdG2u4s3v16xAQMYVq64yJaqrdsd+alStuGJTL1afh0I4sLBmndqU8qrMVU/bf8lLCENAhuCcL4qH39W4C+NyvH4hVqmVO+iuv3yqw4GLnK3D75tHQUktNX+vP+h+zOyp5Pj6mvphuFNVOvrNJj18Yz0VivGo1f2fyG0JeU4CMvNNDe5aXQnlbtFQ64ifOW3uuD4qXe1qdR+6VIPvGqf65o6e2vphzVy/RzNGXqPSxQoqLiZBKcUrqHi5Luq9aLeJkRVqWTpRhRvcoT4jF6jXVV1UptbdGpNpBaT5/PLVVaxsKzXpfLWqlvDKW+1u3RPktTmzh3eR19NJ1+d8VJqZqdbFvbaeBg91oiYgs97TyH4tlFLkUt3wqBWCOzWiZ32Vaz5U145eoYE3X6OyCamqMcj/UoH06UNUqWRReWLilVikgoqVbqhLpu/QnJlDVC6hqM7vNlU3jJyjbm3r6rzL1ivdF5BeJZWrrhKlmuqizr1Uo0ySPOUHa1jg5+p0k0FAIoo4GZAxMV55Esz9Lt76Pe1ViTbpGpO9lmZuN/fT7Mvv1F09qyq+8l0ac/yI5Qr/U9h3Zj9Q36nhPSorvuKdGnX8dZF7dN8a6wGoPyBj01qrZ/qH/vNWLtZFBb2qdvObeRyF3KNpQ7soObGlrg4lNv/AEJAhOOeL4qG3NKpqghqbhf6I+b/fvnpPy/pUUlzBHlr1lf+I4dnKHZA68rO++vx7HfR/FDZHPlqoS8qXVbOBN6tD8QQTkB9HUUDGKyYuQV6z0MTHxahAbGnV7v8XE4iBy6zLudDs1r3X11d86Zs1LPtI4+xbVcrTKMdT2Ls16ebm8ha+UgNXZn+eCVGz0GT4jkB6lFD1Dg1d4T+KOXfaTSruqamu6bmjyVzPwGaKS7lSt6zNcfq6v+mSCl6br68LbSI7IL3mtjWxbv1CiYtVgZiCqtRjhSYFbseMzB05gvw9XdcwQcltl5w4YrnEegq7rFpPzn4Ke4v6tyikhAazNSXHL6I5a0z4BQLSU/4GDV7q/7lITx+ucvHl1XbqmZ8CJyARTZwMyPiqd2rowlc1YcHzGn73KNUu7lFas/makuN1zBmrNmnSov/THT1rm/vrHRqV/UA+d0Cu/6euqeNVWqdgb3IJ8hpI3+vrE1W217NnfqCf9YK6V0sIHJQIcn4YhoAMQf4GZOCk1+5UBc8FGr0p+xjkAX3yfLpuveJitWjZUb1GLtM732XH5TF9/06Wlrz4iQ5+/a6yMu7V6PHpyty8//j15Q7Io1+8rmULn9au4wV5WP99ZYFG9O2mdq07qfeopdr4jf/6f//4BS0Y0UedWzVXq87Xaey6bfrFd86pjvzrJf31vW919NCrGlI+2gLSPDLtvET3LnpV4+//qwbf1EMlvWmqOuD/TrrjZ6zZbBaalzSyb1N5il2vO04bkNs16OIiiq8/S9NyLwhBXgM5b/l9utBbWq0m5Q4LKyCbKi71Kt1y/BGvmay/q0tFs82X/4mADHn8AZnQeKbGWrfzgr9ryJABqpTqVcnL1vmetvJfbo/mrvinJix8Xr2apMrb4sHTB+S6p9S+jFcVrnvh1F8QwV4DufZhNUxKVaN7/E+Zn24ISEQTJwPypNdAmvv21EGtFOdpoz5L9yjj4Ud0cZ0KSilYQWWr1FO5EqmKPVNAZr2iHtW9KtJ1XZB1N1hAvmwubwLymr+dcZ2eM/kGFY2rrIunbw96fjiGgAxBvgfk0W/1+t31lVCqr/78vRVxx/TVUzeoQmoN9V34d73+8nqN71hahds9oF2+TziibdMbKq1hV3WqWkVtrhusAZ0vUIq3mm7/x/fms08NSF+gJnXVim/91//jS0NVPek8dR67VI8/vkQTe9VTq6kf6Mixb7SiZzV1GLpATzz3rNaO72gip5pGvXXIdz2nFZUBefJrIK3YGNCyoOKq3K2xZvGZt2S1ujQ8Xylp56mMWWjKlyqo2DMG5Dbz+UmKbzxPs45fZ2CCvYlmZbpqeUup5cRTF5BZQzvJ4+mgfjmOZM7PXKtWxbyqdMNLZ35k+wcmsgMyx1PYvtmpEVdUUVzha3Vb5h7NmjlBDSoWV0qxqjqvSl2VSIs7c0BaLyUwt8P5/QMvys85QQPyETVKTj3+msvTDQGJaHIuA3L6bW0V7wvIj8yD/GLy1BitMb5nd3ZrQv+Lgh6BrDVkS+Dz39cNzVLkaTD71AMDfzQg129UvyaF5Kk+TuNyHBUN9xCQIcifgIxTUpnqqlu3tqqWSlGxBjdo4dvf+uJPR7ZrZqMEVRn2un73fYJpzE8eUOukC3SP7wilFZAN5EltrDGvBj7n6D49ekmqEjs8rM9MwZ0xII99oaWdU1Tiuqfk61Wfg/rt98BlD+aIxUMbNLxSCGFIQJr5UDe3LRIISLMwXFZB8ZVu0wjfa2f2aMbtF+c6Amm9C7uheizMDkj/YhJXfsiJxSh7bAZk+uzbVTq+gtpOO3FexoKxqugpbS5/5hD5IxNtATnqqgtMQPbRbWte0GXmwWDRjssDf9Znm25pWyhIQJZRq+yAzPo/3+tZC3VadeqfcCIggZA4GZDx1YZr+EPWH+l+TWMmT1OjMl4lN7pPk7I+UP8WqfJcOEHjzRqdsTRTXWoWNmt2zoB8QheX9qpk98c1O/NdTX9sl/+1iua+2WLUa7pv/TZNmTpZLXo+pOnr/1hAZtx/jyp4Cqnu0E1h/9M9OYeADEH+BKRX1Qeu0PN/X6vpV1RRSoU+WrcvkF4/rlWPxCRdkv62tmzZ4p9Ni3VV8UR1X/2DuYA/IJOazAkckbQc1d75LeUtebP+cTCPgDz4ogaVTlCHxV/44zOow/p5/3+0d9dzGlEr8aSn24OK1qewuy7XBOtfAnjoRQ27s6/KJSarUr/nTeRt120dSyiu8jDdbR6pzlv5Z/WoX+KkI5AZD4xWxfhSanzPu0p/7B3NNKfPnXqTiseVUO2Bf9PMrJ2akf6AOnSZptG+N9GEHpDzM5/WJRW9Sql3r0at3K15q/6hPs1Ky1Phdo3I+bR2mCayA9KrhCazNN53O7+sUaOHq3ohj4p1Xq3Z6/6qTuW9Kt55je+o8Zx596lBqfiTAnL+qgWql5ioaje9qrlr3tGM1bs0+rq68iQ3Uddpm5SetVXjxoxQs34mGv9AQM5b+76mL9ukqVMHq4ynstpOeEvTlm3RfWE+MkFAhuqYvnthgq64Yrz+4XvG58wOv79I1102SKv+dcYV1u/Yj3p5ak/1GPOs9ud91RHNmYA0Dw6vqaG4HG9w9RSsogs6TdOI5f7Xns+eeY+qFfbKk1ZKacUvUutL2ymt4lDdffxB/3aNur6pUmKtP0aeqtpDNmte1hsa2K2hUuMK+P6ebGxqDV10+z/Mfdz/h8u99WaeODqZ9aquqGHW+t6ne7PjDg3tVkFxpW7QHXm8se5sh4AMQf4EZI6nsH/doBEXeFT+1hd9rzU8tn+pOnvjVbJmc7Vo0eL4tGzVTdPf+M1cIlhAHtP+pZ2VUPh6PXMgj4A88LSuL5SgS1d+FyQgD2vvE3eqTcWiKlm1vho3qavzUjwEZO7Jek1X1U4KLDJmYswj1FIXqXG/tcdfaD03Y4pqF0+QJ6Wk0orWUdPuXVSsVH/dmR1w5pHmtU1KK876A9XeRuqxYLfmr9+iYf3aq2iC/19CiE2ooJrXrTcx6X9hdfnr/nFiUXlsnmon5DiylWvmzJ2mBuVSzfV4FBcbr+SKV+raeeE/+mhN5AbkR76jytYfEPffzvFKKFpLda5YpHG+xXu3Jgy/UiUTvEosXEIFy1+q9m1rKKnlwycCcv07GnxpNXljzG0aF3gqe82z6t2qqhJj/L9Q4os00cVj31LG+o3qXS9RJXo8eSIgM5eocWrB07wGcqfuNr/wYrO37/h2llbLU14be3ZDQIbqqD5e0FaJiW30wMd5r4a//rWfisRV1vA38niZkOXoPj3cIUkJzdO1O4TejGTOBGSIk7lFkxa9rum+p7F3aW5m7nVnj+Yuf0MTHtly0lHE9Mf+qQmL3tSsnA/is3Zo7jp/nGbPPOsNmGd4ADhv3Tbdl3ny5zgxBGQI8j0gzf/uvq+p+cXUS49bb5Q58LwGlkxW99Xf+c49lT8gE+tO1tbjf/fnkN4eZX5J1R6vLea0Mwbk4c0aU8Or6mM2n/png356Un2KpKnTg3vNNRpHtmtGwyQC8o/Ouvc1ZdFrmrbGurMHW2h2aNYjr2vS8pOPImas2aSJizZouu/z/KfNO+nPRwQuZ04L/ig1MOu3acbiV09ZyMI9kRuQoU3G6o3mF8Nb/n9tJsgvBCs05yx9XROXfnjS7TV3+QaNf3DjSf/ebrBfHnnezudgCEgbjh3Sr78eOsMzPDkd0W+/HAh93Tz0m349FNo1R7J8DcgoGQIyBPkfkOaB5X8eUvukVHVZ+rlZdH7Wi7dWVGqjkXp+X+Avfx/Yr63Pv6KdvhdF+gMyvkCaGt75pPb8clQ/f7hI3UomqM6E93xReOY30RzUxtE15CnZVfe/972OHPle254Ypzse3KpD365S9+Q0dVmyz3yVI/rihXvUpOAZjkAe/V0/fPO1vv78r7q5XIKazXhPX33zjX464J4FLl8DMkom2gMyGoaAhJsQkM4PARmCc74oHt6k0dUT1WzunhOPOo99pVU9Cimp0yOy/vGYYz9s0gPX1FSh+AQVLFZUqQkpOq/lvXrlJ+vCgSOQtfpqVO8aSo2PU2xMsqr0XKj3fvZdm458MEX1klrrgU8DAfnGcFVOvkyPBQ5qHvtpk+Zbr72MiVFsbKySK3XWtNe+1bFjP2jD5FYq4U1W8bKlVbn9XRrSsahvW4MF5KGXb1e5mFxPnxWIV70pHwQPznxAQDo/BGTkDwEJNyEgnR8CMgT5sSgePXRAh3MfpDt6UAcOnfxExqEfP9fef3+iL3/O+WRzztdAHtUvX36sT/b/muspkKM6dPBwjqdQrI9zJ90x/fb1p9r76Tf6Pde2HPxun/bu8//h8WOHD+jgyVf+P4WAdH4IyMgfAhJuQkA6PwRkCP73FsVgb6LB6RCQzg8BGflDQMJNCEjnh4AMwf/eonhUHy+5WvWvXansv/yD0yMgnR8CMvKHgISbEJDODwEZAhbFyEZAOj8EZOQPAQk3ISCdHwIyBCyKkY2AdH4IyMgfAhJuQkA6PwRkCFgUIxsB6fwQkJE/BCTchIB0fgjIELAoRjYC0vkhICN/CEi4CQHp/BCQIWBRjGwEpPNDQEb+EJBwEwLS+SEgQ8CiGNkISOeHgIz8ISDhJgSk80NAhoBFMbIRkM4PARn5Q0DCTQhI54eADAGLYmQjIJ0fAjLyh4CEmxCQzg8BGQIWxchGQDo/BGTkDwEJNyEgnR8CMgQsipGNgHR+CMjIHwISbkJAOj8EZAhYFCMbAen8EJCRPwQk3ISAdH4IyBCwKEY2AtL5ISAjfwhIuAkB6fwQkCFgUYxsBKTzQ0BG/hCQcBMC0vkhIEPAohjZCEjnh4CM/CEg4SYEpPNDQIaARTGyEZDODwEZ+UNAwk0ISOeHgAwBi2JkIyCdHwIy8oeAhJsQkM4PARkCFsXIRkA6PwRk5A8BCTchIJ0fAjIELIqRjYB0fgjIyB8CEm5ihY318xjsZ5UJz1j7l4DMA4tiZGORcX7ccB/idnZ+WCvhFgSk82PtXwIyDyyKkY1Fxvlxw32I29n5Ya2EWxCQzo+1fwnIPLAoRjYWGefHDfchbmfnh7USbkFAOj/W/iUg88CiGNlYZJwfN9yHuJ2dH9ZKuAUB6fxY+5eAzAOLYmRjkXF+3HAf4nZ2flgr4RbZAck4OwRkHsK1EXCnnHcGxrnJb8G2iQn/AG7wzY+HtP2TnxmHx9rPbhWu9YiABAAAiBIEJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCMhwObJdS/tfphuXbNORwEkAAACRiIAMl4MvalDpOJUa9IIOBk4CAACIRARkGB35/Vf9fjTwAQAAQIQiIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWADJcj27W0/2W6cck2HQmcBAAAEIkIyHA5+KIGlY5TqUEv6GDgJAAAgEhEQIbRkd9/1e9HAx8AAABEKAISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAABCSH385rHd2fs84PN/9dCiwx92LgIRj9nz2i17f+g3j8Hz9w8HAHs8fn3/ze9DtYsI7//7818AeB/LPx1/86vudzTg7u/f9Etjj7mVtZzgQkDjFS+/uP+kOwTgzVqjnp63/+jHodjHhHSsigfyWHZDzH/8X49BY+5eADFG4NgLukh2Qwe4gTHjG2r9uCchg28eEZ6z9S0DCDQhI58favwRkiMK1EXAXAtL5sfYvARn5Y+1fAhJuQEA6P9b+JSBDFK6NgLsQkM6PtX8JyMgfa/8SkHADAtL5sfYvARmicG0E3IWAdH6s/UtARv5Y+5eAhBsQkM6PtX8JyBCFayPgLgSk82PtXwIy8sfavwQk3ICAdH6s/UtAhihcGwF3ISCdH2v/EpCRP9b+JSDhBgSk82PtXwIyROHaCLgLAen8WPuXgIz8sfYvAQk3ICCdH2v/EpAhCtdGwF0ISOfH2r8EZOSPtX8JSLgBAen8WPuXgAxRuDYC7kJAOj/W/iUgI3+s/UtAwg0ISOfH2r8EZIjCtRFwFwLS+bH2LwEZ+WPtXwISbkBAOj/W/iUgQxSujYC7EJDOj7V/CcjIH2v/EpBwAwLS+bH2LwEZonBtBNyFgHR+rP1LQEb+WPuXgIQbEJDOj7V/CcgQhWsj4C4EpPNj7V8CMvLH2r8EJNyAgHR+rP1LQIYoXBsBdyEgnR9r/xKQkT/W/iUg4QYEpPNj7V8CMkTh2gi4ixWQfcb8JegdhDn7mbbiA999xw0B2XHQSs1bvyfodjJnPyPnv0lAwhWsgGRdd3b6jf8bARkqAjIyEZDODgEZPUNAwi0ISOeHgLSBgIxMBKSzQ0BGzxCQcAsC0vkhIG0gICMTAensEJDRMwQk3IKAdH4ISBsIyMhEQDo7BGT0DAEJtyAgnR8C0gYCMjIRkM4OARk9Q0DCLQhI54eAtIGAjEwEpLNDQEbPEJBwCwLS+SEgbSAgIxMB6ewQkNEzBCTcgoB0fghIGwjIyERAOjsEZPQMAQm3ICCdHwLSBgIyMhGQzg4BGT1DQMItCEjnh4C0gYCMTASks0NARs8QkHALAtL5ISBtICAjEwHp7BCQ0TMEJNzCsYBc/5GmL35Dkx4288hGzVrr7HoyZ8x1Kn3BUA1fF/z83HPf8n+abduQa97QlBXbgl7+bIaAtIGAjEyOBeTa9zRlsbnzmsVmypL3dF9WkMuEazJXq23FWmo/eUfw8083mZt175w/afh9L2tGZpDzwzARH5Dr3tfUwO08+dHNmh3iQv/HZqfu7ttMpdot1PT1wc4/0+zQ9EX/0MSVu4OcF54hIOEWTgXkrKGXyFOggGJiYs0UUIG4wirTbJyGrQx9XZm7/E1NWbkz6Hm5Z9adHeQpcp1uD2V9Xv9PXVPHqwJm+3JP3AXjdG+YfwcRkDYQkJHJmYDcrls7FFdMAWuRMWPuwLFJFVXjqpWaGuqdOHOLpjyyRenBzss9ax9Sg8SCumh0qI8y92japDtVtYhXntSSSk2MV3yJDrpyzodBLnt2E9kBuUtjrq2t2AIxx2/nGG8pVeo0T2PXBrt8sNmt2Uve0PQ1wc7LPTs0rHsFeepO11QbAZmx7Cld3bamEmM8Or//a8oIcplwDAEJt3AmIPdoxu0XmzXzKt1iBd36bZoydZSqpnhU9pq/hbZWP75V1zdOUGr75Zob9PyTx1ZAmslY875mrnjvxCxdrw7lvSrebX1IX8/OEJA2EJCRyZmA3KZb2haSp9FczbQ+ztysUTdfokJxpdVyYmiRN/fuy5WQ2F03hRIiNgNy3opH1LhIkspdvlbTraBd/Zx61ius+MojdU+YH6VGdkDu1N3X1JCn4jDdbe23rK0af09flY5PUa3bN2pe0M/JNev+pIvLJKjqwDdCCDv7AZmRMUnVi5VSxfa9VC3NawLyVQISEe+cBKR1WtYGXXmhV0ltl/gDbf0Wjb5rlFo0baHKFzRRrfajNOih7GeGPtKk9LXqUtNcvtF4DZn+J9314Pu+dSLj0fW6+vIrVKtWY1Vr1l9XT33bdz/NDsjblj6nATfernY97lKfqW+GGKv/0ux7+6iwt766378r6PlnMwSkDfkRkId+/FKf7dunzz77r7789lcdCZweLkd2LdSlNXvrsc+P+T4+tn+z/rT2Jf37gO/DMDmqX77+zPd97Ms5n32m/T8dDlwm/5yTgLRm9QOq6/Gq5u3v+MNizcsa3L+/GtS7SJVqtFGjqxZozGr/Zec+8pzu7NtaXm8rXTblCQ2f/bJm+oJhj2bel65OF3dStRpNVbvTWA1+aPuJgLxns8aOnqhO3W9R10FrNfE08TlzeFclJnXW9ctPxNSce3spLb6Ous0P70ITVQFpnZb5uNqV8qrM1c8EFvmdmjJ1pjq0vVhVqjfSBc0HqPfMd/0/A1mbNW76TDUs5tV53Rdr2IynNHZp4KmtzNc1+IYb1bBBU51f73J1uOOv5mfgREBOfHCteve+Re2umqCB87eeNlYzFqzWwDnvKCNzrVqZr0NAIhqcm4DcpekzR6lKSmk1GuV/wJgx7y5VKd9JHQcv1uDRM9WySqq8Ne/VOGt9WPe0LqtdQakJMYpJKqlipSroPPMgfu6qJ9ShYpLSag1Ur+EL1Kd3D51XZbDvdY++gEysrDJlK6hSq35q3qiqEuIrqc2krbm2Lcis/6f6NEhVUuN0W89YhDoEpA3nPCAPv6txF8bleB1DrFIrd9Bdf/lU4cquw+9PUO2klpq/96jv4x8zeyo5vr6mfhjGVD28SfdckPP7ODHJvf6k3wMXyy/nJCCz3tfoWzqqYFpr9VpkvQZtj6YO6qASdQfoqruW65Yhg1W1oFelev7ZRMcujenfTiUKpygmJllpJSuoeIXrNcjEZfq8cTo/uZAqdJyg60em6/KOjVSy/TLNtQIywWsuV01Fq3VTy06dVSY5XkU6LAuEZ87ZqRFXVFF8zugxk/HwZFWNL6jGIT8NHtpEV0Bu06QxfVU6sbo6zgjsx7WPqknZBmrWb74GjVuk7k3LK66w+QX02L80b+kDalyutBLjYuRJK2d+oVRRg9s3KGP9Zt3Sobw8JdrrktsX6oZbblOd8i3UY8F2X0DGFauuMiWqq3bHfmpUrbhiUy9Wn4fyeG0jAYko4mRAxhWIU3xCknmA71FMgRgl17xdtz0SeOBnHuSl53gd9OyRlyohoatuDBwcsJ7C7tc4QSnHn8Leo+lDL1Fickf1XXpibUpf85HvAagVkPFxldXsnjf999usl9TjAq/SOq7M8ynp9PThKucpoxbjPwh6/tkOAWnDOQ/IQ29pVNUENZ69U0fM//321Xta1qeS4gr20Kqv/EcMz1bugNSRn/XV59/roP+jsDn06w/64fvv9X32/CdL15ZKVpsFexX4yvnGyYCMifXKaxYaT3ysCeZkle38kMZnHxXM3J7jaYgduqNzaXlqTdakQNTNHdXdLDyXacDxo4jbdGvHkvJUv1fjj4ffLs1ZEzgCmRCnoq0XaKLvkfFuTbq5hTwpPTTwlNfWbdfgDsUUV2O8JuQIyPmPzVNtj1e173w/x2XPfqIhIGNjPPJYv1A88eYXign3plM06viL6ndqbuaJr5kxf5TKe6qqo7lf+07LfEIXl05Q1ZtMOGZfZtEEVfGWVrNxOY4yrP1I9wWOQMaaByI90wOvV125WBeZBx/Vbn7ztEchfUNAIoo4GZDxKV117f2vasKi/9Pdk+9T62oF5Sk/WMNyrLXzrNewP/SKxgy5XKmeDuq7Mvu83AG5Q0O7lVd89XE51vUTc+prIP3rt6fJ/Zqd67Inz1ZzudKKsx7cOvTGPgLShvwNyMBJr92pCp4LNHpT9jHIA/rk+XTdesXFatGyo3qNXKZ3vsuOy2P6/p0sLXnxEx38+l1lZdyr0ePTlbl5//Hryx2QR794XcsWPq1dxwvysP77ygKN6NtN7Vp3Uu9RS7XxG//1//7xC1owoo86t2quVp2v09h12xTaj9JRfbakswoW66X1gevKT04GZHztcbp70asav+A5DR1xh6oV8qhI+6WakeOoYPpjGzVx0Yu6ucv5Jg5PLCKnBGTWC7qsilclejx16utfgrwGMn1qfxXxtlbvHI9q/WMWoPZFFVdzgibmXLBWzVcdE5B1hob30Wo0BGR8uQEavMD8Qln4gkaOHa8GZRKUXHfqyb8Qsj7U9MWv6t4ZI1XZU0FtpwVeFxUkIOfe00OJJ/3SyZ4gr4HMekmXV0tU2V7PnjkMCUhEkXP2Gkgz6dMHqnhcGbWy/grGqmfUq01tpaWWUqnKdVWhnHmwfsaA3K7bOpVUfI6DBzknWEDe1qmEPI3nnTEgMxbPUs2kFFUfdGJdCfcQkDbke0Ae/Vav311fCaX66s/fW+F1TF89dYMqpNZQ34V/1+svr9f4jqVVuN0D2uX7hCPaNr2h0hp2VaeqVdTmusEa0PkCpXir6fZ/fG8++9SA9AVqUlet+NZ//T++NFTVk85T57FL9fjjSzSxVz21mvqBjhz7Rit6VlOHoQv0xHPPau34jirlqaZRbx3yXc8ZHd6iiXUTVHnoa/n+9LXlnL0G8vFdGtu3nuKSummA9XT0ggfUskYZJReqpHJV6qpM0UTFnSkg1z2tS8pbr60L8m6/YAE5bYCKWQG5JHcw7dTwyysprsKdGpXjkWnGQxNVxVNczcdvz3HZs5+oew2kmZnDuyghrqF6LNxtAu8tDb6qjUqkFVXRinVUoXJ5JcadOSDnDO8sj6erbjzl6HGwgHxZPaqbgMzrXaAEJKLIOQ3IWYNV0heQ2zXxpqYm+HpowFL/S0rmTuyrQt6TA9J6F3bKxdkBuUuj+9RSXIkbNCTHdWbPHwtIsy71qqm4Qj018JQHoeEbAtKG/AnIOCWVqa66dWuraqkUFWtwgxa+/a0v/nRku2Y2SlCVYa8fD7Gjnzyg1kkX6B7fEUorIBuYH/bGGvNq4HOO7tOjl6QqscPD+sw04xkD8tgXWto5RSWue0q+XvU5qN9+D1z2YI5YPLRBwyslqNX9H+f5lPRPz96ksomNNWtHGF9neRbOXUDu1vgbGikusZv6r35PNzRNU2KDGZrgi7hAiOQMSOtd2AkmNrMjYv1b6lXXq+SWD2rW8esMjK2A3KPpd7SXN8EsastOnDdrVA/z4MJc/tFwBlZ0BuTsuy5TYnwDX0CmT75BheNrq8t9H/nOm7dkpqp7cwSk9S5sE5BVcgRk+pQbVST+Ql2akfsNTQQkEApnn8K+VH0X+v9I9/g5i3VJ7aKKKzNAQ1bv0tjr6iqu6NW6ZYVZZ9a8pOvbna/4k45A+n83xFW7R2PXbtOM5R8ofa71WsXCqnbdnzR93S7Nun+xunQapxEmGv9QQK5armZFPSrZ48k8Xyd5NkNA2pA/AelV9YEr9Pzf12r6FVWUUqGP1u0LJNqPa9UjMUmXpL+tLVu2+GfTYl1VPFHdV/9gLuAPyKQmcwJHJC1HtXd+S3lL3qx/HMwjIA++qEHmF1uHxV/44zOow/p5/3+0d9dzGlEr8aSn24MKBGyhy1bqi9Nf6Tnl6FPYdcZrtO9fAnhF94wfpzrFPSrY+mFNz9qoPvUTlNhojqaY8EhfvFxtz085+Qik9cjVRMQlJjzmLt+s2et3a8LNLU1U1tbF4/+pues/0sQJY9S010rdZysgTcQ8mqE6qQkq23WZJmXu0dxHHlO7yklKbblQ0055083ZTVQ8hX3eTbrtQet2fl1jp89V84pJSqg5XmPNbTl33NVKia+nbvO2m9h7V8P6tlRqziOQWc/r0speFW6/TDMzt2i69QeG16xV65LxKtR0hu55zETo8mfUr/uN6n3/NvsBmbVNM5dt0rRHFptfKl5V7PNXTV222ZF/QYOADM2RXSs08LJ+eviDEN4OeexHvTy1p3qMeVb781wzj+jfawbrsmsXaUsITwZFMmcC0jw4HHW5kq0/IO57I2iMYpNKq3zT23XTAv/am7H4YbWokKq4pOIqWLiK6vS4RuWTOqrfYyeuY9bkW3VekvW3Y+MCfw/yQ919czeVTAr83WBPaVXtucJ3H589tJO8xfrpjuMBuUO3dy4tb9MHThOQezTtjg7m90RTXWk9AxL0MuEZAtKG/AnIHE9h/7pBIy7wqPytL/pea3hs/1J19sarZM3matGixfFp2aqbpr/xm7lEsIA8pv1LOyuh8PV65kAeAXngaV1fKEGXrvwuSEAe1t4n7lSbikVVsmp9NW5SV+elePIMyENbzNfzltMtz/8cOCX/OROQOzTk0nKKPf6O8zh5C1dXzW7punuV9YvbPJKddJsqpnnlLVhSaaVaqn3nZkrM+caW1U+qa7WCvgUlNqW7Bqwyp2WaR7QdLlRynP9fQogrVE8tRr6qjLWL1SipiJqMPfH0c/qMm1U8oY36nPIaSGt2a9LoAapcyKuYOI9iY5JUtMEI3RH0smc3kR2QuzXuxsa+f5nCfzvHKj61oqq0G6chSwKL95q/6coGpRXvKayChUqrYscBqlOisi6eHgjIx3dp3KD2KhRn/UJJUtWbrD/0bX4+po5SzRKJ/l8osSkq3miMhq3c4Xv5gbfezBOhn/WqrqiRqHK9nwt6ZDHdeiBy/Bde9sSrbK+/h/1IJAEZmkOvDlH5uOIa8GwIfy/NPOh+uIN5QNI8XbvP+OjcckhvjKis+CL99NdfAydFKacCMqQxD+6nPfSqpjxmvVHOPADM3HnKG9wy1mzSxIc2anaOZy7mrXlHExe9pmmrc4bfLs3NzPVMRNZ2zc3xeaeMOX/O2vD/3cfcQ0DakO8Baf53931NlVC0lx633ihz4HkNLJms7qu/8517Kn9AJtadrK3HH+ge0tujqslbe7y2mNPOGJCHN2tMDa+qj9l86p8N+ulJ9SmSpk4P7jXXaBzZrhkNk/IIyB/1twFl5K0zwfe13cKZgAxx1r5rFowNmmk9ulxvvVs31yPG9ds0/eHXNXXVyYvB3JVvaPyit09afDIyt+cKguAL10mzbqumPviKJi3f5tjTmpEdkKHOTs165BVNXLrN/7fiTrmtTOit/KcmLn7v5Kecsj7UtAdf1kRz+xy/HbN2aO66k7+Heeu2K/1Mv1DO0RCQoTqq33/9/YwPtk9y6Df9eijPw49+Rw/ol99CvuaIla8BGSVDQNqQ/wFp1ob/PKT2SanqsvRzHdPPevHWikptNFLP7ws8kj2wX1uff0U7fS+K9AdkfIE0NbzzSe355ah+/nCRupVMUJ0J7/mi8MxvojmojaNryFOyq+5/73sdOfK9tj0xTnc8uFWHvl2l7slp6rJkn/kqR/TFC/eoScEzH4E8+p/F6pSaog4P/yfP10meS/kakFEwBGT0DAEJtyAgnR8C0oZzHpCHN2l09UQ1m7vnRHAd+0qrehRSUqdHZP3jMcd+2KQHrqmpQvEJKlisqFITUnRey3v1yk/WhQNHIGv11ajeNZQaH6fYmGRV6blQ7wWeQT7ywRTVS2qtBz4NBOQbw1U5+TI9FjioeeynTZpvvfYyJkaxsbFKrtRZ0177VseO/aANk1uphDdZxcuWVuX2d2lIx6K+bQ0ekEf04dT68pa+QU//EOIj6XOEgHR2CMjoGQISbkFAOj8EpA3nPCCNo4cO6HDu3jp6UAcOnXwM79CPn2vvvz/Rlz/nfG4452sgj+qXLz/WJ/t/zXX076gOHTyc4zWO1se5E/CYfvv6U+399Bv9nmtbDn63T3v3+f/w+LHDB3TwTIcWjxzQ77m22w0ISGeHgIyeISDhFgSk80NA2pAfAXl2gr2JBrkRkM4OARk9Q0DCLQhI54eAtOF/LyCP6uMlV6v+tSuV/Zd/cCoC0tkhIKNnCEi4BQHp/BCQNvzvBSRCQUA6OwRk9AwBCbcgIJ0fAtIGAjIyEZDODgEZPUNAwi0ISOeHgLSBgIxMBKSzQ0BGzxCQcAsC0vkhIG0gICMTAensEJDRMwQk3IKAdH4ISBsIyMhEQDo7BGT0DAEJtyAgnR8C0gYCMjIRkM4OARk9Q0DCLQhI54eAtIGAjEwEpLNDQEbPEJBwCwLS+SEgbSAgIxMB6ewQkNEzBCTcgoB0fghIGwjIyERAOjsEZPQMAQm3ICCdHwLSBgIyMhGQzg4BGT1DQMItCEjnh4C0gYCMTASks0NARs8QkHALAtL5ISBtICAjEwHp7BCQ0TMEJNyCgHR+CEgbCMjIREA6OwRk9AwBCbcgIJ0fAtIGAjIyEZDODgEZPUNAwi0ISOeHgLSBgIxMBKSzQ0BGzxCQcAsC0vkhIG0gICMTAensEJDRMwQk3IKAdH4ISBsIyMhEQDo7BGT0DAEJtyAgnR8C0gYCMjIRkM4OARk9Q0DCLayAtNadYD+nTHjG2r8EZIgIyMhkBSQLjbNj7V83BCS3s7Nj7V8CEm5AQDo/1v4lIEMUro2AuxCQzo+1fwnIyB9r/xKQcAMC0vmx9i8BGaJwbQTchYB0fqz9S0BG/lj7l4CEGxCQzo+1fwnIEIVrI+AuBKTzY+1fAjLyx9q/BCTcgIB0fqz9S0CGKFwbAXchIJ0fa/8SkJE/1v4lIOEGX/9w0PfzyDg7X357ILDH3cvaznAgIHEKK2xy3iEYZ8Za0PPT59/8HnS7mPDOvz//NbDHASD/WetSOBCQAAAAUYKABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAthCQAAAAsIWABAAAgC0EJAAAAGwhIAEAAGALAQkAAABbCEgAAADYQkACAADAFgISAAAAtrgmIBmGYRiGYZj/nQmHswpIAAAARB8CEgAAALYQkAAAALCFgAQAAIAtBCQAAABsISABAABgCwEJAAAAWwhIAAAA2EJAAgAAwBYCEgAAALYQkAAAALCFgAQAAIAtBCQAAABsISABAABgCwEJAAAAWwhIAAAA2EJAAgAAwBYCEgAAALYQkAAAALCFgAQAAIAN0v8DsBdn0kfjKjwAAAAASUVORK5CYII="}},"execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import math, re, os, time\nimport datetime\nimport tensorflow as tf\nimport numpy as np\nfrom collections import namedtuple\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\n\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\n\nimport gc\ngc.enable()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU or GPU detection","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Competition data access\nTPUs read data directly from Google Cloud Storage (GCS). This Kaggle utility will copy the dataset to a GCS bucket co-located with the TPU. If you have multiple datasets attached to the notebook, you can pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. Use `!ls /kaggle/input/` to list attached datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path() # you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [512, 512] # At this size, a GPU will run out of memory. Use the TPU.\n                        # For GPU training, please select 224 x 224 px image size.\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102\n\n# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n        \n@tf.function\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n\nrng = [i for i in range(20)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization utilities\ndata -> pixels, nothing of much interest for the machine learning practitioner in this section.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Datasets","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\ndef get_training_dataset(batch_size):\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(12753)\n    dataset = dataset.batch(batch_size, drop_remainder=True) # slighly faster with fixed tensor sizes\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(batch_size, ordered=False, repeated=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    if repeated:\n        dataset = dataset.repeat()\n        dataset = dataset.shuffle(3712)\n    dataset = dataset.batch(batch_size, drop_remainder=repeated) # slighly faster with fixed tensor sizes\n    # dataset = dataset.cache()  # seems this is problematic in the setting of this kernel\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(batch_size, ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(batch_size)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\ndef int_div_round_up(a, b):\n    return (a + b - 1) // b\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\n#VALIDATION_STEPS = int_div_round_up(NUM_VALIDATION_IMAGES, BATCH_SIZE)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset visualizations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at training data\ntraining_dataset = get_training_dataset(batch_size=20)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# peer at test data\ntest_dataset = get_test_dataset(batch_size=20)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(test_batch))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Batch Configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_batch_configuration(batch_size_per_replica, batches_per_update):\n\n    # The number of examples for which the training procedure running on a single replica will compute the gradients in order to accumulate them.\n    BATCH_SIZE_PER_REPLICA = batch_size_per_replica\n\n    # The total number of examples for which the training procedure will compute the gradients in order to accumulate them.\n    # This is also used for validation step.\n    BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\n\n    # Accumulate `BATCHES_PER_UPDATE` of gradients before updating the model's parameters.\n    BATCHES_PER_UPDATE = batches_per_update\n\n    # The number of examples for which the training procedure will update the model's parameters once.\n    # This is the `effective` batch size, which will be used in tf.data.Dataset. \n    UPDATE_SIZE = BATCH_SIZE * BATCHES_PER_UPDATE\n\n    # The number of parameter updates in 1 epoch\n    UPDATES_PER_EPOCH = NUM_TRAINING_IMAGES // UPDATE_SIZE\n\n    # The number of batches for a validation step.\n    VALID_BATCHES_PER_EPOCH = NUM_VALIDATION_IMAGES // BATCH_SIZE\n\n    return BATCH_SIZE_PER_REPLICA, BATCH_SIZE, BATCHES_PER_UPDATE, UPDATE_SIZE, UPDATES_PER_EPOCH, VALID_BATCHES_PER_EPOCH","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Optimized custom training loop\nOptimized by calling the TPU less often and performing more steps per call\n## Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_model(learning_rate_scaling=1):\n\n    with strategy.scope():\n\n        pretrained_model = tf.keras.applications.Xception(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n        # pretrained_model = efn.EfficientNetB7(weights='imagenet', include_top=False ,input_shape=[*IMAGE_SIZE, 3])\n        \n        pretrained_model.trainable = True # False = transfer learning, True = fine-tuning\n\n        model = tf.keras.Sequential([\n            pretrained_model,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n        ])\n        model.summary()\n\n        # Instiate optimizer with learning rate schedule\n        class LRSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n            def __call__(self, step):\n                return learning_rate_scaling * lrfn(epoch=step // UPDATES_PER_EPOCH)\n\n        optimizer = tf.keras.optimizers.Adam(learning_rate=LRSchedule())\n\n        # Instantiate metrics\n        train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n        valid_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n        train_loss = tf.keras.metrics.Sum()\n        valid_loss = tf.keras.metrics.Sum()\n\n        # Loss\n        loss_fn = tf.keras.losses.sparse_categorical_crossentropy\n        \n        return model, loss_fn, optimizer, train_accuracy, train_loss, valid_accuracy, valid_loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training routines","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_routines():\n\n    with strategy.scope():\n\n        def train_step_1_forward_backward(images, labels):\n            \"\"\"\n            The procedure to be run on each replica that computes gradients by feedforward and backpropagation.\n\n            The `images` and `labels` must have batch dimension equal to `BATCH_SIZE_PER_REPLICA`.\n            \"\"\"\n\n            with tf.GradientTape() as tape:\n\n                probabilities = model(images, training=True)\n                per_example_loss = loss_fn(labels, probabilities)\n                loss = tf.math.reduce_sum(per_example_loss) / UPDATE_SIZE\n\n            grads = tape.gradient(loss, model.trainable_variables)\n\n            # update metrics\n            train_accuracy.update_state(labels, probabilities)\n            train_loss.update_state(loss)    \n\n            return grads\n\n\n        def train_step_1_update(images, labels):\n            \"\"\"\n            The procedure to be run on each replica that computes gradients\n            in an accumulated way and updates the model's parameter\n            (once the accumulated gradients on each replica are synced across the replicas by summing them).\n\n            The `images` and `labels` must have batch dimension equal to `BATCHES_PER_UPDATE * BATCH_SIZE_PER_REPLICA`.\n            They are splitted into `BATCHES_PER_UPDATE` parts, and each part (which has batch dimension `BATCH_SIZE_PER_REPLICA`)\n            is sent to `train_step_1_forward_backward()` to compute the loss and gradients, then the gradients are added to `accumulated_grads`.\n\n            *** Implementation detail:\n\n                In order to split `images` and `labels` into smaller portions, the easiest way is to do something like `images[start_idx:end_idx]`\n                with `start_idx = BATCHES_PER_UPDATE * batch_idx` and `end_idx = start_idx + BATCHES_PER_UPDATE`.\n\n                However, this gives the following error:\n\n                    Compilation failure: XLA can't deduce compile time constant output shape for strided slice: [?,512,512,3],\n                    output shape must be a compile-time constant.\n\n                Similar error is thrown for `tf.gather(images, tf.range(start_idx, end_idx))`.\n\n                If we use the trick like `images[:BATCH_SIZE_PER_REPLICA]` at the beginning inside the `for` loop\n                and modify `images` by `images = images[BATCH_SIZE_PER_REPLICA:]` at the end inside the `for` loop,\n                we get another error:\n\n                    \"images\" has shape (256, 512, 512, 3) before the loop, but shape (240, 512, 512, 3) after one iteration.\n                    TensorFlow control flow requires it stays the same or be more specific.\n\n                The solution given here is to do the following trick:\n\n                    for batch_idx in tf.range(BATCHES_PER_UPDATE):\n\n                        ...\n\n                        small_images = images[:BATCH_SIZE_PER_REPLICA]\n\n                        ...\n\n                        tf.concat([images[BATCH_SIZE_PER_REPLICA:], small_images], axis=0)\n\n                The idea is to take the first `BATCHES_PER_UPDATE` examples from `images` and pass them to `train_step_1_forward_backward`, and then\n                move this portion to the end of `images`, so `images` always has the same shape, although the content is modified.\n\n            Args:\n\n                images: tf.Tensor with shape [BATCHES_PER_UPDATE * BATCH_SIZE_PER_REPLICA , height, width, depth]\n\n                labels: tf.Tensor with shape [BATCHES_PER_UPDATE * BATCH_SIZE_PER_REPLICA]\n            \"\"\"\n\n            accumulated_grads = [tf.zeros_like(var, dtype=tf.float32) for var in model.trainable_variables]\n\n            for batch_idx in tf.range(BATCHES_PER_UPDATE):\n\n                # This is not working. (Error: output shape must be a compile-time constant.)\n                # start_idx = BATCHES_PER_UPDATE * batch_idx\n                # end_idx = start_idx + BATCHES_PER_UPDATE\n                # small_images = images[start_idx:end_idx]\n                # small_labels = labels[start_idx:end_idx]\n\n                # Take the 1st `BATCH_SIZE_PER_REPLICA` examples.\n                small_images = images[:BATCH_SIZE_PER_REPLICA]\n                small_labels = labels[:BATCH_SIZE_PER_REPLICA]      \n\n                grads = train_step_1_forward_backward(small_images, small_labels)\n\n                accumulated_grads = [x + y for x, y in zip(accumulated_grads, grads)]\n\n                # Move the leading part to the end, so the shape is not changed.\n                images = tf.concat([images[BATCH_SIZE_PER_REPLICA:], small_images], axis=0)\n                labels = tf.concat([labels[BATCH_SIZE_PER_REPLICA:], small_labels], axis=0)\n\n            # Update the model's parameters.\n            optimizer.apply_gradients(zip(accumulated_grads, model.trainable_variables))\n\n\n        @tf.function\n        def train_step_1_epoch(data_iter):\n\n            for _ in tf.range(UPDATES_PER_EPOCH):  \n                strategy.experimental_run_v2(train_step_1_update, next(data_iter))\n                \n                \n        @tf.function\n        def valid_step(data_iter):\n\n            def valid_step_fn(images, labels):\n\n                probabilities = model(images, training=False)\n                per_example_loss = loss_fn(labels, probabilities)\n                loss = tf.math.reduce_sum(per_example_loss) / BATCH_SIZE\n\n                # update metrics\n                valid_accuracy.update_state(labels, probabilities)\n                valid_loss.update_state(loss)\n\n            for _ in tf.range(VALID_BATCHES_PER_EPOCH):\n                strategy.experimental_run_v2(valid_step_fn, next(data_iter))                \n                \n    return train_step_1_epoch, valid_step","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training without gradient accumulation\n\n### BATCH_SIZE_PER_REPLICA = 16\n### BATCHES_PER_UPDATE = 1\n\n* Effective batch size = 16 \\* 8 \\* 8 = 128\n* The original setting","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE_PER_REPLICA, BATCH_SIZE, BATCHES_PER_UPDATE, UPDATE_SIZE, UPDATES_PER_EPOCH, VALID_BATCHES_PER_EPOCH = set_batch_configuration(batch_size_per_replica=16, batches_per_update=1)\nmodel, loss_fn, optimizer, train_accuracy, train_loss, valid_accuracy, valid_loss = set_model(learning_rate_scaling=8)\ntrain_step_1_epoch, valid_step = set_routines()\n\nprint(\"BATCH_SIZE_PER_REPLICA: {}\".format(BATCH_SIZE_PER_REPLICA))\nprint(\"BATCH_SIZE: {}\".format(BATCH_SIZE))\nprint(\"BATCHES_PER_UPDATE: {}\".format(BATCHES_PER_UPDATE))\nprint(\"UPDATE_SIZE: {}\".format(UPDATE_SIZE))\nprint(\"UPDATES_PER_EPOCH: {}\".format(UPDATES_PER_EPOCH))\nprint(\"VALID_BATCHES_PER_EPOCH: {}\".format(VALID_BATCHES_PER_EPOCH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = get_training_dataset(batch_size=UPDATE_SIZE)\ntrain_dist_ds = strategy.experimental_distribute_dataset(train_ds)\ntrain_data_iter = iter(train_dist_ds)\n\nvalid_ds = get_validation_dataset(batch_size=BATCH_SIZE, repeated=True)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_ds)\nvalid_data_iter = iter(valid_dist_ds)\n\nEPOCHS = 20\nfor epoch_idx in range(EPOCHS):\n    \n    s = datetime.datetime.now()\n    \n    train_step_1_epoch(train_data_iter)\n    \n    loss = train_loss.result() / UPDATES_PER_EPOCH\n    acc = train_accuracy.result()\n    \n    print(\"epoch: {}\".format(epoch_idx + 1))\n\n    print(\"train loss: {}\".format(loss))\n    print(\"train accuracy: {}\".format(acc))\n\n    train_loss.reset_states()\n    train_accuracy.reset_states()\n        \n    e = datetime.datetime.now()\n    print(\"elapsed: {}\".format((e-s).total_seconds()))\n    \n    valid_step(valid_data_iter)\n    \n    val_loss = valid_loss.result() / VALID_BATCHES_PER_EPOCH\n    val_acc = valid_accuracy.result()    \n    \n    print(\"valid loss: {}\".format(val_loss))\n    print(\"valid accuracy: {}\".format(val_acc))    \n    \n    valid_loss.reset_states()\n    valid_accuracy.reset_states()    \n    \n    print(\"-\" * 80)\n    \ndel optimizer\ndel model\ndel train_step_1_epoch\ngc.collect()\ntf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training with gradient accumulation\n\n### BATCH_SIZE_PER_REPLICA = 16\n### BATCHES_PER_UPDATE = 8\n\n* Effective batch size = 16 \\* 8 \\* 8 = 1024\n* BATCH_SIZE_PER_REPLICA = 16 with BATCHES_PER_UPDATE = 16 --> Not enough TPU memory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE_PER_REPLICA, BATCH_SIZE, BATCHES_PER_UPDATE, UPDATE_SIZE, UPDATES_PER_EPOCH, VALID_BATCHES_PER_EPOCH = set_batch_configuration(batch_size_per_replica=16, batches_per_update=8)\nmodel, loss_fn, optimizer, train_accuracy, train_loss, valid_accuracy, valid_loss = set_model(learning_rate_scaling=1)\ntrain_step_1_epoch, valid_step = set_routines()\n\nprint(\"BATCH_SIZE_PER_REPLICA: {}\".format(BATCH_SIZE_PER_REPLICA))\nprint(\"BATCH_SIZE: {}\".format(BATCH_SIZE))\nprint(\"BATCHES_PER_UPDATE: {}\".format(BATCHES_PER_UPDATE))\nprint(\"UPDATE_SIZE: {}\".format(UPDATE_SIZE))\nprint(\"UPDATES_PER_EPOCH: {}\".format(UPDATES_PER_EPOCH))\nprint(\"VALID_BATCHES_PER_EPOCH: {}\".format(VALID_BATCHES_PER_EPOCH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = get_training_dataset(batch_size=UPDATE_SIZE)\ntrain_dist_ds = strategy.experimental_distribute_dataset(train_ds)\ntrain_data_iter = iter(train_dist_ds)\n\nvalid_ds = get_validation_dataset(batch_size=BATCH_SIZE, repeated=True)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_ds)\nvalid_data_iter = iter(valid_dist_ds)\n\nEPOCHS = 20\nfor epoch_idx in range(EPOCHS):\n    \n    s = datetime.datetime.now()\n    \n    train_step_1_epoch(train_data_iter)\n    \n    loss = train_loss.result() / UPDATES_PER_EPOCH\n    acc = train_accuracy.result()\n    \n    print(\"epoch: {}\".format(epoch_idx + 1))\n\n    print(\"train loss: {}\".format(loss))\n    print(\"train accuracy: {}\".format(acc))\n\n    train_loss.reset_states()\n    train_accuracy.reset_states()\n        \n    e = datetime.datetime.now()\n    print(\"elapsed: {}\".format((e-s).total_seconds()))\n    \n    valid_step(valid_data_iter)\n    \n    val_loss = valid_loss.result() / VALID_BATCHES_PER_EPOCH\n    val_acc = valid_accuracy.result()    \n    \n    print(\"valid loss: {}\".format(val_loss))\n    print(\"valid accuracy: {}\".format(val_acc))    \n    \n    valid_loss.reset_states()\n    valid_accuracy.reset_states()    \n    \n    print(\"-\" * 80)\n    \ndel optimizer\ndel model\ndel train_step_1_epoch\ngc.collect()\ntf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training with gradient accumulation with even higer effective batch size\n\n### BATCH_SIZE_PER_REPLICA = 8\n### BATCHES_PER_UPDATE = 32\n\n* Effective batch size = 8 \\* 32 \\* 8 = 2048\n* BATCH_SIZE_PER_REPLICA = 8 with BATCHES_PER_UPDATE = 64 --> Not enough TPU memory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE_PER_REPLICA, BATCH_SIZE, BATCHES_PER_UPDATE, UPDATE_SIZE, UPDATES_PER_EPOCH, VALID_BATCHES_PER_EPOCH = set_batch_configuration(batch_size_per_replica=8, batches_per_update=32)\nmodel, loss_fn, optimizer, train_accuracy, train_loss, valid_accuracy, valid_loss = set_model(learning_rate_scaling=1)\ntrain_step_1_epoch, valid_step = set_routines()\n\nprint(\"BATCH_SIZE_PER_REPLICA: {}\".format(BATCH_SIZE_PER_REPLICA))\nprint(\"BATCH_SIZE: {}\".format(BATCH_SIZE))\nprint(\"BATCHES_PER_UPDATE: {}\".format(BATCHES_PER_UPDATE))\nprint(\"UPDATE_SIZE: {}\".format(UPDATE_SIZE))\nprint(\"UPDATES_PER_EPOCH: {}\".format(UPDATES_PER_EPOCH))\nprint(\"VALID_BATCHES_PER_EPOCH: {}\".format(VALID_BATCHES_PER_EPOCH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = get_training_dataset(batch_size=UPDATE_SIZE)\ntrain_dist_ds = strategy.experimental_distribute_dataset(train_ds)\ntrain_data_iter = iter(train_dist_ds)\n\nvalid_ds = get_validation_dataset(batch_size=BATCH_SIZE, repeated=True)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_ds)\nvalid_data_iter = iter(valid_dist_ds)\n\nEPOCHS = 20\nfor epoch_idx in range(EPOCHS):\n    \n    s = datetime.datetime.now()\n    \n    train_step_1_epoch(train_data_iter)\n    \n    loss = train_loss.result() / UPDATES_PER_EPOCH\n    acc = train_accuracy.result()\n    \n    print(\"epoch: {}\".format(epoch_idx + 1))\n\n    print(\"train loss: {}\".format(loss))\n    print(\"train accuracy: {}\".format(acc))\n\n    train_loss.reset_states()\n    train_accuracy.reset_states()\n        \n    e = datetime.datetime.now()\n    print(\"elapsed: {}\".format((e-s).total_seconds()))\n    \n    valid_step(valid_data_iter)\n    \n    val_loss = valid_loss.result() / VALID_BATCHES_PER_EPOCH\n    val_acc = valid_accuracy.result()    \n    \n    print(\"valid loss: {}\".format(val_loss))\n    print(\"valid accuracy: {}\".format(val_acc))    \n    \n    valid_loss.reset_states()\n    valid_accuracy.reset_states()    \n    \n    print(\"-\" * 80)\n    \ndel optimizer\ndel model\ndel train_step_1_epoch\ngc.collect()\ntf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training with gradient accumulation with even higer effective batch size + larger learning rate\n\n### BATCH_SIZE_PER_REPLICA = 8\n### BATCHES_PER_UPDATE = 32\n\n* Effective batch size = 8 \\* 32 \\* 8 = 2048\n* scale the learning rate by 16 times of the original learning rate (because the effective batch size is 16 times larger)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE_PER_REPLICA, BATCH_SIZE, BATCHES_PER_UPDATE, UPDATE_SIZE, UPDATES_PER_EPOCH, VALID_BATCHES_PER_EPOCH = set_batch_configuration(batch_size_per_replica=8, batches_per_update=32)\nmodel, loss_fn, optimizer, train_accuracy, train_loss, valid_accuracy, valid_loss = set_model(learning_rate_scaling=16)\ntrain_step_1_epoch, valid_step = set_routines()\n\nprint(\"BATCH_SIZE_PER_REPLICA: {}\".format(BATCH_SIZE_PER_REPLICA))\nprint(\"BATCH_SIZE: {}\".format(BATCH_SIZE))\nprint(\"BATCHES_PER_UPDATE: {}\".format(BATCHES_PER_UPDATE))\nprint(\"UPDATE_SIZE: {}\".format(UPDATE_SIZE))\nprint(\"UPDATES_PER_EPOCH: {}\".format(UPDATES_PER_EPOCH))\nprint(\"VALID_BATCHES_PER_EPOCH: {}\".format(VALID_BATCHES_PER_EPOCH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_ds = get_training_dataset(batch_size=UPDATE_SIZE)\ntrain_dist_ds = strategy.experimental_distribute_dataset(train_ds)\ntrain_data_iter = iter(train_dist_ds)\n\nvalid_ds = get_validation_dataset(batch_size=BATCH_SIZE, repeated=True)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_ds)\nvalid_data_iter = iter(valid_dist_ds)\n\nEPOCHS = 20\nfor epoch_idx in range(EPOCHS):\n    \n    s = datetime.datetime.now()\n    \n    train_step_1_epoch(train_data_iter)\n    \n    loss = train_loss.result() / UPDATES_PER_EPOCH\n    acc = train_accuracy.result()\n    \n    print(\"epoch: {}\".format(epoch_idx + 1))\n\n    print(\"train loss: {}\".format(loss))\n    print(\"train accuracy: {}\".format(acc))\n\n    train_loss.reset_states()\n    train_accuracy.reset_states()\n        \n    e = datetime.datetime.now()\n    print(\"elapsed: {}\".format((e-s).total_seconds()))\n    \n    valid_step(valid_data_iter)\n    \n    val_loss = valid_loss.result() / VALID_BATCHES_PER_EPOCH\n    val_acc = valid_accuracy.result()    \n    \n    print(\"valid loss: {}\".format(val_loss))\n    print(\"valid accuracy: {}\".format(val_acc))    \n    \n    valid_loss.reset_states()\n    valid_accuracy.reset_states()    \n    \n    print(\"-\" * 80)\n    \ndel optimizer\ndel model\ndel train_step_1_epoch\ngc.collect()\ntf.keras.backend.clear_session()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}