{"cells":[{"metadata":{},"cell_type":"markdown","source":"# File Upload/Download to GitHub from Kaggle Notebooks\n![githubkaggle.jpeg](attachment:githubkaggle.jpeg)\nComputer vision challenge such as this, with 100GB sized dataset and a state of the art base model, will hit many developers straight out of the park. Relieing on local bandwidth and computing power will no longer an option. and cloud is the way to go. However cloud charges can be an inhibiting factor to rapidly prototype multiple models on this kind of challenge. \n\nIt will be a blessing to be able to use both kaggle and colab runtimes along with GitHub.\nThe following solution can be used in current GLD or any other future projects and can certainly add a feather to ones *data engineering* capabilities. This can help in overcoming the limitations of Kaggle/Colab by using GitHub as an intermediate storage. \n\n> This notebook is an experiment in trying GitHub as one more tool to be considered in data pre/processing activites along with GCP Cloud, and not as a complete alternative.\n\n# Things required to understand this notebook\n* Python/Conversions btwn(Numpy, Pandas, strings, tensors)\n* GitHub API/Account\n* Basic level of Tensorflow usage\n* Kaggle/Colab directory structure\n* Matplotlib to view images for sanity check\n* UNIX/bash commands \n\n# GitHub Vs GCP\n* Data preprocessing activities involves reading the dataset multiple times. This can generate huge data transfer charges from the cloud/GCP. Using GitHub incurs no data transfer and storage chages.\n* Training a model involves reading/writing checkpoints, 100's of EPOCHs over dataset, which can again generate considerable cloud storage and data transfer bills.\n* However GCP can be an alternative for solutions beyond GitHub's 100GB/repo or 100MB/file limit.\n* Competition rules should allow one to upload dataset to public repositories.\n\n# What does this notebook do?\nResize each image to size ~(224,320) preserving the aspect ratio, before uploading to GitHub. After resizing the images, the resultant dataset size is ~15GB, which can easily be accomadated in either kaggle/colab or can be downloaded locally. Either plain images or tar of multiple files can be uploaded\n\n# Motivation: create small dataset\nGLD dataset size, 99GB, has been a huge limiting factor in many ways to compete in this image recognition challenge, in a way\n* It occupies the whole available user disk space *on kaggle*, which means, no space left out for data manipulation or intermediate data generation related activites.(5GB working directory data limit anyway)\n* Not enough Kaggle computing time to validate various ensembling or stacking models\n* SSH is not allowed from Kaggle notebook to Colab notebook server(worked briefly and stopped working from next day)\n* Can't use *google colab* to train custom models, as the GLD competition dataset size will easily cross the available limits(77GB for CPU/TPU, 37GB for GPU)\n* Cannnot edit/remove files in input Dataset. Can only remove entire input dataset, but not individual files, to make space for new edited files.\n\n\n*Note* Internet switch should be \"on\" to use this notebook","attachments":{"githubkaggle.jpeg":{"image/jpeg":"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCADNAPYDASIAAhEBAxEB/8QAHQAAAgIDAQEBAAAAAAAAAAAABQYEBwADCAIJAf/EAFcQAAAFAQUDBggJCQQIBgMBAAECAwQFAAYREhMUByIjFSEkMjNDCDE0QlJTYmNBRFFyc4KDkqIWJVRhk7LC0vA1cYGjZHSRs8PT4vIJF0WElOOhtNHz/8QAHAEAAgMBAQEBAAAAAAAAAAAABAUCAwYACAEH/8QANREAAQMDAwIFAwIFBAMAAAAAAQACAwQREgUTISIxBhQjQVEyM2FCcRYkUlNiFYGR8Aeh0f/aAAwDAQACEQMRAD8AeUdM2c8mNoxfo+Zxlu9VPvdb0aMH8mQdfY1Akk+kr6ngL9z+H8Ne2ybZtn9J8o/rcrz6969H4IqwX0rZDv8AULUyWeIHAbUsNl/0Vr5P23rKJMDuaFnf0LpGbke2nxy76TptT5Pl1Px/sKVAe6lyi502Qv2Ky3zOrTOyR07ZBsFIp4/hIZodpiMlQbiCDkKFLLtqjSto20a21P8AuaByj7Uj0b9t7ygfKzZ7iopaV8iLY22pQc/o/wD21AcJ6Vvp+BkZyi336hs5FzwK0yT7TOaKjhk3E0jpetJ7xRz/AGY5z+kZnbUbYWEBy108a24+RwclbtVaW3Lts2c/5yNOVjbVRzZu9bSbrpudwf8AorRSvnjhzjRWoZ7HpqspuHkoOS00m28no9CKNnLlD1/8lJ7na3GW4tbKtWsYuguwRTRzllicVLH+E1M7Z9yZkaXIz+xye7p7nPsM3FOhZ6aMWmQ5MhV5xtZpe0i7BZu80bPyjcP1yF87B1sFNskg2cyOpa56/fe7pPczLWNkUHTqTyF0PlV9jzDVGf8AhCWJjHLGMk3PHcI5yK2b99P5xKTVTKqXiNiFqmTRybiKzrttylqXORkcPO96rjwkwfhpNtvHmQFcCgIo++R4eb8+gk3tgslKWus5AwTpeW5fWzlmaKPDatcBjZn7/wB2rmslYSSnYkbRSJJKVjlk10ixTQSigoJDCIGFZS5JIC3Dd1jDeNxB8daTSaOrqcKd7Oe6A1zV6PRaE1lU6zB8Akk/AABJP4AuqIiXKjLywoh3NfskZKXyGpCuB0+WsjkK5XFJvfwU5bTyQ6sNZy1kXAN4kjtJdRZFBRQ5b01zkKInUMYcQlAL7rgv8QBTds9sNZtFCJWcRyb6VlEFJEpHShyN2LRITFFdXBvnExiiUhCiW8Q5x5wrVx6fL5gxC1x7+yxUvjXSzosOuAnamxwFupxfw0Bvyfj27mwBSTGspeZc/m9kIL6NRbJEPQqzIWHOhDBaOakUUUYJmo7WRVD2DYz+1u5tMyAwrSUTjGUZEgtJlUJHu48VgbrroAUyjNVJY5zEUEpiXCBxAcZAu5+Y9IyjKYj2thD2ej1XM+wUUdJLqnK2aMEwEpzqiS45gMYpilIUQERDxheFRm02u8wyOMjHvf2Ss+PtOZSvqBk1zHiMsLesSOti3H3LgQQRcEG97XKoeYsTEWicr5CyIIOEc7JSAivC6yWP5+M5qFxKTcXWpgmvkGYitk9mslg7PBV1L2Ss9EySFnU4CMj1Z5mKEY9iTLJIOTNk8INFU1TnMQwEMGEQOIDjKF3PzJ1mLCRNhzkkpFs7XcSrwWzCIQANU7WvABTEwgIEIAmLeYQHnEAAB57nzBNG4Rd7+6g/xfoklDPqM7jG6F2MgcCHNcbWFhe5dcY43yuLXQlxNtnca+EEUnC6DLKZIrB8ZwcIhz1RM68cLukHMcwhGErkps1pJ4BPzfjJv4Me4Xr110wb2TZJvIpJhZFMyZyldtkk3zwEDc1wGVBdIt4YgvMQgh8gjSXtW2Z2StFZ2VWlLNRiaiLJFdJzHOFFGj9AypkzFOmqXGUwCQbxxf7QEBpxQyGHi91ip9Uikmja+GRgkNmlzLAm1+bXLb2/UB8d1zo2IlZORQbru05BYeM8WzM1NXNP2hPn4y0CjtnMpMah0wOdbXrZw54dqrgLXVJrK7NrQvLHOIHZrAClah1ySlymk/TWa5RkSAB8p7hVLvlG8CkvuHdC+jbiMsg/i10oSCslKljkDOhbtySTVfLSKKhspRRdQmICgJrhAL7vhG4BKmn3vrSh/iKmZJPGyKQ7B67NBtdod7Ek9JB4B/54XBVoI+fhJxeBc2WHRZPGeLfpW9wMFAWkO9cLyrsW57hklB7D0EU67fWsBZS0to7NKlZu3EJMEXeggocM8FiZgGQE5QuMOJMLjAULymAbgHmBwY2YsSionZdrZayJJhijn8mdPBUFFEgUFEy5lsOaJRAOdPDiC6+g3RsecijHa7TFkc8AdIx7cwWtv08c+3z2+r8L5xSqzZo5Eqq4gHf/AEv9Yqyr18IUkNYOTj7WQLEpIG1iOcVJZMFVUHaJjlVTMYAADdYDAYAC8pi8wDeAZVG2tBQzwajTR1dM7KN4Dmn5B5CuuefOfirb3Oct3vnbn3Khokctf/ccb7+7Ri2cH2A9xnJ53uutjwfgoIdP4zpvJ++rz5DOvSH3GIqzfft6MQko25S9+443ZcPf9ClJmvpfp6arMKalz/q+WtXT/QqH8MTlER2pc9J7Htvx0cy22p1Llt6ypDRPTBqa8uXWpz21It66z07zUSJYm+jZH+j0HRTcufjP0K1T5VT/APxoIzfaVt5Uh/w6YQsfgndLH6KybfdG0tCpO0ThwFzjmWqA8tBqpKgkqv0mmlLSo1kfRdY86VGrttSv0hFRHO+f6FCnhOjL+Udjk+83CYfvVpy+jL9KXz69/wCs5+Q4W/yifvU329tioz3H7aW4qKjbHuejZCGo4yy3a5vz63M7XOdTpXVpl36DdbOR6GT5u/8AUPW6ShpLUPm0nJof2kpyasij8ROThYy+l1qUrQp8mdK1K6C+d2KPzPPPTWN7JEG8vp/tpwf7QnMFG6WTdP8AI4mStko/OJj/AB4/qVVdvFHNsIT8r4OTY65uimisj3mVjxHUJ8zz6AyUq2nJtjGOXWQg4WTzllluH1/w0SbQfLvKulk8hBgiotwVicJX1eD9ymEcbKf1FW+fznppKspOzlhdpEUId+snxkUe1SJun/frvnYTaGSWnWbJVqsCK6DgQUOO4A5ZjAIe0Yph/wBlcOHj3NqUXs1quwRTVR/5ZPaPgrtHYHESTZtDWztDaFoziSNlcsH71uiosYqRkwEhDGA5wAd28oDz3hRTWvqq6GSMdiL2+LpBr9XTaZotV554aC1wBcQBcg2Fz7lAtobtFKwNiGyqWYC5XRP1FEztQAEf1X1YJl1oY1pgZlBVdlYWJRZgTmIVU4MDhfd5uacQN+oTUuTlmHVuNlcFHwDuIcPY4jpRdmq/QTc3lVUOUCpHOBzcxr90Bo5YJ9HWqhk5MzgStHUF+TMyqUonFguich2yypSgIgkYEUS47rgEpr7uatWxpE0gI7jj88LzdSVEUvhHw9VxvDo6aSAy2IOAwc27rdrFwJv2HJ4C0WQRkIixNkFFHQLqsrVtDIq34lBVFMgrCc3nGE5AG/5BLVpmFVLbfbZFECiLKzTRBAPEBSm0ZzAH94mMH+I0AbR0YK1l2bFw3XhLMOhlJB4gFzU6pClBBsmfmBRU4piJsIjcCl/mjcURforWsJtAzTKx0tFcjyzoqZjgzckMRRFVUCgIgmbKSLiuuAQNfdzVM3EOyfrLSg69pqtdl8RQ80bKunJePpIZE5jn37FrXPaC7sLHngrxtCVXbsrFyRClKqhaZsqjhHnKYwAIh+AlR5gxU9tjZG8FEoSzbgzQADdxCisYTAA/IZQw/qwh8gURlwj56Zs0nEujSMJZh0aYkn5CiZA6wYclqmbxHOYSCAgURuA9/mjcLeLjN2khrXWeR5RlYtBVq9iymAq0jHKYynMiA9Y5QVULd4x3BABwjVdKdmKOGQ2fY/8Af+/Cq16ln1WvrdeoWmSlZVUrziLh7Ym2kc0D6g0uHa9y0gdlVbGxr+Utk5CzwZLhwGEqSigFKph5xETDzAABfRacsDa60DBGx8Pb5i/RdrZh4Yk6iHHEeZMgCbxXjzAPy01qWdhEZ8ZplbViwwnAU0XahmbtELgASmTUADAYLrxw3hf4hHx1umplzFWstdbwjh6lGOoKPiWsssmqgR06xsUjlTMoBROI4FBG7xlKI+LnoukjMbAHDlajXvFTJqqki0qWOXdeGuA6i1pBJf0u4AsAbjuRz7EWxSdsA2TclNEVnJJxcW6B1cshjgdoAFMcAHCAiHjuG6/xDRo0QIN3aFjoaJK8kUDtVjpy+odAkoUSHBFFRJuUVBKJgABERvNzBfdUOTdxtmJ/Zgxl5liC0LMquJEpXCZxZkFVuIZuARwDcQ3MNwhcN4V+rRNj0rQJz/8A5ktnQA5K4O1ZmM4WNca+5MpCiJjDdcF9wXjziAUWbElYNktBU6pq8VRXbLXOYOHxi42WA8uBPHIOJHx3VY7Z+BDWcm9n716nBRhRYp3gKb1q7KYVVAXu5gOYxzGASiJRALgHcEAM7G7KrMGTXaI6k15e0E3GmmyGcqJNmjEmYZsRRdwopiObGQDYCkvHEULxvG6PtFWUhbMSKU+3GPkrV2gTlysFOZRqgmC92MvmifUmuKNw3JgIgACW9qhUIpHZUz2eKFcoyCtllH/LeAchNqUx3OQPPd1i4czx49y74arw/Unem1k7vDlEyod5Z0hbHw2xxJLW4gh2Jc0Ai4sL+y5y8Ia2UGVrB2Os+/TfNIEFCrOhIJSuHShhMocgGuECAGEhbwARAl4gF9wZVIzzF6+nnjYygp5Bx+Hx1lCGUkre6fQwaZSx0dMLMYA0D8AWC7teL8pttT7mlV4+1Of0Xj9j9F/NTZalDoy/RshD+Q9Aljtv/kZfGrzWz+4vS9L6jEtkJVhWSY6ZtqdTS2tHdG1LZqh75H1VTIpdy26Nqf6JRj37jF08G6nxzMaXIobMT7nU6VtWlbTuagPOfo3H7uhoIWIJlKxaZWSc/GW1L2Y50y7WjD9pqnPSaDrL9J0v0lMI0bHBZQ1mPYdK8orTyd0bU0bbfo2mz6nO2LZtfpuBwa+iqfG9XYKrHP8AaVTHKf5tX7fo/YrIrdlj3tz0aMT0U2/RuP31B4p05bZ7bsKexybkaAw23pSttMaXpTnt8lPJWRW4mUTdJj9E2+eliYkm0nZtd1ql9C4RzkfWbm6ej1sHWlknzqTjPfZ3eexuebQF+76Tpm3QEHHbdTMSSITEdM9MYPoVEn1qvVorSuYrv0H+Wtk52V1+oof8FZMR1pLITaDbsNRqGaOT3qR/T9Lc8+jzlCSk22qbdI0/2XU/k3NylU9o411ntbTNUMhBbJzlluIljJ+7TSN75eEuOESYoC1FnCyYNwbZCDdFTg5PoE/gPVt7Ptts3srjmjBO0tpIeNklnL0goySgtSioW8CgkUQAgl+HmrmGKYzfJr7855DGPzEc5ZHtce7l05ozElGZDWT/ADtoMvORzvTJh3PwVbLTSQm0DiC74VjJIqmLGpaCPyr1t14S21lsCjN5buZWbvCmIdqD1YCmRMFwgI4rhAQG4ap5rtUtbZCQCcsxMO4sQ8SrdYyZw9O4Sjf5tSmC/wCUzZeCk3XHyeCt3ft/NwUqrWVbalBq6k112P8AX4aY6dJK84zOJKUanRwRNvAwAH8LoOxm1XaTb54Clu7RvXqa6JBZHOudUd3xnvMI1fVm7TzEG1bDGLZz1bxFSUFAwf4hXJtjF3MY5Y6bgIOEcn7n89XkS0zly202p8n9SjQ9a2bdyjPKhRQQbODwLK0bd2tnH5WLiQkHC6C6I3CooY1yhT+mA0jJOkYlRBRFMxXDdLK5lu7638NLExPyOpQbOuOxb6jJRWx9qcheJUmYU01iV7TQTnXziDxNHk33SqxS5nzSb56gGvtk43KGqoY8cGiwTAptxt7FNtKwte9apAIEwHXUObAHWEoAN1UvtmtvP2ms4s7fyb8XQLJgiIKCuqqoU/ixj1abZVT/AEX1n0lLFv4NtGOUIxrx+hJrLPPWulSYj4PmdStNRzOd3KytZSMYbgJfs34WfhFoRSZZS01ocSQCgRMj9UBOCZ8AgG9zGq023hL26WTlGMzam0x1W6ChLiSSgppL3FMU5rzbyXWrniVhuQsjS8CKyeDxs1RVXz1FT+l1MHsYKy0JOU22pjJzQL6LjcHiLb9NnuKRzMjHZM9prZOVJNw3dzZ5lQw56S4plFNMTkxnIOHzy9WtDTblaxnBjZpe0UiECY5yCxByfIxG58R078N9/wAN1U7aTlKMhNLGcDUdtQ1zFSUnG/6v3NfGPQrwnaZRVlFUJJoZNdN2lmgJG+InP8hvhrKL2OGPgoNGLZJtnSQ8UGr5XLBC/wBGsqeCpXbb7UOWyEZqV33B7bvPax0CZsZFs200m21HfI/wUztj/GXPb1CngkuByY2QXQX7bO+eXfrytn17a9JRv2vTQqNRcOW+obNug8TO9Zm7vULXlE/Jjn7FPJRqS8Qc6lB027Dvq0yX7demDFey6mEd6n4t9DUZ470zm9z3FLZH2qtJpnUavnxCKbxmsisfLV1GIpyfOJg/FXt/KtnWe11Wf/xUv++mLKJcxEpGc1PNUAjvUudU6+2+ipeOvqnPqNRl5KPvaYYFRt5K6a8fuaKfCynYrAirb/8AYqe/I50y+m7fJUyfublT4eO6TXj8n3LaSX0zngOFs7jdpm+rJ7NKN9j3qh87I34IYzYC5aoOZO7PcIp530uDf/FXkkBG6nU6XPpnbRvRulV7JHabpLn6aq31r/61Q+qjVJ2zs45bTa7rS8Bv2K2Tmp/XqqH7Fs2bdK4+oy/QS65zcT5vmV2QzTbNs9s5+mWzvu1Re0XZtCRjlB1pl8jOU7bs0sZ8O56Jt+mul6uzPYkVF/MdCp+bQ03SXPYcTyPhJu1TkxbnzCYMdVc8Q1TbVSbVBdBwsoj9rTPtFgLSctMZJq5XfoN41Nmj7pIm8rjJ6Rzn/DSTFQdrZ2SfaWTQyMnsfdHP6P1K39GyPDcD0lqnvz23sXiNTkpOSQ5T/wDh/M6m/RiS5NdTb7kycz0Ojs1u6USdHPhy9/zfbrc2sxJdO5TdZ67DjcFHiejv/UoDaQkbqV9M28oW42d7Hrfv0wj9R6rL308CfDxznZ5bZ9ZmdyGC7DjZOcR1m4yFwb5Otjx0SkmkbJuX3Rc9BxwclHhdfdP82qijXzlrNsdS6Xz2+Wjk+qSJ1E6t2zbvoy+l4/ffS4KvezbwkQ1HOypZsSIrYxNzGNui+QsMtnxluJ1OFgJ53UqxY2S6N0rt6plgu5cyXSmq6GoR6Hk+wsXHj+pVkMF9TkabI+2+tgr5O/cUGen6bE4P5XkyN5TdOfIEc5bvU9w+HL/HQpzORsH0mTcr6Fv6lHN88pSYCfwVklJNuRJWC4GRL5ec87zKIcpjpk9k5ydegMa7c6nU8fpHbLd31DFOn87BgqNKqJ/T6E/wkq2ctkJOT6B67O7RLrVukmlkp3P9f2yKyPzKVWElGzkau1cxi+R3KyyPRnfm5hK9udTBZ/boLt/U/MxUwpanbes1qDFWlvLOOXPJVmXLpfkp/l61H9Eatz4ssntKnOT7tGEbMxs7rmrboD71yPC8wvmfXolPSPKdrULTSbpBBBhDaNFmzx9LVO5zcz8BKjMH3KefJ6XjuMzOR9b1eoenbPUYsvJmsW2ctmsau5k4xBdBuimjnLetqk7fnhLMyS7Zy2XQQzs7WM1uJ9Hg6lXeeZctY3TNXPA7HJ7vK6xKQJuDjbTTa8ZONchdusn9HvkL1PZ8yro2KGaqC0U81lGbNw3cAiAh4kqyrAd7Dx1qulOgsxv4JM3Ly/1VlRwXZsX0IORvpxctg4zjtqELL/Fe/wAngrUXMm55FRHj5/vv4aEHQryoybr3F6LgLF4RP8WrwRBz5NW5spqqkkU/+mihIjnvSrPR2lbdF7dxS8zQc/17dWyLXU530NLh4BzqdNpeBTSCt20LdkiT0WOqcoOaZ4eKctvKfsa3HitN0rv/AHP4KJRp9U26U2qc8+4uYzbR6KX0zZAKNRbZtJZ/NQJg07Cj0Uyu8m7+s9VemlFdh7Lc5g/9G4FRn5239mOfjHBptjUGwhpvseL+Ohs210rlB1+j0r80/NJGVp+2q8mpVw0z4xrGroSuTnZOSfsyHKXr+xjLS9ZBw5tLZtGDk5LPm10VNYt3bVUhzcf9yrEfT3Kkgu4c+pTS4SXzvxVT3RoK2y/RV0F362TnI9nlHPh3/wAFaKk64bp7SiSSHr+tUzbCDm4y1sq2k8hfjKZyyPC8wpaAtnzaDcoNm0FkIOEVNaijgzMreLl/jp/212ckW1tnwh/6ujnI5PaZpCFL+M+L7tVjbB1Nxmf0bp3fZyPe+3/PW4oT5hjAr5vTG4g9p3ca1bLtYKTYIIaNNbtu1Sxmx/gqq5h9CaaznIcFoHzdmojJvM7N1auPtMHmmISnOyT6NczcrJzrbIX0aiOT3aWAhsG7S3MWYbOchzwMhwj2KPe/M9r2K2GnllO/bWb1APqGZqe85E/KR9+TOevB52czWeeU5WAvDV+vjpnZyOl6V2CH01V7ZvTRjldq5c8DJUW/kTpkkj8mSWp4C/uVkfTw4Nz0iY6bTeo/BK2fyzNxWWzdSTVyu10uQ+b9sit2iXnHT9nzD022YmI1r+fXLbPlW/GZI93m+ZjqmW1rnMnNryc66/Or/jLLetVptZtOU3KEm1arr6BbO7b2MO+TzqX1MH9avjqmSKwmB3LmNX1LlBdfJUW7HtVesfB/JQfP5TcrxjXP0PbPVvVedkE9o5+vUZZ8502ljHWQ+ccbOW+KJfzUYjUNLkeob/5vpqH9o9Vwemuqv5hOdlVOTHKGqjM9i3RU6H610fqbnqqVcDmTkuU9VkIZPGR9a63cCmPzS4NzBQ2EtbN8tzjaTjF0GLBZvyM8/S0lSG8/5/3KMP5Fs6z3LV0vkOOCtwe1wf8AXR4SGq+2g79pGwcl0Vqu3XztZxnh+1P3ZPRLXhbozlDkxrkcHueL18Ruv9So08+kuRH3KcHn6dZTkxFFbNUdqk7Lf82hsISSbSSGpk18jRZOd73H2daCD7ay9UiTxNs6balt2+TnfRJEJiV3PqUkrOpJta19qZNBfUIp1ZGr/OWpbdv2KPB7rB5/snpPePmzq1q/RtRqGai2Tk5XU9A/z/MoqBBYKSvJNpOLSbaPVLt1RuU+VO7mrKjgARKQO27ro9xEB+kuEw1lRUMF9B7Qw4TjZ9GcpaHPRU6Yj8Vxk69LcrB8hNulOs9f+i07P5JtBh0njoIdt1KDz0q2k2y7lt7uvF9LPP8AR+hfs+nTTxyf4JMRTr0i0HlHUj2GdwaiZ/b1M5V/SeBWhjYtU9jyiJjtuBpvoa8S57gocR3+wqYR1qfKasweqdt6DutTpl9NGa5dDsUc7Kzfr1PIppnNYsppnNRj/wD3Uax6K+4irZ1pejanvs7jUYjnTh10mN7DJTpSI70znybgUzQkrGtm6EY2c8ZdFTIR92QlC1MPRuSJRWwbfWti1o3Ln82d/wCur8PaObGai40IRivFv0XHKTvV8RqrgLlYUu9Kf0qXJtOSjGz513GTnI5Pae9x0HbW0cg2fXNUF127Pgo94rg7urKSiZJ6mCrfp7JWbkakuV9U50vYadZTvvq0GtTfOuV7NNsjgM85FbO7Vz5xPnE3D0mxc/N9ObTjZBBfWPEUUc7NzWuMuBT51T7PWnjWslyY1yEF9Z23qkj4cdPX6Y+njTeMdCcJ5jGybli6cxmfoOMjk9p1MX79UVbxpqWzHlyMfoL8Rn5mZlY+0P7OCrvfvgByvpuO+cZjzJ90QhS5nzepSlaHU8mrzjl1x37JSNzkUeyVOTcyiH+fVWnySU8isw6MFSaNlY1t0ZzGLroOFk0feb5C5W/6NVvPQDaMz4N1wGLhZRHO3+EqTvCezXQ7CSm9SvZmTyNC/RTWZrIokSU1RCFKTH6JarfaQ0knVm2MZJts983RUzlvVeadM/tEPWx06qfv+oktbBuRv21TjZpGunOp5T4+dnZP4cwh6kz3Sm2pdZ+f233K0xUM5jPKv2PujkqS8QbcB13+T2NbGORZeQbke2lJyp2DrVZH9YquCxPKTZsh/Wakcm5VYnjWzWS8lQyG/ByafIR9pmyDlr7xHJ9Vjq+b1EDSsfGrCZzP5t5MatW+hzk1uN2m4TK65+tXuHfOXLl81a5/QEU1lvoj0ttl/wBF4+n7H3VSWz5z8ZdUE9maYpncu+jLuWuf2P79ZAyWpjfKkMjufepUKZoOdMu2/bI+toq2Qbcdq2a5GnRURRyfW+Z9Wph6UVq3P0NLnunXAQ9d3aW/SY5O5bOUOTGueu4y0e24eUdbCdSrj2YzliW1v4prbltnxTjMRzluzzcGEih/Zx0vbS2mziM2ozlmbDahdCIRT1iP6I6V3jpk9IvVP9+joNQO9sLO1VN0biW1l9U26V8X9StxPR/fwb9LZ15L8pOU+gN9OyTzkVlicXrfephYSs3GNnzWMcoIIOGWjk/epYymyyYy+x16WyaZra1By6syg/YuIxRnkrLH4qvrMfpE9Cm8L0revU09gIhFI6Thcyy1wvEHiVyaS15rso/nFw3VlTZFqs0Fu3exOqVKkBhUyfHi56yq81DBdw2klWzqN1Pf9it9LUWNkQcx2moUzvdN12sm15kFk8n3tR2btt/X8FeWfKsHQvREFLHGzbXuSJ/SNe3h3LaN6NkZ+T31QFnzbTanS/8A1V4WkQcxqGp7Dho/a0VHGjkXrYwU+M0BjXbaT+M/bUbZ8znTf51G/bXPUNWY1Nq30GMaugggzaO9X3aqiuZuF+blFqejptNqnXxeocwp0bTan+vYqG5lfitfSzd+2qI434Iqt5Sg27hej0Cu2bOfi+Rk8H1iX1qVSSP6N6njVmqbeVaXj8P/AKKrmg3GbahNS7rMFJtPP9JXg2zXI06Kn7LzKqt4vyY50v8AW/3dEpV3pbSfbKLZ3z949abVIOWvSf0jjZ1PdOg8ngqWM22baDuV/sO+oOeS6T0nQ5DhZOt0kpS88PqXOlcts/s1v5K1EMe6gtzberORTcyeRym66c375HhcIhMR9/0aDubearorrP5K4fGziZnU/CbqUtrT7mMbdFc+ULZP3909BHKjlrkaV0h2yecssjw1d8vXJVLNI3H7iMfVI28kY2dkuU3LldhqHjNHse1zT4dz0S9SlW3kbNjs4fTjnjr8pvM71iXVLuF85Lr01LINpOS5caxmfON0VM5ZHs2iWMpj4CejgJWWMtG2dW10znQL6h75Yt2eV6s/snqMofGQ/wDpVBkZ9C5+eaaTkuiuvJ2af9YPNoasf/SkOkZnGR7T69NW12DjYy1r5rpV0NQsotwVu68xMns1SGfJaZd1x0P+b6ytjp38xGsXqdT5ORWQdo25SQ0znjuONkrUejUNNnteB/XUqpbMWu6Tppzp/Zo51WoR3G8Br2+o9TRUkckaopp2VHqMTI2fRrpt5Lx87Ro5K2Vletx1MOg2jHPSnSHY52T/AMShqy/xngep/er9Zx8c6mQtJ2Cz9mnHLIrdmikXezPxVQrnvTMTUtuiuvpux4n1KJIu+jf6vS8i70vlTnPX4fGWWPwvN3K3Ik0rZDVZCC7j/ekw1Th6iX1Knralz5LkLoN8xbJ/FuH9itLDosku50vTpdZN4s89aqTdJm17Io2+KuUPtq8eUyTHTZCCDftsnvacxhZqqPXgp9vF/wApptC005GRLB83RT/s3Hps3BhVUOT2zkpbtPI6qSs421LBBBgi4R95mnPizKPEQjdT0ltwMlTg0pTanKc2u6a2ZYRLFgspyMistxHbU+UU6mPzjY/Mo+mZ0Jc9HrPNpm282uZs6j27xuzTzFnzg5QVTxmwYS/BWUoScurHkSKqcCKCHOZLxF9isq7YUM11zPWqc6born6ZGg8JaNy2jdK2c5HacbJ4mb7FL02+0rZdq6bZD7ho/j8yvFk3zmccRVmoN0hrl1slHO+sY6n4D15wFL0fQvSW/AH+ons4t5OPfSWo6agsnwvdn8/51ApWV1OQ27hutwUajWhPHNpFeMbWuYzq6GXrFma2am1VOTqHoItJVfBT7b/UChvskZuRpnbTmmcoNuP0jvqcIeZbOc91VVoyNM8Ou54HSqMqYGYLoXpknrTQlmo1eStLJIRSGcmiis87PNV3S0NZ2xsS5coRjW2ES/fOONkou81TK8/BVBeG+S0klYmzrprGLrxbB2os8ed2ljwkSTV+tVKeCS1Edq+qFuPR414AZSXZK4MJcRva3q02neG4JNL89n1rPVusz0+pspI2Lu55JOYPXTjmcQ0KCKfQ1sCWV1uIc3nY/wCGsRtByZ/vvv0mPF20m2Xg5xrnsXHByfW9U1Yddtx3LrPyG+Xko+q93SvyLJFoXz7aKnfcpySDmvFs19TGoNmuQghrFEc5ZY/Zbpsz756XjzOlkl/68ytMlM6rI+hyf5KJFLtoXfUBy6+g6P8AZUE1bZzn9FcIIZ3fLfuVPP0pt0rgL99Q1zpuBqvXdjTeB6VzrxGyTbj9F4/+5qMsp8Zats9dujwUe7S839ygh0JuM6K6nUF+NwVvdbuDHWP3znybgIL0xghS986ntrTabXcpteA4jFEUVkVjpKb+6qn7JT0Hcoamb6Lnodysisjw1Wqu8RTH+CtJ1GzrI5T7Bwjxlve+srdGr/m1j+c+BxEVkck/mdSumg/oXbm4km07785L9K16HYo/REJuYKWCKcpueldhk0em4ptx2rX7Ghp+i/T07pMBGs7XPeXpVtDZxs1yHUY2+xrw2n+TI1Bt8e4mT7rfpkOg5+n4Ne4qKhHWf0VDP7bOop70Exn9tHrMPo2TbPtTOIMF26KfBW73/sqxSbNbbNoRed0y6DHtkf8AS2pN465PZIT92quRim0ZJLyemyPUrfv12dYLa41nNgaFiIyNY2ytUxZKMxh9WRq80xsSWBI5t3cNgN8zFWc1WaqgLNhn1JhCWRw9a5tRP8a7hwsmij7r06RrW7YvyQm+Q+TEM9usnrP6+ZV27OLb2Zsw2t/ZnaXsrXirRCyUh2SyOBynHOTtvN3urjOQ+Kqutbs6sVtMbRMmDpdjaNuzTaTLxFbNbO1SEw7no7lOaRzA526lladxnoJjWkm2mQk2znPYuOMj9EeiUau24Hbr+p+l3evXizFi20HZtjB9ugwR4K3rceKpMbYq0nIjHTNUGC7d4otJrPFu1SwG4BCejjOQ1XsnjSd7Fkwh+bV3P7HO7NLBQp/DtuTUIzVcBvxs7vNzqUw/kBaSTz2vQENOs3yfVqq+wfzcHXqGeyU3ZltFaqcYSzGXeZLPR90qfrqY6aUU8aXTwvQCUg1pyIauo+SyS4gvDver51ZTi/ssnY6bfwNoElJM7UEEQObqmwp3YwrKN3mJeWPTPaSY1U2v0ngN1uhf6qTqJ+1S8SV+NNnOnXbrZ2cj2lVutablMGPJjZdDT5iLxbO7XfxEUITzS4NyjEa+00a+k+BkN8vORWW4nFPhJgJ51fij9PfF6a/e/OskemizrSGsk2WjLNxosUF1s1br8VXqk6/s0a1XxqkZg++M6nPQ4a1G2z74r/k1SbmTcl7opj48OEyNj9h8YphgZVs19eh7lb7tJLZfS0YRfaWq6ll2K+CROclJRsm2Xg5NtrmL9HJeIrdmqkek+Hs7C2HbLxtkI1BixccbJR7T65+uatxFNS5+xrcRPTOff537LH/DUYJZKePy+fCnMI5H7n61JWfaVsv0npzhk3RR4PpnKY+A/m7hKFP5xy66V2FTG0jynrm36Atk/akwm/joa8aaVtTCB6g9RnL7VdJrSi6c6lfVOeA4y0fotzf/AB14RUbab+u1/lrS8X7f7POpilyzPctei9/2Od/xKCOV3LrIdcDIrc8U6Mu57jtvd/XobKr9GYuW3kL/AC1mayPepbxfu7lEwMQs71AI10vlLpdwhkqdt3te1l9K2QdarI/r+iVGcrtmuQ61PA/nrGz7pK/cadamjEvevflP/uP5KNrL6pt5V5B/m1ARQc6nlNr2DfT9z2uaQxfq79M9jEI3lvSycYh0hHsVlu6PvV9qfTZuBTpmbj8FXs2hqvJXWn7NbO+v2dLyzTlNyh6/JU+j3D79WdtRhnMFNvmrVqhkP+My9XlHJipJswxbdO1TnoL/AKZneq3Cly/r10dUx8OaGng9TbQFnGtuTdTpuO4/3ROpWlgu502latW/SP8AK/rfp5krJNnUlym1bL5+SnwUe6S9YehR2mmz2zZqh771iVSjq45OEO+lfGoCx/t/c1hCfGu3Q7n1iVaVkPjTnsHHvvQoCdflzIc2Zk8hDWfS5vm4KawwbrElnftvT5rmznP7D8fZE7z5tB7MIubT2tXk4NtyTBh22T/6huYf3SVJfxrmTz4xq68n/tN5udESPukQJ6WPfrGakk2bMW2lQYLsOMsiitw+v/JXwwsjYq/uPV6wln438kmM41k+OvwUc7s0s3dImc9TJiz83BOV2sZ0fXoqSWc8xq5qp906n1DpfiqtDzkbZCEQsNx0PymRbySK2/2R+umf0epjq3dlkPfCL7PbTSaD6w7h4pJRrxZ2dWRaJYEjHQxeic5PxVnJ3+TZvyfQmj6WCowZGkl5bWbaxq7ngLrt0VFsnvFcrr7npefW5hOW2k42KcxkmwidRDJvHkatHeSKnxFyCeiUh8B/rU4bWbJNtntvn2qc58G4R1msW9VjLwyYOrgwYD+xVbxqclONkJN1OeTrKLZzNYmW7xk9PzSn3MHzadadWwVkLJ41mK6CSnfgl3attBkpWLi3E6dtJxTjsnCaWSoVyliKcmHD1d+sp9QlXiscggtZSLduW/MDcAzipJDzF3jefu736zBWUw33pdgucrPW/balBtJ5GRk9j/P6VFXMw2k5L82NvJ+xW7vfqn4R3pnKDpzx1/fU4MLRyXJum0vHzuMst/BSiq0xmeca39FW+irOsk6bOWyH2dO0q0jWs2+bRjlddi3W4P3P56pmz0x0nTanyhH+OrFRfOdMh0bgVjtQpdt60NLPuMTJruw/Y0eh+ldGcucj1NJ+r7CMbNe+/wA0/UqejJcmTel79usn23sH/npdIz00dC/bTO51MFJaVz2HrvdH7ymckU5dNtTpeBk9tWWts/2EY67dujnIvPxET+bTbs6YuZyxL7oy75eJRTZ8FbKUVxnL+5u/dpPNW7cOaYZsjF5EkuUOje/yU1uDUB+u21KDV1/Sp+onVuT2yORjLJr2kvz0W6Ka3vNw/FPVETC7ngNf6S9BSr9IrWVj/rVHmI5WemozxeN1PRvtq8TC8bwNM6z6gOVKhnU/0WtQxiFzUOYU5TbabTZ7FfM1ndeyROoaiLcW4xzd1eLBHobNHukurgJRvkpzJtujNs9fvvV9f0/Spc/J1u1kUJvj67J5OyVleGt1jfVpjG9lrIKeF60ZDnyZ1x/+V7FaSSraMcoSbrsG/bZ3468OWjmM0M7x3GozEftSHw9T0a1PrNRsjHPY2TBdBB/lrZyXaJJY9zcphGY/dB7ciYDp6qSX/wBSZ5OT2aqXFMRT7lbrPR0lOTaEZBNs+VcLJs8nO+qdOtMCxbOnOldOl2DHRs2esR4qiSRM3iYPO3D1GsJav8kHTGdknPKqDDM/OKPRVM1I+FufCfrGPU2wTSxv2EDVVsNNhmp79jqbW6a0zl+w07zk1Zb1WDdzCEP5tbnNg7SRnRnUZ0HtkVkezdpEPv4Kzadt3s3tM2kdJ6Avkt2fGRyuL5++Tc8+mGzdrW042XbOpN+hoNQizWW7pXeLuEP5uAmPB8yl8gqqdjNxiMpZ6Ws/Wiuyi09krHtpWzNuWrhfl5FNGSyeLm+cRMhe6LVe2hYtnUkv+TMZqIrW5KPrPoz0wv4Cycm5ivyGnX7+VnuC8RWRylNcTdzMfml36CxtkLbOXMU5s010+vePEUVvi2pZENjIf7hi4/ToCmY0z76sqn9GCrxZrpXK+m7dwtxs5bN+zJR7Z1ZFs6baXl1CJX1qcazWyeyVVPvrnJ6PmfWrTbCw9rdnvJX5TNsjl5nywz43atTnw7/ompb5cctvXoaj1K3oHxVsIXvqIPTespOY4pPUV62/2GWAsPsdZbXrDW5nXzKXkk45GNWZ5ieb3qbjzi4DkPve1VPnXc8B01a8DO6bnYOy9j0qCHt/aTk1eMaya6CHEW0ayx1W2afr7laW0k5dNv2aP36Io6N9PD6/WgZq2EP6E2rWnktSg6nJ1d++cZbPOye66pEyE+iPT/Yy2NpIO235M7M5Plbpuj1mjzW2UTDgUwH3ylJv46odaRctptDS9ug9TyVluzSVwYt/6lNsbK7UbM2kY25sy1Xz3CyazJ5Go9r53ZecY/n0PXUscsdlZRVzM8117bDavsdtNZNj+XDnPlF1uD0Q6TeQT+MKJY9zDuYf++ufJ6Md2ntpZaF2eOIpCUB67dojdpUnTYuPAudI5uEVIhhLv9fBU/artTtrabZvFNtquwXkrQLaOMtGz4TfN3uGdr1N8+PH9WqTjbQws5FsSztkUHD3Pbi8ndYfUKsSHxHaJJdRLc3M2l2h6X5NmbP/AKidT1BlQzB7FZVmNqLizmoSb2yB6sCqqIisTDcBT7+AfhLirKStpFrIW1tt5a09jrEQ9mYRdQqMZEtUMtJBqUBwGOT4VTeMxvlrK0LWcfQlHpqnEnTkzcHGq7AVFfeZtHYuOkfKtSgv33/ZV1eC94OKe3sJJjB8mtOSmZF3ijsHLtRcxjYdxNskqoN3jEcIAAeMQ5q6Dh/AnhHSJ14raJBOSJiUmNvFy6hUjlC8QvBlzeMK6dkzx6QVD/EOk6VKINRqWRvPIDnAEj5AJF1yjFH1TZBtGRuQu3zFniy2DLysZcrK/HRlFzJ6lDpA9H41XZbPwSJeCZupuz8/H2gSjkxB6mwVUBRu3AbzGOgsRNXCHwmAggX4RCnuxHgybPpPZ9EWsmbcvhWdqlbuSR0W2clZuTHUKQpwM8TU5ypia/LAP1jSCbS5pX4Acp6zxholHQnUzO10ANi5vUB+5be35v2XNh5XS6Fy2dcfOT7but8vE/r0asDa5JRoWkfNmuRn614i8WZ4MtbNIU3CP6OPzvaoDtRspJbN7Wv7KkIDjkx4s0zSIXAplnEuIA+ABuvoMkk9LziRFuv74KRVFMWOa09wtjHO2Roe3sV0bJStm3WzeDdNpNBCVYM26LxFZbif9WCiuy7ahprSIRnAQ5eW4yPZfaY/NN/NXLLR88QyGxZFFwv70O6PiNl0fjblUlmS7EzgXCOT2vESx4d8lZ2TQGuL4/lGSagJINsr6F7V51u6sI9sTFj+dF0E0WYe7Pvdb5hD1w8/km0n0l1kIdzk+qpvt5tjtsvsRg0pCSULa5hIptQepAXM3SK41PaTOlgL8+qvPIoyzoguGg6l0IBnI8JPKwehQ2geHJdLY5kn9TkHSOip4C0Fe5KV6NpWrVDvON92oca+1TlfpXk6Kn7lWlaHwdJeJsataY8nGqOmbJCReRhFDi7bNlsGWqcBIBLhzU7ylOJi4wxAHPVFOWMsVQwNGqmMuZeZFDm3q2jNLniIDwgqfX6Gva51NIHhpLTYg2cDYg29we49kx/l/JNrNrwTb4x+0V6vEpVmLQOePGNvU8FZbtPSzPm0JlZ87GPGKeNEFhbcZFZLtMr+iUGTmnbzIkylEf8AlYNyj4NPVdVq/wDbT9Y+3+pbLuZ2MXX0+X2PZpOvVq+iU9T3ltbNus/VQegQcIqI5KPaZRD9njqtF3L1o2XaimbIcdsiiHaqk6mP5lDmS783iTM4Qb/q709X/wCkMkfuIf8A118bME08sNnTlBy6a+TrcFbO9DdPgqp5q1EjByL4RcuFnucpoks3hNU8Zt/D6dMq8w/K4vdqmDQI5KKNxMpLGfF1fbqppBYRk1w98P79aGhpdpZXV6rzLGI08tG55N5M0yC/fLLLdpmn/ip3shtQkXMau2kmyD9cctHjfMwkU+pQzZHsYthtinQgrLsXL1y4EcJEiX7oBeJhHxAUAAREw3AAAIiN1dc7Pv8Aw7XyTZyeRtpA6wEwKJGxXj8Elb7947RuqnzfqMNF1NIyZmD1nG65T6O8GeVrL9snAX/a55SDsiXtttChJz8kMiJ5AhlHknJLLESbq96kmc/rTkIclfv/AJ+WjjI1eTjbX8hWjiHrfkyBs20RSgVeqqqu4S85U+I+PB18NNm0DwK7R2bg5mRs1bSNkUWCIuHrds+VSUTSTEAA526pU1FAAohfcURC++66lbZb4GlqNpdmkJ4sxHR5nLo7CMbv1zpqyLkoEEySOEhigPEIGJQSEvMAYr7wBW3R4w5x9k+Pi+jnpfNNnYY72yyGNyQAL3te5At8kKudou223+0x0xk7XybHPiEdGjktCNcpLreZ3W/Ve8pOf0r6amyQ2dyDeTXRTaOxMgcSHQwjxBDx31eGxXwN3+02y0hbZ3aGOiolm4Fu41Dd6scggQpxOYGzdUCEuMG8cSgIgN19w0bBTwwdEaX1+oiOIz1bw1g7kkAD9yey5jOu5dNtK67Bvxkf4/4K3EXkmvSYx15Qioj9L/R662tp4ECdkLFjbyPtfESsWU4NhErJ8BVzHEwXlFw3TIe4S84FMIhcHN47mk3/AIfcbFKpR9qLcw5ZV22EsYmCEuoAOhuwqonIzEq5QvDDgExRvAQEQEKLS1+vaXHSNqn1DBG7gOzbif2N7H/lcZNk5Jy5QayXAWf5fBzuHm9XMq8vB78Ii3+w2yT7TQUE/QQk9ZkzyJ83cRMlwvZ3z0q7edg01sDnS2KXdlmF1Fm6+JuVUoIorFxCBiKkIoQ13jKchRD+4QGo+zg7KVfq2Y2hpGcRXJq6PKV3EaJeYc56BraWGoZtvTGmma9oc03BVjba/Ce2leELGrwdpV0YmD6OstDs0OHmpHxJL4z75THOeqdLs7s458mnNOh32dg4qp/3cFdhWA8EfYzIWEhZiWtVKzar16WNWWYAmUjVZRRTJIOML1ABNO/EA3X3hzVyRtbsGGz22cpZ5EwqpsXTluVTIuxqEMJQNd8F91dTUDKOHbgQg1mnrZ5KaN3XHbIWItft3HN7dxdSFdn7huNze1ArAPwq+P8ADWUotHrfGbU5993rqyrcHq/NXj4C7JZhtysKcXIpgeYjAwelxyc1dsw3KUhs5GJgLYMYd8naB0usRWYSYnMkKKAFNvnKJgvKYOa/xDVAeCnsntJA7ZrIyUpYCWjm4SzFYVFWp0ilUKuQREcXwc1X0jKW4h9kpnVg2Uku7PaN2RcWKYmNgBBvhA2HnuvEbv8AGqaeQviJcsp4uEv8WaUKMtDsKj6gSO0fcAg/+0xKpncOWNqHEgMuyshCuCz0qQxliLYgVAjUFB51hEpykEQvAAOIX4S31UWyGQVfWIncQJgmFookxMHiG8rrn/8AxT23mrbBY9pbWbjF4y0baXQaMFzMgbLvETpKisRQoAGcUBIiG8A8yogN4DdXhWyMRZNW2zWCSKkyC2DFJJIgbiRSi63Cj8IFvwgP6qvwa5zXj2WIr5H0mma/p9QAZiWyvc36DmBji3u0gN5BLiTzkb2Cepsig9q23K3hZhJ87LFyCyhWjMxCKLLKPiolATnASkIXMxGMIDcAUQtz4LezuSYsIODQkY15KZrOLcBKJvmyrogFEEFABuiZMTYigBgxheYPGADdYENZk1ndpMxMt1Sqytq7XOYhqq4IYyTNHMAypxTAQBUw5pAKU27ujeAiICDfMnM5tHs3cEfuVESWsRRyDtmiJAMB0BE4AgiQbxAQDeE3iqltHC4HJo5Wv8TeLa+illfpsr/QMLXNtHtgvc24cXeoS5jh9PA457rmjZB4Ltml45qra87567kGyz9Fk3XTaEQZpqikK6yx01BDEoUSlIVMwjdfeF4VYjPwXtlyc0wkjjJRya7pJu2Is+TcpSB1CnyypuCIlwbyZiiJkxuELrqabNwtoUIOHlZaTXiXzMijRhKNSJuSKNVFVTFSdNjGvwCcF8JhAQEAEMJsIDUlCAey1so1CVBu0k7OSUbIAeOOoDF+1XWTAFARNzJKXnIO6UoCAmASgIXjKOjgawANREmv6tHrJgr5nsjke5sRaI3ROGBs0mxe2QEE9XSS2w44SVtQUsfbXZi8l9p64RaUWpKpwR0JIwmQFNBMUmhG4tz40ymKkUOIn1zCNw3iPAliJtmptLYNZaQUSjTPEyuFEigY5EubEJQHmEQLeIBXWvhE2cl57Y/HnjDgUG81L477/hK1+SuLIiz8onatkzcgljKoGUiqie/eH2arnhifIC4cpn/47qat/hunmnmdI54uS63Fyb2sB7883PPdfTy3IWbmLES9k4s7hu2bQUa8TmdQk4O7YatqkkmsGkRVECFUKbDiuAUiBzgACGtKAsvCxMrswbppK2INBcpvHBpQW2qMo7biR4Y2nUADhhSTKTLHCUTb14mEV87V6nYKVapAOoLs7jAEMXPzPmV/OP6r68WlavXmz+cbNCCK59njDCUN4bgftBHxePmAaMDRe6wj62ulhjibO5v8+5l2hgJaC4i9m2JuAbkcnvdfPnbyyspF7TpiO2aywu4ZF0Zs0dL9qoT5b8Jb+e8L8Jb7r7gvuq9vB38FaLtfEMbUW4ePQJKqqoso1oJE1HGUXEqqKim4igQL71Bxc5TABd0wl5omY6WYWqFrJAbmW58SX66+jWz4oSWzizZIJXnXsDLN2olG8QWKs8MsXm84SiYbvkOWoNawnstz4n1Wq02khbSus6WWOPKwOIe4AuseL24Fxa5HB7JTtX4JOySXYtwsg9WjDuFxZNHASab5iZ2YBEEVzFRRUQE1w4TCQxeYee4DGLJsh4MuxlnA2cRkrP2hk5WWcnjHgN5NFuZq/KIYkshZsACAAcnPn3DeN4k8QeNk0RaRls3tSlKlWMDqVjUWhOffcXriAE+UcOP7wfLV1qmSVthHqtjlOVTaK/MBynxAY3R7zAPyCN4h/fVoA+FjtZ13VdP87RxVDiYTTlr7MytK/BzHdOJ4FwcQee/Cpi1HgibInbNKHaQUrCuJZwLNnMcuspNsDswbhV00EwEhRG7nBQRALxAD4RCvmrths07s9tOlIh4imksD0yKiZA5ymAbhAf8AEK+pGy1WVVs875TMoN1sIXLAw+LceX3f7Ar5/wDhhR5o/bTaCRMW4BlnJP8ANNUgn2kVVYNQrdOqZjK2LbLS4NDutpJBwa0EAjji/PN12B4H1hI+D2U2cZIXtnVvpYycm6T7QsYgZMoJlH4AE4rHMHiMKaV/iqi9rfhXbQ7SW7WZQT12wimipU2Ma1WMRBqkBxApCEAbuYPGPjEbxERERGug/BQtXFTOyKw8sqtmN7Iy6sZMELvGQaOTgomqIfIbE5L/AHp+0Fc8Wq2NWq2W7aiqykSm+TZPyKimoTMQcpgYDAPyHKctwgPiEpv119d3S7w8M/EGpuqPuAxht/aPAFtvwXZE2/Ve/IVtWc8KG0cjsykbL23gJOYl12zhnHyajq/IRXKQDkOBiiYxQEmIoAYLhMb5a6F2CK2PZ2DsewhxkHZ5+WSaLOBcIlViJJbARRVBNRofDcUUxAxVRE2WURwCAXT059k/Xs/LqQTRjGStm5GedxwxrERIdsDwcshxbXgQ2lL4yiNxh5x5qB7IZJtNO7PzbOOQj0ZDaQk5Kzbp4EW+NRM2WmUOYCFvAAD5ArkDq2tNm02tlo4zG+KZjX3DHBz8oweDkLYlvNgbgW5C87H7PRGz9nAWgsIojIzdo5ptHyKorihkgC4HMyDcESkUuTxqXmvKF2G4RAYMBF7OLO2/kY2wUmZdVxZWaPIN9XqiNVAbK4SFUyEb7yhfhwjd8o31I2MJii0s4mcBKYu0BpiKPm8yfNVabD7Oz7Dabap7JlUFILPTIAY3wXtFKirJHTyVesiSVxaxgAabWF4i74vwSff35uUwbadlu1HaBZPZ48sDByj1s1hVUnB2bdRQoH17scIiUB57jFG79YVYGzmzFprJI7LoC1rB0zfIzLy5JdM5D5QmbgUQAwANw3D/AI30s25t2OzmyNkDRlhYiQGRijunLleOSXUMprXSfOJgERuKkQP8KbrHSaVoXuzO03IEfGPH0m4IqVk0K3KoBBRwXlL47sRuf9dfMheyz9bJXnwdQtkjYI7UvIeS62cVunAD9+rj8oUbYvsWeR0VNS9lJGTXlJMkW7MEsLczdZU58IlTM3OBy4CgIjmX3iIXB460yng/7DrNWaXll7Iu5I8K+Fu5SSlBZiHEMVMSlFuoU4DdiHfAQEfF8IlLMqvfyaZg/OYQStrG3CcuG4MC1GLasnbSzduRcJGIB5JuICPnDnmqAsbGyea1rOoUkesGGYgwMY6PhpxJYSe7TcEj9V/wVATshHbO4uQsvCMhBhH2+Z6VBMbzmIGrAAvNcGIQAPkC/wCSpe07ZXs5kXL6dtXsxMi3O4Md29SkmkiZqdQ9wGVbkKBgLiEAEQOIXiAX3iACdtyQpH8hcHONvWQjz3/C7r06cokjtpYTLM+NNs5Mg6EbiCU6oEImIfLjMQQr7fgBD1bqqKv1aspZzG6OON3ZhaSGONnZNJtx+ktPPdcrbQNilirMTqkatZuHUKAFUSVTSvIqmYoGIoUfhKYpimD9QhWVY+2Z2SKCzEe6bKqPiwLI6qRhEogUyd5DBf8ABhEpf7wrKHfLi4hfoem1T62jhqZG4l7WuI+CQDb/AGVg7NJKw9ilpUdm20PlY8ooiYGDidbGXaKgPjbKrEXIBRDzMBRDmEDB8O2xm0Syli5Y9hnpdpNj13jgyxAM8ZrtFVBAAxZ5WolARAof7Kom1TWyzVkL4tkWAqrutKkPOXL9sbrsY/33USb2FtLYxwV1F7QpB0dlm5aMijqW6WIbxykxOAk/xMakdLqcmOLUdqXhzT9SqG1VSwl7QQDk4WB72sRa/F7d7C/ZdAWm2obFbGuGttrY2tevl0FdOzfTDs70Gzi+/AmmimCZT38/VG4efx89EIm2eyfblBi2gpycUYRjkq6xmrRNikouXFhUAy7cxzCAHELsd36q55tB4Sdt4YTNbTQlmZtuqqRcU047ShmpjiKfrH56umyu0iVtfDs5B00QbgqnmCmlzBfRba6Qm5KCPhjSzTPpNkYPN3DkZH/I3u7/AHJVj2pJZe1EYixtAd4YGK4OyyaTpFByiuAjxMxJIpCjcIB2fwAPjABoLKt7Ivm0fLOJO1sg8inBH7F+EsVdbGUbwAgqJmSEL7h3SFvEOe+q/lbUOZkqke6ao6ZPuwDmN/f8tbV5+Ri0SqtiNAFK67gj5vi86pCtlHuqpvC+kVjzJPAHOIAJJNzj9JJvckezjz+U8llYGXea1m0mIbTtSMxIkcjYqiRTnOGYkdA6IiBlDiAlTKAYhuAKTLYRVrJBwCljZZqwUUdIu3Lt46z3Tg6PYlMfCUoEJ5pCEKHivvuLcHidoMnaNsu6Fqg3Lhy8seIF3+NTC2gkTKrooZCIZeZupAPPUDVzuFroqLQqCKcVLYhmCSCSTYkWLgCbBxBILgLm5ueSpyr+0Fl4mUk4qCFAzgyi7xMJu+IzzluMcGwkA1w+iKohcAAN4BXKNlbHpxdtIh3HPrI2mGMelfyTJs+Icz1PGBgTECgOADJgKd9w+PxDXVnLJGvWjkFdP48Xef31isoxaF1Q2ah9R6xFoVEfw1z6l77F3srdO06l0mMw0UYY0m9h25+Pj9hwlx1tPQi5BxOPdldo81ZkSPWYunzJNuo1KYhio5XJuEAAyZBvAANeF999IsNt1l4wiMpJ7NZwF2Sa7Vg9g5bSgLXGY4IL4klMYFEwgAhhG4CgPiAatTlZhp9LyOXT4lFMrOG6/e/VXOm0PaWpBbMpTa3EwKDZ9JF5PbtU1jgk1TxGTv8AeD5143c9Xtq5Sl/8O6UYXQOiu1zsyCXHr75XJuHX7kcn3VHeELbK3+0C3722zqybhmR3lFRFJIyyQFQSImURMYRHEJSAJjDzmMIj8NPOxPwl5HZ1CN4C01nBfMGrrUNRZnMgu0cZYCdVFUt4lEQKUBAQMUQALy3gAgB2MWgthtOZTKluLTrSDODaI6VmCJE0wOcxiifm+G4oU0TEPHItF48rcNMwJjTTv5hUvEcY/KO8NW70gN00mo6aspzSzsDmG3B5HHb/AI7j4PKsy3fhcNW1n0LTWUstIiLIwnReSC6axWRxAMSqKKKKKYKiIAAHMUwhcAhzgAhss74VNhG1ibNSzmx8wRaFV1ZTMJRsiJ3QiAisdJRmoYxhApAEDHMXd5gAOavG3CxUdAbMbN2XZGDC4USTXWFMOICSYqDeXxbxjDfVJsLCR8qokwlHa6rdXLvKkOUb7wUNDqEsjb/5WQb/AA5pb6d1M6LpcQ53Ju5wtYuN7kiwtcm1h8BXg98MvZfJNUWpdl01GJt3RHoFjJFqhmOCCfKOqJmRhxgChguAQJz8xArk3wnrfwO2S1slaKIgCwGvK3vZqqkUMZQpBA6p1CJplLjMAmHCQoXm8VXEvsysuy6Ki1EDZiieZfz3Y6p3aXs8aWv2hNIhrIGigaQTt0kokkBx4AqCYprxDEB8IfN+C+mTZS7urodKpaSd9VE2z32yNyb27XuT29ktbCtsE/sCtUg4Xm0k414RRB+KZAet1URARM3WRHmPecoDcPiEAEBAQAQ+ilidszm2baLhrHbKZyWkkY9u9dsGku3yI5JdTAQ6YPG7gyaQnx+I4AGEfHXycsNZRxb1y71k2shlJioHDBTnMa4fGNM7LadtCs/LOrO2ctdIxCbwEo5wsyVyzqIpHwlII+hesqbD4rzUSg6vS6StmEszOoCwcLhwHwHCxt+L2XbtsPDXsTLxkg+aum8ZMHaOonNllHcg8STOCiKiaBkylbkASnUDshuxCIXDz0z7NPCgsDaSzrAGNlJhFWxazRUy7GSbImcOiJpgD0xDsDnMYwkEwlOc4AICAc3NXyxcu3caddm3cGwZh07/ANRTXBV4+DPBnte2tG1dSbhua5NfMSHnvy1Qu/uqDnENuqf4e019OaTaAY52RAuMnccuIILjwOST2HwF3krtvsutNyAnswRpArnbOk27J6Rs7QeIlEuoIqVEiZTGAw3lBEA5i3c4Xi7qeEbZOXhlIl9HyZwVIqmDpSXbHeKN1iFKqTOVanEC3FDmSBPxjfffXGcfZaMZmj2rszh83dJqKGSWVG6+++mqMstZJSNU1NnG62Xl5N5zcP8Au56odI8dl9l0PT5pHyvj6ntDXG5BcBxYkG544uebcdlelsdq1hX9jWllmlm1zJt1Mto5lnzVRZoQDmUOQgptklBATHMNxjiAXjcFHJTb7snIRhaVVlLRwwK5VGbWLkGRUmh1DgG6TRCJ+cpedQTmEAC8Rrm+Ns1Ecprl5MY5Xo5R7/vY636GL1P9mo95Udx97quTQtOlpGUD47xMti3J1hb6ff8ATYYj2sLWsr5mdvdgnkvIspKIQCGekb57Mr5JuukukAgRxiKmQhFBxHDmTw3HEMPw1Pk/CA2WTMaazk7IWikGixSqkO6teRRRM5RC4ecgpB4rrxSEfkEPh5iujT6rUwjBbRqZY3pXZwe1X6dGFYoqrIQLQBSy7ub26iJnhfKvw5pVe8vqYQ4ubiSbklvsD82ubE8j5XRUj4UtkJ1+q1lbMAo1Bw3WI4NMIA4VXSMqJHGYiVJMFBKsN4ZYgIFARvNeYSU94SNiZNqqK4O5UGwg6BCUnEFGh1A8RzJIopCqIc92Mxg5+cBC8B5pJHvEZnTFVjhRyyKYeTi4rzePexVPI1bYzLZQXq+PmD/+Vdk5Rm0LTah2csQcSADe5yDe2XPVb/K66VhNo+zrbvCIx9u7QsyWihzmMqKjlNBRRM3nJCHMJOcoHAOYDYKyuaHLTJODcDJmApjjiMiURHnrKFLL8pxlZf/Z"}}},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport shutil\n# Dataset parameters:\nINPUT_DIR = os.path.join('..', 'input')\n\nDATASET_DIR = os.path.join(INPUT_DIR, 'landmark-recognition-2020')\nTEST_IMAGE_DIR = os.path.join(DATASET_DIR, 'test')\nTRAIN_IMAGE_DIR = os.path.join(DATASET_DIR, 'train')\nTRAIN_LABELMAP_PATH = os.path.join(DATASET_DIR, 'train.csv')\nupload_tar_or_plainImages = True #set False to upload plain images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport PIL\nfrom PIL import Image\nimport tensorflow as tf\n\ndef load_image_tensor(image_path):\n  return tf.convert_to_tensor(\n      np.array(PIL.Image.open(image_path).convert('RGB')))\n\ndef to_hex(image_id) -> str:\n  return '{0:0{1}x}'.format(image_id, 16)\n\ndef get_image_path(subset, image_id):\n  name = to_hex(image_id)\n  return os.path.join(DATASET_DIR, subset, name[0], name[1], name[2],'{}.jpg'.format(name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#for file upload, download, decode\nimport base64\nimport io\n!pip install PyGithub requests\nfrom github import Github\nfrom pprint import pprint\n\ndef get_repo(username=\"uname\",password=\"pass\"):\n    # authenticate to github\n    g = Github(username, password)\n    # get the authenticated user\n    user = g.get_user()\n    repos = user.get_repos()\n    !git config --global user.email \"XXXXX@gmail.com\"\n    return(repos[0]) #return the first available repo, change index as per your needs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Copy CSV files to dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"import csv\nImgPerClass_columns = ['imagecount','path']\ndef write_images_perclass_csv(count_path_dictionary,csv_file):\n    try:\n        with open(csv_file, 'w') as csvfile:\n            writer = csv.DictWriter(csvfile, fieldnames=ImgPerClass_columns)\n            writer.writeheader()\n            for imgcnt, pathArray in count_path_dictionary.items():\n                for imgpath in pathArray[0]:\n                    writer.writerow({'path':imgpath, 'imagecount':imgcnt})#path becomes the key as it is unique\n            csvfile.close()\n    except IOError:\n        print(\"I/O error while creating csv file\")\n    \ndef upload_csv_gitGCP(repo,bucket_name,csvfilename):\n    try:\n        if(upload_to_git):\n            with open(csvfilename, 'rb') as csvFile:\n                repo.create_file(csvfilename, \"initial commit\", csvFile.read())\n        if(upload_to_gcp):\n            upload_blob(bucket_name, csvfilename, csvfilename)\n    except:\n        print(\"upload CSV error\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below, reads entire image list from the dataset and iterate through each image and upload to Git. To check the upload process with just few(100) images, we have taken head(100) into dataframe from csv, before for loop is added. Edit this line, to upload the entire dataset. \n\n# Caution\nUploading individual images can be a quite time taking. To make this script run faster, tar and upload one single file, for every ~10,000 image files. set upload_tar_or_plainImages to True to upload tar files.\n"},{"metadata":{},"cell_type":"markdown","source":"# Preprocess images into groups, for k-fold training later\nCreate one tar file for each set of images, having same number of images per class/landmark for ex: if landmark 9 & 10, has 2 images each in traning set. All these 4 images will be part of one tar/group file.\nThis way, we can train model with all classes having only 2 images once, and in next round train model with all classes having 3 images in training set etc ....This will create 430 groups. However we shall combine few sets having less image count per landmark, to restrict groups to around 50\ndict = {imgcnt: list of image paths}"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport pathlib\nfrom pathlib import Path\nfrom collections import defaultdict\ndef prepare_imageGroups():\n    traincsv = pd.read_csv(DATASET_DIR + '/train.csv')#Remove head to upload whole dataset\n    sorteddf = traincsv.groupby('landmark_id').count()\n    imgdict = defaultdict(list)\n    for landmark,imgcnt in (traincsv.groupby('landmark_id').count()).iterrows():\n        landmarks = traincsv.loc[traincsv['landmark_id'] == landmark]\n        imgidlist = list(landmarks['id'])\n        imgdict[imgcnt['id'].astype(str)].extend(imgidlist)#image count in a class becomes key\n    #replace imageid with image path\n    for imgcnt, landmark_list in imgdict.items():\n        imgdict[imgcnt] = [Path(get_image_path('train',int(landmark,16))) for landmark in landmark_list]\n        print(\"total {} images under all classes having {} images in each class\".format(len(imgdict[imgcnt]),imgcnt))\n        #some sets have 30k images, divide them into smaller lists\n        setsize = len(imgdict[imgcnt])/3000 + 1\n        imgdict[imgcnt] = np.array_split(imgdict[imgcnt],setsize)\n    return imgdict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image resize and save to working/Git directory\nIf in case of plain images, upload to repository is done below. If uploading tar files, it is done in a create_tarzip function"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import random\nimport subprocess\n\ndef upload_imageset_repo(image_paths,repo):\n  #image_paths = [x for x in pathlib.Path(TRAIN_IMAGE_DIR).rglob('*.jpg')]\n  for index,image_path in enumerate(image_paths):\n    #print(index, \"original image:\",load_image_tensor(image_path).shape, tf.size(load_image_tensor(image_path)), image_path)\n    resized_image = tf.image.resize(load_image_tensor(image_path), (224,320), method='bilinear', preserve_aspect_ratio=True)\n    #print(\"resized image:\", resized_image.shape, tf.size(resized_image),\"\\n\")\n\n    np_array = resized_image.numpy()#.eval(session=tf.compat.v1.Session())\n    imgarray = Image.fromarray(np_array.astype('uint8'),mode='RGB')\n    filepath = os.path.normpath(image_path).replace(DATASET_DIR+\"/\",'')  \n    try:\n        directory = os.path.dirname(filepath)\n        pathlib.Path(directory).mkdir(parents=True, exist_ok=True) \n        imgarray.save(filepath)\n    except:\n        print(\"file save failed: \",filepath)\n\n    if(upload_tar_or_plainImages == False): #False to upload plain images\n        try:\n            with open(filepath, \"rb\") as imageFile:\n                repo.create_file(filepath, \"initial commit\", imageFile.read())\n                imageFile.close()\n            subprocess.call(['rm', filepath])\n        except:\n            print(\"Git repo file create exception \", filepath)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tarzip and upload to Git or GCP\nTar all the files in this set, upload to Git and clear all the local files. Clearing local files is important to not to exceed working directory limits. Files paths need to be relative paths, before sending to tar. This helps while untar into some other VM or floder path. Use below command to untar and merge all files under same tree structure\n> !ls gld/*.gz |xargs -n1 tar -P -xvzf"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tarfile\nupload_to_git = True\nupload_to_gcp = False\ndef create_tarzip(directory_to_tar,tarfilename): \n    #os.system(\"!find \"+directory_to_tar+\" -name *.jpg | tar -cvzf \"+tarfilename+\" -T -\")\n\n    def make_tarfile(output_filename, source_dir):\n        with tarfile.open(output_filename, \"w:gz\") as tar:\n            tar.add(source_dir, arcname=os.path.basename(source_dir))\n    make_tarfile(tarfilename,directory_to_tar)\n    try:\n        if(upload_to_git):\n            with open(tarfilename, \"rb\") as tarFile:\n                repo.create_file(tarfilename, \"initial commit\", tarFile.read())\n                tarFile.close()\n        if(upload_to_gcp):\n            upload_blob(bucket_name, tarfilename, tarfilename)\n        #os.system(\"!rm -rf \"+directory_to_tar+\"train/\")\n    except:\n        print(\"Git repo file create exception \", tarfilename, Path(tarfilename).stat().st_size>>20 )\n    subprocess.run(['rm', tarfilename])\n    subprocess.call(['rm', '-rf', directory_to_tar+'train'])\n    #shutil.rmtree(directory_to_tar+\"train\")\n    \n#get all tar files\n#untar and mege folders, similar to original dataset\ndef get_tarzip_from_repo(repo):\n    !git clone https://github.com/jkreddy123/gld.git\n    basedirpath = \"/kaggle/working/gld\"\n    tar_paths = [x for x in pathlib.Path(basedirpath).rglob('*.tar.gz')]\n    for tar_path in tar_paths:\n        subprocess.call(['tar', '-xvkf', tar_path]) # use -P if error with /","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot images for sanity check\nDownload few samples images from GitHub and display them using matplotlib for sanity check and correctness. Change samples_to_plot to download/plot desired number of images."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\n\ndef plot_images(image_paths,samples_to_plot):\n    plt.figure()\n    f, axarr = plt.subplots(samples_to_plot,1) \n\n    #plot few samples, by downloading from GiHub\n    #image_paths used from the previous cell\n    img_paths = random.sample(image_paths,samples_to_plot)\n    for index, img_path in enumerate(img_paths):\n        filepath = os.path.normpath(img_path).replace(DATASET_DIR+\"/\",'')\n        git_file = repo.get_contents(filepath)\n        file_data_string = git_file.content\n        image_data = base64.b64decode(file_data_string)\n        image = Image.open(io.BytesIO(image_data))\n\n        axarr[index].imshow(image)\n    plt.draw()\n    plt.show()\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Deleting all files from Git\nCall below function to delete all the files in the repository. It can be used while testing or to remove complete dataset after project is over. Clone repo to local and prepare list of filenames to delete. CD into repository home directory and edit email, username/XXXX, and gitname/gld(twice) before calling"},{"metadata":{"trusted":true},"cell_type":"code","source":"def delete_files_in_repo(repo):\n    !git clone https://github.com/jkreddy123/gld.git\n    basedirpath = \"/kaggle/working/gld\"\n    if(upload_tar_or_plainImages == False):\n        image_paths = [x for x in pathlib.Path(basedirpath).rglob('*.jpg')]\n    else:\n        image_paths = [x for x in pathlib.Path(basedirpath).rglob('*.tar.gz')]#*.tar.gz incase of tarballs\n    for image_path in image_paths:\n      filepath = str(image_path).replace(basedirpath+\"/\",\"\")#format as per repo path\n      print(\"removing {} from git\".format(str(filepath)))\n      #os.system(\"!git rm -f \"+filepath)\n      try:\n        contents = repo.get_contents(filepath)\n        repo.delete_file(contents.path, \"remove file\", contents.sha)\n      except:\n        print(\"repo file delete failed: \",filepath)\n    #!git commit\n    #!git push -f origin master\n    shutil.rmtree(basedirpath,ignore_errors=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Upload to GCP storage"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas_profiling as pp\n# Set your own project id here\nPROJECT_ID = 'mlkaggle-288509'\nbucket_name = 'gldv2-train-validate-1580470tfrecords' #should be lower case letters\n\nfrom google.cloud import storage\nstorage_client = storage.Client(project=PROJECT_ID)\n\ndef create_bucket(dataset_name):\n    \"\"\"Creates a new bucket. https://cloud.google.com/storage/docs/ \"\"\"\n    bucket = storage_client.create_bucket(dataset_name)\n    print('Bucket {} created'.format(bucket.name))\n\ndef upload_blob(bucket_name, source_file_name, destination_blob_name):\n    \"\"\"Uploads a file to the bucket. https://cloud.google.com/storage/docs/ \"\"\"\n    bucket = storage_client.get_bucket(bucket_name)\n    blob = bucket.blob(destination_blob_name)\n    blob.upload_from_filename(source_file_name)\n    print('File {} uploaded to {}.'.format(\n        source_file_name,\n        destination_blob_name))\ndef list_blobs(bucket_name):\n    \"\"\"Lists all the blobs in the bucket. https://cloud.google.com/storage/docs/\"\"\"\n    blobs = storage_client.list_blobs(bucket_name)\n    for blob in blobs:\n        print(blob.name)\n        \ndef download_to_kaggle(bucket_name,destination_directory,file_name):\n    \"\"\"Takes the data from your GCS Bucket and puts it into the working directory of your Kaggle notebook\"\"\"\n    os.makedirs(destination_directory, exist_ok = True)\n    blobs = storage_client.list_blobs(bucket_name)\n    for blob in blobs:\n        full_file_path = os.path.join(destination_directory, blob.name)\n        blob.download_to_filename(full_file_path)\n \n#from colab use the below lines to upload file to GCP bucket\n#from google.colab import auth\n#auth.authenticate_user()\n#project_id = 'XXXXXXXX'\n#bucket_name = 'gldv2-train-validate-1580470tfrecords'\n#!gcloud config set project {project_id}\n#!gsutil cp  ReducedGLD2020.tar.gz gs://{bucket_name}/ReducedGLD2020.tar.gz","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Main code\n*Note:* Replace XXXX with your GitHub account details, for the below code to work."},{"metadata":{"trusted":true},"cell_type":"code","source":"#log the time elapsed, useful when running in background/batch\nfrom timeit import default_timer as timer\nfrom datetime import timedelta\nstart = timer()\nrepo = get_repo(username=\"jkreddy123\",password=\"XXXXX\")\nimagedict = prepare_imageGroups()\nfor images_per_landmark, image_paths_arr in imagedict.items():\n    for index, image_paths in enumerate(image_paths_arr):\n        print(\"uploading {} images of classes having {} images each\".format(len(image_paths),images_per_landmark))\n        upload_imageset_repo(image_paths,repo)\n        create_tarzip(\"/kaggle/working/\",str(images_per_landmark)+\"-\"+str(index)+\"images\"+str(len(image_paths))+\".tar.gz\")\n \nif(False):\n    csv_file = \"images_per_class.csv\"\n    write_images_perclass_csv(imagedict,csv_file)\n    upload_csv_gitGCP(repo,bucket_name,csv_file)\n    upload_csv_gitGCP(repo,bucket_name,\"train.csv\")#take care of input path name\n    get_tarzip_from_repo(repo)#get all tarfiles and unzip into same folder structure\n    plot_images(image_paths,4)#download the uploaded images and plot\n    delete_files_in_repo(repo)#clear local and github repo\n\nend = timer()\nprint('time taken to upload files to GitHub ', timedelta(seconds=end-start))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pls Upvote, comment if you find this analysis and code useful in your Machine Learning tasks.\nThanks."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}