{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"*Poonam Ligade*\n\n*1st Feb 2017*\n\n\n----------\n\n\nThis notebook is like note to self.\n\nI am trying to understand various components of Artificial Neural Networks aka Deep Learning.\n\nHope it might be useful for someone else here.\n\nI am designing neural net on MNIST handwritten digits images to identify their correct label i.e identify the number in an image.\n\nYou must have guessed its an image recognition task.\n\nMNIST is called Hello world of Deep learning.\n\nLets start!!\n\nThis notebook is inspired from [Jeremy's][1] [Deep Learning][2] mooc and [Deep learning with python][3] book by Keras author [François Chollet][4] .\n\n\n  [1]: https://www.linkedin.com/in/howardjeremy/\n  [2]: http://course.fast.ai/\n  [3]: https://www.manning.com/books/deep-learning-with-python\n  [4]: https://research.google.com/pubs/105096.html","metadata":{"_cell_guid":"84d4608d-4cc3-fcbb-57fb-61f07ad7d020","_uuid":"6407080d145a62b4803d7f159c00118a056a7b5f"}},{"cell_type":"markdown","source":"**Import all required libraries**\n===============================","metadata":{"_cell_guid":"654456b6-e648-0379-0d66-1cc97af6d00d","_uuid":"6b48ce0e361bdb67689dd2f254ecedd9ade1f5ff"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense , Dropout , Lambda, Flatten\nfrom keras.optimizers import Adam ,RMSprop\nfrom sklearn.model_selection import train_test_split\nfrom keras import  backend as K\nfrom keras.preprocessing.image import ImageDataGenerator\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.","metadata":{"_cell_guid":"e5b02688-c589-5a89-e11c-837c6a99eb6e","_uuid":"f043e48097bfd98e41710142dd8aac41fa88a801","execution":{"iopub.status.busy":"2022-07-21T07:34:42.422023Z","iopub.execute_input":"2022-07-21T07:34:42.422326Z","iopub.status.idle":"2022-07-21T07:34:42.501180Z","shell.execute_reply.started":"2022-07-21T07:34:42.422275Z","shell.execute_reply":"2022-07-21T07:34:42.499935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What does the Adam Optimizer do?**\n\nAdam is a replacement optimization algorithm for stochastic gradient descent for training deep learning models. Adam combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization algorithm that can handle sparse gradients on noisy problems\n\n**Adam Configuration Parameters**\n\n* **alpha**. Also referred to as the learning rate or step size. The proportion that weights are updated (e.g. 0.001). Larger values (e.g. 0.3) results in faster initial learning before the rate is updated. Smaller values (e.g. 1.0E-5) slow learning right down during training\n\n* **beta1**. The exponential decay rate for the first moment estimates (e.g. 0.9).\n\n* **beta2**. The exponential decay rate for the second-moment estimates (e.g. 0.999). This value should be set close to 1.0 on problems with a sparse gradient (e.g. NLP and computer vision problems).\nepsilon. Is a very small number to prevent any division by zero in the implementation (e.g. 10E-8).\n\nfor a gentle intro abour Adam Optimizer, please follow the below link. it gives you a good explanation about Adam and how it works:\n\nhttps://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/\n\n* Adam is a replacement optimization algorithm for stochastic gradient descent for training deep learning models.\n* Adam combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimization algorithm that can handle sparse gradients on noisy problems.\n* Adam is relatively easy to configure where the default configuration parameters do well on most problems","metadata":{}},{"cell_type":"markdown","source":"**What is Keras backend K?**\n\nWhat is a \"backend\"? Keras is a model-level library, providing **high-level** building blocks for developing deep learning models. It does not handle itself low-level operations such as tensor products, convolutions and so on","metadata":{}},{"cell_type":"markdown","source":"Keras ImageDataGenerator is a gem! It lets you **augment** your images in real-time while your model is still training! You can apply any random transformations on each training image as it is passed to the model. This will not only make your model robust but will also save up on the overhead memory!\n\n**https://www.analyticsvidhya.com/blog/2020/08/image-augmentation-on-the-fly-using-keras-imagedatagenerator/#:~:text=Keras%20ImageDataGenerator%20is%20a%20gem,up%20on%20the%20overhead%20memory!**","metadata":{}},{"cell_type":"markdown","source":"**Load Train and Test data**\n============================","metadata":{"_cell_guid":"22a7fd70-ab61-432d-24cb-93e558414495","_uuid":"62fbd0fe9c338b7ac0b04e688c8ee7947e6170f7"}},{"cell_type":"code","source":"# create the training & test sets, skipping the header row with [1:]\ntrain = pd.read_csv(\"../input/train.csv\")\nprint(train.shape)\ntrain.head()","metadata":{"_cell_guid":"05226b08-226a-1a00-044d-a0e6b2101388","_uuid":"4eff577bcd43479a3b7e91180393cbad9fcfca33","execution":{"iopub.status.busy":"2022-07-21T07:34:42.503756Z","iopub.execute_input":"2022-07-21T07:34:42.504322Z","iopub.status.idle":"2022-07-21T07:34:45.409241Z","shell.execute_reply.started":"2022-07-21T07:34:42.504030Z","shell.execute_reply":"2022-07-21T07:34:45.408377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test= pd.read_csv(\"../input/test.csv\")\nprint(test.shape)\ntest.head()","metadata":{"_cell_guid":"2ec570a6-b41a-2139-5e0e-4941c4f0a9d0","_uuid":"67f0854ad0d812a1395130144a0adef9966fec88","execution":{"iopub.status.busy":"2022-07-21T07:34:45.412581Z","iopub.execute_input":"2022-07-21T07:34:45.413025Z","iopub.status.idle":"2022-07-21T07:34:47.324972Z","shell.execute_reply.started":"2022-07-21T07:34:45.412833Z","shell.execute_reply":"2022-07-21T07:34:47.324153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = (train.iloc[:,1:].values).astype('float32') # all pixel values\ny_train = train.iloc[:,0].values.astype('int32') # only labels i.e targets digits\nX_test = test.values.astype('float32')","metadata":{"_cell_guid":"1ae10fe0-dde9-7659-f53d-1a1bd625cfb1","_uuid":"bdffbed77ce62da528c60e43f2b1bea9f57fcdbc","execution":{"iopub.status.busy":"2022-07-21T07:34:47.326431Z","iopub.execute_input":"2022-07-21T07:34:47.326981Z","iopub.status.idle":"2022-07-21T07:34:47.442122Z","shell.execute_reply.started":"2022-07-21T07:34:47.326930Z","shell.execute_reply":"2022-07-21T07:34:47.441240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train","metadata":{"_cell_guid":"250b1126-ce1d-6d3f-9736-2504f7a1e098","_uuid":"5e3e1e3574c3e019eadfd14e4dda41fd15b4de2a","execution":{"iopub.status.busy":"2022-07-21T07:34:47.446284Z","iopub.execute_input":"2022-07-21T07:34:47.446564Z","iopub.status.idle":"2022-07-21T07:34:47.452781Z","shell.execute_reply.started":"2022-07-21T07:34:47.446511Z","shell.execute_reply":"2022-07-21T07:34:47.451744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train","metadata":{"_cell_guid":"e0f15f8a-ac08-540a-58db-dab989cc687c","_uuid":"4c96cf1c9cdc364ae3faff6b8c3c97aa7fa982d4","execution":{"iopub.status.busy":"2022-07-21T07:34:47.456343Z","iopub.execute_input":"2022-07-21T07:34:47.457195Z","iopub.status.idle":"2022-07-21T07:34:47.463087Z","shell.execute_reply.started":"2022-07-21T07:34:47.456969Z","shell.execute_reply":"2022-07-21T07:34:47.462123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:47.464983Z","iopub.execute_input":"2022-07-21T07:34:47.465697Z","iopub.status.idle":"2022-07-21T07:34:47.472382Z","shell.execute_reply.started":"2022-07-21T07:34:47.465594Z","shell.execute_reply":"2022-07-21T07:34:47.471476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:47.474160Z","iopub.execute_input":"2022-07-21T07:34:47.474866Z","iopub.status.idle":"2022-07-21T07:34:47.481691Z","shell.execute_reply.started":"2022-07-21T07:34:47.474815Z","shell.execute_reply":"2022-07-21T07:34:47.480664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The output variable is an integer from 0 to 9. This is a **multiclass classification** problem.","metadata":{"_cell_guid":"c2c91588-5547-353a-7f92-39600027438e","_uuid":"f01a969286e62fa5ffe37031ed6d4aea947b59a8"}},{"cell_type":"code","source":"print(X_train.shape)\nprint(y_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:47.483532Z","iopub.execute_input":"2022-07-21T07:34:47.484337Z","iopub.status.idle":"2022-07-21T07:34:47.490295Z","shell.execute_reply.started":"2022-07-21T07:34:47.484276Z","shell.execute_reply":"2022-07-21T07:34:47.489438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Visualization\nLets look at 3 images from data set with their labels.","metadata":{"_cell_guid":"60957d82-c76f-4822-28ff-def7011a34fa","_uuid":"da0573528e8c6c3dd2b0e0cf33c600ad3f14466d"}},{"cell_type":"code","source":"#Convert train datset to (num_images, img_rows, img_cols) format \n\n#print(X_train)\nX_train = X_train.reshape(X_train.shape[0], 28, 28)\n#print(X_train)\nfor i in range(6, 9):\n    plt.subplot(330 + (i+1))\n    plt.imshow(X_train[i], cmap=plt.get_cmap('gray'))\n    plt.title(y_train[i]);","metadata":{"_cell_guid":"1541678d-a08b-d2b2-1e1e-eabf882baaec","_uuid":"7998af1ce3c065c4a54a73cd97fed9afebde7e96","execution":{"iopub.status.busy":"2022-07-21T07:34:47.492347Z","iopub.execute_input":"2022-07-21T07:34:47.493201Z","iopub.status.idle":"2022-07-21T07:34:48.116865Z","shell.execute_reply.started":"2022-07-21T07:34:47.493144Z","shell.execute_reply":"2022-07-21T07:34:48.115864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.121460Z","iopub.execute_input":"2022-07-21T07:34:48.121915Z","iopub.status.idle":"2022-07-21T07:34:48.134628Z","shell.execute_reply.started":"2022-07-21T07:34:48.121856Z","shell.execute_reply":"2022-07-21T07:34:48.133385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"list(range(6, 9))","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.139257Z","iopub.execute_input":"2022-07-21T07:34:48.139700Z","iopub.status.idle":"2022-07-21T07:34:48.151270Z","shell.execute_reply.started":"2022-07-21T07:34:48.139538Z","shell.execute_reply":"2022-07-21T07:34:48.149882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#expand 1 more dimention as 1 for colour channel gray\nX_train = X_train.reshape(X_train.shape[0], 28, 28,1)\nX_train.shape","metadata":{"_cell_guid":"6be2f3e9-42eb-85b6-9162-c25e4d706155","_uuid":"4051a0e6612b8e6d4b8aef6a4d131be621cd3a14","execution":{"iopub.status.busy":"2022-07-21T07:34:48.152777Z","iopub.execute_input":"2022-07-21T07:34:48.153541Z","iopub.status.idle":"2022-07-21T07:34:48.166160Z","shell.execute_reply.started":"2022-07-21T07:34:48.153037Z","shell.execute_reply":"2022-07-21T07:34:48.164986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train[:1].shape","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.171111Z","iopub.execute_input":"2022-07-21T07:34:48.171757Z","iopub.status.idle":"2022-07-21T07:34:48.178312Z","shell.execute_reply.started":"2022-07-21T07:34:48.171428Z","shell.execute_reply":"2022-07-21T07:34:48.177520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = X_test.reshape(X_test.shape[0], 28, 28,1)\nX_test.shape","metadata":{"_cell_guid":"6949468c-fd27-19c5-15c7-0b357a961003","_uuid":"6d4c1323f1fa3f89a16532fed893ec5d72051bcb","execution":{"iopub.status.busy":"2022-07-21T07:34:48.179811Z","iopub.execute_input":"2022-07-21T07:34:48.180472Z","iopub.status.idle":"2022-07-21T07:34:48.189993Z","shell.execute_reply.started":"2022-07-21T07:34:48.180423Z","shell.execute_reply":"2022-07-21T07:34:48.189161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Preprocessing the digit images**\n==================================","metadata":{"_cell_guid":"1232c385-3cb2-56fd-4d1d-f027df7bc78e","_uuid":"185d620525e041eb61aabce19e8536614ab50870"}},{"cell_type":"markdown","source":"**Feature Standardization**\n-------------------------------------\n\nIt is important preprocessing step.\nIt is used to centre the data around zero mean and unit variance.\nHaving features on a similar scale can help the gradient descent converge more quickly towards the minima.\n\nWhat is Normalization?\n\nNormalization is a scaling technique in which values are shifted and rescaled so that they end up ranging between 0 and 1. It is also known as **Min-Max scaling**\n\n\nhttps://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/","metadata":{"_cell_guid":"6fcc1f9e-1586-e393-49ba-50c73564e0ed","_uuid":"b8847f48f7408c93ce795db16f30c1b7c6a8cf89"}},{"cell_type":"code","source":"mean_px = X_train.mean().astype(np.float32)\nstd_px = X_train.std().astype(np.float32)\n\ndef standardize(x): \n    return (x-mean_px)/std_px","metadata":{"_cell_guid":"a3f837ef-0373-8d91-46e6-30992cf73166","_uuid":"528a370b381c91b73131a8c7a4217968278696c8","execution":{"iopub.status.busy":"2022-07-21T07:34:48.193054Z","iopub.execute_input":"2022-07-21T07:34:48.194049Z","iopub.status.idle":"2022-07-21T07:34:48.339132Z","shell.execute_reply.started":"2022-07-21T07:34:48.193995Z","shell.execute_reply":"2022-07-21T07:34:48.338413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_px","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.341733Z","iopub.execute_input":"2022-07-21T07:34:48.342251Z","iopub.status.idle":"2022-07-21T07:34:48.348041Z","shell.execute_reply.started":"2022-07-21T07:34:48.342201Z","shell.execute_reply":"2022-07-21T07:34:48.347233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"std_px","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.349585Z","iopub.execute_input":"2022-07-21T07:34:48.350151Z","iopub.status.idle":"2022-07-21T07:34:48.358392Z","shell.execute_reply.started":"2022-07-21T07:34:48.350095Z","shell.execute_reply":"2022-07-21T07:34:48.357610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*One Hot encoding of labels.*\n-----------------------------\n\nA one-hot vector is a vector which is 0 in most dimensions, and 1 in a single dimension. In this case, the nth digit will be represented as a vector which is 1 in the nth dimension. \n\nFor example, 3 would be [0,0,0,1,0,0,0,0,0,0].","metadata":{"_cell_guid":"725c55fc-9742-a63c-9822-c67ab0c773ee","_uuid":"532d3f3bd26b0dfb42bc0c96e9710269234fae9b"}},{"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\ny_train= to_categorical(y_train)\nnum_classes = y_train.shape[1]\nnum_classes","metadata":{"_cell_guid":"c879f076-b3dd-6cb1-e2d9-2f404f2ed132","_uuid":"41bb3082e71111d73dd0432f9f60261f5be05e15","execution":{"iopub.status.busy":"2022-07-21T07:34:48.359901Z","iopub.execute_input":"2022-07-21T07:34:48.360361Z","iopub.status.idle":"2022-07-21T07:34:48.369013Z","shell.execute_reply.started":"2022-07-21T07:34:48.360168Z","shell.execute_reply":"2022-07-21T07:34:48.368076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.370581Z","iopub.execute_input":"2022-07-21T07:34:48.371107Z","iopub.status.idle":"2022-07-21T07:34:48.378644Z","shell.execute_reply.started":"2022-07-21T07:34:48.371028Z","shell.execute_reply":"2022-07-21T07:34:48.377636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets plot 10th label.","metadata":{"_cell_guid":"4d76fb04-57fc-e802-6d91-06ece552686b","_uuid":"429e528f5bf36152cd9e0b2acaa457525a202171"}},{"cell_type":"code","source":"y_train[9]","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.380257Z","iopub.execute_input":"2022-07-21T07:34:48.380785Z","iopub.status.idle":"2022-07-21T07:34:48.387279Z","shell.execute_reply.started":"2022-07-21T07:34:48.380732Z","shell.execute_reply":"2022-07-21T07:34:48.386398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.title(y_train[9])\nplt.plot(y_train[9])\nplt.xticks(range(10));","metadata":{"_cell_guid":"1c927e75-08d2-d539-54f3-71ab0308fec1","_uuid":"b3ad8362611417de16de730cc55a6ff6309766f4","execution":{"iopub.status.busy":"2022-07-21T07:34:48.388686Z","iopub.execute_input":"2022-07-21T07:34:48.389268Z","iopub.status.idle":"2022-07-21T07:34:48.613626Z","shell.execute_reply.started":"2022-07-21T07:34:48.388965Z","shell.execute_reply":"2022-07-21T07:34:48.612868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Oh its 3 !","metadata":{"_cell_guid":"4e130661-9f09-d9a9-d49b-7274ef13927f","_uuid":"40aecbff9b92269d438c384ac429bfa47ab37dda"}},{"cell_type":"markdown","source":"**Designing Neural Network Architecture**\n=========================================","metadata":{"_cell_guid":"6a89dcdd-7b68-6ed1-2c39-b3a1edb3e7be","_uuid":"dc7ece2b7ee08767b664149d67d922d8c1d0bbb1"}},{"cell_type":"code","source":"# fix random seed for reproducibility\nseed = 43\nnp.random.seed(seed)","metadata":{"_cell_guid":"39107235-d87a-af4d-44fb-80c9c3aa0212","_uuid":"1070353d05490ccec23933c62f11cdfd2d7e5032","execution":{"iopub.status.busy":"2022-07-21T07:34:48.614945Z","iopub.execute_input":"2022-07-21T07:34:48.615386Z","iopub.status.idle":"2022-07-21T07:34:48.619821Z","shell.execute_reply.started":"2022-07-21T07:34:48.615330Z","shell.execute_reply":"2022-07-21T07:34:48.618958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Linear Model*\n--------------","metadata":{"_cell_guid":"a8b65f54-398b-267f-e31a-313210450f54","_uuid":"62606ecbb1d7e259850aebf8a8514e54263a2a06"}},{"cell_type":"code","source":"from keras.models import  Sequential\nfrom keras.layers.core import  Lambda , Dense, Flatten, Dropout\nfrom keras.callbacks import EarlyStopping\nfrom keras.layers import BatchNormalization, Convolution2D , MaxPooling2D","metadata":{"_cell_guid":"5dbe450c-845f-aaa2-dbde-21414a91d8c1","_uuid":"5f54b59d89cd4e43dd129d9b133950ba83b5cad8","execution":{"iopub.status.busy":"2022-07-21T07:34:48.621177Z","iopub.execute_input":"2022-07-21T07:34:48.621666Z","iopub.status.idle":"2022-07-21T07:34:48.631064Z","shell.execute_reply.started":"2022-07-21T07:34:48.621577Z","shell.execute_reply":"2022-07-21T07:34:48.629985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before proceeding there are few terms that you should be familiar with:\n\n* **Epoch** – The number of times the algorithm runs on the whole training dataset.\n\n* **Sample** – A single row of a dataset.\n\n* **Batch** – It denotes the number of samples to be taken to for updating the model parameters.\n\n* **Learning rate** – It is a parameter that provides the model a scale of how much model weights should be updated.\n\n* **Cost Function/Loss Function** – A cost function is used to calculate the cost that is the difference between the predicted value and the actual value.\n\n* **Weights/ Bias** – The learnable parameters in a model that controls the signal between two neurons.","metadata":{}},{"cell_type":"markdown","source":"Lets create a simple model from **Keras Sequential layer**.\n\n1. **Lambda** layer performs simple arithmetic operations like sum, average, exponentiation etc.\n\n In 1st layer of the model we have to define input dimensions of our data in (rows,columns,colour channel) format.\n (In theano colour channel comes first)\n\n\n2. **Flatten** will transform input into 1D array.\n\n\n3. **Dense** is fully connected layer that means all neurons in previous layers will be connected to all neurons in fully connected layer.\n In the last layer we have to specify output dimensions/classes of the model.\n Here it's 10, since we have to output 10 different digit labels.","metadata":{"_cell_guid":"5c3f674f-f3fc-9614-f2d4-056c3e3ad633","_uuid":"ff25a88562237e84e44f20b38079f2b44a394d2c"}},{"cell_type":"code","source":"model= Sequential()\nmodel.add(Lambda(standardize,input_shape=(28,28,1)))\nmodel.add(Flatten())\nmodel.add(Dense(10, activation='softmax'))\nprint(\"input shape \",model.input_shape)\nprint(\"output shape \",model.output_shape)","metadata":{"_cell_guid":"a2c27783-3cfa-e907-4749-1e340a513f26","_uuid":"fb79b4558335446a722542c8bc06288e96781423","execution":{"iopub.status.busy":"2022-07-21T07:34:48.632458Z","iopub.execute_input":"2022-07-21T07:34:48.632937Z","iopub.status.idle":"2022-07-21T07:34:48.688284Z","shell.execute_reply.started":"2022-07-21T07:34:48.632842Z","shell.execute_reply":"2022-07-21T07:34:48.687476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.689561Z","iopub.execute_input":"2022-07-21T07:34:48.689985Z","iopub.status.idle":"2022-07-21T07:34:48.698357Z","shell.execute_reply.started":"2022-07-21T07:34:48.689936Z","shell.execute_reply":"2022-07-21T07:34:48.697385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.700303Z","iopub.execute_input":"2022-07-21T07:34:48.700805Z","iopub.status.idle":"2022-07-21T07:34:48.849231Z","shell.execute_reply.started":"2022-07-21T07:34:48.700745Z","shell.execute_reply":"2022-07-21T07:34:48.848267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***Compile network***\n-------------------\n\nBefore making network ready for training we have to make sure to add below things:\n\n 1.  **A loss function**: to measure how good the network is\n    \n 2.  **An optimizer**: to update network as it sees more data and reduce loss value\n    \n 3.  **Metrics**: to monitor performance of network","metadata":{"_cell_guid":"260645fb-61b7-68e9-6826-047b97436c14","_uuid":"2dd7f371688dd2590de94a94814799654595e55d"}},{"cell_type":"markdown","source":"\nDeep learning is the subfield of machine learning which is used to perform complex tasks such as speech recognition, text classification, etc. A deep learning model consists of activation function, input, output, hidden layers, loss function, etc. Any deep learning model tries to generalize the data using an algorithm and tries to make predictions on the unseen data. We need an algorithm that maps the examples of inputs to that of the outputs and an optimization algorithm. An optimization algorithm finds the value of the parameters(weights) that minimize the error when mapping inputs to outputs. These optimization algorithms or optimizers widely affect the accuracy of the deep learning model. They as well as affect the speed training of the model. But first of all the question arises what an optimizer really is?\n\nWhile training the deep learning model, we need to modify each epoch’s weights and minimize the loss function. **An optimizer is a function or an algorithm that modifies the attributes of the neural network, such as weights and learning rate**. T**hus, it helps in reducing the overall loss and improve the accuracy**. The problem of choosing the right weights for the model is a daunting task, as a deep learning model generally consists of millions of parameters. It raises the need to choose a suitable optimization algorithm for your application. Hence understanding these algorithms is necessary before having a deep dive into the field.\n\nYou can use different optimizers to make changes in your weights and learning rate. However, choosing the best optimizer depends upon the application. As a beginner, one evil thought that comes to mind is that we try all the possibilities and choose the one that shows the best results. This might not be a problem initially, but when dealing with hundreds of gigabytes of data, even a single epoch can take a considerable amount of time. So randomly choosing an algorithm is no less than gambling with your precious time that you will realize sooner or later in your journey.\n\nDifferent optimizers are used in building a deep learning model and the factors that could make you choose an optimizer instead of others for your application.\n\n1. Gradient Descent\n2. Stochastic Gradient Descent\n3. Stochastic Gradient descent with momentum\n4. Mini-Batch Gradient Descent\n5. Adagrad\n6. **RMSProp**\n7. AdaDelta\n8. **Adam**","metadata":{}},{"cell_type":"code","source":"from keras.optimizers import RMSprop\nmodel.compile(optimizer=RMSprop(lr=0.001),\nloss='categorical_crossentropy',\nmetrics=['accuracy'])","metadata":{"_cell_guid":"9d1d1af9-b2a8-e3b9-6eaf-100d08fe83aa","_uuid":"4bb75be10b9eec8bcdfe48665f639bf326a5f5fc","execution":{"iopub.status.busy":"2022-07-21T07:34:48.851038Z","iopub.execute_input":"2022-07-21T07:34:48.851566Z","iopub.status.idle":"2022-07-21T07:34:48.894105Z","shell.execute_reply.started":"2022-07-21T07:34:48.851510Z","shell.execute_reply":"2022-07-21T07:34:48.893425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A Comprehensive Guide on Deep Learning Optimizers\n\nhttps://www.analyticsvidhya.com/blog/2021/10/a-comprehensive-guide-on-deep-learning-optimizers/","metadata":{}},{"cell_type":"code","source":"from keras.preprocessing import image\ngen = image.ImageDataGenerator()","metadata":{"_cell_guid":"db3b4be6-4f72-c6cc-65cd-b45978db2462","_uuid":"51f82558d87e95fa5c146b0469ab6c8b42e13bcf","execution":{"iopub.status.busy":"2022-07-21T07:34:48.895845Z","iopub.execute_input":"2022-07-21T07:34:48.896286Z","iopub.status.idle":"2022-07-21T07:34:48.903327Z","shell.execute_reply.started":"2022-07-21T07:34:48.896103Z","shell.execute_reply":"2022-07-21T07:34:48.902604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(gen)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.904828Z","iopub.execute_input":"2022-07-21T07:34:48.905095Z","iopub.status.idle":"2022-07-21T07:34:48.912192Z","shell.execute_reply.started":"2022-07-21T07:34:48.905045Z","shell.execute_reply":"2022-07-21T07:34:48.911388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.913935Z","iopub.execute_input":"2022-07-21T07:34:48.914520Z","iopub.status.idle":"2022-07-21T07:34:48.921881Z","shell.execute_reply.started":"2022-07-21T07:34:48.914447Z","shell.execute_reply":"2022-07-21T07:34:48.920981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:48.929656Z","iopub.execute_input":"2022-07-21T07:34:48.929951Z","iopub.status.idle":"2022-07-21T07:34:48.937736Z","shell.execute_reply.started":"2022-07-21T07:34:48.929901Z","shell.execute_reply":"2022-07-21T07:34:48.936104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cross Validation ","metadata":{"_uuid":"841a6f3b78b607e142f3e18d88bd7957202e4dcb"}},{"cell_type":"markdown","source":"Whenever we build any machine learning model, we feed it with initial data to train the model. And then we feed some unknown data (test data) to understand how well the model performs and generalized over unseen data. If the model performs well on the unseen data, it’s consistent and is able to predict with good accuracy on a wide range of input data; then this model is stable.\n\nBut this is not the case always! Machine learning models are not always stable and we have to evaluate the stability of the machine learning model. That is where Cross Validation comes into the picture.\n\n“In simple terms, Cross-Validation is a technique used to assess how well our Machine learning models perform on unseen data”\n\nAccording to Wikipedia, **Cross-Validation is the process of assessing how the results of a statistical analysis will generalize to an independent data set**.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX = X_train\ny = y_train\nX_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.10, random_state=42)\nbatches = gen.flow(X_train, y_train, batch_size=64)\nval_batches=gen.flow(X_val, y_val, batch_size=64)","metadata":{"_cell_guid":"9071d720-da50-8530-e9f3-1f0c37aac7ff","_uuid":"0cff7e02b1ee8894b4ee9080b9268558aaa4e7c5","execution":{"iopub.status.busy":"2022-07-21T07:34:48.939941Z","iopub.execute_input":"2022-07-21T07:34:48.940406Z","iopub.status.idle":"2022-07-21T07:34:49.278553Z","shell.execute_reply.started":"2022-07-21T07:34:48.940238Z","shell.execute_reply":"2022-07-21T07:34:49.277639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# gen.","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:49.280855Z","iopub.execute_input":"2022-07-21T07:34:49.281397Z","iopub.status.idle":"2022-07-21T07:34:49.286600Z","shell.execute_reply.started":"2022-07-21T07:34:49.281345Z","shell.execute_reply":"2022-07-21T07:34:49.284678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**How do you get data from a generator object?**\n\nTo get values from the generator object, call the next() method on the generator object or loop through the generator object.\n\n","metadata":{}},{"cell_type":"code","source":"batches.data_format","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:49.288339Z","iopub.execute_input":"2022-07-21T07:34:49.288881Z","iopub.status.idle":"2022-07-21T07:34:49.298245Z","shell.execute_reply.started":"2022-07-21T07:34:49.288655Z","shell.execute_reply":"2022-07-21T07:34:49.297415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(batches.batch_size)\nprint(batches.n)\nprint(batches.index_generator)\nprint(batches.image_data_generator)\nprint(batches.sample_weight)\nprint(batches.total_batches_seen)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:49.300308Z","iopub.execute_input":"2022-07-21T07:34:49.300566Z","iopub.status.idle":"2022-07-21T07:34:49.308276Z","shell.execute_reply.started":"2022-07-21T07:34:49.300509Z","shell.execute_reply":"2022-07-21T07:34:49.307173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:34:49.309589Z","iopub.execute_input":"2022-07-21T07:34:49.310050Z","iopub.status.idle":"2022-07-21T07:34:49.318986Z","shell.execute_reply.started":"2022-07-21T07:34:49.310000Z","shell.execute_reply":"2022-07-21T07:34:49.317939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# performing data argumentation by training image generator\nhistory=model.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=3, \n                    validation_data=val_batches, validation_steps=val_batches.n)\n\n# https://www.geeksforgeeks.org/keras-fit-and-keras-fit_generator/","metadata":{"_cell_guid":"20e08e2a-a394-bb70-69f1-be0fdab4f9ab","_uuid":"6b23c282e2772b1ee482596131d6f1d3494c3bce","execution":{"iopub.status.busy":"2022-07-21T07:34:49.320741Z","iopub.execute_input":"2022-07-21T07:34:49.321281Z","iopub.status.idle":"2022-07-21T07:42:56.021313Z","shell.execute_reply.started":"2022-07-21T07:34:49.320998Z","shell.execute_reply":"2022-07-21T07:42:56.020439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_dict = history.history\nhistory_dict.keys()","metadata":{"_cell_guid":"9f344366-c372-0b04-b7e0-860778d4bfd3","_uuid":"6900e38c62028692b9f101b94730c527129675cc","execution":{"iopub.status.busy":"2022-07-21T07:42:56.022668Z","iopub.execute_input":"2022-07-21T07:42:56.022967Z","iopub.status.idle":"2022-07-21T07:42:56.029755Z","shell.execute_reply.started":"2022-07-21T07:42:56.022918Z","shell.execute_reply":"2022-07-21T07:42:56.028957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_dict","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.031358Z","iopub.execute_input":"2022-07-21T07:42:56.031914Z","iopub.status.idle":"2022-07-21T07:42:56.044310Z","shell.execute_reply.started":"2022-07-21T07:42:56.031643Z","shell.execute_reply":"2022-07-21T07:42:56.043608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_dict.keys()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.045675Z","iopub.execute_input":"2022-07-21T07:42:56.045915Z","iopub.status.idle":"2022-07-21T07:42:56.054049Z","shell.execute_reply.started":"2022-07-21T07:42:56.045874Z","shell.execute_reply":"2022-07-21T07:42:56.053162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#type(history_dict)\ntype(history)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.055578Z","iopub.execute_input":"2022-07-21T07:42:56.056063Z","iopub.status.idle":"2022-07-21T07:42:56.065621Z","shell.execute_reply.started":"2022-07-21T07:42:56.056016Z","shell.execute_reply":"2022-07-21T07:42:56.064945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nloss_values = history_dict['loss']\nval_loss_values = history_dict['val_loss']\nepochs = range(1, len(loss_values) + 1)\n\n# \"bo\" is for \"blue dot\"\nplt.plot(epochs, loss_values, 'bo')\n# b+ is for \"blue crosses\"\nplt.plot(epochs, val_loss_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\n\nplt.show()","metadata":{"_cell_guid":"df40f5fc-586a-1fae-025e-ee508a8d9b71","_uuid":"c4b26ff79e0f186212266b60d03611ad58d0d5e3","execution":{"iopub.status.busy":"2022-07-21T07:42:56.067761Z","iopub.execute_input":"2022-07-21T07:42:56.068012Z","iopub.status.idle":"2022-07-21T07:42:56.308445Z","shell.execute_reply.started":"2022-07-21T07:42:56.067941Z","shell.execute_reply":"2022-07-21T07:42:56.307326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(loss_values)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.313805Z","iopub.execute_input":"2022-07-21T07:42:56.314259Z","iopub.status.idle":"2022-07-21T07:42:56.326820Z","shell.execute_reply.started":"2022-07-21T07:42:56.314098Z","shell.execute_reply":"2022-07-21T07:42:56.325903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.clf()   # clear figure\nacc_values = history_dict['acc']\nval_acc_values = history_dict['val_acc']\n\nplt.plot(epochs, acc_values, 'bo')\nplt.plot(epochs, val_acc_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\n\nplt.show()","metadata":{"_cell_guid":"1ed6b756-00c2-d08c-c596-0ce496ec3d04","_uuid":"fc9be5b885360ca9972e0d6b5da1ea36dc12cb5f","execution":{"iopub.status.busy":"2022-07-21T07:42:56.330754Z","iopub.execute_input":"2022-07-21T07:42:56.332447Z","iopub.status.idle":"2022-07-21T07:42:56.584385Z","shell.execute_reply.started":"2022-07-21T07:42:56.332182Z","shell.execute_reply":"2022-07-21T07:42:56.583486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fully Connected Model\n\n**What is fully connected layer?**\n\nA fully connected layer multiplies the input by a weight matrix and then adds a bias vector. The convolutional (and down-sampling) layers are followed by one or more fully connected layers. As the name suggests, all neurons in a fully connected layer connect to all the neurons in the previous layer.\n\nNeurons in a fully connected layer have full connections to all activations in the previous layer, as seen in regular Neural Networks. \nAdding another Dense Layer to model.\n\n\n**What is fully connected layer in CNN?**\n\nFully Connected Layer is simply, feed forward neural networks. Fully Connected Layers form the last few layers in the network. The input to the fully connected layer is the output from the final Pooling or Convolutional Layer, which is flattened and then fed into the fully connected layer.\n\nhttps://towardsdatascience.com/convolutional-neural-network-17fb77e76c05\n\nhttps://www.analyticsvidhya.com/blog/2021/07/convolution-neural-network-better-understanding/\n\nHere below, we are attempting to build the same previous model using FC function ","metadata":{"_uuid":"64ec304e056ec0c9e33fe94ea2315cbf65a7fbff"}},{"cell_type":"code","source":"def get_fc_model():\n    model = Sequential([\n        Lambda(standardize, input_shape=(28,28,1)),\n        Flatten(),\n        Dense(512, activation='relu'),\n        Dense(10, activation='softmax')\n        ])\n    model.compile(optimizer='Adam', loss='categorical_crossentropy',\n                  metrics=['accuracy'])\n    return model","metadata":{"_uuid":"9556f3de5bd370bcddc70a81910eb2104624e3a3","execution":{"iopub.status.busy":"2022-07-21T07:42:56.589579Z","iopub.execute_input":"2022-07-21T07:42:56.592144Z","iopub.status.idle":"2022-07-21T07:42:56.602304Z","shell.execute_reply.started":"2022-07-21T07:42:56.592075Z","shell.execute_reply":"2022-07-21T07:42:56.601077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.608559Z","iopub.execute_input":"2022-07-21T07:42:56.611615Z","iopub.status.idle":"2022-07-21T07:42:56.625937Z","shell.execute_reply.started":"2022-07-21T07:42:56.611550Z","shell.execute_reply":"2022-07-21T07:42:56.624606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:42:56.630478Z","iopub.execute_input":"2022-07-21T07:42:56.632995Z","iopub.status.idle":"2022-07-21T07:42:56.788369Z","shell.execute_reply.started":"2022-07-21T07:42:56.632935Z","shell.execute_reply":"2022-07-21T07:42:56.787587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fc = get_fc_model() # get an instance of fc\nfc.optimizer.lr=0.01","metadata":{"_uuid":"1901b6805f878ac6ed4efafbaf15bf003d505654","execution":{"iopub.status.busy":"2022-07-21T07:42:56.790866Z","iopub.execute_input":"2022-07-21T07:42:56.791414Z","iopub.status.idle":"2022-07-21T07:42:56.877553Z","shell.execute_reply.started":"2022-07-21T07:42:56.791342Z","shell.execute_reply":"2022-07-21T07:42:56.876821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history=fc.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=1, \n                    validation_data=val_batches, validation_steps=val_batches.n)","metadata":{"_uuid":"5fb346c542c8920fac61ddc5df44b2136969a6e9","execution":{"iopub.status.busy":"2022-07-21T07:42:56.879317Z","iopub.execute_input":"2022-07-21T07:42:56.879699Z","iopub.status.idle":"2022-07-21T07:45:51.141522Z","shell.execute_reply.started":"2022-07-21T07:42:56.879639Z","shell.execute_reply":"2022-07-21T07:45:51.140613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fc","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.144818Z","iopub.execute_input":"2022-07-21T07:45:51.145069Z","iopub.status.idle":"2022-07-21T07:45:51.151670Z","shell.execute_reply.started":"2022-07-21T07:45:51.145024Z","shell.execute_reply":"2022-07-21T07:45:51.150680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_dict = history.history\nhistory_dict.keys()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.153311Z","iopub.execute_input":"2022-07-21T07:45:51.154000Z","iopub.status.idle":"2022-07-21T07:45:51.160156Z","shell.execute_reply.started":"2022-07-21T07:45:51.153947Z","shell.execute_reply":"2022-07-21T07:45:51.159235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nloss_values = history_dict['loss']\nval_loss_values = history_dict['val_loss']\nepochs = range(1, len(loss_values) + 1)\n\n# \"bo\" is for \"blue dot\"\nplt.plot(epochs, loss_values, 'bo')\n# b+ is for \"blue crosses\"\nplt.plot(epochs, val_loss_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.161940Z","iopub.execute_input":"2022-07-21T07:45:51.162631Z","iopub.status.idle":"2022-07-21T07:45:51.368658Z","shell.execute_reply.started":"2022-07-21T07:45:51.162582Z","shell.execute_reply":"2022-07-21T07:45:51.367764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.clf()   # clear figure\nacc_values = history_dict['acc']\nval_acc_values = history_dict['val_acc']\n\nplt.plot(epochs, acc_values, 'bo')\nplt.plot(epochs, val_acc_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.370107Z","iopub.execute_input":"2022-07-21T07:45:51.370618Z","iopub.status.idle":"2022-07-21T07:45:51.580954Z","shell.execute_reply.started":"2022-07-21T07:45:51.370570Z","shell.execute_reply":"2022-07-21T07:45:51.580132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convolutional Neural Network\nCNNs are extremely efficient for images.\n","metadata":{"_uuid":"46b81b17854a98f2b380da694691502c1e583bfb"}},{"cell_type":"code","source":"from keras.layers import Convolution2D, MaxPooling2D\n\ndef get_cnn_model():\n    model = Sequential([\n        Lambda(standardize, input_shape=(28,28,1)),\n        Convolution2D(32,(3,3), activation='relu'),\n        Convolution2D(32,(3,3), activation='relu'),\n        MaxPooling2D(),\n        Convolution2D(64,(3,3), activation='relu'),\n        Convolution2D(64,(3,3), activation='relu'),\n        MaxPooling2D(),\n        Flatten(),\n        Dense(512, activation='relu'),\n        Dense(10, activation='softmax')\n        ])\n    model.compile(Adam(), loss='categorical_crossentropy',\n                  metrics=['accuracy'])\n    return model","metadata":{"_uuid":"fd0ab0de6bdbf7addf7515c8d7b59d8d17fef8e7","execution":{"iopub.status.busy":"2022-07-21T07:45:51.582407Z","iopub.execute_input":"2022-07-21T07:45:51.582903Z","iopub.status.idle":"2022-07-21T07:45:51.594091Z","shell.execute_reply.started":"2022-07-21T07:45:51.582856Z","shell.execute_reply":"2022-07-21T07:45:51.592867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model= get_cnn_model()\nmodel.optimizer.lr=0.01","metadata":{"_uuid":"b5baeaed50c9f03c68900461555f4b7831a7e7c7","execution":{"iopub.status.busy":"2022-07-21T07:45:51.595697Z","iopub.execute_input":"2022-07-21T07:45:51.596303Z","iopub.status.idle":"2022-07-21T07:45:51.749156Z","shell.execute_reply.started":"2022-07-21T07:45:51.596253Z","shell.execute_reply":"2022-07-21T07:45:51.748446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.750520Z","iopub.execute_input":"2022-07-21T07:45:51.750798Z","iopub.status.idle":"2022-07-21T07:45:51.760801Z","shell.execute_reply.started":"2022-07-21T07:45:51.750755Z","shell.execute_reply":"2022-07-21T07:45:51.759929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:45:51.761922Z","iopub.execute_input":"2022-07-21T07:45:51.762186Z","iopub.status.idle":"2022-07-21T07:45:51.941313Z","shell.execute_reply.started":"2022-07-21T07:45:51.762144Z","shell.execute_reply":"2022-07-21T07:45:51.940549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history=model.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=1, \n                    validation_data=val_batches, validation_steps=val_batches.n)","metadata":{"_uuid":"a3d15eeead8b26c41b53d084dc111ae9e667a169","execution":{"iopub.status.busy":"2022-07-21T07:45:51.944677Z","iopub.execute_input":"2022-07-21T07:45:51.944942Z","iopub.status.idle":"2022-07-21T07:50:22.845454Z","shell.execute_reply.started":"2022-07-21T07:45:51.944889Z","shell.execute_reply":"2022-07-21T07:50:22.844201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nloss_values = history_dict['loss']\nval_loss_values = history_dict['val_loss']\nepochs = range(1, len(loss_values) + 1)\n\n# \"bo\" is for \"blue dot\"\nplt.plot(epochs, loss_values, 'bo')\n# b+ is for \"blue crosses\"\nplt.plot(epochs, val_loss_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Loss')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:50:22.847009Z","iopub.execute_input":"2022-07-21T07:50:22.847319Z","iopub.status.idle":"2022-07-21T07:50:23.051278Z","shell.execute_reply.started":"2022-07-21T07:50:22.847270Z","shell.execute_reply":"2022-07-21T07:50:23.050236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.clf()   # clear figure\nacc_values = history_dict['acc']\nval_acc_values = history_dict['val_acc']\n\nplt.plot(epochs, acc_values, 'bo')\nplt.plot(epochs, val_acc_values, 'b+')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T07:50:23.052866Z","iopub.execute_input":"2022-07-21T07:50:23.053417Z","iopub.status.idle":"2022-07-21T07:50:23.382261Z","shell.execute_reply.started":"2022-07-21T07:50:23.053356Z","shell.execute_reply":"2022-07-21T07:50:23.377633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Augmentation\nIt is tehnique of showing slighly different or new images to neural network to avoid overfitting. And  to achieve better generalization.\nIn case you have very small dataset, you can use different kinds of data augmentation techniques to increase your data size. Neural networks perform better if you provide them more data.\n\nDifferent data aumentation techniques are as follows:\n1. Cropping\n2. Rotating\n3. Scaling\n4. Translating\n5. Flipping \n6. Adding Gaussian noise to input images etc.\n\nTo dive deeply, please check the link:\n\nhttps://www.analyticsvidhya.com/blog/2021/03/image-augmentation-techniques-for-training-deep-learning-models/\n","metadata":{"_uuid":"e2891c7e434a2022ee182a0e9bd243a876532dcc"}},{"cell_type":"code","source":"gen =ImageDataGenerator(rotation_range=8, width_shift_range=0.08, shear_range=0.3,\n                               height_shift_range=0.08, zoom_range=0.08)\nbatches = gen.flow(X_train, y_train, batch_size=64)\nval_batches = gen.flow(X_val, y_val, batch_size=64)","metadata":{"_uuid":"daa409b92678202cf7c751371b7ba17fb14aa2ac","execution":{"iopub.status.busy":"2022-07-21T07:50:23.386707Z","iopub.execute_input":"2022-07-21T07:50:23.387174Z","iopub.status.idle":"2022-07-21T07:50:23.397621Z","shell.execute_reply.started":"2022-07-21T07:50:23.386980Z","shell.execute_reply":"2022-07-21T07:50:23.396760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.optimizer.lr=0.001\nhistory=model.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=1, \n                    validation_data=val_batches, validation_steps=val_batches.n)","metadata":{"_uuid":"f21ba7b8d77a37bee6e8238a8f517b654ae3f0a0","execution":{"iopub.status.busy":"2022-07-21T07:50:23.402385Z","iopub.execute_input":"2022-07-21T07:50:23.404734Z","iopub.status.idle":"2022-07-21T08:05:23.426062Z","shell.execute_reply.started":"2022-07-21T07:50:23.404679Z","shell.execute_reply":"2022-07-21T08:05:23.423870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Adding Batch Normalization\n\nBN helps to fine tune hyperparameters more better and train really deep neural networks.\n\nBatch Normalization is also a regularization technique, but that doesn’t fully work like l1, l2, dropout regularizations but by adding Batch Normalization we reduce the internal covariate shift and instability in distributions of layer activations in Deeper networks can reduce the effect of overfitting and works well with generalization data. So we can use Batch Normalization as a Regularization technique.\n\nhttps://analyticsindiamag.com/hands-on-guide-to-implement-batch-normalization-in-deep-learning-models/#:~:text=Batch%20Normalization%20is%20also%20a,of%20overfitting%20and%20works%20well\n\nhttps://www.analyticsvidhya.com/blog/2021/03/introduction-to-batch-normalization/","metadata":{"_uuid":"538f504c44e14d389c70b2f35b7225de61b9015d"}},{"cell_type":"markdown","source":"The problem we have in neural networks is the internal covariate shift. When we are training our neural network, the distribution of data changes and the model trains slower. This problem is framed as an internal covariate shift. To maintain the similar distribution of data we use batch normalization by normalizing the outputs using mean=0, standard dev=1 (μ=0,σ=1). By using this technique, the model is trained faster and it also increases the accuracy of the model compared to a model that does not use the batch normalization.\n\n**When to use Batch Normalization?**\n\nWe can use Batch Normalization in Convolution Neural Networks, Recurrent Neural Networks, and Artificial Neural Networks. In practical coding, we add Batch Normalization after the activation function of the output layer or before the activation function of the input layer. Mostly researchers found good results in implementing Batch Normalization after the activation layer.\n\nBy using Batch Normalization we can set the learning rates high which speeds up the Training process. Due to the flexibility of mean and variance for every mini-batch, it provides better learning and increases the accuracy of the model.","metadata":{}},{"cell_type":"code","source":"from keras.layers.normalization import BatchNormalization\n\ndef get_bn_model():\n    model = Sequential([\n        Lambda(standardize, input_shape=(28,28,1)),\n        Convolution2D(32,(3,3), activation='relu'),\n        BatchNormalization(axis=3),\n        Convolution2D(32,(3,3), activation='relu'),\n        MaxPooling2D(),\n        BatchNormalization(axis=3),\n        Convolution2D(64,(3,3), activation='relu'),\n        BatchNormalization(axis=3),\n        Convolution2D(64,(3,3), activation='relu'),\n        MaxPooling2D(),\n        Flatten(),\n        BatchNormalization(),\n        Dense(512, activation='relu'),\n        BatchNormalization(),\n        Dense(10, activation='softmax')\n        ])\n    model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])\n    return model","metadata":{"_uuid":"8b72580fbb06f5f4f769c514cb0d7d2f15aa2c2f","execution":{"iopub.status.busy":"2022-07-21T08:05:23.433282Z","iopub.execute_input":"2022-07-21T08:05:23.436092Z","iopub.status.idle":"2022-07-21T08:05:23.449840Z","shell.execute_reply.started":"2022-07-21T08:05:23.433811Z","shell.execute_reply":"2022-07-21T08:05:23.446779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-07-21T08:05:23.454956Z","iopub.execute_input":"2022-07-21T08:05:23.455195Z","iopub.status.idle":"2022-07-21T08:05:23.465588Z","shell.execute_reply.started":"2022-07-21T08:05:23.455150Z","shell.execute_reply":"2022-07-21T08:05:23.464577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Previous model without BN:**\n\n![![image.png](attachment:986f0b34-2362-4b75-a0ed-f84f66306c1b.png)","metadata":{},"attachments":{"986f0b34-2362-4b75-a0ed-f84f66306c1b.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAjkAAAGSCAYAAAAB2NDdAAAgAElEQVR4nOzdP4+rSP4G+mfv7Asg7OySmeA3UoXsRiRHcmhpEoKNx5WMVGG/hA5LmsCliTcgGclZI52EaJfMJe0EhNyss+YNzHADsA0Yu7Ebu93085GOdGxsqCqq4Ft/cP+tLMsSRERERBPz/3x0AoiIiIiugUEOERERTRKDHCIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSWKQQ0RERJPEIIeIiIgmiUEOERERTdLfh3zIeB5075YAv/yc4Nff+r/3f9/+wh/f++Oof/zrn/jvv//Tu+3Pbz/ih+//69/pz78g+O1XJL0bf8JP+B2/96cG3/76A/3JYT5O5eNkWv/8hh9/+I7+XF4nH1cp84vzcWd1Zyr5YBv4Evn4Cm0AKkImRf82urq/8W9XERER0RRxuoqIiIgmiUEOERERTdLkgxxrPMi4+PA0GPuhSSC6qiKW8FjJiejODA5yPuNFrIglQqvxNHea7yKWtw06hIyAUOKDYy36MFWd87z6n4wxuCpYc97nz0jP+W2gkw+PdZq2LIzXrBsf37k8rUAsDSxQtbFPdm+j4aY7klPEeFQCkZnDefvTVyYgIwH1OPbNij4Da3woESHLMmRZhkgo+J/womqNj/Ui3eUjywzmH9+46G4E0GldN1INKP9TjGAXLxaB6350MuhKxglyihjS6+mpFjFkp7dXxLLVMy1i2Yj+68h6+10Zwza2n9MzsJECdIjmg3vVsXyoBNBhpzf6RlqrKadmb6WR1lP52BIhNBSiT9DoaURFDKMVosYjpCLUCHQKi8MR0v3ruq6FGkgU/G3d2n72VPsoYshuW6pfn2wDpzOClzfr7ktjpKfZBrq9/HPaefu79z06QDuOj0UA2JdmnTwymnmyDhyvO9Z4MHG9X2NhTfWZtwKrfRvQCD0PvkqQKP8KI6Z0F8qBXp+X5Wy1GfTZzWpWbj+6Wc3K5fPrdi/l83K/rdys2vtsvn59Lpezxndfn8vlbFUOS8Fr+bxclrvDHmxrpKGT7mNp3axm5Wy232frs6fy0TzyGWVIE7FZlbPlc9muiptyVdelbp04qCO93y937aPdXur20W0rB23neBs4aXvM2eF3X5+Xrffbbenws+12fSQfB+34VLumj7Wv02VZ1ufx+Llq3iPOudY36051TV6Vm8b3T9W7zo7KZd2uWmmhyRn0Y4BvszBe2PrBwEAXAByIUAOPKYr5HE6RYg2Np7pTa1MNaMBrfxHVNwFAQW7Hw505TDY0PTnyRMA/cyj9VFqrpD3thueFr5CkOQAxIB91Fh4EkJ+XJqLjGiNEjo9FsMbLNbuizhwmy7DrXSOAThtTVirCNjmuGyDJcwAC1fqHavRon/TttlP5SLFOEiS+10pG66t0RxKo3bnq1I0T94jKsWv96boT6BAC6f775ozkigc42xFK/4zv0acySpBjTQjd+FVHa7x9XXN8LOAjshnClzWweGrd+FWU4W5+DPKNtJ5yV/mg++G6CJIcORr35eIFFucH4fdDQGYZwljCj+ybv+ZaxI9Q0Eizan1cEUv4+cBDBRrpXayro7d1A5u9k/eIE95Vd/r32Aqadh3TxIPVKQwXmU3OCGtyqkh4t3CriGFafwPCwVwq6NQgUmIfraPq7emwZ/3Ku7lwA3ukV+vgQTTmijvbjqW1rUBsNJRfNdih+eACty/I8bEINMxujUGB+LGzXsy+7NawPba6rGgFSSfZCAqLRuC0rf/V8dp7PdUGhsvz5M3P7D4nHupAxSLq5rGpmQ/nASJReOQ6nE/urXvEcWfVnUEczE2GVAcIdIosi6CgEGUZA5yJOm8kR4etKZlq9KIKDFTooxqoVNA6wLr5PeFDhXUk33jbmRtEykPY2GkwSjTt4EEkMGmBec++RKgBf5veTu/jSFoBVIvTmuncjrAPykeBdJ1ASDakr8XB3ETIvX3dgYqQ1XXDmUsoFaIa5VeIIoUkbX59DtmsW62/g6MbdU4hqnu71XcUwnrqQEURVGunb7SBXofTDUP/Jo8INQJ/e+0IoLXqTNseyQcEZKoh/UbZQSHKJGerPpUB94gj3q47RKfd6G9XVRdI3HJKp4gh/Rzy7Atif1qt8WDcdwRg1sAzLofeaRwX1+87M5V8ENFdusnv5BSxgQ40wltexZw5nrRFeOZjgddJq4UJLfQTAxwiIqJbGenpqiOsqX7nozUEfTvO3CDKPUR2wAjSFdNqTTUyxClfIiKi27nRdBURERHRbU33zzoQERHRl8Ygh4iIiCaJQQ4RERFNEoMcIiIimiQGOURERDRJDHKIiIhokob9Tk4RQ/rdv3+z/bMOPX8lttqITLontglY4yHs/g2T+g/y5Se2ObvftGltrH6aHifS6jIfl+bjZFrzW+fjSmV+aT7ure5MJR9sA18gH1+jDfAn0j4OfyeHiIiIJokjOcwHe3932vv78vlgG/gC+fgabYAjOR+HIzlEREQ0SVx4TERERJPEIIeIiIgmiUEOERERTRKDHCIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkDQ5yrPHgedU/Y8c5eBFLeAN3ds5nr84aeDJGcd2DwHgS8aCDnPPZArEc+lka3U3qDl2NNfdzHSKiNw0OcoTMkGUpdHDF1NxAM1i7KGArYsjQQj81f8XynCDjozmYPy2w9g14qb6xg7pTIJYePK9xLooY0rvXc7NNb/VPtip8e5s3VnsoYshj+7Sm1ZZHCz52++05D0IiQtjJO1UszHuurURX8OWmq6pgrf4XKWhzTq+6QPyoICKD+Wf+nW5njidtEfIqdEPH604QWJhPcNMs4kesF2nVdlINqMdd0NHalmXIIgEVvbd+FYgf11ik1T6rQ27bq4UJgWh7vCyFtubdgVU1YuwijdTRzwgZQTTyTkAV5IZAtD0fERB+lo4fTdk4QU6rt+Xth+OLGFLGiE3dC7P15xrD9YGLRg+w3SiKWO726Xf+YMi7R2QAFC8WEA/D/66IjaCgEYpzjtLu3TR7h9Z4MHFdJsbu8rTPi8BDbo70VJv7DdH6kynHzkeDM5dQ+v03BRroRN0RUkKs095gu13POyM+u7bV0wY6dWCMeNaZG5hthOb4WAQJ8rx++SCQ7G78BWKjEbjue4+IudkHhY6/QJDkqA7pwg30PlC3EVQi8PDOzoczN8je/FtDAqHGCEHchHTrt02hkWCd8gJDH6w8y2v5vJyVq83pT21W9Wden8vlbFYun1/LzWpWzmarcvP6XC5nq7LavCxns/3+Xp+X5ax+8fq8LGfL5/K1PNx2mKznctn47BupK1ezWTmb1ekZ9J19GpbPfUfZlKvZsuzddGIf7TLZl1O1vU7nLl/NY1Tb9sVx+vi789Hzfn9+aGz9dWffntptpqqXm9WsVedbbaKuM7vtm1WnrjTrdvf1KBnapbP93qzVpsc95OE1oGpDs9a1YhSb1enrQ1/+v7C+a/fm1DWb6EZGmq5qj1aEGrAv2wheQdZdsUCHOOjIqgiyftPxFwjsCwoUSNcJlDzeo2qO8ni+QrLr4b1FQO6G1IHwjPUPeZ5AnN1VbK9V8FWCZNv9RbNM9uW0FzTWbwj4qu452xQ6ODWidOp87LlucGZe6FJv1R0RatjW1GmBFxtAN05yeyQDABSibeNx3f02m0JDI2yN9Fn0VIELWRhfQURy356tgefXU0uphg27a3bee0gDX4l9fut2FSJClmWIhIJ/y7VMzsPhtYxgjQc/l8jMHO5HJ4YII01XWRNCq2g3H39iOnscRYxHBeh6rj5LNYJL9iNC6GDMi/+hIn6EgkaabdcVBNc7WO3m54Pez5lDijXSfKT9Nc5/9W+sdWQWxqvWXsjGXd6mGoF+qo7hzGEiheTIFNz5hzSoDtkIqooU62Qf5AlZBf4pZ5A+hPMgAB3CuCmy+pzkeTLClCXR+4wQ5BR4sdhX5iKG0Se/cJSNFLDw4cDBgwD09oplTXtNTp4jwX7+3UYKSXdnww541jy+6wa9IyKn5HnSWPdjEXXWFg1WxDBawReoe+1rpNv1D7K5Jmf4+bhsZIouMaTuCF9AmXX9ysGDSFrrPmykkCj/7REE10Wgw9PrcOoniM4abSliyJ4AZ6s5Qlm8dA/e8yTZoEPKwwBnp9lByZEnh9vNkTVp71a8wAYuRyu2hA+FAAu/vp4UMYxuvCb6IH8f9rECsfSxuz8nHjQAFWWQwsFcKqjQhwcAUNA6wPrYrrp0CG97E1YRsrq7KWQE5dXbAo00UvDT+nMihA58hPUXA62hBk1WVb3Q3T0/0EiztxYZ7jkPAolJUcz7vpNA+d7+ZaCRmjlEqBH42zwG0Fph4LxaZ58BdGqqC70zx5New6+3qSiFxuM2lQPPh0WqA7jpwQa6gtN1pyZCaGgoCEjUoxPSh7erAhqpGTBJUo+keKHXruvNBbWuiwDbwGTYxMu2M5H07PcgrVCIWm2r6rggqQOTQY1u2ylIdm29OmQKM5/DRDm8RpsLdIp28bhwq0xicC6tgRfujxV6+rDsABTpGomQwx9amDwBmS4gfQ/VwHF1vfrUT6HSJPytLMvyoxPxeVTBXi77e7KfSRHLau78s2fk07i3ulOlZ71I909MXVvftNOVFbGEv14cBCnvY2E8A5c3caK79+V+J+d9HMyfNOxn//2HIsZjaxEnXd/91J1q0f4tA5x62uiWAU79CP34AU615s1u1x8R0V2b2EhOZzqqZcThU2vgpf4nHQUpEMtH4Im90A/xqesO8fwRfS4TC3KIiIiIKpyuIiIioklikENERESTxCCHiIiIJolBDhEREU0SgxwiIiKaJAY5RERENEkMcoiIiGiSGOQQERHRJDHIISIioklikENERESTxCCHiIiIJolBDhEREU0SgxwiIiKapL8P+ZDxPOjeLQF++TnBr7/1f+//vv2FP773x1H/+Nc/8d9//6d325/ffsQP3//Xv9Off0Hw269Iejf+hJ/wO37vTw2+/fUH+pPDfJzKx8m0/vkNP/7wHf25vE4+rlLmF+fjzurOVPLBNvAl8vEV2gBUhEyK/m10dX8ry7L86EQQERERjY3TVURERDRJDHKIiIhokqYR5BQxpGdgb3U8a+DJGMWtjjeqArGUiD9n4j+/T1136CRr4JmbXYWIaIC7DnKs8eB59b/RLh4FYlntU15ypy9iyNBCP83hdN/3Gum99o2sdbxm0GJhmuk4yKeD+dMCa/+GQSFVDurOti42zsWtA/az7NtOf/tpbx+lyR6t5zVrxm1zb7Xj5vG650lIRAgvu65MQOt6fXD+T9edIpa3u3bS11Leq82qnK022xflajYrdy+7Xp/L5WxVHtvc2Gm9n9fyeTkrl8+vZyaq+t5BOjarcnYqfaN7LZ+Xy3Kb/NfnZTlbPpfVy025mu23bfPczevr87JRvnR9fXWnrofL5f78DK7Lt/f63E3n2/XsnUc8Uc/r11ctq+4525SrxvE2q1lPG+q2vy9qszo4V0frTs9neW2isZw1ktOO1Ju9mHaU3ozErfFg4kbvaLetGnFoRfvNXqyQjcfuBHwF2Jei89l6n7468hhql4DMMlz8NJ+NoKARdr5vU41Ap0f3e7TcihhSxojNYe+niOVBj8aabQ/IwdwYzOuhJMdfIEhy5L1HF5CRQrJOW/ty5hJKG05b3cqRugMAQkqIzvnZfe2CurPbfrRXfRlnbmD2lQ6LIEGebxOaQqtov30Up+p5gXQN6FTieg/n5siTAK67fS0gs/3xRHVR6pw3gVADKrrPsbhbKV4sIB52o93H606B2GgouR/dTNcJoNM7Hc2kz2ZwkGONh9BqpFmGLMuQ7Rp7gVj6WC/S+v0MkVDwG1dVrdZYpBmyLIWGQtX+BUIdQKeNUCldI1F+z0XLItUBFn7dSIoY0t/uM0OWagQXZf88xYtFsPDb01QoULXn/ou7NR5CRLuySbVF2AxeEgW13R4paFNtc+YSKlkjbUxDtcqgmYKj5VZz3Z4gSMBXCdZp0f8dGlV/3dkS8MW2XexdWncAC+PnkLu2GgHhyFNgRYp1ouDXlc6mGoH70pgqHX/dV6ueFynWiQDSxjTHSFPa+6mTEFY/4VjcZlPde04df4HgS96k91PlvhKIjvX6WnWnGUhaGM9HLiMoWLzw0kQjGBjkVDfYg3UoAKpKqiAbV4JuDyfYXSgcPIj9iEw1mrC9GNQ9s4OuboFYti821cVOHr34XEueJz3BTI48OfaNAi82aOXpcNRF7S8GrWCk6hHughCbQvfl2ZrTF5QTXDc4+zt0mf66sydCDWuaI3fvqDs2hYZGuAs4QuhRbxoWxlcQUXsUJVE5/G1gFQmoxxHXVvTWc401nupALoW24WgjVtt9LtZ+7xqbIpYIrcZT30XIebji6NI9E/vAOgLC3rVl/XUHeTWK76bvGGkn6vHBC48FfKWRWlTRPRZoD1TsR4nGHQYfkws36EyljcTxF0A9jWFTDeV3Wr818EIgyt4Yss9zJIELd/QU0micOaRYI81H2p/ajwBV/8xInQILU1W6g5tRoMN9PewdPbz0kMfqebNz1e5AjcPBXCokuzm5ShFL+OsFUtPX6SMAgAihg25g3Vd3XLhBAhXmkNs6WrzAQuBEn4BosIFBTjW10T/P7MINNMyut1PNsR4fmu/sOdSwqUWRriFk86JRDV32BTjOg2jM2VY9g2RYRt7FdYOei2h9IVSPPcPzDh5Eu9xspE5PLbW+Xt/4CovUttdzFLEcFuAUMWTYnPPee2t0gcbTX3fahC+gzLp+9Y6647oI9BujGvVTQmc9CVTEkEcCHOG3130dTqH2PEk26JBH6rnjYxHUHaQqQz3TufX0ycVP69TXsv2iHFjjvR3gFC+wX71TYSOopBGoHK07DvxFADTqylnXSKI3DPrbVQAgZAotfXje9h1VX3gczE2E3POx3xQhG9ptdHwsrA/faqRm/3YRm+rvZanGfrfHFBKR8hB6GkAAnUZQfjrgYNXIkErql0m1b9Vz0e5N6oNAYlIU884FTkhkkYHn71OKQCM188NyCzRSM7z5Cl/A80OoKGsFgJFKACR1GWx3ncLMASCB2qUlgE6znl58dVNwhxQbvdvRutMkQmhoKAhI9LS5oXXHmcNECl7Y+JtzdX3cHdt1EQD1KMWw+mijqjOR9O1XSETCg380rdVIC5K6dz/o8nCqnjuYP2lI39tdH1TUrefVKCvqEaUhuSxiCX93gaj2uctGEcNUFyU0m3r3+lGkayRCfrFRnmqUplUvsn19O1l35gaR8fb1nH/riUbEv111lipIyuXnnzcuYgk/l7yY3My91Z0PmAoeOr06ottPLVkYz8BNx5oeJKL3uOsfA7w/Ve/Rhp/8F4OLGI8XLlamS91P3ameHrplgFNPG90ywKkfob/12hlrTj+RRUS3NbGRnM6QaUsAPVbvyhp4qf9JR0EKxPIReGJP80N86rpDJ/HcEt2diQU5RERERBVOVxEREdEkMcghIiKiSWKQQ0RERJPEIIeIiIgmiUEOERERTRKDHCIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSWKQQ0RERJPEIIeIiIgmiUEOERERTdLfh3zIeB5075YAv/yc4Nff+r/3f9/+wh/f++Oof/zrn/jvv//Tu+3Pbz/ih+//69/pz78g+O1XJL0bf8JP+B2/96cG3/76A/3JYT5O5eNkWv/8hh9/+I7+XF4nH1cp84vzcWd1Zyr5YBv4Evn4Cm0AKkImRf82urq/lWVZfnQiiIiIiMbG6SoiIiKaJAY5RERENEnTCHKKGNIzsLc6njXwZIziVscbVYFYSsSfM/Gf36euO3SSNfDMza5CRDTAXQc51njwvPrfWBePIobc7tO74GZfxJChhX6awzm6X+/6N7Kj+bAwzXR4HmQrkw7mTwus/RsGhVQ5qDsFYunBawbotw7Yz7JNb1+92qrr31j1f0B7LWIJz/MwyiXirXZsTaNtdc6TkIgQHimXL6BVNt1zdWndaX/voms2fW3lvdqsytlqs31Rrmazcvey6/W5XM5W5bHNjQ+Wz8tl+fy6/dqynC2fy9fBiXotn5c96disytmp9I3uVD425Wq237Ytu+VzO5evz8tG+dL19dWd6r3lcrk/P4Pr8u29PnfT2axnZbnL42p1Zrs6esS32+tmVc6Wq3LV1y5HOX5zv5ty1Tg3m9Wspw11298X0a0PnXp8ad1pfa8sO/cForedNZLTGllp9WI60XYjErfGg4kbvaPdtipqb/W+mr1YIRuP3Qn4CrAvReez9T59deQx1C4Hc2Mwr4dgHH+BIMmRDy6ACAoaYedpQJtqBDrFsacEj5ZbEUPKGHFj+7Y8ilge9CKt2faAzsmHgIwUknXa2pczl1DasFd0K0fqDgAIKSE652f3tQvqzm67d2TbhZy5gdlXOiyCBHnePOQjlIgg/fcfqz7IG/XcwoQW+imEO9YhW3LkSQB3t3MBmUnsrkrVRalz3gRCDajoPsfiribPkSi5O1dV/bDYXrIvrTvOg0CiHuvrVIHYaAT7E0L0psFBjjUeQquRZhmyLEO2a+wFYuljvUjr9zNEQsFvXFW1WmORZsiyFBoKVfsXCHUAnTZCpXSNRPk4vA9YpDrAwq8bSRFD+tt9ZshSjeCCzB8/3pHPv1gEC789TYUCLxYQD07vd6zxECLalU2qLcJm8JIoqO32SEGbapszl1DJGmljGqpVBufkw3V7giABXyVYp4xybqG/7mwJ+GLbLvYurTuAhfFzyF1bjYBw5CmwIsU6UfC3la6I8agEoiv+Hki3nlsTwuqn/Y11rOPU01+ed3r/NtW959TxFwh0eqdTjlfiuu082wgqaQcyO+fUHSGRpQusfQ+e5yOX2T5YIhpgYJBT3WAP1qEAqHo7CrJR8bo9nGB3oXDwIPYjMtVowrZhFEjXgD7o6haIZftiU13s5PsubtbAP/OinOdJTzCTI0+OfaPAiw1aeTrsjap9GlrBSNUj3AUhNoXuy/MF+dhy3eDs79Bl+uvOngg1rGmvRbi47tgUGhrhbiQnhMa+V/1+FsZXEFGjo/O4xiKVgzsM5x+yU8+tQYjoKjc8Z27q4DDFYu33rh8pYonQajz1Hd95uF453Ctnjidt93UudaGD5ijY1pl1xxp42w5tqmHDY+t5iPoN+sXj6xHwVYjUSgg3xRoLPLVX8+5GiUa9mFkDLwSibIyLsgs3ANYv1TTSmBx/ATymKOZz5KmG8rP2B4bmI8+RBC6eRk0djcqZQwqJNF+Ms7+r/cqqhakq3X56tu61w/dan/Q9QDU/d/Ehu/W8mrZAAni68bnEgw40UtPXGbuEg7lUUGkONFpYEUv468WIx5mGKjjcvrIwnkB74Pn8uuPXSwGqy/8cJlLwTHVNZNnTEANHcqqpjf55ZhduoGF20XU9b3p0aL6z51DDphZFuoaQzYprYbz+AMd5EMBuBKjqGSTDMlINRV8Y4Lhu0F4XVKUGc6ka88btbQ+iXW42UsOnyJw5pFgjLSxS217PMTgfRQwZaih5eFF4a3SBxtNfd9qEL6DMun71jrrjugh0eHodTv0kzFm94iKG7N6kgGpKYTc1Vk2dIaimtvef63mSbNAh++q5g7lpHC9LoYPqppiZ7jXkPU96Ha4BscZ7O8ApXmAD90rrhD6D7eh7uD9n76g7SWPOq3j5UpOANILBIzlCptDSh7cLuFV94XEwNxFyz8d+U4Rs6MiL42NhffhWIzX7t4vYVH8vSzX2uz2mkIiUh9DTAALoNILy0wEHs4hUAiCpv1sJ9LCRIudBIOnrRQiJLDLwmr2Rukd5UG6BRmqGh1fCF/D8ECrKWhfv4/kAgARql5YAOs16pvaqKUh3SLHRux2tO00ihIaGgoBET5sbWnecuscbNv7mXHeEw3URYHsDGVYfbVR1JpJT+z2eKDwIAEk9bTbo8vC+9rodZUU9jTckl0Us4atk91pFGXZFXsQw1UWpNfDQHa0q0jUSIb/YSEM16r4tula54fK603/f4SgODce/XXWWqiHncoQh+A9WxBJ+LvmH427m3urOlaaCTxl1mniY208tWRjPwE3N6Auiieh8d/1jgPfHwfxJw4af/AepbvAkDHXdT92pnh66ZYBTTxvdMsCpH6G/9dqZaz3xRUSXmdhITrWwTfduC6DH6l1ZAy/1P+koSIFYPgJP7Gl+iE9dd+gknluiuzOxIIeIiIiowukqIiIimiQGOURERDRJDHKIiIhokhjkEBER0SQxyCEiIqJJYpBDREREk8Qgh4iIiCaJQQ4RERFNEoMcIiIimiQGOURERDRJDHKIiIhokhjkEBER0SQxyCEiIqJJ+vuQDxnPg+7dEuCXnxP8+lv/9/7v21/443t/HPWPf/0T//33f3q3/fntR/zw/X/9O/35FwS//Yqkd+NP+Am/4/f+1ODbX3+gPznMx6l8nEzrn9/w4w/f0Z/L6+TjKmV+cT7urO5MJR9sA18iH1+hDUBFyKTo30ZX97eyLMuPTgQRERHR2DhdRURERJPEIIeIiIgm6csFOdZ48Lzqn7F9nygQSw+ejFHcOnFFDOkZ9CbrGqz5mHyOoYgh6/N4LA/Vub5heQ7xmcucTrMGXv9FhYg+yJcLcoTMkGUpdHC7YzYDq/EugnUw5nmQ8QW3zCKGDC300xxOc3+N9I0TJFgYr5H/S9Pb5cxhsgxZpN6/r6GagZUncXY2jpV5s4xvHeieqYjl0cDy5vX8veej93DNffbk05pGXe6cJyERIRynfn9K7bbOeI/uwZcLct7mYG4yZGZ7I3onaxAiQpZlyLIISocjNH4L4/nI5aXBWoH4UUFEBvNWJhUU0vrCbZFCYZwQIoBOs6oMUg0o/yYXwCqglRjnuYYC8eMaizofqQbU4zkjMsfKHAgCC/MJbn/XzvcAACAASURBVIzWePBzibSv0t28nr/3fByxDZ6zujMEhWiXDwsTAlG9PVIaYSeTQkYQ6nGcgOtTKRDLEIiyXR1AOFLgSfQOg4KcIpaQcVxF6TKGrXtzux7L0d5PHdl3RwcGDNdXx7S7XtxBT63Vo+r0GjrpGdyzau6zc/GyxoOJj0+RtHqxzfQK2Xh8UMBXgH1pfLOZVl8deZy2S0BmGS5+KtFGUNAIW9/PkSeA7wOpBWDT6sWJPG6LqOrdN85Pnafem5zjYxE0yuDkuSoa5/+MKcRTU1lFDCljxMemLTv1ap8mB3OzD1Acf4EgyZEPSQ9wpMwrQkqIdXpiyq1n5OCtfHTKdYygUsjs+KOwb9Xzy454op6/83wMkiNPArhuMz37oFlUmeycN4FQAyoaocA/k279tik0EqxTRjn0wcoBXp+X5Wy2LJ9fN+VqNitnq0313mrT+/nNalbuN1XfWW3KstysytnyuXwdfMz9flrH26zK2WxVbvYfLpezZfn82vl/9/j7vZfPy+57nWN3Nm5Ws7oMDr//+rxs5OvUvjflqpm2blpfn8tlM19vqo61fB5Soo1vPS97vrMpV7NVuSk35Wq1KTerVbkpX8vnZbMsWzspl41zuS+Dbv5P5PnkuTrM22Y1O6xzp+pU37bX53I5m7Xr0vYznfScquOnth37/GGZ78tq12YadaCb31Y9O5WP3bncFUTn9fu063ufzjl//xHfrOfnno+TR6uvPbPZ6WNuVke2n92OP79m+W/rx2bEc0J0qeHTVUrWvaYA+qA72p6LDXWzFycgUw0bevBCIDpnGkhFu16c4y8Q1L2m4sUi0OF+GsLxsQgS5DmAPEeyS2t1/HF6lUCgn+r9OngQp/d5uK0azrW7fQBFuu6k9TbyPIF4OHZQgdA1MG54MM2zW4+xHXVq9JyduUEkFHzPx3qRdnrfCZS//d4ai7TugZ88VznyREE2Cqe/53wJhWibQNc9PQLQdzxr4Cux38cAp8scEKGGNc1RpwIvtt3WDkcrjuTDptDQCHdtMoSGxQhNYIDDen51F5yPU5y52U1XLdZ+70hwEUuEVuOpL5POw0hTpJ/PdkozM3O4H50YIgz8xeO3WBNCN37V0RoPpvuhIECQ1BfaG9/Ur83xFwiUgu/Vb6gIWeviVyCW1c3f3DqiGap4gQ1chKgu8tX5a0+rPSpAp1l18ypiSD8/2E0QBEjyHGhd5gPo9HAtyt1xfCwCBbU/kYiyTlBuTRWsj7bWZ3vsOaSQSPPFOPv7kF9Z/YB6fq3zAQBwMJcKKs3RrM9FLOGvF0jHWrc3Ac6DAFQIo9PdtS/PEwTu0wenjL66ERYeF3ixQLCduC5iGN3cbmH8NRZPBiYSFy8OtJECFj4cVA0qUdF+fYKNoBIFX6Dq0WrTWh9idICFf73LkY0UxG7BXXfdQrV4su/C7zwIQO8X+prBa3Lex3WD80e28hwJBLaDETZqp7WIJUJEMOYJ2g5cdHryXLlwA91YkFsgNhpBXQeuwkZQItqfx86Ns4jlxTfUIWUufAFl1vUrBw8iaa3tsJFCovy3j+26CN5a+FuvPRrvSaDj9bzS8yTZO719PuxuHeFluazr3H5RTjVS8VaAU3cY3GPbp0j4UGhcZ29w3SUaYoSRnLq3E/qo+r8KWgeoLtUWxgthdQrpoHrEUnjwPTXsRqFDeNuAqTk6IiRSLeF72x53NVIggOrpiCiHt+uNAyqqRx/qnqZK6g1J9Te5VFQtbixiCX+/sTr2gB6xCDWM78Frvll/r4hN9Xe/lN/YXudfSETKQ+jpOg8RlJ++VSo9+aj2vc3HW5wHgcSkKOZDeqL1NKAIoQO/TisQaA1VT5xU5SYQZdXB508a0vcgdQozP5WQU+fKwdxEyL1GuTXrgDXwwn007XsAAl3dfE5tO5VVEUIbH177RNZ11SJSCYBkVwZVOQwbtRhU5iKEhoaCgAQgZAotG+kJNFIz5ATPYSIFL2z8zblu/l0XAdAz6nZct334VaV7u55XicKDAHDWaO6pej7kfLhwq0xiaC7beayOtSvyXQeuMWqLw3ZXpGskQn6xUR4BmS4gfa9+IvOTjN7S9H30oqBjxlxIeG0HCxAPFtTem9MLr7+sg4XKly3s7ndvZT5m3gbqPjBwA28vkh7b2Iuuieg9+Ds571ZN17V0pnbuj4P5k4bl71i0FIcnEnmCkwuGh7ufMq8WkN9y7Uw9bXS1tTM96kfob712xpobL7omopM+7K+Qd4eFW1SE1DXVKv3P8CfqixiytZ7mvUO11TSf7t024jCwNfBS/3OU8U10pkcwfDpqMJb5dPHcEt2dDwtyiIiIiK6J01VEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSWKQQ0RERJPEIIeIiIgmiUEOERERTRKDHCIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSWKQQ0RERJP09yEfMp4H3bslwC8/J/j1t/7v/d+3v/DH9/446h//+if+++//9G7789uP+OH7//p3+vMvCH77FUnvxp/wE37H7/2pwbe//kB/cpiPU/k4mdY/v+HHH76jP5fXycdVyvzifNxZ3ZlKPtgGvkQ+vkIbgIqQSdG/ja7ub2VZlh+dCCIiIqKxcbqKiIiIJolBDhEREU3SNIKcIob0DOytjmcNPBmjuNXxRlUglhLx50z85/ep6w6dZA08c7OrEBENcNdBjjUePK/+N9LFo7VPz8PZuy1iyNBCP83hdN9v7PfqN7LW8ZpBi4Xx2nmUrYjGwfxpgbV/w6CQKgd1p0AsPXjNAP3WAfuZilgerd+7baPW/20Zdesx0K3r48UX+2Me2291Hel0FoREhLAnnV/F8HPl9dbx+jOtutM+F333gVa9Y5BJXeW92qzK2WqzfVGuZrNy97Lr9blczlblsc0nj7F8Ll8Hf+G1fF72pGOzKmen0je61/J5uSyf64S/Pi8b+diUq9l+27bsls/tXL4+LxvlS9fXV3eq95bL5f78XFqXb2CzmpWz1aZT33YbW++NU7+27b4up1Yd7pZnt96/75jd9tJU5W115HhjpeOzOXWuqrrTrA6vz8v+87lq16NtnWvup/m9zWp25jWcvpqzRnLaoyDNSLwTbTcicWs8mLgx6rDbVkXtrcC72YsVsvHYnYCvAPtSdD5b79NXRx5DPa14sYB4aI/InCyACAoaYedpQJtqBDrFsacEj5ZbEUPKGLE57DUWsTzoDVuz7SE5mBuDeZ1wx18gSHLkvUcXkJFCsk5b+3LmEkobTlvdypG6AwBCSojO+dl97YK6s9s+8iiHkNmRR2ELxEZDyf0IVbpOAJ2+c0RKQGZZf7vqlqdNoZFgnb6zQtsUWkUw8yNXhSLGoxKIpH80zaEGVPTVRhROnCsArhtAh9v6axGpBOJhX8ZF/AglIrSLtcCLDaAbjUb4Ckme168sUq0Qmfnwazh9OYODHGs8hFYjzTJkWYYsk6iqXoFY+lgv0vr9DJFQ8BtXVa3WWKQZsiyFhkLV/gVCHUCnjVApXSNRPg7biUWqAyz8uioXMaS/3WeGLNUIBmd5P2zqK4HojN8vKF4sgoXfaVAFqlipv5lZ4yFEtCubVFuEzeAlUVDb7ZGCNtU2Zy6hkjX21+xOGTRTcLTcaq7bEwQJ+GqEmwIN0l93tgR8sW0Xe5fWHcDC+Dnkrq1GQHjNKbAceRLAdetjez5yGUHB4uVK1avZQSliCc+4iHTQuAFexqYagfvSmFppTkkViB8VRCSPtzXUnY53B3jT4swNsggI62DdTRsB0S5w7JaqgwfRvEZVwTTsS1XPbQoduHhpBPpfd6qQjhkY5FQ32IN1KACqC5yCbPR8RDXssrsYB/qpHnVw8CD2IzLVaML2YlAgXaMVtaN+P5Yh7G4f25u6xLHO1mlif/GPgPCM9Q95nvQEMzny5Ng3Dnsih6Muat+4W8FI1SPcNXCbQvfl2Zqzg7Ut1w3O/g5dpr/u7IlQw5r2WoSL645NoaHrG4oHzwuhrxhw7OTVSGzrBnZl1njwc4nMzOGOtM9E5fB31wgB9VidlyJ+xHpxfMR2x3k4GQR9RUUs4YVAtC1Tfzu6WCB+XGOR9geOQkYQyq/r8SOwUO3R90Qh9/fBvFCPHJ2mlkG/eHw9Ar4KkVoJ4aZYY4Gn9mre3SjR0eHjdx0+hA4eq4v/xbt34QbA+n076eX4C+AxRTGfI081lJ+1P2BNfeE43bNEniMJXDyNmjoalTOHFBJpvhhnfzf9lVUXbpBAhQJRZqq6WLzAQqBn4HEUzoMAVAijU2T1tSHPEwTu+2t5oMN9e9oFjxapSpDAh9f8sO/xF23fVE1fqqiuG0Ii1RZ+aiGRQiVJVY4NVbFmkKKeBtvuKZYI3HD/waA5BVzVwzwHGGXS1sCRnGpqo3+e2YUbaJhd+FwNKR4fmu/sOdSwqUWRriFkc6SoGvbuC3CcB9GY77cwF67JgY2gEoETHewW1w3a64Kq1GAuFZLeHkQ13NosNxup01NLra/PIcUaaWGR2vZ6jn3P6I0Ap4ghw+Z6ib23RhdoPP11p034Asqs61fvqDuui0CHp9fhWDPi8L4DfxEAjbQdprXnSbL3ED4U2lPY5mA6t+9pnbd2216/tp8KbowA16MGCgF02rNOqXiBDdzRRpamoln/8+3wt5CNMq2mXRFUyyIO4sZ61Ho3ayD89pR+kWKdKPgMcKhh8EiOkCm09OHtAm5V32AdzE2E3Gv0cFS06129yfGxsD58q5Ga/dtFbKq/l6WaPaf6mEIiUh5CTwMIoNMIyk8HHMzCeOH+73AFGmk2fNGa8yCQmGpkpfUdIZFFBl6zNxJopGZ+WG6BRmqGt0LhC3h+CBVlrQAwUgmApC6D7a5TmDkAJFC7tFQX4sPTUU1BukOKjd7taN1pEiE0NBQEJHra3NC648xhIgUv9Np1vblA03URAPUalmH1sYglfJXUr5Kq812PYjhzg8h4+7QejG5UU9VI7Bkjp9VI7v6Q1bVg18NPF5C+B1VlEDo1nXpejbKinsYblEshEQlvP7BwZnsF6sBIyC+2GPbUueq5RwwtV2vghXr/ndb1WkBGonHdreoAYxxq4t+uOkvVkHN5uzUH11LEslrL8Nkz8mncW9258lRwn6HTqyMqYgl/vWgHeFdlYTwD9yDgIqKPcNc/Bnh/HMyfNGz4yX8x+OjTDHQ991N3qh9Pu2WAU08b3TLAqR+hv22AA1jTfkiCiD7WxEZyOtNRLX3D2ZcexsBL/U86ClIglo/AE3uaH+JT1x06ieeW6O5MLMghIiIiqnC6ioiIiCaJQQ4RERFNEoMcIiIimiQGOURERDRJDHKIiIhokhjkEBER0SQxyCEiIqJJYpBDREREk8Qgh4iIiCaJQQ4RERFNEoMcIiIimiQGOURERDRJDHKIiIhokv4+5EPG86B7twT45ecEv/7W/73/+/YX/vjeH0f941//xH///Z/ebX9++xE/fP9f/05//gXBb78i6d34E37C7/i9PzX49tcf6E8O83EqHyfT+uc3/PjDd/Tn8jr5uEqZX5yPO6s7U8kH28CXyMdXaANQETIp+rfR1f2tLMvyoxNBRERENDZOVxEREdEkMcghIiKiSZpGkFPEkJ6BvdXxrIEnYxS3Ot6oCsRSIv6cif/8PnXd+Rqs8WBudjEhomu66yDHGg+eV/8b+apTxBKed8HFrIghQwv9NIfTfd9rpPfaN7LW8ZpBi4VppsPzIFsRjYP50wJr/4ZBIVUO6k6BWHrwmgH6rQP2s3Tr1jjp3LbF/vr6Hs309gT22zbUuQgIGQEhOwLnal2ve6+tx87Hth30X+tP3QeuV3doMsp7tVmVs9Vm+6JczWbl7mXX63O5nK3KY5t7971clavliX32H6h87vvOZlXOTqVvdK/l83JZPr/Wr56X5Wz5XFYvN+Vqtt+2Lbvl/o39d26XYOqtO9V7y+Vyf37Orcs3tFm10//6vDyoV2d7fS6Xu7pbllV9HSH/r8/l8mSbrNrJanWkHWxWjTZFZ+uW34nzsVnNWudgs2pcr07dB65Vd2hSzhrJaUfqzV5cJxJvjGJY48HEjVGH3bYqqm8F5s1erJCNx+4EfAXYl6Lz2XqfvjryGGpvLmBCC/0Uwj0n8wBgIyhohJ2nAW2qEegUx54SPFpuRQwpY8TmsPdTxPJgNMiabU/FwdwYzOuhJMdfIEhy5L1HF5CRQrJOW/ty5hJKG/ZWb+VI3QEAISVE5/zsvnZB3dltP9mrPp/rBtDhNg0WkUogHpw3vvUG5wEiUXisK2IRG+jAPb9tdhTpGjjZJkNY/YTw2IFECA2F6D6H1O5e8WIB8bAb7T5+Pgq82AC60TCEr5Dkef3ixH3gSnWHJmZoNLRZzY70bOreaKNH14zMN6tZOduNLLR7s93RhOOjC53RidfncnnwelgEv+8lHBmVOaG/53p6P91eSmvUpe7d7La3ej99IzLN1+10tXs7nc8dKZ9Wj4mu6q26sxslaZyr99Wd5vkesYdbj1rOjtTFy1TlMJvNRhpdrMv1uS6j2aycNfPfKKtTI5oc7TxXNdJyUN5vnI/2daiuC733mr5r4Nh1h6Zm4EiORaqDw3UoAIAceaIg5/stogq3dz3TQD/Vow4OHsQ+Eq9GE9K6Z1ggXaMV0aN+P5ZVr2t7iCJdI1ES83M7kdYgRARz9hcred7Xc82RJ8e+cdhLORx1UYi2PRXXbWwTCDWwTutStCl0X56tga/Efh9ncN3g7O/QZfrrzp4INaxpjty9o+7YFBoa4W4kJ4SGRXMg9BJFLOGFQJRlyCIB5Y8xQmRhPB/rRYosS6FtONp6Nr0GnrIMWZYh1RahsdXxQiAyfdeyNueBP+B2HgFZl3cWAWFnzVb/+ajWQAnl13X1EVio1ihQ5fA+cM26Q9PxwQuPBXylkVoARYo1FvDbq3kRy6oSXxqYtPZlNKDDujH5UAmgw/cuEnbhBp2ptJE4/gKopzFsqqH87jyZqW86Eicvx3mOhMO4982ZQ4o10nyk/amoutns/pnzOwUtBdJ1AhXVdU3IquOTvjPKsSl0oPE0d1BNw0ZQyRrpCM1JyX0g41S9K/x/sWkFgH51EehfmEyXEyF00A6s+85HtbkRHGUGPiwC123s7Mh94Ip1h6ZjYJAj4KsEqneC2oUbaJjdFaIKJoKF/2ZPCah7sKlFka4hZLN3tY/SuwGO8yCA3QiQhRm0JsfB3DQv+il0AKgoQzagVwdUIx+HwYyDuVRI1GPPRdLBg2iXm40UEuWfDkp2X69vfIVFatvrOfa96jcCnCKGDHXrArP11ugCjae/7rQJX0CZdf3qHXXHdRHo8PQoizUXPY3SzEN+MITZ87TYEM3RqeLl8Ltnp9WBv2gHYDatrkn/79y0gr+0ugj0BoHFS/dmS4PZCCoReKhH8I+dj4OrTz0yvZ8ZOH4fAPB23aEvb9DfrgIAIVNo6cPztu+o+gZbRdC552O/KUI2tNvo+FhYH77VSM3+7SI21d/LUo39bo8pJCLlIfQ0gAA6jaD8dGhWLuY8CCQmRTHvBAxCIosMPH+fUgQaqZkfllugkZrhw+DCF/D8ECrKWgFgpBIASV0G212nMHMASKB2aQmg06ynF19NQbrXLzbCibrTJEJoaCgISPS0uaF1x5nDRApe2Pibc3V93B3bdREA9QLPIfWxp50fpKeajkZS9+AH9XIkUi3he41WHnXq69lpBZz5U7vszrkmAdiOXAnJTsAwFsYL2/Uta4zcnDof1sALde/33roPvFl36Mvj3646SzVsmsvs6FMbn0URS/i55B+Ou5l7qztjTgU3DJ1CPcuV0nqKNfCM2w4MiejTuesfA7w/DuZPGvaz/1BYEePxwsXKdKn7qTvVD6iNHTTUP/Q2coBznbS+ZfszEwxwiD67iY3kdIZMWwLo9L2LL7eHMfBS/5OOghSI5SPwNFJZ0Hk+dd35GqzxkPr3MuJGRO8xsSCHiIiIqMLpKiIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSWKQQ0RERJPEIIeIiIgmiUEOERERTRKDHCIiIpokBjlEREQ0SQxyiIiIaJIY5BAREdEkMcghIiKiSfr7kA8Zz4Pu3RLgl58T/Ppb//f+79tf+ON7fxz1j3/9E//99396t/357Uf88P1//Tv9+RcEv/2KpHfjT/gJv+P3/tTg219/oD85zMepfJxM65/f8OMP39Gfy+vk4yplfnE+7qzuTCUfbANfIh9foQ1ARcik6N9GV/e3sizLj04EERER0dg4XUVERESTxCCHiIiIJunLBTnWePC86p+xfZ8oEEsPnoxR3DpxRQzpGfQm6xqs+Zh8jqGIIevzeCwP1bm+YXkO8ZnL/Iuw5ti1gYg+my8X5AiZIctS6OB2x2wGVt5YV8/mTd6TiM+9axYxZGihn+Zwqjeq4K6RvnGCBAvjNfLveZBnJ7aHM4fJMmSRev++hrpWmTfL+NaB7iW25TByJFDVtwvK9fgeG3WvZ79H8iFkBIRjpuOraLd1Bop0D75ckPM2B3OTITPbG9E7WYMQEbIsQ5ZFUDocofEXiB/XWKQZsixDqgH1eM7oQIH4UUFEBvNWJhUU0voGa5FCYZwQIoCu05qlGlD+TS6AVUArMc5zDdcqcyAILMynuaNaGH8NoYJR91rEEuFo9Q11ABMCUV3vsm65n8qHgIzEmef3qysQy2Z5M1Ck+zAoyCliCRnHVZQuY9hYtnvkrR5uc/qgjuy7owMDhuurY9q6p9vTE7OmNTrQuml20jN45KC5z85d2BoPJj4+RdIarWmmV8jG44MCvgLsS+ObzbT66sjjtF0O5mZ/0Xb8BYIkRz4sl4CNoKARtu7+OfIE8H0gtQBsWr04kcdtERWxbJ+fOk+9gYzjYxE0yuDkuSoa5/+MKcRTU1lFDCljxMemLTv1ap+ma5R5RUgJsU5PTLlt/3VGfE7lo1OuYwWV1oSw+gmhO87+AABFjEclEEn/7c8O3WW6BnSKY0/uvpkPEUJDIeJoxDDd+m1TaCRYp4xy6IOVA7w+L8vZbFk+v27K1WxWzlab6r3Vpvfzm9Ws3G+qvrPalGW5WZWz5XP5OviY+/20jrdZlbPZqtzsP1wuZ8vy+bXz/+7x93svn5fd9zrH7mzcrGZ1GRx+//V52cjXqX1vylUzbd20vj6Xy2a+Bjp1Lo59fvncPQubcjVblZtyU65Wm3KzWpWb8rV8XjbLsrWTctk4l/sy6Ob/RJ5PnqtqP810blazw3yeqlN9216fy+Vs1q5L28900nOqXMcp831Z7dpMow5089uqZ6fysTuXu4LovL5Q4xjn5v+4Zn3p1JX37vO5LqPZrH29GJiP8fI4fc2y2tbTDcuP7sDw6Sol615sAH3QHW3PxYa6OVohIFMNG3rwQiA6ZxpIRbuemOMvENgXFACKF4tAh/tpCMfHIkiQ5wDyHMkurdXxD0ZPLhTop3q/Dh7E6X0ebquGc+1uH1Vvs53WC1gDXwlEZ/zYVJ4nEA/HDioQugbGDQ+meYp6BG836tQYyXDmBpFQ8D0f60W3B51A+dvvrbFI6xGRk+cqR54oyEbhiGrjCFMIal9ernt6RKbveKOXOSBCDWuao04FXmy7rR2OHh3Jh02hoRHu2mQIDYv3NQELc277HaCIH3vqyzj0GnjKttOLFqGxOCcfzgN/wO1c1njwc4nMzOF+dGKIMPAXj99iTQjd+FVHazyY7oeCAEFSX2jHvEreAcdfIFAKvle/oSJkrcilQCyrm795V0TTYU0VOI6x7qR4gQ1chKgClur8tafVHhWg06wKSooY0s8PdhMEAZI8B1opCqDTw7Uod8fxsQgU1P5EIso6N8Mxy7x17DmkkEjzxTj7G/lXVovYQCMBPN14N4Gn33NuLSKVIIEPr/m2742SfiX35855EMD6Bf9fvL5CPsh5EIAKYXS6u/bleYLAffrglNFXN8LC4wIvFghct34ZwzSvH/UCv8WTgXnHYj4bKWDhw0HVoBIV7dcn2AgqUfAFqh6tNq31IUYHWPjXu3rZSEHsFtxlnYuzhfH6AxznQQB6v9DXDF6TU4+qXHizdd3g/JGtPEcCge1ghI3aaa0WjkYw5gnaDlxcffJcuXAD3ViQWyA2GkFdB67CRlAi2p/HTtleu8yFL6DMun7l4EEkUI1FITZSSJT/9rFdF8FbC9zrtUdD16s5c9MolwypDqpApLWgt+dpsZMEZGOfWRZBbRepN9vQmWkFHPiLADptlF1a1Z3/d1A+6ty82P11jU4TPhQa19kbXHeJhhhhJMfBXCqocNsbU9A6QHWptjBeCKtTSAeAkIiEB99Tw24UOtx3uJqjI0Ii1RK+t+3/Vb0wAVSPFkc5PH/fN1RRPfpQj6iopN6QVH+TS0UZpKhuYv5+Y3XsAT1KEWoY32v3RuvvVT1gAKrZW63zLyQi5SH0dJ2HCMpP3yoVbHvAQFJ/ty4FPWykyHkQSEyKYj5k6qGeBhQhdODvjhdoDVVPnFTlJhBlVTnNnzSk70HqFGZ+KiGnzpWDuYmQe41ya9YBa+CF+7z7HoBAIzVzOKe2ncqqCKGND699Iuu6eoMyFyE0NBQEJAAhU2jZSE+gkZoB4ZUzh4kUvLDxN+e6+XddBEDPqNt7VNO4GHvE9oK0OvOndtkdjK6+pUC6TiAkb9LDCMh0Ael79RNyHBmjO/HRi4KO+UyL/jar9gLZwwW19+b0wusv62Ch8uHi58vdW5mPmbeG7kMBo7hSWk854yEJIrpf/J2cd6um61o6Uzv3x8H8ScPydyxaisMTiTzByQXDw91PmVcLyMdeI1Y/fDDyeqXrpPUtFqb1o41E9Fl92F8hb08NdagIqWuqVfqf4U/UFzFkaz3Ne4dqq2k+3bttxGFga+Cl/uco2zWi2gAAIABJREFU45voTGdi+HTUYCzzu2eNh9TPrvLEFxHd1ocFOURERETXxOkqIiIimiQGOURERDRJDHKIiIhokhjkEBER0SQxyCEiIqJJYpBDREREk8Qgh4iIiCaJQQ4RERFNEoMcIiIimiQGOURERDRJDHKIiIhokhjkEBER0SQxyCEiIqJJ+vuQDxnPg+7dEuCXnxP8+lv/9/7v21/443t/HPWPf/0T//33f3q3/fntR/zw/X/9O/35FwS//Yqkd+NP+Am/4/f+1ODbX3+gPznMx6l8nEzrn9/w4w/f0Z/L6+TjKmV+cT7urO5MJR9sA18iH1+hDUBFyKTo30ZX97eyLMuPTgQRERHR2DhdRURERJPEIIeIiIgm6ewgxxoPnlf/M3b75uF7X1URQ3oGw0qhQCy35SkRF1dO23tZw/P7XtbAkzHu/VTfjSKGZHkR0YXOC3KsQWg10ixDlmX7xVRCIssypDo4PwWnLvq3viE0g7WLAjYL4yuISGK3zKyIIZv79DzIXTTjYG4yZFkEdXZiqwDpWBKLWI4fkAiJCGEj/XSWIoYMLfTTHE71Rh3kNoLis4Lk2ytiWdXjnna529azvbdz1FF9phPsO3M8LdbwGVx/AhamcZ3jKaN7cFaQU7xYQDzUF+gJqoO16l8Epc1ZoyvWhLA6xeFCeoVot98MZv55S1DICEI93v+o090pED8qiMige/qDwMJ8ggK1xoOfy/7OjDXw14tdByhtBibWIETUaFfhwQ2wiCVCqN5g35k/QdvD79A9KRDLEIj210+En2B0miZvUJCz7aH5KgF0eN70SnckY9fDq6P+UAOJgt/q5Z3a1r/f/SYL4xnEjV7lRSMPxQssBB6GxiNFDKMV5IgBTKtn3MhH9b6P6nR0prvqcmmfq3YZtHrVjR53EUvIOG70xvpGFARCDaiId5yz2AgKGmHPk6RCSoh12jti2TpX3REfGSM2R3rOR9vH5YTMjjwKWyA2GkruR6jSdQLotEqvkI3vCfgKsC+N3BYxHpVAJP0jR3YwlwracNrqbnXrt02hkWCd8ozRByvP8Pq8LGerzcXby7IsN6tZ2frIZlXOls/la/+Hj2zblKvZqtz0vt6Uq9ls/73X53LZ+uxpm9WsnM2qf29k5TCtfV94fS6Xs/0++/e7KVezZfncWwjNzzTz8Vo+L4+n8di5eH1elsvGgZqvX5+XrfRtVrPWZ9t5Gl6mdFju9bu7c7hrF42y3axmrXP4+rzs1OvG9lZbOdU+xslLu10262/V/labY3W6+36zHp9qB0PaCH2U5vVmWz82A+4HRNc26McA38fCeGHrxwQDXQDvmfSyKTQ04LX2iv1ug/26B+cBAmu8DDxk1VvdptuDibKe6adDxYtF4IZHtipEWWOdzlDWVKNZO808XqLqYSeJD6+VvBzYpk5Fu/y6boAkb2zbch4gkF+aiC8pzxMI//iJE6GGeYxRPG3fKfBiA+infdk7/gKBypFje0YUov3JQpDU295sH1eSx5DhGos0w9yxMAcfqKY0rE4h63QU8SPWixTmzcbhwg3GTjCNzRqvmpo0AkW8/ujkEA37xeP3sCaEbvziozVez8XvAlf/FUmBUAd4HBodjc7ChBpqF2RV03DvF0Cnh+tC6IM5c0ghkeaLcfZ3019ZdeEGCVQoEGWmCsDq6d59XFcgln4V0Owqn0WkEiToBN2+x1+J/WScBwGoEEanyOrzm+cJAvfpjW8SXdeVfyenwIsFAtetX8Yw3b8P0eiBHji2zXUR9CxeHFd1ARYDF+U4D6Ie9RhJ8QKLAPuiM50/reHgQXTWNnTSA/vSWcPg4EEkUI/vXNtQvMAGLtz37OOLcd3g6LnaEr6AMtveb32uGmufbKSQKP/tEcEh7aN+knCcJ+Uc+IsAaKStnVYL43UDHAAQkI0F+dVThgF02rf2J0eenLFGjm5L+FAIsNhGtUUMoxuviT7IKEFO78JkY7FdMJgov3rPz7HoPpnhzCGVRti3uPjYNmcOE6nGotv+R1rP1V3k6abDpqoA1DeW9MxHf7ePXIbQSKD8xgJiZw6ptu958PMFukUnQg1sy7a7EFyE0Ngv2t7ezIRMW+9fsii1SNdIpvyU3RU4DwLJkcXFO6KqB8n2pUyh7X7xeGg10iEVckj7cF0EwFmBeaudbx8IqCuPMzeI0Egr9iMx2wB9dx04uqj9BJtCM7C+YwIyXWBdX688f40FR4zpDvBvV43IGg/G7fZWp6aaNnN5ATtTNV2TyzMC56vqmz66V/dWdkT0WfDPOoyo+g0Zf9K/51H9FtATA5yzOZg/adg7+O2Q7U8QfI4Ap1qcrETEAIeIzvaFRnIOn/LaG3ExbhFDPgJPZj696Rxr4KU+F4S+B8vwPFNuT0R0dV8oyCEiIqKvhNNVRERENEkMcoiIiGiSGOQQERHRJDHIISIioklikENERESTxCCHiIiIJolBDhEREU0SgxwiIiKaJAY5RERENEkMcoiIiGiSGOQQERHRJDHIISIioklikENEREST9PchHzKeB927JcAvPyf49bf+7/3ft7/wx/f+OOof//on/vvv//Ru+/Pbj/jh+//6d/rzLwh++xVJ78af8BN+x+/9qcG3v/5Af3KYj1P5OJnWP7/hxx++oz+X18nHVcr84nzcWd2ZSj7YBr5EPr5CG4CKkEnRv42u7m9lWZYfnQgiIiKisXG6ioiIiCaJQQ4RERFN0oVBjoXxJOJi3MTcFWvgyRh3m8Uihrzn9FG/e69XH8EaeMZ+dCqIaIImPZJTxBKe5+Hs62cRQ4YW+mkOB0AV1HnwGv/kR0d4zhxPizV83hw+j4N6VSCWHjzPwDY/03x9R7btaf+v09Gxpn7/MP3WNL7XrbNCIkL48W2K3mFblxv/WsF8e3v/ua6vsye+d1Dnihjykm30ZUw3yLEG/lpABed+sUD8qCAig7nTfD+ATjNkWYYsiyCU/+EXZWf+BG3D84M4+gDH6hUQBBbms1yBVfT/t3f/vK3raBrAn8XsB1CZUl1UzAAstVupuYBLA9Oo2HrCZgCW+QgpCUwRYuop1AzgLgJuo2pH3SGwU6hUme7oC8x4C+oPJcuOnDg3jvz8gAMch5ZEypT16qUstsdAhaoa2tLkEpEJUWbqcBlrkCLrjx2lD/uskBmEeuSJ6JtTWTX0D9MF80CTP2K3Ld3fSw0cfNYNcpkCatx/RstVFapMQGVd57Ew8Q7b7nu53GIXdwH2qTK6JWcEOX42Iz34SfnoSs2LxK2JYHIvop6k6kfLTaLt8ZXjOR3UwqQW+ilFuLyB7aIZFDTSk7/4E5CZQrEr+7Ycq2uTS8g89/bdpB391e/cFc44g3QYVAXYSAVtOPxx9U70KyElhNeXRouNjo9JxkfmyL3yUeAwuop9RzbzTMHGjE5qI0J6P6EViBVgX6etFUg1vBMYrUmwMTBdRBzE2CYF6noob/JHKJFBxpPl7gSKPiBqkBuNJAzbZQy0kkOgXe5QQKO0p8votiwMcixMlALZkMnw4+0mlzDhEG2X2x0evROyVl1EXUJDIRsiABirUc5cGcIaxLX0InggXfhNbU0Kq58OrpiXaF4tkm08/2XtC0MkRY16QV0Lpfp9lyntXbU3yI31MkTV8EWABrk0CPuyEtvdzJWuiKGKHUpGOVftdL8SiIV3XLSsibwMSIVSW6T+RUKhoLryzA92LUxcQ1bD8Yr0QlexOj0+7LSIRakTbOPDPRHEWyS65NX2N6bTBYF1U2JXKMRd3NvkeFQC2dyzZIRsszARoihGLYfvyLou+oDHmghxLZG1AfSpMrotix4GCFtCJxrlbHajQbkrUBQxIv/PqgbgFkj6gCPAnQB2rw2AAAjuIIoUcZRAl+M0vi01oIHITxklGu2SJ+rapsY3AfCO/EZdFxAzX8AnN3mirgAAlaE7fsMwQVHXcPsmwJ0okMYREl16AQ7aL4ICRTzaq/5ubYUIk7OqS1/grX4lUg3zmKN56v7S4NUm0E/Dhx3EWyTKBdbur2o4MXhBt7Al9GGHhH7z4Dkt2BhUVffKXfjIcNJvT3JDElaXkHOLBHcQ7rKBvp0AG1Nh6B4GUSoRltPhWQsTK4isavtwg/xxh21pxl9ro/W4C0GDHDKOIEfflTVymbohrU0Aa/yFT5XRrVgW5LzpMEhZRkBWFWR3c1nhxnS7723//8u4dCaKyfd7Ebkg7Vg6/T3qGkUS4glAfaKub4VZQlaoZDfcVYyfjnnpOtP1CjaQQqKst5dZ36c/ZbUddlr8/ga5jLHbnhMU0bclYqjD28/7EYG+a9oMqiiAycVcHLnv1LjUSHTZnls2MJlCZEo0m427YFQpRFbBCMBdGAAiDhDieBndlmXDVWGIpB8ScVdjQwzhshHq8SP3hbirgFInfToxDBPos1Ps7dVENQzx6KS9GW5hsBCGydspzSaHTDWUdOt8X10nNd8YVKVGYl/dfgzuIAo1GvabV6MuBO547F61Jf1KxALK7NpX7XHljWHZTKFQ8fwV73hjSGZu7h1p7wV7983zTQ5zZNhpZmMw0YIAp3mFTcLz76Ojq9PkBjrZou8eTQ45DXAANxxVed/ZmXIXd9XwvsK7ead5HTp1EG+RwBv2shlUOwx2qoxuy7JMTrDBk94hbqNtlZXQeOyLhSyhZTwKxhdlYaxBlHopl0SjNEG7SYNMRUi9lMzBkM4nCO4EivZKYbylAqpvoPulVX8f3bvr6q5svKWgS9NuV0CWGjL2hwEVskqOT3K2hE5ClGe2k35bx/uVR6TQ0FAQkBiOq6jvdhqlWfAtHbRXvKk359w0KxiGSABv6PRtTS4Rq6Jb4Th7OzmW00j320RuXD3U6b7clDsUQjJz+S25TN3QPcb9zWYKBYDiVJ+ccXAMQCGr2mWCDUzmgvWhrO1Tp8ropnDuqgPuYK3luUNlX+E71fXWXdtndW3DRxYmMjP3cBARvd96n5PzbgE2Txo2vf6HR/U/u7yKkyaddj39yt3/dU0Bzsd+EUlEdMz3y+Q0OWTsUp+HLpiStAZRGX/yzZsf0OSQj8ATb0z+Xq69X30F7hMi+iTfL8ghIiIiWoDDVURERLRKDHKIiIholRjkEBER0SoxyCEiIqJVYpBDREREq8Qgh4iIiFaJQQ4RERGtEoMcIiIiWiUGOURERLRKDHKIiIholRjkEBER0SoxyCEiIqJVYpBDREREq/SfS95kogh6tiTBn/9U4C9/nV/u97/8G//8dT6O+q//+W/842//O1v2r1/+gN/9+n/zK/3Tn5H89S8oZgv/iD/i7/j7fG3wy7//ifnqsB2n2nGyrv/6BX/43a+Yb+XntONT9vm723FlfWct7eAxcBPtuIVjACpDJcV8GX26/9jv9/uvrgQRERHRpXG4ioiIiFaJQQ4RERGt0juDHAsTSeTNZStzVaxBJHNcbRObHPKa60fzrr1ffRJrIhj71bUgoluzwkxOg1xGiCLv37knlSaHTC300wYBABfUjdcpvzrCCzZ42u4Q88zxfRz0q66vGlj/Pf7rK9Pk8ugx1ZfNlAuZAenKL4xu2tvfu0f7TpNDLlnuPd/ldPNWGOQ4KqtQVe0/051UlmiQPyqIzGAzWiiBLrt1ZhAq/vJAJ9g8QduUV8jfwrF+BSSJhfkGZ39rIsS1RKmTuULEuy3K9pgrDwJwAZkJqEeepNbs2Pfuyb4TbGC6ZaoSGgqZ7Rd8o18RnXZGkONnM9KDn5RbMx+JWxPB5F6kPonER8tNhsBGEfxvdXVrMyhopCd/8ScgM4ViV/ZtOVbXJpeQee7tu0k7rDmRIRpnkA6DqgAbqaANTxxX70S/ElJCeH1ptNjo+JhkfGSO3CsfffdPro4vcV4QsjryU9gGudFQcshQlbsC0OW4r4t0fAKjm3G870zVqIsEYQgs7ldEJywMcixMlALZkMlQXmmTS5iw7CP4crvDo3dC1mqHbTkTpTc5jNV9lF5V3lWuNYhrOVwVZEB6xje1Tt/3Bd+8WiTb+O3MTxgiKWrUC+paKNXvu0xp76q9QW6slyGqYPod0CCXBmE5XOFsd4+H6X4RQxU7lIxyrtrpfiUQi8OTvzURUmTDcaUtUv8ioVBQXXnmB7sWJq4hq+F4RfqZFwn+icnCRDFqmUHB4nXULwPE2wS65Clqrd79vdtfJKaw+qk9DyztV0THLXoYIGwJnWiUs4G4i66LIkbk/1nVANwCSd9pA9wJYPfaAAiA4A6iSBFHCXQ5TuPbUgMaiPyUUaLRLnlCgI2pUPUrMohSibA8HCaYU9cFRLx8cOutugIAVIbuIiYMExR1DbdvAtyJAmkcIdGlF+AAaErsigJFPNqr/m5thQiTs6pLX+CtfiVSDfOYo3nq/tLg1SbQT8OHHcRbJMoF1u6vCtnQsfqgW9gS+rBDQr998HxMnUOm7oJmE1iYmbcEdwLuyoDW5WPfu8HGoKoAd3EXQ6KE2bSFC/oV0THLgpw3HQYpywjIqoLsblor3Jhu973t///dRAz1GdewdY0iCfEE9519rK5vXXC4NG53JVOMn46ZaJRn3U9E31awgRQSZb29zPp+06eshgiTAioVyCrjArDmFRYCZ14v0Fq8+3vXDcGrsgb7FV3CsuGqMETSD4k0yKV/T47LRnzshkJ3FVDqBLbNQ4ZhAn2BFHuTG+hku/igCMOhDidWCpkOY8WXqGuwMahKjcS+uv0Y3EEUajTsN69GXQjc8aC/akv6lYgFlNm1r9rjyhvDsplCoWK8GbqEIRL9xg3p7b1gl7l53g1Dwavbsbo2rxaJG3+gFTv3e9dbErnRbR9Z3q+IjlmWyQk2eNI7xO3QicpKaDz2xUKW0DKGP7KyKAtjDaLUS6knGqUJ2k0aZCpC6qXcD4Z0Zrl0pyr8dS7PhgR3AoUp0WymyxRQfQPdL626qry/ru5eJ28p6NK02xWQpYaM/WFAhayS4wPcltBJiHJh++hrHO9XHpFCQ0NBQGI4rqK+22mUZsHXe7CByRSi1JtzbnochCESwBs6fVuTS8T9gVW4473NGAUbg8xEQ11nM0luaFtIRuTrc/p791TfGZe5c0fXzZf1K6LjOHfVAXew1vICQ2Wf7jvV9dZd22fl6rPbLgnGL8QaRCbkECwR/WZW+5yc9wuwedKw3+DBZU3+CCWyKzlp0mnX06/c/V+/cYADCzN6ECIR0ef7fpmcJoeMFYrZwpnhnPeyBlEZX29qtMkhH4EnXhV/L9ferz6JNRHK+FqyWER0K75fkENERES0AIeriIiIaJUY5BAREdEqMcghIiKiVWKQQ0RERKvEIIeIiIhWiUEOERERrdKyaR2OPJvGTd0weZz3UIhKhifKBKyJ4M/qAKB/HHh9oiyYTgfhCt0koThR15DteG87Tta1/q3b8Un7/L3tuLa+s5Z28Bi4gXbcxjHAZ5l9HT4nh4iIiFaJmRy2g1d/V3r1d/Pt4DFwA+24jWOAmZyvw0wOERERrRJvPCYiIqJVYpBDREREq8Qgh4iIiFaJQQ4RERGtEoMcIiIiWiUGOURERLRKDHKIiIhola43yGlyyChCFEWIZI7mq+tDRERE38qiIKfJpQs2Jv9kviz0aHKJyNjzahZsYKoKVabOW+4mNMjl5PMYBYIW5q3PqQsiz/hcxv3AYLSkNcfL/IA1kljYbYiIiD5k0bQOwcagqgB38jQIS4MNn1P95dxjyKd/bZDLFFaXqDZB+zqGufPfa2HiHYRKDh5tfpQ1iJVAVhmI9nUk8/aR5RYmBbKqggDc481NjEoKt/3HHbZlBRO4QCl+zPmocyIi+nQXGq6aZBa6rEJ7BR+rAtDpTGZhnHE4yAC8iwvErJ9Z8LIV06zUKMthDSJjYc1Q3i86ykb4mRO3vbxdr7Fdm4a2+Ouby2S4Ol0qw1GjLhRkH4UGiLcJ7KuX5zEprH5CGi5fa/NqkegUfZwkYqiiRu1eQFayLxOxAuxru38CbMwQFAfxFkm/HBER0ee5QJDjMgW7bYmqqlBVFTKhEBvbDzmVOnGTmLXlpj8BC8j2b1VVodQW5iJneo3UhCirClWVQWnTBxAuK9VtM4NQ2Tiw0inKuC3PFLRpg5lu+MxrY2aH7alaotQJdGoQlhkULF4bAE0OY3VblwpVdbksmE5ngjGECBONsn9tkakCRV23Lw1Sq/F0ZiWCO4FiV/ZDYk1uoLs2TthSI9nGs5maptyhUPEQLBEREX2SZbOQn9RmDsxwShOxAoy7kj99Kp2Z8VXVwIdPgQn0UzccIhCrAmW32oOZaxNov6KJRtqnJCQq073PwkQpRkvqZlhHKoASgJLYBEC/WHAHUaSIo3YG3JkdMgwHLhVgYyr0i1iDKJXtMGKAzZOGjCNEAAAFrRMkCF0bUiCr3L45K5wUEpmIEEdd2zUUatxN2tPkEqnVKOVMQ/shL4Y4RET0+S4Q5Lxfkz9CQaPsTrq5RFx/5hYtTKq9e1ncUNOiJU0KrbL2PhM3BLVsSZetkt2QXnHsXpoPEDGUn49qs05D3SOIOGizLwUQ+aFagUgfD8BGm5EVKtm+aHLIJETqlTe5RLzbzt9vYw2iFMi8YS0iIqLPdIHhKjc8MgwzNcjNeLgiuBPePRqDui4Acde+zw2rXFyTw2iFWABoXmGRIAy7IjPKzJxYCV4tkAwLwixb0OOyL6Ue3x/jVvexe3Ka3EAnW8QzQUqXWUnFdKjOH0b0A5zu/qpT90dZmFhByCGYsSY6GuA0uWSAQ0REv7kLZHICbEyGOorb4RG4E6efFhApNGJvqKOE2QQQqUYSp21iIYHWCv0dqZNhpTgCkOiFv8opoLqNwWUpBAAEG0ilkHZlSkMnu2VtlAoqjUdDQEuWPBgeSzRK83YLTpsM8032S5NLd7M34D4Lc05oEeBOACja+236qvrDdQl0WQ2BUR/0qf4zdpuuIEUXvBZIvQxS1weIiIg+y3/s9/v9V1fisvgz9w/j0BIREa3A9T7xmL5A+/N3BjhERLQCX3rjMV2b7iZpIiKi72+Fw1VEREREHK4iIiKilWKQQ0RERKt0FUGOm9vpEvNWAe7m2ePPnJmdl4qIiIhWZ1GQM53UcnZyy29CyApVVUInX12T72Y6meqpiU/nA9amn8T0fduc9rfRxKeTlZ4qIyKi27Do11XD3Eqf8wya0XQBdMWOTP/Qz0nVPnTRGkQyHz+40RrEOwGVFAu31SCXKawu2wdLugcgmrt2SgxrkCJDVQl0Dyo08YIyIiK6GZcZrrIGkbHzQ0FNDulf/ct8mN7BL/P/DnfVL/Pcu5IfZwdGV+oH2QGBu9q860p+tN5JnY5xdbXtdAjRZIqGaQbEb0eDXErkdn4/TDNoXSZjtG9kDtu+T/pTa8hj2/TqdKEMR/Nqkeh0eK6OiKGKun94tZszzEI/pQgXr7Wd+LWPqALEW29KDCH7ecTcJKxYVkZERLdjf5Yf++f7h/3Lz+mfn/f39/f75x/e64eX/fRtrsh73/74+3++PIzW+eP5fv9wsOH+zfuHfvkf++f7e299c3X+uX95OKzHz5eH0Tamr4+Z1vXny8P+/qCRc+t09RjqOl+vfd+O5/2PfnsP+5efbVuff4y3+eP56PaHdd2/8Z4jy/T/vH06+fyG+nXF3Wd3qn1T0/eeqvORfvlmGRERrdnlHgaYuEkgAbgr6X6Kbn/Oo+6to0mRjlNZP8QQhgmKugbafMFofib3ZgylCfRTN1QiEKsC5VB4RINyV6AovDm4AEC9ueBBXYN4i2TnJiQNpvNMzaxT9RNdukk8e9N5r5BAN3DZECWxCQCDBDoVQOm9LQyRpCkirY48ufg9D/0bL9PkEnFs3PqFRCYib24yDYUadwGGoaN2yGm5AJsnDRlHo/nCkoNc0DCsJQ+61KkyIiJau09/4rE1KbTK+uEDayKYN5Z5U5PjUWGYJLLJIeP6o2vF0XtOPqDJH6GgUVYukGlyiWVVtTCpbie5bF9HC/dcsIGpKnTDUukntCuIt0i8AanRfVVNDpmESNsZ6VEAkR+rFRH0kslW+3Y41kQQo6nWXQC5285N9nmqjIiIbsEn/4S8wasFkjBsX3azVX9QXaOAcJkCADZTKI5WIYfRCvGbyZgAd6KAelx2H84pNlPANkYAoK4LQNy1J/NuRu4FmldYJBh2ncH5u85lXzJVoK5HNfzwPTk2UyhUPJMlsjCxgpAbBG1mqqq6f+5XbSqrUE1uSn7r13pNLpFaL1sICxMdC2JOlRER0a345ExOgI1UUGk8GnLYdcWT4Zg4AtBe4Z8kUugkRtqmB7rhkUEB1Y2dtFmMdpBrPHRURNBAny0RsoSWcT/sAq/sTTodshWqG54BRKqRxF1ZAq0VRlU9JthAKoW0q4zS0Mnu9DKtg6E8laH68C+LJsOOXnZuXJYMGbalwhAJMBqOBCbtUBkq45e1QZ/yhxfd8Fx4oow/sCIiuh2cu+oC3BCU/4seOg+HloiI6PKu4onHdLvcz+QZ4BAR0eV9+o3HRKcMD5okIiK6LA5XERER0SpxuIqIiIhWiUEOERERrdJtBjntnFmXmkW9yeXxZ86cmJ+LiIiIPs+yIGfmYW3WXC5IOLnd7x4YtE/trTL11TX5lkYTpl5oQlEiIroNyzM5SQKoDKs4zbSBB3+yfN2aXLp5r9onJmdIPz+wJiKi1ThjuGoLqS3K2SinQS69K24v+2JNBJOfO2TTTjuQaqBQiA+u5BvkUiK38+t1z14Z6jPNQHV/nyYGTtbVH3aaaUcSwtsHEovPxZP1LktWuHmsbJthO8hynKqrNYiMPbIf2v3e/zN9UDvaN97yxnrrPbLP/fLlyRiLTMFNPtq+LjVQ7MrvndkjIqLfzqK5yn887+8fXvY/f77sHx5e9j/3+/2P5/v9w8vP/X7/c//y0P2/e/v9/v75x/D/+4e9K3bvbYuWb/egwK1nKDsCrn5CAAADkUlEQVS13h/75/vn/Y+Z5afvP6euP56Hsp8vD/v7+8nrw5XPtGVat7m6HmuT3/4f++e+3qfruv/xPKrr8X3s2tF9rm7fPO9//HzZP9y7z3vcB45vf3a7b/n5sn/o9sXPl/3D/cP+5Yf3NyIiojec9zDAYAMpImS2Qtz/sUZdKEgzDP2IWAHmtb/iTvRTO5dRgDsB7F4bwM3JPZ5L6sz5hZTsJnl0E0H2JnNiAQl0t8k3HK/rZO4mAIm/UpX1c1wF8RbJzrX/5CZtCQ09maJ7aV0T6Keu/QKxKlDWaKd+eqOuiTfRpZCo+snNp58HANWvFIlOIVACUJCbAMN08m5y0zSOkOgjTy4WElU/TfkZrEGUAlllIJp8mPeMiIjoDWc/8VikGiazXpDzEe0s1RdZV8fCpNqbWNMN7Xx4rSaF9ialtCbCx9eKyUSXl/Heujb5IxQ0ysoFT25OrmXbFLJCJbuhwuLj7QruIKCQmqE+qGsUSYin96+ViIhuyPk/IQ9ibK3x7q0IESYapr8Ho0FuNJJtvCRxcloYIinqRZN295pXWCQIw/ZlNyP1hzR4tUAyrBTmxEptpoAl7Q9DJDo94z6VY9XLYbRCLM6vq6+uC0DctfW2yEYpnWWCjUFVaiT2dXzvzNn35AjECl4/umC/IiKim/COuasCbKSASgsk2/a1yVBHMaLuLSpDdYlfLgUbSBUh7YZzlmQHgg2kUkjjtjZKQyfdIMdkOKaIoAEv63N0pdhIBZV2bVTQOhkPneh0GHXy2z8ZOosjAIlGaTYIgg1MphCl0RCIdWWnWwmggOraiAS6NO2g0oK6HiFSjSTu2pFAa4VlEeZ0eMzV56M9QMgSWxkjan99f3QojIiIaAbnrvqW3BBcWBrwnE9ERDTvNp94TERERKvHIIeIiIhWicNVREREtErM5BAREdEqMcghIiKiVWKQQ0RERKvEIIeIiIhWiUEOERERrRKDHCIiIlolBjlERES0SgxyiIiIaJUY5BAREdEqMcghIiKiVWKQQ0RERKvEIIeIiIhWiUEOERERrdJ/LnmTiSLo2ZIEf/5Tgb/8dX653//yb/zz1/k46r/+57/xj7/972zZv375A3736//Nr/RPf0by17+gmC38I/6Iv+Pv87XBL//+J+arw3acasfJuv7rF/zhd79ivpWf045P2efvbseV9Z21tIPHwE204xaOAagMlRTzZfTp/h/PNDwkskqBlgAAAABJRU5ErkJggg=="}}},{"cell_type":"code","source":"import tensorflow as tf\ntf.keras.utils.plot_model(model)","metadata":{"execution":{"iopub.status.busy":"2022-07-21T08:05:23.467012Z","iopub.execute_input":"2022-07-21T08:05:23.467452Z","iopub.status.idle":"2022-07-21T08:05:23.741833Z","shell.execute_reply.started":"2022-07-21T08:05:23.467387Z","shell.execute_reply":"2022-07-21T08:05:23.739881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install --upgrade tensorflow","metadata":{"execution":{"iopub.status.busy":"2022-07-21T08:05:23.745614Z","iopub.execute_input":"2022-07-21T08:05:23.745976Z","iopub.status.idle":"2022-07-21T08:05:23.752866Z","shell.execute_reply.started":"2022-07-21T08:05:23.745921Z","shell.execute_reply":"2022-07-21T08:05:23.751656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model= get_bn_model()\nmodel.optimizer.lr=0.01\nhistory=model.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=1, \n                    validation_data=val_batches, validation_steps=val_batches.n)","metadata":{"_uuid":"78e382d0b3de14312e762edc480b5d215be82269","execution":{"iopub.status.busy":"2022-07-21T08:05:23.755039Z","iopub.execute_input":"2022-07-21T08:05:23.755482Z","iopub.status.idle":"2022-07-21T08:22:51.309048Z","shell.execute_reply.started":"2022-07-21T08:05:23.755310Z","shell.execute_reply":"2022-07-21T08:22:51.305311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submitting Predictions to Kaggle.\nMake sure you use full train dataset here to train model and predict on test set.\n","metadata":{"_uuid":"8e4b16516a57e152a911f6e7ba7f4d70ff204512"}},{"cell_type":"code","source":"model.optimizer.lr=0.01\ngen = image.ImageDataGenerator()\nbatches = gen.flow(X, y, batch_size=64)\nhistory=model.fit_generator(generator=batches, steps_per_epoch=batches.n, epochs=3)","metadata":{"_uuid":"0fc055b482971b36561aaf9421c8a9c53df2900b","execution":{"iopub.status.busy":"2022-07-21T08:22:51.313100Z","iopub.execute_input":"2022-07-21T08:22:51.313343Z","iopub.status.idle":"2022-07-21T08:46:48.468260Z","shell.execute_reply.started":"2022-07-21T08:22:51.313297Z","shell.execute_reply":"2022-07-21T08:46:48.467579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = model.predict_classes(X_test, verbose=0)\n\nsubmissions=pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n                         \"Label\": predictions})\nsubmissions.to_csv(\"DR.csv\", index=False, header=True)","metadata":{"_cell_guid":"c2841d54-f3dd-1ee8-a30d-4457dec0a67a","_uuid":"4262c6bfb15ec96993e83bd2a2552eadf14fb33d","execution":{"iopub.status.busy":"2022-07-21T08:46:48.471473Z","iopub.execute_input":"2022-07-21T08:46:48.471779Z","iopub.status.idle":"2022-07-21T08:46:50.749558Z","shell.execute_reply.started":"2022-07-21T08:46:48.471730Z","shell.execute_reply":"2022-07-21T08:46:50.748736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"More to come . Please upvote if you find it useful.\n\nYou can increase number of epochs on your GPU enabled machine to get better results.","metadata":{"_cell_guid":"0e6213b0-fc56-658d-46e3-4a5dcb7148ce","_uuid":"3a9a548a2080ebf61b2ce35db78f0c9520c1358c"}},{"cell_type":"markdown","source":"Dear Reader,\n\nkindly note that in addition to uderstanding code, i highly advise if the terms and notions of ML and DL are well understood thats why you can have a logical understanding of what happening and more important the WHY of each line of code. If you feel that you are new to ML area, please find below this link, it will resume everything around the ML/DL notion:\n\n**https://www.analyticsvidhya.com/blog/2022/01/machine-learning-algorithms/**\n\nafter 10 years of experience, the best way to study, is to learn by coding, so you can dive deeply inside the sea of data and their unbelivale worlds.\n\nHappy coding!!!!","metadata":{}},{"cell_type":"markdown","source":"# Steps to Consider in developing Machine Learning Models\n\nFor any machine learning models, there are few essential steps to consider before it is deployed for the service. These series of steps are similar to all the machine learning models that you develop. Now let us look at the steps in order.\n\n1. **Collecting Data**: For any Machine Learning model data is the main resource that needs to be considered, and models might need a vast amount of data. Collecting data is one of the difficult jobs. There are free services and paid services which offer such data if needed, where these data might be structured data in the form of JSON or XML or CSV ..etc. But if someone tries to collect data on there own, Then they need to consider that the data they collected might be an unstructured form.\n\n2. **Data Preparation**: Now data been collected but this data might contain some corrupted information or missing values, That is, in the process of data collection someone might have unknowingly changed the data which is human error or some columns might be empty, which should be solved at this stage. with the help of statistics, we can solve this problem using mean, median or mode and other statistical methods.\n\n3. **Data Analysis**: Data need to be analysed at this stage, that is, if it as any hidden relation between features in the dataset. With the correct feature engineering with domain knowledge, we can solve 70% of the problem. If we overcome the data analysis stage then the work has almost been completed. According to the survey, 70% of the data scientist time is spent on data analysis and feature engineering.\n\n4. **Training Model**: Post data analysis the given data is divided into Train, Test and validation data where:\n\n* Training dataset: 65% of the data is used for the training algorithm.\n\n* Validation dataset: 5% — 10% of data is used for validation of the algorithm. The validation process is used in the tuning process.\n\n* Test dataset: 30% of data is used to test the algorithm performance.\n\nthen the training dataset is given as an input to the algorithm to learn. the algorithm might be anyone like Regression, Classification, KNN ..etc.\n\n5. **Testing Model**: Once the algorithm shows a good performance on the training dataset. Then we test it on the test dataset, where the algorithm should predict the outcome of the test dataset. If the performance is seen good even on test data, then we proceed to the final step.\n\n6. **Deploy Model**: In the final stage, we deploy the model to the cloud where the algorithm can serve its purpose, even the model can be used as an API.\n\nHere are the 6 steps involved in developing the machine learning model","metadata":{}},{"cell_type":"markdown","source":"The best explanation i had ever seen for CNN\n\nhttps://www.analyticsvidhya.com/blog/2021/07/convolution-neural-network-better-understanding/","metadata":{}}]}