{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NOTE :- IF you are looking at this like a beginner's tutorial, please read every line carefully. Download and edit for better view.\n\n### While going through tutorials we all have been there getting bored and skipping lines just to get to the core of it (includig me) but lees we hat the core is in the lines. I'll try my best to provide as much information as I can about the code and process because this is what I want to learn and how I have learned- \"*what is this line of code doing, why this parameter, how is this happening, why did we do this*\" and all the waves of questions that arise from the very process of implementation of a code be it a simple (\"Hello World\") or a very Deep Network.\n\n## Another Note:\n#### I just assume that you know and have practiced a bit of (*if you are here, I'm sure you know it all*) in Python-3,Numpy,jupyter-lab,Pandas, and basics of Machine Learning, sklearn, Deep Learning, Neural Networks, Convolution Neural Networks & keras. If not, just in case, I highly recommend learning these in the order.\n\n### Last Important Note: \n#### This might not be the most elegent notebook and efficient code out there but WE as a beginners, care less about the efficiency but the leraning curve. I myself have self studued all the concepts by MYSELF withthe help of this online coding community so here is a little effort for that coomunity and the Data Science enthusiasts yet to come. If you see something that you can not understand in a *loop/method*, just implement it pieces by pieces yourself. You'll develop the core understanding of the libraries as well as the process of how to do it. If I can save one fellow coder a trouble I'de be repaying my debt to the community. Please feel free to correct,comment and contact me if you see anything wrong or something that can be replaced with something better as I am a learner myslf too."},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nstart_time = time.time() # We will count the total execution time of this script","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Problem\nWe have a set of almost 250000  single channel (^gray scale) handwritten images of dimensions `(137*236)`, converted to digital format, and then converted to pixel values where pixel values range from 0-255 (after exploring the dataset we'll know) and saved as the 32332 columns `(137*236=32332)`\n\n^ Every color is made up of Red,Green,Blue (3 channels) and in computers we represent these colors by numbers ranging from (0-255) where 0 represents White and 255 represents Black. Each color has it's own values in case of 3 channel images but in grayscale, we just have one channel so we get B&W pictures. To get a better understanding of how does all of this makes sense, just [visit this link](https://www.youtube.com/watch?v=bwb4r3UVKko)"},{"metadata":{},"cell_type":"markdown","source":"## Solution\nWe are trying to find the root,vowel and consonent in each image that will be presented to us. For that we'll build a Neural Network and more specifically a Convolution Network. NN, CNN works on finding and memorizing the patterns like we humans do but subconciously. We do not have to apply a lot of load because we have memorized and learned to memorize the patterns from billions of years of evolution and we are trying to teach this this to computers within what, a hundred years or less. Not Fair! And I can not give you a whole lot of Deep Learning in this mere notebook but before we start into the solution, I highly rcommend and I repeat, I HIGHLY RECOMMEND to watch these [Neural Network series](https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw/playlists?view=50&sort=dd&shelf_id=20) and [Fundamentals of Deep Learning and Neural Networks](https://www.youtube.com/playlist?list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU) where mazing people giving you *INSIGHTS* of what happens *ACTUALLY INSIDE* the mathemetics of something , how things work, what are *PRACTICAL SIGNIFICANCE* of formulas we use and all the things. I have nothing to get from this but I had a sudden revealation from these such as **Neural Networks JUST find the patterns and we have high hopes that they'll find and to our surprise, they even just do. NN don't even know what they found.**  And some amazing visual explanations of **Working of Gradients**, **Significance of Eignveectors and EignValues and why and how are they used, How Convolution , Pooling ACTUALLY works** and so on. Few of the videos on Keras are [available here](https://www.youtube.com/playlist?list=PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL). I am just trying to give you the best of I have after searching through tons of tutorials and blogs."},{"metadata":{},"cell_type":"markdown","source":"## Import  Libraries\nImport all the needed libraries all at one place so that you don't have to be confused. The moment you need some new library, you come up here and import it"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt  # plotting \nimport matplotlib.font_manager as fm # to plot the font\nfrom tqdm.auto import tqdm # see progress bar\nimport PIL.Image as Image, PIL.ImageDraw as ImageDraw, PIL.ImageFont as ImageFont # Darw picture from font\nfrom skimage.transform import resize # Resizing of image\nfrom cv2 import resize as cv2_resize # resizng of image\nfrom keras.preprocessing.image import ImageDataGenerator  # image augmentation on training images ONLY\nfrom sklearn.model_selection import train_test_split  # splitting the data\nimport keras.backend as K # for custom metrices implementations and other processes that we define\nfrom keras.layers import Dense,BatchNormalization,Input,Dropout,Conv2D,Flatten,MaxPool2D,LeakyReLU # keras layers\nfrom keras.models import Model #Model class\nfrom keras.optimizers import Adam #optimizer\nfrom keras.callbacks import ReduceLROnPlateau,ModelCheckpoint,EarlyStopping \n# Call backs acts like milestones and if/else while model is being trained\nimport gc # garbage collector\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Files, Paths & Outputs\nIf you have any problem while finding a file,data and path, you can find it easily on the screen printed above. Kaggle saves the output *only when you commit and can be accessed as the output of that successful comitted version*. So Check the the data under your commit histories and you can use the data of that kernal for a new kernal."},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"TRAIN_IMG_PATH = '../input/bengaliai-cv19/train_image_data_' # just for training data\nTEST_IMG_PATH = '../input/bengaliai-cv19/test_image_data_'\nFILE_TYPE = '.parquet'\nfinal_w = final_h = 92 \n# these are the final dimensions of an image. We resize the image so that it fits in out memory and uses less\n# computational power. You can surely try different dimensions. There is not BEST size\n\nBATCH_SIZE = 128\n# our model processes data in the batches. Rather than computing *learnable* parameters and tweaking them\n# using gradients (Search for it in case). it searches for parameters in *batches*. batch=1 = single image\n\ntrain_classes = pd.read_csv(\"../input/bengaliai-cv19/train.csv\")\n# train_classes have our respective label for every image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing Images \n***DO NOT IMPORT ALL OF *.parquet* FILES AT ONCE. IT'LL MAKE YOUR KERNAL CRASH AT ONE POINT OF TIME LATER***.\nSo instead of viewing the results from the original images for EDA, perform it on the *'grapheme'* column and view some results by converting characters to images using *matplotlib*\n\nBelow is a code just to see that there is no difference betwwen the two. Put any value of **i** that exists in **df.index**We will delete this dataframe later and import it again. This is not a good practice though I did it just to show how to do (what I'll do later) in one go"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_parquet(TRAIN_IMG_PATH+'0'+FILE_TYPE) # import any random image file given in input\ndf.head()\n# print the first 5. We have 32333 columns. First one is 'image_id' and from the second column to last\n# (in our case it is marked as 0-32331) we have pixels of image ranging in 0-255.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Labels info\ntrain_classe will tell us how many classes are there, what are the min,max classes and some other information about the Data distrubution on the data we have"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_classes.info(),) # basic info such  rows,columns,each one's data type and memory used\nprint('*'*50)\nprint(train_classes.head()) # there are 5 columns corresponding to each image. Each image has a unique \n# 'image_id' and belongs to atleast one of the 3 classes. Classes ranges are in different. Last columns is\n# what tell us which 'grapheme' is used in image.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('_.'*50)\nprint('Insights on the labels and classes\\n')\nprint('Unique classes in df\\n',train_classes.iloc[:df.shape[0],1:4].nunique())  \n# number of unique classes in each number. See the Difference. One  class has over 160 values but second one \n# merely has 7 classes. \nprint('*'*75)\nprint('Description of Train Classes',train_classes.iloc[:df.shape[0],1:4].describe())\n# See all the important description about classes. This is not necessary but it is to show that dataframe has very good\n# method for showing important values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plotting \nThis function plots the images side by side. One from the DataFrame and other one from the 'grapheme' used in. We have a '.ttf' file for this. \"*.ttf file is a font file format created by Apple, but used on both Macintosh and Windows platforms. It can be resized to any size without losing quality and looks the same when printed as it does on the screen.*\""},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_comp(i,filepath,pixel_df,class_df):\n    '''\n    Function that plots the comparision between the given Glapheme Character using the character to image \n    conversion function and the second one using the pixel of images given for the same in .parquet file\n    \n    input: \n        i: index of the image in the dataframes given\n        fontpath: path of the font file\n        pixel_df: dataframe that contains pixel values of columns\n        class_df: dataframe whose ['grapheme'] column is perfectly aligned to the index of df_pixel\n    '''\n    fontpath = filepath  # we will set the file path in arguments\n    char = class_df.loc[i]['grapheme'] # get the character used in 'glapheme'\n    \n    print(\"Grapheme used and given with classes: %s\" %char)\n    fig = plt.figure(figsize=(10,3)) \n    # set the figure size's dimensions to (width=10,height=3). Out 'N' subplots will acquire the areas this\n    # area accordingly each one having same area\n    \n    ax1 = plt.subplot(121) # first subplot of the 1 row, 2 columns subplots\n    ax1.set_title('Image of the Grapheme using Pixels') # set title of the first subplot\n    ax1.imshow(pixel_df.iloc[[i],:].values.reshape(137,236),cmap='gray', vmin=0,vmax=255)\n    # show the image which is as index *i* of the input DataFrame. As we have a continuous values of pixels,\n    # we have to convert it to the dimensions of the given input already given to us. So we use (137,236)\n    # specified by one who gave us the data. cmap='gray' depicts Black & White \n    \n    ax2 = plt.subplot(122) # second subplot of the 1 row, 2 columns\n    font_prop = fm.FontProperties(fname=fontpath) # get the properties of the fonts\n    ax2.text(0.15,0.35,char, fontproperties=font_prop,size=75) # plot the text as an image\n    ax2.set_title('Image of the Grapheme from Fonts')\n    \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filepath = '/kaggle/input/bangla-font/font2.ttf'\nplot_comp(np.random.randint(50000),filepath,df.iloc[:,1:],train_classes) # generalises the function for any \nplot_comp(np.random.randint(50000),filepath,df.iloc[:,1:],train_classes) # df so a bit slow","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Comparison that our data is sorted in a certain that is can be aligned\\n')\nprint('Classes tail that ends at the last element in df\\n',train_classes.iloc[:df.shape[0],:4].tail())\nprint('*'*75)\nprint('Tail of df\\n',df.tail())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model \nFor this problem, we have to answer what is better? Taking 3 different vehicles or sharing a common with friends when all we have to do is to go to a common place,common theatre but side-by-side seats. I think the later one got your attention. Right! So we will be asking from our models to do the common work so that we can use the common works done by common the lower layers and use it in out output. **Only if the 3 labels are independent of each other and do not affect each other's output in any sense**\n\nWe can build our model by 2 different styles. Sequential Model (Easy to implement but not customizable) and using Functional API (Very customizable and relatively not so difficult to implement). So we'll be using Functional API. from Keras. If you have any queries, you can check the keras' official github & keras documentation. If you have queries regarding *what and why about*  this `Layer()(model)`, you can see the [stackoverflow answer](https://stackoverflow.com/questions/59420425/meaning-and-working-of-a-function-fvariable-v-a-function-followed-by-varia) regarding this one. \n\n**Layers in Keras are by default named as `layer_type_position` suc as first `Conv2d()` and `Dense()` are named as *conn2d_1* and *dense_1* unless specified by `name='something_else` in layer. You can access flattened layers by `model.layers`.** Let us just get the summary of layers  and activation functions as they appear in our network.\n\n**Input Layer** (`Input()`: The input layer of a neural network is composed of artificial input neurons, and brings the initial data into the system for further processing by subsequent layers of artificial neurons. The input layer is the very beginning of the workflow for the artificial neural network.\n\n**Convuolution Layer** (`Conv2D()`): Convolutional layers are the layers where filters are applied to the original image, or to other feature maps in a deep CNN. In other words, these layers summarize the presence of features in an input image or extract features.\n\n**ReLu** (`activation='relu'`): ReLu will convert every negative value to 0 and use the +ive value as it is. {-1,0,-12.9} will be outputted to 0 but {1,2,76,0.1} will be outputted as they are {1,2,76,0.1}\n\n**LeakyRelu** (`LeakyRelu()`): Leaky version of a Rectified Linear Unit (ReLu). It allows a small gradient when the unit is not active. With a Leaky ReLU, you won't face the **dead ReLU** (or “dying ReLU”) problem which happens when your ReLU always have values under 0.\n\n**Normalization** (`BatchNormalization()`): Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1. For example, if the values from the previous layers in range -100 to 100000, it'll normalize the entire batch.\n\n**Pooling** (`MaxPool2D()`): Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map. Max Pooling calculate the maximum value for each patch of the feature map or in very very simple words for example, it'll return a Maximum value of each `2*2` matrix `Max(00,01,10,11)` that is present in each `N*N` it encounters **IFF** step=1. For images we use 2D and for videos and sound, 3D.\n\n**Dropout** (`Dropout()`): Dropout Layer will drop `n` proportion of values it gets from the previous layers. It is a method of preventing *Overfitting*. IF Dropout is set to .4, it'll randomly drop the weights 40% of neurons which were provided by the previous layer. Dropout may be implemented on any or all hidden layers in the network as well as the visible or input layer. It is not used on the output layer.\n\n**Flatten** (`Flatten()`): Flatten layer performs a flatten operation on a tensor, reshapes the tensor to have the shape that is equal to the number of elements contained in tensor non including the batch dimension. Not getting it right? Well for example, `Flatten()` will take a tensor of any shape and transform it into a one dimensional tensor (plus the samples dimension) but keeping all values in the tensor. For example a tensor (samples, 10, 20, 1) will be flattened to (samples, 10 * 20 * 1). Simply, **(N-D tensor collapsed into 1-D)**.\n\n**Sotmax** (`activation='softmax'`): Softmax activation function gives the probability of each class. Let us suppose that in each class (in our problem we have 169,11,7 classes respectively for each label) we get a probability, the class assigned to the image will be the class with the highest probability.\n\n### NOTE:\nBefore you go on searching and wondering about why these many or so less layers, why these many numbers of neurons, why droput is set to be at this fraction, why normalization and why/why not's of ocean, these are just trial and error parameter tunings and testing with permutations which cost us very less computational power and time. There is not panacea. You have to check it for yourself and see what fits better. But for more details on this one you can read research paper from [Yoshua Bengio](https://www.quora.com/profile/Yoshua-Bengio) on [Practical recommendations for gradient-based training of deep architectures](https://arxiv.org/abs/1206.5533) and [Jeff Heaton](https://github.com/jeffheaton)'s research on [The Number of Hidden Layers](https://www.heatonresearch.com/2017/06/01/hidden-layers.html)"},{"metadata":{"trusted":true},"cell_type":"code","source":"inputs = Input(shape = (final_w, final_h, 1)) \n# Input layer just takes into account of the size of the input \n\nmodel = Conv2D(filters=64, kernel_size=(5,5), padding='SAME', activation='relu')(inputs) \n# filter size is 64 and kernal is a (3,3) matrix. padding='SAME' means that at each side, o's are padded\n# to each image so that the corner bits get same attention as the middle ones.\n\nmodel = Conv2D(filters=64, kernel_size=(3, 3), padding='SAME')(model)\nmodel = LeakyReLU(alpha=0.17)(model) \n# using LeakyRelu as a new layer. We could also use Relu() as layer too instead of activation='relu'\n\nmodel = Conv2D(filters=64, kernel_size=(3, 3), padding='SAME')(model)\nmodel = LeakyReLU(alpha=0.13)(model)\nmodel = BatchNormalization(momentum=0.15)(model)\n# momentum=0.15 means move the mean and variance by a factor of 0.15\n\nmodel = MaxPool2D(pool_size=(2, 2))(model) \n# Get maximum value of every 2-by-2 matrix present in matrix it gets and return a new matrix of these values\nmodel = Conv2D(filters=64, kernel_size=(3, 3), padding='SAME', activation='relu')(model)\nmodel = BatchNormalization(momentum=0.15)(model)\nmodel = Dropout(rate=0.3)(model) # Drop weights from 30% of neurons from previous layer\nmodel = Conv2D(filters=32, kernel_size=(3, 3), padding='SAME')(model)\nmodel = LeakyReLU()(model)\nmodel = BatchNormalization(momentum=0.15)(model)\nmodel = MaxPool2D(pool_size=(3, 3))(model)\nmodel = Conv2D(filters=64, kernel_size=(5, 5), padding='SAME', activation='relu')(model)\nmodel = Conv2D(filters=64, kernel_size=(2, 2), padding='SAME', activation='relu')(model)\nmodel = BatchNormalization(momentum=0.15)(model)\nmodel = MaxPool2D(pool_size=(2, 2))(model)\nmodel = Dropout(rate=0.4)(model)\nmodel = Conv2D(filters=64, kernel_size=(2, 2), padding='SAME', activation='relu')(model)\nmodel = MaxPool2D(pool_size=(2, 2))(model)\nmodel = Flatten()(model) # Convert into 1-D tensor\nmodel = Dense(512)(model) \nmodel = LeakyReLU()(model)\nmodel = Dropout(rate=0.35)(model)\nmodel = BatchNormalization(momentum=0.15)(model)\nmodel = Dropout(rate=0.3)(model)\ndense = Dense(192, activation = \"relu\")(model) \n\nout1 = Dense(168, activation = 'softmax',name='out_1')(dense) # names of output layers. We need these names\nout2 = Dense(11, activation = 'softmax',name='out_2')(dense)  # as they act as the keys for mapping output\nout3 = Dense(7, activation = 'softmax',name='out_3')(dense)   # to each later. See in the model.fit()\n\nmodel = Model(inputs=inputs, outputs=[out1,out2,out3]) # final line of our model construction code\n# tell the system only the start and end, it'll find the path","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom metric\nIn keras we can implement a custom metric as only few of the common metrices are available though you can get those metrices later too. We need custom metrices because it is the problem's demand. For example, in case of Cancer detection, we can use Recall rather than accuracy because a False Positive (model says you have Cancer when you actually don't) will lead to few more tests from doctors but a False Negative(model says you are fit but you are not rally) will lead to a delayed treatment that can be harmful.\n\nIn our case, the organizers have already given us the metric on which they'll be evaluating our scores so why not to use it from start when we will be judged in the end on this basis?? (Wish we could have same for the life too). \n\nMetrices is keras can be implemented using ```keras.backend```\n `For Later`"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compiling\nUnlike other frameworks like PyTorch, Keras is not so pythonic in case of dynamic behaviour. You have to compile it before you can train your model. Compilation will calculate all the required numbers of parameters to train, the loss function to use, assign them to each layer, get a memory space from os, assign accuracy metric etc.\n\nWe are using Adam as an optimizer by defining `optimizer='adam`. You can also use the `Adam()` imported already and set the learning rate. Optimizers help in reducing the number of computations drastically.\n\nCategorical loss is used as the loss function. Simply, it'll tell us the average difference between the classes that were actually there and those were predicted by the model and if we get a difference in the values, scold our model and ask every neuron in process to tweak their output so that the resulting somehow becomes the original.**Every neuron involved in producing that final value will ask every other previous value to tell teak parameters. This process goes from the output layer to input layer and is called *Backpropagation*.**\n\nAccuracy is the metric we are using. It is simply a measure of how accurae we were in predictions.\n\n**We can use multiple metrices and define each output layer's loss and metric seperately by using the `loss={'layer_1_name':[list,of,metrices]}`.** For this very demo, you can see a very good example of [using muti-loss multi-metrics for multi-output NN](https://github.com/keras-team/keras/issues/10306)\n\n\n**We can also define our own custom metrices and loss functions.** You can follow [custom loss function example](https://stackoverflow.com/questions/59603353/using-multi-output-labels-in-keras-imagedatagenerator-flow-and-using-model-fit) and [custom metric in keras example](https://datascience.stackexchange.com/questions/45165/how-to-get-accuracy-f1-precision-and-recall-for-a-keras-model)"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Callbacks\nCallbacks are a method of invisibly inserting our commands basically if/else conditions to show,stop or do something when some conditions are met. We want model to show or do something while in training, we do not want to interrupt training, we use callbacks. We can use any number of callbacks but we'll be using 3 in our case. We are using callbacks *as whole* but can use these on each individual output layers too as `monitor='val_loss_out_1'` i.e `MetricToWatch_LayerName`\n\n**ReduceLROnPlateau** will reduce the leraning rate of optimizer by a defined factor if any of the output  metric/average metric hits plateau. It means that if validation loss is not improving or if accuracy is not improving, decrease the learning rate.\n\n**EarlyStopping** is used to avoid resource wastage basically. It'll stop the model from training if given parameters like accuracy is not improving or loss is not decreasing ^significantly. **We define the significance**\n\n**ModelCheckPoint** is used to save the whole model or just the weights if our model improves by the criteria of improvement defined."},{"metadata":{"trusted":true},"cell_type":"code","source":"R_LR_P = ReduceLROnPlateau(monitor='val_out_1_loss',patience=3,verbose=1,factor=0.5,min_lr=0.00001)\n# if validation loss of out_1 is not decreasing for 3 consecutive epochs, decrease the learning rate by 0.5 \n# given if learning rate is above 0.00001 and give us little insight on what has happened verbose=1\n\nES = EarlyStopping(monitor='val_loss',patience=4, min_delta=0.0025,restore_best_weights=True)\n# stop the model from fitting data if validation loss has not decreased by 0.0025 in the last 5 epochs and \n# restore best weights for the next time\n\nMCP = ModelCheckpoint('best_model_weight.h5', monitor ='val_loss', verbose =1, \n                      save_best_only = True, save_weights_only=True)\n# save the weights in a file name specified only if the validation loss of out_1 layer has improved from \n# last save. out_1 because it's recall matters twice \n\ncallbacks = [R_LR_P,ES,MCP]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{},"cell_type":"markdown","source":"### Resize\nWe need to resize our images. We have many reasons for these but the main 2 reasons in our case are computational power and memory. You an not load all the images,resize,reshape, train-test-split and fit the model. (Not in this case, Try doing it). `(137*236=32332)` pixels take more memory and computational power than that of `(74*74=5476)` pixels. Obviously we lose a bit of information but the ratio between (computation & memory) to the (performace & information) will be decreased greatly. We can get a very good result by losing a very little information."},{"metadata":{"trusted":true},"cell_type":"code","source":"def resize_image(df_in,curr_w,curr_h,res_w,res_h,lib='cv2'):\n    '''\n    Resizes all the images in a Dataframe where the columns represent the pixels of GRAYSCALE image\n    and returns the resized DataFrame\n    Input:\n        df_if: Dataframe whose columns represents the pixels of image\n        curr_w,curr_h: Current Width and Height or dimensions of image\n        res_w,res_h: Desired Width and Height of the output image\n        lib: library to be used for resizing. 'skimage' for 'scikit-image' and default='cv2' for Open CV\n    Output:\n        resized:\n            Dataframe like df_in but with changed Width and Height\n    '''\n    \n    resized = {} # empty dictonary which stores the values of pixels for each image\n    if lib=='skimage':\n        for i in range(df_in.shape[0]): # iterate through all the images in dataframe\n            image = resize(df_in.loc[df.index[i]].values.reshape(curr_w,curr_h),(res_w,res_h,1))\n            # apply resize transformations on per-row basis \n            resized[df_in.index[i]] = image.reshape(-1)  # reshape accordingly\n        resized = pd.DataFrame(resized).T # resizing swaps the rows to columns so Transpose sets to default\n    \n    else: \n        for i in range(df_in.shape[0]):\n            image = cv2_resize(df_in.loc[df.index[i]].values.reshape(curr_w,curr_h),(res_w,res_h))\n            resized[df_in.index[i]] = image.reshape(-1)\n        resized = pd.DataFrame(resized).T \n    return resized","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Generator\n`ImageDataGenerator()` is a very good tool for generating and augmenting images. What it does is to take all the image files either from directory or in form of rank-4 metrices (`number_of_pictures,width,height,channel`) and maps to the corresponding labels. We can pass in the images ourselves too but what this generator best is that it augments the images. It can resize,crop,shift,flip,rotate  iamges for us and so on. Why do we need to? Good one! So common that even Facebook has the answer,literally.\n\nNeural network is no magic but it just learns the images somehow. (Must read paper from [Chiyuan Zhang](http://pluskid.org/) on [deep learning requires rethinking](https://arxiv.org/abs/1611.03530) tells that a Neural Network crams the wrong data too much that it tells them correctly. If given enough time,data and wrong label to a network, it'll learn to classify dog as a car with 100% accuracy)\n\nSo if it learn to classify a dog that is in the center of image everytime facing right, it won;t be able to classify the same dog in corner, or facing left and so on. So to make our model robust, we need augmentations."},{"metadata":{"trusted":true},"cell_type":"code","source":"class CustomDataGenerator(ImageDataGenerator):\n    '''\n    This class extends the ImageDataGenerator but as the parent class only map 1 class label to each image\n    For example it can only map if a picture of car is black or white but we are trying to map it to\n    N classes so that it can override the default flow() and provide a mapping of a car to color,model,\n    company etc. Specially useful if you have different losses for each class so you have to pass a dict\n    of y_labels\n    \n    This code's credit goes to - https://github.com/keras-team/keras/issues/12639\n    '''\n    \n    def flow(self,x,y=None,batch_size=BATCH_SIZE,shuffle=True,sample_weight=None,seed=None,save_to_dir=None,\n             save_prefix='',save_format='png',subset=None): \n        '''\n        Function takes data & label arrays, generates batches of augmented data (#official keras Documents)\n        Input:\n            x: Flow method looks for Rank-4 numpy array. i.e (number_of_images,width,height,channels)\n            y: dictonary which maps each picture to its ONE-HOT ENCODES respective classes such as  \n            if Image1 is associated to 3 classes in a way ->[0,1,2] and Image2 is associated as [3,4,5] so\n            the y will be as y={'y1':to_categorical([0,3]),'y2':to_categorical([1,4])...and so on} \n            others: default settings of parameters in the original flow() method\n        Output:\n            Just like the default flow(), it'll generate an instance of image array x  but instead of a \n            single y-label/class mapping it'll produce a a dictonary as label_dict that contains mapping \n            of all the classes for that image\n        '''\n\n        labels_array = None # all the labels array will be concatenated in this single array\n        key_lengths = {} \n        # define a dict which maps the 'key' (y1,y2 etc) to lengths of corresponding label_array\n        ordered_labels = [] # to store the ordering in which the labels Y were passed in this class\n        for key, label_value in y.items():\n            if labels_array is None:\n                labels_array = label_value \n                # for the first time loop, it's empty, so insert first element\n            else:\n                labels_array = np.concatenate((labels_array, label_value), axis=1) \n                # concat each array of y_labels \n                \n            key_lengths[key] = label_value.shape[1] \n            # key lengths will be different for different range of classes in each class due to_categorical \n            # ONE-HOT encodings. Ex- some have 2 classes (red,yellow) but other can have 4 \n            # (Audi,BMW,Ferrari,Toyota) so we have to keep track  due to inner working of super().flow()\n            ordered_labels.append(key)\n\n\n        for x_out, y_out in super().flow(x, labels_array, batch_size=batch_size):\n            label_dict = {} # final dictonary that'll be yielded\n            i = 0 # keeps count of the ordering of the labels and their lengths\n            for label in ordered_labels:\n                target_length = key_lengths[label]\n                label_dict[label] = y_out[:, i: i + target_length] \n                # Extract to-from the range of length of labels values. That is why we had ordered_labels\n                # and key_lengths It'll extract the elements ordering vise else there will be conflict\n                i += target_length\n\n            yield x_out, label_dict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training\nWe do our work in loop. There are 2 methods that I could come up with.\n* Import data, resize, merge in 4 iterations and then train the whole data at once\n* Import data, resize, train  in 4  different iterations\n\nEach method has its own drawback. First method can not perform well on or above images which are resized as `(64*64)` but with second, we lose the element of randomness as we just have 1/4 of the original sample to choose from. I chose to go for second. So we'll be applying the process as\n\n1. Perform Import\n2. Merge\n3. Extract Labels\n4. Resize \n5. Reshape \n6. Train_Test Split \n7. Train\n8. Delete\n9. Repeat"},{"metadata":{},"cell_type":"markdown","source":"## Garbage Collection \nEven though Grabage collection is done automatically in Python by the `gc` and in periods but we'll use the `gc.collect()` method to force the collector but it is thought to be a bad practice to use *del* because it removes the reference of that object from the memory.The community still stands divided on this whether to delete and force collector free up memory. In my case, it has been a journey of luck. Sometimes I get the freed space, sometimes I don't but we are forcing the garbage collector to collect and free up space for the sake of it."},{"metadata":{"trusted":true},"cell_type":"code","source":"frame = pd.DataFrame({'-1':['File -1']}) # just a random dataframe nothing much\ndrop_cols = ['image_id','grapheme_root','vowel_diacritic','consonant_diacritic','grapheme'] \n# columns to be dropped after merging with dataframe so that we can have only numerical pixel values\n\nj=-1\nfor i in tqdm(range(0,4)):\n    j+=1\n    gc.collect() # calling garbage collector to free up memory from last delete\n    \n    print('Importing file %s'%j)\n    df = pd.merge(pd.read_parquet(TRAIN_IMG_PATH+str(j)+FILE_TYPE),train_classes,on='image_id')\n    # Pandas automically assigns the correct columns by looking at the values in 'image_id' so we don't\n    # have to worry about alignment if the 'image_id' in train_classes are correct to the respect classes\n    \n    grapheme = pd.get_dummies(df['grapheme_root']).values # get the numerical values of classes and convert\n    vowel = pd.get_dummies(df['vowel_diacritic']).values  # them to categorical so that [0,1,2] becomes \n    consonant = pd.get_dummies(df['consonant_diacritic']).values # [[1,0,0],[0,1,0],[0,0,1]] where the size of the\n    # later depends on the maximum number that is present in the original matrix. Position of 1 defines the \n    # presence of the elemnt in the list. For example if [1,3] is original array, then one-hot encoded array\n    # will look something like {0:[0,0,0,0],1:[0100],2:[0,0,0,0],3:[0,0,0,1]}\n    # NOTE- this is not the actual representatin but the values are correct .I can not plot a 2-D array in #s\n    \n    print('Dropping Columns.....')\n    df.drop(drop_cols,axis=1,inplace=True) # remove all the columnsthat are not of use\n    \n    print('Resizing....')\n    df = (resize_image(df_in=df,curr_w=137,curr_h=236,\n                           res_w=final_w,res_h=final_h)/255.).astype('float32')\n    \n    # resize the images and then divide each pixel by 255 to limit the range from 0-255 to 0-1 and convet as\n    # float32 (memory efficient)\n    \n    print('Reshaping into Rank-4..... ')\n    df = df.values.reshape(-1, final_w,final_h,1) # reshape into rank-4 matrix \n    # -1 checks for the best dimension. It'll be equal to number of elements  or df.shape[0] in our case\n    \n    print('Splitting Train-Test...')\n    df, x_test, y_train_graph, y_test_graph, y_train_vowel, y_test_vowel, y_train_conso, y_test_conso =\\\n    train_test_split(df, grapheme,vowel,consonant, test_size=0.15, random_state=13)\n    # split the data in training and test to cross-validate our model's performance. random_state produces\n    # the same data every time for reproducibility so it is kind of pseudo randomness\n    \n    del grapheme\n    del vowel\n    del consonant\n    gc.collect()\n    \n    print('Fitting into Generator.....')\n    aug_generator = CustomDataGenerator(featurewise_center=False,samplewise_center=False,\n                                        featurewise_std_normalization=False,samplewise_std_normalization=False,\n                                        zca_whitening=False,rotation_range=13, zoom_range = 0.17,\n                                        width_shift_range=0.18,height_shift_range=0.16,horizontal_flip=False,\n                                        vertical_flip=False) \n    \n    #featurewise_center : would have set input mean to 0 over the dataset if stated True\n    #samplewise_center : would have set each sample mean to 0 if stated True\n    #featurewise_std_normalization : would have divided inputs by std of the dataset if set True\n    #samplewise_std_normalization : would have divided each input by its std if True\n    #rotation_range=8 : will rotate the image in range 8 degrees\n    #zoom_range = 0.15 : will zoom in or out the image propotional to original image\n    #width_shift_range=0.15 : will shift the width 0.15 points respective to total width\n    \n    aug_generator.fit(df)\n    # this will JUST calculate parameters required (PCA, ZCA and others if True) no transformations performed\n    \n    print('Starting Training........')\n    history = model.fit_generator(aug_generator.flow(df, {'out_1': y_train_graph, 'out_2': y_train_vowel, \n                                                     'out_3': y_train_conso}, batch_size=BATCH_SIZE),\n                              epochs = 27, validation_data = (x_test, [y_test_graph, y_test_vowel, y_test_conso]),\n                              steps_per_epoch=df.shape[0] //128 ,callbacks=callbacks)\n    \n    stopped_at = ES.stopped_epoch # gives you >=1 at which epoch model stopped due to early stopping\n    \n    # labels are givesn as a dictonary parameter to the fit_generator because it is multi output problem. \n    # epoch is the iteration for which we will train the model. At each epoch, same images will be shown to\n    # model and the weights and biases will be adjusted accordingly. More epochs than required can lead model\n    # to overfitting. Overfitting is cramming in a sense. Out of Notebook question, 0 marks scored.\n    \n    del df\n    del x_test\n    del y_train_graph\n    del y_test_graph\n    del y_train_vowel\n    del y_test_vowel\n    del y_train_conso\n    del y_test_conso\n    frame[j]='Files completed is'+str(j) # nothing\n    frame.to_csv('files.csv') # saving nothing is like committing when offline and preparing a log if crash happens\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('best_model_weight.h5') \n\n# load the best weights so far so that we don't have over fitting by the time the mode lhad stopped ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predictions and Submissions"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_dict = {\n    'grapheme_root': [],\n    'vowel_diacritic': [],\n    'consonant_diacritic': []\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"components = ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']\ntarget=[] # model predictions placeholder\nrow_id=[] # row_id place holder\nfor j in range(4):\n    print('Processing file %s'%j)\n    df = pd.read_parquet(TEST_IMG_PATH+str(j)+FILE_TYPE)\n    df.set_index('image_id', inplace=True)\n    index_values = df.index.values\n\n    df = (resize_image(df_in=df,curr_w=137,curr_h=236,\n                           res_w=final_w,res_h=final_h)/255.).astype('float32')\n    df = df.values.reshape(-1, final_w, final_h, 1)\n    \n    preds = model.predict(df)\n\n    for i, p in enumerate(preds_dict):\n        preds_dict[p] = np.argmax(preds[i], axis=1)\n\n    for k,id in enumerate(index_values):  \n        for i,comp in enumerate(components):\n            id_sample=id+'_'+comp\n            row_id.append(id_sample)\n            target.append(preds_dict[comp][k])\n    \n    del df\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({'row_id': row_id,'target':target},columns = ['row_id','target'])\nsubmission.to_csv('submission.csv',index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"exec_time = time.time()-start_time\nprint('Total Execution time of Script is',exec_time)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What next?\n## What Next?\nFrom this point onwards, you can make the models more robust and efficient using different techniques such as adding more layers until the model reaches saturation point, using dimentionality reduction techniques such as PCA, ISOmap to reduce the overhead by keepong the most significant data pixels and tweaking the parameters of your model.\n\nTo get the best of your model and knowledge, read more apply the most. You can follow Medium blogs and one of my favourites blogs is [Machine Leraning Mastry](https://machinelearningmastery.com/) and obviously you can always follow and ask questions on [Stackoverflow](https://stackoverflow.com/) and [Cross Validated](https://stats.stackexchange.com/). \nWish you all the best!\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Thank you all for coping with my gramatical errors."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}