{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Data preparation using TensorFlow\n\nI recently tried my skill in a competition on Kaggle called \"Global Wheat Detection\". To be fair I am fairly new in the field and as a beginner, while learning I focused on the model, how to train them etc. But a crucial step which most online courses don't pay attention to is called data pre-processing also termed as data-pipeline. it includes formatting your data so that it can be fed to your model. \n\nEach model is different and they require data in a specific format. reading images with cv2.imread() and then appending them to list will do the job but not efficiently because it will eat up so much RAM that you will have no choice but to reduce batch_size. if an image shape is (128,128,3) it will take somewhere around 10-30 kb. but when you load images as NumPy array it's size will vary according to it's NumPy data type. when you normalize images before feeding them default datatype is float32 (also referred to as single floating point precision). one value for this float 32 takes around 4 bytes, Hence after normalizing your image will take \n\n>total values in an image = 128*128*3 = 49152\n>size of numpy array = 49152*4 = 196608 bytes = 192 kb\n\nThis dataset has 1580470 images, that means,\n\n>size required = 192 * 1580470 = 303450240 kb ~ 289 GB of memory\n\nI don't think any consumer-grade GPU has that much of RAM, As you can see this is why using NumPy array is not optimal<font size=\"4\">\n\nI myself in that old competition did the same thing used 256*256*3 images, but there were only 3422 images which I augmented using hence total images were 6844 it took me 6-7 GB of memory and I had to train the network on a batch size of 16 ultimately I failed miserably but I learned a lot from it and started looking for better ways to implement this. I will share a way which is suitable for all kind of classification tasks hope you will enjoy it\n\nIf You liked it hit the little \" **^** \" icon at upper right (I guess that's an upvote!)\nSo let's dive in!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Step 1 - import the required libraries","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom IPython.display import Image, display\nimport tensorflow as tf\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2 - Set directories\n\n* csv_path - path to the train.csv provided in dataset\n* base_directory - path to the train folder which contains all the sub-directories ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_path = \"../input/landmark-recognition-2020/train.csv\"\nbase_directory = \"../input/landmark-recognition-2020/train\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3 - Read train.csv into pandas dataframe\n* columns - tells what are the column names of the csv \n* annotations - We are loading train.csv as a pandas dataframe in this variable \n* data_frame - This is the dictionary of lists in which we will save our processed values from annotations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = ['id','landmark_id']\nannotations = pd.read_csv(csv_path, usecols=columns)\n\ndata_frame = {\"image_dir\":[],\"landmark_id\":[]}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4 - Process the annotations data frame\n* First we will loop over all rows of the data frame by storing \"id\" column value of the current row in image_id and \"landmark_id\" column value of the current row in land_id variable\n* Then in the second line, we create an absolute path to the image. As explained in the data section first three-letter in id represent subdirectory structure that's what we are doing (* refer to data tab on competitions homepage)\n* On line 3rd and 4th we store this absolute path in our dictionary which we defined earlier \"data_frame\"","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for image_id, land_id in zip(annotations[\"id\"],annotations[\"landmark_id\"]):\n    image_dir = \"{}/{}/{}/{}/{}.jpg\".format(base_directory,image_id[0],image_id[1],image_id[2],image_id)\n    data_frame[\"image_dir\"].append(image_dir)\n    data_frame[\"landmark_id\"].append(land_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5 - Save the new CSV \n* On the first line we first convert our dictionary into pandas data frame we call it \"df\"\n* On the second line we save it as \"train_data.csv\" (we set index = False because we don't want an index in our CSV)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(data_frame)\ndf.to_csv(\"train_data.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Verify the algorithm\nAfter you make something it's always good practice to verify the algorithm using a small sample from data let's do that\nfollowing cell loads our new \"train_data.csv\" into a pandas data frame called data_csv\n(note = make sure you set dtype=str if the data frame is used to generate data for training )","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_csv_path = \"./train_data.csv\"\n\ncolumns = ['image_dir','landmark_id']\ndata_csv = pd.read_csv(data_csv_path, usecols=columns,dtype=str)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let us check is everything is assigned properly \nFollowing cell prints\n* first 7 entries of train.csv\n* first 7 entries of train_data.csv\n* last 7 entries of train.csv\n* last 7 entries of train_data.csv\n\nWe'll manually check this 7 values if id-label pair is correct","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# let us check is everthing is assigned properly\n# This line just tell pandas to print whole string \npd.options.display.max_colwidth = 100\nprint(\"First 7 entries of original train.csv\")\nprint(annotations.head(7))\nprint(\"First 7 entries of our processed train_data.csv\")\nprint(data_csv.head(7))\nprint(\"Last 7 entries of original train.csv\")\nprint(annotations.tail(7))\nprint(\"Last 7 entries of our processed train_data.csv\")\nprint(data_csv.tail(7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hurray! our algorithms seems to be doing perfectly\nsince labels are verified, let's verify that directories \nfollowing cell displays images if images are displayed that means directories are also correct","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's visualize some images from our csv\ntail_part = data_csv.tail(5)\nfor image,label in zip(tail_part[\"image_dir\"],tail_part[\"landmark_id\"]):\n    display(Image(image))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Great Everything seems to be working fine\nnow just summerise how many images and classes the data contain and we will compare this after we generated image_flow","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"no_of_classes=len(annotations[\"landmark_id\"].unique())\nno_of_images=len(annotations[\"id\"].unique())\nprint(\"There are total {} images belonging to {} classes\".format(no_of_images,no_of_classes))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Great !\nNow we know that there are 1580470 images which belongs to 81313 classes","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Step 6 - prepare data for training\n\nTensorflow is very generous it had provided us with a very handy tool called ImageDataGenerator\nThis tool performs mainly two functions \n* Augment the images as specified by the user \n* Load images in batches rather than loading them all at once.\n\nwe will look into both in following cells","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### To use this feature we follow the following steps \n\n1. create a datagen object which dictates how much images will be augmented \n\n    Following cell does just that, In this cell, I have mentioned only some of the augmentation option TensorFlow provides you. What each of the argument does is pretty much self explanatory. In case you want full info on what each one does and what more options are available click [here](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator)\n\n\n\nAlso do not forget to import the function using\n\n>from tensorflow.keras.preprocessing.image import ImageDataGenerator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n    rescale=1./255,\n    shear_range=0.2,\n    rotation_range=30,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    zoom_range=0.2,\n    horizontal_flip=True,\n    validation_split=0.1,\n    fill_mode='nearest')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Now augmentation is taken care of let's look into how to load images from our \"data_csv\" data frame Basically what this function does is load specified images at a time and won't load all of them at once. How many images to load is controlled by hyper-parameter batch_size. for example, if you specified batch_size = 32 then the function will first load 32 images after training on them is finished discard them and load new 32 images. This happens until all images are done training that is the completion of one epoch. Also in some datasets, all images are of different sizes this is a problem because CNN models require images of the same shape this also is taken care of for you in this method.\n\n    okay, once you understand that. Let's look at the method, \n   \n    To generate this flow of images from the data frame, We will call the \"flow_from_dataframe\" method on \"datagen\" object we created in the previous cell. As ImageDataGenerator this method also has different parameter you can learn more about them [here](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_dataframe)\n\n    \n   Some of the important one are :-\n\n* dataframe - put your dataframe here\n\n* x_col - specify column name which contain absolute directories of images\n\n* y_col - specify colum name which contain respective class_id\n\n* target_size - size of images all images will be processed to be of the size you define here\n\n* color_mode - specify color mode see [here](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_dataframe)\n\n* class_mode - specify class mode see [here](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_dataframe)\n\n* batch_size - specify the batch size\n\n* subset - wheter \"training\" or \"validation\" (only works if \"validation_split=0.1\" parameter is provided while creating datagen object see [here](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_dataframe))\n\n    \n\n    \n\nRun the following cell note that this will take a while so if it appears as frozen it's not! let it work. After all, it is working on 1.5 million images!!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 32\ntarget_shape=(64,64)\n\ntrain_generator = datagen.flow_from_dataframe(\n        dataframe = data_csv,\n        x_col = \"image_dir\",\n        y_col = \"landmark_id\",\n        target_size = target_shape,\n        color_mode = \"rgb\",\n        class_mode = \"categorical\",\n        batch_size = batch_size,\n        subset = 'training'\n)\nvalidation_generator = datagen.flow_from_dataframe(\n        dataframe = data_csv,\n        x_col = \"image_dir\",\n        y_col = \"landmark_id\",\n        target_size = target_shape,\n        color_mode = \"rgb\",\n        class_mode = \"categorical\",\n        batch_size = batch_size,\n        subset = 'validation'\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"remember from before we counted 1580470 images belonging to 81313 class?\n\nLet's verify if eveything is loaded accurately \n\n>classes = 81313 (verified)\n\n>images = train_images + validation images = 1422423 + 158047 = 1580470 (verified)\n\nCongratulations! Everything working perfectly\n\nNow the question is how to train them?\n\nprocedure is almost the same with some adjustment.\n\nlet's create a super simple model \n\nnote that even such a simple model will have lots of parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mymodel = tf.keras.Sequential([\n    tf.keras.layers.InputLayer(input_shape=(64,64,3)),\n    tf.keras.layers.MaxPooling2D(2),\n    tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation=\"relu\"),\n    tf.keras.layers.MaxPooling2D(2),\n    tf.keras.layers.Conv2D(filters=4, kernel_size=3, activation=\"relu\"),\n    tf.keras.layers.MaxPooling2D(2),\n    tf.keras.layers.Conv2D(filters=8, kernel_size=3, activation=\"relu\"),\n    tf.keras.layers.MaxPooling2D(2),\n    tf.keras.layers.Flatten(),\n    tf.keras.layers.Dense(256, activation=\"sigmoid\"),\n    tf.keras.layers.Dense(no_of_classes, activation=\"softmax\"),\n])\nmymodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Then Compile the model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mymodel.compile(optimizer=\"Adam\", loss=\"categorical_crossentropy\", metrics=['categorical_accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Finally! train the model ...\n\nhere syntax is little different \n\n* first argument is \"train_generator\" created in previous sevtions\n\n* epoch - no of epoch to train your model\n\n* steps_per_epochs - This should be equal to number of images // batch_size \n\n    Avoid hardcoding this values to avoid errors use train_generator.samples instead to find number of images\n\n* validation_data - specify validation generator object\n\n* validation_ steps - should be equal to (validation_generator.samples//batch_size)\n\n\n\nI have trained only for 1 epoch because training an accurate model is out of the scope of this notebook it is something you have to figure out. this is a competition after all and also even one epoch will take a long time to train","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Convert this cell into the code cell to run \nI am not running following command because of time required due to large data even 1 epoch is taking 3hrs to complete\n\nYou can use fit function like below to train network\n\nmymodel.fit(train_generator, \n            epochs=1,\n            steps_per_epoch=train_generator.samples//batch_size, \n            validation_data = validation_generator,\n            validation_steps=validation_generator.samples//batch_size)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Congratulation!! you have learned how to train a simple classification model But do know that this notebook is about loading data that you can use no problem but the model created in this model won't reach much accuracy in this case. classifying landmarks into 81k classes is no joke. This can not be achieved with a simple classification problem. You will have to implement things like DeLF (DEep Local Features) or think of something new. But don't be discouraged keep trying new things and pushing boundaries of Deep learning\n\n\n\n## ALL THE BEST FOR COMPETITION !!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}