{"cells":[{"metadata":{},"cell_type":"markdown","source":"Based on the good response of my first notebook on creating and training a ResNet50 from scratch, I am continuing this series with creating MiniGoogLeNet from scratch using Tensorflow. You can find my original kernel on building ResNet50 from scratch here:\nhttps://www.kaggle.com/sandy1112/create-and-train-resnet50-from-scratch\n\nIf you are liking this series of building popular architectures from scratch, please upvote. That would motivate me to add more architectures in upcoming versions\n\nThis kernel just uses top 100 classes. For actual competition we would need to model all classes with neccessary methods to handle class imbalance","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**The MiniGoogLeNet architecture is based on the paper - 'Understanding deep learning requires rethinking generalization' by Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals, published in 2016**\n\nThe link for the original paper can be found here:\n\nhttps://arxiv.org/abs/1611.03530\n\nThe snapshot of the architecture is given below:","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/minigooglenetarch/minigooglenet_architecture.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Importing the required libraries**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport tensorflow as tf\nimport cv2\nimport skimage.io\nfrom skimage.transform import resize\nfrom imgaug import augmenters as iaa\nfrom sklearn import preprocessing\nfrom sklearn.preprocessing import LabelBinarizer,LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.layers import Input, Add, Dense, Dropout, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D,GlobalAveragePooling2D,Concatenate,concatenate, ReLU, LeakyReLU,Reshape, Lambda\nfrom tensorflow.keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import Adam,SGD\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential, load_model, Model\nfrom tensorflow.keras.callbacks import LearningRateScheduler\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras import metrics\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications.imagenet_utils import preprocess_input\nfrom tensorflow.keras.initializers import glorot_uniform\nfrom tqdm import tqdm\nimport imgaug as ia\nfrom imgaug import augmenters as iaa\nfrom PIL import Image\nimport keras.backend as K\nK.set_image_data_format('channels_last')\nK.set_learning_phase(1)\n\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Path retrieval**\n\n\nTaken from the public kernel...thanks for sharing this. It was quite helpful for me. Please upvote that kernel if you also found that helpful\nhttps://www.kaggle.com/derinformatiker/landmark-retrieval-all-paths","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/landmark-retrieval-2020/train.csv\")\ndef get_paths(sub):\n    index = [\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\",\"a\",\"b\",\"c\",\"d\",\"e\",\"f\"]\n\n    paths = []\n\n    for a in index:\n        for b in index:\n            for c in index:\n                try:\n                    paths.extend([f\"../input/landmark-retrieval-2020/{sub}/{a}/{b}/{c}/\" + x for x in os.listdir(f\"../input/landmark-retrieval-2020/{sub}/{a}/{b}/{c}\")])\n                except:\n                    pass\n\n    return paths","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = train\n\nrows = []\nfor i in tqdm(range(len(train))):\n    row = train.iloc[i]\n    path  = list(row[\"id\"])[:3]\n    temp = row[\"id\"]\n    row[\"id\"] = f\"../input/landmark-retrieval-2020/train/{path[0]}/{path[1]}/{path[2]}/{temp}.jpg\"\n    rows.append(row[\"id\"])\n    \nrows = pd.DataFrame(rows)\ntrain_path[\"id\"] = rows","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Setting up some basic model specs**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 128\nseed = 42\nshape = (64, 64, 3) ##desired shape of the image for resizing purposes\nval_sample = 0.1 # 10 % as validation sample\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = pd.read_csv('../input/landmark-retrieval-2020/train.csv')\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"k =train[['id','landmark_id']].groupby(['landmark_id']).agg({'id':'count'})\nk.rename(columns={'id':'Count_class'}, inplace=True)\nk.reset_index(level=(0), inplace=True)\nfreq_ct_df = pd.DataFrame(k)\nfreq_ct_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels = pd.merge(train,freq_ct_df, on = ['landmark_id'], how='left')\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_ct_df.sort_values(by=['Count_class'],ascending=False,inplace=True)\nfreq_ct_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Due to Kernel runtime limits, for illustrating the example let's run this for top 100 landmark categories**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"freq_ct_df_top100 = freq_ct_df.iloc[:100]\ntop100_class = freq_ct_df_top100['landmark_id'].tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"top100class_train = train_path[train_path['landmark_id'].isin (top100_class) ]\ntop100class_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Setting up some helper functions**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def getTrainParams():\n    data = top100class_train.copy()\n    le = preprocessing.LabelEncoder()\n    data['label'] = le.fit_transform(data['landmark_id'])\n    lbls = top100class_train['landmark_id'].tolist()\n    lb = LabelBinarizer()\n    labels = lb.fit_transform(lbls)\n    \n    return np.array(top100class_train['id'].tolist()),np.array(labels),le","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Credit to Michal Haltuf from whose kernel I had first learnt Data Generators two years back.\nYou can check out his kernel here: https://www.kaggle.com/rejpalcz/cnn-128x128x4-keras-from-scratch-lb-0-328","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Landmark2020_DataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, paths, labels, batch_size, shape, shuffle = False, use_cache = False, augment = False):\n        self.paths, self.labels = paths, labels\n        self.batch_size = batch_size\n        self.shape = shape\n        self.shuffle = shuffle\n        self.use_cache = use_cache\n        self.augment = augment\n        if use_cache == True:\n            self.cache = np.zeros((paths.shape[0], shape[0], shape[1], shape[2]), dtype=np.float16)\n            self.is_cached = np.zeros((paths.shape[0]))\n        self.on_epoch_end()\n    \n    def __len__(self):\n        return int(np.ceil(len(self.paths) / float(self.batch_size)))\n    \n    def __getitem__(self, idx):\n        indexes = self.indexes[idx * self.batch_size : (idx+1) * self.batch_size]\n\n        paths = self.paths[indexes]\n        X = np.zeros((paths.shape[0], self.shape[0], self.shape[1], self.shape[2]))\n        # Generate data\n        if self.use_cache == True:\n            X = self.cache[indexes]\n            for i, path in enumerate(paths[np.where(self.is_cached[indexes] == 0)]):\n                image = self.__load_image(path)\n                self.is_cached[indexes[i]] = 1\n                self.cache[indexes[i]] = image\n                X[i] = image\n        else:\n            for i, path in enumerate(paths):\n                X[i] = self.__load_image(path)\n\n        y = self.labels[indexes]\n                \n        if self.augment == True:\n            seq = iaa.Sequential([\n                iaa.OneOf([\n                    iaa.Fliplr(0.5), # horizontal flips\n                    \n                    iaa.ContrastNormalization((0.75, 1.5)),\n                    iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.5),\n                    iaa.Multiply((0.8, 1.2), per_channel=0.2),\n                    \n                    iaa.Affine(rotate=0),\n                    #iaa.Affine(rotate=90),\n                    #iaa.Affine(rotate=180),\n                    #iaa.Affine(rotate=270),\n                    iaa.Fliplr(0.5),\n                    #iaa.Flipud(0.5),\n                ])], random_order=True)\n\n            X = np.concatenate((X, seq.augment_images(X), seq.augment_images(X), seq.augment_images(X)), 0)\n            y = np.concatenate((y, y, y, y), 0)\n        \n        return X, y\n    \n    def on_epoch_end(self):\n        \n        # Updates indexes after each epoch\n        self.indexes = np.arange(len(self.paths))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __iter__(self):\n        \"\"\"Create a generator that iterate over the Sequence.\"\"\"\n        for item in (self[i] for i in range(len(self))):\n            yield item\n            \n    def __load_image(self, path):\n        image_norm = skimage.io.imread(path)/255.0\n        \n\n        im = resize(image_norm, (shape[0], shape[1],shape[2]), mode='reflect')\n        return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nlabls = top100class_train['landmark_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's build the building blocks of a MiniGoogLNet model one by one**\n\nCredit: Dr. Adrain Rosebrock for his course on Computer Vision. It helped me a lot in clearing my concepts. You may check out his course on:\nhttps://www.pyimagesearch.com/\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The entire MiniGoogLeNet architecture is built on three main modules:\n\n1. Conv Module - this module is responsible for applying a convolution, followed by Batch Normalization and finally with an activation.\n2. Inception Module - this is more like a mini-inception module which consists of two branches. The first branch is a conv module  that learns 1X1 filters. The second brandch is also a Conv Module that learns 3X3 filters.\n3. Downsample Module - this module is responsible for reducing the spatial dimensions of an input volume. The first branch of this module learns a set of K, 3X3 filters using stride of 2X2. On top of this Max pooling is applied with window size of 3X3 and stride of 2X2. Finally outputs of Conv 3X3 and Pool are concated.\n\nThe architecture can now be compiled as follows:\n\n\nInput --> Conv Module --> Inception Module --> Inception Module --> Downsample Module -->Inception Module-->Inception Module-->Inception Module-->Inception Module-->Downsample Module-->Inception Module-->Inception Module-->Average pooling-->Dropout-->Flatten-->Dense-->Activation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def conv_module(x,K,kX,kY, stride, chanDim, padding=\"same\"):\n        x = Conv2D(K, (kX, kY), strides=stride, padding=padding)(x)\n        x = BatchNormalization(axis=chanDim)(x)\n        x = Activation(\"relu\")(x)\n        return x\n\ndef inception_module(x, numK1x1, numK3x3, chanDim):\n    conv_1x1 = conv_module(x, numK1x1, 1, 1,(1, 1), chanDim)\n    conv_3x3 = conv_module(x, numK3x3, 3, 3,(1, 1), chanDim)\n    x = concatenate([conv_1x1, conv_3x3], axis=chanDim)\n    return x\n\ndef downsample_module(x, K, chanDim):\n    conv_3x3 = conv_module(x, K, 3, 3, (2, 2),-1, padding=\"valid\")\n    pool = MaxPooling2D((3, 3), strides=(2, 2))(x)\n    x = concatenate([conv_3x3, pool], axis=chanDim)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build(width, height, depth, classes):\n    inputShape = (height, width, depth)\n    chanDim = -1\n    \n    inputs = Input(shape=inputShape)\n    x = conv_module(inputs, 96, 3, 3, (1, 1),chanDim)\n    x = inception_module(x, 32, 32, chanDim)\n    x = inception_module(x, 32, 48, chanDim)\n    x = downsample_module(x, 80, chanDim)\n    x = inception_module(x, 112, 48, chanDim)\n    x = inception_module(x, 96, 64, chanDim)\n    x = inception_module(x, 80, 80, chanDim)\n    x = inception_module(x, 48, 96, chanDim)\n    x = downsample_module(x, 96, chanDim)\n    x = inception_module(x, 176, 160, chanDim)\n    x = inception_module(x, 176, 160, chanDim)\n    x = AveragePooling2D((7, 7))(x)\n    x = Dropout(0.5)(x)\n    x = Flatten()(x)\n    x = Dense(classes)(x)\n    x = Activation(\"softmax\")(x)\n    model = Model(inputs, x, name=\"googlenet\")\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.metrics import categorical_accuracy,top_k_categorical_accuracy\ndef top_5_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = build(shape[0],shape[1],shape[2], nlabls)\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc',top_5_accuracy])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths, labels,_ = getTrainParams()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"keys = np.arange(paths.shape[0], dtype=np.int)  \nnp.random.seed(seed)\nnp.random.shuffle(keys)\nlastTrainIndex = int((1-val_sample) * paths.shape[0])\n\npathsTrain = paths[0:lastTrainIndex]\nlabelsTrain = labels[0:lastTrainIndex]\n\npathsVal = paths[lastTrainIndex:]\nlabelsVal = labels[lastTrainIndex:]\n\nprint(paths.shape, labels.shape)\nprint(pathsTrain.shape, labelsTrain.shape, pathsVal.shape, labelsVal.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = Landmark2020_DataGenerator(pathsTrain, labelsTrain, batch_size, shape, use_cache=False, augment = True, shuffle = True)\nval_generator = Landmark2020_DataGenerator(pathsVal, labelsVal, batch_size, shape, use_cache=False, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the idea of this kernel was to illustrate the architecture in depth and how to create it from scratch, we will run this for only one epoch. In real world, one will need to train this over a large number of epochs to get good results.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 1\nuse_multiprocessing = True \nworkers = 1 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_cnn = model.fit_generator(\n    train_generator,\n    steps_per_epoch=len(train_generator),\n    validation_data=val_generator,\n    validation_steps=64,\n    #class_weight = class_weights,\n    epochs=epochs,\n    #callbacks = [clr],\n    use_multiprocessing=use_multiprocessing,\n    #workers=workers,\n    verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('MiniGoogLeNet.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**If you found this useful, an upvote would be much appreciated.**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}