{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Inference Notebook is found here\n[Inference Notebook](https://www.kaggle.com/shubham219/experiment-with-models-in-keras-inference)"},{"metadata":{},"cell_type":"markdown","source":"This is my first Computer Vision Competition. This is notebook is just for some random experiments and to see how things impacts the model and accuracy. \n\nIf you have any suggestion please comment and please upvote. "},{"metadata":{},"cell_type":"markdown","source":"# Updates \n\n## Update 1 - No Image Augmentation  - VGG16 Base(freeze)\nModel was trained on VGG16 base with only one dense layer. VGG16 parameters are freezed and only dense parameters are allowed to trained. Model was slightly overfitting and the rank on public leader board is 0.64\n\n## Update 2 - Image Augmentation - VGG16 Base(freeze)\nModel was trained on VGG16 base with only one dense layer. VGG16 parameters are freezed and only dense parameters are allowed to trained. Using Image Augmentation Validation Loss and Validation Accuracy are less than from the previous model(Without Augmentation) but model performed well and accuracy improved to 0.65 on public leader board\n\n## Update 3 - Image Augmentation With Class Weights\nModel was trained on VGG16 base with only one dense layer and putting some class weights. VGG16 parameters are freezed and only dense parameters are allowed to trained. Model did not perform well and accuracy got dropped to 0.50\n\n## Update 4 - Few Layers Unfreezed\nModel was trained on VGG16 base with only one dense layer but this time unfreezing the last convolution block and training it with low learning rate but the model is trainined first on freezed layer **Model performed really well and got accuracy of 0.763 from 0.65**\n\n## Update 5 - Trying Model Ensembling\nAccuracy Jumed to 0.803"},{"metadata":{},"cell_type":"markdown","source":"# Importing All The Required Liraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/efficientnet-keras-source-code/repository/qubvel-efficientnet-c993591\n!pip install -U efficientnet\n!pip install /kaggle/input/keras-pretrained-imagenet-weights/image_classifiers-1.0.0-py3-none-any.whl\n!pip install image-classifiers\n!pip install keras-applications==1.0.8","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nimport shutil\nfrom functools import partial\nfrom matplotlib import pyplot as plt\nfrom sklearn.model_selection import train_test_split \nfrom sklearn.utils import class_weight\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import KFold\nfrom classification_models.tfkeras import Classifiers\n\nprint(\"Tensorflow version -\",tf.__version__)\nprint(\"Python version\")\n!python --version","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU Config"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(\"Device:\", tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint(\"Number of replicas:\", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Variables"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"EPOCHS = 100\nIMAGE_SIZE = [512, 512]\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nBUFFER_SIZE = 32\nFOLD = 5\nSEED = (2, 3)\nBATCH_SIZE = AUG_BATCH = 16*strategy.num_replicas_in_sync\nGCS_PATH_ORG = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_COMB= KaggleDatasets().get_gcs_path('cassava-old-new-data-600-800')\n\nloss_list = []\nacc_list = []\nval_acc_list = []\nval_loss_list = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading Metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\n\nwith open('../input/cassava-leaf-disease-classification/label_num_to_disease_map.json') as file:\n    text = file.read()\nprint(text)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Distribution Of Classes\nDataset is very imbalanced"},{"metadata":{"trusted":true},"cell_type":"code","source":"figure = plt.figure(figsize=(8,4))\n(data['label'].value_counts()/len(data)*100).plot(kind='bar')\nplt.title(\"Distribution of Classes\")\nplt.ylabel('% count of classes')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Augmentation & Other Functions\nBelow Transformation Code is taken from [Notebook](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Learning rate\nLR = 0.01\n# Test time augmentation rounds\nTTA = 10\n# Verbosity\nVERBOSE = 2\n# Number of classes\nN_CLASSES = 5\n\ndef onehot(image,label):\n    CLASSES = 5\n    return image,tf.one_hot(label,CLASSES)\n\ndef cutmix(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        # MAKE CUTMIX LABEL\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2\n\n\ndef mixup(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n        # MAKE CUTMIX LABEL\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],CLASSES)\n            lab2 = tf.one_hot(label[k],CLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image2,label2\n\n\n\ndef data_augment(image, target):\n    \n    # For Generating A Random Vaue between 0 and 1\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n    p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n    p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n    p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype = tf.float32)\n            \n    # Flips\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    \n    if p_spatial > 0.75:\n        image = tf.image.transpose(image)\n        \n    # Rotates\n    if p_rotate > 0.75:\n        image = tf.image.rot90(image, k = 3) # rotate 270º\n    elif p_rotate > 0.5:\n        image = tf.image.rot90(image, k = 2) # rotate 180º\n    elif p_rotate > 0.25:\n        image = tf.image.rot90(image, k = 1) # rotate 90º\n        \n    # Pixel-level transforms\n    if p_pixel_1 >= 0.4:\n        image = tf.image.random_saturation(image, lower = 0.7, upper = 1.3)\n    if p_pixel_2 >= 0.4:\n        image = tf.image.random_contrast(image, lower = 0.8, upper = 1.2)\n    if p_pixel_3 >= 0.4:\n        image = tf.image.random_brightness(image, max_delta = 0.1)\n        \n    # Crops\n    if p_crop > 0.7:\n        if p_crop > 0.9:\n            image = tf.image.central_crop(image, central_fraction = 0.7)\n        elif p_crop > 0.8:\n            image = tf.image.central_crop(image, central_fraction = 0.8)\n        else:\n            image = tf.image.central_crop(image, central_fraction = 0.9)\n    elif p_crop > 0.4:\n        crop_size = tf.random.uniform([], int(IMAGE_SIZE[0] * 0.8), IMAGE_SIZE[0], dtype = tf.int32)\n        image = tf.image.random_crop(image, size = [crop_size, crop_size, 3])\n    \n    image = tf.image.resize(image, size = IMAGE_SIZE)\n\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n\n    return image, target\n\n\n\ndef transform(image,label):\n    # THIS FUNCTION APPLIES BOTH CUTMIX AND MIXUP\n    DIM = IMAGE_SIZE[0]\n    CLASSES = 5\n    SWITCH = 0.5\n    CUTMIX_PROB = 0.666\n    MIXUP_PROB = 0.666\n    # FOR SWITCH PERCENT OF TIME WE DO CUTMIX AND (1-SWITCH) WE DO MIXUP\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image4 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(AUG_BATCH,CLASSES))\n    return image4,label4\n\n\n\ndef get_val_tta(filenames, ordered = True):\n    dataset = load_dataset(filenames, ordered = ordered)\n    dataset = dataset.map(data_augment, num_parallel_calls = AUTOTUNE)\n    dataset = dataset.repeat()\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset\n\n\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading TF Records"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Decode the data\ndef decode_image(image):\n    print(\"Reading Image\")\n    image = tf.image.decode_jpeg(image,channels=3)\n    image = tf.cast(image, tf.float32)\n    image = tf.image.resize(image, IMAGE_SIZE)\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example, labeled):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.int64)\n    } if labeled else {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string)\n    }\n    \n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    if labeled:\n        label = tf.cast(example['target'], tf.int32)\n        return image, label\n    idnum = example['image_name']\n    return image, idnum\n\ndef load_dataset(filenames, labeled=True):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False  # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(\n        filenames\n    )  # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(\n        ignore_order\n    )  # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(\n        partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE\n    )\n    # returns a dataset of (image, label) pairs if labeled=True or just images if labeled=False\n    return dataset\n\ndef get_training_dataset(filenames, labeled=True ,augment=False, one_hot=True):\n    \n    dataset = load_dataset(filenames, labeled=labeled)\n   \n    if augment:\n        dataset = dataset.map(data_augment, num_parallel_calls = AUTOTUNE)\n        dataset = dataset.repeat()\n        dataset = dataset.batch(AUG_BATCH)\n#         dataset = dataset.map(transform, num_parallel_calls = AUTOTUNE)\n\n    if one_hot:\n        dataset = dataset.map(onehot,num_parallel_calls=AUTOTUNE\n                             )\n#     dataset = dataset.cache()\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n   \n    return dataset\n\ndef get_validation_dataset(filenames, labeled=True,augment=False, one_hot=True):\n    \n    dataset = load_dataset(filenames, labeled=labeled)\n    \n    if augment:\n        dataset = dataset.map(data_augment)\n        dataset = dataset.batch(BATCH_SIZE)\n#         dataset = dataset.map(transform)\n    \n    dataset = dataset.batch(BATCH_SIZE)\n    \n    if one_hot:\n        dataset = dataset.map(onehot,num_parallel_calls=AUTOTUNE\n                             )\n#     dataset = dataset.cache()\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n   \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Splitting the Data Into Train And Test"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Old and New iMages\nx_train, x_test = train_test_split(tf.io.gfile.glob(GCS_PATH_COMB + '/*.tfrec'),\n                                   test_size=0.3,\n                                   random_state=123)\n\n\n# Creatig the data Set \ntrain_ds = get_training_dataset(x_train, augment=True)\nvalid_ds = get_validation_dataset(x_test, augment=False, one_hot=True)\n\n\n# Counting the Number Of Files\ntrain_files_cnt = sum([int(i) for i in re.findall('train\\d+-(\\d+)', str(x_train))])\nvalid_files_cnt = sum([int(i) for i in re.findall('train\\d+-(\\d+)', str(x_test))])\n\nprint(f'Number of Training And Validation Files are - {train_files_cnt} and {valid_files_cnt}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing The Training Dataset\nSome of the images contains lots of leaves and image is taken from far\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"figure = plt.figure(figsize=(10,10))\n\nfor image, label in train_ds.take(1):\n    print(label.shape)\n    for i in range(9):\n        plt.subplot(330+1+i)\n        plt.imshow(np.array(image[i]).astype('uint8'))\n#         plt.title(int(label[i]))\n        plt.axis('off')\n        \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing The Validation Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"figure = plt.figure(figsize=(10,10))\n\nfor image, label in valid_ds.take(1):\n    print(image.shape)\n    print(label.shape)\n    for i in range(9):\n        plt.subplot(330+1+i)\n        plt.imshow(np.array(image[i]).astype('uint8'))\n#         plt.title(int(label[i]))\n        plt.axis('off')\n        \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Preparation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(input_shape):\n    \n    inputs = tf.keras.Input(shape=input_shape)\n    \n    #Rescaling\n    scales_input = tf.keras.layers.experimental.preprocessing.Rescaling(1./255)(inputs)\n    \n    # Model 1\n    base1 = conv_base1(scales_input)\n    base1_x = tf.keras.layers.GlobalAveragePooling2D()(base1)\n    base1_x = tf.keras.layers.Dense(256)(base1_x)\n    base1_x = tf.keras.layers.Activation('sigmoid')(base1_x)\n    \n    \n    # Model 2\n    base2 = conv_base2(scales_input)\n    base2_x = tf.keras.layers.GlobalAveragePooling2D()(base2)\n    base2_x = tf.keras.layers.Dense(256)(base2_x)\n    base2_x = tf.keras.layers.Activation('sigmoid')(base2_x)\n    \n    # Model 3\n    base3 = conv_base3(scales_input)\n    base3_x = tf.keras.layers.GlobalAveragePooling2D()(base3)\n    base3_x = tf.keras.layers.Dense(256)(base3_x)\n    base3_x = tf.keras.layers.Activation('sigmoid')(base3_x)\n  \n    #Concatenation\n    models = tf.keras.layers.concatenate([base1_x, base2_x, base3_x], axis=-1)\n    x = tf.keras.layers.BatchNormalization()(models)\n    x = tf.keras.layers.Dense(4096)(x)      \n    x = tf.keras.layers.LeakyReLU(0.2)(x)\n    x = tf.keras.layers.Dropout(0.5)(x)\n    \n    x = tf.keras.layers.Dense(512)(x)      \n    x = tf.keras.layers.LeakyReLU(0.2)(x)\n    x = tf.keras.layers.Dropout(0.5)(x)\n \n    x = tf.keras.layers.Dense(256)(x)      \n    x = tf.keras.layers.LeakyReLU(0.2)(x)\n    x = tf.keras.layers.Dropout(0.5)(x)\n    x = tf.keras.layers.BatchNormalization()(x)\n    \n    \n    output = tf.keras.layers.Dense(5, activation='softmax')(x)\n    \n    model = tf.keras.Model(inputs, output)\n    \n    return model\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Pretrained Convolution Base\nFreezing the layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet.tfkeras import EfficientNetB7\nclass_wg_root = '/kaggle/input/keras-pretrained-imagenet-weights/'\n\n\nwith strategy.scope():\n    \n    # Model base 1\n    conv_base1 = EfficientNetB7(include_top=False,\n                                weights='imagenet'\n                               )\n\n    conv_base1.trainable=False\n     \n    # Model base 2\n    SeResNeXT50, preprocess_input = Classifiers.get('seresnext50')\n    SRNXT = SeResNeXT50(input_shape=(*IMAGE_SIZE, 3), include_top=False, weights=None)\n    SRNXT.load_weights(class_wg_root + 'seresnext50_imagenet_1000_no_top.h5')\n    conv_base2 = SRNXT\n\n    conv_base2.trainable=False\n\n    conv_base3 = tf.keras.applications.InceptionResNetV2(include_top=False, \n                                                         weights='imagenet'\n                                                        )\n    conv_base3.trainable=False\n\n    model = make_model([*IMAGE_SIZE,3])\n    \n    model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001),\n                  metrics = tf.keras.metrics.CategoricalAccuracy(),\n                  loss = tf.keras.losses.CategoricalCrossentropy()\n                 )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Network Architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"display(tf.keras.utils.plot_model(model))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5 Fold Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"cv = KFold(n_splits=FOLD, shuffle=True, random_state=123)\n\nX = pd.Series(tf.io.gfile.glob(GCS_PATH_COMB + '/*.tfrec'))\n\nfor n_fold, (train, test) in enumerate(cv.split(X)):\n    \n    print(\"Fold No \", n_fold+1)\n    print(\"Train Indexes -\", train)\n    print(\"Test Indexes -\", test)\n    # Creatig the data Set \n    train_ds = get_training_dataset(X[train.tolist()], augment=True)\n    valid_ds = get_validation_dataset(X[test].tolist(), augment=False, one_hot=True)\n    \n    history = model.fit(train_ds,\n                        validation_data=valid_ds,\n                        epochs=EPOCHS,\n                        steps_per_epoch = train_files_cnt//BATCH_SIZE,\n                        batch_size=BATCH_SIZE,\n                        callbacks=[tf.keras.callbacks.ModelCheckpoint(f'model_v0.77.h5',\n                                                                      save_best_only=True),\n                                   \n                                   tf.keras.callbacks.EarlyStopping(patience=5,\n                                                                    restore_best_weights=True)\n                        ]\n         )\n    \n    acc_list.append(np.mean(history.history['categorical_accuracy']))\n    loss_list.append(np.mean(history.history['loss']))\n    val_acc_list.append(np.mean(history.history['val_categorical_accuracy']))\n    val_loss_list.append(np.mean(history.history['val_loss']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,5))\nplt.plot(list(range(FOLD)), acc_list, label='Training Accuracy')\nplt.plot(list(range(FOLD)), val_acc_list, label='Validation Accuracy')\nplt.xlabel('N-Folds')\nplt.ylabel('Mean Accuracy')\nplt.legend()\nplt.show()\n\nplt.figure(figsize=(8,5))\nplt.plot(list(range(FOLD)), loss_list, label='Training Loss')\nplt.plot(list(range(FOLD)), val_loss_list, label='Validation Loss')\nplt.xlabel('N-Folds')\nplt.ylabel('Mean Loss')\nplt.legend()\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss Vs Epochs"},{"metadata":{},"cell_type":"markdown","source":"# Unfreezing few conv layers now"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of layer in conv base 1 are ', len(conv_base1.layers))\nprint('Number of layer in conv base 2 are ', len(conv_base2.layers))\nprint('Number of layer in conv base 3 are ', len(conv_base3.layers))\n\nnot_to_train = 0.85\n# Training only 20% of the lower layers\nconv_base1_to_train = int(not_to_train*len(conv_base1.layers))\nconv_base2_to_train = int(not_to_train*len(conv_base2.layers))\nconv_base3_to_train = int(not_to_train*len(conv_base3.layers))\n\nprint(\"\\nFrom layer numbers to train\")\nprint(conv_base1_to_train)\nprint(conv_base2_to_train)\nprint(conv_base3_to_train)\n\nconv_base1.trainable=True\nconv_base2.trainable=True\nconv_base3.trainable=True\n\n# # adding regularization\n# regularizer = tf.keras.regularizers.l1(0.00001)\n\n# for layer in model.layers:\n#     for attr in ['kernel_regularizer']:\n#         if hasattr(layer, attr):\n#           setattr(layer, attr, regularizer)\n\ndef unfreeze_model(model, layers_to_train):\n    \n    for layer in model.layers:\n        if isinstance(layer, tf.keras.layers.BatchNormalization): \n            layer.trainable = False\n  \n    for layer in model.layers[:layers_to_train]:   \n        if not isinstance(layer, tf.keras.layers.BatchNormalization): \n            layer.trainable = False\n\nunfreeze_model(conv_base1, conv_base1_to_train)\nunfreeze_model(conv_base2, conv_base2_to_train)\nunfreeze_model(conv_base3, conv_base3_to_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# initial_learning_rate = 0.01\n# epochs = 100\n# decay = initial_learning_rate / epochs\n# def lr_time_based_decay(epoch, lr):\n#     return lr * 1 / (1 + decay * epoch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_list = []\nacc_list = []\nval_acc_list = []\nval_loss_list = []\n\n\ncv = KFold(n_splits=FOLD, shuffle=True, random_state=123)\n\n# X = pd.Series(tf.io.gfile.glob(GCS_PATH_COMB + '/*.tfrec'))\n\nwith strategy.scope():\n\n    model.compile(optimizer = tf.keras.optimizers.Adam(1e-4),\n                  metrics = tf.keras.metrics.CategoricalAccuracy(),\n                  loss = tf.keras.losses.CategoricalCrossentropy()\n                 )\n\nfor n_fold, (train, test) in enumerate(cv.split(X)):\n    \n    print(\"Fold No \", n_fold+1)\n    print(\"Train Indexes -\", train)\n    print(\"Test Indexes -\", test)\n    # Creatig the data Set \n    train_ds = get_training_dataset(X[train.tolist()], augment=True)\n    valid_ds = get_validation_dataset(X[test].tolist(), augment=False, one_hot=True)\n    \n    # .map(lambda x, y : (x, tf.one_hot(tf.cast(y, tf.uint8), depth=5)))\n    \n    history = model.fit(train_ds,\n                    validation_data=valid_ds,\\\n                    epochs=EPOCHS,\n                    steps_per_epoch = train_files_cnt//BATCH_SIZE,\n                    batch_size=BATCH_SIZE,\n                    callbacks=[tf.keras.callbacks.ModelCheckpoint(f'fineTuned_v0.77.h5', save_best_only=True),\n                               tf.keras.callbacks.EarlyStopping(patience=15, restore_best_weights=True),\n                               tf.keras.callbacks.ReduceLROnPlateau(factor=0.2,\n                                                                    patience=7,\n                                                                    verbose=1,\n                                                                    min_lr=0.00001)\n\n                                 \n                    ]\n         )\n    \n    acc_list.append(np.mean(history.history['categorical_accuracy']))\n    loss_list.append(np.mean(history.history['loss']))\n    val_acc_list.append(np.mean(history.history['val_categorical_accuracy']))\n    val_loss_list.append(np.mean(history.history['val_loss']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8,5))\nplt.plot(list(range(FOLD)),acc_list, label='Training Accuracy')\nplt.plot(list(range(FOLD)), val_acc_list, label='Validation Accuracy')\nplt.xlabel('N-Folds')\nplt.ylabel('Mean Accuracy')\nplt.legend()\nplt.show()\n\nplt.figure(figsize=(8,5))\nplt.plot(list(range(FOLD)), loss_list, label='Training Loss')\nplt.plot(list(range(FOLD)), val_loss_list, label='Validation Loss')\nplt.xlabel('N-Folds')\nplt.ylabel('Mean Loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}