{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nTo start with fact\n\n**Petals to the Metal** is actually one of the book from the series The Adventure Zone...\nTo find more about it please check this [link](https://www.amazon.com/Adventure-Zone-Petals-Metal/dp/1250232635)...\nKeeping this aside, let's get to classifying some flowers\n\n\n\n\n
"},{"metadata":{},"cell_type":"markdown","source":"Please upvote this kernel if you like it. It motivates me to produce more quality content :)"},{"metadata":{},"cell_type":"markdown","source":"This competition comes under **Getting Started Competition** and the main aim is to understand how to use TPU's when there is large amount of data...\nThere is around 4.8GB of data and it contains TFRecords of train, validation, test and a sample_submission.csv\nOur task at hand is to build a classifier which will classify 104 different types of flowers...\n\nSo without any more delay lets get into it.."},{"metadata":{},"cell_type":"markdown","source":"# Update Log\n\n### V7\n* Adding DenseNet201 to the notebook \n* Visualizing the results\n\n### V9\n* Improving notebook aesthetics\n* Added visualizations for loss\n\n### V12\n* Hiding unnecessary code and adding and minor code changes"},{"metadata":{},"cell_type":"markdown","source":"\n
\n

Contents

\n
"},{"metadata":{},"cell_type":"markdown","source":"# Data Preparation\n\n## Importing Dependencies \n\n Back to the table of contents"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras.applications import DenseNet201\n\nimport math, os, re, warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom IPython.display import SVG\n\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.utils import class_weight\nfrom sklearn.metrics import classification_report, confusion_matrix\nfrom tensorflow.keras import optimizers, applications, Sequential, layers\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\nimport tensorflow as tf, tensorflow.keras.backend as K\nfrom tensorflow.keras.models import Model\n\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\ndef seed_everything(seed):\n np.random.seed(seed)\n tf.random.set_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 42\nseed_everything(seed)\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The below code is used to detect hardware and check whether our TPU is working or not... \nThe output returned tells us about appropriate distribution strategy \n\nIf the output is **8 replicas** then TPU is switched on and working fine. \nIf the output is **1 replica** then TPU is not switched on and you can switch it on through **Settings->Accelerator->TPU v3-8**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Setting the parameters \nBack to the table of contents"},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCHS = 20\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nWARMUP_LEARNING_RATE = 1e-4 * strategy.num_replicas_in_sync\nWARMUP_EPOCHS = 3\nHEIGHT = 512\nWIDTH = 512\nIMAGE_SIZE = [224, 224]\nCHANNELS = 3\nN_CLASSES = 104\nES_PATIENCE = 6\nRLROP_PATIENCE = 3\nDECAY_DROP = 0.3\n\nmodel_path = 'DenseNet201_%sx%s.h5' % (HEIGHT, WIDTH)\n\nGCS_PATH = KaggleDatasets().get_gcs_path() + '/tfrecords-jpeg-%sx%s' % (HEIGHT, WIDTH)\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below defined are the 104 classes of flowers which most of the humans cannot classify but a machine can... \nSuch advanced is the field of Computer Vision."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"CLASSES = [\n 'pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea', \n 'wild geranium', 'tiger lily', 'moon orchid', 'bird of paradise', 'monkshood', \n 'globe thistle', 'snapdragon', \"colt's foot\", 'king protea', 'spear thistle', \n 'yellow iris', 'globe-flower', 'purple coneflower', 'peruvian lily', \n 'balloon flower', 'giant white arum lily', 'fire lily', 'pincushion flower', \n 'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy', \n 'prince of wales feathers', 'stemless gentian', 'artichoke', 'sweet william', \n 'carnation', 'garden phlox', 'love in the mist', 'cosmos', 'alpine sea holly', \n 'ruby-lipped cattleya', 'cape flower', 'great masterwort', 'siam tulip', \n 'lenten rose', 'barberton daisy', 'daffodil', 'sword lily', 'poinsettia', \n 'bolero deep blue', 'wallflower', 'marigold', 'buttercup', 'daisy', \n 'common dandelion', 'petunia', 'wild pansy', 'primula', 'sunflower', \n 'lilac hibiscus', 'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', \n 'pink-yellow dahlia', 'cautleya spicata', 'japanese anemone', 'black-eyed susan', \n 'silverbush', 'californian poppy', 'osteospermum', 'spring crocus', 'iris', \n 'windflower', 'tree poppy', 'gazania', 'azalea', 'water lily', 'rose', \n 'thorn apple', 'morning glory', 'passion flower', 'lotus', 'toad lily', \n 'anthurium', 'frangipani', 'clematis', 'hibiscus', 'columbine', 'desert-rose', \n 'tree mallow', 'magnolia', 'cyclamen ', 'watercress', 'canna lily', \n 'hippeastrum ', 'bee balm', 'pink quill', 'foxglove', 'bougainvillea', \n 'camellia', 'mallow', 'mexican petunia', 'bromelia', 'blanket flower', \n 'trumpet creeper', 'blackberry lily', 'common tulip', 'wild rose']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below is the code for the Learning Rate that will be used to train the model..\nNote that we are not starting with a high learning rate because we are fine tuning a model and if we use high learning rate at the beginning then it might break the pretrained weights...\nTo know more about Learning rate warm-up please refer to [this answer](https://stackoverflow.com/questions/55933867/what-does-learning-rate-warm-up-mean)"},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate = 3e-5 * strategy.num_replicas_in_sync\nlr_start = 0.00000001\nlr_min = 0.000001\nlr_max = 3e-5 * strategy.num_replicas_in_sync\nlr_rampup_epochs = 3\nlr_sustain_epochs = 0\nlr_exp_decay = .8\n\ndef lrfn(epoch):\n if epoch < lr_rampup_epochs:\n lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n lr = lr_max\n else:\n lr = (lr_max - lr_min) * lr_exp_decay**(epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n return lr\n \n \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(21 if EPOCHS<21 else EPOCHS)]\ny = [lrfn(x) for x in rng]\n\n\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualizing anything makes understanding better so to understand how learning rate will change over the epochs please hover over the plot"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Scatter(x=rng, y=y,\n mode='lines+markers',\n line=dict(color='royalblue', width=4)))\nfig.update_layout(\n title='Learning Rate Schedule',\n title_x=0.5,\n xaxis_title=\"Range of epochs\",\n yaxis_title=\"Learning rate in 10^-6\",\n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',\n font=dict(\n size=18,\n color=\"red\"\n )\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions \nBack to the table of contents"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Datasets utility functions\nAUTO = tf.data.experimental.AUTOTUNE # instructs the API to read from multiple files if available.\n\n\ndef decode_image(image_data):\n '''\n This method is used to read the input bytes string and\n detects whether an image is a BMP, GIF, JPEG, or PNG,\n and performs the appropriate operation to convert \n a Tensor of type dtype.\n '''\n image = tf.image.decode_jpeg(image_data, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [HEIGHT, WIDTH, 3])\n return image\n\ndef read_labeled_tfrecord(example):\n '''\n This method is used to read an example of train tfrecord \n or validation tfrecord and the output given is an image\n with its label.\n '''\n LABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"class\": tf.io.FixedLenFeature([], tf.int64), # shape [] means single element\n }\n example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n label = tf.cast(example['class'], tf.int32)\n return image, label\n\ndef read_unlabeled_tfrecord(example):\n '''\n Nearly same as `read_labeled_tfrecord`, but in this\n the test tfrecord is read and we do not have test labels\n so the output is image and id_number.\n '''\n UNLABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"id\": tf.io.FixedLenFeature([], tf.string), # shape [] means single element\n # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n }\n example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n idnum = example['id']\n return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n '''\n Used to load the train, validation and test dataset\n '''\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False \n # disable order, increase speed\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) \n # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) \n # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n return dataset\n\ndef data_augment(image, label):\n '''\n Performing different types of augmentations on the data...\n '''\n image = tf.image.random_flip_left_right(image, seed=seed)\n image = tf.image.random_flip_up_down(image, seed=seed)\n image = tf.image.random_saturation(image, lower=0, upper=2, seed=seed)\n image = tf.image.random_crop(image, size=[int(HEIGHT*.8), int(WIDTH*.8), CHANNELS], seed=seed)\n\n return image, label\n\ndef get_training_dataset(do_aug=True):\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n if do_aug: \n dataset = dataset.map(transform, num_parallel_calls=AUTO)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) \n # prefetch next batch while training (autotune prefetch buffer size)\n return dataset\n\ndef get_training_dataset_preview(ordered=True):\n dataset = load_dataset(TRAINING_FILENAMES, labeled=True, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache()\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef get_validation_dataset(ordered=False):\n dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache()\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef get_test_dataset(ordered=False):\n dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef count_data_items(filenames):\n # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization functions \nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n images, labels = data\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n numpy_labels = [None for _ in enumerate(numpy_images)]\n # If no labels, only image IDs, return None for labels (this is the case for test data)\n return numpy_images, numpy_labels\n\ndef disp_images(databatch):\n row = 2; col = 4;\n FIGSIZE = 16.0\n subplot=(row,col,1)\n plt.figure(figsize=(FIGSIZE,FIGSIZE/col*row))\n images, _ = batch_to_numpy_images_and_labels(databatch)\n for j in range(row*col):\n plt.subplot(row,col,j+1)\n plt.axis('off')\n plt.imshow(images[j,])\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentation Functions \nBack to the table of contents\n\n\n\nI will also be adding cutmix, mix-up in the future versions if this notebook recieves a good response"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n # returns 3x3 transformmatrix which transforms indicies\n \n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n shear = math.pi * shear / 180.\n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n rotation_matrix = tf.reshape( tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3] )\n \n # SHEAR MATRIX\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = tf.reshape( tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3] ) \n \n # ZOOM MATRIX\n zoom_matrix = tf.reshape( tf.concat([one/height_zoom,zero,zero, zero,one/width_zoom,zero, zero,zero,one],axis=0),[3,3] )\n \n # SHIFT MATRIX\n shift_matrix = tf.reshape( tf.concat([one,zero,height_shift, zero,one,width_shift, zero,zero,one],axis=0),[3,3] )\n \n return K.dot(K.dot(rotation_matrix, shear_matrix), K.dot(zoom_matrix, shift_matrix))\n\n\ndef transform(image,label):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated, sheared, zoomed, and shifted\n DIM = IMAGE_SIZE[0]\n XDIM = DIM%2 #fix for size 331\n \n rot = 15. * tf.random.normal([1],dtype='float32')\n shr = 5. * tf.random.normal([1],dtype='float32') \n h_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n w_zoom = 1.0 + tf.random.normal([1],dtype='float32')/10.\n h_shift = 16. * tf.random.normal([1],dtype='float32') \n w_shift = 16. * tf.random.normal([1],dtype='float32') \n \n # GET TRANSFORMATION MATRIX\n m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(m,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image,tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3]),label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Train data\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\ntrain_dataset = get_training_dataset_preview(ordered=True)\ny_train = next(iter(train_dataset.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\n\n# Validation data\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nvalid_dataset = get_validation_dataset(ordered=True)\ny_valid = next(iter(valid_dataset.unbatch().map(lambda image, label: label).batch(NUM_VALIDATION_IMAGES))).numpy()\n\n# Test data\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\ntest_dataset = get_test_dataset(ordered=True)\n\nprint('Dataset:\\n {} training images,\\n {} validation images,\\n {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizations\n## Training Images\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"disp_images(next(iter(train_dataset.unbatch().batch(8))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Validation Images\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"disp_images(next(iter(valid_dataset.unbatch().batch(8))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test Images\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"disp_images(next(iter(test_dataset.unbatch().batch(8))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Augmentations\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for i in range(2):\n row = 2; col = 4;\n all_elements = get_training_dataset(do_aug=False).unbatch()\n one_element = tf.data.Dataset.from_tensors( next(iter(all_elements)) )\n augmented_element = one_element.repeat().map(transform).batch(row*col)\n\n for (img,label) in augmented_element:\n plt.figure(figsize=(16,int(16*row/col)))\n for j in range(row*col):\n plt.subplot(row,col,j+1)\n plt.axis('off')\n plt.imshow(img[j,])\n plt.show()\n break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's look at how many flowers of each type are there in the test and train dataset"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_agg = np.asarray([[label, (y_train == index).sum()] for index, label in enumerate(CLASSES)])\nvalid_agg = np.asarray([[label, (y_valid == index).sum()] for index, label in enumerate(CLASSES)])\nfig = go.Figure(data=[\n go.Bar(name='Train', x=train_agg[...,1], y=train_agg[...,0],orientation='h',\n marker=dict(color='rgba(102, 255, 102, 0.5)')),\n go.Bar(name='Validation',x=valid_agg[...,1], y=valid_agg[...,0],orientation='h',\n marker=dict(color='rgba(255, 102, 102, 0.5)'))\n])\nfig.update_layout(\n title='Train and Validation Class distribution',\n title_x=0.5,\n barmode='stack',\n xaxis_title=\"\",\n yaxis_title=\"\",\n font=dict(\n size=10,\n color=\"royalblue\"\n ),\n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',\n autosize=False,\n width=800,\n height=1500,\n margin=dict(\n l=0,\n r=0,\n b=0,\n t=40,\n pad=4\n ))\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the above plot please not that by toggling the **Legend** we can see that the x axis values will change..."},{"metadata":{},"cell_type":"markdown","source":"# Modelling\nBack to the table of contents"},{"metadata":{},"cell_type":"markdown","source":"The first model we will be dealing with is **DenseNet201** \nRecent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. DenseNet connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - DenseNet network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers.\n\nFor more information on DenseNet201 please follow this [link](https://www.kaggle.com/pytorch/densenet201)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model(input_shape, N_CLASSES):\n base_model = applications.DenseNet201(weights='imagenet', \n include_top=False,\n input_shape=input_shape)\n\n base_model.trainable = False # Freeze layers\n model = tf.keras.Sequential([\n base_model,\n layers.GlobalAveragePooling2D(),\n layers.Dense(N_CLASSES, activation='softmax')\n ])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Warm-up the top layers\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with strategy.scope():\n model = create_model((None, None, CHANNELS), N_CLASSES)\n \nmetric_list = ['sparse_categorical_accuracy']\n\noptimizer = optimizers.Adam(lr=WARMUP_LEARNING_RATE)\nmodel.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=metric_list)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As I said earlier in the notebook that anything that is visualized makes understanding things more better so let's visualize the model architecture\n## Fundamental DenseNet Block\nBack to the table of contents"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(Model(model.layers[0].input, model.layers[0].layers[13].output), dpi=75).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above image shows the fundamental block in the DenseNet architecture. The architecture mainly involves Convolution, Maxpooling, ReLU, and concatenation.\n## Model Architecture\nBack to the table of contents"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model, dpi=75).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nwarmup_history = model.fit(x=get_training_dataset(), \n steps_per_epoch=STEPS_PER_EPOCH, \n validation_data=get_validation_dataset(),\n epochs=WARMUP_EPOCHS, \n verbose=2).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Remember the learning rate plot where there was a linear increase for first three epochs...\nThe above code is warming up the top layers so that we can fine tune it in the next step\n## Fine Tuning all the layers\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for layer in model.layers:\n layer.trainable = True # Unfreeze layers\n\ncheckpoint = ModelCheckpoint(model_path, monitor='val_loss', mode='min', save_best_only=True)\nes = EarlyStopping(monitor='val_loss', mode='min', patience=ES_PATIENCE, \n restore_best_weights=True, verbose=1)\nlr_callback = LearningRateScheduler(lrfn, verbose=1)\n\ncallback_list = [checkpoint, es, lr_callback]\n\noptimizer = optimizers.Adam(lr=learning_rate)\nmodel.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=metric_list)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"history = model.fit(x=get_training_dataset(), \n steps_per_epoch=STEPS_PER_EPOCH, \n validation_data=get_validation_dataset(),\n callbacks=callback_list,\n epochs=EPOCHS, \n verbose=2).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize the results\nBack to the table of contents"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def display_training_curves(training, validation):\n fig = go.Figure()\n \n fig.add_trace(\n go.Scatter(x=np.arange(1, EPOCHS+1), mode='lines+markers', y=training, marker=dict(color=\"dodgerblue\"),\n name=\"Train\"))\n \n fig.add_trace(\n go.Scatter(x=np.arange(1, EPOCHS+1), mode='lines+markers', y=validation, marker=dict(color='red'),\n name=\"Val\"))\n if training != history['loss']:\n fig.update_layout(title_x=0.5,title_text='Accuracy vs Epochs', \n yaxis_title='Accuracy', xaxis_title=\"Epochs\",\n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',)\n else:\n fig.update_layout(title_x=0.5,title_text='Loss vs Epochs', \n yaxis_title='Loss', xaxis_title=\"Epochs\",\n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',)\n fig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_training_curves(history['sparse_categorical_accuracy'],history['val_sparse_categorical_accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"display_training_curves(history['loss'],history['val_loss'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"acc_df = pd.DataFrame(np.transpose([[*np.arange(1, 20).tolist()*3], [\"Train\"]*19 + [\"Val\"]*19 + [\"Benchmark\"]*19,\n history['sparse_categorical_accuracy'] + history['val_sparse_categorical_accuracy'] + [1.0]*19]))\nacc_df.columns = [\"Epochs\", \"Stage\", \"Accuracy\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(acc_df, x=\"Accuracy\", y=\"Stage\", animation_frame=\"Epochs\", title=\"Accuracy vs. Epochs\", color='Stage',\n color_discrete_map={\"Train\":\"dodgerblue\", \"Val\":\"red\", \"Benchmark\":\"seagreen\"}, orientation=\"h\")\n\nfig.update_layout(\n xaxis = dict(\n autorange=False,\n range=[0, 1]\n )\n)\n\nfig.update_layout(title_x=0.5, \n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please click the **PLAY** button to understand how Train accuracy and validation accuracy change over the time"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"acc_df = pd.DataFrame(np.transpose([[*np.arange(1, 20).tolist()*2], [\"Train\"]*19 + [\"Val\"]*19,\n history['loss'] + history['val_loss'] ]))\nacc_df.columns = [\"Epochs\", \"Stage\", \"Loss\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(acc_df, x=\"Loss\", y=\"Stage\", animation_frame=\"Epochs\", title=\"Loss vs. Epochs\", color='Stage',\n color_discrete_map={\"Train\":\"dodgerblue\", \"Val\":\"red\"}, orientation=\"h\")\n\nfig.update_layout(\n xaxis = dict(\n autorange=False,\n range=[0, 1]\n )\n)\n\nfig.update_layout(title_x=0.5, \n paper_bgcolor='rgb(252, 252, 255)',\n plot_bgcolor='rgb(248, 248, 255)',)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the above animation that after certain epochs there is no decrease in the validation loss\nAlso, for the first few epochs there is no decrease in the loss for both training and validation"},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n1. [Awesome notebook by Chris Deotte](https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96)\n2. [Work of DimitreOliveira](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline)\n3. [Excellent visualizations by Tarun](https://www.kaggle.com/tarunpaparaju/plant-pathology-2020-eda-models)\n4. [Notebook Aesthetics by Ouassim](https://www.kaggle.com/ishivinal/machine-learning-model-evaluation-metrics)\n5. [Plotly](https://plotly.com/)\n\n\n\nBack to the table of contents"},{"metadata":{},"cell_type":"markdown","source":"## In the upcoming versions of this notebook I will be adding different models and their performance..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}