{"cells":[{"metadata":{},"cell_type":"markdown","source":"# SIIM-ISIC Competition Submission\n## Authors - Sarthak Khandelwal, Sergei Issaev","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n* [Introduction](#1)\n* [Import Required Libraries](#2)\n* [Configration](#3)\n* [Load the Data](#4)\n* [Utility Functions](#5)\n * [Decoding the Data](#5)\n * [Convert TfRecord to tf.Example](#6)\n * [Preparing Dataset](#7)\n * [Data Augmentation](#8)\n * [Define Dataset](#9) \n * [Get Images Count](#10)\n * [Build Model](#11)\n * [Define Learning Rate](#11)\n * [Build Base Model](#12)\n* [Load Dataset](#13)\n* [Class Imbalance](#14)\n* [Train Model](#15)\n* [Visualize Scores](#16)\n* [Make Submission](#17)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1) Introduction \n\nBasically this problem is about image classification where we have to find the probability of a patient having `Melanoma` given images of his `skin lesions`. This is accomplished by classifying the sample as malignant. One thing to note here is that we don't have to predict the classes but the probabilty of positive class (malignant) and hence we won't have to apply any thresholds here. Just return score calculated by model.
\n\nWhat you will learn in this notebook:
\n- EfficientNet - A family of image classification models which acheive state-of-the-art accuracy.\n- How to Work with TPU.\n- Transfer Learning\n- Working with tensorflow input pipelines.\n- How to handle data in `tfrecord` format.\n- How to deal with Class Imbalance.\n\n**Please let me know in the Comments about what you learnt from the notebook that is not in the above list :)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Installing EfficientNet module \n!pip install efficientnet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import Required Libraries \nTo enable training with TPU's we need to configure them first.\n\nHere's what each line of its configration works:\n- `TPUClusterResolver` : Locates the cluster on the network\n- `experimental_connect_to_cluster`: Will make devices on the cluster available to use\n- `initialize_tpu_system`: Initialize the TPU devices.\n- `TPUStrategy`: Contains necessary distributed training code that will work on TPUs with their 8 compute cores ","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Computation Specific\nimport re\nimport os\nimport numpy as np\nimport pandas as pd\nfrom functools import partial\n\n# Machine Learning Specific\nimport tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import auc\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.callbacks import CSVLogger\n\n\n# Augmentation and Visualization Specific\nimport imgaug\nimport matplotlib.pyplot as plt\n\n# Pretrained Model\nimport efficientnet.tfkeras as efn\n\n\nTPU_used = True\nif TPU_used:\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Device:', tpu.master())\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n except:\n strategy = tf.distribute.get_strategy()\n print('Number of replicas:', strategy.num_replicas_in_sync)\n \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2) Configration \nLets define some configrations used throughout the notebook.\n- `AUTOTUNE` : Tune values dynamically at runtime.\n- `GCS_PATH` : Get the Corresponding Google Cloud Storage path where this dataset is hosted.\n- `BATCH_SIZE`: Set batch size for the model.\n- `IMAGE_SIZE` : Set the shape of the images which are used throughout the notebook.
\n\n**Because TPUs are very fast, many models ported to TPU end up with a data bottleneck. And hence its a good practice to read data from GCS (Google Cloud Storage). TPUs read training data exclusively from GCS.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path()\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3) Load the data \nNow we will load in our data. For this notebook, we will be importing the TFRecord Files.\n- `glob` would return a list of filenames matching a certain pattern provided to it.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec'),\n test_size=0.2, random_state=42\n)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')\nprint('Train TFRecord Files:', len(TRAINING_FILENAMES))\nprint('Validation TFRecord Files:', len(VALID_FILENAMES))\nprint('Test TFRecord Files:', len(TEST_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4) Utility Functions ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Decoding the data\n\n- Tensorflow model needs data to be Tensors or numpy arrays for training.
\n\n- Here we are using `tf.image.decode_jpeg` to decode the image into JPEG format and as this would be an RGB image, we kept number of channels to be 3.
\n\n- We would also normalize the images such that pixel values would now range from `[0,1]` instead of `[0,255]`.\n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we load in our data, we need both our ```X``` and our ```Y```. The X is our image; the model will find features and patterns in our image dataset. We want to predict Y, the probability that the lesion in the image is malignant. When we input our training dataset into the model, it is necessary know what the labels of our images are. For our testing set, we do not need to feed in the labels because the model will predict it for us. In this particular dataset, our label is the ```target``` feature. We will to through our TFRecords and parse out the image and the target values.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Convert TfRecord to tf.Example \n- The Below Function is used to convert the `.tfrecord` file into `tf.Example`. This gives us features and labels of our dataset, feature being the image and label being the class-id (0 or 1) for benign or malignant.
\n- As we know that we would not be needing labels for our test set and hence we would have two formats that data resides in our `.tfrecord` files namely labelled and unlabelled data. Unlabelled is used for testing and its format also would be different and hence we are toggling between different formats based on whether the given image is labelled or not. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(example, labeled):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } if labeled else {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"image_name\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n if labeled:\n label = tf.cast(example['target'], tf.int32)\n return image, label\n idnum = example['image_name']\n return image, idnum","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Prepare Dataset \n- Our dataset is not ordered in any meaningful way, so the order can be ignored when loading our dataset for training. By ignoring the order and reading files as soon as they come in, it will take a shorter time to load the data.\n- Note that the above function `read_tfrecord` is called from `load_dataset` to parse `TfRecord Dataset` to get `tf.Example` object.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n ignore_order = tf.data.Options()\n dataset = tf.data.TFRecordDataset(filenames,num_parallel_reads=AUTOTUNE) \n dataset = dataset.map(partial(read_tfrecord, labeled=labeled),num_parallel_calls=AUTOTUNE)\n if not ordered:\n ignore_order.experimental_deterministic = False \n dataset.with_options(ignore_order)\n return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" ## 4.4 Data augmentation\n\nImage augmentation introduces new images into our data, and generally prevents the model from overfitting and leads to higher accuracy on testing datasets. We will perform following augmentations in the dataset.\n\n- `tf.image.random_flip_left_right`: Randomly flips the image either to left or right.\n- `tf.image.random_saturation`: Adds random saturation to image. Sometimes beneficial if image is deem or fade.\n- `tf.image.random_brightness`: Adds random brightness to the image.\n- `tf.image.random_flip_up_down`: Randomly flips the image either to up or down.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def augmentation_pipeline(image,label):\n image = tf.image.convert_image_dtype(image,tf.float32)\n image =tf.image.random_flip_up_down(image)\n image = tf.image.random_brightness(image,max_delta=0.5) \n return image,label\n\n#image = tf.image.random_brightness(image,max_delta=0.3)\n#image = tf.image.random_flip_left_right(image)\n#image =tf.image.random_flip_up_down(image)\n#image = tf.image.random_saturation(image,3,8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.5 Define Dataset\n\nWe define a class named `Dataset` through which we would be getting different datasets for `training`, `validation` and `testing`.
\nHere we have implemented a function named `get_dataset` within this class which returns dataset for one of the given mode [`train`,`valid`,`test`].\n\n- Note that most of data preprocessing steps are same for all the three modes. We have also seperated the ones which are specific to a particular mode. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Dataset():\n \n def __init__(self,batch_size):\n self.batch_size = batch_size\n \n def get_dataset(self,filenames,labeled,ordered,mode=\"train\"):\n \n assert(mode in [\"train\",\"valid\",\"test\"]) # Checking if the mode is one of the correct ones.\n \n # Prepairing dataset according to the provided mode.\n \n dataset = load_dataset(filenames, labeled=labeled,ordered=ordered)\n if mode==\"train\":\n dataset = dataset.map(augmentation_pipeline)\n dataset = dataset.repeat()\n dataset = dataset.shuffle(2048)\n \n dataset = dataset.batch(self.batch_size)\n dataset = dataset.prefetch(AUTOTUNE)\n \n if mode==\"valid\":\n dataset = dataset.cache()\n \n return dataset\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.6 Get total number of Train, Validation and Test images.\nBelow functions uses a python module named `re` to count the number of images from the given list of Images.
\nFor more details on `re` or `Regular Expressions` refer [this](https://docs.python.org/3/library/re.html).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALID_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint(\n 'Dataset: {} training images, {} validation images, {} unlabeled test images and the steps per epoch is {}'.format(\n NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES, STEPS_PER_EPOCH\n )\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.7 Build the Model\n\n- **The following set of utility functions would help us building our model for the given task.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 4.7.1 Define Learning Rate\n\nThe following function allows for the model to change the learning rate as it runs each epoch. Having a learning rate that is too high will prevent our model from converging. However, having a learning rate that is too small will cause our model to run for far too long. With this function, the model will know how to change its learning rate after each epoch and update the learning rate itself to allow for increased efficiency while still allowing the model to converge.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**We will define two approaches to which changes learning rate dynamically. First we should understand what they are.**\n\n### #1. Time based\nThis method deals with reducing the learning rate over time.
\nFor example consider we reduce learning rate as epochs increases by the rule -
\n \n    **learning_rate = learning_rate / (1 + decay * epoch )**
\n \nLet the initial Learning-rate is `0.1` and decay be `0.001` LR at first five epochs are:
\n\nepoch-1 = 0.1
\nepoch-2 = 0.0999000999
\nepoch-3 = 0.0997006985
\nepoch-4 = 0.09940249103
\nepoch-5 = 0.09900646517
\n\n\n![image.png](attachment:image.png)\nExample taken from [here](https://machinelearningmastery.com/using-learning-rate-schedules-deep-learning-models-python-keras/). \n","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWMAAAEaCAYAAADE7zboAAAABHNCSVQICAgIfAhkiAAAIABJREFUeF7tnQ98VdWV71f+EEgAEwMx2BICto0hUZSAgnSqaEMbUcFpn2grFem0qE8elEdHbXFeP1NT+sB5D8pj3kdwfI7KTKQTrFShUJg4OsUKjUowkhiMIQGBJCQGE5AgSd7ZB4IkNzd339xz79333O/+fPK55ty19177uzY/T/bZe52YLqsIBQIQgAAEwkogNqy90zkEIAABCNgEEGMmAgQgAAEDCCDGBgQBFyAAAQggxswBCEAAAgYQQIwNCAIuQAACEECMmQMQgAAEDCCAGBsQBFyAAAQggBgzByAAAQgYQAAxNiAIuAABCEAAMWYOQAACEDCAAGJsQBBwAQIQgABizByAAAQgYAABxNiAIOACBCAAAcSYOQABCEDAAAKIsQFBwAUIQAACiDFzAAIQgIABBBBjA4KACxCAAAQQY+YABCAAAQMIIMYGBAEXIAABCCDGzAEIQAACBhBAjA0IAi5AAAIQQIyZAxCAAAQMIIAYGxAEXIAABCCAGDMHIAABCBhAADE2IAi4AAEIQAAxZg5AAAIQMIAAYmxAEHABAhCAAGLMHIAABCBgAAHE2IAg4AIEIAABxJg5AAEIQMAAAoixAUHABQhAAAKIMXMAAhCAgAEEEGMDgoALEIAABBBj5gAEIAABAwggxgYEARcgAAEIIMbMAQhAAAIGEECMDQgCLkAAAhBAjJkDEIAABAwggBgbEARcgAAEIIAYMwcgAAEIGEAAMTYgCLgAAQhAADFmDkAAAhAwgABibEAQcAECEIAAYswcgAAEIGAAAcTYgCDgAgQgAAHEmDkAAQhAwAACiLEBQcAFCEAAAogxcwACEICAAQQQYwOCgAsQgAAE4iMFQWVlpRQWFkpTU5NkZGTIwoULZcKECR7uNzY22t9t3LjR4zudC7srG+WVPYel/OAncsnQQTIoLlYuSUqQjLQkyclIkStHJ8u4UcN0msIGAhCAgDaBiBHjoqIimTNnjuTn58vmzZuluLjYQ4x37dol69evl+bmZm0AFxuu2VwhJWVHL1z69OTn9n83fdouNcda5Y336u3f4+JiZFz6cLnp6nS55ZrLZeiQiMF48XD5bwhAwCACEbNMUVVVZQtxUlKSFBQUSHV1tQfGLVu2yBNPPOFxXeeCuiO+WIj7q9PR0SUfHvlUntl+QO5d+YYsffovUnqgSU6ePttfNb6DAAQg4JVAxNzStbS0SGJioj2Q5OTkPu9+ly9f7nWgvr7Y/cFxXyZev68+2iqFRWWSEB8nP/jmFdwteyXFFxCAgDcCESPG8fE9XY2JifE2pgFdr2/5bED1Lq505myHfbf83I4Dcu2oDpk8uiPgNmkAAm4loP7CpXxBIGLEODU1Vdra2mT48OHS2toqKSkpjsYxPSVR3q9tcaTNs50ipUfipLwxQZb8dY5MyU5zpF0agQAE3EsgYtaMs7OzpaSkRNrb2+3PrKwsR6My5cqRjranGjv9eYf8+rfvyd8+U2o9AGxzvH0ahAAE3EMgYsR45syZ9na12bNn2zsp1M6K7jJjxoyAI6LuXtXOiGCUAx9/Ko/+v7fl928dCkbztAkBCLiAQEyXVVwwDseG8I3b58mNdy22lyxirXXpTofxjB+TIkvuzJHLUoY45jMNQQACkU8AMe4Vw8mTJ0tpaemFqw0tp+0ta+9UN8lbFY1S13hS2q3lh0BK0uB4+e/WWvLkLOeXRgLxi7oQgED4CCDGvdj3FuO+QqNO5217+4j8uaJBOjoH/ofFHVMy5G++/bW+uuAaBCAQZQQQ4wGIcXcVdcf86p5DsvnPh+RU+8AOfFxn3R3/xFq24BRflP3LY7gQ6EUAMQ5AjC8W5X/fe1Q2vlEzoFN4KtdF4X15CDL/PCEQxQQiZjeFyTFSd7WzpmbI+kXTZOZ1o/12VW17+8m6PaLWpykQgEB0EkCMHYy7EuUFt2bJqgXXS+ZlQ/1qufHEaVmyfg/7kf2ihjEE3EMAMQ5CLNWyw28enCL33DTOr9bVGvSy599BkP2ihjEE3EEAMQ5iHJUYr37geiuBkD7mU5YgqztklUWOAgEIRA8BfZWIHiaOjnRs+jBZ+1+nypg0/5Yt/uGl9+VgPUeoHQ0GjUHAYAKIcQiCo07brXloil/HrT+3sg09Zh2h1s2xHIJh0AUEIBBEAohxEOH2bnrR7PFy941jJS5WL/2nSjS0/g9V3CH3BsnvEHAhAcQ4xEH93vQr5OE7srV7PX2mQ5Y9x0M9bWAYQiBCCSDGYQicyg63/P5JMniQHn61y+Jxa5cF+5DDECy6hECICOipQYiciaZucsYky9997xpJTIjTGrYS5CesVzvxnj0tXBhBIOIIIMZhDNlVYy+19yPr3iEfsjLGqTtkBDmMQaNrCASJAGIcJLC6zaqdFuoOWbeoo9PqoR4FAhBwFwHE2IB4qjtkdYR6iOaSxevvHbNffEqBAATcQwAxNiSW6gj14/dM0Pbmld2HRP1QIAABdxBAjA2Ko7pD/tmcqyVWcx/yP+/8kB0WBsUPVyAQCAHEOBB6QairXoy6eHaOVssdHV2y+KndJBbSooURBMwmgBgbGJ+brk7XPjr9mXUoZM3v9xs4ClyCAAT8IYAY+0MrhLbq6HT+tZdr9ah2WDzNDgstVhhBwFQCiLGpkbH8WjhrvKisbzply18Os8NCBxQ2EDCUAGJsaGC63fr53RNk1KWJWl6q3RXqLpkCAQhEHgHE2PCYqUMhv/zBRO09yL/+7T52WBgeU9yDQF8EEOO+qBh2TQmyereeTlHJhJQgc2RahxY2EDCHAGJsTiz69URlenv4dr3Um2qpYtvbH/fbHl9CAAJmEUCMzYpHv97MyPuS5Gam9GvT/eW/vvYR79HTIoURBMwggBibEQdtL9SBkNEjk3zad3R2WcsV7/FAzycpDCBgBgHE2Iw4aHuh1o/VC0513zj93E4SCmnDxRACYSSAGIcRfiBd/+CbX9GqvvejT+TF12u0bDGCAATCRwAxDh/7gHq+Y0qG3HPTOK02lBjvrmzUssUIAhAIDwHEODzcHelVifG1V6RqtbXuDx9o2WEEAQiEhwBiHB7ujvU6L/+rWm01t56Rp7YiyFqwMIJAGAggxmGA7mSXKil94X0TtXIgbyv9mPVjJ+HTFgQcJBDvYFt+NVVZWSmFhYXS1NQkGRkZsnDhQpkwoeebLrzZzJgxo0dfsbGxsn37dqmvr5e5c+f2+G7Hjh1++RWJxiop/bcmfknroIdaP9Zda45EFvgMgUglEDYxLioqkjlz5kh+fr5s3rxZiouLPcTYm83FAvvqq6/KyZMnbf579+4VJdSPPPJIpMZjwH6r3RVvf9gkjSdO+2xj5b+Vy8N3ZMvQIWELv08fMYBAtBEI2zJFVVWVLcRJSUlSUFAg1dXVHux92Rw6dEh27txpi7oqZWVlkpeX59FONFxQwqoyvF2SNMjncN+saCDdpk9KGEAgtATCdmvU0tIiiYnnUkMmJydLc3Ozx8h92axbt07mzZsnMTExdt19+/bJsWPHZM2aNTJ27Fj56U9/KmPGjPFo19eFbdu2+TIx9vtJo2LltY98h7Wk7KikdtTJyKQuY8eCY+4moG7CKF8Q8P2vNki04uN7dt0tqBd315/NwYMHpbGxUSZOnHihyoYNG+z/7urqktraWlm5cqWsXbvW7xFE8iRR07vhuXfk/doWn+P+8LN0+euZV7Fc4ZMUBhAIPoGwLVOkpqZKW9u5ROitra2SkuKZAKc/G7U8MX369D4JKWHPzMy0BTkai25C+r3VzXb+CgoEIBB+AmET4+zsbCkpKZH29nb7MyvLM19vfzbqYV1OTs+3KM+fP1/UDozOzk6pqamR3Nzc8BMOgwdq/fjRu67W6rn84CeiliwoEIBAeAmETYxnzpwpGzdulNmzZ9s7Kbofwikc3VvX+rOpq6uz14UvLmoXxapVq+w2169fL0uXLg0v3TD2rvYfT7lypJYH263cxySj10KFEQSCRiDGWl/lCc5FeCdPniylpaVBAx7KhpXAFhaVScWhEz67VXuV1eERCgQgEB4CYbszDs9wo6tXtVzx+Peu0Ro0yxVamDCCQNAIIMZBQ2tGw0qQVYY3nfLSruh84KnDBhsIBJsAYhxswga0r44/T/7aCJ+eHD5+Sh5//l2fdhhAAALOE0CMnWdqXIvdyxVDEuJ8+qaWK0hG7xMTBhBwnABi7DhScxu88wa904ivsdXN3CDimWsJIMauDa3nwNRyhc7bpetbTnMYxBMfVyAQVAKIcVDxmtf4r+blyZc13i6tXtPEcoV58cMj9xJAjN0bW68j+0ZuutfvLv4CMdbChBEEHCGAGDuCMbIaUcsVY9OHaTn91BZe1aQFCiMIBEgAMQ4QYKRWX/3A9TJOQ5C3WUeluUOO1CjjdyQRQIwjKVoO+3rLtZdrtajEmNwVWqgwgsCACSDGA0YX+RXVyTyd3RVqpJs4nRf5AWcERhNAjI0OT/CdU7mPdZYr1FFpliuCHw96iF4CiHH0xt4euTqdp/u2aCXGDdYeZAoEIOA8AcTYeaYR1+KU7DTt5YqX/1wXcePDYQhEAgHEOBKiFAIf1XLFl0ck+exp618O82YQn5QwgID/BBBj/5m5soZarnjotiu1xrZmc4XUHDv3/kKtChhBAAI+CSDGPhFFj4F624fuYRDemxc984KRhoYAYhwazhHTi1quSBma4NNflit8IsIAAn4RQIz9wuV+48tShsiTP5rsc6AdnV3CcoVPTBhAQJsAYqyNKnoM05KHaC9XvPj6R9EDhpFCIIgEEOMgwo3kphfPzpGkwfE+h1B6oEnU20EoEIBAYAQQ48D4ubb2uFHDRCUT8lXUcoV6bx67K3yR4nsI9E8AMe6fT1R/q9aPp1w5UosByxVamDCCgFcCiLFXNHyhCNxz0xVaIHZ/cJyj0lqkMIJA3wQQ4765cPU8AbVcUXjfRImPi/HJZMn6PSxX+KSEAQT6JoAY982FqxcRUIdBbrxqlE8mKufxmt/v92mHAQQg4EkAMfZkwpU+CKjcxzpFPcg7WM9RaR1W2EDgYgKIMfNBi4Barlg0e7zExfpervj7f9nLcoUWVYwg8AUBxJjZoE3glmsul5nXjfZp/0nbGZYrfFLCAAI9CSDGzAi/CChB1ilquYLDIDqksIHAOQKIMTPBLwLdyxU6lVb9bj/LFTqgsIGARQAxZhr4TUDdHX/n65k+6zW1trNc4ZMSBhA4RwAxZiYMiEDBpC9r1VPLFeQ+1kKFUZQTME6MKysrZe7cuXLrrbfKggULZN++fR4h8mZTX18vM2bM6PHjUZkLjhBQR6X/5ttf02rrX177iNN5WqQwimYCxolxUVGRzJkzRzZt2iQ333yzFBcXe8THm83evXttId6xY8eFH4/KXHCMgNp7fO/0K3y21/Rpu5VM6B1Rh0IoEIBA3wSME+OqqirJz8+XpKQkKSgokOrqag/PvdmUlZVJXl6ehz0XgkfgrhvHaqXabGg5zXJF8MJAyy4gYJwYt7S0SGJioo02OTlZmpubPTB7s1FLGlu3bpVZs2bJokWLpK6uzqMuF5wn8L3p47QaffH1GpYrtEhhFI0EfGcPDzGV+PieLsXEeJ748mazYcMG29uuri6pra2VlStXytq1a/0ewbZt2/yuE80VBlmDv350nOw5HNcvBrVM8d/+8U35/rVnZHD/pv22w5fuIKD+8qV8QcA4MU5NTZW2tjYZPny4tLa2SkpKike8fNkoAc/MzLQFeSCFSeI/NfXP6vsr3pBT7f2vC7d3iNSezZRFt433vxNqQMDFBIxbpsjOzpaSkhJpb2+3P7Oysjzwe7OZP3++qJ0WnZ2dUlNTI7m5uR51uRA8Aj8q0NtdsWt/A8sVwQsDLUcogRjrT/ouk3xXD+FWrFhhrxWPGDFCli1bJjk5ObaL3TslvNlUVFTI6tWr5ciRI7YQL126VNLS0vwa3uTJk6W0tNSvOhh/QUCtC6sfX2XokHhZteB6UVvkKBCAgIhxYhzuoCDGgUfgkWdKperjT302pPIkq8T1FAhAgOPQzIEgEPiuxlFp1a1KJKS2vFEgAAHEmDkQBAJTstNENxn94qd2k0woCDGgycgjYNwDvMhDiMd9EVBHpW8Y73u9/rMzHSQT6gsg16KOAGIcdSEP3YBv00hEr7xRyYT+9H5D6ByjJwgYSAAxNjAobnFJPaDTTSb0D5vKZXdlo1uGzjgg4DcBxNhvZFTwh4BaO555nV66zTW/ryCZkD9wsXUVAcTYVeE0czB33pCplUxIHZf+1//4yMxB4BUEgkwAMQ4yYJoX+2CH7um8LXsOyzPbD4ANAlFHADGOupCHZ8DqVU0P356t1fkruw/xMlMtUhi5iQBi7KZoGj6WaTmXSVqy3vHnF984aPhocA8CzhJAjJ3lSWv9EFD5KBbPHi+DB/nOn6lO5z3+/Lv9tMZXEHAXAcTYXfE0fjRqu9uKH07S8lMJMi8z1UKFkQsIIMYuCGKkDWFs+jDJzfTMU93XONTDPPJX9EWGa24jgBi7LaIRMp7Fs3O0t7stWb8HQY6QuOLmwAkgxgNnR80ACKjtbk8vniZDEnyvH6v9x+v/8EEAvVEVAuYTQIzNj5FrPVQP9O68YYzW+N7+sEnUljcKBNxKADF2a2QjZFz33DROa/1YvY9GrR+TvyJCAoubfhNAjP1GRgWnCfxqXp5cMWq4VrP/+3f7tewwgkCkEUCMIy1iLvX3h9/6qtbI2j/vkJ/+UykJhbRoYRRJBBDjSIqWi331J93mh0c+tRLSV7iYBkOLRgKIcTRG3dAxq3Sbug/01NrxK2/xQM/QUOLWAAggxgOARpXgEbjrG2O181c888cDnNALXihoOcQEEOMQA6e7/gmo7W4/v3uCDLM+dcrTf6jiQIgOKGyMJ4AYGx+i6HNw3Khh8vN7JmgNXL3Q9Bcb3kWQtWhhZDIBxNjk6ESxbzljUkStIeuUo82fya9/u48dFjqwsDGWAGJsbGhwTL3M9NuT9N6fp94w/ewO3hDCrIlcAohx5MYuKjx/6LYrZcxlQ7XGuvPdo7JmM1vetGBhZBwBxNi4kOBQbwJL7syVRI2EQqqeyn/MkeneBPk9EgggxpEQpSj3UT3Q+82DUyQuNkaLxMricgRZixRGJhFAjE2KBr54JaBSbt4/Q+/IdEdnl/VA7z1eauqVJl+YSAAxNjEq+NQnAbW7QneHhWqg8EV2WPQJkotGEkCMjQwLTnkjoHZY3HjVKG9f97h+2tqD/OPfvClqpwUFAqYTQIxNjxD+eRB4YGaW9pHpU+1nZfnGfR5tcAECphFAjE2LCP74JKCOTK9+4HrJGKm35a3xxGl5+B/f4pSeT7IYhJNA2MS4srJS5s6dK7feeqssWLBA9u3zvHvxZrN//3558MEH5fbbb5eHHnpIqqqqbIb19fUyY8aMHj/hhEvfwSOgBPnvvn+N1ktNlRcfN50StcuCAgFTCYRNjIuKimTOnDmyadMmufnmm6W4uNiDkTebJ598Uu6//3556aWX5LbbbhP1uyp79+61hXjHjh0Xfjwa5YJrCKgdFotnj5dBcXrTWOVB/tmzb3Ns2jUzwF0D0ZvFQRizupvNz8+XpKQkKSgokOrqao9evNk8++yzMnXqVElISLDF9/jx43bdsrIyycvL82iHC+4lMCU7TZ780WTtAVYcOiGPP/+Otj2GEAgVgbCJcUtLiyQmJtrjTE5OlubmZo8x69iUl5fLtGnT7LpqqWPr1q0ya9YsWbRokdTV1Xm0yQX3ERibPkzULgvdonZXLH5qD3fIusCwCwkBvaSxQXAlPr5n1zExnqerfNm0trbKyy+/LI899pjt4YYNG+zPLutVwrW1tbJy5UpZu3at395v27bN7zpUCC+BQVb3+V+Jk53VcVqO1DZYgrz2dfkvV32uZY+R8wTUX8SULwiETYxTU1Olra1Nhg8fLkpUU1JSPOLSn426k163bp0sXLhQhg7t+VRdCXtmZqYtyAMpTJKBUDOjzsjXa+RF60enHD8VI9vrUqXwvjxRDwQpEAgngbAtU2RnZ0tJSYm0t7fbn1lZWR4cvNmUlpZKYWGhvaMiPT39Qr358+eL2oHR2dkpNTU1kpub69EmF9xN4J6bxon60S1qyWLBmjdZstAFhl3QCMRYf9J3Ba31fhpWD9tWrFhhrxWPGDFCli1bJjk5OXaN7h0R3mzuvfdeaWho6NG62kFRUVEhq1evliNHjthCvHTpUklLS+vHC8+vJk+eLErsKZFN4Pl/r5aXdun/ZZSWPER+NS9P1A4NCgTCQSBsYhyOwer0iRjrUIoMm2e2H5BXduu/QTppcLwtyCpLHAUCoSYQtmWKUA+U/qKPgNphcecNY7QHro5O/+0zpWR70yaGoZMEEGMnadKWcQRU2s1brrlc26+zHZ3WPuR3yYesTQxDpwggxk6RpB1jCSyyTul99+uZfvmn8iG/YK07UyAQKgKIcahI009YCfzgm1/x6w5ZObvJegC48t/IZxHWwEVR54hxFAU72oeq7pB/+C39k3qK15sVDXbGN3IiR/vsCf74EePgM6YHgwjMmpohSpT9KSrj27Ln3pHSqnM5UPypiy0EdAkgxrqksHMNAfVAb/n9k2TwIP3pr3ZaqNc4qe1yFAgEg4D+bAxG77QJgTARyBmTLP9z/mRJTNDLZdHtptq3/D+s3RYnT58Nk+d061YCiLFbI8u4fBJQhzv+6Sdfl1GXnsse6LPCeYN9Bz+R+av+JPvrWnSrYAcBnwQQY5+IMHAzAZUg6H/9+DrJzfRMVNXfuM983ik//+d3WLboDxLf+UUAMfYLF8ZuJKAEWR2Dvu360X4PTy1bLPy/7LbwGxwVPAggxh5IuBCtBH5ckOVXkvpuToePn5Il6/fIBg6JROvUcWTciLEjGGnELQTumJJhv3k6wY+dFt1jL7YOiTy0lrtkt8yFUI8DMQ41cfoznoB6jdPah6aK+vS3HG0+d5f81NYP2HHhL7wotyeFZq8JQArNKP8X0Wv46q0hum8O6U1uiLVtbsGtWX4fw+7dDr9HBwHEGDGOjpkewCjLra1sK4vL5dNTA3tf3uiRQ2Xpd3LJkxxADKKhKmKMGEfDPA94jOqQxxNFZVJ56MSA27oua6Soh4S8TWTACF1dETFGjF09wZ0e3CtvHZIXSqrlzNnOATetjmOrxPe8BHXACF1ZETFGjF05sYM5qIaW0/KrF8uktuFkQN0oUfY3aVFAHVLZaAKIMWJs9AQ12Tl14OP5ndXyufV2kEDKtPGXycN3ZHOnHAhEF9RFjBFjF0zj8A1B3SWv+t37UhHAWnK39+PHpMgD1u6LgWypCx8BenaKAGKMGDs1l6K6nd2VjfJ/XqmUts8GtuPiYnjpVuKiu/5qrNwwPo275SiaVYgxYhxF0z24Q1U7LtSe5C17DktnV1fAnQ2Ki5VJXxshd984jm1xAdM0vwHEGDE2f5ZGmIdKlNds3i+7P3DuzSCXDh8st0wYJd+xXqzKLowImxCa7iLGiLHmVMHMXwIH69vkNy/vlxrr06kSEyMyJm2o/FVuusy8bjTC7BRYA9pBjBFjA6ahu11Q785bv61K1MM+p8tlKYlys3XHPDU7jaUMp+GGuD3EGDEO8ZSL3u5Kyo7Kb984KMc++SwoEFKGJsjlI5Lku9ZSxviMZO6ag0I5eI0ixohx8GYXLfdJQInyq7sPy0fHWvv83qmLIy4ZLFdlXipXj72UnRlOQQ1iO4gxYhzE6UXT/RFQCYiK/1Qr71mfHZ2B777ory/1nXrwlz062d7HnPfVEX6/aspX+3wfGAHEuBc/UmgGNqGo7T8BtZa8yUpM/x/7jkq79W69UJUY62lgytBBcom1vDHBuntW685Jg+NZew5VAHr1gxgjxmGaenTbm4DaEvdG+TF5reyYVH38ae+vQ/p7qrWVLnW4tQZ9aZJM/GqqJCclSIa1i4OMc8ELA2KMGAdvdtHygAnUHGuTnXuPyI53j4h6E7UpRd1Nq6T5ydYddeqwwTLCEm0l0DnWUW610KK23SHYA4sWYowYD2zmUCtkBNR+5Y1v1Mh7NZ9Im3X3HAklLjZGBluiPcxap1b/PfKSITIoPlZGW7s91Gda8hD7Tjs3MyUShhMSHxFjxDgkE41OnCGg7pjVbgz1o5Y1Ir2ou+hFs8bLVdaadbQXV4lxZWWlFBYWSlNTk2RkZMjChQtlwoQJfsWYB3h+4cI4jASUMP/n+/Xy9oHjAedWDuMw7F0eqxZcH/XLG64S41/84hcyadIkyc/Pl82bN0tFRYX88pe/9GueIcZ+4cLYIAIqc9w71c3ybnVTUE77BXOoU64cKT+7278bp2D6E46248PRabD6rKqqkkcffVSSkpKkoKBAXn311WB1RbsQMI7AFGtrmvrpLv9ZXm/nxXhzf0PQTv05BcHJ/B1O+RTqdlwlxi0tLZKYmGgzTE5Olubm5lDzpD8IGEPgG1eli/q575tfsX3aX3fCEuVTUn6wRcprPzHq7rnts8hf/w408K4S4/j4nsNR23AGUtRSBQUC0UAgPulSGXTJl2RI+niJS0y2/nuUxA1JDvnQrx7LrgpXiXFqaqq0tbXJ8OHDpbW1VVJS/A9waWlpyCciHULANAJqp0aNlTvjA+vwifpsPNEuTa3t0tLWbh/ddiB3fo8h33PTFaYhCLk/rhLj7OxsKSkpsdeL1WdWVlbIgdIhBNxAQO1wUNvNvG05U0e4Pzh8QhKsPcPqoeHnZzuk+mirnLWE+kTbGfvz9JkOnyjU8evFs8dzBNsi5ardFGVlZbJixQp7rXjEiBGybNkyycnJ8TkhMIAABIJHQAl3XUObfXLvQ0uwVf6NLuvWWu0xVlnlOLF3jr2rxDh404mWIQABCASXQGxwm6d1CEAAAhDQIYAY61DCBgIQgECd4H76AAAEmUlEQVSQCSDGQQZM8xCAAAR0CCDGOpSwgQAEIBBkAq7a2hYIKyeSDAXSv5N19+/fL2vWrJHDhw/bCZOWLFlib/Orr6+XuXPn9uhqx44dTnYd9La8jSHS4zdjxowe7GJjY2X79u0RHbPGxkY7WdfGjRsvjM1bnLxdD/qEMqgDxPh8MIqKimTOnDkXkgwVFxf7nfHNlLg++eST8sADD0heXp788Y9/FPX7008/LXv37hX1j/6RRx4xxVW//fA2hkiP38X/U1Q5VU6ePGmz8TZev8GFuMKuXbtk/fr1HikJvMXJ2/UQux3W7limOI9fJRlS2d66kwxVV1eHNTCBdP7ss8/K1KlTJSEhwRbf48eP282pfdhKoCO5eBuDW+J36NAh2blzp31jEMkx27JlizzxxBMeU81bnLxd92jAxRcQ4/PBdWuSofLycpk2bZo9yn379snWrVtl1qxZsmjRIqmrq4u4qe1tDG6J37p162TevHnSnVfF23hND9zy5ctlzJgxHm56i5O36x4NuPgCyxTng+tUkiGT5orKz/Hyyy/LY489Zru1YcMG+1OdfqqtrZWVK1fK2rVrTXLZpy/exuCG+B08eFDUOuvEiRMvcPA2Xp+gDDXwFidv1w0dRlDc4s74PNbuJEPq14EmGQpKhAbYqDoSroRWPUAZOnRoj1bUXVdmZqYtyJFaeo/BDfFTyxPTp0/vMyS9x9unUQRc9BYnb9cjYEiOuYgYn0fZnWSovb094pMMqcxz6vVTDz74oKSnp1+YLPPnzxf11Lqzs1NqamokNzfXsYkUqoa8jcEN8VMP63rnUvE23lDxdrofb3Hydt3p/k1uj9wU56PjpiRD9957rzQ0NPSYd+ppvXoN1erVq+XIkSO2EC9dulTS0r54M4TJE7XbN29jcEP81Fr+Cy+8YL8Ywdd4IyFWykf1APninSLe4uTteqSM0wk/EWMnKNIGBCAAgQAJsEwRIECqQwACEHCCAGLsBEXagAAEIBAgAcQ4QIBUhwAEIOAEAcTYCYq0AQEIQCBAAohxgACpDgEIQMAJAoixExRpAwIQgECABBDjAAFSHQIQgIATBBBjJyjSBgQgAIEACSDGAQKkOgQgAAEnCCDGTlCkDQhAAAIBEkCMAwRIdQhAAAJOEECMnaBIGxCAAAQCJIAYBwiQ6hCAAAScIIAYO0GRNiAAAQgESAAxDhAg1SEAAQg4QQAxdoIibUAAAhAIkABiHCBAqkMAAhBwggBi7ARF2oAABCAQIAHEOECAVIcABCDgBAHE2AmKtAEBCEAgQAKIcYAAqQ4BCEDACQKIsRMUaQMCEIBAgAQQ4wABUh0CEICAEwQQYyco0gYEIACBAAkgxgECpDoEIAABJwggxk5QpA0IQAACARJAjAMESHUIQAACThBAjJ2gSBsQgAAEAiSAGAcIkOoQgAAEnCCAGDtBkTYgAAEIBEgAMQ4QINUhAAEIOEEAMXaCIm1AAAIQCJAAYhwgQKpDAAIQcIIAYuwERdqAAAQgECABxDhAgFSHAAQg4AQBxNgJirQBAQhAIEACiHGAAKkOAQhAwAkC/x8XvjeE26JjYgAAAABJRU5ErkJggg=="}},"execution_count":null},{"metadata":{},"cell_type":"markdown","source":"By looking above graph we can say that Learning Rate changes continuously in Time Based Approach of Schedulling Learning Rate.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Note that learning rate first increase till 10 epochs and then reduces as the epoch increases.\ndef build_lrfn(lr_start=0.00001, lr_max=0.000075, lr_min=0.0000001, lr_rampup_epochs=5, lr_sustain_epochs=0, lr_exp_decay=.8):\n lr_max = lr_max * strategy.num_replicas_in_sync\n def lrfn(epoch):\n if epoch < lr_rampup_epochs:\n lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n lr = lr_max\n else:\n lr = (lr_max - lr_min) * lr_exp_decay ** (epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n return lr\n return lrfn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## #2. Drop Based\n\nIn this approach we change learning rate conditionally. This function introduces discontinuous graph.
\n\nFor example if we decrease the learning rate in interval of 10 epochs and considering 100 epochs in total we would get the following graph. Also the learning rate is `0.1` intially.\n\n![image.png](attachment:image.png)\nExample taken from [here](https://machinelearningmastery.com/using-learning-rate-schedules-deep-learning-models-python-keras/). ","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWMAAAEbCAYAAAAPs+VNAAAABHNCSVQICAgIfAhkiAAAHUxJREFUeF7t3X9sFOeZwPFn7bXxDxwbE4dyrTHJKcYxiatQ7ojQ3RV69tVKr/BXfa1Aoih3av6wSBEVSo+TKiVWKshJpYh/IKoiXZEolalwmiCoqaWcLpXIWXe2S7DjnM8YOAdjbNzYBBts780zsMTLsl7bM559Z+c7FbU9877vvO/nmTxez7wzE4pYi7AggAACCKRUICOle2fnCCCAAAK2AMmYAwEBBBAwQIBkbEAQ6AICCCBAMuYYQAABBAwQIBkbEAS6gAACCJCMOQYQQAABAwRIxgYEgS4ggAACJGOOAQQQQMAAAZKxAUGgCwgggADJmGMAAQQQMECAZGxAEOgCAgggQDLmGEAAAQQMECAZGxAEuoAAAgiQjDkGEEAAAQMESMYGBIEuIIAAAiRjjgEEEEDAAAGSsQFBoAsIIIAAyZhjAAEEEDBAgGRsQBDoAgIIIEAy5hhAAAEEDBAgGRsQBLqAAAIIkIw5BhBAAAEDBEjGBgSBLiCAAAIkY44BBBBAwAABkrEBQaALCCCAAMmYYwABBBAwQIBkbEAQ6AICCCBAMuYYQAABBAwQIBkbEAS6gAACCJCMOQYQQAABAwRIxgYEgS4ggAACJGOOAQQQQMAAAZKxAUGgCwgggADJmGMAAQQQMECAZGxAEOgCAgggQDLmGEAAAQQMECAZGxAEuoAAAgiQjDkGEEAAAQMESMYGBIEuIIAAAiRjjgEEEEDAAAGSsQFBoAsIIIAAyZhjAAEEEDBAgGRsQBDoAgIIIEAy5hhAAAEEDBAgGRsQBLqAAAIIkIw5BhBAAAEDBEjGBgSBLiCAAAIkY44BBBBAwAABkrEBQaALCCCAQNgvBF1dXdLQ0CBDQ0NSWloq9fX1UlVVFdf9wcFBe9uJEyfits1lxfmuQfnth1flwqWbkpOdKZOTEcnMDNlVp6at7zNCMnF3yt42fmdK8nPCcmt88sHPS7KsOla5sFVOFy3/1MoC+e7frJZnVy+bSxcogwACARQIRazFD+P+yU9+Il/72tekurpampqapLOzU1577bWYrn/wwQdy9OhR6e/vl+bm5nkP61BTp7S0fzrvenOtsGvrM/KNr66ca3HKIYBAgAR8c5qiu7vbTsR5eXlSW1srPT09cWF677335PXXX49bP5cV+ol4MROx9uEXZz+R6yPjc+kOZRBAIGACvknGIyMjkpuba4ensLBQhoeH40L1xhtvyKpVq+LWz2XF+Y9vzKWYozJ6OuP8x4OO2qAyAgikp4BvzhmHw7FdDYXunZN1KywDI7fdamrWdtovdEnWzY9mLcNGBIIgoH/hsnwh4JtkXFxcLGNjY1JQUCCjo6NSVFTkahxXFOXKR30jrrb5qMa++myF1G4ofdQm1iGAQIAFfHOaoqKiQlpaWmRiYsL+Wl5e7mrYNqx53NX2HtVY3pKwbFhT8qhNrEMAgYAL+CYZv/jii/Z0ta1bt0pjY6PU1dU9CF1NTY3jMG6oKFn0mQ7/WPu0PFGU47ivNIAAAukn4JupbV7R//Xf75C/+c4r9ikLe57xVER0yrCeotbvw9acY51HnJWZkWCecYa9PdOqoHUyrP/Tecbf+/qTzDP2KojsBwEfCpCMHwra+vXrpbW11YehpMsIIOBnAd+cpvAzMn1HAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAgGTsATK7QAABBJIJkIyTCbEdAQQQ8ECAZOwBMrtAAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAgGTsATK7QAABBJIJkIyTCbEdAQQQ8ECAZOwBMrtAAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAgGTsATK7QAABBJIJkIyTCbEdAQQQ8ECAZOwBMrtAAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAgGTsATK7QAABBJIJkIyTCbEdAQQQ8ECAZOwBMrtAAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAgGTsATK7QAABBJIJkIyTCbEdAQQQ8ECAZOwBMrtAAAEEkgmQjJMJsR0BBBDwQIBk7AEyu0AAAQSSCZCMkwmxHQEEEPBAIOzBPh65i66uLmloaJChoSEpLS2V+vp6qaqqiimbqExNTU1MuYyMDDl79qwMDAzI9u3bY7Y1Nzc/cv+sRAABBEwSSFkyPn78uNTV1Ul1dbU0NTVJY2NjXDJOVGZmgn333Xfl1q1btmlbW5toot67d69JxvQFAQQQSCqQstMU3d3ddiLOy8uT2tpa6enpietssjJXrlyRc+fO2Uldl/b2dlm3bl1cO6xAAAEETBdI2SfjkZERyc3NtX0KCwtleHg4zipZmSNHjsiOHTskFArZdTs6OuTatWty6NAhWb16tfzoRz+SVatWxbWbbMWZM2eSFWE7Agg4FNAPYSxfCKQsGYfDsbuOJtSZwZmtzKVLl2RwcFCef/75B1WOHTtmfx+JRKSvr08OHDgghw8fnne8OUjmTUYFBBBwKJCy0xTFxcUyNjZmd390dFSKiorihjJbGT09sWnTprg6ukITe1lZmZ2QWRBAAAE/CKQsGVdUVEhLS4tMTEzYX8vLy+O8ZiujF+sqKytj6uzcuVN0Bsb09LT09vbK2rVr49pkBQIIIGCiQMqS8YsvvignTpyQrVu32jMpohfhFCk6dW22MpcvX7bPC89cdBbFz372M7vNo0ePyp49e0w0p08IIIBAnEDIOr8aiVsb4BXr16+X1tbWAAswdAQQSIVAyj4Zp2Kw7BMBBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESoBkHKhwM1gEEDBVgGRsamToFwIIBEqAZByocDNYBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESoBkHKhwM1gEEDBVgGRsamToFwIIBEqAZByocDNYBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESoBkHKhwM1gEEDBVgGRsamToFwIIBEqAZByocDNYBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESoBkHKhwM1gEEDBVgGRsamToFwIIBEqAZByocDNYBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESoBkHKhwM1gEEDBVgGRsamToFwIIBEqAZByocDNYBBAwVYBkbGpk6BcCCARKgGQcqHAzWAQQMFWAZGxqZOgXAggESiBs2mi7urqkoaFBhoaGpLS0VOrr66Wqqiqmm4nKDAwMyPbt22PKNjc3p2yIt8Yn5Vfv98qFvpvSe21McrIzZfzOVMzXycmIZGaGZHJ6WkLW/zIzQjJlfR/OzLDL5ueERduJ1s0OZ8h0RGTZ0mx5amWBvPR3T8sTRTkpGyM7RgABdwSMS8bHjx+Xuro6qa6ulqamJmlsbIxLxonKtLW1SU1Njezdu9cdHQetaALdffRDuT4y/qAVTa66PPx1cjpaJCKT94pYX+99o+3MrHPnfuHBP42L/rtw6ab8uO45eXb1Mge9pSoCCKRawLjTFN3d3XYizsvLk9raWunp6YkzSlSmvb1d1q1bF1c+FSsONV2MScSL1QdN1ofe6Vys5mkXAQQ8EjAuGY+MjEhubq49/MLCQhkeHo6jSFSmo6NDTp8+LVu2bJFdu3bJ5cuX4+p6teL8xze82pWd9PUTMgsCCPhXwLjTFOFwbJdCoVCcbqIyx44ds8tGIhHp6+uTAwcOyOHDh+PqJ1tx5syZZEVm3d7/mf6O85b29//xn3K168H5jln7x0YETBDQv3xZvhDwNmPMQb64uFjGxsakoKBARkdHpaioKK5WsjKawMvKyuyEvJDFjYPknddaFrLrBdf527/6C84bL1iPigikXsC40xQVFRXS0tIiExMT9tfy8vI4pURldu7cKTrTYtqajdDb2ytr166Nq+vVig1rHvdqV1JSmEMi9kybHSGwOAIh6096a6KUOYtehNu/f799rnj58uWyb98+qaystDuoMyV0qlqiMp2dnXLw4EHp7++3E/GePXukpKRkXoNbv369tLa2zqvOowrrhbUfHvnQnvGwmEvekrD88z8wm2IxjWkbAS8EjEvGXgx6tn24lYx1H9F5xn+0Lq5dGkg8zzjD+vtkypo8rKfHdZ7xpPV9VoJ5xlnWPGP9/VmUv0T+fOVSeemb5cwzni2gbEPAJwLGnTP2iducuqk3bLz0zafnVJZCCCAQbAHjzhkHOxyMHgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqQDIOauQZNwIIGCVAMjYqHHQGAQSCKkAyDmrkGTcCCBglQDI2Khx0BgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqQDIOauQZNwIIGCVAMjYqHHQGAQSCKkAyDmrkGTcCCBglQDI2Khx0BgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqQDIOauQZNwIIGCVAMjYqHHQGAQSCKkAyDmrkGTcCCBglQDI2Khx0BgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqQDIOauQZNwIIGCVAMjYqHHQGAQSCKkAyDmrkGTcCCBglQDI2Khx0BgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqQDIOauQZNwIIGCVAMjYqHHQGAQSCKkAyDmrkGTcCCBglQDI2Khx0BgEEgipAMg5q5Bk3AggYJUAyNiocdAYBBIIqEA7qwP0+7t5rY9I7MCoDN2/L5HREIhGRvCVh+XxiUooLlsjN0QnJyc6UyamIZFi/cqendcQRybR+mLJ+yApnxpUdvzMlX348T54tWyZPFOX4nYj+I+ArgZQl466uLmloaJChoSEpLS2V+vp6qaqqisFLVObixYty6NAhuXr1ql139+7dUl5eLgMDA7J9+/aYNpqbm30VkLl09hdnP5Hfnr8yl6ILLvPSN5+Wb28oXXB9KiKAwPwEUnaa4vjx41JXVycnT56UzZs3S2NjY1zPE5V588035fvf/7785je/kW9961uiP+vS1tYmNTU1ogk4+i+uUZ+v8CIRK5Hu53zXoM+16D4C/hFIWTLu7u6W6upqycvLk9raWunp6YlTS1Tm7bfflhdeeEGys7Pt5Hvjxg27bnt7u6xbty6unXRZcX1kfNE/Ec+0OvROZ7rQMQ4EjBdIWTIeGRmR3NxcG6iwsFCGh4fjsOZS5sKFC7Jx40a7bkdHh5w+fVq2bNkiu3btksuXL8e16ecVvddGPe3+rfFJ0V8ALAggsPgCKTtnHA7H7joUCsWNNlmZ0dFROXXqlLz66qt23WPHjtlfI9bVrL6+Pjlw4IAcPnw4rt1kK86cOZOsSEq2997U353ehuz999+XgiXW1UEWBFwW0L+IWb4Q8Pa/7BnyxcXFMjY2JgUFBaJJtaioKC4us5XRT9JHjhyxL/zl5+fH1NXEXlZWZifkhSymHiT6KfXsJ39YyJAWVEdnZ3xn6zcWVJdKCCAwP4GUnaaoqKiQlpYWmZiYsL/qbIiHl0RlWltb7ZkYL7/8sqxYseJBtZ07d4rOwJi2pm719vbK2rVrH27S1z/rdDMvZzi8svUZX3vReQT8JBCy/qRPyd+gerFt//799rni5cuXy759+6SystK2i86ISFRm27Ztcv369RhnnT3R2dkpBw8elP7+fjsR79mzR0pKSuYVj/Xr14sme5OXxZ5RoZ+Iv7fpSU8Tv8ne9A0BLwRSloy9GNxC9uGHZKzjmnnTRzgzQ25bN2zkWjd56Ndl+dly89Ydyc3KlLvWDSHWZutGD5GQddNHhnXTh/7lEK0zs+ztu1Oy5suPyZNfKuCmj4UcPNRBwIFAys4ZO+gzVS2BJ7+01P7HggAC6SGQsnPG6cHHKBBAAAF3BEjG7jjSCgIIIOBIgGTsiI/KCCCAgDsCJGN3HGkFAQQQcCRAMnbER2UEEEDAHQGSsTuOtIIAAgg4EiAZO+KjMgIIIOCOAMnYHUdaQQABBBwJkIwd8VEZAQQQcEeAZOyOI60ggAACjgRIxo74qIwAAgi4I8CzKdxxTItW9M0e+qLT3oExGf5sQp8sZD1gSB80FJI7d6ett01nyIT1dWlulnxulc0KW2+atp5AFLK2T1vl9E3U2VkZEtE6mSEZv3Ovjj6k6JnSQtmw5nF5dvWytLBiEAi4LUAydlvUp+3pU+D+5d/+SzQhL8byP/2f2Ylen8esb55mQQCBWAFOU3BE2An4p7/uWLREPJNYEzJvneagQyBegGQcbxK4Nec/HvT0xaPvWAmZBQEEYgVIxhwRniZi5f6obwR1BBB4SIBkzCHhuUBJYY7n+2SHCJguQDI2PUIe9G/Dmvm9J9Bpl55jRoVTQuqnoQDJOA2DOt8h6eubvHrrtL7slNkU840Q5YMgwNS2IER5DmPUBJmfE5Zfvd87h9ILK7K2rEhe2Vpp74cFAQRiBXg79ENHhF/eDr2YB7LOOY4mTJ32pt9fH7ltvzW699qofePGhUs37Z+vj4zb26Pzk6PfP1GUY5eN1tGvJOHFjBpt+12AZEwy9vsxTP8RSAsB/l5MizCaPQj9FN3Sfk2uDn0ug9YnbL1lWm+d1tusQ6GQ9f20ZGXqrdZT1i3XYRm3vi6xykzc0a+ZctfaHrZur45EQtbt2fp9hnV79pQ8UZQrRUuz5dt/+RVuszb7EKB3cxAgGc8BiSILF9A77n5x9pN5NHBnzmVvjt0rq3f06Tlvry5CzrmDFERgHgLMppgHFkXnJ6DnnueXiOfX/szSuh/9BM6CgF8FSMZ+jZwP+t3S/qmnvTz/8Q1P98fOEHBTgGTspiZtxQj8rzWbwsvF6/15OTb2lf4CnDNO/xinbIQrrAtsXj6HYty64Hfuvz+1LwhmWBcHo0uGdZHw7tSUdZEwU27fmdTHNEvE+qdf9WKg/rs1fldyrRtSdNGLhU+uKLCm5S190AbfILDYAiTjxRYOcPv6MHkvT1X0fDoqh3/b6Zr4hooS2bXlGeZHuyZKQ7MJcJpiNh22ORLQZKYJ2a+LztL46a//6Nfu02+fCXDTx0MB4w48949gvcX6922fyuCfxufUePQ0wpwKe1DoS8ty5XHrSXP6ail9DVUkErFfOTVtfRVr7nOG9ZHGnisdzrTK6CmSDBm9fVcKrNdT6XxoLZtpnSrRNrx8DogHNOzCRQGSMcnYxcPJm6b+6ed/mHNi96ZH89vLyuJc2bb5KVm2dIn1nkC90SUz4Vdt+fbEpH0+e3h0QooL4utoGX3uB4u/BThn7O/4BbL3i/WePq8wPx2+Lf968iNXd6cXIUtL8qQwP1vy7t/FGN1B2LqY+bmV9K0P5/adjtZHe+sCZsi+UHl30npbrLXeuhfS3q4/68XNx6xP9csfWyJrvlJo/wLQvwZ00a9612T0F4TW0U16kVT/UtBtWib6S2ZmnWjZ6POs9fklLF8IpFUy7urqkoaGBhkaGpLS0lKpr6+Xqqoq4p1mAs+tLhLmFMcGVU+T6E02bi96B+ViLt/46kr77kkeImX9QltMaK/bPn78uNTV1cnJkydl8+bN0tjY6HUX2J8HAt/9+lMe7IVdeCGgs212H/3Qk5fhejEeJ/tIq2Tc3d0t1dXVkpeXJ7W1tdLT0+PEhrqGCuhFsB/XPSf6oHoW/wvoY1gX8znafhFKq6N5ZGREcnNzbfvCwkIZHh72Sxzo5zwFdNrcQesZyRf6bsrVG5/bT3zT85Y620GfBpdpnQ+9PTFlt6qnSTOtGQ56TlOfAhctqz9H62i9f//jNeumkHt15tkdijsU0DgGfUmrZBwOxw5HLyYsZNHpbSzBE8gq/DNZ8fUfBm/gBox4Mc53GzCseXUhrZJxcXGxjI2NSUFBgYyOjkpR0fyn+7S2ts4LkMLpJaBPfvt5U6evp875MSJMzbNuw/dj4BL1uaKiQlpaWuzzxfq1vLw8UVHWI/BIAX2l1FuvbLRfJzXzVVMrrBs/Bm7eth9or+vzc7Ie1I++liq6LVo2+sqppdY0sZ+fuii9A+7PdnjkIHy4csuGUh/22t0up9VNH+3t7bJ//377XPHy5ctl3759UllZ6a4YrSGwQAG9vVoTss4zHrceWKTJ++qNW/Y8Xp07rW83Gb8z/eC8dn5OprWnkP0QI/sNKFYd/arnvHWurp4r123d//fZAntkRjWd3rZr6zNmdCaFvUirZJxCR3aNQEoFHn6wvn5y10Stn8rHrFuz9QKmvrJKb9XWW7aj27XT0e/1SXd6y7fWuWHdut7dPyrXhvXiqNYTa5v+u3cruLalbepT8B7Ly5ahz8btm0Q++1x/Yeht4dbFUfs2cb1Ieu8Cqta5Mxmx+6M3nKwszrOfXaIXY1msX7vW3TL3bq1BAwEEEEAgZQJpNc84ZYrsGAEEEHAoQDJ2CEh1BBBAwA0BkrEbirSBAAIIOBRIq6ltTizS6SFDFy9elEOHDsnVq1ftBybt3r3bnuY3MDAg27dvj2Fqbm52wuZ53URj8Hv8ampqYiz1QtvZs2d9HbPBwUH7YV0nTpx4MLZEcUq03vMDLIU7JBnfx48+ZEifbdHU1GQ/ZMivT3x788035Qc/+IGsW7dOfve734n+/NZbb0lbW5vof/R79+5N4SHnbNeJxuD3+M38pfjuu+/KrVu3bKhE43WmuPi1P/jgAzl69GjcIwkSxSnR+sXvqTl74DTF/Vik00OG3n77bXnhhRckOzvbTr43btx7hb3Ow9YE7ecl0RjSJX5XrlyRc+fO2U8f1CXReE2P4XvvvSevv/56XDcTxSnR+rgG0ngFyfh+cNP1IUMXLlyQjRs32qPs6OiQ06dPy5YtW2TXrl1y+fJl3x3aicaQLvE7cuSI7Nixw35Iu59j9sYbb8iqVavijq9EcUq0Pq6BNF7BaYr7wXXrIUMmHSv6fI5Tp07Jq6++anfr2LFj9ledWt7X1ycHDhyQw4cPm9TlpH1JNIZ0iN+lS5dEz7M+//zzDxwSjTcplKEFEsUp0XpDh7Eo3eKT8X3W6EOG9MeFPmRoUSK0wEb1lnBNtHoBJT8/P6YV/dRVVlZmJ2S/Lg+PIR3ip6cnNm3a9MiQPDzeRxbywcpEcUq03gdDcq2LJOP7lNGHDE1MTPj+IUP65Dl9/dTLL78sK1aseHCw7Ny5U/Sqtd4O29vbK2vXrnXtQPKqoURjSIf46cW6h5+lkmi8Xnm7vZ9EcUq03u39m9wet0Pfj046PWRo27Ztcv369ZjjTq/Wd3Z2ysGDB6W/v99OxHv27JGSEn89FyDRGNIhfnou/5e//KX9YoTokmi8JieVmX3TC8gzZ4okilOi9X4Zpxv9JBm7oUgbCCCAgEMBTlM4BKQ6Aggg4IYAydgNRdpAAAEEHAqQjB0CUh0BBBBwQ4Bk7IYibSCAAAIOBUjGDgGpjgACCLghQDJ2Q5E2EEAAAYcCJGOHgFRHAAEE3BAgGbuhSBsIIICAQwGSsUNAqiOAAAJuCJCM3VCkDQQQQMChAMnYISDVEUAAATcESMZuKNIGAggg4FCAZOwQkOoIIICAGwIkYzcUaQMBBBBwKEAydghIdQQQQMANAZKxG4q0gQACCDgUIBk7BKQ6Aggg4IYAydgNRdpAAAEEHAqQjB0CUh0BBBBwQ4Bk7IYibSCAAAIOBUjGDgGpjgACCLghQDJ2Q5E2EEAAAYcCJGOHgFRHAAEE3BAgGbuhSBsIIICAQwGSsUNAqiOAAAJuCJCM3VCkDQQQQMChAMnYISDVEUAAATcESMZuKNIGAggg4FCAZOwQkOoIIICAGwIkYzcUaQMBBBBwKEAydghIdQQQQMANAZKxG4q0gQACCDgUIBk7BKQ6Aggg4IYAydgNRdpAAAEEHAqQjB0CUh0BBBBwQ4Bk7IYibSCAAAIOBUjGDgGpjgACCLgh8P9wYckNs7xFCwAAAABJRU5ErkJggg=="}},"execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def Callbacks():\n cb = []\n \n # Drop based LR Scheduling.\n reducelr = ReduceLROnPlateau(monitor=\"val_loss\", mode=\"min\", patience=5, min_lr = 1e-06, factor=0.2)\n log = CSVLogger(\"Melanoma_classification.csv\")\n cb.append(reducelr)\n cb.append(log)\n return cb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.7.2 Build base model\n\nHere we will use `EfficientNetB0` pretrained model with custom architecture on the top.
\nWhat we have just done is called `Transfer Learning`. For more info on transfer learning we recommend [this](https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a) article.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(output_bias = None, metrics = None):\n \n if output_bias is not None:\n output_bias = tf.keras.initializers.Constant(output_bias)\n \n base_model = efn.EfficientNetB7(input_shape=(*IMAGE_SIZE, 3),\n include_top=False,\n weights=\"imagenet\")\n \n base_model.trainable = False\n \n model = tf.keras.Sequential([\n base_model,\n tf.keras.layers.Conv2D(64,kernel_size=(3,3),activation=\"relu\"),\n tf.keras.layers.GlobalAveragePooling2D(),\n tf.keras.layers.Dense(32,activation=\"relu\"),\n tf.keras.layers.Dropout(0.2),\n tf.keras.layers.Dense(16,activation=\"relu\"),\n tf.keras.layers.Dense(8,activation=\"relu\"),\n tf.keras.layers.Dense(1, activation='sigmoid',\n bias_initializer=output_bias\n )\n ])\n \n lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(\n 0.001,\n decay_steps=(NUM_TRAINING_IMAGES // BATCH_SIZE)*100,\n decay_rate=1,\n staircase=False\n )\n\n model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lr_schedule),\n loss='binary_crossentropy',\n metrics=[metrics])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5) Load Dataset\n- **Now we will load the training as well as validation dataset.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = Dataset(BATCH_SIZE)\ntrain_dataset = dataset.get_dataset(TRAINING_FILENAMES,labeled = True,ordered = False,mode=\"train\")\nvalid_dataset = dataset.get_dataset(VALID_FILENAMES,labeled = True,ordered = False,mode=\"valid\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6) Checking for Class Imbalance and correcting it\n\nLet's have a look at our dataset to see if it has class Imbalance or not.\n- What is Class Imbalance though? \n
Basically it is when the ratio between number of samples between classes is either too large or too   small.
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest_csv = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n\ntotal_img = train_csv['target'].size\n\nmalignant = np.count_nonzero(train_csv['target'])\nbenign = total_img - malignant\n\nprint('Examples:\\n Total: {}\\n Positive: {} ({:.2f}% of total)\\n'.format(\n total_img, malignant, 100 * malignant / total_img))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **As you can see that 1.76% of malignant samples out of 100% data we have. This clearly is a sign of class imbalanace.**
\n\n- **We would correct it by setting initial bias and class weights for the model.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 6.1 Set initial bias\n\nWe want to set the correct initial bias for our model so that it will not waste time figuring out that there are not many malignant images in our dataset. We want our output layer to reflect the inbalance that we have in our data.\n\nThe correct bias to set can be derived from:\n
p0 = pos/(pos+neg) = 1/(1+ e-b0)\nb0 = -log(1/p0 -1)\nb0 = log(pos/neg)
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"initial_bias = np.log([malignant/benign])\ninitial_bias","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 Set class weights\n\nSince there are not enough malignant images, we want these malignant images to have more weight in our model. By increasing the weight of these malignant images, the model will pay more attention to them, and this will help balance out the difference in quantity.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"weight_for_0 = (1 / benign)*(total_img)/2.0 \nweight_for_1 = (1 / malignant)*(total_img)/2.0\n\nclass_weight = {0: weight_for_0, 1: weight_for_1}\n\nprint('Weight for class 0: {:.2f}'.format(weight_for_0))\nprint('Weight for class 1: {:.2f}'.format(weight_for_1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **Define learning rate and other configrations for model training.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lrfn = build_lrfn()\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7) Start Training\n\nHere we would take `AUC (Area Under the Curve)` as evaluation metric for our model becuase if the model would have `accuracy` as evaluation metric then it would give false confidence on the model.With predicting all examples as `benign (0)` the model could get **98%** accuracy which is false.\nMoreover the competition scores the model by finding the area under the ROC curve, which is also why our metric will be set to `AUC`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model = make_model(metrics=tf.keras.metrics.AUC(name='auc'),output_bias=initial_bias)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"callback = Callbacks()\n\nhistory = model.fit(\n train_dataset, epochs=20,\n steps_per_epoch=STEPS_PER_EPOCH,\n validation_data=valid_dataset,\n validation_steps=VALID_STEPS,\n callbacks= [tf.keras.callbacks.CSVLogger(\"Melanoma_classification.csv\")],\n class_weight=class_weight,\n verbose=1\n)\n# tf.keras.callbacks.LearningRateScheduler(lrfn)\n# tf.keras.callbacks.CSVLogger(\"Melanoma_classification_new.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8) Visualize Scores\n\n- **In the following function we would visualize AUC score and loss of the model for both training and validation mode.** ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## Visualizing Model Performance\ndef visualize(epochs,x,y,label=\"\"):\n plt.plot(epochs,x,marker=\"o\",c=\"red\",label=f\"Training {label}\")\n plt.plot(epochs,y,marker=\"x\",c=\"green\",label=f\"Validation {label}\")\n plt.legend()\n plt.grid(False)\n\nvisualize_df = pd.read_csv(\"/kaggle/working/Melanoma_classification.csv\")\nepochs = visualize_df[\"epoch\"]\nauc = visualize_df[\"auc\"]\nval_auc = visualize_df[\"val_auc\"]\nloss = visualize_df[\"loss\"]\nval_loss = visualize_df[\"val_loss\"]\n\nfig = plt.figure(figsize=(12,4))\nplt.subplot(1,2,1)\nvisualize(epochs,auc,val_auc,label=\"AUC\")\nplt.subplot(1,2,2)\nvisualize(epochs,loss,val_loss,label=\"LOSS\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 9) Making Submission\n\n- **Let's make a submission file for our Test Dataset by predicting the correct classes for samples in Test Dataset.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting test Dataset.\n# Note that here test dataset is ordered. \ntest_ds = dataset.get_dataset(TEST_FILENAMES,labeled=False,ordered=True,mode=\"test\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef submit(filename=\"submission.csv\"):\n print('Generating submission.csv file...')\n print('Computing predictions...')\n test_images_ds = test_ds.map(lambda image, idnum: image)\n probabilities = model.predict(test_images_ds)\n repr(\"Prediction done\")\n test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n pred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\n sub = pd.read_csv(\"/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv\")\n del sub['target']\n sub = sub.merge(pred_df, on='image_name')\n sub.to_csv(open(filename,\"w\"), index=False)\n print(\"Submitted Successfully.....\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submit()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Thanks For Reading :)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}