{"cells":[{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"**_Note: If you like this kernel and/or choose to fork it, please appreciate the hard work by up-voting the kernel with the ^ button above._**\n\nFollow me on Twitter @Urengaraju\n"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"%%HTML\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **SIIM**\n\n\nThe Society for Imaging Informatics in Medicine (SIIM) is the leading healthcare professional organization for those interested in the current and future use of informatics in medical imaging. The society's mission is to advance medical imaging informatics across the enterprise through education, research, and innovation in a multi-disciplinary community"},{"metadata":{},"cell_type":"markdown","source":"\n\nImage Source : siim.org"},{"metadata":{},"cell_type":"markdown","source":"# **ISIC :**\n\nThe International Skin Imaging Collaboration: Melanoma Project is an academia and industry partnership designed to facilitate the application of digital skin imaging to help reduce melanoma mortality. When recognized and treated in its earliest stages, melanoma is readily curable. Digital images of skin lesions can be used to educate professionals and the public in melanoma recognition as well as directly aid in the diagnosis of melanoma through teledermatology, clinical decision support, and automated diagnosis.\n"},{"metadata":{},"cell_type":"markdown","source":"# **Skin Cancer :**\n\nSkin cancer is the most prevalent type of cancer. Melanoma, specifically, is responsible for 75% of skin cancer deaths, despite being the least common skin cancer. The American Cancer Society estimates over 100,000 new melanoma cases will be diagnosed in 2020. It's also expected that almost 7,000 people will die from the disease. As with other cancers, early and accurate detection—potentially aided by data science—can make treatment more effective.\n"},{"metadata":{},"cell_type":"markdown","source":"# **Problem Statement :**\n\nIn this workshop , you’ll identify melanoma in images of skin lesions and you’ll use images within the same patient and determine which are likely to represent a melanoma. Using patient-level contextual information may help the development of image analysis tools, which could better support clinical dermatologists.\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"# **Installing the Libraries**"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-29T17:35:07.768661Z","iopub.status.busy":"2020-09-29T17:35:07.767806Z","iopub.status.idle":"2020-09-29T17:35:29.46515Z","shell.execute_reply":"2020-09-29T17:35:29.464182Z"},"papermill":{"duration":21.729797,"end_time":"2020-09-29T17:35:29.465301","exception":false,"start_time":"2020-09-29T17:35:07.735504","status":"completed"},"tags":[],"id":"bqeZJ9juDVzv","trusted":true},"cell_type":"code","source":"!pip install -q --upgrade pip\n!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-09-29T17:35:29.509576Z","iopub.status.busy":"2020-09-29T17:35:29.508858Z","iopub.status.idle":"2020-09-29T17:35:36.648898Z","shell.execute_reply":"2020-09-29T17:35:36.648021Z"},"papermill":{"duration":7.165158,"end_time":"2020-09-29T17:35:36.649056","exception":false,"start_time":"2020-09-29T17:35:29.483898","status":"completed"},"tags":[],"id":"C5TEL9qDDV0F","trusted":true},"cell_type":"code","source":"import re\nimport os\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom functools import partial\nimport efficientnet.tfkeras as efn\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"Tensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads.It's easy to run replicated models on Cloud TPU using High-level Tensorflow APIs .\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"# Performance : \n\nTPU can achieve a high computational throughput on massive multiplications and additions for neural networks , at blazingly fast speeds with much less power consumption and smaller footprint.\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"📌 **TPU Initialization:** TPUs are usually on Cloud TPU workers and hence have to be connected to remote clusters and then initialized .\n\n📌 **Distribution strategies :** A distribution strategy is an abstraction that can be used to drive models on CPU, GPUs or TPUs. "},{"metadata":{},"cell_type":"markdown","source":"Credit : The code has been adapted from @mgornergooglestater kernel on Flower classification on TPUs"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-09-29T17:35:36.785423Z","iopub.status.busy":"2020-09-29T17:35:36.732271Z","iopub.status.idle":"2020-09-29T17:35:41.909695Z","shell.execute_reply":"2020-09-29T17:35:41.91053Z"},"papermill":{"duration":5.242582,"end_time":"2020-09-29T17:35:41.910774","exception":false,"start_time":"2020-09-29T17:35:36.668192","status":"completed"},"tags":[],"id":"hyQk56psDV0U","outputId":"57f42752-745e-490e-aa3a-a30c605875bc","trusted":true},"cell_type":"code","source":"try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Device:', tpu.master())\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\nNote: TPUs read data directly from Google Cloud Storage (GCS) This Kaggle utility will copy the dataset to a GCS bucket co-located with the TPU. If you have multiple datasets attached to the notebook, you can pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. Use !ls /kaggle/input/ to list attached datasets.\n
"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"execution":{"iopub.execute_input":"2020-09-29T17:35:41.968507Z","iopub.status.busy":"2020-09-29T17:35:41.967789Z","iopub.status.idle":"2020-09-29T17:35:42.291702Z","shell.execute_reply":"2020-09-29T17:35:42.290784Z"},"papermill":{"duration":0.360735,"end_time":"2020-09-29T17:35:42.29187","exception":false,"start_time":"2020-09-29T17:35:41.931135","status":"completed"},"tags":[],"id":"1TNm_-hkDV0o","trusted":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TF Records :\n\n📌TF-Record is Tensorflow’s own binary storage format.\n\n📌Any byte-string that can be decoded in TensorFlow could be stored in a TFRecord file. Examples include: Lines of text, JSON ,encoded image data, or serialized tf.Tensors .\n\n📌A TFRecord file contains a sequence of records and can only be read sequentially.Each record contains a byte-string, for the data-payload, plus the data-length, and CRC32C (32-bit CRC using the Castagnoli polynomial) hashes for integrity checking.\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"execution":{"iopub.execute_input":"2020-09-29T17:35:42.5044Z","iopub.status.busy":"2020-09-29T17:35:42.503397Z","iopub.status.idle":"2020-09-29T17:35:42.507098Z","shell.execute_reply":"2020-09-29T17:35:42.507707Z"},"papermill":{"duration":0.196123,"end_time":"2020-09-29T17:35:42.507902","exception":false,"start_time":"2020-09-29T17:35:42.311779","status":"completed"},"tags":[],"id":"SUucXwzfDV05","outputId":"32c83bbf-f221-4408-dc25-98e4673d8a8e","trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec'),\n test_size=0.1, random_state=42\n)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')\nprint('Train TFRecord Files:', len(TRAINING_FILENAMES))\nprint('Validation TFRecord Files:', len(VALID_FILENAMES))\nprint('Test TFRecord Files:', len(TEST_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\nImage Source : mc.ai"},{"metadata":{},"cell_type":"markdown","source":"# TF-Data\n\n"},{"metadata":{},"cell_type":"markdown","source":"📌The tf.data API enables you to build complex input pipelines from simple, reusable pieces. The tf.data API also makes it possible to handle large amounts of data, read from different data formats, and perform complex transformations.\n\n📌The tf.data API introduces a tf.data.Dataset abstraction that represents a sequence of elements, in which each element consists of one or more components. For example, in an image pipeline, an element might be a single training example, with a pair of tensor components representing the image and its label.\n\n🎯 **Input Pipeline :**\n\n tf.data.TFRecordDataset() is used to create an input pipeline for data stored in TFRecord format .\n \n🎯 **Transformations :**\n\nThe Dataset object can be transformed into a new Dataset by chaining method calls on the tf.data.Dataset object . Some of the transformations which can be applied are Dataset.map() , Dataset.batch() , Dataset.shuffle() , Dataset.prefetch() .\n\nThe Dataset object is a Python iterable which it possible to consume its elements using a for loop .\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_to_tensor(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(example, labeled):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.int64)\n } if labeled else {\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"image_name\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_to_tensor(example['image'])\n if labeled:\n label = tf.cast(example['target'], tf.int32)\n return image, label\n idnum = example['image_name']\n return image, idnum","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-29T17:35:42.664302Z","iopub.status.busy":"2020-09-29T17:35:42.663307Z","iopub.status.idle":"2020-09-29T17:35:42.666786Z","shell.execute_reply":"2020-09-29T17:35:42.66616Z"},"papermill":{"duration":0.032205,"end_time":"2020-09-29T17:35:42.666907","exception":false,"start_time":"2020-09-29T17:35:42.634702","status":"completed"},"tags":[],"id":"o3mSVhP0DV1o","trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False # disable order, increase speed\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Augmentation :\n\n🎯 Data augmentation is a technique through which one can increase the size of the data for the training of the model without adding the new data. Techniques like padding, cropping, rotating, and flipping are the most common methods that are used over the images to increase the data size. \n\n
\nNote: Both Keras Preprocessing Layers and tf.image can be used for data augmentation. For finer control , tf.image is preferred.\n
\nCyclic learning rates is a learning rate scheduling technique for (1) faster training of a network and (2) a finer understanding of the optimal learning rate. Cyclic learning rates have an effect on the model training process known somewhat fancifully as\"superconvergence\" \n
\n\nSource : @residentmario Notebook"},{"metadata":{},"cell_type":"markdown","source":"\n\nImage Source : https://arxiv.org/abs/1803.09820"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-29T17:35:43.036264Z","iopub.status.busy":"2020-09-29T17:35:43.035479Z","iopub.status.idle":"2020-09-29T17:35:43.039165Z","shell.execute_reply":"2020-09-29T17:35:43.038431Z"},"papermill":{"duration":0.034425,"end_time":"2020-09-29T17:35:43.039298","exception":false,"start_time":"2020-09-29T17:35:43.004873","status":"completed"},"tags":[],"id":"R7qwaU6HDV3f","trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=0.000075, lr_min=0.000001, lr_rampup_epochs=20, lr_sustain_epochs=0, lr_exp_decay=.8):\n lr_max = lr_max * strategy.num_replicas_in_sync\n def lrfn(epoch):\n if epoch < lr_rampup_epochs:\n lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n lr = lr_max\n else:\n lr = (lr_max - lr_min) * lr_exp_decay ** (epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n return lr\n return lrfn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EfficientNets\n\n**Compound Model Scaling: A Better Way to Scale Up CNNs**\n\nEfficientNets uses a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient The first step in the compound scaling method is to perform a grid search to find the relationship between different scaling dimensions of the baseline network under a fixed resource constraint.This determines the appropriate scaling coefficient for each of the dimensions mentioned above. We then apply those coefficients to scale up the baseline network to the desired target model size or computational budget.\n\nThis compound scaling method consistently improves model accuracy and efficiency for scaling up existing models such as MobileNet (+1.4% imagenet accuracy), and ResNet (+0.7%), compared to conventional scaling methods.\n\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"**EfficientNet Architecture :**\n\nThe effectiveness of model scaling also relies heavily on the baseline network which is developed by performing a neural architecture search using the AutoML MNAS framework, which optimizes both accuracy and efficiency (FLOPS). The resulting architecture uses mobile inverted bottleneck convolution (MBConv), similar to MobileNetV2 and MnasNet, but is slightly larger due to an increased FLOP budget.The baseline network is then scaled up to obtain a family of models, called EfficientNets."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"📌 **EfficientNet Performance**\n\n
\nNote:\nThe EfficientNet models achieve both higher accuracy and better efficiency over existing CNNs like ResNet , DenseNet etc , reducing parameter size and FLOPS by an order of magnitude. \n