{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.039382,"end_time":"2020-11-19T21:45:23.042097","exception":false,"start_time":"2020-11-19T21:45:23.002715","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Introduction\n\n\n\n**TPUs with TensorFlow**  \nWe'll be using TensorFlow and Keras to build our computer vision model, and using TPUs to both train our model and make predictions. If you'd like to learn about more about TPUs be sure to check out our **[Learn With Me: Getting Started with Tensor Processing Units (TPUs)](https://youtu.be/1pdwRQ1DQfY)** video.  \n\n**References**  \nThis notebook was built using the following amazing resources created by Kagglers\n- **Martin Gorner:** [Getting Started With 100 Flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu)\n- **Amy Jang:** [TensorFlow + Transfer Learning: Melanoma](https://www.kaggle.com/amyjang/tensorflow-transfer-learning-melanoma)\n- **Phil Culliton:** [A Simple TF 2.1 Notebook](https://www.kaggle.com/philculliton/a-simple-tf-2-1-notebook)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"papermill":{"duration":0.035886,"end_time":"2020-11-19T21:45:23.118837","exception":false,"start_time":"2020-11-19T21:45:23.082951","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Tensor Processing Units (TPUs)\n\nTensor Processing Units (TPUs) are hardware accelerators that are specialized for deep learning tasks. All Kagglers have 30 hours of free TPU time each week, and can use up to 3 hours in a single session (although if you'd like to increase your TPU quota consider submitting an exemplary TPU notebook to our **[TPU Star program](https://www.kaggle.com/tpu-prize)**!)   \n\nYou can read through the Kaggle documentation on TPUs **[here](https://www.kaggle.com/docs/tpu)**, and check out the TPU Star notebooks **[here](https://www.kaggle.com/tpu-stars)**."},{"metadata":{"papermill":{"duration":0.037375,"end_time":"2020-11-19T21:45:23.192515","exception":false,"start_time":"2020-11-19T21:45:23.15514","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Set up environment"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:45:23.274004Z","iopub.status.busy":"2020-11-19T21:45:23.273154Z","iopub.status.idle":"2020-11-19T21:45:30.119096Z","shell.execute_reply":"2020-11-19T21:45:30.119725Z"},"papermill":{"duration":6.890298,"end_time":"2020-11-19T21:45:30.119979","exception":false,"start_time":"2020-11-19T21:45:23.229681","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import math, re, os, warnings, random\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow import keras\nfrom functools import partial\nfrom sklearn.model_selection import train_test_split\n\n\nfrom sklearn.utils import class_weight\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_score\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import optimizers, applications, Sequential, losses, metrics\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\n\nprint(\"Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.037444,"end_time":"2020-11-19T21:45:30.195328","exception":false,"start_time":"2020-11-19T21:45:30.157884","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Detect TPU\nWhat we're doing with our code here is making sure that we'll be sending our data across a TPU. What you're looking for is a printout of `Number of replicas: 8`, corresponding to the 8 cores of a TPU. If your printout instead says `Number of replicas: 1` you likely do not have TPUs enabled in your notebook.   \n\nTo enable TPUs navigate to the panel on the right and click on `Accelerator`. Choose TPU from the dropdown.  \n\nIf you'd like more TPU troubleshooting and optimization guidelines check out our **[Learn With Me: Troubleshooting and Optimizing TPUs video](https://youtu.be/BSeWHzjMHMU)**.  "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:45:30.373218Z","iopub.status.busy":"2020-11-19T21:45:30.372403Z","iopub.status.idle":"2020-11-19T21:45:34.382672Z","shell.execute_reply":"2020-11-19T21:45:34.382035Z"},"papermill":{"duration":4.150374,"end_time":"2020-11-19T21:45:34.382816","exception":false,"start_time":"2020-11-19T21:45:30.232442","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.038122,"end_time":"2020-11-19T21:45:34.458722","exception":false,"start_time":"2020-11-19T21:45:34.4206","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Set up variables\nWe'll set up some of our variables for our notebook here. "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:45:34.555293Z","iopub.status.busy":"2020-11-19T21:45:34.541822Z","iopub.status.idle":"2020-11-19T21:47:59.71579Z","shell.execute_reply":"2020-11-19T21:47:59.714961Z"},"papermill":{"duration":145.219568,"end_time":"2020-11-19T21:47:59.715925","exception":false,"start_time":"2020-11-19T21:45:34.496357","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nGCS_PATH = KaggleDatasets().get_gcs_path()\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [512, 512]\nCLASSES = ['0', '1', '2', '3', '4']\nEPOCHS = 25\nHEIGHT = 512\nWIDTH = 512\nCHANNELS = 3\n","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.038439,"end_time":"2020-11-19T21:47:59.869037","exception":false,"start_time":"2020-11-19T21:47:59.830598","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Decode the data\nIn the code chunk below we'll set up a series of functions that allow us to convert our images into tensors so that we can utilize them in our model. We'll also normalize our data. Our images are using a \"Red, Blue, Green (RBG)\" scale that has a range of [0, 255], and by normalizing it we'll set each pixel's value to a number in the range of [0, 1]. "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:47:59.952622Z","iopub.status.busy":"2020-11-19T21:47:59.951868Z","iopub.status.idle":"2020-11-19T21:47:59.954997Z","shell.execute_reply":"2020-11-19T21:47:59.955558Z"},"papermill":{"duration":0.04859,"end_time":"2020-11-19T21:47:59.955731","exception":false,"start_time":"2020-11-19T21:47:59.907141","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.039515,"end_time":"2020-11-19T21:48:00.034969","exception":false,"start_time":"2020-11-19T21:47:59.995454","status":"completed"},"tags":[]},"cell_type":"markdown","source":"You might notice that this function accounts for unlabeled images. This is because our test image doesn't have any labels.  "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:00.123967Z","iopub.status.busy":"2020-11-19T21:48:00.123143Z","iopub.status.idle":"2020-11-19T21:48:00.126902Z","shell.execute_reply":"2020-11-19T21:48:00.126284Z"},"papermill":{"duration":0.052475,"end_time":"2020-11-19T21:48:00.127039","exception":false,"start_time":"2020-11-19T21:48:00.074564","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def read_tfrecord(example, labeled):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.int64)\n    } if labeled else {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"image_name\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    if labeled:\n        label = tf.cast(example['target'], tf.int32)\n        return image, label\n    idnum = example['image_name']\n    return image, idnum","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.038946,"end_time":"2020-11-19T21:48:00.205445","exception":false,"start_time":"2020-11-19T21:48:00.166499","status":"completed"},"tags":[]},"cell_type":"markdown","source":"We'll use the following function to load our dataset. One of the advantages of a TPU is that we can run multiple files across the TPU at once, and this accounts for the speed advantages of using a TPU. To capitalize on that, we want to make sure that we're using data as soon as it streams in, rather than creating a data streaming bottleneck."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:00.325942Z","iopub.status.busy":"2020-11-19T21:48:00.324875Z","iopub.status.idle":"2020-11-19T21:48:00.327502Z","shell.execute_reply":"2020-11-19T21:48:00.328493Z"},"papermill":{"duration":0.073623,"end_time":"2020-11-19T21:48:00.328703","exception":false,"start_time":"2020-11-19T21:48:00.25508","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTOTUNE) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:00.683596Z","iopub.status.busy":"2020-11-19T21:48:00.607588Z","iopub.status.idle":"2020-11-19T21:48:00.687244Z","shell.execute_reply":"2020-11-19T21:48:00.686445Z"},"papermill":{"duration":0.225941,"end_time":"2020-11-19T21:48:00.687385","exception":false,"start_time":"2020-11-19T21:48:00.461444","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"TRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n    tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/ld_train*.tfrec'),\n    test_size=0.35, random_state=5\n)\n\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test_tfrecords/ld_test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.038372,"end_time":"2020-11-19T21:48:00.765394","exception":false,"start_time":"2020-11-19T21:48:00.727022","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Adding in augmentations \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image, label):\n    p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    # Shear\n    if p_shear > .2:\n        if p_shear > .6:\n            image = transform_shear(image, HEIGHT, shear=20.)\n        else:\n            image = transform_shear(image, HEIGHT, shear=-20.)\n            \n    # Rotation\n    if p_rotation > .2:\n        if p_rotation > .6:\n            image = transform_rotation(image, HEIGHT, rotation=45.)\n        else:\n            image = transform_rotation(image, HEIGHT, rotation=-45.)\n            \n    # Flips\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    if p_spatial > .75:\n        image = tf.image.transpose(image)\n        \n    # Rotates\n    if p_rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        \n    # Pixel-level transforms\n    if p_pixel_1 >= .4:\n        image = tf.image.random_saturation(image, lower=.7, upper=1.3)\n    if p_pixel_2 >= .4:\n        image = tf.image.random_contrast(image, lower=.8, upper=1.2)\n    if p_pixel_3 >= .4:\n        image = tf.image.random_brightness(image, max_delta=.1)\n        \n    # Crops\n    if p_crop > .7:\n        if p_crop > .9:\n            image = tf.image.central_crop(image, central_fraction=.6)\n        elif p_crop > .8:\n            image = tf.image.central_crop(image, central_fraction=.7)\n        else:\n            image = tf.image.central_crop(image, central_fraction=.8)\n    elif p_crop > .4:\n        crop_size = tf.random.uniform([], int(HEIGHT*.6), HEIGHT, dtype=tf.int32)\n        image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n            \n    image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data augmentation @cdeotte kernel: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\ndef transform_rotation(image, height, rotation):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    rotation = rotation * tf.random.uniform([1],dtype='float32')\n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shear(image, height, shear):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly sheared\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    shear = shear * tf.random.uniform([1],dtype='float32')\n    shear = math.pi * shear / 180.\n        \n    # SHEAR MATRIX\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3])    \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.038742,"end_time":"2020-11-19T21:48:00.930185","exception":false,"start_time":"2020-11-19T21:48:00.891443","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Define data loading methods\nThe following functions will be used to load our `training`, `validation`, and `test` datasets, as well as print out the number of images in each dataset."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.018921Z","iopub.status.busy":"2020-11-19T21:48:01.017912Z","iopub.status.idle":"2020-11-19T21:48:01.02164Z","shell.execute_reply":"2020-11-19T21:48:01.020852Z"},"papermill":{"duration":0.052326,"end_time":"2020-11-19T21:48:01.021791","exception":false,"start_time":"2020-11-19T21:48:00.969465","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)  \n    dataset = dataset.map(data_augment, num_parallel_calls=AUTOTUNE)  \n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.109616Z","iopub.status.busy":"2020-11-19T21:48:01.108559Z","iopub.status.idle":"2020-11-19T21:48:01.111418Z","shell.execute_reply":"2020-11-19T21:48:01.111986Z"},"papermill":{"duration":0.049787,"end_time":"2020-11-19T21:48:01.112145","exception":false,"start_time":"2020-11-19T21:48:01.062358","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALID_FILENAMES, labeled=True, ordered=ordered) \n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.204402Z","iopub.status.busy":"2020-11-19T21:48:01.203418Z","iopub.status.idle":"2020-11-19T21:48:01.207545Z","shell.execute_reply":"2020-11-19T21:48:01.20682Z"},"papermill":{"duration":0.050395,"end_time":"2020-11-19T21:48:01.207665","exception":false,"start_time":"2020-11-19T21:48:01.15727","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.301631Z","iopub.status.busy":"2020-11-19T21:48:01.30084Z","iopub.status.idle":"2020-11-19T21:48:01.304479Z","shell.execute_reply":"2020-11-19T21:48:01.303807Z"},"papermill":{"duration":0.05422,"end_time":"2020-11-19T21:48:01.304611","exception":false,"start_time":"2020-11-19T21:48:01.250391","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.39253Z","iopub.status.busy":"2020-11-19T21:48:01.391743Z","iopub.status.idle":"2020-11-19T21:48:01.395039Z","shell.execute_reply":"2020-11-19T21:48:01.395972Z"},"papermill":{"duration":0.051209,"end_time":"2020-11-19T21:48:01.396198","exception":false,"start_time":"2020-11-19T21:48:01.344989","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALID_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nprint('Dataset: {} training images, {} validation images, {} (unlabeled) test images'.format(\n    NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.041086,"end_time":"2020-11-19T21:48:01.478205","exception":false,"start_time":"2020-11-19T21:48:01.437119","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Brief exploratory data analysis (EDA)\nFirst we'll print out the shapes and labels for a sample of each of our three datasets:"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:01.575573Z","iopub.status.busy":"2020-11-19T21:48:01.574436Z","iopub.status.idle":"2020-11-19T21:48:18.597144Z","shell.execute_reply":"2020-11-19T21:48:18.596204Z"},"papermill":{"duration":17.07767,"end_time":"2020-11-19T21:48:18.597304","exception":false,"start_time":"2020-11-19T21:48:01.519634","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in get_training_dataset().take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())\nprint(\"Validation data shapes:\")\nfor image, label in get_validation_dataset().take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Validation data label examples:\", label.numpy())\nprint(\"Test data shapes:\")\nfor image, idnum in get_test_dataset().take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.044101,"end_time":"2020-11-19T21:48:18.6862","exception":false,"start_time":"2020-11-19T21:48:18.642099","status":"completed"},"tags":[]},"cell_type":"markdown","source":"The following code chunk sets up a series of functions that will print out a grid of images. The grid of images will contain images and their corresponding labels."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:18.782198Z","iopub.status.busy":"2020-11-19T21:48:18.781353Z","iopub.status.idle":"2020-11-19T21:48:18.808005Z","shell.execute_reply":"2020-11-19T21:48:18.807301Z"},"papermill":{"duration":0.077342,"end_time":"2020-11-19T21:48:18.808133","exception":false,"start_time":"2020-11-19T21:48:18.730791","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_plant(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_plant(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:18.904519Z","iopub.status.busy":"2020-11-19T21:48:18.903424Z","iopub.status.idle":"2020-11-19T21:48:18.956871Z","shell.execute_reply":"2020-11-19T21:48:18.956219Z"},"papermill":{"duration":0.104176,"end_time":"2020-11-19T21:48:18.957003","exception":false,"start_time":"2020-11-19T21:48:18.852827","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# load our training dataset for EDA\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:20.176923Z","iopub.status.busy":"2020-11-19T21:48:20.175758Z","iopub.status.idle":"2020-11-19T21:48:22.374002Z","shell.execute_reply":"2020-11-19T21:48:22.374605Z"},"papermill":{"duration":3.371477,"end_time":"2020-11-19T21:48:22.374778","exception":false,"start_time":"2020-11-19T21:48:19.003301","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# run this cell again for another randomized set of training images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.086379,"end_time":"2020-11-19T21:48:22.547481","exception":false,"start_time":"2020-11-19T21:48:22.461102","status":"completed"},"tags":[]},"cell_type":"markdown","source":"You can also modify the above code to look at your `validation` and `test` data, like this:"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:22.732786Z","iopub.status.busy":"2020-11-19T21:48:22.731592Z","iopub.status.idle":"2020-11-19T21:48:22.769732Z","shell.execute_reply":"2020-11-19T21:48:22.768929Z"},"papermill":{"duration":0.136485,"end_time":"2020-11-19T21:48:22.769878","exception":false,"start_time":"2020-11-19T21:48:22.633393","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# load our validation dataset for EDA\nvalidation_dataset = get_validation_dataset()\nvalidation_dataset = validation_dataset.unbatch().batch(20)\nvalid_batch = iter(validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:22.946423Z","iopub.status.busy":"2020-11-19T21:48:22.945571Z","iopub.status.idle":"2020-11-19T21:48:26.010139Z","shell.execute_reply":"2020-11-19T21:48:26.010819Z"},"papermill":{"duration":3.155802,"end_time":"2020-11-19T21:48:26.010993","exception":false,"start_time":"2020-11-19T21:48:22.855191","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# run this cell again for another randomized set of training images\ndisplay_batch_of_images(next(valid_batch))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:26.368142Z","iopub.status.busy":"2020-11-19T21:48:26.367301Z","iopub.status.idle":"2020-11-19T21:48:26.410393Z","shell.execute_reply":"2020-11-19T21:48:26.411021Z"},"papermill":{"duration":0.232531,"end_time":"2020-11-19T21:48:26.411209","exception":false,"start_time":"2020-11-19T21:48:26.178678","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# load our test dataset for EDA\ntesting_dataset = get_test_dataset()\ntesting_dataset = testing_dataset.unbatch().batch(20)\ntest_batch = iter(testing_dataset)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:26.744678Z","iopub.status.busy":"2020-11-19T21:48:26.743913Z","iopub.status.idle":"2020-11-19T21:48:27.89988Z","shell.execute_reply":"2020-11-19T21:48:27.900494Z"},"papermill":{"duration":1.333241,"end_time":"2020-11-19T21:48:27.900651","exception":false,"start_time":"2020-11-19T21:48:26.56741","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# we only have one test image\ndisplay_batch_of_images(next(test_batch))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.230199,"end_time":"2020-11-19T21:48:28.353794","exception":false,"start_time":"2020-11-19T21:48:28.123595","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Building the model\n## Learning rate schedule\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:28.8307Z","iopub.status.busy":"2020-11-19T21:48:28.829632Z","iopub.status.idle":"2020-11-19T21:48:28.833152Z","shell.execute_reply":"2020-11-19T21:48:28.832481Z"},"papermill":{"duration":0.248904,"end_time":"2020-11-19T21:48:28.83328","exception":false,"start_time":"2020-11-19T21:48:28.584376","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"lr_scheduler = keras.optimizers.schedules.ExponentialDecay(\n    initial_learning_rate=1e-5, \n    decay_steps=10000, \n    decay_rate=0.9)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.22538,"end_time":"2020-11-19T21:48:29.285377","exception":false,"start_time":"2020-11-19T21:48:29.059997","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Building our model\nIn order to ensure that our model is trained on the TPU, we build it using `with strategy.scope()`.    \n\nWe will use _pre-trained model_ (ResNet50) as our base model \n\nNote that we're using `sparse_categorical_crossentropy` as our loss function, because we did _not_ one-hot encode our labels."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:29.74622Z","iopub.status.busy":"2020-11-19T21:48:29.745069Z","iopub.status.idle":"2020-11-19T21:48:49.158244Z","shell.execute_reply":"2020-11-19T21:48:49.157378Z"},"papermill":{"duration":19.661572,"end_time":"2020-11-19T21:48:49.158413","exception":false,"start_time":"2020-11-19T21:48:29.496841","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():       \n    img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.resnet50.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n    \n    base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)\n    base_model.trainable = False\n    \n    model = tf.keras.Sequential([\n        tf.keras.layers.BatchNormalization(renorm=True),\n        img_adjust_layer,\n        base_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(1024, activation='relu'),\n        #tf.keras.layers.BatchNormalization(renorm=True),\n        #tf.keras.layers.Dropout(0.5)(x)\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')  \n    ])\n    \n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=lr_scheduler, epsilon=0.001),\n        loss='sparse_categorical_crossentropy',  \n        metrics=['sparse_categorical_accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.174404,"end_time":"2020-11-19T21:48:49.513099","exception":false,"start_time":"2020-11-19T21:48:49.338695","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Train the model\nAs our model is training you'll see a printout for each epoch, and can also monitor TPU usage by clicking on the TPU metrics in the toolbar at the top right of your notebook."},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:49.868188Z","iopub.status.busy":"2020-11-19T21:48:49.867432Z","iopub.status.idle":"2020-11-19T21:48:49.93567Z","shell.execute_reply":"2020-11-19T21:48:49.936263Z"},"papermill":{"duration":0.249051,"end_time":"2020-11-19T21:48:49.936435","exception":false,"start_time":"2020-11-19T21:48:49.687384","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# load data\ntrain_dataset = get_training_dataset()\nvalid_dataset = get_validation_dataset()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T21:48:50.304573Z","iopub.status.busy":"2020-11-19T21:48:50.303472Z","iopub.status.idle":"2020-11-19T22:04:46.794408Z","shell.execute_reply":"2020-11-19T22:04:46.795473Z"},"papermill":{"duration":956.681695,"end_time":"2020-11-19T22:04:46.795744","exception":false,"start_time":"2020-11-19T21:48:50.114049","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALID_STEPS = NUM_VALIDATION_IMAGES // BATCH_SIZE\n\nhistory = model.fit(train_dataset, \n                    steps_per_epoch=STEPS_PER_EPOCH, \n                    epochs=EPOCHS,\n                    validation_data=valid_dataset,\n                    validation_steps=VALID_STEPS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T22:04:57.050462Z","iopub.status.busy":"2020-11-19T22:04:57.0494Z","iopub.status.idle":"2020-11-19T22:04:57.053166Z","shell.execute_reply":"2020-11-19T22:04:57.053855Z"},"papermill":{"duration":1.31245,"end_time":"2020-11-19T22:04:57.054025","exception":false,"start_time":"2020-11-19T22:04:55.741575","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# print out variables available to us\nprint(history.history.keys())","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T22:04:59.5746Z","iopub.status.busy":"2020-11-19T22:04:59.573814Z","iopub.status.idle":"2020-11-19T22:04:59.983142Z","shell.execute_reply":"2020-11-19T22:04:59.982506Z"},"papermill":{"duration":1.671861,"end_time":"2020-11-19T22:04:59.983272","exception":false,"start_time":"2020-11-19T22:04:58.311411","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# create learning curves to evaluate model performance\nhistory_frame = pd.DataFrame(history.history)\nhistory_frame.loc[:, ['loss', 'val_loss']].plot()\nhistory_frame.loc[:, ['sparse_categorical_accuracy', 'val_sparse_categorical_accuracy']].plot();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Model Save\nmodel.save('model_cassava.h5')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.326243,"end_time":"2020-11-19T22:05:02.628032","exception":false,"start_time":"2020-11-19T22:05:01.301789","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Making predictions\nNow that we've trained our model we can use it to make predictions! "},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T22:05:05.184942Z","iopub.status.busy":"2020-11-19T22:05:05.183725Z","iopub.status.idle":"2020-11-19T22:05:05.18757Z","shell.execute_reply":"2020-11-19T22:05:05.186823Z"},"papermill":{"duration":1.270192,"end_time":"2020-11-19T22:05:05.187694","exception":false,"start_time":"2020-11-19T22:05:03.917502","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# this code will convert our test image data to a float32 \ndef to_float32(image, label):\n    return tf.cast(image, tf.float32), label","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T22:05:07.746039Z","iopub.status.busy":"2020-11-19T22:05:07.744935Z","iopub.status.idle":"2020-11-19T22:05:22.234912Z","shell.execute_reply":"2020-11-19T22:05:22.235492Z"},"papermill":{"duration":15.776858,"end_time":"2020-11-19T22:05:22.235661","exception":false,"start_time":"2020-11-19T22:05:06.458803","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) \ntest_ds = test_ds.map(to_float32)\n\nprint('Computing predictions...')\ntest_images_ds = testing_dataset\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.271799,"end_time":"2020-11-19T22:05:24.759257","exception":false,"start_time":"2020-11-19T22:05:23.487458","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Creating a submission file\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-11-19T22:05:27.316025Z","iopub.status.busy":"2020-11-19T22:05:27.315202Z","iopub.status.idle":"2020-11-19T22:05:28.241598Z","shell.execute_reply":"2020-11-19T22:05:28.24078Z"},"papermill":{"duration":2.185537,"end_time":"2020-11-19T22:05:28.241723","exception":false,"start_time":"2020-11-19T22:05:26.056186","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"print('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":1.255302,"end_time":"2020-11-19T22:05:30.746339","exception":false,"start_time":"2020-11-19T22:05:29.491037","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Be aware that because this is a code competition with a hidden test set, internet and TPUs cannot be enabled on your submission notebook. Therefore TPUs will only be available for training models. For a walk-through on how to train on TPUs and run inference/submit on GPUs, see our [TPU Docs](https://www.kaggle.com/docs/tpu#tpu6)."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}