{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n
\n

Cassava Leaf Disease - TPU Tensorflow - Training

\n
\n\n#### This an introdutory notebook with a quick EDA using the `tfrecords` and a simple demonstration on how to use deep learning with TPU for image classification.\n\n- [Inference notebook](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-inference/data)\n- Dataset source `resized` [128x128](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128), [256x256](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256), [384x384](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384), [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512)\n- Dataset source `center cropped` [128x128](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128), [256x256](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256), [384x384](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384), [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512)\n- Dataset source [discussion thread](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198744)\n- Dataset [creation source](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256)\n\n> This was based on the [competition baseline](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease) and on one of my previous works [Flower Classification with TPUs - EDA and Baseline](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline), from the [Petals to the Metal: Flower Classification on TPU](https://www.kaggle.com/c/tpu-getting-started) competition.","metadata":{"papermill":{"duration":0.034254,"end_time":"2020-10-10T21:25:09.791326","exception":false,"start_time":"2020-10-10T21:25:09.757072","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Dependencies","metadata":{"papermill":{"duration":0.033986,"end_time":"2020-10-10T21:25:09.858267","exception":false,"start_time":"2020-10-10T21:25:09.824281","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install --quiet efficientnet","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math, os, re, warnings, random\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.utils import class_weight\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_score\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import optimizers, applications, Sequential, losses, metrics\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\nimport efficientnet.tfkeras as efn\n\ndef seed_everything(seed=0):\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings('ignore')","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-10-10T21:25:09.94291Z","iopub.status.busy":"2020-10-10T21:25:09.94211Z","iopub.status.idle":"2020-10-10T21:25:26.556399Z","shell.execute_reply":"2020-10-10T21:25:26.555607Z"},"papermill":{"duration":16.665386,"end_time":"2020-10-10T21:25:26.556557","exception":false,"start_time":"2020-10-10T21:25:09.891171","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Hardware configuration","metadata":{"papermill":{"duration":0.03302,"end_time":"2020-10-10T21:25:26.623975","exception":false,"start_time":"2020-10-10T21:25:26.590955","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# TPU or GPU detection\n# Detect hardware, return appropriate distribution strategy\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:26.705656Z","iopub.status.busy":"2020-10-10T21:25:26.704813Z","iopub.status.idle":"2020-10-10T21:25:31.882446Z","shell.execute_reply":"2020-10-10T21:25:31.881768Z"},"papermill":{"duration":5.225184,"end_time":"2020-10-10T21:25:31.882571","exception":false,"start_time":"2020-10-10T21:25:26.657387","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model parameters","metadata":{"papermill":{"duration":0.034123,"end_time":"2020-10-10T21:25:31.952284","exception":false,"start_time":"2020-10-10T21:25:31.918161","status":"completed"},"tags":[]}},{"cell_type":"code","source":"BATCH_SIZE = 16 * REPLICAS\nLEARNING_RATE = 3e-5 * REPLICAS\nEPOCHS = 30\nHEIGHT = 512\nWIDTH = 512\nCHANNELS = 3\nN_CLASSES = 5\nES_PATIENCE = 10\nN_FOLDS = 5","metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.028802Z","iopub.status.busy":"2020-10-10T21:25:32.027887Z","iopub.status.idle":"2020-10-10T21:25:32.031045Z","shell.execute_reply":"2020-10-10T21:25:32.030416Z"},"papermill":{"duration":0.044623,"end_time":"2020-10-10T21:25:32.031164","exception":false,"start_time":"2020-10-10T21:25:31.986541","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data\n\nHere we have three options of data sources\n- Original TFRecods data\n- Original JPG images converted to TFRecods and resized\n- Original JPG images converted to TFRecods square center cropped and resized\n\nWe are using the last option","metadata":{}},{"cell_type":"code","source":"def count_data_items(filenames):\n n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\n\ndatabase_base_path = '/kaggle/input/cassava-leaf-disease-classification/'\ntrain = pd.read_csv(f'{database_base_path}train.csv')\nprint(f'Train samples: {len(train)}')\n\n# GCS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification') # Original dataset\n# GCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-{HEIGHT}x{WIDTH}') # Only resized\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}') # Center croped and resized\n\n# TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/*.tfrec') # Original TFRecords\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\n\nprint(f'GCS: train images: {NUM_TRAINING_IMAGES}')\ndisplay(train.head())\n\nCLASSES = ['Cassava Bacterial Blight', \n 'Cassava Brown Streak Disease', \n 'Cassava Green Mottle', \n 'Cassava Mosaic Disease', \n 'Healthy']","metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.118263Z","iopub.status.busy":"2020-10-10T21:25:32.115381Z","iopub.status.idle":"2020-10-10T21:25:32.677665Z","shell.execute_reply":"2020-10-10T21:25:32.677043Z"},"papermill":{"duration":0.61129,"end_time":"2020-10-10T21:25:32.677805","exception":false,"start_time":"2020-10-10T21:25:32.066515","status":"completed"},"tags":[],"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentation","metadata":{}},{"cell_type":"code","source":"def data_augment(image, label):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n \n # Shear\n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, HEIGHT, shear=20.)\n else:\n image = transform_shear(image, HEIGHT, shear=-20.)\n \n # Rotation\n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, HEIGHT, rotation=45.)\n else:\n image = transform_rotation(image, HEIGHT, rotation=-45.)\n \n # Flips\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_flip_up_down(image)\n if p_spatial > .75:\n image = tf.image.transpose(image)\n \n # Rotates\n if p_rotate > .75:\n image = tf.image.rot90(image, k=3) # rotate 270º\n elif p_rotate > .5:\n image = tf.image.rot90(image, k=2) # rotate 180º\n elif p_rotate > .25:\n image = tf.image.rot90(image, k=1) # rotate 90º\n \n # Pixel-level transforms\n if p_pixel_1 >= .4:\n image = tf.image.random_saturation(image, lower=.7, upper=1.3)\n if p_pixel_2 >= .4:\n image = tf.image.random_contrast(image, lower=.8, upper=1.2)\n if p_pixel_3 >= .4:\n image = tf.image.random_brightness(image, max_delta=.1)\n \n # Crops\n if p_crop > .7:\n if p_crop > .9:\n image = tf.image.central_crop(image, central_fraction=.6)\n elif p_crop > .8:\n image = tf.image.central_crop(image, central_fraction=.7)\n else:\n image = tf.image.central_crop(image, central_fraction=.8)\n elif p_crop > .4:\n crop_size = tf.random.uniform([], int(HEIGHT*.6), HEIGHT, dtype=tf.int32)\n image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n \n image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n\n return image, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Auxiliary functions","metadata":{}},{"cell_type":"code","source":"# data augmentation @cdeotte kernel: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\ndef transform_rotation(image, height, rotation):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly rotated\n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n rotation = rotation * tf.random.uniform([1],dtype='float32')\n # CONVERT DEGREES TO RADIANS\n rotation = math.pi * rotation / 180.\n \n # ROTATION MATRIX\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shear(image, height, shear):\n # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n # output - image randomly sheared\n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n shear = shear * tf.random.uniform([1],dtype='float32')\n shear = math.pi * shear / 180.\n \n # SHEAR MATRIX\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3]) \n\n # LIST DESTINATION PIXEL INDICES\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # FIND ORIGIN PIXEL VALUES \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Datasets utility functions\ndef decode_image(image_data):\n \"\"\"\n 1. Decode a JPEG-encoded image to a uint8 tensor.\n 2. Cast tensor to float and normalizes (range between 0 and 1).\n 3. Resize and reshape images to the expected size.\n \"\"\"\n image = tf.image.decode_jpeg(image_data, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n \n image = tf.image.resize(image, [HEIGHT, WIDTH])\n image = tf.reshape(image, [HEIGHT, WIDTH, 3])\n return image\n\ndef read_tfrecord(example, labeled=True):\n \"\"\"\n 1. Parse data based on the 'TFREC_FORMAT' map.\n 2. Decode image.\n 3. If 'labeled' returns (image, label) if not (image, name).\n \"\"\"\n if labeled:\n TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string), \n 'target': tf.io.FixedLenFeature([], tf.int64), \n }\n else:\n TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string), \n 'image_name': tf.io.FixedLenFeature([], tf.string), \n }\n example = tf.io.parse_single_example(example, TFREC_FORMAT)\n image = decode_image(example['image'])\n if labeled:\n label_or_name = tf.cast(example['target'], tf.int32)\n else:\n label_or_name = example['image_name']\n return image, label_or_name\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n \"\"\"\n Create a Tensorflow dataset from TFRecords.\n \"\"\"\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n dataset = dataset.with_options(ignore_order)\n dataset = dataset.map(lambda x: read_tfrecord(x, labeled=labeled), num_parallel_calls=AUTO)\n return dataset\n\ndef get_dataset(FILENAMES, labeled=True, ordered=False, repeated=False, augment=False):\n \"\"\"\n Return a Tensorflow dataset ready for training or inference.\n \"\"\"\n dataset = load_dataset(FILENAMES, labeled=labeled, ordered=ordered)\n if augment:\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n if repeated:\n dataset = dataset.repeat()\n if not ordered:\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-10-10T21:25:32.781506Z","iopub.status.busy":"2020-10-10T21:25:32.777062Z","iopub.status.idle":"2020-10-10T21:25:32.784982Z","shell.execute_reply":"2020-10-10T21:25:32.78432Z"},"papermill":{"duration":0.072304,"end_time":"2020-10-10T21:25:32.785102","exception":false,"start_time":"2020-10-10T21:25:32.712798","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualization utility functions\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n images, labels = data\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n numpy_labels = [None for _ in enumerate(numpy_images)]\n # If no labels, only image IDs, return None for labels (this is the case for test data)\n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n if correct_label is None:\n return CLASSES[label], True\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n plt.subplot(*subplot)\n plt.axis('off')\n plt.imshow(image)\n if len(title) > 0:\n plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', \n fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n \"\"\"This will work with:\n display_batch_of_images(images)\n display_batch_of_images(images, predictions)\n display_batch_of_images((images, labels))\n display_batch_of_images((images, labels), predictions)\n \"\"\"\n # data\n images, labels = batch_to_numpy_images_and_labels(databatch)\n if labels is None:\n labels = [None for _ in enumerate(images)]\n \n # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n rows = int(math.sqrt(len(images)))\n cols = len(images)//rows\n \n # size and spacing\n FIGSIZE = 13.0\n SPACING = 0.1\n subplot=(rows,cols,1)\n if rows < cols:\n plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n else:\n plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n \n # display\n for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n title = '' if label is None else CLASSES[label]\n correct = True\n if predictions is not None:\n title, correct = title_from_label_and_target(predictions[i], label)\n dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n \n #layout\n plt.tight_layout()\n if label is None and predictions is None:\n plt.subplots_adjust(wspace=0, hspace=0)\n else:\n plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n plt.show()\n \n# Visualize model predictions\ndef dataset_to_numpy_util(dataset, N):\n dataset = dataset.unbatch().batch(N)\n for images, labels in dataset:\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n break; \n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n label = np.argmax(label, axis=-1)\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(label, str(correct), ', shoud be ' if not correct else '',\n correct_label if not correct else ''), correct\n\ndef display_one_flower_eval(image, title, subplot, red=False):\n plt.subplot(subplot)\n plt.axis('off')\n plt.imshow(image)\n plt.title(title, fontsize=14, color='red' if red else 'black')\n return subplot+1\n\ndef display_9_images_with_predictions(images, predictions, labels):\n subplot=331\n plt.figure(figsize=(13,13))\n for i, image in enumerate(images):\n title, correct = title_from_label_and_target(predictions[i], labels[i])\n subplot = display_one_flower_eval(image, title, subplot, not correct)\n if i >= 8:\n break;\n \n plt.tight_layout()\n plt.subplots_adjust(wspace=0.1, hspace=0.1)\n plt.show()\n\n\n# Model evaluation\ndef plot_metrics(history):\n metric_list = [m for m in list(history.keys()) if m is not 'lr']\n size = len(metric_list)//2\n fig, axes = plt.subplots(size, 1, sharex='col', figsize=(20, size * 4))\n if size > 1:\n axes = axes.flatten()\n else:\n axes = [axes]\n \n for index in range(len(metric_list)//2):\n metric_name = metric_list[index]\n val_metric_name = metric_list[index+size]\n axes[index].plot(history[metric_name], label='Train %s' % metric_name)\n axes[index].plot(history[val_metric_name], label='Validation %s' % metric_name)\n axes[index].legend(loc='best', fontsize=16)\n axes[index].set_title(metric_name)\n if 'loss' in metric_name:\n axes[index].axvline(np.argmin(history[metric_name]), linestyle='dashed')\n axes[index].axvline(np.argmin(history[val_metric_name]), linestyle='dashed', color='orange')\n else:\n axes[index].axvline(np.argmax(history[metric_name]), linestyle='dashed')\n axes[index].axvline(np.argmax(history[val_metric_name]), linestyle='dashed', color='orange')\n\n plt.xlabel('Epochs', fontsize=16)\n sns.despine()\n plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:32.896825Z","iopub.status.busy":"2020-10-10T21:25:32.886455Z","iopub.status.idle":"2020-10-10T21:25:32.900548Z","shell.execute_reply":"2020-10-10T21:25:32.899792Z"},"papermill":{"duration":0.08061,"end_time":"2020-10-10T21:25:32.900668","exception":false,"start_time":"2020-10-10T21:25:32.820058","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA\n\n## First let's look at some samples from each set\n\n### Class map\n```\n0: Cassava Bacterial Blight (CBB)\n1: Cassava Brown Streak Disease (CBSD)\n2: Cassava Green Mottle (CGM)\n3: Cassava Mosaic Disease (CMD)\n4: Healthy\n```","metadata":{}},{"cell_type":"markdown","source":"### Training samples","metadata":{"papermill":{"duration":0.035974,"end_time":"2020-10-10T21:25:50.332948","exception":false,"start_time":"2020-10-10T21:25:50.296974","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_dataset = get_dataset(TRAINING_FILENAMES, ordered=True)\ntrain_iter = iter(train_dataset.unbatch().batch(20))\n\ndisplay_batch_of_images(next(train_iter))\ndisplay_batch_of_images(next(train_iter))","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:50.419126Z","iopub.status.busy":"2020-10-10T21:25:50.418355Z","iopub.status.idle":"2020-10-10T21:25:52.964466Z","shell.execute_reply":"2020-10-10T21:25:52.965082Z"},"papermill":{"duration":2.589363,"end_time":"2020-10-10T21:25:52.965239","exception":false,"start_time":"2020-10-10T21:25:50.375876","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The important thing here is that the image quality seems to be very consistent over the dataset, this will make things easier for the model to generalize.\n\nOn the other side, some of the images have a lot of leaves, some are far from the plant, and a few don't have leaves at all.","metadata":{"papermill":{"duration":0.141933,"end_time":"2020-10-10T21:26:01.051276","exception":false,"start_time":"2020-10-10T21:26:00.909343","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Label distribution","metadata":{"papermill":{"duration":0.143843,"end_time":"2020-10-10T21:26:01.335026","exception":false,"start_time":"2020-10-10T21:26:01.191183","status":"completed"},"tags":[]}},{"cell_type":"code","source":"label_count = train.groupby('label', as_index=False).count()\nlabel_count.rename(columns={'image_id': 'Count', 'label': 'Label'}, inplace=True)\nlabel_count['Label'] = label_count['Label'].apply(lambda x: CLASSES[x])\n\nfig, ax = plt.subplots(1, 1, figsize=(18, 10))\nax = sns.barplot(x=label_count['Count'], y=label_count['Label'], palette='viridis')\nax.tick_params(labelsize=16)\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:26:01.627182Z","iopub.status.busy":"2020-10-10T21:26:01.626121Z","iopub.status.idle":"2020-10-10T21:26:05.595561Z","shell.execute_reply":"2020-10-10T21:26:05.596158Z"},"papermill":{"duration":4.122119,"end_time":"2020-10-10T21:26:05.596329","exception":false,"start_time":"2020-10-10T21:26:01.47421","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see the dataset is very unbalanced, most of the samples belong to the `Cassava Mosaic Disease`, but all the other classes sizes are much closer.","metadata":{"papermill":{"duration":0.150306,"end_time":"2020-10-10T21:26:05.896739","exception":false,"start_time":"2020-10-10T21:26:05.746433","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Learning rate schedule\n\nWe are going to use a `cosine learning rate schedule with a warm-up phase`, this may be a good idea since we are using a pre-trained model, the warm-up phase will be useful to avoid the pre-trained weights degradation resulting in catastrophic forgetting, during the schedule the learning rate will slowly decrease to very low values, this helps the model to land on more stable weights.","metadata":{"papermill":{"duration":0.151388,"end_time":"2020-10-10T21:30:00.486807","exception":false,"start_time":"2020-10-10T21:30:00.335419","status":"completed"},"tags":[]}},{"cell_type":"code","source":"LR_START = 1e-8\nLR_MIN = 1e-8\nLR_MAX = LEARNING_RATE\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nN_CYCLES = .5\n\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n progress = (epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) / (EPOCHS - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)\n lr = LR_MAX * (0.5 * (1.0 + tf.math.cos(math.pi * N_CYCLES * 2.0 * progress)))\n if LR_MIN is not None:\n lr = tf.math.maximum(LR_MIN, lr)\n \n return lr\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\n\nsns.set(style='whitegrid')\nfig, ax = plt.subplots(figsize=(20, 6))\nplt.plot(rng, y)\n\nprint(f'{EPOCHS} total epochs and {NUM_TRAINING_IMAGES//BATCH_SIZE} steps per epoch')\nprint(f'Learning rate schedule: {y[0]:.3g} to {max(y):.3g} to {y[-1]:.3g}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:30:00.81171Z","iopub.status.busy":"2020-10-10T21:30:00.803307Z","iopub.status.idle":"2020-10-10T21:30:01.170422Z","shell.execute_reply":"2020-10-10T21:30:01.169829Z"},"papermill":{"duration":0.532334,"end_time":"2020-10-10T21:30:01.170556","exception":false,"start_time":"2020-10-10T21:30:00.638222","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n\nHere I am using EfficientNet, compared to many of the popular computer vision models it has a very good trade-off, between `number of parameters` and `accuracy` on `ImageNet`, I am using the implementation from this [GitHub](https://github.com/qubvel/efficientnet) repository.\n\n![](https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Flower%20Classification%20with%20TPUs/EfficientNet_comparison.jpg)","metadata":{"papermill":{"duration":0.149079,"end_time":"2020-10-10T21:26:06.196269","exception":false,"start_time":"2020-10-10T21:26:06.04719","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def model_fn(input_shape, N_CLASSES):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB3(input_tensor=input_image, \n include_top=False, \n weights='noisy-student', \n pooling='avg')\n\n model = Sequential([\n base_model,\n L.Dropout(.25),\n L.Dense(N_CLASSES, activation='softmax', name='output')\n ])\n\n optimizer = optimizers.Adam(lr=LEARNING_RATE)\n model.compile(optimizer=optimizer, \n loss=losses.SparseCategoricalCrossentropy(), \n metrics=['sparse_categorical_accuracy'])\n \n return model","metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:26:06.501284Z","iopub.status.busy":"2020-10-10T21:26:06.500306Z","iopub.status.idle":"2020-10-10T21:26:06.503182Z","shell.execute_reply":"2020-10-10T21:26:06.502614Z"},"papermill":{"duration":0.159056,"end_time":"2020-10-10T21:26:06.50331","exception":false,"start_time":"2020-10-10T21:26:06.344254","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{"papermill":{"duration":0.152125,"end_time":"2020-10-10T21:30:01.481735","exception":false,"start_time":"2020-10-10T21:30:01.32961","status":"completed"},"tags":[]}},{"cell_type":"code","source":"skf = KFold(n_splits=N_FOLDS, shuffle=True, random_state=seed)\noof_pred = []; oof_labels = []; history_list = []\n\nfor fold,(idxT, idxV) in enumerate(skf.split(np.arange(15))):\n if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n print(f'\\nFOLD: {fold+1}')\n print(f'TRAIN: {idxT} VALID: {idxV}')\n\n # Create train and validation sets\n TRAIN_FILENAMES = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxT])\n VALID_FILENAMES = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxV])\n np.random.shuffle(TRAINING_FILENAMES)\n ct_train = count_data_items(TRAIN_FILENAMES)\n \n ## MODEL\n K.clear_session()\n with strategy.scope():\n model = model_fn((None, None, CHANNELS), N_CLASSES)\n \n model_path = f'model_{fold}.h5'\n es = EarlyStopping(monitor='val_sparse_categorical_accuracy', mode='max', \n patience=ES_PATIENCE, restore_best_weights=True, verbose=1)\n\n ## TRAIN\n history = model.fit(x=get_dataset(TRAIN_FILENAMES, labeled=True, ordered=False, repeated=True, augment=True), \n validation_data=get_dataset(VALID_FILENAMES, labeled=True, ordered=True, repeated=False, augment=False), \n steps_per_epoch=(ct_train // BATCH_SIZE), \n callbacks=[es, LearningRateScheduler(lrfn, verbose=0)], \n epochs=EPOCHS, \n verbose=2).history\n \n history_list.append(history)\n # Save last model weights\n model.save_weights(model_path)\n\n # OOF predictions\n ds_valid = get_dataset(VALID_FILENAMES, labeled=True, ordered=True, repeated=False, augment=False)\n oof_labels.append([target.numpy() for img, target in iter(ds_valid.unbatch())])\n x_oof = ds_valid.map(lambda image, image_name: image)\n oof_pred.append(np.argmax(model.predict(x_oof), axis=-1))\n \n ## RESULTS\n print(f\"#### FOLD {fold+1} OOF Accuracy = {np.max(history['val_sparse_categorical_accuracy']):.3f}\")","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model loss graph","metadata":{"papermill":{"duration":0.157699,"end_time":"2020-10-10T22:09:56.87745","exception":false,"start_time":"2020-10-10T22:09:56.719751","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for fold, history in enumerate(history_list):\n print(f'\\nFOLD: {fold+1}')\n plot_metrics(history)","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:09:57.231883Z","iopub.status.busy":"2020-10-10T22:09:57.231124Z","iopub.status.idle":"2020-10-10T22:09:57.902666Z","shell.execute_reply":"2020-10-10T22:09:57.901809Z"},"papermill":{"duration":0.861991,"end_time":"2020-10-10T22:09:57.902823","exception":false,"start_time":"2020-10-10T22:09:57.040832","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model evaluation\n\nNow we can evaluate the performance of the model, first, we can evaluate the usual metrics like, `accuracy`, `precision`, `recall`, and `f1-score`, `scikit-learn` provides the perfect function for this `classification_report`.\n\nWe are evaluating the model on the `OOF` predictions, it stands for `Out Of Fold`, since we are training using `K-Fold` our model will see all the data, and the correct way to evaluate each fold is by looking at the predictions that are not from that fold.\n\n## OOF metrics","metadata":{"papermill":{"duration":0.180457,"end_time":"2020-10-10T22:09:58.291265","exception":false,"start_time":"2020-10-10T22:09:58.110808","status":"completed"},"tags":[]}},{"cell_type":"code","source":"y_true = np.concatenate(oof_labels)\ny_preds = np.concatenate(oof_pred)\n\nprint(classification_report(y_true, y_preds, target_names=CLASSES))","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:44.510949Z","iopub.status.busy":"2020-10-10T22:10:44.510261Z","iopub.status.idle":"2020-10-10T22:10:49.740311Z","shell.execute_reply":"2020-10-10T22:10:49.739452Z"},"papermill":{"duration":5.399145,"end_time":"2020-10-10T22:10:49.740438","exception":false,"start_time":"2020-10-10T22:10:44.341293","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Confusion matrix\n\nLet's also take a look at the confusion matrix, this will give us an idea about what classes the model is mixing or having a hard time.","metadata":{"papermill":{"duration":0.165203,"end_time":"2020-10-10T22:10:50.079293","exception":false,"start_time":"2020-10-10T22:10:49.91409","status":"completed"},"tags":[]}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(20, 12))\ntrain_cfn_matrix = confusion_matrix(y_true, y_preds, labels=range(len(CLASSES)))\ntrain_cfn_matrix = (train_cfn_matrix.T / train_cfn_matrix.sum(axis=1)).T\ntrain_df_cm = pd.DataFrame(train_cfn_matrix, index=CLASSES, columns=CLASSES)\nax = sns.heatmap(train_df_cm, cmap='Blues', annot=True, fmt='.2f', linewidths=.5).set_title('Train', fontsize=30)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:50.445555Z","iopub.status.busy":"2020-10-10T22:10:50.444837Z","iopub.status.idle":"2020-10-10T22:10:54.125002Z","shell.execute_reply":"2020-10-10T22:10:54.12552Z"},"papermill":{"duration":3.872244,"end_time":"2020-10-10T22:10:54.125651","exception":false,"start_time":"2020-10-10T22:10:50.253407","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize predictions\n\nFinally, it is a good practice to always inspect some of the model's prediction by looking at the data, this can give an idea if the model is getting some predictions wrong because the data is really hard, of if it is because the model is actually bad.\n\n\n### Class map\n```\n0: Cassava Bacterial Blight (CBB)\n1: Cassava Brown Streak Disease (CBSD)\n2: Cassava Green Mottle (CGM)\n3: Cassava Mosaic Disease (CMD)\n4: Healthy\n```\n\n\n## Train set","metadata":{"papermill":{"duration":0.168779,"end_time":"2020-10-10T22:10:58.678136","exception":false,"start_time":"2020-10-10T22:10:58.509357","status":"completed"},"tags":[]}},{"cell_type":"code","source":"x_samp, y_samp = dataset_to_numpy_util(train_dataset, 18)\n\nx_samp_1, y_samp_1 = x_samp[:9,:,:,:], y_samp[:9]\nsamp_preds_1 = model.predict(x_samp_1, batch_size=9)\ndisplay_9_images_with_predictions(x_samp_1, samp_preds_1, y_samp_1)\n\nx_samp_2, y_samp_2 = x_samp[9:,:,:,:], y_samp[9:]\nsamp_preds_2 = model.predict(x_samp_2, batch_size=9)\ndisplay_9_images_with_predictions(x_samp_2, samp_preds_2, y_samp_2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:59.029492Z","iopub.status.busy":"2020-10-10T22:10:59.027464Z","iopub.status.idle":"2020-10-10T22:11:21.88443Z","shell.execute_reply":"2020-10-10T22:11:21.885182Z"},"papermill":{"duration":23.038156,"end_time":"2020-10-10T22:11:21.885383","exception":false,"start_time":"2020-10-10T22:10:58.847227","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]}]}