{"cells":[{"metadata":{"papermill":{"duration":0.034254,"end_time":"2020-10-10T21:25:09.791326","exception":false,"start_time":"2020-10-10T21:25:09.757072","status":"completed"},"tags":[]},"cell_type":"markdown","source":"
\n
\n

Cassava Leaf Disease - TPU Tensorflow - Training

\n
\n\n#### This an introdutory notebook with a quick EDA using the `tfrecords` and a simple demonstration on how to use deep learning with TPU for image classification.\n\n- [Inference notebook](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-inference/data)\n- Dataset source `resized` [128x128](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-128x128), [256x256](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-256x256), [384x384](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-384x384), [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-512x512)\n- Dataset source `center cropped` [128x128](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-128x128), [256x256](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-256x256), [384x384](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-384x384), [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tfrecords-center-512x512)\n- Dataset source [discussion thread](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198744)\n- Dataset [creation source](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256)\n\n> This was based on the [competition baseline](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease) and on one of my previous works [Flower Classification with TPUs - EDA and Baseline](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline), from the [Petals to the Metal: Flower Classification on TPU](https://www.kaggle.com/c/tpu-getting-started) competition."},{"metadata":{"papermill":{"duration":0.033986,"end_time":"2020-10-10T21:25:09.858267","exception":false,"start_time":"2020-10-10T21:25:09.824281","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Установка зависимостей"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install --quiet efficientnet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Импорт библиотек "},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-10-10T21:25:09.94291Z","iopub.status.busy":"2020-10-10T21:25:09.94211Z","iopub.status.idle":"2020-10-10T21:25:26.556399Z","shell.execute_reply":"2020-10-10T21:25:26.555607Z"},"papermill":{"duration":16.665386,"end_time":"2020-10-10T21:25:26.556557","exception":false,"start_time":"2020-10-10T21:25:09.891171","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import math, os, re, warnings, random\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.utils import class_weight\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_score\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import optimizers, applications, Sequential, losses, metrics\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\nimport efficientnet.tfkeras as efn\n\n# Задаем seed, что бы рандом отрабатывал одинаково \ndef seed_everything(seed=0):\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.03302,"end_time":"2020-10-10T21:25:26.623975","exception":false,"start_time":"2020-10-10T21:25:26.590955","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Конфигурация оборудования на котором будут проходить испытания"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:26.705656Z","iopub.status.busy":"2020-10-10T21:25:26.704813Z","iopub.status.idle":"2020-10-10T21:25:31.882446Z","shell.execute_reply":"2020-10-10T21:25:31.881768Z"},"papermill":{"duration":5.225184,"end_time":"2020-10-10T21:25:31.882571","exception":false,"start_time":"2020-10-10T21:25:26.657387","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Определяем TPU или GPU будет использоваться\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.034123,"end_time":"2020-10-10T21:25:31.952284","exception":false,"start_time":"2020-10-10T21:25:31.918161","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Параметры модели"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.028802Z","iopub.status.busy":"2020-10-10T21:25:32.027887Z","iopub.status.idle":"2020-10-10T21:25:32.031045Z","shell.execute_reply":"2020-10-10T21:25:32.030416Z"},"papermill":{"duration":0.044623,"end_time":"2020-10-10T21:25:32.031164","exception":false,"start_time":"2020-10-10T21:25:31.986541","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 16 * REPLICAS\nLEARNING_RATE = 3e-5 * REPLICAS\nEPOCHS = 30\nHEIGHT = 512\nWIDTH = 512\nCHANNELS = 3\nN_CLASSES = 5\nES_PATIENCE = 10\nN_FOLDS = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Загружаем данные\n\nУ нас есть три варианта источников данных:\n- Исходные данные TFRecods\n- Исходные изображения JPG преобразованы в TFRecods и изменен размер изображения\n- Исходные изображения JPG, преобразованы в TFRecods и обрезаны края, изменен размер\n\nМы будем использовать последний вариант"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.118263Z","iopub.status.busy":"2020-10-10T21:25:32.115381Z","iopub.status.idle":"2020-10-10T21:25:32.677665Z","shell.execute_reply":"2020-10-10T21:25:32.677043Z"},"papermill":{"duration":0.61129,"end_time":"2020-10-10T21:25:32.677805","exception":false,"start_time":"2020-10-10T21:25:32.066515","status":"completed"},"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Считаем количество элементов\ndef count_data_items(filenames):\n n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\n# Загружаем данные из соревнования\ndatabase_base_path = '/kaggle/input/cassava-leaf-disease-classification/'\ntrain = pd.read_csv(f'{database_base_path}train.csv')\nprint(f'Train samples: {len(train)}')\n\n# GCS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification') # Original dataset\n# GCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-{HEIGHT}x{WIDTH}') # Only resized\n\n# Изображение с обрезанными краями и измененным размером\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-tfrecords-center-{HEIGHT}x{WIDTH}')\n\n# TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/*.tfrec') # Original TFRecords\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\n\nprint(f'GCS: train images: {NUM_TRAINING_IMAGES}')\ndisplay(train.head())\n\n# Классы которые у нас есть\nCLASSES = ['Cassava Bacterial Blight', \n 'Cassava Brown Streak Disease', \n 'Cassava Green Mottle', \n 'Cassava Mosaic Disease', \n 'Healthy']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Применение различных аугментаций"},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image, label):\n p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n \n # Сдвиг\n if p_shear > .2:\n if p_shear > .6:\n image = transform_shear(image, HEIGHT, shear=20.)\n else:\n image = transform_shear(image, HEIGHT, shear=-20.)\n \n # Случайное количество раз поворачиваем на +-45 градусов\n if p_rotation > .2:\n if p_rotation > .6:\n image = transform_rotation(image, HEIGHT, rotation=45.)\n else:\n image = transform_rotation(image, HEIGHT, rotation=-45.)\n \n # Отражение\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_flip_up_down(image)\n if p_spatial > .75:\n image = tf.image.transpose(image)\n \n # Вращение\n if p_rotate > .75:\n image = tf.image.rot90(image, k=3) # rotate 270º\n elif p_rotate > .5:\n image = tf.image.rot90(image, k=2) # rotate 180º\n elif p_rotate > .25:\n image = tf.image.rot90(image, k=1) # rotate 90º\n \n # Вырезка пикселей\n if p_pixel_1 >= .4:\n image = tf.image.random_saturation(image, lower=.7, upper=1.3)\n if p_pixel_2 >= .4:\n image = tf.image.random_contrast(image, lower=.8, upper=1.2)\n if p_pixel_3 >= .4:\n image = tf.image.random_brightness(image, max_delta=.1)\n \n # Обрезка изображения\n if p_crop > .7:\n if p_crop > .9:\n image = tf.image.central_crop(image, central_fraction=.6)\n elif p_crop > .8:\n image = tf.image.central_crop(image, central_fraction=.7)\n else:\n image = tf.image.central_crop(image, central_fraction=.8)\n elif p_crop > .4:\n crop_size = tf.random.uniform([], int(HEIGHT*.6), HEIGHT, dtype=tf.int32)\n image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n \n image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n\n return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Вспомогательные функции"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def transform_rotation(image, height, rotation):\n # На входе получаем изображение, а не пакет изображений\n # На выходе получаем рандомно повернутое изображение\n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n rotation = rotation * tf.random.uniform([1],dtype='float32')\n # Преобразовываем градусы в радианы\n rotation = math.pi * rotation / 180.\n \n # Матрица вращения\n c1 = tf.math.cos(rotation)\n s1 = tf.math.sin(rotation)\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n # Список пикселей назначения\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # Повернуть пикселы назначения на пиксели исхода\n idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # Найти исходные значения пикселей \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shear(image, height, shear):\n # На входе получаем изображение, а не пакет изображений\n # На выходе получаем рандомно обрезанное изображение\n DIM = height\n XDIM = DIM%2 #fix for size 331\n \n shear = shear * tf.random.uniform([1],dtype='float32')\n shear = math.pi * shear / 180.\n \n # Матрица сдвига\n one = tf.constant([1],dtype='float32')\n zero = tf.constant([0],dtype='float32')\n c2 = tf.math.cos(shear)\n s2 = tf.math.sin(shear)\n shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3]) \n\n # Список пикселей назначения\n x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n z = tf.ones([DIM*DIM],dtype='int32')\n idx = tf.stack( [x,y,z] )\n \n # Повернуть пикселы назначения на пиксели исхода\n idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n idx2 = K.cast(idx2,dtype='int32')\n idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n \n # Найти исходные значения пикселей \n idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n d = tf.gather_nd(image, tf.transpose(idx3))\n \n return tf.reshape(d,[DIM,DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-10-10T21:25:32.781506Z","iopub.status.busy":"2020-10-10T21:25:32.777062Z","iopub.status.idle":"2020-10-10T21:25:32.784982Z","shell.execute_reply":"2020-10-10T21:25:32.78432Z"},"papermill":{"duration":0.072304,"end_time":"2020-10-10T21:25:32.785102","exception":false,"start_time":"2020-10-10T21:25:32.712798","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Служебные функции\ndef decode_image(image_data):\n \"\"\"\n 1. Decode a JPEG-encoded image to a uint8 tensor.\n 2. Cast tensor to float and normalizes (range between 0 and 1).\n 3. Resize and reshape images to the expected size.\n \"\"\"\n image = tf.image.decode_jpeg(image_data, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n \n image = tf.image.resize(image, [HEIGHT, WIDTH])\n image = tf.reshape(image, [HEIGHT, WIDTH, 3])\n return image\n\ndef read_tfrecord(example, labeled=True):\n \"\"\"\n 1. Parse data based on the 'TFREC_FORMAT' map.\n 2. Decode image.\n 3. If 'labeled' returns (image, label) if not (image, name).\n \"\"\"\n if labeled:\n TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string), \n 'target': tf.io.FixedLenFeature([], tf.int64), \n }\n else:\n TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string), \n 'image_name': tf.io.FixedLenFeature([], tf.string), \n }\n example = tf.io.parse_single_example(example, TFREC_FORMAT)\n image = decode_image(example['image'])\n if labeled:\n label_or_name = tf.cast(example['target'], tf.int32)\n else:\n label_or_name = example['image_name']\n return image, label_or_name\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n \"\"\"\n Create a Tensorflow dataset from TFRecords.\n \"\"\"\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n dataset = dataset.with_options(ignore_order)\n dataset = dataset.map(lambda x: read_tfrecord(x, labeled=labeled), num_parallel_calls=AUTO)\n return dataset\n\ndef get_dataset(FILENAMES, labeled=True, ordered=False, repeated=False, augment=False):\n \"\"\"\n Return a Tensorflow dataset ready for training or inference.\n \"\"\"\n dataset = load_dataset(FILENAMES, labeled=labeled, ordered=ordered)\n if augment:\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n if repeated:\n dataset = dataset.repeat()\n if not ordered:\n dataset = dataset.shuffle(2048)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:32.896825Z","iopub.status.busy":"2020-10-10T21:25:32.886455Z","iopub.status.idle":"2020-10-10T21:25:32.900548Z","shell.execute_reply":"2020-10-10T21:25:32.899792Z"},"papermill":{"duration":0.08061,"end_time":"2020-10-10T21:25:32.900668","exception":false,"start_time":"2020-10-10T21:25:32.820058","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Служебные функции визуализации\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n images, labels = data\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n if numpy_labels.dtype == object:\n numpy_labels = [None for _ in enumerate(numpy_images)]\n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n if correct_label is None:\n return CLASSES[label], True\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n plt.subplot(*subplot)\n plt.axis('off')\n plt.imshow(image)\n if len(title) > 0:\n plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', \n fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n \"\"\"This will work with:\n display_batch_of_images(images)\n display_batch_of_images(images, predictions)\n display_batch_of_images((images, labels))\n display_batch_of_images((images, labels), predictions)\n \"\"\"\n images, labels = batch_to_numpy_images_and_labels(databatch)\n if labels is None:\n labels = [None for _ in enumerate(images)]\n \n rows = int(math.sqrt(len(images)))\n cols = len(images)//rows\n \n FIGSIZE = 13.0\n SPACING = 0.1\n subplot=(rows,cols,1)\n if rows < cols:\n plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n else:\n plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n \n\n for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n title = '' if label is None else CLASSES[label]\n correct = True\n if predictions is not None:\n title, correct = title_from_label_and_target(predictions[i], label)\n dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n \n #layout\n plt.tight_layout()\n if label is None and predictions is None:\n plt.subplots_adjust(wspace=0, hspace=0)\n else:\n plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n plt.show()\n \n# Визуализация прогноза модели\ndef dataset_to_numpy_util(dataset, N):\n dataset = dataset.unbatch().batch(N)\n for images, labels in dataset:\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n break; \n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n label = np.argmax(label, axis=-1)\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(label, str(correct), ', shoud be ' if not correct else '',\n correct_label if not correct else ''), correct\n\ndef display_one_flower_eval(image, title, subplot, red=False):\n plt.subplot(subplot)\n plt.axis('off')\n plt.imshow(image)\n plt.title(title, fontsize=14, color='red' if red else 'black')\n return subplot+1\n\ndef display_9_images_with_predictions(images, predictions, labels):\n subplot=331\n plt.figure(figsize=(13,13))\n for i, image in enumerate(images):\n title, correct = title_from_label_and_target(predictions[i], labels[i])\n subplot = display_one_flower_eval(image, title, subplot, not correct)\n if i >= 8:\n break;\n \n plt.tight_layout()\n plt.subplots_adjust(wspace=0.1, hspace=0.1)\n plt.show()\n\n\n# Оценка модели\ndef plot_metrics(history):\n metric_list = [m for m in list(history.keys()) if m is not 'lr']\n size = len(metric_list)//2\n fig, axes = plt.subplots(size, 1, sharex='col', figsize=(20, size * 4))\n if size > 1:\n axes = axes.flatten()\n else:\n axes = [axes]\n \n for index in range(len(metric_list)//2):\n metric_name = metric_list[index]\n val_metric_name = metric_list[index+size]\n axes[index].plot(history[metric_name], label='Train %s' % metric_name)\n axes[index].plot(history[val_metric_name], label='Validation %s' % metric_name)\n axes[index].legend(loc='best', fontsize=16)\n axes[index].set_title(metric_name)\n if 'loss' in metric_name:\n axes[index].axvline(np.argmin(history[metric_name]), linestyle='dashed')\n axes[index].axvline(np.argmin(history[val_metric_name]), linestyle='dashed', color='orange')\n else:\n axes[index].axvline(np.argmax(history[metric_name]), linestyle='dashed')\n axes[index].axvline(np.argmax(history[val_metric_name]), linestyle='dashed', color='orange')\n\n plt.xlabel('Epochs', fontsize=16)\n sns.despine()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Имеющиеся классы изображений\n```\n0: Cassava Bacterial Blight (CBB)\n1: Cassava Brown Streak Disease (CBSD)\n2: Cassava Green Mottle (CGM)\n3: Cassava Mosaic Disease (CMD)\n4: Healthy\n```"},{"metadata":{"papermill":{"duration":0.035974,"end_time":"2020-10-10T21:25:50.332948","exception":false,"start_time":"2020-10-10T21:25:50.296974","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Обучающие данные"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:50.419126Z","iopub.status.busy":"2020-10-10T21:25:50.418355Z","iopub.status.idle":"2020-10-10T21:25:52.964466Z","shell.execute_reply":"2020-10-10T21:25:52.965082Z"},"papermill":{"duration":2.589363,"end_time":"2020-10-10T21:25:52.965239","exception":false,"start_time":"2020-10-10T21:25:50.375876","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train_dataset = get_dataset(TRAINING_FILENAMES, ordered=True)\ntrain_iter = iter(train_dataset.unbatch().batch(20))\n\ndisplay_batch_of_images(next(train_iter))\ndisplay_batch_of_images(next(train_iter))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.143843,"end_time":"2020-10-10T21:26:01.335026","exception":false,"start_time":"2020-10-10T21:26:01.191183","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Распределение Лейблов"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:26:01.627182Z","iopub.status.busy":"2020-10-10T21:26:01.626121Z","iopub.status.idle":"2020-10-10T21:26:05.595561Z","shell.execute_reply":"2020-10-10T21:26:05.596158Z"},"papermill":{"duration":4.122119,"end_time":"2020-10-10T21:26:05.596329","exception":false,"start_time":"2020-10-10T21:26:01.47421","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"label_count = train.groupby('label', as_index=False).count()\nlabel_count.rename(columns={'image_id': 'Count', 'label': 'Label'}, inplace=True)\nlabel_count['Label'] = label_count['Label'].apply(lambda x: CLASSES[x])\n\nfig, ax = plt.subplots(1, 1, figsize=(18, 10))\nax = sns.barplot(x=label_count['Count'], y=label_count['Label'], palette='viridis')\nax.tick_params(labelsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.150306,"end_time":"2020-10-10T21:26:05.896739","exception":false,"start_time":"2020-10-10T21:26:05.746433","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Набор данных не стабилизирован, большинство изображений относятся к `Cassava Mosaic Disease`, остальные примерно равные."},{"metadata":{"papermill":{"duration":0.151388,"end_time":"2020-10-10T21:30:00.486807","exception":false,"start_time":"2020-10-10T21:30:00.335419","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## График обучения"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:30:00.81171Z","iopub.status.busy":"2020-10-10T21:30:00.803307Z","iopub.status.idle":"2020-10-10T21:30:01.170422Z","shell.execute_reply":"2020-10-10T21:30:01.169829Z"},"papermill":{"duration":0.532334,"end_time":"2020-10-10T21:30:01.170556","exception":false,"start_time":"2020-10-10T21:30:00.638222","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"LR_START = 1e-8\nLR_MIN = 1e-8\nLR_MAX = LEARNING_RATE\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nN_CYCLES = .5\n\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n progress = (epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) / (EPOCHS - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)\n lr = LR_MAX * (0.5 * (1.0 + tf.math.cos(math.pi * N_CYCLES * 2.0 * progress)))\n if LR_MIN is not None:\n lr = tf.math.maximum(LR_MIN, lr)\n \n return lr\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\n\nsns.set(style='whitegrid')\nfig, ax = plt.subplots(figsize=(20, 6))\nplt.plot(rng, y)\n\nprint(f'{EPOCHS} total epochs and {NUM_TRAINING_IMAGES//BATCH_SIZE} steps per epoch')\nprint(f'Learning rate schedule: {y[0]:.3g} to {max(y):.3g} to {y[-1]:.3g}')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.149079,"end_time":"2020-10-10T21:26:06.196269","exception":false,"start_time":"2020-10-10T21:26:06.04719","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Модель\n\nЗдесь я использую EfficientNet, по сравнению со многими популярными моделямиу неё очень хороший компромисс между количеством параметров и точностью в ImageNet, я использую реализацию из этого репозитория [GitHub](https://github.com/qubvel/efficientnet) repository.\n\n![](https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Flower%20Classification%20with%20TPUs/EfficientNet_comparison.jpg)"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:26:06.501284Z","iopub.status.busy":"2020-10-10T21:26:06.500306Z","iopub.status.idle":"2020-10-10T21:26:06.503182Z","shell.execute_reply":"2020-10-10T21:26:06.502614Z"},"papermill":{"duration":0.159056,"end_time":"2020-10-10T21:26:06.50331","exception":false,"start_time":"2020-10-10T21:26:06.344254","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def model_fn(input_shape, N_CLASSES):\n input_image = L.Input(shape=input_shape, name='input_image')\n base_model = efn.EfficientNetB3(input_tensor=input_image, \n include_top=False, \n weights='noisy-student', \n pooling='avg')\n\n model = Sequential([\n base_model,\n L.Dropout(.25),\n L.Dense(N_CLASSES, activation='softmax', name='output')\n ])\n\n optimizer = optimizers.Adam(lr=LEARNING_RATE)\n model.compile(optimizer=optimizer, \n loss=losses.SparseCategoricalCrossentropy(), \n metrics=['sparse_categorical_accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.152125,"end_time":"2020-10-10T21:30:01.481735","exception":false,"start_time":"2020-10-10T21:30:01.32961","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Обучение"},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"skf = KFold(n_splits=N_FOLDS, shuffle=True, random_state=seed)\noof_pred = []; oof_labels = []; history_list = []\n\nfor fold,(idxT, idxV) in enumerate(skf.split(np.arange(15))):\n if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n print(f'\\nFOLD: {fold+1}')\n print(f'TRAIN: {idxT} VALID: {idxV}')\n\n # Создание наборов для обучения и проверки\n TRAIN_FILENAMES = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxT])\n VALID_FILENAMES = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxV])\n np.random.shuffle(TRAINING_FILENAMES)\n ct_train = count_data_items(TRAIN_FILENAMES)\n \n K.clear_session()\n with strategy.scope():\n model = model_fn((None, None, CHANNELS), N_CLASSES)\n \n model_path = f'model_{fold}.h5'\n es = EarlyStopping(monitor='val_sparse_categorical_accuracy', mode='max', \n patience=ES_PATIENCE, restore_best_weights=True, verbose=1)\n\n # Обучаем модель\n history = model.fit(x=get_dataset(TRAIN_FILENAMES, labeled=True, ordered=False, repeated=True, augment=True), \n validation_data=get_dataset(VALID_FILENAMES, labeled=True, ordered=True, repeated=False, augment=False), \n steps_per_epoch=(ct_train // BATCH_SIZE), \n callbacks=[es, LearningRateScheduler(lrfn, verbose=0)], \n epochs=EPOCHS, \n verbose=2).history\n \n history_list.append(history)\n # Сохраняем результат\n model.save_weights(model_path)\n \n ds_valid = get_dataset(VALID_FILENAMES, labeled=True, ordered=True, repeated=False, augment=False)\n oof_labels.append([target.numpy() for img, target in iter(ds_valid.unbatch())])\n x_oof = ds_valid.map(lambda image, image_name: image)\n oof_pred.append(np.argmax(model.predict(x_oof), axis=-1))\n \n ## Выводим результат\n print(f\"#### FOLD {fold+1} OOF Accuracy = {np.max(history['val_sparse_categorical_accuracy']):.3f}\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.157699,"end_time":"2020-10-10T22:09:56.87745","exception":false,"start_time":"2020-10-10T22:09:56.719751","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## График потерь"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:09:57.231883Z","iopub.status.busy":"2020-10-10T22:09:57.231124Z","iopub.status.idle":"2020-10-10T22:09:57.902666Z","shell.execute_reply":"2020-10-10T22:09:57.901809Z"},"papermill":{"duration":0.861991,"end_time":"2020-10-10T22:09:57.902823","exception":false,"start_time":"2020-10-10T22:09:57.040832","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"for fold, history in enumerate(history_list):\n print(f'\\nFOLD: {fold+1}')\n plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.180457,"end_time":"2020-10-10T22:09:58.291265","exception":false,"start_time":"2020-10-10T22:09:58.110808","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Оценка модели"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:44.510949Z","iopub.status.busy":"2020-10-10T22:10:44.510261Z","iopub.status.idle":"2020-10-10T22:10:49.740311Z","shell.execute_reply":"2020-10-10T22:10:49.739452Z"},"papermill":{"duration":5.399145,"end_time":"2020-10-10T22:10:49.740438","exception":false,"start_time":"2020-10-10T22:10:44.341293","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"y_true = np.concatenate(oof_labels)\ny_preds = np.concatenate(oof_pred)\n\nprint(classification_report(y_true, y_preds, target_names=CLASSES))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.165203,"end_time":"2020-10-10T22:10:50.079293","exception":false,"start_time":"2020-10-10T22:10:49.91409","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Confusion матрица"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:50.445555Z","iopub.status.busy":"2020-10-10T22:10:50.444837Z","iopub.status.idle":"2020-10-10T22:10:54.125002Z","shell.execute_reply":"2020-10-10T22:10:54.12552Z"},"papermill":{"duration":3.872244,"end_time":"2020-10-10T22:10:54.125651","exception":false,"start_time":"2020-10-10T22:10:50.253407","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(20, 12))\ntrain_cfn_matrix = confusion_matrix(y_true, y_preds, labels=range(len(CLASSES)))\ntrain_cfn_matrix = (train_cfn_matrix.T / train_cfn_matrix.sum(axis=1)).T\ntrain_df_cm = pd.DataFrame(train_cfn_matrix, index=CLASSES, columns=CLASSES)\nax = sns.heatmap(train_df_cm, cmap='Blues', annot=True, fmt='.2f', linewidths=.5).set_title('Train', fontsize=30)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.168779,"end_time":"2020-10-10T22:10:58.678136","exception":false,"start_time":"2020-10-10T22:10:58.509357","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n## Карта классов \n```\n0: Cassava Bacterial Blight (CBB)\n1: Cassava Brown Streak Disease (CBSD)\n2: Cassava Green Mottle (CGM)\n3: Cassava Mosaic Disease (CMD)\n4: Healthy\n```\n\n\n## Набор тестов"},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T22:10:59.029492Z","iopub.status.busy":"2020-10-10T22:10:59.027464Z","iopub.status.idle":"2020-10-10T22:11:21.88443Z","shell.execute_reply":"2020-10-10T22:11:21.885182Z"},"papermill":{"duration":23.038156,"end_time":"2020-10-10T22:11:21.885383","exception":false,"start_time":"2020-10-10T22:10:58.847227","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"x_samp, y_samp = dataset_to_numpy_util(train_dataset, 18)\n\nx_samp_1, y_samp_1 = x_samp[:9,:,:,:], y_samp[:9]\nsamp_preds_1 = model.predict(x_samp_1, batch_size=9)\ndisplay_9_images_with_predictions(x_samp_1, samp_preds_1, y_samp_1)\n\nx_samp_2, y_samp_2 = x_samp[9:,:,:,:], y_samp[9:]\nsamp_preds_2 = model.predict(x_samp_2, batch_size=9)\ndisplay_9_images_with_predictions(x_samp_2, samp_preds_2, y_samp_2)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}