{"cells":[{"metadata":{"papermill":{"duration":0.034254,"end_time":"2020-10-10T21:25:09.791326","exception":false,"start_time":"2020-10-10T21:25:09.757072","status":"completed"},"tags":[]},"cell_type":"markdown","source":"<center><img src=\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Cassava%20Leaf%20Disease%20Classification/banner.png\" width=\"1000\"></center>\n<br>\n<center><h1>Cassava Leaf Disease - Training with TPU v2 Pods</h1></center>\n<br>\n\n#### This is based on my previous work [Cassava Leaf Disease - TPU Tensorflow - Training](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-tensorflow-training)\n\n\n- [Inference notebook](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-tpu-v2-pods-inference)\n- Dataset source `center cropped` [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-center-512x512) - `divided by classes` [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-classes-512x512)\n- Dataset source `external data` `center cropped` [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-50-tfrecords-external-512x512) - `divided by classes` [512x512](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-ext-50-tfrec-classes-512x512)\n- Dataset source [discussion thread](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198744)\n- Dataset [creation source](https://www.kaggle.com/dimitreoliveira/cassava-leaf-disease-stratified-tfrecords-256x256)"},{"metadata":{},"cell_type":"markdown","source":"### What is a TPU pod?\n\n> To accelerate the largest-scale machine learning (ML) applications deployed today and enable rapid development of the ML applications of tomorrow, Google created custom silicon chips called Tensor Processing Units ([TPUs](https://cloud.google.com/tpu/docs/tpus)). When assembled into multi-rack ML supercomputers called Cloud TPU Pods.\n\n\n<center><img src=\"https://miro.medium.com/max/890/1*16HkeV33jzWruFoVYokVlQ.png\" width=\"800\"></center>\n<br>\n\n\n### What’s in a Cloud TPU Pod\n> A single Cloud TPU Pod can include more than 1,000 individual TPU chips which are connected by an ultra-fast, two-dimensional toroidal mesh network, as illustrated below. The TPU software stack uses this mesh network to enable many racks of machines to be programmed as a single, giant ML supercomputer via a variety of flexible, high-level APIs.\n\n\n\n\n> References:\n- [Use TPUs](https://www.tensorflow.org/guide/tpu)\n- [Better performance with the tf.data API](https://www.tensorflow.org/guide/data_performance)\n- [Custom training with tf.distribute.Strategy](https://www.tensorflow.org/tutorials/distribute/custom_training)\n- [Cloud TPU](https://cloud.google.com/tpu)\n- [Google’s scalable supercomputers for machine learning, Cloud TPU Pods, are now publicly available in beta](https://cloud.google.com/blog/products/ai-machine-learning/googles-scalable-supercomputers-for-machine-learning-cloud-tpu-pods-are-now-publicly-available-in-beta)"},{"metadata":{},"cell_type":"markdown","source":"Improvements\n - `Custom training loop`: Using a custom training loop greatly improves the training time and resource usage.\n - `Maximize MXU and minimize Idle time`: I have made a few adjustments to the Tensorflow pipeline to improve performance.\n\nExperiments\n - Small improvements using external data (2019 competition).\n - Small improvements from using `CCE label smoothing`.\n - Small improvements from using `CutOut`.\n - Small improvements from `oversmapling` classes `0`, `1`, `2` and `4`.\n - Small improvements from keeping `batch normalization` layers frozen.\n - No relevant improvements from using `class weights`.\n - No relevant improvements from using `MixUp`.\n - No relevant improvements from using different backbones.\n - Worse performance by using different image resolution even the default `EfficientNet` input size.\n - Changing `Sparse CCE` to `CCE` has no impact, as expected.\n - Was not able to make progressive unfreezing work.\n - Changing the `learning rate` batch wise seems more efficent than epoch wise, specially for the warm up phase."},{"metadata":{"papermill":{"duration":0.033986,"end_time":"2020-10-10T21:25:09.858267","exception":false,"start_time":"2020-10-10T21:25:09.824281","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Зависимости"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"#Сеть построена на EfficientNet\n!pip install --quiet efficientnet","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-10-10T21:25:09.94291Z","iopub.status.busy":"2020-10-10T21:25:09.94211Z","iopub.status.idle":"2020-10-10T21:25:26.556399Z","shell.execute_reply":"2020-10-10T21:25:26.555607Z"},"papermill":{"duration":16.665386,"end_time":"2020-10-10T21:25:26.556557","exception":false,"start_time":"2020-10-10T21:25:09.891171","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#Библиотеки\nimport math, os, re, warnings, random, time\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import classification_report, confusion_matrix\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import optimizers, Sequential, losses, metrics, Model\nfrom tensorflow.keras.callbacks import EarlyStopping\nimport efficientnet.tfkeras as efn\n\n#Определяется сид\ndef seed_everything(seed=0):\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.03302,"end_time":"2020-10-10T21:25:26.623975","exception":false,"start_time":"2020-10-10T21:25:26.590955","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Hardware configuration\n\nNote that we have `32` cores, this is because the `TPU v2 Pod` have more cores than a single `TPU v3` which has `8` cores."},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:26.705656Z","iopub.status.busy":"2020-10-10T21:25:26.704813Z","iopub.status.idle":"2020-10-10T21:25:31.882446Z","shell.execute_reply":"2020-10-10T21:25:31.881768Z"},"papermill":{"duration":5.225184,"end_time":"2020-10-10T21:25:31.882571","exception":false,"start_time":"2020-10-10T21:25:26.657387","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# TPU or GPU detection\n# Detect hardware, return appropriate distribution strategy\n\n#Настраиваем TPU\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.034123,"end_time":"2020-10-10T21:25:31.952284","exception":false,"start_time":"2020-10-10T21:25:31.918161","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Параметры модели"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.028802Z","iopub.status.busy":"2020-10-10T21:25:32.027887Z","iopub.status.idle":"2020-10-10T21:25:32.031045Z","shell.execute_reply":"2020-10-10T21:25:32.030416Z"},"papermill":{"duration":0.044623,"end_time":"2020-10-10T21:25:32.031164","exception":false,"start_time":"2020-10-10T21:25:31.986541","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#Параметры для модели\nBATCH_SIZE = 8 * REPLICAS #Batch size = 64\nLEARNING_RATE = 1e-5 * REPLICAS\nEPOCHS = 10\nHEIGHT = 512\nWIDTH = 512\nHEIGHT_RS = 512\nWIDTH_RS = 512\nCHANNELS = 3\nN_CLASSES = 5\nN_FOLDS = 5\nFOLDS_USED = 5\nES_PATIENCE = 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Загрузка данных"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:25:32.118263Z","iopub.status.busy":"2020-10-10T21:25:32.115381Z","iopub.status.idle":"2020-10-10T21:25:32.677665Z","shell.execute_reply":"2020-10-10T21:25:32.677043Z"},"papermill":{"duration":0.61129,"end_time":"2020-10-10T21:25:32.677805","exception":false,"start_time":"2020-10-10T21:25:32.066515","status":"completed"},"tags":[],"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#Используются 4 датасета\n#2 с текущего соревнования и 2 с прошедшего\n#По одному с классами и без\n#Все изображения square center cropped\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r'-([0-9]*)\\.').search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\n\ndatabase_base_path = '/kaggle/input/cassava-leaf-disease-classification/'\ntrain = pd.read_csv(f'{database_base_path}train.csv')\nprint(f'Train samples: {len(train)}')\n\n# GCS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification') # Original dataset\nGCS_PATH = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-50-tfrecords-center-{HEIGHT}x{WIDTH}') # Center croped and resized (50 TFRecord)\nGCS_PATH_EXT = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-50-tfrecords-external-{HEIGHT}x{WIDTH}') # Center croped and resized (50 TFRecord) (External)\nGCS_PATH_CLASSES = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-50-tfrecords-classes-{HEIGHT}x{WIDTH}') # Center croped and resized (50 TFRecord) by classes\nGCS_PATH_EXT_CLASSES = KaggleDatasets().get_gcs_path(f'cassava-leaf-disease-ext-50-tfrec-classes-{HEIGHT}x{WIDTH}') # Center croped and resized (50 TFRecord) (External) by classes\n\n# FILENAMES_COMP = tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/*.tfrec') # Original TFRecords\nFILENAMES_COMP = tf.io.gfile.glob(GCS_PATH + '/*.tfrec')\nFILENAMES_2019 = tf.io.gfile.glob(GCS_PATH_EXT + '/*.tfrec')\n\nFILENAMES_COMP_CBB = tf.io.gfile.glob(GCS_PATH_CLASSES + '/CBB*.tfrec')\nFILENAMES_COMP_CBSD = tf.io.gfile.glob(GCS_PATH_CLASSES + '/CBSD*.tfrec')\nFILENAMES_COMP_CGM = tf.io.gfile.glob(GCS_PATH_CLASSES + '/CGM*.tfrec')\nFILENAMES_COMP_CMD = tf.io.gfile.glob(GCS_PATH_CLASSES + '/CMD*.tfrec')\nFILENAMES_COMP_Healthy = tf.io.gfile.glob(GCS_PATH_CLASSES + '/Healthy*.tfrec')\n\nFILENAMES_2019_CBB = tf.io.gfile.glob(GCS_PATH_EXT_CLASSES + '/CBB*.tfrec')\nFILENAMES_2019_CBSD = tf.io.gfile.glob(GCS_PATH_EXT_CLASSES + '/CBSD*.tfrec')\nFILENAMES_2019_CGM = tf.io.gfile.glob(GCS_PATH_EXT_CLASSES + '/CGM*.tfrec')\nFILENAMES_2019_CMD = tf.io.gfile.glob(GCS_PATH_EXT_CLASSES + '/CMD*.tfrec')\nFILENAMES_2019_Healthy = tf.io.gfile.glob(GCS_PATH_EXT_CLASSES + '/Healthy*.tfrec')\n\n\nTRAINING_FILENAMES = (FILENAMES_COMP + \n                      FILENAMES_2019 + \n                      (2 * FILENAMES_COMP_CBB) + \n                      (2 * FILENAMES_2019_CBB) + \n                      (2 * FILENAMES_COMP_CBSD) + \n                      (2 * FILENAMES_2019_CBSD) + \n                      (2 * FILENAMES_COMP_CGM) + \n                      (2 * FILENAMES_2019_CGM) + \n                      (2 * FILENAMES_COMP_Healthy) + \n                      (2 * FILENAMES_2019_Healthy))\n\n#Количество изображений\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\n\nprint(f'GCS: train images: {NUM_TRAINING_IMAGES}')\ndisplay(train.head())\n\n#5 классов с названиями\nCLASSES = ['Cassava Bacterial Blight', \n           'Cassava Brown Streak Disease', \n           'Cassava Green Mottle', \n           'Cassava Mosaic Disease', \n           'Healthy']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Аугментация"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Сначала генерируются рандомные значения для разных аугментаций\n#Затем по этим значениям вычисляется, использовать конкретную аугментацию или нет, если да, то как\ndef data_augment(image, label):\n    p_rotation = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_1 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_2 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_pixel_3 = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_shear = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    #Кастомная функция\n    #Сдвиг\n    if p_shear > .2:\n        if p_shear > .6:\n            image = transform_shear(image, HEIGHT, shear=20.)\n        else:\n            image = transform_shear(image, HEIGHT, shear=-20.)\n    \n    #Кстомная функция\n    # Поворот\n    if p_rotation > .2:\n        if p_rotation > .6:\n            image = transform_rotation(image, HEIGHT, rotation=45.)\n        else:\n            image = transform_rotation(image, HEIGHT, rotation=-45.)\n            \n    # Отражение\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    if p_spatial > .75:\n        image = tf.image.transpose(image)\n        \n    # Вращение\n    if p_rotate > .75:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .25:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        \n    # Вырезка пикселей\n    if p_pixel_1 >= .4:\n        image = tf.image.random_saturation(image, lower=.7, upper=1.3) #Рандомная насыщенность цвета\n    if p_pixel_2 >= .4:\n        image = tf.image.random_contrast(image, lower=.8, upper=1.2)   #Рандомный контраст\n    if p_pixel_3 >= .4:\n        image = tf.image.random_brightness(image, max_delta=.1)        #Рандомная яркость\n        \n    # Обрезка изображений\n    if p_crop > .6:\n        if p_crop > .9:\n            image = tf.image.central_crop(image, central_fraction=.5)\n        elif p_crop > .8:\n            image = tf.image.central_crop(image, central_fraction=.6)\n        elif p_crop > .7:\n            image = tf.image.central_crop(image, central_fraction=.7)\n        else:\n            image = tf.image.central_crop(image, central_fraction=.8)\n    elif p_crop > .3:\n        crop_size = tf.random.uniform([], int(HEIGHT*.6), HEIGHT, dtype=tf.int32)\n        image = tf.image.random_crop(image, size=[crop_size, crop_size, CHANNELS])\n            \n    image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n\n    if p_cutout > .5:\n        image = data_augment_cutout(image)\n        \n    return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Вспомогательные функции"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Кастомные функции для аугментации\n# data augmentation @cdeotte kernel: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\ndef transform_rotation(image, height, rotation):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated\n    \n    #Результат - поворот изображения на рандомное количество градусов\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    rotation = rotation * tf.random.uniform([1],dtype='float32')\n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    \n    # ROTATION MATRIX\n    c1 = tf.math.cos(rotation)\n    s1 = tf.math.sin(rotation)\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    rotation_matrix = tf.reshape(tf.concat([c1,s1,zero, -s1,c1,zero, zero,zero,one],axis=0),[3,3])\n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(rotation_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])\n\ndef transform_shear(image, height, shear):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly sheared\n    \n    #Результат - рандомный Shear\n    DIM = height\n    XDIM = DIM%2 #fix for size 331\n    \n    shear = shear * tf.random.uniform([1],dtype='float32')\n    shear = math.pi * shear / 180.\n        \n    # SHEAR MATRIX\n    one = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)\n    shear_matrix = tf.reshape(tf.concat([one,s2,zero, zero,c2,zero, zero,zero,one],axis=0),[3,3])    \n\n    # LIST DESTINATION PIXEL INDICES\n    x = tf.repeat( tf.range(DIM//2,-DIM//2,-1), DIM )\n    y = tf.tile( tf.range(-DIM//2,DIM//2),[DIM] )\n    z = tf.ones([DIM*DIM],dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(shear_matrix,tf.cast(idx,dtype='float32'))\n    idx2 = K.cast(idx2,dtype='int32')\n    idx2 = K.clip(idx2,-DIM//2+XDIM+1,DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES \n    idx3 = tf.stack( [DIM//2-idx2[0,], DIM//2-1+idx2[1,]] )\n    d = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM,DIM,3])\n\n# CutOut - рандомное вырезание небольших прямоугольников\ndef data_augment_cutout(image, min_mask_size=(int(HEIGHT * .1), int(HEIGHT * .1)), \n                        max_mask_size=(int(HEIGHT * .125), int(HEIGHT * .125))):\n    p_cutout = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    if p_cutout > .85: # 10~15 cut outs\n        n_cutout = tf.random.uniform([], 10, 15, dtype=tf.int32)\n        image = random_cutout(image, HEIGHT, WIDTH, \n                              min_mask_size=min_mask_size, max_mask_size=max_mask_size, k=n_cutout)\n    elif p_cutout > .6: # 5~10 cut outs\n        n_cutout = tf.random.uniform([], 5, 10, dtype=tf.int32)\n        image = random_cutout(image, HEIGHT, WIDTH, \n                              min_mask_size=min_mask_size, max_mask_size=max_mask_size, k=n_cutout)\n    elif p_cutout > .25: # 2~5 cut outs\n        n_cutout = tf.random.uniform([], 2, 5, dtype=tf.int32)\n        image = random_cutout(image, HEIGHT, WIDTH, \n                              min_mask_size=min_mask_size, max_mask_size=max_mask_size, k=n_cutout)\n    else: # 1 cut out\n        image = random_cutout(image, HEIGHT, WIDTH, \n                              min_mask_size=min_mask_size, max_mask_size=max_mask_size, k=1)\n\n    return image\n\ndef random_cutout(image, height, width, channels=3, min_mask_size=(10, 10), max_mask_size=(80, 80), k=1):\n    assert height > min_mask_size[0]\n    assert width > min_mask_size[1]\n    assert height > max_mask_size[0]\n    assert width > max_mask_size[1]\n\n    for i in range(k):\n      mask_height = tf.random.uniform(shape=[], minval=min_mask_size[0], maxval=max_mask_size[0], dtype=tf.int32)\n      mask_width = tf.random.uniform(shape=[], minval=min_mask_size[1], maxval=max_mask_size[1], dtype=tf.int32)\n\n      pad_h = height - mask_height\n      pad_top = tf.random.uniform(shape=[], minval=0, maxval=pad_h, dtype=tf.int32)\n      pad_bottom = pad_h - pad_top\n\n      pad_w = width - mask_width\n      pad_left = tf.random.uniform(shape=[], minval=0, maxval=pad_w, dtype=tf.int32)\n      pad_right = pad_w - pad_left\n\n      cutout_area = tf.zeros(shape=[mask_height, mask_width, channels], dtype=tf.uint8)\n\n      cutout_mask = tf.pad([cutout_area], [[0,0],[pad_top, pad_bottom], [pad_left, pad_right], [0,0]], constant_values=1)\n      cutout_mask = tf.squeeze(cutout_mask, axis=0)\n      image = tf.multiply(tf.cast(image, tf.float32), tf.cast(cutout_mask, tf.float32))\n\n    return image","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-10-10T21:25:32.781506Z","iopub.status.busy":"2020-10-10T21:25:32.777062Z","iopub.status.idle":"2020-10-10T21:25:32.784982Z","shell.execute_reply":"2020-10-10T21:25:32.78432Z"},"papermill":{"duration":0.072304,"end_time":"2020-10-10T21:25:32.785102","exception":false,"start_time":"2020-10-10T21:25:32.712798","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Datasets utility functions\n\n# Вспомогательные функции для работы с датасетом\n\ndef decode_image(image_data):\n    \"\"\"\n        Decode a JPEG-encoded image to a uint8 tensor. Декодирование JPEG в unit8\n    \"\"\"\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    return image\n\ndef scale_image(image, label):\n    \"\"\"\n        Cast tensor to float and normalizes (range between 0 and 1). Приведение тензора к float и нормализация\n    \"\"\"\n    image = tf.cast(image, tf.float32)\n    image /= 255.0\n    return image, label\n\ndef prepare_image(image, label):\n    \"\"\"\n        Resize and reshape images to the expected size. Изменение размера изображения\n    \"\"\"\n    image = tf.image.resize(image, [HEIGHT_RS, WIDTH_RS])\n    image = tf.reshape(image, [HEIGHT_RS, WIDTH_RS, 3])\n    return image, label\n\ndef read_tfrecord(example, labeled=True):\n    \"\"\"\n        1. Parse data based on the 'TFREC_FORMAT' map.\n        2. Decode image.\n        3. If 'labeled' returns (image, label) if not (image, name). Читает tfrec и возварщает изображение с label либо name\n    \"\"\"\n    if labeled:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'target': tf.io.FixedLenFeature([], tf.int64), \n        }\n    else:\n        TFREC_FORMAT = {\n            'image': tf.io.FixedLenFeature([], tf.string), \n            'image_name': tf.io.FixedLenFeature([], tf.string), \n        }\n    example = tf.io.parse_single_example(example, TFREC_FORMAT)\n    image = decode_image(example['image'])\n    if labeled:\n        label_or_name = tf.cast(example['target'], tf.int32)\n        # One-Hot Encoding needed to use \"categorical_crossentropy\" loss\n        label_or_name = tf.one_hot(tf.cast(label_or_name, tf.int32), N_CLASSES)\n    else:\n        label_or_name = example['image_name']\n    return image, label_or_name\n\ndef get_dataset(FILENAMES, labeled=True, ordered=False, repeated=False, \n                cached=False, augment=False):\n    \"\"\"\n        Return a Tensorflow dataset ready for training or inference. Подготавливает датасет к работе\n    \"\"\"\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False\n        dataset = tf.data.Dataset.list_files(FILENAMES)\n        dataset = dataset.interleave(tf.data.TFRecordDataset, num_parallel_calls=AUTO)\n    else:\n        dataset = tf.data.TFRecordDataset(FILENAMES, num_parallel_reads=AUTO)\n        \n    dataset = dataset.with_options(ignore_order)\n    \n    dataset = dataset.map(lambda x: read_tfrecord(x, labeled=labeled), num_parallel_calls=AUTO)\n    \n    if augment:\n        dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n        \n    dataset = dataset.map(scale_image, num_parallel_calls=AUTO)\n    dataset = dataset.map(prepare_image, num_parallel_calls=AUTO)\n    \n    if not ordered:\n        dataset = dataset.shuffle(2048)\n    if repeated:\n        dataset = dataset.repeat()\n        \n    dataset = dataset.batch(BATCH_SIZE)\n    \n    if cached:\n        dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef unfreeze_model(model):\n    # Unfreeze layers while leaving BatchNorm layers frozen\n    \n    #Размораживает модель, кроме слоев BatchNormalization\n    for layer in model.layers:\n        if not isinstance(layer, L.BatchNormalization):\n            layer.trainable = True\n        else:\n            layer.trainable = False\n                \ndef unfreeze_block(model, block_name=None, n_top=3):\n    # Unfreeze layers while leaving BatchNorm layers frozen\n    \n    #Размораживает слои в блоке, кроме BatchNormalization\n    for layer in model.layers[:-n_top]:\n        if isinstance(layer, L.BatchNormalization):\n            layer.trainable = False\n        else:\n            if block_name and (block_name in layer.name):\n                layer.trainable = True","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:25:32.896825Z","iopub.status.busy":"2020-10-10T21:25:32.886455Z","iopub.status.idle":"2020-10-10T21:25:32.900548Z","shell.execute_reply":"2020-10-10T21:25:32.899792Z"},"papermill":{"duration":0.08061,"end_time":"2020-10-10T21:25:32.900668","exception":false,"start_time":"2020-10-10T21:25:32.820058","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# Visualization utility functions\n\n# Вспомогательные функции для визуализации\n\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', \n                  fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    labels = np.argmax(labels, axis=-1)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n    \n# Visualize model predictions\ndef dataset_to_numpy_util(dataset, N):\n    dataset = dataset.unbatch().batch(N)\n    for images, labels in dataset:\n        numpy_images = images.numpy()\n        numpy_labels = labels.numpy()\n        break;  \n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    label = np.argmax(label, axis=-1)\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(label, str(correct), ', shoud be ' if not correct else '',\n                                correct_label if not correct else ''), correct\n\ndef display_one_flower_eval(image, title, subplot, red=False):\n    plt.subplot(subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    plt.title(title, fontsize=14, color='red' if red else 'black')\n    return subplot+1\n\ndef display_9_images_with_predictions(images, predictions, labels):\n    subplot=331\n    plt.figure(figsize=(13,13))\n    for i, image in enumerate(images):\n        title, correct = title_from_label_and_target(predictions[i], labels[i])\n        subplot = display_one_flower_eval(image, title, subplot, not correct)\n        if i >= 8:\n            break;\n              \n    plt.tight_layout()\n    plt.subplots_adjust(wspace=0.1, hspace=0.1)\n    plt.show()\n\n\n# Model evaluation\ndef plot_metrics(history):\n    fig, axes = plt.subplots(2, 1, sharex='col', figsize=(20, 8))\n    axes = axes.flatten()\n    \n    axes[0].plot(history['loss'], label='Train loss')\n    axes[0].plot(history['val_loss'], label='Validation loss')\n    axes[0].legend(loc='best', fontsize=16)\n    axes[0].set_title('Loss')\n    axes[0].axvline(np.argmin(history['loss']), linestyle='dashed')\n    axes[0].axvline(np.argmin(history['val_loss']), linestyle='dashed', color='orange')\n    \n    axes[1].plot(history['accuracy'], label='Train accuracy')\n    axes[1].plot(history['val_accuracy'], label='Validation accuracy')\n    axes[1].legend(loc='best', fontsize=16)\n    axes[1].set_title('Accuracy')\n    axes[1].axvline(np.argmax(history['accuracy']), linestyle='dashed')\n    axes[1].axvline(np.argmax(history['val_accuracy']), linestyle='dashed', color='orange')\n\n    plt.xlabel('Epochs', fontsize=16)\n    sns.despine()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training data samples (with augmentation)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#Подготавливается датасет и показываются примеры\ntrain_dataset = get_dataset(FILENAMES_COMP, ordered=True, augment=True)\ntrain_iter = iter(train_dataset.unbatch().batch(20))\n\ndisplay_batch_of_images(next(train_iter))\ndisplay_batch_of_images(next(train_iter))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Датасет текущего соревнования"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ds_comp = get_dataset(FILENAMES_COMP)\nlabels_comp = [target.numpy() for img, target in iter(ds_comp.unbatch())]\nlabels_comp = np.argmax(labels_comp, axis=-1)\n\nfig, ax = plt.subplots(1, 1, figsize=(18, 8))\nax = sns.countplot(y=labels_comp, palette='viridis')\nax.tick_params(labelsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Датасет соревнования 2019\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ds_2019 = get_dataset(FILENAMES_2019)\nlabels_2019 = [target.numpy() for img, target in iter(ds_2019.unbatch())]\nlabels_2019 = np.argmax(labels_2019, axis=-1)\n\nfig, ax = plt.subplots(1, 1, figsize=(18, 8))\nax = sns.countplot(y=labels_2019, palette='viridis')\nax.tick_params(labelsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dataset oversampled"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#Изображения из недостающих классов дубируются и подвергаются аугментации, что лучше балансирует классы\nFILENAMES_COMP_OVER = (FILENAMES_COMP + \n                       FILENAMES_2019 + \n                       (2 * FILENAMES_COMP_CBB) + \n                       (2 * FILENAMES_2019_CBB) + \n                       (2 * FILENAMES_COMP_CBSD) + \n                       (2 * FILENAMES_2019_CBSD) + \n                       (2 * FILENAMES_COMP_CGM) + \n                       (2 * FILENAMES_2019_CGM) + \n                       (2 * FILENAMES_COMP_Healthy) + \n                       (2 * FILENAMES_2019_Healthy))\n\nds_comp = get_dataset(FILENAMES_COMP_OVER)\nlabels_comp = [target.numpy() for img, target in iter(ds_comp.unbatch())]\nlabels_comp = np.argmax(labels_comp, axis=-1)\n\nfig, ax = plt.subplots(1, 1, figsize=(18, 8))\nax = sns.countplot(y=labels_comp, palette='viridis')\nax.tick_params(labelsize=16)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.151388,"end_time":"2020-10-10T21:30:00.486807","exception":false,"start_time":"2020-10-10T21:30:00.335419","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Learning rate schedule\n\nWe are going to use a `cosine learning rate schedule with a warm-up phase`, this may be a good idea since we are using a pre-trained model, the warm-up phase will be useful to avoid the pre-trained weights degradation resulting in catastrophic forgetting, during the schedule the learning rate will slowly decrease to very low values, this helps the model to land on more stable weights."},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-10-10T21:30:00.81171Z","iopub.status.busy":"2020-10-10T21:30:00.803307Z","iopub.status.idle":"2020-10-10T21:30:01.170422Z","shell.execute_reply":"2020-10-10T21:30:01.169829Z"},"papermill":{"duration":0.532334,"end_time":"2020-10-10T21:30:01.170556","exception":false,"start_time":"2020-10-10T21:30:00.638222","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#Warm-up - период \"разминки\", который снижает вероятность излишней фокусировки\n#сети на конкретных особенностях изображений в начале обучения\nlr_start = 1e-8\nlr_min = 1e-8\nlr_max = LEARNING_RATE\nnum_cycles = 1.\nwarmup_epochs = 1   #Одна warm-up эпоха\nhold_max_epochs = 0\ntotal_epochs = EPOCHS\nwarmup_steps = warmup_epochs * (NUM_TRAINING_IMAGES//BATCH_SIZE)\ntotal_steps = total_epochs * (NUM_TRAINING_IMAGES//BATCH_SIZE)\n\n@tf.function\ndef lrfn(step):\n    if step < warmup_steps:\n        lr = (lr_max - lr_start) / warmup_steps * step + lr_start\n    else:\n        progress = (step - warmup_steps) / (total_steps - warmup_steps)\n        lr = lr_max * (0.5 * (1.0 + tf.math.cos(np.pi * ((num_cycles * progress) % 1.0))))\n        if lr_min is not None:\n            lr = tf.math.maximum(lr_min, float(lr))\n\n    return lr\n\n\n# rng = [i for i in range(total_epochs)]\nrng = [i for i in range(total_steps)]\ny = [lrfn(tf.cast(x, tf.float32)) for x in rng]\n\nsns.set(style='whitegrid')\nfig, ax = plt.subplots(figsize=(20, 6))\nplt.plot(rng, y)\n\nprint(f'{total_steps} total steps and {NUM_TRAINING_IMAGES//BATCH_SIZE} steps per epoch')\nprint(f'Learning rate schedule: {y[0]:.3g} to {max(y):.3g} to {y[-1]:.3g}')","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.149079,"end_time":"2020-10-10T21:26:06.196269","exception":false,"start_time":"2020-10-10T21:26:06.04719","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Модель"},{"metadata":{"execution":{"iopub.execute_input":"2020-10-10T21:26:06.501284Z","iopub.status.busy":"2020-10-10T21:26:06.500306Z","iopub.status.idle":"2020-10-10T21:26:06.503182Z","shell.execute_reply":"2020-10-10T21:26:06.502614Z"},"papermill":{"duration":0.159056,"end_time":"2020-10-10T21:26:06.50331","exception":false,"start_time":"2020-10-10T21:26:06.344254","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"#Определяется модель EfficientNet\ndef model_fn(input_shape, N_CLASSES):\n    inputs = L.Input(shape=input_shape, name='input_image')\n    base_model = efn.EfficientNetB4(input_tensor=inputs, \n                                    include_top=False, \n                                    weights='noisy-student', \n                                    pooling='avg')\n    base_model.trainable = False\n\n    x = L.Dropout(.5)(base_model.output)\n    output = L.Dense(N_CLASSES, activation='softmax', name='output')(x)\n    model = Model(inputs=inputs, outputs=output)\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.152125,"end_time":"2020-10-10T21:30:01.481735","exception":false,"start_time":"2020-10-10T21:30:01.32961","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Training"},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"#При использовании K-Fold датасет разделяется на несколько равных частей (в данном случае - 5)\n#Первая часть ипользуется как датасет валидации, а все остальные - как обучающий датасет\n#После обучения следующая часть становится датасетом валидации, обучение начинается снова,\n#пока все части датасета не побывали валидирующими и обучающими\n#Таким образом сеть может обучаться и валидироваться на всем датасете\n\nskf = KFold(n_splits=N_FOLDS, shuffle=True, random_state=seed)\noof_pred = []; oof_labels = []; history_list = []\n\nfor fold,(idxT, idxV) in enumerate(skf.split(np.arange(50))):\n    if fold >= FOLDS_USED:\n        break\n    if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    K.clear_session()\n    print(f'\\nFOLD: {fold+1}')\n    print(f'TRAIN: {idxT} VALID: {idxV}')\n\n    # Create train and validation sets\n    #Создание тренировочных и валидирующих датасетов\n    FILENAMES_COMP = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_2019 = tf.io.gfile.glob([GCS_PATH_EXT + '/Id_train%.2i*.tfrec' % x for x in idxT])\n\n    FILENAMES_COMP_CBB = tf.io.gfile.glob([GCS_PATH_CLASSES + '/CBB%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_COMP_CBSD = tf.io.gfile.glob([GCS_PATH_CLASSES + '/CBSD%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_COMP_CGM = tf.io.gfile.glob([GCS_PATH_CLASSES + '/CGM%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_COMP_Healthy = tf.io.gfile.glob([GCS_PATH_CLASSES + '/Healthy%.2i*.tfrec' % x for x in idxT])\n    \n    FILENAMES_2019_CBB = tf.io.gfile.glob([GCS_PATH_EXT_CLASSES + '/CBB%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_2019_CBSD = tf.io.gfile.glob([GCS_PATH_EXT_CLASSES + '/CBSD%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_2019_CGM = tf.io.gfile.glob([GCS_PATH_EXT_CLASSES + '/CGM%.2i*.tfrec' % x for x in idxT])\n    FILENAMES_2019_Healthy = tf.io.gfile.glob([GCS_PATH_EXT_CLASSES + '/Healthy%.2i*.tfrec' % x for x in idxT])\n\n    TRAIN_FILENAMES = (FILENAMES_COMP + \n                       FILENAMES_2019 + \n                       (2 * FILENAMES_COMP_CBB) + \n                       (2 * FILENAMES_2019_CBB) + \n                       (2 * FILENAMES_COMP_CBSD) + \n                       (2 * FILENAMES_2019_CBSD) + \n                       (2 * FILENAMES_COMP_CGM) + \n                       (2 * FILENAMES_2019_CGM) + \n                       (2 * FILENAMES_COMP_Healthy) + \n                       (2 * FILENAMES_2019_Healthy))\n    \n    VALID_FILENAMES = tf.io.gfile.glob([GCS_PATH + '/Id_train%.2i*.tfrec' % x for x in idxV])\n    np.random.shuffle(TRAIN_FILENAMES)\n    \n    ct_train = count_data_items(TRAIN_FILENAMES)\n    ct_valid = count_data_items(VALID_FILENAMES)\n    \n    step_size = (ct_train // BATCH_SIZE)\n    valid_step_size = (ct_valid // BATCH_SIZE)\n    total_steps=(total_epochs * step_size)\n    warmup_steps=(warmup_epochs * step_size)\n    \n    \n    # Build TF datasets\n    #Построение датасетов\n    train_ds = strategy.experimental_distribute_dataset(get_dataset(TRAIN_FILENAMES, repeated=True, augment=True))\n    valid_ds = strategy.experimental_distribute_dataset(get_dataset(VALID_FILENAMES, ordered=True, repeated=True, cached=True))\n    train_data_iter = iter(train_ds)\n    valid_data_iter = iter(valid_ds)\n    \n    \n    # Step functions\n    #Пошаговые функции\n    @tf.function\n    def train_step(data_iter):\n        def train_step_fn(x, y):\n            with tf.GradientTape() as tape:\n                probabilities = model(x, training=True)\n                loss = loss_fn(y, probabilities, label_smoothing=.3)\n            gradients = tape.gradient(loss, model.trainable_variables)\n            optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n            # update metrics\n            train_accuracy.update_state(y, probabilities)\n            train_loss.update_state(loss)\n        for _ in tf.range(step_size):\n            strategy.experimental_run_v2(train_step_fn, next(data_iter))\n\n    @tf.function\n    def valid_step(data_iter):\n        def valid_step_fn(x, y):\n            probabilities = model(x, training=False)\n            loss = loss_fn(y, probabilities)\n            # update metrics\n            valid_accuracy.update_state(y, probabilities)\n            valid_loss.update_state(loss)\n        for _ in tf.range(valid_step_size):\n            strategy.experimental_run_v2(valid_step_fn, next(data_iter))\n    \n    \n    # Model\n    #Описание модели\n    model_path = f'model_{fold}.h5'\n    with strategy.scope():\n        model = model_fn((None, None, CHANNELS), N_CLASSES)\n        unfreeze_model(model) # unfreeze all layers except \"batch normalization\"\n        \n        optimizer = optimizers.Adam(learning_rate=lambda: lrfn(tf.cast(optimizer.iterations, tf.float32)))\n        loss_fn = losses.categorical_crossentropy\n\n        train_accuracy = metrics.CategoricalAccuracy()\n        valid_accuracy = metrics.CategoricalAccuracy()\n        train_loss = metrics.Sum()\n        valid_loss = metrics.Sum()\n    \n    \n    # Setup training loop\n    #Создание круга обучения\n    step = 0\n    epoch_steps = 0\n    patience_cnt = 0\n    best_val = 0\n    history = {'loss': [], 'val_loss': [], 'accuracy': [], 'val_accuracy': []}\n\n    ### Train model\n    #Обучение модели\n    for epoch in range(EPOCHS):\n        epoch_start_time = time.time()\n\n        # Run training step\n        #Выполнение одного шага\n        train_step(train_data_iter)\n        epoch_steps += step_size\n        step += step_size\n            \n\n        # Validation run at the end of each epoch\n        #Валидация в конце эпохи\n        if (step // step_size) > epoch:\n            # Validation run\n            #Запуск валидации\n            valid_epoch_steps = 0\n            valid_step(valid_data_iter)\n            valid_epoch_steps += valid_step_size\n\n            # Compute metrics\n            #Расчет метрик\n            history['accuracy'].append(train_accuracy.result().numpy())\n            history['loss'].append(train_loss.result().numpy() / (BATCH_SIZE * epoch_steps))\n            history['val_accuracy'].append(valid_accuracy.result().numpy())\n            history['val_loss'].append(valid_loss.result().numpy() / (BATCH_SIZE * valid_epoch_steps))\n\n            # Report metrics\n            #Отчет по метрикам\n            epoch_time = time.time() - epoch_start_time\n            print(f'\\nEPOCH {epoch+1}/{EPOCHS}')\n            print(f'time: {epoch_time:0.1f}s',\n                  f\"loss: {history['loss'][-1]:0.4f}\",\n                  f\"accuracy: {history['accuracy'][-1]:0.4f}\",\n                  f\"val_loss: {history['val_loss'][-1]:0.4f}\",\n                  f\"val_accuracy: {history['val_accuracy'][-1]:0.4f}\",\n                  f'lr: {lrfn(tf.cast(optimizer.iterations, tf.int32).numpy()):0.4g}')\n\n            # Early stopping monitor\n            #Ранняя остановка\n            if history['val_accuracy'][-1] >= best_val:\n                best_val = history['val_accuracy'][-1]\n                model.save_weights(model_path)\n                print(f'Saved model weights at \"{model_path}\"')\n                patience_cnt = 1\n            else:\n                patience_cnt += 1\n            if patience_cnt > ES_PATIENCE:\n                print(f'Epoch {epoch:05d}: early stopping')\n                break\n\n                \n            # Set up next epoch\n            #Подготовка следующей эпохи\n            epoch = step // step_size\n            epoch_steps = 0\n            train_accuracy.reset_states()\n            train_loss.reset_states()\n            valid_accuracy.reset_states()\n            valid_loss.reset_states()\n    \n    \n    ### RESULTS\n    #Результаты\n    print(f\"#### FOLD {fold+1} OOF Accuracy = {np.max(history['val_accuracy']):.3f}\")\n    \n    history_list.append(history)\n    # Load best model weights\n    #Загрузка весов\n    model.load_weights(model_path)\n\n    # OOF predictions\n    #Предсказания\n    ds_valid = get_dataset(VALID_FILENAMES, ordered=True)\n    oof_labels.append([target.numpy() for img, target in iter(ds_valid.unbatch())])\n    x_oof = ds_valid.map(lambda image, target: image)\n    oof_pred.append(np.argmax(model.predict(x_oof), axis=-1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# График потерь"},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold, history in enumerate(history_list):\n    print(f'\\nFOLD: {fold+1}')\n    plot_metrics(history)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}