{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n
\n

Flower with TPUs - K-Fold optimized training loops

\n
\n\n#### This notebooks implements Martin Gorner's suggestions to improve training time, here I'll also show how to incorporate the optimizations inside a tipical K-Fold training loop. From the [reference notebook](https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu):\n\n#### Optimizations:\n- use dataset.batch(BATCH_SIZE, drop_remainder=True)\nThe training dataset is infinitely repeated so drop_remainder=True should not be needed. However, whith the setting, Tensorflow produces batches of a known size and although XLA (the TPU compiler) can now handle variable batches, it is slightly faster on fixed batches.\nOn the validation dataset, this setting can drop some validation images. It is not the case here because the validation dataset happens to contain an integral number of batches.\n\n#### Optimizations specific to the TPU-optimized custom training loop:\n- The training and validation step functions run multiple batches at once. This is achieved by placing a loop using `tf.range()` in the step function. The loop will be compiled to (thanks to ` @tf.function`) and executed on TPU.\n- The validation dataset is made to repeat indefinitely because handling end-of-dataset exception in a TPU loop implemented with `tf.range()` is not yet possible. Validation is adjusted to always use exactly or more than the entire validation dataset. This could change numerics. It happens that in this example, the validation dataset is used exactly once per validation.\n- The validation dataset iterator is not reset between validation runs. Since the iterator is passed into the step function which is then compiled for TPU (thanks to ` @tf.function`), passing a fresh iterator for every validation run would trigger a fresh recompilation. With a validation at the end of every epoch this would be slow.\n- Losses are reported through Keras metrics. It is possible to return values from step function and return losses in that way. However, in the optimized version of the custom training loop, using `tf.range()`, aggregating losses returned from multiple batches becomes impractical.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Dependencies","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install --quiet efficientnet\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport os, re, math, warnings, time\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import classification_report, confusion_matrix\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.models import load_model\nfrom tensorflow.keras import optimizers, applications, Sequential, layers, metrics, losses\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\nimport efficientnet.tfkeras as efn\n\ndef seed_everything(seed=0):\n np.random.seed(seed)\n tf.random.set_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nseed = 0\nseed_everything(seed)\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TPU configuration","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# TPU or GPU detection\n# Detect hardware, return appropriate distribution strategy\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model parameters","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 16 * strategy.num_replicas_in_sync\nEPOCHS = 10\nLEARNING_RATE = 3e-5 * strategy.num_replicas_in_sync\nHEIGHT = 512\nWIDTH = 512\nCHANNELS = 3\nN_CLASSES = 104\nN_FOLDS = 5\nFOLDS_USED = 3\n\nGCS_PATH = KaggleDatasets().get_gcs_path() + '/tfrecords-jpeg-%sx%s' % (HEIGHT, WIDTH)\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec') + tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n\nCLASSES = [\n 'pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea', \n 'wild geranium', 'tiger lily', 'moon orchid', 'bird of paradise', 'monkshood', \n 'globe thistle', 'snapdragon', \"colt's foot\", 'king protea', 'spear thistle', \n 'yellow iris', 'globe-flower', 'purple coneflower', 'peruvian lily', \n 'balloon flower', 'giant white arum lily', 'fire lily', 'pincushion flower', \n 'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy', \n 'prince of wales feathers', 'stemless gentian', 'artichoke', 'sweet william', \n 'carnation', 'garden phlox', 'love in the mist', 'cosmos', 'alpine sea holly', \n 'ruby-lipped cattleya', 'cape flower', 'great masterwort', 'siam tulip', \n 'lenten rose', 'barberton daisy', 'daffodil', 'sword lily', 'poinsettia', \n 'bolero deep blue', 'wallflower', 'marigold', 'buttercup', 'daisy', \n 'common dandelion', 'petunia', 'wild pansy', 'primula', 'sunflower', \n 'lilac hibiscus', 'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', \n 'pink-yellow dahlia', 'cautleya spicata', 'japanese anemone', 'black-eyed susan', \n 'silverbush', 'californian poppy', 'osteospermum', 'spring crocus', 'iris', \n 'windflower', 'tree poppy', 'gazania', 'azalea', 'water lily', 'rose', \n 'thorn apple', 'morning glory', 'passion flower', 'lotus', 'toad lily', \n 'anthurium', 'frangipani', 'clematis', 'hibiscus', 'columbine', 'desert-rose', \n 'tree mallow', 'magnolia', 'cyclamen ', 'watercress', 'canna lily', \n 'hippeastrum ', 'bee balm', 'pink quill', 'foxglove', 'bougainvillea', \n 'camellia', 'mallow', 'mexican petunia', 'bromelia', 'blanket flower', \n 'trumpet creeper', 'blackberry lily', 'common tulip', 'wild rose']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Datasets utility functions\nAUTO = tf.data.experimental.AUTOTUNE # instructs the API to read from multiple files if available.\n\ndef decode_image(image_data):\n image = tf.image.decode_jpeg(image_data, channels=3)\n image = tf.cast(image, tf.float32) / 255.0\n image = tf.reshape(image, [HEIGHT, WIDTH, 3])\n return image\n\ndef read_labeled_tfrecord(example):\n LABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"class\": tf.io.FixedLenFeature([], tf.int64), # shape [] means single element\n }\n example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n label = tf.cast(example['class'], tf.int32)\n return image, label\n\ndef read_unlabeled_tfrecord(example):\n UNLABELED_TFREC_FORMAT = {\n \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n \"id\": tf.io.FixedLenFeature([], tf.string), # shape [] means single element\n # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n }\n example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n image = decode_image(example['image'])\n idnum = example['id']\n return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n ignore_order = tf.data.Options()\n if not ordered:\n ignore_order.experimental_deterministic = False # disable order, increase speed\n\n dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n return dataset\n\ndef data_augment(image, label):\n p_spatial = tf.random.uniform([1], minval=0, maxval=1, dtype='float32', seed=seed)\n p_spatial2 = tf.random.uniform([1], minval=0, maxval=1, dtype='float32', seed=seed)\n p_pixel = tf.random.uniform([1], minval=0, maxval=1, dtype='float32', seed=seed)\n p_crop = tf.random.uniform([1], minval=0, maxval=1, dtype='float32', seed=seed)\n \n ### Spatial-level transforms\n if p_spatial >= .2: # flips\n image = tf.image.random_flip_left_right(image, seed=seed)\n image = tf.image.random_flip_up_down(image, seed=seed)\n \n if p_crop >= .7: # crops\n if p_crop >= .95:\n image = tf.image.random_crop(image, size=[int(HEIGHT*.6), int(WIDTH*.6), CHANNELS], seed=seed)\n elif p_crop >= .85:\n image = tf.image.random_crop(image, size=[int(HEIGHT*.7), int(WIDTH*.7), CHANNELS], seed=seed)\n elif p_crop >= .8:\n image = tf.image.random_crop(image, size=[int(HEIGHT*.8), int(WIDTH*.8), CHANNELS], seed=seed)\n else:\n image = tf.image.random_crop(image, size=[int(HEIGHT*.9), int(WIDTH*.9), CHANNELS], seed=seed)\n image = tf.image.resize(image, size=[HEIGHT, WIDTH])\n \n ## Pixel-level transforms\n if p_pixel >= .4: # pixel transformations\n if p_pixel >= .85:\n image = tf.image.random_saturation(image, lower=0, upper=2, seed=seed)\n elif p_pixel >= .65:\n image = tf.image.random_contrast(image, lower=.8, upper=2, seed=seed)\n elif p_pixel >= .5:\n image = tf.image.random_brightness(image, max_delta=.2, seed=seed)\n else:\n image = tf.image.adjust_gamma(image, gamma=.6)\n\n return image, label\n\ndef get_training_dataset(filenames):\n dataset = load_dataset(filenames, labeled=True)\n dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n dataset = dataset.repeat() # the training dataset must repeat for several epochs\n dataset = dataset.shuffle(2048)\n# dataset = dataset.batch(BATCH_SIZE, drop_remainder=True) # slighly faster with fixed tensor sizes\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n return dataset\n\ndef get_validation_dataset(filenames, ordered=True, repeated=False):\n dataset = load_dataset(filenames, labeled=True, ordered=ordered)\n if repeated:\n dataset = dataset.repeat()\n dataset = dataset.shuffle(2048)\n# dataset = dataset.batch(BATCH_SIZE, drop_remainder=repeated)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.cache()\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef get_test_dataset(filenames=TEST_FILENAMES, ordered=True):\n dataset = load_dataset(filenames, labeled=False, ordered=ordered)\n dataset = dataset.batch(BATCH_SIZE)\n dataset = dataset.prefetch(AUTO)\n return dataset\n\ndef count_data_items(filenames):\n # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\ndef int_div_round_up(a, b):\n return (a + b - 1) // b","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Visualization utility functions\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef plot_metrics(history, metric_list):\n fig, axes = plt.subplots(len(metric_list), 1, sharex='col', figsize=(24, 12))\n axes = axes.flatten()\n \n for index, metric in enumerate(metric_list):\n axes[index].plot(history[metric], label='Train %s' % metric)\n axes[index].plot(history['val_%s' % metric], label='Validation %s' % metric)\n axes[index].legend(loc='best', fontsize=16)\n axes[index].set_title(metric)\n\n plt.xlabel('Epochs', fontsize=16)\n sns.despine()\n plt.show()\n\n \ndef dataset_to_numpy_util(dataset, N):\n dataset = dataset.unbatch().batch(N)\n for images, labels in dataset:\n numpy_images = images.numpy()\n numpy_labels = labels.numpy()\n break; \n return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n label = np.argmax(label, axis=-1)\n correct = (label == correct_label)\n return \"{} [{}{}{}]\".format(CLASSES[label], str(correct), ', shoud be ' if not correct else '',\n CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower_eval(image, title, subplot, red=False):\n plt.subplot(subplot)\n plt.axis('off')\n plt.imshow(image)\n plt.title(title, fontsize=14, color='red' if red else 'black')\n return subplot+1\n\ndef display_9_images_with_predictions(images, predictions, labels):\n subplot=331\n plt.figure(figsize=(13,13))\n for i, image in enumerate(images):\n title, correct = title_from_label_and_target(predictions[i], labels[i])\n subplot = display_one_flower_eval(image, title, subplot, not correct)\n if i >= 8:\n break;\n \n plt.tight_layout()\n plt.subplots_adjust(wspace=0.1, hspace=0.1)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## About the datasets","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Train data\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\ntrain_dataset = get_training_dataset(TRAINING_FILENAMES)\ny_train = next(iter(train_dataset.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\nprint('Number of training images %d' % NUM_TRAINING_IMAGES)\n\n# Test data\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Number of test images %d' % NUM_TEST_IMAGES)\ntest_dataset = get_test_dataset(ordered=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model(input_shape, N_CLASSES):\n base_model = efn.EfficientNetB0(weights='noisy-student', \n include_top=False,\n input_shape=input_shape)\n\n model = tf.keras.Sequential([\n base_model,\n layers.GlobalAveragePooling2D(),\n layers.Dense(N_CLASSES, activation='softmax')\n ])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Learning rate schedule","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"LR_START = 0.00000001\nLR_MIN = 0.000001\nLR_MAX = LEARNING_RATE\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\n@tf.function\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n return lr\n \nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\n\nsns.set(style=\"whitegrid\")\nfig, ax = plt.subplots(figsize=(20, 6))\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Optimized training loop","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"history_list = []\nmodel_path_list = []\nkfold = KFold(N_FOLDS, shuffle=True, random_state=seed)\nfor n_fold, (trn_ind, val_ind) in enumerate(kfold.split(TRAINING_FILENAMES)):\n if n_fold < FOLDS_USED:\n tf.tpu.experimental.initialize_tpu_system(tpu) # clear TPU memory\n print(f'##### FOLD {n_fold} #####') \n ### Data\n fold_train_filenames = np.asarray(TRAINING_FILENAMES)[trn_ind]\n fold_valid_filenames = np.asarray(TRAINING_FILENAMES)[val_ind]\n train_size = count_data_items(fold_train_filenames)\n validation_size = count_data_items(fold_valid_filenames)\n \n # distribute the datset according to the strategy\n train_dist_ds = strategy.experimental_distribute_dataset(get_training_dataset(fold_train_filenames))\n # Hitting End Of Dataset exceptions is a problem in this setup. Using a repeated validation set instead.\n # This will introduce a slight inaccuracy because the validation dataset now has some repeated elements.\n valid_dist_ds = strategy.experimental_distribute_dataset(get_validation_dataset(fold_valid_filenames, repeated=True))\n \n train_data_iter = iter(train_dist_ds) # the training data iterator is repeated and it is not reset\n # for each validation run (same as model.fit)\n valid_data_iter = iter(valid_dist_ds) # the validation data iterator is repeated and it is not reset\n # for each validation run (different from model.fit whre the \n # recommendation is to use a non-repeating validation dataset)\n # recommendation is to use a non-repeating validation dataset)\n\n STEPS_PER_TPU_CALL = STEPS_PER_EPOCH = train_size//BATCH_SIZE\n VALIDATION_STEPS_PER_TPU_CALL = validation_size//BATCH_SIZE\n \n # Step functions (must be called everytime for each fold)\n @tf.function\n def train_step(data_iter):\n def train_step_fn(images, labels):\n with tf.GradientTape() as tape:\n probabilities = model(images, training=True)\n loss = loss_fn(labels, probabilities)\n grads = tape.gradient(loss, model.trainable_variables)\n optimizer.apply_gradients(zip(grads, model.trainable_variables))\n train_accuracy.update_state(labels, probabilities)\n train_loss.update_state(loss)\n for _ in tf.range(STEPS_PER_TPU_CALL):\n strategy.experimental_run_v2(train_step_fn, next(data_iter))\n\n @tf.function\n def valid_step(data_iter):\n def valid_step_fn(images, labels):\n probabilities = model(images, training=False)\n loss = loss_fn(labels, probabilities)\n valid_accuracy.update_state(labels, probabilities)\n valid_loss.update_state(loss)\n for _ in tf.range(VALIDATION_STEPS_PER_TPU_CALL):\n strategy.experimental_run_v2(valid_step_fn, next(data_iter))\n \n ### Model\n model_path = 'model_%sx%s_fold_%d.h5' % (HEIGHT, WIDTH, (n_fold+1))\n with strategy.scope():\n model = create_model((None, None, CHANNELS), N_CLASSES)\n \n # Instiate optimizer with learning rate schedule\n class LRSchedule(optimizers.schedules.LearningRateSchedule):\n def __call__(self, step):\n return lrfn(epoch=step//STEPS_PER_EPOCH)\n optimizer = optimizers.Adam(learning_rate=LRSchedule())\n \n train_accuracy = metrics.SparseCategoricalAccuracy()\n valid_accuracy = metrics.SparseCategoricalAccuracy()\n train_loss = metrics.Sum()\n valid_loss = metrics.Sum()\n\n loss_fn = losses.sparse_categorical_crossentropy\n\n step = 0\n epoch = 0\n epoch_steps = 0\n epoch_start_time = time.time()\n history = {'loss': [], 'val_loss': [], 'sparse_categorical_accuracy': [], 'val_sparse_categorical_accuracy': []}\n \n ### Train model\n while True:\n # run training step\n train_step(train_data_iter)\n epoch_steps += STEPS_PER_TPU_CALL\n step += STEPS_PER_TPU_CALL\n\n # validation run at the end of each epoch\n if (step // STEPS_PER_EPOCH) > epoch:\n # validation run\n valid_epoch_steps = 0\n for _ in range(int_div_round_up(validation_size, BATCH_SIZE*VALIDATION_STEPS_PER_TPU_CALL)):\n valid_step(valid_data_iter)\n valid_epoch_steps += VALIDATION_STEPS_PER_TPU_CALL\n\n # compute metrics\n history['sparse_categorical_accuracy'].append(train_accuracy.result().numpy())\n history['val_sparse_categorical_accuracy'].append(valid_accuracy.result().numpy())\n history['loss'].append(train_loss.result().numpy() / (BATCH_SIZE*epoch_steps))\n history['val_loss'].append(valid_loss.result().numpy() / (BATCH_SIZE*valid_epoch_steps))\n\n # report metrics\n epoch_time = time.time() - epoch_start_time\n print('\\nEPOCH {:d}/{:d}'.format(epoch+1, EPOCHS))\n print('time: {:0.1f}s'.format(epoch_time),\n 'loss: {:0.4f}'.format(history['loss'][-1]),\n 'sparse_categorical_accuracy: {:0.4f}'.format(history['sparse_categorical_accuracy'][-1]),\n 'val_loss: {:0.4f}'.format(history['val_loss'][-1]),\n 'val_sparse_categorical_accuracy: {:0.4f}'.format(history['val_sparse_categorical_accuracy'][-1]))\n print('LearningRate: {:0.4g}'.format(lrfn(epoch)))\n\n # set up next epoch\n epoch = step // STEPS_PER_EPOCH\n epoch_steps = 0\n epoch_start_time = time.time()\n train_accuracy.reset_states()\n valid_accuracy.reset_states()\n valid_loss.reset_states()\n train_loss.reset_states()\n if epoch >= EPOCHS:\n break\n \n history_list.append(history)\n model_path_list.append(model_path)\n model.save(model_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model loss graph","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.set(style=\"whitegrid\")\nfor index, history in enumerate(history_list):\n print(f'##### FOLD {index} #####')\n plot_metrics(history, metric_list=['loss', 'sparse_categorical_accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make predictions","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Train predictions\ncomplete_dataset = get_validation_dataset(TRAINING_FILENAMES, ordered=True)\nx_complete = complete_dataset.map(lambda image, label: image)\ncomplete_preds = np.zeros((NUM_TRAINING_IMAGES, N_CLASSES))\n# Test predictions\nx_test = test_dataset.map(lambda image, idnum: image)\ntest_preds = np.zeros((NUM_TEST_IMAGES, N_CLASSES))\n\nfor model_path in model_path_list:\n print(model_path)\n ### Model\n with strategy.scope():\n model = load_model(model_path)\n \n complete_preds += model.predict(x_complete) / FOLDS_USED\n test_preds += model.predict(x_test) / FOLDS_USED\n \ncomplete_preds = np.argmax(complete_preds, axis=-1)\ntest_preds = np.argmax(test_preds, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model evaluation","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"y_complete = next(iter(complete_dataset.unbatch().map(lambda image, label: label).batch(NUM_TRAINING_IMAGES))).numpy()\nprint(classification_report(y_complete, complete_preds, target_names=CLASSES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Confusion matrix\n\nI have split confusion matrices into 3 parts to make it clearer, the first plot has classes 1 from 34, the second plot 35 to 69 and the third has the remaining ones.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(24, 45))\n\ncfn_matrix = confusion_matrix(y_complete, complete_preds, labels=range(len(CLASSES)))\ncfn_matrix = (cfn_matrix.T / cfn_matrix.sum(axis=1)).T\n\ndf_cm1 = pd.DataFrame(cfn_matrix[:34], index=CLASSES[:34], columns=CLASSES)\ndf_cm2 = pd.DataFrame(cfn_matrix[34:68], index=CLASSES[34:68], columns=CLASSES)\ndf_cm3 = pd.DataFrame(cfn_matrix[68:], index=CLASSES[68:], columns=CLASSES)\n\nsns.heatmap(df_cm1, cmap=\"Blues\", ax=ax1).set_title('Labels (1:35)', fontsize=30)\nsns.heatmap(df_cm2, cmap=\"Blues\", ax=ax2).set_title('Labels (35:69)', fontsize=30)\nsns.heatmap(df_cm3, cmap=\"Blues\", ax=ax3).set_title('Labels (69:)', fontsize=30)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize predictions","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"x_samp, y_samp = dataset_to_numpy_util(complete_dataset, 9)\nsamp_preds = model.predict(x_samp, batch_size=9)\ndisplay_9_images_with_predictions(x_samp, samp_preds, y_samp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test set predictions","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\nsubmission = pd.DataFrame(test_ids, columns=['id'])\nsubmission['label'] = test_preds\nsubmission.to_csv('submission.csv', index=False)\ndisplay(submission.head(10))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}