{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Warning\n\n====================================================================================================<br>\nIMPORTANT UPDATE May 7, 2020:<br>\nKaggle has disallowed the use of the following 5 external datasets: ImageNet, Oxford 102 Category Flowers, TF Flowers, Open Images, and iNaturalist. Kaggle has added the following new rule:<br>\n<br>\n**Training on any examples included in the test set will result in disqualification.**<br>\n<br>\nDo not select a final submission that trains on these 5 datasets.<br>\nConsidering code below has no reference to these 5 image datasets, they could be used to only refer to how multiple kernel outputs can be used together. Having said that apparently there are better ways to connect output from multiple kernel outputs. Possibly adding the outputs to your own custom dataset and use it for connecting them together.<br>\n<br>\n====================================================================================================<br>\nRefer to the discussions specifically the [point noted by Kaggle Team member](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329#836349)\n<br>"},{"metadata":{},"cell_type":"markdown","source":"## Description\n[Version 11 of this kernel](https://www.kaggle.com/haveri/flowerflowerwhoareyou-onlysubmissions-ensembling?scriptVersionId=32998379) ensembles outputs of 4 other kernels to create a submission for this competition. The 4 kernels used are<br>\nS1 - [EfficientNet-With-All-5-Imagesets-S1](https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1?scriptVersionId=32838132). Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00005 * num_replicas, LR_EXP_DECAY = 0.8. Epochs = 50.<br>\nS2 - Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00005 * num_replicas, LR_EXP_DECAY = 0.75. Epochs = 50.<br>\nS3 - 1 DenseNet201. Images of [224,224], LR_MAX = 0.00005 * num_replicas, LR_EXP_DECAY = 0.8. Epochs = 50.<br>\nS4 - 1 DenseNet201. Images of [224,224], LR_MAX = 0.00005 * num_replicas, LR_EXP_DECAY = 0.75. Epochs = 50.<br>\n<p>\n[Version 14 of this kernel](https://www.kaggle.com/haveri/flowerflowerwhoareyou-onlysubmissions-ensembling?scriptVersionId=33348004) ensembles outputs of 15 other kernels to create a submission for this competition. These 15 kernels used are<br>\nS2 - Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.75. Epochs = 50.<br>\nS3 - Uses 1 DenseNet201. Images of [224,224], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.8. Epochs = 50.<br>\nS4 - Uses 1 DenseNet201. Images of [224,224], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.75. Epochs = 50.<br>\nS5 - Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 50.<br>\nS6 - Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.90, EPOCHS = 50.<br>\nS7 - Uses 1 DenseNet201. Images of [224,224], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 50.<br>\nS8 - Uses 1 DenseNet201. Images of [224,224], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.90, EPOCHS = 50.<br>\nS9 - Uses 1 EfficientNetB7. Images of [224,224], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.8. Epochs = 50.<br>\nS10 - Uses 1 DenseNet201. Images of [224,224], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.8. Epochs = 50.<br>\nS11 - Uses 1 EfficientNetB7. Images of [512,512], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 16.<br>\nS12 - Uses 1 DenseNet201. Images of [512,512], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 25.<br>\nS13 - Uses 1 EfficientNetB7. Images of [331,331], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 30.<br>\nS14 - Uses 1 DenseNet201. Images of [331,331], LR_MAX = 0.00004 \\* num_replicas, LR_EXP_DECAY = 0.85, EPOCHS = 40.<br>\nS15 - Uses 1 EfficientNetB7. Images of [331,331], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.80, EPOCHS = 35.<br>\nS16 - Uses 1 DenseNet201. Images of [331,331], LR_MAX = 0.00005 \\* num_replicas, LR_EXP_DECAY = 0.80, EPOCHS = 45.<br>\n<p>\nVersion 16 of this kernel. Adding more description/explanation.<br>\n"},{"metadata":{},"cell_type":"markdown","source":"## Imports and Initialization\nRefer to [Getting started code](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu)"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import math, re, gc\nimport numpy as np # linear algebra\nimport pickle\nfrom datetime import datetime, timedelta\nimport tensorflow as tf\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint('TensorFlow version', tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The original code incorrectly used \"!ls -ltr \\$GCS_DS_PATH\". Instead you should use \"!gsutil ls \\$GCS_PATH\". Refer to [Five flowers train save and reload on TPU](https://www.kaggle.com/mgornergoogle/five-flowers-train-save-and-reload-on-tpu)"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint('Replicas:', strategy.num_replicas_in_sync)\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('flower-classification-with-tpus')\nprint(GCS_DS_PATH)\n!gsutil ls -l $GCS_DS_PATH\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"start_time = datetime.now()\nprint('Time now is', start_time)\nend_training_by_tdelta = timedelta(seconds=8400)\nthis_run_file_prefix = start_time.strftime('%Y%m%d_%H%M_')\nprint(this_run_file_prefix)\n\nIMAGE_SIZE = [224, 224] # [512, 512]\n\nEPOCHS = 12\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n\nGCS_PATH_SELECT = {\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec')\n\nCLASSES = ['pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea', 'wild geranium', 'tiger lily', 'moon orchid', 'bird of paradise', 'monkshood', 'globe thistle', # 00 - 09\n           'snapdragon', \"colt's foot\", 'king protea', 'spear thistle', 'yellow iris', 'globe-flower', 'purple coneflower', 'peruvian lily', 'balloon flower', 'giant white arum lily', # 10 - 19\n           'fire lily', 'pincushion flower', 'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy', 'prince of wales feathers', 'stemless gentian', 'artichoke', 'sweet william', # 20 - 29\n           'carnation', 'garden phlox', 'love in the mist', 'cosmos', 'alpine sea holly', 'ruby-lipped cattleya', 'cape flower', 'great masterwort', 'siam tulip', 'lenten rose', # 30 - 39\n           'barberton daisy', 'daffodil', 'sword lily', 'poinsettia', 'bolero deep blue', 'wallflower', 'marigold', 'buttercup', 'daisy', 'common dandelion', # 40 - 49\n           'petunia', 'wild pansy', 'primula', 'sunflower', 'lilac hibiscus', 'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', 'pink-yellow dahlia', # 50 - 59\n           'cautleya spicata', 'japanese anemone', 'black-eyed susan', 'silverbush', 'californian poppy', 'osteospermum', 'spring crocus', 'iris', 'windflower', 'tree poppy', # 60 - 69\n           'gazania', 'azalea', 'water lily', 'rose', 'thorn apple', 'morning glory', 'passion flower', 'lotus', 'toad lily', 'anthurium', # 70 - 79\n           'frangipani', 'clematis', 'hibiscus', 'columbine', 'desert-rose', 'tree mallow', 'magnolia', 'cyclamen ', 'watercress', 'canna lily', # 80 - 89\n           'hippeastrum ', 'bee balm', 'pink quill', 'foxglove', 'bougainvillea', 'camellia', 'mallow', 'mexican petunia', 'bromelia', 'blanket flower', # 90 - 99\n           'trumpet creeper', 'blackberry lily', 'common tulip', 'wild rose'] # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels = 3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n#\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'class': tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label\n#\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'id': tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum\n#\n\ndef load_dataset(filenames, labeled = True, ordered = False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads = AUTO)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls = AUTO)\n    return dataset\n#\n\ndef data_augment(image, label):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_saturation(image, 0, 2)\n    return image, label\n#\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled = True)\n    dataset = dataset.map(data_augment, num_parallel_calls = AUTO)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n#\n\ndef get_validation_dataset(ordered = False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled = True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n#\n\ndef get_test_dataset(ordered = False):\n    dataset = load_dataset(TEST_FILENAMES, labeled = False, ordered = ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n#\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n#\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered = True)\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ds = get_test_dataset(ordered = True)\n\ntest_images_ds = test_ds.map(lambda image, idnum: image)\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Alternative Implementation\nIf you train a model using TPU and reload the same model in another kernel to make your predictions, [there are some issues as discussed here](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148615). To overcome this you need to make a copy of the model into CPU and save it before [loading it in another kernel as explained here](https://www.kaggle.com/mgornergoogle/five-flowers-train-save-and-reload-on-tpu).<br>\n<p>\nI had myself faced this issue and so was saving certain information in a format that did not depend on the model. Information saved in kernel duing training the model included as can be seen in [EfficientNet-With-All-5-Imagesets-S1](https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1?scriptVersionId=32838132).<br>\n* cm_correct_labels\n* cm_predictions\n* test_ids\n* val_probabilities\n* test_probabilities"},{"metadata":{},"cell_type":"markdown","source":"## Multiple Kernels v/s Using a Dataset\nIt may help using a private dataset to accumulate kernel outputs at a single place instead of using multiple kernels to ensemble. When I started I could only think of using multiple kernels. Maybe explore using a dataset and update this section."},{"metadata":{},"cell_type":"markdown","source":"## Retrieve Outputs from 4 Trained Kernels\nThe format of the information stored during training of the model helps ensemble multiple kernel outputs irrespective of the size of images used. Usually if you had trained a model, saved it and retrived it for ensembling you would need a lot more complex code to combine/ensemble models trained using different image sizes. But storing just the validation and test probabilities allows combining multiple models irrespective of the resolution of images used for training."},{"metadata":{"trusted":true},"cell_type":"code","source":"start_time = datetime.now()\nthis_run_file_prefix = start_time.strftime('%Y%m%d_%H%M_')\ntest_vals_pickle_outputs = ['/kaggle/input/efficientnet-with-all-5-imagesets-s2/20200428_0931_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s3/20200428_1755_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s4/20200429_0633_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s5/20200430_1549_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s6/20200430_1554_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s7/20200430_1821_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s8/20200430_1826_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s9/20200501_0701_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s10/20200501_0705_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s11/20200502_0517_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s12/20200502_0810_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s13/20200504_1036_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s14/20200504_1038_tests_vals_0.pkl', '/kaggle/input/efficientnet-with-all-5-imagesets-s15/20200505_0235_tests_vals_0.pkl', '/kaggle/input/densenet-with-all-5-imagesets-s16/20200505_0237_tests_vals_0.pkl']\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_results_from_pickle(filename):\n    pklfile = open(filename, 'rb')\n    test_results = pickle.load(pklfile)\n    pklfile.close()\n    return test_results\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading the information gathered during training models saved in the pickle file. Both predictions for validation image set and test image set are done after we get them using an ordered dataset. So the order is believed to be correct. But it has been noticed that if you train 2 different models each using a GPU and the other using a TPU, the order will not match because of the fact that strategy.num_replicas_in_sync for both are not same."},{"metadata":{"trusted":true},"cell_type":"code","source":"no_of_models = len(test_vals_pickle_outputs)\ncm_correct_labels_results = [0] * no_of_models\ncm_predictions_results = [0] * no_of_models\ntest_ids_results = [0] * no_of_models\nval_probabilities = [0] * no_of_models\ntest_probabilities = [0] * no_of_models\ntest_predictions = [0] * no_of_models\n#\nfor j in range(no_of_models):\n    test_results = get_results_from_pickle(test_vals_pickle_outputs[j])\n    [cm_correct_labels_results[j], cm_predictions_results[j], test_ids_results[j], val_probabilities[j], test_probabilities[j]] = test_results\n    test_predictions[j] = np.argmax(test_probabilities[j], axis = -1)\n#\nfor j in range(no_of_models):\n    cm_labels_compare = cm_correct_labels == cm_correct_labels_results[j]\n    test_ids_compare = test_ids == test_ids_results[j]\n    if not cm_labels_compare.all():\n        print('Some probelm with cm_correct_labels_results in', j)\n    if not test_ids_compare.all():\n        print('Some probelm with test_ids_results in', j)\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ensembling multiple model results helps you get better overall prediction results. The ensembling could be a simple one by adding all probability results for the different classes. Or you could try to find the optimum combination by finding the best alpha to use to ensemble the different models [as explained here](https://www.kaggle.com/wrrosa/tpu-enet-b7-densenet/output) under the section \"Finding best alpha\". This kernel provides ready code to combine two/three models. For finding best alpha for more than three models you will need to change them."},{"metadata":{"trusted":true},"cell_type":"code","source":"cm_probabilities = np.zeros((val_probabilities[0].shape)) # = val_probabilities[0] + val_probabilities[1] + val_probabilities[2]\nfor j in range(no_of_models):\n    cm_probabilities = cm_probabilities + val_probabilities[j]\n\ncm_predictions = np.argmax(cm_probabilities, axis = -1)\nprint('Correct labels: ', cm_correct_labels_results[0].shape, cm_correct_labels_results[0])\nprint('Predicted labels: ', cm_predictions.shape, cm_predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getFitPrecisionRecall(correct_labels, predictions):\n    score = f1_score(correct_labels, predictions, labels = range(len(CLASSES)), average = 'macro')\n    precision = precision_score(correct_labels, predictions, labels = range(len(CLASSES)), average = 'macro')\n    recall = recall_score(correct_labels, predictions, labels = range(len(CLASSES)), average = 'macro')\n    return score, precision, recall\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for j in range(no_of_models):\n    model_predictions = np.argmax(val_probabilities[j], axis = -1)\n    score, precision, recall = getFitPrecisionRecall(cm_correct_labels_results[0], model_predictions)\n    print('For model: {}, f1 score: {:.4f}, precision: {:.4f}, recall: {:.4f}'.format(j, score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cmat = confusion_matrix(cm_correct_labels_results[0], cm_predictions, labels = range(len(CLASSES)))\nscore, precision, recall = getFitPrecisionRecall(cm_correct_labels_results[0], cm_predictions)\ncmat = (cmat.T / cmat.sum(axis = -1)).T\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint('f1 score: {:.6f}, precision: {:.6f}, recall: {:.6f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_submission_file_not_the_right_way(filename, probabilities, test_ids):\n    predictions = np.argmax(probabilities, axis = -1)\n    print('Generating submission file...', filename)\n    np.savetxt(filename, np.rec.fromarrays([test_ids, predictions]), fmt = ['%s', '%d'], delimiter = ',', header = 'id,label', comments = '')\n#\n#\ndef create_submission_file(filename, probabilities):\n    predictions = np.argmax(probabilities, axis = -1)\n    print('Generating submission file...', filename)\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n\n    np.savetxt(filename, np.rec.fromarrays([test_ids, predictions]), fmt = ['%s', '%d'], delimiter = ',', header = 'id,label', comments = '')\n#\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def combine_two(correct_labels, probability_0, probability_1):\n    print('Start. ', datetime.now())\n    alphas0_to_try = np.linspace(0, 1, 101)\n    best_score = -1\n    best_alpha0 = -1\n    best_alpha1 = -1\n    best_precision = -1\n    best_recall = -1\n    best_val_predictions = None\n\n    for alpha0 in alphas0_to_try:\n        alpha1 = 1.0 - alpha0\n        probabilities = alpha0 * probability_0 + alpha1 * probability_1 #\n        predictions = np.argmax(probabilities, axis = -1)\n\n        score, precision, recall = getFitPrecisionRecall(correct_labels, predictions)\n        if score > best_score:\n            best_alpha0 = alpha0\n            best_alpha1 = alpha1\n            best_score = score\n            best_precision = precision\n            best_recall = recall\n            best_val_predictions = predictions\n    #\n    return best_alpha0, best_alpha1, best_val_predictions, best_score, best_precision, best_recall","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def combine_three(correct_labels, probability_0, probability_1, probability_2):\n    print('Start. ', datetime.now())\n    alphas0_to_try = np.linspace(0, 1, 101)\n    alphas1_to_try = np.linspace(0, 1, 101)\n    best_score = -1\n    best_alpha0 = -1\n    best_alpha1 = -1\n    best_alpha2 = -1\n    best_precision = -1\n    best_recall = -1\n    best_val_predictions = None\n\n    for alpha0 in alphas0_to_try:\n        for alpha1 in alphas1_to_try:\n            if (alpha0 + alpha1) > 1.0:\n                break\n\n            alpha2 = 1.0 - alpha0 - alpha1\n            probabilities = alpha0 * probability_0 + alpha1 * probability_1 + alpha2 * probability_2\n            predictions = np.argmax(probabilities, axis = -1)\n\n            score, precision, recall = getFitPrecisionRecall(correct_labels, predictions)\n            if score > best_score:\n                best_alpha0 = alpha0\n                best_alpha1 = alpha1\n                best_alpha2 = alpha2\n                best_score = score\n                best_precision = precision\n                best_recall = recall\n                best_val_predictions = predictions\n    #\n    return best_alpha0, best_alpha1, best_alpha2, best_val_predictions, best_score, best_precision, best_recall","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Submission from Retrieved Test Probabilities\nThis is done using simple ensembling without finding the best alpha to combine multiple models."},{"metadata":{"trusted":true},"cell_type":"code","source":"probabilities = np.zeros((test_probabilities[0].shape)) # = test_probabilities[0] + test_probabilities[1] + test_probabilities[2]\nfor j in range(no_of_models):\n    probabilities = probabilities + test_probabilities[j]\n\ncreate_submission_file('submission.csv', probabilities)\n#create_submission_file_not_the_right_way('submission.csv', probabilities, test_ids_results[0])\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While the earlier version of this kernel used the functions combine_two, combine_three and get_best_combination while combining outputs from multiple models, version 11 does not use it."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_best_combination(no_models, cm_correct_labels, val_probabilities, test_probabilities):\n    best_fit_score = -10000.0\n    best_predictions = 0\n    choose_filename = ''\n\n    curr_predictions = np.argmax(val_probabilities[0], axis = -1)\n    score, precision, recall = getFitPrecisionRecall(cm_correct_labels, curr_predictions)\n    print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))\n    filename = this_run_file_prefix + 'submission_0.csv'\n    if best_fit_score < score:\n        best_fit_score = score\n        best_predictions = curr_predictions\n        choose_filename = filename\n        create_submission_file('./submission.csv', test_probabilities[0])\n    create_submission_file(filename, test_probabilities[0])\n\n    if no_models > 1:\n        curr_predictions = np.argmax(val_probabilities[1], axis = -1)\n        score, precision, recall = getFitPrecisionRecall(cm_correct_labels, curr_predictions)\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))\n        filename = this_run_file_prefix + 'submission_1.csv'\n        if best_fit_score < score:\n            best_fit_score = score\n            best_predictions = curr_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', test_probabilities[1])\n        create_submission_file(filename, test_probabilities[1])\n\n    if no_models > 2:\n        curr_predictions = np.argmax(val_probabilities[2], axis = -1)\n        score, precision, recall = getFitPrecisionRecall(cm_correct_labels, curr_predictions)\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))\n        filename = this_run_file_prefix + 'submission_2.csv'\n        if best_fit_score < score:\n            best_fit_score = score\n            best_predictions = curr_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', test_probabilities[2])\n        create_submission_file(filename, test_probabilities[2])\n\n    if no_models > 1:\n        best_alpha0, best_alpha1, best_val_predictions, best_score, best_precision, best_recall = combine_two(cm_correct_labels, val_probabilities[0], val_probabilities[1])\n        print('For indx', [0, 1], 'best_alpha0:', best_alpha0, 'best_alpha1:', best_alpha1, '. ', datetime.now())\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(best_score, best_precision, best_recall))\n        combined_probabilities = best_alpha0 * test_probabilities[0] + best_alpha1 * test_probabilities[1]\n        filename = this_run_file_prefix + 'submission_01.csv'\n        if best_fit_score < best_score:\n            best_fit_score = best_score\n            best_predictions = best_val_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', combined_probabilities)\n        create_submission_file(filename, combined_probabilities)\n\n    if no_models > 2:\n        best_alpha0, best_alpha1, best_val_predictions, best_score, best_precision, best_recall = combine_two(cm_correct_labels, val_probabilities[0], val_probabilities[2])\n        print('For indx', [0, 2], 'best_alpha0:', best_alpha0, 'best_alpha1:', best_alpha1, '. ', datetime.now())\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(best_score, best_precision, best_recall))\n        combined_probabilities = best_alpha0 * test_probabilities[0] + best_alpha1 * test_probabilities[2]\n        filename = this_run_file_prefix + 'submission_02.csv'\n        if best_fit_score < best_score:\n            best_fit_score = best_score\n            best_predictions = best_val_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', combined_probabilities)\n        create_submission_file(filename, combined_probabilities)\n\n        best_alpha0, best_alpha1, best_val_predictions, best_score, best_precision, best_recall = combine_two(cm_correct_labels, val_probabilities[1], val_probabilities[2])\n        print('For indx', [1, 2], 'best_alpha0:', best_alpha0, 'best_alpha1:', best_alpha1, '. ', datetime.now())\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(best_score, best_precision, best_recall))\n        combined_probabilities = best_alpha0 * test_probabilities[1] + best_alpha1 * test_probabilities[2]\n        filename = this_run_file_prefix + 'submission_12.csv'\n        if best_fit_score < best_score:\n            best_fit_score = best_score\n            best_predictions = best_val_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', combined_probabilities)\n        create_submission_file(filename, combined_probabilities)\n\n        best_alpha0, best_alpha1, best_alpha2, best_val_predictions, best_score, best_precision, best_recall = combine_three(cm_correct_labels, val_probabilities[0], val_probabilities[1], val_probabilities[2])\n        print('For indx', [0, 1, 2], 'best_alpha0:', best_alpha0, 'best_alpha1:', best_alpha1, 'best_alpha2:', best_alpha2, '. ', datetime.now())\n        print('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(best_score, best_precision, best_recall))\n        combined_probabilities = best_alpha0 * test_probabilities[0] + best_alpha1 * test_probabilities[1] + best_alpha2 * test_probabilities[2]\n        filename = this_run_file_prefix + 'submission_012.csv'\n        if best_fit_score < best_score:\n            best_fit_score = best_score\n            best_predictions = best_val_predictions\n            choose_filename = filename\n            create_submission_file('./submission.csv', combined_probabilities)\n        create_submission_file(filename, combined_probabilities)\n#\n    cmat = confusion_matrix(cm_correct_labels, best_predictions, labels = range(len(CLASSES)))\n    cmat = (cmat.T / cmat.sum(axis = -1)).T\n    display_confusion_matrix(cmat, score, precision, recall)\n#\n    print('Best score from all combination was', best_fit_score, '. For submission file used is', choose_filename)\n    return best_fit_score, best_predictions\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_predictions = cm_predictions\nrun_this = False\nif no_of_models > 1 and run_this:\n    bp = get_best_combination(no_of_models, cm_correct_labels_results[0], val_probabilities, test_probabilities)\n#    bp = get_best_combination(no_of_models, cm_correct_labels, val_probabilities, test_probabilities)\n    best_predictions = bp\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#images_ds_unbatched = images_ds.unbatch()\n#cm_images_ds_numpy = next(iter(images_ds_unbatched.batch(NUM_VALIDATION_IMAGES))).numpy()\nuse_correct_labels = cm_correct_labels_results[0]\nuse_val_predictions = best_predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#print('type of labels_ds is {}'.format(type(labels_ds)))\nprint('type of use_val_predictions is {}. shape of use_val_predictions is {}'.format(type(use_val_predictions), use_val_predictions.shape))\n#print('type of use_correct_labels is {}, cm_images_ds_numpy is {}'.format(type(use_correct_labels), type(cm_images_ds_numpy)))\n#print('shape of use_correct_labels is {}, cm_images_ds_numpy is {}'.format(use_correct_labels.shape, cm_images_ds_numpy.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Additional Statistics\nWhile the F1 score gives a good indication of how your model performs with the validation images, this gives you the exact number of correct predictions for validation image set. The kernel [EfficientNet-With-All-5-Imagesets-S1](https://www.kaggle.com/haveri/efficientnet-with-all-5-imagesets-s1?scriptVersionId=32838132) at the very end, also demonstrates how you can create the statistics and save in a csv file. It can then be used offline to see which class of images are adversely affecting your overall performance. Since they are the ones which will bring down your overall score you could use the information to augument only images for that class. It will help keeping the total images to train lower and bring in more benefits to the model by improving performance for that class id. I leave this exercise for others to try something I myself couldn't get to."},{"metadata":{"trusted":true},"cell_type":"code","source":"correct_labels_cnt = 0\nincorrect_labels_cnt = 0\ncorrect_labels = []\nincorrect_labels = []\nvals_actual_true = {}\nvals_tp = {}\nvals_fn = {}\nvals_fp = {}\nfor i in range(len(CLASSES)):\n    vals_actual_true[i] = 0\n    vals_tp[i] = 0\n    vals_fn[i] = 0\n    vals_fp[i] = 0\n\nfor i in range(len(use_correct_labels)):\n    correct_label = use_correct_labels[i]\n    predict_label = use_val_predictions[i]\n    vals_actual_true[correct_label] = vals_actual_true[correct_label] + 1\n    if use_val_predictions[i] != use_correct_labels[i]:\n        incorrect_labels_cnt = incorrect_labels_cnt + 1\n        incorrect_labels.append(i)\n        vals_fn[correct_label] = vals_fn[correct_label] + 1\n        vals_fp[predict_label] = vals_fp[predict_label] + 1\n    else:\n        correct_labels_cnt = correct_labels_cnt + 1\n        correct_labels.append(i)\n        vals_tp[correct_label] = vals_tp[correct_label] + 1\n#        print(i)\n#\nprint('Number of correct_labels is {}, incorrect_labels is {}'.format(correct_labels_cnt, incorrect_labels_cnt))\n#print('Correct labels', correct_labels)\nprint('Incorrect labels', incorrect_labels)\n#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## What Next\nPossibly create more similar models and ensemble all of them together to improve the score. While the models used for training used images of [224, 224] size possibly train models using larger images and ensemble them together with the smaller images."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":4}