{"cells":[{"metadata":{},"cell_type":"markdown","source":"***Credits ***\n* [CutMix](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) Amazing kernel on CutMix\n* [All in one]( https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/199119) Amazing collection of notebooks i learned from \n* [Nature Paper](https://www.nature.com/articles/s41598-020-59108-x) A good paper\n\n* [Inference Kernel](https://www.kaggle.com/venkat555/cassava-leaf-disease-tpu-tensorflow-inference) Inference\n\nTODO\n* Confirm if the label adjustment in CutMix is correct \n* Emsemble"},{"metadata":{},"cell_type":"markdown","source":"## Setup Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Using https://www.kaggle.com/ryanholbrook/create-your-first-submission\n# as the base \n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport math, re, os\nimport tensorflow as tf\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\nfrom sklearn.model_selection import train_test_split \n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# TPU or GPU detection\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"try: # detect TPUs\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() # TPU detection\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept ValueError: # no TPU found, detect GPUs\n    strategy = tf.distribute.MirroredStrategy() # for GPU or multi-GPU machines\n    #strategy = tf.distribute.get_strategy() # default strategy that works on CPU and single GPU\n    #strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy() # for clusters of multi-GPU machines\n\nprint(\"Number of accelerators: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Competition data access\n\nTPUs read data directly from Google Cloud Storage (GCS). This Kaggle utility will copy the dataset to a GCS bucket co-located with the TPU. If you have multiple datasets attached to the notebook, you can pass the name of a specific dataset to the get_gcs_path function. The name of the dataset is the name of the directory it is mounted in. Use !ls /kaggle/input/ to list attached datasets."},{"metadata":{},"cell_type":"markdown","source":"# Configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path(\"cassava-leaf-disease-classification\") # you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\nprint( \"GCS_DS_PATH :{} \".format(GCS_DS_PATH))\nGCS_TRAIN_PATH = KaggleDatasets().get_gcs_path(\"cassava-leaf-disease-tfrecords-512x512\") # you can list the bucket with \"!gsutil ls $GCS_DS_PATH\"\nprint( \"GCS_TRAIN_PATH :{} \".format(GCS_TRAIN_PATH))\n\nIMAGE_SIZE = [512, 512] # At this size, a GPU will run out of memory. Use the TPU.\n                        # For GPU training, please select 224 x 224 px image size.\nSEED =555    \nBATCH_SIZE = 32 * strategy.num_replicas_in_sync\nAUG_BATCH = BATCH_SIZE\nFOLDS = 3\nEPOCHS = 20\nFIRST_FOLD_ONLY = False\n\n# use all tfrec files except ld_train15 as the training\n#TRAINING_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/train_tfrecords/*[!train15]*.tfrec')\n# use only tfrec file ld_train15 as the validation\n#VALIDATION_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/train_tfrecords/ld_train15*.tfrec')\n# TRAINING_FILENAMES, VALIDATION_FILENAMES = train_test_split(\n#     tf.io.gfile.glob(GCS_TRAIN_PATH + '/*.tfrec'),\n#     test_size=0.35, random_state=5\n# )../input/cassava-leaf-disease-tfrecords-512x512\ntraining_filenames = []\ntraining_filenames.append(GCS_TRAIN_PATH + '/*.tfrec')\n#testing ../input/cassava-leaf-disease-tfrecords-512x512\n#training_filenames.append(GCS_L2Y_PATH + '/merged_train_tfrecs/merged_train_tfrecs/100174*.tfrecords')\n#training_filenames.append(GCS_L2Y_PATH + '/merged_train_tfrecs/merged_train_tfrecs/*.tfrecords')\n\nTRAINING_FILENAMES = tf.io.gfile.glob(training_filenames)\nTEST_FILENAMES = tf.io.gfile.glob(GCS_DS_PATH + '/test_tfrecords/*.tfrec') # predictions on this dataset should be submitted for the competition\nprint(\" [file names] train {} , test : {}\".format(TRAINING_FILENAMES,TEST_FILENAMES))\n# inferred from \n# label_num_to_disease_map.json\nCLASSES = ['0',\n'1',\n'2',\n'3',\n'4']  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# raw_dataset = tf.data.TFRecordDataset('gs://kds-c58ccbc5333ad4d5db3a728ecc7032c4457f5af522071f79a7e5386c/merged_train_tfrecs/merged_train_tfrecs/1002088496.tfrecords')\n# sum(1 for _ in raw_dataset)\n# # for raw_record in raw_dataset.take(1):\n#   example = tf.train.Example()\n#   example.ParseFromString(raw_record.numpy())\n#   print(example)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def onehot(image,label):\n    NUMCLASSES = len(CLASSES)\n    return image,tf.one_hot(label,NUMCLASSES)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CutMix "},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\ndef cutmix(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = IMAGE_SIZE[0]\n    NUMCLASSES = len(CLASSES) \n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.round(tf.math.maximum(0,y-WIDTH//2))\n        yb = tf.math.round(tf.math.minimum(DIM,y+WIDTH//2))\n        xa = tf.math.round(tf.math.maximum(0,x-WIDTH//2))\n        xb = tf.math.round(tf.math.minimum(DIM,x+WIDTH//2))\n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        # MAKE CUTMIX LABEL\n        a = tf.cast((xb-xa)*(yb-ya)/DIM/DIM,tf.float32)\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],NUMCLASSES)\n            lab2 = tf.one_hot(label[k],NUMCLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        \n        #tf.print(\"{} {} {}\".format(a,lab1,lab2))\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,NUMCLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MixUp"},{"metadata":{"trusted":true},"cell_type":"code","source":"def mixup(image, label, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    DIM = IMAGE_SIZE[0]\n    NUMCLASSES = len(CLASSES) \n    \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n        # MAKE CUTMIX LABEL\n        if len(label.shape)==1:\n            lab1 = tf.one_hot(label[j],NUMCLASSES)\n            lab2 = tf.one_hot(label[k],NUMCLASSES)\n        else:\n            lab1 = label[j,]\n            lab2 = label[k,]\n        labs.append((1-a)*lab1 + a*lab2)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,NUMCLASSES))\n    return image2,label2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\ndef transform(image,label):\n    # THIS FUNCTION APPLIES BOTH CUTMIX AND MIXUP\n    DIM = IMAGE_SIZE[0]\n    NUMCLASSES = len(CLASSES)\n    SWITCH = 0.5\n    CUTMIX_PROB = 0.666\n    MIXUP_PROB = 0.666\n    # FOR SWITCH PERCENT OF TIME WE DO CUTMIX AND (1-SWITCH) WE DO MIXUP\n    image2, label2 = cutmix(image, label, CUTMIX_PROB)\n    image3, label3 = mixup(image, label, MIXUP_PROB)\n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        P = tf.cast( tf.random.uniform([],0,1)<=SWITCH, tf.float32)\n        imgs.append(P*image2[j,]+(1-P)*image3[j,])\n        labs.append(P*label2[j,]+(1-P)*label3[j,])\n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image4 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label4 = tf.reshape(tf.stack(labs),(AUG_BATCH,NUMCLASSES))\n    #tf.print(\"label4 :\",label4,output_stream=sys.stdout)\n    return image4,label4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization utilities\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data    \n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        #print(\"label {}\".format(label))\n        title = '' if label is None else CLASSES[np.argmax(label,axis=-1)]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dataset Transformation Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_float32_2(image, label):\n    max_val = tf.reduce_max(label, axis=-1,keepdims=True)\n    cond = tf.equal(label, max_val)\n    label = tf.where(cond, tf.ones_like(label), tf.zeros_like(label))\n    return tf.cast(image, tf.float32), tf.cast(label, tf.int32)\n\ndef to_float32(image, label):\n    return tf.cast(image, tf.float32), label\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\" : tf.io.FixedLenFeature([], tf.string , default_value='deadbeef.jpg' ),\n        \"target\": tf.io.FixedLenFeature([], tf.int64 , default_value=None),  # shape [] means single element\n        #\"label\": tf.io.FixedLenFeature([], tf.int64 , default_value=None) \n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    image_name = example['image_name']\n\n    #label= tf.cast(example['label'], tf.int32) \n    label= tf.cast(example['target'], tf.int32) \n\n    return image,label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    image_name = example['image_name']\n    return image, image_name # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    # RandomCrop, VFlip, HFilp, RandomRotate\n    image = tf.image.rot90(image,k=np.random.randint(4))\n    image = tf.image.random_flip_left_right(image , seed=SEED)\n    image=  image = tf.image.random_flip_up_down(image, seed=SEED)\n    IMG_SIZE=IMAGE_SIZE[0]\n    # Add 6 pixels of padding\n    image = tf.image.resize_with_crop_or_pad(image, IMG_SIZE + 6, IMG_SIZE + 6) \n    # Random crop back to the original size\n    image = tf.image.random_crop(image, size=[IMG_SIZE, IMG_SIZE, 3])\n    image = tf.image.random_brightness(image, max_delta=0.5) # Random brightness\n    image = tf.image.random_saturation(image, 0, 2, seed=SEED)\n    image = tf.image.adjust_saturation(image, 3)\n    \n    #image = tf.image.central_crop(image, central_fraction=0.5)\n    return image, label   \n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_training_dataset(dataset, do_aug=True):\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.batch(AUG_BATCH)\n    if do_aug: dataset = dataset.map(transform, num_parallel_calls=AUTO) # note we put AFTER batching\n    dataset = dataset.unbatch()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    #the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n    #c = 0\n    #for filename in filenames:\n    #    c += sum(1 for _ in tf.data.TFRecordDataset(filename))\n    #return c\nprint( 'Training filenames {}'.format(TRAINING_FILENAMES))\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint(\"STEPS_PER_EPOCH {}\".format(STEPS_PER_EPOCH))\n#NUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\n#print('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\nprint('Dataset: {} training images,  {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data dump\nprint(\"Training data shapes:\")\ntraining_raw_dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\nfor image, label in get_training_dataset(training_raw_dataset, do_aug=False).take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())\n#print(\"Validation data shapes:\")\n#for image, label in get_validation_dataset().take(3):\n#    print(image.numpy().shape, label.numpy().shape)\n#print(\"Validation data label examples:\", label.numpy())\nprint(\"Test data shapes:\")\nfor image, image_name in get_test_dataset().take(3):\n    print(image.numpy().shape )\n#print(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling "},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras import regularizers\n\n\ndef get_model():\n    lr_scheduler = keras.optimizers.schedules.ExponentialDecay(\n        initial_learning_rate=1e-5, \n        decay_steps=10000, \n        decay_rate=0.9)\n\n    with strategy.scope():\n        #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.xception.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n        #pretrained_model = tf.keras.applications.Xception(weights='imagenet', include_top=False)\n\n        #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.vgg16.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n        #pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)\n\n        #img_adjust_layer = tf.keras.layers.Lambda(tf.keras.applications.resnet50.preprocess_input, input_shape=[*IMAGE_SIZE, 3])\n        #pretrained_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)\n        pretrained_model = tf.keras.applications.DenseNet201(input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3),weights='imagenet', include_top=False)\n\n        #pretrained_model.trainable = False # False = transfer learning, True = fine-tuning\n        pretrained_model.trainable = True\n\n\n        model = tf.keras.Sequential([\n            #img_adjust_layer,\n            pretrained_model,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.Dropout(0.5),\n            tf.keras.layers.Dense(1024,activation='relu'  ),\n            tf.keras.layers.Dense(512,activation='relu'  ),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.Dropout(0.5),\n            tf.keras.layers.Dense(len(CLASSES), activation='softmax',dtype='float32')\n        ])\n\n\n    custom_loss = tf.keras.losses.CategoricalCrossentropy( from_logits=False,\n                                                   label_smoothing=0.1, name='categorical_crossentropy' ) \n    opt = tf.keras.optimizers.Adam(lr=0.00001) \n    model.compile(\n        optimizer=opt,\n        loss = custom_loss,\n        metrics=['categorical_accuracy']\n    )\n    return model\n\nbase_model = get_model()\n# Let's take a look to see how many layers are in the base model\nprint(\"Number of layers in the base model: \", len(base_model.layers))\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fine-tune from this layer onwards\nfine_tune_at = -8\n\n# Freeze all the layers before the `fine_tune_at` layer\nfor layer in base_model.layers[:fine_tune_at]:\n  layer.trainable =  False\n\n# Freeze all the layers before the `fine_tune_at` layer\nfor layer in base_model.layers[fine_tune_at:]:\n  layer.trainable =  True\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's take a look to see how many layers are in the base model\nprint(\"Number of layers in the base model: \", len(base_model.layers))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_model.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_validation_dataset_for_kfold(dataset, do_onehot=True):\n    dataset = dataset.batch(BATCH_SIZE)\n    if do_onehot: dataset = dataset.map(onehot, num_parallel_calls=AUTO) # we must use one hot like augmented train data\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run train and predict\n# FOLDS = 3  \n# FIRST_FOLD_ONLY = True\n# EPOCHS=10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import KFold\n\ndef train_cross_validate(folds = 5):\n    histories = []\n    models = []\n    # Define per-fold score containers\n    acc_per_fold = []\n    loss_per_fold = []\n    early_stopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 10)\n    kfold = KFold(folds, shuffle = True, random_state = SEED)\n    for f, (trn_ind, val_ind) in enumerate(kfold.split(TRAINING_FILENAMES)):\n        print(); print('#'*25)\n        print('### FOLD',f+1)\n        print('#'*25)\n        train_dataset = load_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[trn_ind]['TRAINING_FILENAMES']), labeled = True)\n        val_dataset = load_dataset(list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[val_ind]['TRAINING_FILENAMES']), labeled = True, ordered = True)\n        model = base_model\n        data_for_validation = get_validation_dataset_for_kfold(val_dataset)\n        history = model.fit(\n            get_training_dataset(train_dataset),\n            steps_per_epoch = STEPS_PER_EPOCH,\n            epochs = EPOCHS,\n            callbacks = [lr_callback],# early_stopping],\n            validation_data = data_for_validation,\n            verbose=2\n        )\n        scores = model.evaluate(data_for_validation, verbose=0)\n        #print(scores)\n        print(f'Score for fold {f+1}: {model.metrics_names[0]} of {scores[0]}')\n        acc_per_fold.append(scores[1] * 100)\n        loss_per_fold.append(scores[0])\n        model.save(\"model-fold{}.h5\".format(f+1))\n        models.append(model)\n        histories.append(history)\n        if FIRST_FOLD_ONLY: break\n    return histories, models,acc_per_fold, loss_per_fold\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_and_predict(folds = 5):\n    test_ds = get_test_dataset(ordered=True) #map(data_augment, num_parallel_calls=AUTO) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n    test_images_ds = test_ds.map(lambda image, idnum: image)\n    print('Start training %i folds'%folds)\n    histories, models,acc_per_fold,loss_per_fold  = train_cross_validate(folds = folds)\n    # == Provide average scores ==\n    print('------------------------------------------------------------------------')\n    print('Score per fold')\n    for i in range(0, len(acc_per_fold)):\n        print('------------------------------------------------------------------------')\n        print(f'> Fold {i+1} - Loss: {loss_per_fold[i]} - Accuracy: {acc_per_fold[i]}%')\n    print('------------------------------------------------------------------------')\n    print('Average scores for all folds:')\n    print(f'> Accuracy: {np.mean(acc_per_fold)} (+- {np.std(acc_per_fold)})')\n    print(f'> Loss: {np.mean(loss_per_fold)}')\n    print('------------------------------------------------------------------------')\n    \n    print('Computing predictions...')\n    # get the mean probability of the folds models\n    if FIRST_FOLD_ONLY: probabilities = np.average([models[i].predict(test_images_ds) for i in range(1)], axis = 0)\n    else: probabilities = np.average([models[i].predict(test_images_ds) for i in range(folds)], axis = 0)\n    predictions = np.argmax(probabilities, axis=-1)\n    print('Generating submission.csv file...')\n    test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n    np.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\n    return histories, models\n    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train and Predict"},{"metadata":{"trusted":true},"cell_type":"code","source":"histories, models = train_and_predict(folds = FOLDS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for history in histories : \n    #print(history)\n    display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211)\n    display_training_curves(history.history['categorical_accuracy'], history.history['val_categorical_accuracy'], 'accuracy', 212)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Confusion matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nall_labels = []; all_prob = []; all_pred = []\nkfold = KFold(FOLDS, shuffle = True, random_state = SEED)\nfor j, (trn_ind, val_ind) in enumerate( kfold.split(TRAINING_FILENAMES) ):\n    print('Inferring fold',j+1,'validation images...')\n    VAL_FILES = list(pd.DataFrame({'TRAINING_FILENAMES': TRAINING_FILENAMES}).loc[val_ind]['TRAINING_FILENAMES'])\n    NUM_VALIDATION_IMAGES = count_data_items(VAL_FILES)\n    cmdataset = get_validation_dataset_for_kfold(load_dataset(VAL_FILES, labeled = True, ordered = True), do_onehot=False)\n    images_ds = cmdataset.map(lambda image, label: image)\n    labels_ds = cmdataset.map(lambda image, label: label).unbatch()\n    all_labels.append( next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() ) # get everything as one batch\n    prob = models[j].predict(images_ds)\n    all_prob.append( prob )\n    all_pred.append( np.argmax(prob, axis=-1) )\n    if FIRST_FOLD_ONLY: break\ncm_correct_labels = np.concatenate(all_labels)\ncm_probabilities = np.concatenate(all_prob)\ncm_predictions = np.concatenate(all_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visual Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# dataset = get_validation_dataset()\n# dataset = dataset.unbatch().batch(100)\n# batch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # run this cell again for next set of images\n# images,labels = next(batch)\n# probabilities = model.predict(tf.cast(images, tf.float32))\n# predictions = np.argmax(probabilities, axis=-1)\n# display_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}