{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Cassava Leaf Disease Classification\n\nAs the second-largest provider of carbohydrates in Africa, `cassava` is a key food security crop grown by smallholder farmers because it can withstand harsh conditions. At least 80% of household farms in Sub-Saharan Africa grow this starchy root, but `viral diseases` are major sources of poor yields. With the help of data science, it may be possible to identify common diseases so they can be treated.\n\nExisting methods of `disease detection` require farmers to solicit the help of government-funded agricultural experts to visually inspect and diagnose the plants. This suffers from being labor-intensive, low-supply and costly. As an `added challenge`, effective solutions for farmers must perform well under significant constraints, since African farmers may only have access to mobile-quality cameras with low-bandwidth.\n\nThis notebook is for the Kaggle's [Cassava Leaf Disease Classification](https://www.kaggle.com/c/cassava-leaf-disease-classification) competition where we have to identify the type of disease present on a Cassava Leaf image.\n\n![](https://media.giphy.com/media/JYc3Q2iiHgayI/giphy.gif)\n\n**More info about the competition**\n\nIn this competition, we introduce a dataset of `21,367 labeled images` collected during a regular survey in Uganda. Most images were `crowdsourced` from farmers taking `photos` of their gardens, and `annotated by experts` at the National Crops Resources Research Institute (NaCRRI) in collaboration with the AI lab at Makerere University, Kampala. This is in a format that most realistically represents what farmers would need to diagnose in real life.\n\nYour task is to `classify each cassava image into four disease categories or a fifth category indicating a healthy leaf`. With your help, farmers may be able to quickly identify diseased plants, potentially saving their crops before they inflict irreparable damage.","metadata":{}},{"cell_type":"markdown","source":"**Getting packages**","metadata":{}},{"cell_type":"code","source":"import os\nimport re\nimport sys\nimport math\nimport json\nimport random\nimport itertools\nfrom functools import partial\n\nimport cv2\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import KFold, train_test_split\nfrom sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score\n\nimport albumentations as A \n\nimport tensorflow as tf\nfrom tensorflow.keras.optimizers import Adam, RMSprop\nfrom tensorflow.keras import Input, Sequential, mixed_precision\nfrom tensorflow.keras.optimizers.schedules import ExponentialDecay\nfrom tensorflow.keras.applications import EfficientNetB3, InceptionV3\nfrom tensorflow.keras.layers import Dense, Dropout, GlobalAveragePooling2D, Activation\nfrom tensorflow.keras.callbacks import Callback, EarlyStopping, LearningRateScheduler, ModelCheckpoint, ReduceLROnPlateau\n\nfrom kaggle_datasets import KaggleDatasets\n\n\nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🏂 Setting Up Things","metadata":{}},{"cell_type":"markdown","source":"**Configuring Hardware**, things configured here are `Mixed Precision`, `XLA` and `TPU`.\n\nTo know about [Mixed Precision](https://www.tensorflow.org/guide/mixed_precision) and [XLA](https://www.tensorflow.org/xla) follow the official `Tensorflow` docs via the `links`.\n\nHere I've diabled `XLA` and `Mixed Precision` as I was facing issue (`InvalidArgumentError`) while loading the saved model and doing predictions (stage where I had issue).\n\n```bash\nInvalidArgumentError: No OpKernel was registered to support Op 'Conv2D' used by {{node sequential/inception_v3/conv2d/Conv2D}} with these attrs: [dilations=[1, 1, 1, 1], T=DT_BFLOAT16, strides=[1, 2, 2, 1], data_format=\"NHWC\", use_cudnn_on_gpu=true, explicit_paddings=[], padding=\"VALID\"]\nRegistered devices: [CPU]\nRegistered kernels:\n  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_BFLOAT16, DT_HALF]\n  device='CPU'; T in [DT_INT32]\n  device='CPU'; T in [DT_DOUBLE]\n  device='CPU'; T in [DT_FLOAT]\n  device='CPU'; T in [DT_HALF]\n\n\t [[sequential/inception_v3/conv2d/Conv2D]] [Op:__inference_predict_function_11991]\n```\n\nTo view kernel which used this go to `version 5` of this kernel.","metadata":{}},{"cell_type":"code","source":"tf.keras.backend.clear_session()\n\nXLA_ACCELERATE = False\nMIXED_PRECISION = False\n\n# Enabling XLA\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n        \ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    \n    # Enabling mixed precision for TPUs\n    if MIXED_PRECISION:\n        mixed_precision.set_global_policy('mixed_bfloat16')\n        \nexcept Exception as e:\n    strategy = tf.distribute.get_strategy()\n\n    # Enabling mixed precision for CPUs/GPUs\n    if MIXED_PRECISION:\n        mixed_precision.set_global_policy('mixed_float16')\n\n        \nprint('Number of replicas:', strategy.num_replicas_in_sync)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Base Config**","metadata":{}},{"cell_type":"code","source":"class BaseConfig:\n    SEED = 101\n    \n    TRAIN_CSV = '../input/cassava-leaf-disease-classification/train.csv'\n    TRAIN_IMG_PATH = '../input/cassava-leaf-disease-classification/train_images/'\n    TEST_IMG_PATH = '../input/cassava-leaf-disease-classification/test_images/'\n    CLASS_MAP = '../input/cassava-leaf-disease-classification/label_num_to_disease_map.json'\n    \n    AUTOTUNE = tf.data.experimental.AUTOTUNE\n\n    IMAGE_SIZE = [512, 512]\n\n    FOLDS = 3\n    EPOCHS = 100\n    CLASSES = ['0', '1', '2', '3', '4']\n    \n    GCS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\n\n    # Getting training and testing tfrecord filenames\n    TRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train_tfrecords/ld_train*.tfrec')\n    TEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test_tfrecords/ld_test*.tfrec')\n    \n    \n    # ==========================\n    # Setting up batch size\n    # ==========================\n    \n    # When XLA is enabled, the batch size should be multiple of \n    # 128 (for TPU) or 64 (for CPU/GPU) as per Tensorflow docs\n    if strategy.num_replicas_in_sync == 8:\n        # BATCH_SIZE = 128 * strategy.num_replicas_in_sync\n        \n        # Due to running out of memory, using batch size as multiple of 8\n        BATCH_SIZE = 8 * strategy.num_replicas_in_sync\n    else:\n        # BATCH_SIZE = 64 * strategy.num_replicas_in_sync\n        \n        # Due to running out of memory, using batch size as multiple of 8\n        BATCH_SIZE = 8 * strategy.num_replicas_in_sync","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Understand the use of `tf.data.experimental.AUTOTUNE`**\n\nThe `tf.data` API provides a software pipelining mechanism through the `tf.data.Dataset.prefetch` transformation, which can be used to **decouple the time when data is produced from the time when data is consumed**. In particular, the transformation uses a background thread and an internal buffer to prefetch elements from the input dataset ahead of the time they are requested. The number of elements to prefetch should be equal to (or possibly greater than) the number of batches consumed by a single training step. You could either **manually tune** this value, or set it to `tf.data.experimental.AUTOTUNE` which will prompt the tf.data runtime to tune the value dynamically at runtime.\n\n`tf.data.experimental.AUTOTUNE` defines appropriate number of processes that are free for working. This can be used to get the `number of processes` that can be used for `transorfmation`.","metadata":{}},{"cell_type":"code","source":"def seed_all(s):\n    random.seed(s)\n    np.random.seed(s)\n    tf.random.set_seed(s)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['PYTHONHASHSEED'] = str(s)\n\n\n# Seeding all\nseed_all(BaseConfig.SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Dataset\ndf = pd.read_csv(BaseConfig.TRAIN_CSV)\ndf.label = df.label.astype('str')\n\nprint(f'Dataset size: {len(df)}')\ndf.sample(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Label\nname_of_diseases = pd.read_json(BaseConfig.CLASS_MAP, typ='series')\nname_of_diseases","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the `label` `count` in our `dataset`","metadata":{}},{"cell_type":"code","source":"sns.countplot(x='label', data=df, palette='icefire')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Plot images**","metadata":{}},{"cell_type":"code","source":"def plot_imgs(img_ids, labels):\n    plt.figure(figsize=(20, 18))\n    \n    num_samples = len(labels)\n    for idx, (img_id, label) in enumerate(zip(img_ids, labels)):\n        num_cols = 3\n        num_rows = (num_samples // num_cols) + (num_samples % num_cols)\n        plt.subplot(num_rows, num_cols, idx + 1)\n        \n        img = cv2.imread(BaseConfig.TRAIN_IMG_PATH + '/' + img_id)\n        \n        # Reversing the color channle from BGR to RGB\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(img)\n        plt.title(name_of_diseases[int(label)], fontsize=18)\n        plt.axis('off')\n        \n    plt.show()\n    \n    \ntmp_df  = df.sample(15)\nimg_ids = tmp_df.image_id.values\nlabels  = tmp_df.label.values\nplot_imgs(img_ids, labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🪂 Data Preparation\n\n**Functions to work with tfrecords**","metadata":{}},{"cell_type":"code","source":"def decode_image(img):\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    img = tf.reshape(img, [*BaseConfig.IMAGE_SIZE, 3])\n    return img\n\n\ndef read_tfrecord(example, labeled):\n    if labeled:\n        tfrecord_format = {\n            'image': tf.io.FixedLenFeature([], tf.string),\n            'target': tf.io.FixedLenFeature([], tf.int64)\n        }\n    else:\n        tfrecord_format = {\n            'image': tf.io.FixedLenFeature([], tf.string),\n            'image_name': tf.io.FixedLenFeature([], tf.string)\n        }\n    \n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    \n    if labeled:\n        label = tf.cast(example['target'], tf.float32)\n        return image, label\n\n    idnum = example['image_name']\n    return image, idnum\n\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Creating option for tf.data.Dataset\n    ignore_order = tf.data.Options()\n    \n    if not ordered:\n        # The outputs produced will not be in deterministic order\n        # default is True\n        ignore_order.experimental_deterministic = False\n        \n    # Creating dataset using multiple TFRecord files and with AUTOTUNE\n    # it'll use all the processes for transformation\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=BaseConfig.AUTOTUNE)\n    \n    # Using data as soon as it streams in, rather than in its original order\n    # if the ordered=False\n    dataset = dataset.with_options(ignore_order)\n    \n    dataset = dataset.map(\n        partial(read_tfrecord, labeled=labeled),\n        num_parallel_calls=BaseConfig.AUTOTUNE\n    )\n\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_augment(img, label):\n    img = tf.image.random_flip_left_right(img)\n    img = tf.image.random_flip_up_down(img)\n    img = tf.image.random_crop(img, size=[*BaseConfig.IMAGE_SIZE, 3])\n    return img, label","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Getting dataset\n\n\n''' Not using the albumentation's augmentations because while using it,\n    it's giving the error below\n    \n    UnavailableError: failed to connect to all addresses\n    Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n    :{\"created\":\"@1618946008.948648282\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1618946008.948644997\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n'''\n\ndef get_training_dataset(dataset, augment=True):\n    if augment:\n        dataset = dataset.map(data_augment, num_parallel_calls=BaseConfig.AUTOTUNE)\n        \n    # Repeating training dataset infinitely\n    dataset = dataset.repeat()\n    \n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BaseConfig.BATCH_SIZE)\n    \n    if augment:\n        dataset = dataset.map(transform, num_parallel_calls=BaseConfig.AUTOTUNE)\n        # dataset = albu_transforms_train(dataset)\n        pass\n    \n    # Prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(BaseConfig.AUTOTUNE)\n    return dataset\n\n\ndef get_validation_dataset(dataset, augment=True):\n    dataset = dataset.batch(BaseConfig.BATCH_SIZE)\n\n    if augment:\n        #dataset = albu_transforms_valid(dataset)\n        pass\n    \n    # Caching the elements in the dataset (this cache will persist until\n    # the cache file is removed)\n    dataset = dataset.cache()\n    \n    dataset = dataset.prefetch(BaseConfig.AUTOTUNE)\n    return dataset\n\n\ndef get_test_dataset(ordered=False, augment=False):\n    dataset = load_dataset(BaseConfig.TEST_FILENAMES, labeled=False, ordered=ordered)\n    \n    dataset = dataset.batch(BaseConfig.BATCH_SIZE)\n    \n    if augment:\n        #dataset = albu_transforms_inference(dataset)\n        pass\n    \n    dataset = dataset.prefetch(BaseConfig.AUTOTUNE)\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_data_items(filenames):\n    n = [\n        int(re.compile(r'-([0-9]*)\\.').search(filename).group(1))\n        for filename in filenames\n    ]\n    \n    return np.sum(n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# training, validation slipt from the training dataset\nTRAIN_VAL_SPLIT = 0.8\n\nNUM_TRAINING_IMAGES   = int(count_data_items(BaseConfig.TRAINING_FILENAMES) * TRAIN_VAL_SPLIT)\nNUM_VALIDATION_IMAGES = int(count_data_items(BaseConfig.TRAINING_FILENAMES) * (1 - TRAIN_VAL_SPLIT))\nNUM_TEST_IMAGES       = count_data_items(BaseConfig.TEST_FILENAMES)\n\nprint(f'Training set size: {NUM_TRAINING_IMAGES}')\nprint(f'Validation set size: {NUM_VALIDATION_IMAGES}')\nprint(f'Testing set size: {NUM_TEST_IMAGES}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Augmentation\n\n![](https://media.giphy.com/media/3ohzdQhmr2YrxHT45y/giphy.gif)","metadata":{}},{"cell_type":"code","source":"# Augmentation for training \ndef albu_transforms_train(train_ds):\n    transforms = A.Compose([\n        A.RandomResizedCrop(BaseConfig.IMAGE_SIZE[0], BaseConfig.IMAGE_SIZE[1]),\n        A.HorizontalFlip(p=0.5),\n        A.VerticalFlip(p=0.5),\n        A.ShiftScaleRotate(p=0.5),\n        A.RandomRotate90(p=0.5),\n        A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n        A.RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n        A.RandomShadow(shadow_roi=(0, 0.5, 1, 1), shadow_dimension=5, p=0.5),\n        A.CoarseDropout(p=0.5),            \n        A.Resize(BaseConfig.IMAGE_SIZE[0], BaseConfig.IMAGE_SIZE[1]),\n        A.ToFloat()\n    ], p=1.)\n    \n    def aug_fn(image):\n        data = {\"image\":image}\n        aug_data = transforms(**data)\n        aug_img = aug_data[\"image\"]\n        aug_img = tf.cast(aug_img, tf.float32)\n        return aug_img\n\n    def process_data(image, label):\n        aug_img = tf.numpy_function(func=aug_fn, inp=[image], Tout=tf.float32)\n        return aug_img, label\n    \n    def set_shapes(img, label, img_shape=(*BaseConfig.IMAGE_SIZE,3)):\n        img.set_shape(img_shape)\n        return img, label\n    \n    ds_alb = train_ds.map(partial(process_data), num_parallel_calls=BaseConfig.AUTOTUNE).prefetch(BaseConfig.AUTOTUNE)\n    ds_alb = ds_alb.map(set_shapes, num_parallel_calls=BaseConfig.AUTOTUNE)\n    ds_alb = ds_alb.repeat()\n    ds_alb = ds_alb.batch(BaseConfig.BATCH_SIZE)\n    print(len(list(ds_alb)))\n    return ds_alb\n\n\n# Augmentation for validation\ndef albu_transforms_valid(val_ds):\n    transforms = A.Compose([\n        A.ToFloat(),\n        A.CenterCrop(BaseConfig.IMAGE_SIZE[0], BaseConfig.IMAGE_SIZE[1], p=1.),\n        A.Resize(BaseConfig.IMAGE_SIZE[0], BaseConfig.IMAGE_SIZE[1])\n    ], p=1.)\n    \n    def aug_fn(image):\n        data = {\"image\":image}\n        aug_data = transforms(**data)\n        aug_img = aug_data[\"image\"]\n        aug_img = tf.cast(aug_img, tf.float32)\n        return aug_img\n\n    def process_data(image, label):\n        aug_img = tf.numpy_function(func=aug_fn, inp=[image], Tout=tf.float32)\n        print(aug_img.shape)\n        return aug_img, label\n    \n    def set_shapes(img, label, img_shape=(*BaseConfig.IMAGE_SIZE, 3)):\n        img.set_shape(img_shape)\n        return img, label\n    \n    ds_alb = val_ds.map(partial(process_data), num_parallel_calls=BaseConfig.AUTOTUNE).prefetch(BaseConfig.AUTOTUNE)\n    ds_alb = ds_alb.map(set_shapes, num_parallel_calls=BaseConfig.AUTOTUNE).batch(BaseConfig.BATCH_SIZE)\n    return ds_alb\n\n\n# Augmentation for Inference\ndef albu_transforms_inference(test_ds):\n    transforms = A.Compose([\n        A.Transpose(p=0.5),\n        A.VerticalFlip(p=0.5),\n        A.HorizontalFlip(p=0.5),\n        A.RandomRotate90(p=0.5),\n        A.ShiftScaleRotate(p=0.5),\n        A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n        A.ToFloat(),\n        A.RandomResizedCrop(BaseConfig.IMAGE_SIZE[0], BaseConfig.IMAGE_SIZE[1])\n    ], p=1.)\n    \n    def aug_fn(image):\n        data = {\"image\":image}\n        aug_data = transforms(**data)\n        aug_img = aug_data[\"image\"]\n        aug_img = tf.cast(aug_img, tf.float32)\n        return aug_img\n\n    def process_data(image, label):\n        aug_img = tf.numpy_function(func=aug_fn, inp=[image], Tout=tf.float32)\n        return aug_img, label\n    \n    def set_shapes(img, label, img_shape=(*BaseConfig.IMAGE_SIZE, 3)):\n        img.set_shape(img_shape)\n        return img, label\n    \n    ds_alb = test_ds.map(partial(process_data), num_parallel_calls=BaseConfig.AUTOTUNE).prefetch(BaseConfig.AUTOTUNE)\n    ds_alb = ds_alb.map(set_shapes, num_parallel_calls=BaseConfig.AUTOTUNE).batch(BaseConfig.BATCH_SIZE)\n    return ds_alb","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Applying sophisticated agumentation**\n\n- CutMix\n- MixUp\n- FixM\n\nFor more info have a look at the following kernels, [link](https://www.kaggle.com/saife245/cutmix-vs-mixup-vs-gridmask-vs-cutout) and [link](https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu) and for more info about these transformation follow this blog via this [link](https://medium.com/@virajbagal12/mixed-sample-data-augmentation-721e65093fcf).","metadata":{}},{"cell_type":"markdown","source":"**CutMix Agumentation**","metadata":{}},{"cell_type":"code","source":"def CutMix(image, label, DIM, PROBABILITY = 1.0):\n#     # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n#     # output - a batch of images with cutmix applied\n#     CLASSES = len(BaseConfig.CLASSES)\n#     BATCH_SIZE = BaseConfig.BATCH_SIZE\n    \n#     imgs = []; labs = []\n#     for j in range(len(image)):\n#         # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n#         P = tf.cast(tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n        \n#         # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n#         k = tf.cast(tf.random.uniform([], 0, BATCH_SIZE), tf.int32)\n        \n#         # CHOOSE RANDOM LOCATION\n#         x = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n#         y = tf.cast(tf.random.uniform([], 0, DIM), tf.int32)\n        \n#         # this is beta dist with alpha=1.0\n#         b = tf.random.uniform([], 0, 1)\n        \n#         WIDTH = tf.cast(DIM * tf.math.sqrt(1-b), tf.int32) * P\n#         ya = tf.math.maximum(0, y-WIDTH // 2)\n#         yb = tf.math.minimum(DIM, y+WIDTH // 2)\n#         xa = tf.math.maximum(0, x-WIDTH // 2)\n#         xb = tf.math.minimum(DIM, x+WIDTH // 2)\n        \n#         # MAKE CUTMIX IMAGE\n#         one = image[j, ya:yb, 0:xa, :]\n#         two = image[k, ya:yb, xa:xb ,:]\n#         three = image[j, ya:yb, xb:DIM, :]\n#         middle = tf.concat([one, two, three], axis=1)\n#         img = tf.concat([image[j, 0:ya, :, :], middle, image[j, yb:DIM, :, :]], axis=0)\n#         imgs.append(img)\n        \n#         # MAKE CUTMIX LABEL\n#         a = tf.cast(WIDTH*WIDTH/DIM/DIM, tf.float32)\n#         labs.append((1-a)*label[j] + a*label[k])\n            \n#     # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n#     image2 = tf.reshape(tf.stack(imgs), (BATCH_SIZE, DIM, DIM, 3))\n#     label2 = tf.reshape(tf.stack(labs), (BATCH_SIZE, CLASSES))\n\n\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    DIM = BaseConfig.IMAGE_SIZE[0]\n    CLASSES = len(BaseConfig.CLASSES)\n    AUG_BATCH = BaseConfig.BATCH_SIZE\n        \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        # MAKE CUTMIX LABEL\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        labs.append((1-a)*label[j] + a*label[k])\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,))\n    \n    return image2, label2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**View cutmix tranformations**","metadata":{}},{"cell_type":"code","source":"def apply_cutmix_on_example(img, label):\n    img, label = CutMix(img, label, BaseConfig.IMAGE_SIZE[0], 1)\n    return img, label\n\nrow = 4\ncol = 4\nrow = min(row, BaseConfig.BATCH_SIZE//col)\n\nall_elements = get_training_dataset(load_dataset(BaseConfig.TRAINING_FILENAMES), augment=False).unbatch()\naugmented_element = all_elements.repeat().batch(BaseConfig.BATCH_SIZE).map(apply_cutmix_on_example)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row, col, j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MixUp Augmentation**","metadata":{}},{"cell_type":"code","source":"def MixUp(image, label, DIM, PROBABILITY=1.0):\n#     # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n#     # output - a batch of images with mixup applied\n#     CLASSES = len(BaseConfig.CLASSES)\n    \n#     imgs = []; labs = []\n#     for j in range(len(image)):\n#         # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n#         P = tf.cast( tf.random.uniform([], 0, 1) <= PROBABILITY, tf.int32)\n                   \n#         # CHOOSE RANDOM\n#         k = tf.cast(tf.random.uniform([], 0, len(image)), tf.int32)\n        \n#         # this is beta dist with alpha=1.0\n#         a = tf.random.uniform([], 0, 1) * P\n                    \n#         # MAKE MIXUP IMAGE\n#         img1 = image[j,]\n#         img2 = image[k,]\n#         imgs.append((1-a)*img1 + a*img2)\n                    \n#         # MAKE CUTMIX LABEL\n#         labs.append((1-a)*label[j] + a*label[k])\n            \n#     # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n#     image2 = tf.reshape(tf.stack(imgs), (len(image), DIM, DIM, 3))\n#     label2 = tf.reshape(tf.stack(labs), (len(image), CLASSES))\n\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    DIM = BaseConfig.IMAGE_SIZE[0]\n    CLASSES = len(BaseConfig.CLASSES)\n    AUG_BATCH = BaseConfig.BATCH_SIZE\n        \n    imgs = []; labs = []\n    for j in range(AUG_BATCH):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,AUG_BATCH),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n        # MAKE CUTMIX LABEL\n        labs.append((1-a)*label[j] + a*label[k])\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(AUG_BATCH,DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(AUG_BATCH,))\n    \n    return image2, label2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def apply_mixup_on_example(img, label):\n    img, label = MixUp(img, label, BaseConfig.IMAGE_SIZE[0], 1)\n    return img, label\n\nrow = 4\ncol = 4\nrow = min(row, BaseConfig.BATCH_SIZE//col)\n\nall_elements = get_training_dataset(load_dataset(BaseConfig.TRAINING_FILENAMES), augment=False).unbatch()\naugmented_element = all_elements.repeat().batch(BaseConfig.BATCH_SIZE).map(apply_mixup_on_example)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row, col, j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**FMix Augmentation**","metadata":{}},{"cell_type":"code","source":"sys.path.insert(0, '/kaggle/input/pyutils')\nfrom fmix_utils import sample_mask\n\ndef FMix(image, label, DIM,  alpha=1, decay_power=3, max_soft=0.0, reformulate=False):\n    lam, mask = sample_mask(alpha, decay_power, (DIM, DIM), max_soft, reformulate)\n    \n    index = tf.constant(np.random.permutation(BaseConfig.BATCH_SIZE))\n    # Using the below line gives error so using the BaseConfig.BATCH_SIZE,\n    # image.shape[0] also represents batch size but since the tf.data.Dataset\n    # used here have a used the .repeat() method without parameter on them \n    # which repeats the dataset infinitely\n    # index = tf.constant(np.random.permutation(int(image.shape[0])))\n    \n    mask  = np.expand_dims(mask, -1)\n    \n    # samples \n    image1 = image * mask\n    image2 = tf.gather(image, index) * (1 - mask)\n    image3 = image1 + image2\n\n    # labels\n    label1 = label * lam \n    label2 = tf.gather(label, index) * (1 - lam)\n    label3 = label1 + label2 \n    \n    return image3, label3","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def apply_fmix_on_example(img, label):\n    img, label = FMix(img, label, BaseConfig.IMAGE_SIZE[0])\n    return img, label\n\nrow = 4\ncol = 4\nrow = min(row, BaseConfig.BATCH_SIZE//col)\n\nall_elements = get_training_dataset(load_dataset(BaseConfig.TRAINING_FILENAMES), augment=False).unbatch()\naugmented_element = all_elements.repeat().batch(BaseConfig.BATCH_SIZE).map(apply_fmix_on_example)\n\nfor (img, label) in augmented_element:\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        plt.subplot(row, col, j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In this function all the `sophisticated` augmentations are applied\ndef transform(image, label):\n    DIM = BaseConfig.IMAGE_SIZE[0]\n    CLASSES = len(BaseConfig.CLASSES)\n    \n    SWITCH = 0.5\n    CUTMIX_PROB = 0.2\n    MIXUP_PROB = 0.1\n    \n    # FOR SWITCH PERCENT OF TIME WE DO CUTMIX AND (1-SWITCH) WE DO MIXUP\n    image2, label2 = CutMix(image, label, DIM, CUTMIX_PROB)\n    image3, label3 = MixUp(image, label, DIM, MIXUP_PROB)\n    \n    # Not using FMix as it doesn't give good results, the loss become NaN and \n    # accuracy was very bad \n    # if np.random.rand() > 0.1:\n    #     image4, label4 = FMix(image, label, DIM)\n    \n    imgs = []; labs = []\n    for j in range(BaseConfig.BATCH_SIZE):\n        P = tf.cast(tf.random.uniform([], 0, 1) <= SWITCH, tf.float32)\n        \n        # While using FMix uncomment 2 lines below and comment the other 2 lines\n        # imgs.append(P*image2[j,] + (1-P)*image3[j,] + image4[j,])\n        # labs.append(P*label2[j,] + (1-P)*label3[j,] + label4[j,])\n        \n        imgs.append(P*image2[j,] + (1-P)*image3[j,])\n        labs.append(P*label2[j,] + (1-P)*label3[j,])\n                \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image5 = tf.reshape(tf.stack(imgs), (BaseConfig.BATCH_SIZE, DIM, DIM, 3))\n    label5 = tf.reshape(tf.stack(labs), (BaseConfig.BATCH_SIZE, 1))\n    # label5 = tf.cast(label5, tf.int32)\n    \n    return image5, label5    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🏋️‍♀️ Modelling\n\n![](https://media.giphy.com/media/3o7TKWC4IgROm4Qdc4/giphy.gif)","metadata":{}},{"cell_type":"markdown","source":"**Building the CustomCallback for getting more info on model's performance**\n","metadata":{}},{"cell_type":"code","source":"class CustomCallback(Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        loss      = logs['loss']\n        accuracy  = logs['accuracy']\n        \n        # Validation info\n        val_loss      = logs['val_loss']\n        val_accuracy  = logs['val_accuracy']\n        \n        info = {\n            'loss': round(loss, 5),\n            'accuracy': round(accuracy, 4),\n            \n            'val_loss': round(val_loss, 5),\n            'val_accuracy': round(val_accuracy, 4),\n        }\n        \n        print(f'\\n{json.dumps(info, indent=2)}')\n        print()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Other callbacks\n\n# Since the competition needs the best accuracy score\n# therefore monitoring accuracy\nmodel_checkpoints = ModelCheckpoint(\n    'model.h5', save_best_only=True,\n    monitor='val_accuracy', mode='max',\n    verbose=1\n    \n)\n\nearly_stopping = EarlyStopping(\n    monitor='val_loss', mode='min',\n    patience=7, restore_best_weights=True,\n    verbose=1\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Learning Rate Scheduler**","metadata":{}},{"cell_type":"code","source":"scheduler = ExponentialDecay(\n    initial_learning_rate=1e-3,\n    decay_steps=10000,\n    decay_rate=0.9\n)\n\n\n# ==============================\n# Custom Learning Rate Schedule\n# ==============================\n\n# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\nLR_START = 1e-8\nLR_MIN = 1e-8\nLR_MAX = 3e-5 * strategy.num_replicas_in_sync\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nN_CYCLES = .5\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        progress = (epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) / (BaseConfig.EPOCHS - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)\n        lr = LR_MAX * (0.5 * (1.0 + tf.math.cos(math.pi * N_CYCLES * 2.0 * progress)))\n        if LR_MIN is not None:\n            lr = tf.math.maximum(LR_MIN, lr)\n            \n    return lr\n    \nlr_scheduler = LearningRateScheduler(lrfn)\n\n\nrng = [i for i in range(25 if BaseConfig.EPOCHS<25 else BaseConfig.EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# All the callbacks\nCALLBACKS = [\n    model_checkpoints,\n    # early_stopping,\n    lr_scheduler,\n    CustomCallback()\n]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OPTIMIZER = Adam()\nLOSS = 'sparse_categorical_crossentropy'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n    with strategy.scope():\n        inputs = Input(shape=[*BaseConfig.IMAGE_SIZE, 3])\n        base_model = InceptionV3(weights='imagenet', include_top=False)\n        base_model.trainable = True\n        \n        model_layers = [\n            inputs,\n            base_model,\n            GlobalAveragePooling2D(),\n            Dropout(0.25)\n        ]\n        \n        if MIXED_PRECISION:\n            # Since using mixed precision we have to use tf.float32 as dtype in our last layer\n            # So that's why breaking them in to Dense (with no activation) and Activation layer\n            model_layers.extend([\n                Dense(len(BaseConfig.CLASSES)),\n                Activation('softmax', dtype='float32')\n            ])\n        else:\n            model_layers.extend([\n                Dense(len(BaseConfig.CLASSES), activation='softmax')\n            ])\n        \n        model = Sequential(model_layers)\n        \n        # Metrics\n        METRICS = ['accuracy']\n\n    model.compile(optimizer=OPTIMIZER, loss=LOSS, metrics=METRICS)         \n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training without cross validaiton**","metadata":{}},{"cell_type":"code","source":"def train_without_cross_validation():\n    # splitting the data for training and validation\n    TRAINING_FILENAME, VAL_FILENAME = train_test_split(\n        tf.io.gfile.glob(BaseConfig.GCS_PATH + '/train_tfrecords/ld_train*.tfrec'),\n        test_size=0.2, random_state=0\n    )\n    \n    # Creating pd.Dataframe \n    train_df = pd.DataFrame({ 'TRAINING_FILENAMES': TRAINING_FILENAME })['TRAINING_FILENAMES']\n    val_df   = pd.DataFrame({ 'TRAINING_FILENAMES': VAL_FILENAME })['TRAINING_FILENAMES']\n    \n    # Loading datasets\n    train_dataset = load_dataset(list(train_df), labeled=True)\n    val_dataset   = load_dataset(list(val_df), labeled=True, ordered=True)\n    \n    # Getting nums of steps per epoch and validation step per epoch as \n    # we are used tf.data.Dataset.repeat() method on our dataset, so \n    # we need to tell our algorithm when to stop iterating through \n    # the dataset as repeat() without parameter will generate dataset \n    # of infinite length\n    \n    train_files = list(train_df)\n    val_files   = list(val_df)\n    num_train_imgs = count_data_items(train_files)\n    num_val_imgs   = count_data_items(val_files)\n    steps_per_epoch = num_train_imgs // BaseConfig.BATCH_SIZE\n    validation_steps_per_epoch = num_val_imgs // BaseConfig.BATCH_SIZE\n    \n    # Data pre-processing + Augmentation\n    train = get_training_dataset(train_dataset)\n    val   = get_validation_dataset(val_dataset)\n\n    # Getting the model\n    model = build_model()\n    \n    # Training the model\n    history = model.fit(\n        train,\n        steps_per_epoch=steps_per_epoch,\n        epochs=BaseConfig.EPOCHS,\n        callbacks=CALLBACKS,\n        validation_data=val,\n        validation_steps=validation_steps_per_epoch,\n        verbose=1\n    )\n    \n    return history, model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training with cross validation**","metadata":{}},{"cell_type":"code","source":"def train_with_cross_validation(folds=BaseConfig.FOLDS):\n    models = []\n    histories = []\n    kFold = KFold(folds, shuffle=True, random_state=0)\n    \n    for fold, (train_idx, val_idx) in enumerate(kFold.split(BaseConfig.TRAINING_FILENAMES)):\n        print()\n        print('#'*25)\n        print(f'### FOLD - {fold + 1} ###')\n        print('#'*25)\n        \n        # Splitting the data for training and validation\n        train_df = pd.DataFrame({ 'TRAINING_FILENAMES': BaseConfig.TRAINING_FILENAMES }).loc[train_idx]['TRAINING_FILENAMES']\n        val_df   = pd.DataFrame({ 'TRAINING_FILENAMES': BaseConfig.TRAINING_FILENAMES }).loc[val_idx]['TRAINING_FILENAMES']\n        \n        # Loading datasets\n        train_dataset = load_dataset(list(train_df), labeled=True)\n        val_dataset   = load_dataset(list(val_df), labeled=True, ordered=True)\n        \n        # Getting nums of steps per epoch and validation steps\n        train_files = list(train_df)\n        val_files   = list(val_df)\n        num_train_imgs = count_data_items(train_files)\n        num_val_imgs   = count_data_items(val_files)\n        steps_per_epoch = num_train_imgs // BaseConfig.BATCH_SIZE\n        validation_steps_per_epoch = num_val_imgs // BaseConfig.BATCH_SIZE\n        \n        # Data pre-processing + Augmentation\n        train = get_training_dataset(train_dataset)\n        val   = get_validation_dataset(val_dataset)\n\n        # Getting the model\n        model = build_model()\n\n        # Training the model\n        history = model.fit(\n            train,\n            steps_per_epoch=steps_per_epoch,\n            epochs=BaseConfig.EPOCHS,\n            callbacks=CALLBACKS,\n            validation_data=val,\n            validation_steps=validation_steps_per_epoch,\n            verbose=1\n        )\n        \n        models.append(model)\n        histories.append(history)\n        \n    return histories, models","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**While training make predictions for each model in the respective fold**","metadata":{}},{"cell_type":"code","source":"def train_cv_and_predict(folds=BaseConfig.FOLDS):\n    # Since we are splitting the dataset and iterating \n    # separately on imgs and ids, order matters\n    test_ds = get_test_dataset(ordered=True)\n    test_img_ds = test_ds.map(lambda img, idnum: img)\n    \n    print(f'Start training {folds} folds')\n    histories, models = train_with_cross_validation(folds=folds)\n    print('Computing predictions...')\n    \n    # Get the mean probability of the folds models\n    probabilities = np.average(\n        [models[i].predict(test_img_ds) for i in range(folds)],\n        axis=0\n    )\n    \n    predictions = np.argmax(probilities, axis=-1)\n    \n    print('Generating submission.csv file...')\n    test_ids_ds = test_ds.map(lambda img, idnum: idnum).unbatch()\n    \n    # All in one batch\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n    \n    # Submitting the results\n    np.savetxt(\n        'submission.csv',\n        np.rec.fromarrays([test_ids, predictions]),\n        fmt=['%s', '%d'],\n        delimiter=',',\n        header='image_id,label',\n        comments=''\n    )\n    \n    return histories, models","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Training and making prediction on model without cross validaiton**","metadata":{}},{"cell_type":"code","source":"def train_and_predict():\n    # Since we are splitting the dataset and iterating\n    # separately on imgs and ids, order matters\n    test_ds = get_test_dataset(ordered=True)\n    test_img_ds = test_ds.map(lambda img, idnum: img)\n    \n    print('Start training')\n    history, model = train_without_cross_validation()\n    print('Computing predictions...')\n    \n    # Get the mean probabiliy of the folds models\n    probabilities = model.predict(test_img_ds)\n    predictions   = np.argmax(probabilities, axis=-1)\n    \n    print('Generating submission.csv file...')\n    test_ids_ds = test_ds.map(lambda img, idnum: idnum).unbatch()\n    \n    # All in one batch\n    test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U')\n    \n    # Submitting the results\n    np.savetxt(\n        'submission.csv',\n        np.rec.fromarrays([test_ids, predictions]),\n        fmt=['%s', '%d'],\n        delimiter=',',\n        header='image_id,label',\n        comments=''\n    )\n    \n    return history, model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history, model = train_without_cross_validation()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧬 Evaluation\n\n![](https://media.giphy.com/media/3orieSdZDhn7I6gViw/giphy.gif)","metadata":{}},{"cell_type":"markdown","source":"**Accuracy**","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['accuracy'][1:], label='train acc')\nplt.plot(history.history['val_accuracy'][1:], label='validation acc')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend(loc='lower right')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loss**","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['loss'][1:], label='train loss')\nplt.plot(history.history['val_loss'][1:], label='validation loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend(loc='upper right')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# confusion matrix\ndef display_confusion_matrix(cm, classes, normalize=False, title='Confusion matrix', cmap=sns.cubehelix_palette(start=.5, rot=-.5, as_cmap=True)):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    \n    if normalize:\n        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n        print(\"Normalized confusion matrix\")\n    else:\n        print('Confusion matrix, without normalization')\n\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n\n    fmt = '.2f' if normalize else 'd'\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, format(cm[i, j], fmt), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inferring validation images","metadata":{}},{"cell_type":"markdown","source":"Since using `tfrecords` we have to extra step to create `confusion matrix`","metadata":{}},{"cell_type":"code","source":"# Labels, probabilities and predictions\nall_labels = []\nall_prob = []\nall_pred = []","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Inferring validation images...')\n\n# Getting the validation set ready (predictions will be done on \n# and it will be used to plot the confusion matrix)\nTRAINING_FILENAMES, VALID_FILENAMES = train_test_split(\n    tf.io.gfile.glob(BaseConfig.GCS_PATH + '/train_tfrecords/ld_train*.tfrec'),\n    test_size=0.2, random_state=0\n)\n\n# Getting filenames and dataset size\nVAL_FILES = list(pd.DataFrame({'TRAINING_FILENAMES': VALID_FILENAMES})['TRAINING_FILENAMES'])\nNUM_VALIDATION_IMAGES = count_data_items(VAL_FILES)\nNUM_VALIDATION_IMAGES","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading validation set\ncmdataset = get_validation_dataset(load_dataset(VAL_FILES, labeled=True, ordered=True))\n\n# validation imgs\nimages_ds = cmdataset.map(lambda image, label: image)\n\n# validation labels (all in one batch)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\nall_labels.append(next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Predicting\nprob = model.predict(images_ds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Probabilities\nall_prob.append(prob)\n\n# Predicted labels\nall_pred.append(np.argmax(prob, axis=-1))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cm_correct_labels = np.concatenate(all_labels)\ncm_probabilities = np.concatenate(all_prob)\ncm_predictions = np.concatenate(all_pred)\n\nprint(all_labels)\nprint(cm_correct_labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classes_num = len(BaseConfig.CLASSES)\n\ncmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(classes_num))\nscore = f1_score(cm_correct_labels, cm_predictions, labels=range(classes_num), average='weighted')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(classes_num), average='weighted')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(classes_num), average='weighted')\n\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_confusion_matrix(cmat, BaseConfig.CLASSES, True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If this kernel helped you then don't forget to 🔼 `upvote` and share your 🎙 `feedback` on improvements of the kernel.\n\n![](https://media.giphy.com/media/1ncv8cQdrpLjzn4wOz/giphy.gif)\n\n---","metadata":{}}]}