{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**finale update**\n\n- In the below section, I was **wrong** about the official `EfficientNet`. The weird performace was due to the normalizing the data in data-loader; currently it's not needed becuase this network has built-in normalizatio layer. However, I'm not sure if this will be remained or removed in future. \n\n---\n\n## About this kernel\n- **Augmentations**: Advance augmentation like `CutMix`, `MixUp`, `FMix` are used. \n- **Data Generator**: A Keras custom data generator is implemented.\n- **Training**: Previously We've used official `EfficientNet` models but due to their weird performance it's been changed. \n- **Training Mechansim**: A convininet OOP oriented training mechanism is implemented for K-Fold training and Out-of-fold validation and TTA over validation set.\n- **Submission**: Test Time Augmentation is applied to each K-Fold splits and ended up with group Ensemble. For TTA `albumentation` library is used. \n\n\n## Logs\n- **update 2**: Used un-official efficientnet implementation\n- **update 1**: Added Symmetric Cross Entropy Loss for dealing with noisy labels \n\nSymmetric Cross Entropy Loss, [implementation](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/208324). \n\n---","metadata":{}},{"cell_type":"markdown","source":"## Cassava Leaf Disease Classification\n\nHi,\nThis is a complete **GPU** baseline starter in `tf.keras` for this **Cassava Leaf Disease Classification** problem. Here `jpg` samples will be used for modeling in ene-to-end. Main approaches are:\n\n```\n- Stratify-KFold (K=5)\n- Image Augmentation\n    - General Augmentaiton (Albumentation)\n    - CutMix \n    - MixUp\n    - FMix\n    - Mosaic (will not use, incomplete implementation)\n- Image Modeling\n    - EfficientNet (Modified) + Custom Top Layers (Attention Mechanisms) + Softmax \n- Training\n    - 5 Fold Training \n    - OOF Evaluation: Optimizing Metrics - Out-of-Fold Weights Ensemble\n- Inference\n    - Ensemble\n    - Test Time Augmentation\n```\n\nThe implementations of `CutMix` and `MixUp` augmentation are taken from [Chris Deotte](https://www.kaggle.com/cdeotte) and integrated into a custom [tf.keras.utils.Sequence](https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence) generator. For modeling, `EfficientNets` are typically used but feel free to change with other architecture in the config class. We've tried to integrate duel attention mechanism on top of the base model. Next,a loop training process is done over the number of folds. In inference time, the saved weights with the respected model architectures and input sizes is loaded and run **TTA** for each model followed by ensemble. ","metadata":{}},{"cell_type":"code","source":"!pip install /kaggle/input/kerasapplications -q\n!pip install /kaggle/input/efficientnet-keras-source-code/ -q --no-deps","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-08-05T14:56:47.403352Z","iopub.execute_input":"2021-08-05T14:56:47.403693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom glob import glob\nimport albumentations as A \nfrom pylab import rcParams\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nimport os, gc, cv2, random, warnings, math, sys, json, pprint\n\n# sklearn\nfrom sklearn.utils import class_weight\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import accuracy_score, balanced_accuracy_score\n\n# tf \nimport tensorflow as tf\nimport efficientnet.tfkeras as efn\nfrom tensorflow.keras import backend as K\n\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \nwarnings.simplefilter('ignore')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# helper function to plot sample \ndef plot_imgs(dataset_show, row, col):\n    rcParams['figure.figsize'] = 20,10\n    for i in range(row):\n        f, ax = plt.subplots(1,col)\n        for p in range(col):\n            idx = np.random.randint(0, len(dataset_show))\n            img, label = dataset_show[idx]\n            ax[p].grid(False)\n            ax[p].imshow(img[0])\n            ax[p].set_title(idx)\n    plt.show()\n    \n\ndef visulize(path, n_images, is_random=True, figsize=(16, 16)):\n    plt.figure(figsize=figsize)\n    \n    w = int(n_images ** .5)\n    h = math.ceil(n_images / w)\n    \n    image_names = os.listdir(path)\n    for i in range(n_images):\n        image_name = image_names[i]\n        if is_random:\n            image_name = random.choice(image_names)\n            \n        img = cv2.imread(os.path.join(path, image_name))\n        plt.subplot(h, w, i + 1)\n        plt.imshow(img)\n        plt.xticks([])\n        plt.yticks([])\n    plt.show()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Enabling Mixed Precision and Accelerated Linear Algebra**","metadata":{}},{"cell_type":"code","source":"MIXED_PRECISION = True\nXLA_ACCELERATE  = False # Didn't work; Dunno Why!\n\nGPUS = tf.config.experimental.list_physical_devices('GPU')\nif GPUS:\n    try:\n        for GPU in GPUS:\n            tf.config.experimental.set_memory_growth(GPU, True)\n            logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n            print(len(GPUS), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\") \n    except RuntimeError as  RE:\n        print(RE)\n\nif MIXED_PRECISION:\n    policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    tf.keras.mixed_precision.experimental.set_policy(policy)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n    \nstrategy = tf.distribute.get_strategy()\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}') \nprint(\"Tensorflow version \" + tf.__version__)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Basic Config**","metadata":{}},{"cell_type":"code","source":"class BaseConfig(object):\n    SEED  = 101\n    TRAIN_DF       = '../input/cassava-leaf-disease-classification/train.csv'\n    TRAIN_IMG_PATH = '../input/cassava-leaf-disease-classification/train_images/'\n    TEST_IMG_PATH  = '../input/cassava-leaf-disease-classification/test_images/'\n    CLASS_MAP      = '../input/cassava-leaf-disease-classification/label_num_to_disease_map.json'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_all(s):\n    random.seed(s)\n    np.random.seed(s)\n    tf.random.set_seed(s)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['PYTHONHASHSEED'] = str(s) \n    \nseed_all(BaseConfig.SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**General Overview**","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(BaseConfig.TRAIN_DF)\nassert df.shape[0] == len(df.image_id.unique()) , \"NOT ALL ID UNIQUE\"\nprint(df.info())\ndf.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Significant Class Imbalance**","metadata":{}},{"cell_type":"code","source":"with open(os.path.join(BaseConfig.CLASS_MAP)) as file:\n    pprint.pprint(json.loads(file.read()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp_df = df.copy()\ntemp_df[['CBB', 'CBSD', \n         'CGM', 'CMD', 'Healthy']] = pd.get_dummies(temp_df[\"label\"])\n\nfig = go.Figure(data=[go.Pie(labels=temp_df.columns[2:],values=temp_df.iloc[:, 2:].sum().values)])\nfig.show()\n\ndel temp_df","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Displaying Samples**","metadata":{}},{"cell_type":"code","source":"visulize(BaseConfig.TRAIN_IMG_PATH, 9, is_random=True)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentation\n\nThe `albumentation` is primarily used. However, for more details augmentation, plese refer to my another work, [here](https://www.kaggle.com/ipythonx/tf-keras-sota-augmentation-in-sequence-generator). ","metadata":{}},{"cell_type":"code","source":"import albumentations as A \n\n# For Training \ndef albu_transforms_train(data_resize): \n    return A.Compose([\n            A.RandomResizedCrop(data_resize, data_resize),\n            A.HorizontalFlip(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.ShiftScaleRotate(p=0.5),\n            A.RandomRotate90(p=0.5),\n            A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, \n                                 val_shift_limit=0.2, p=0.5),\n            A.RandomBrightnessContrast(brightness_limit=(-0.1,0.1), \n                                       contrast_limit=(-0.1, 0.1), p=0.5),\n            A.RandomShadow(shadow_roi=(0, 0.5, 1, 1), \n                           shadow_dimension=5, p=0.5),\n            A.CoarseDropout(p=0.5),            \n            A.ToFloat()\n        ], p=1.)\n\n# For Validation \ndef albu_transforms_valid(data_resize): \n    return A.Compose([\n            A.ToFloat(),\n            A.CenterCrop(data_resize, data_resize, p=1.),\n            A.Resize(data_resize, data_resize),\n        ], p=1.)\n\n# For Inference (TTA)\ndef albu_transforms_inference(data_resize): \n    return A.Compose([\n            A.Transpose(p=0.5),\n            A.VerticalFlip(p=0.5),\n            A.HorizontalFlip(p=0.5),\n            A.RandomRotate90(p=0.5),\n            A.ShiftScaleRotate(p=0.5),\n            A.HueSaturationValue(hue_shift_limit=0.2, \n                                 sat_shift_limit=0.2, \n                                 val_shift_limit=0.2, p=0.5),\n            A.ToFloat(),\n            A.RandomResizedCrop(data_resize, data_resize)\n        ], p=1.)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**CutMix** Augmentation","metadata":{}},{"cell_type":"code","source":"def CutMix(image, label, DIM, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with cutmix applied\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(len(image)):\n        # DO CUTMIX WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.int32)\n        \n        # CHOOSE RANDOM IMAGE TO CUTMIX WITH\n        k = tf.cast( tf.random.uniform([],0,len(image)),tf.int32)\n        \n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        \n        b = tf.random.uniform([],0,1) # this is beta dist with alpha=1.0\n        \n        WIDTH = tf.cast( DIM * tf.math.sqrt(1-b),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        \n        # MAKE CUTMIX IMAGE\n        one = image[j,ya:yb,0:xa,:]\n        two = image[k,ya:yb,xa:xb,:]\n        three = image[j,ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        img = tf.concat([image[j,0:ya,:,:],middle,image[j,yb:DIM,:,:]],axis=0)\n        imgs.append(img)\n        \n        # MAKE CUTMIX LABEL\n        a = tf.cast(WIDTH*WIDTH/DIM/DIM,tf.float32)\n        labs.append((1-a)*label[j] + a*label[k])\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(len(image),DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(len(image),CLASSES))\n    \n    return image2,label2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**MixUp** Augmentation","metadata":{}},{"cell_type":"code","source":"def MixUp(image, label, DIM, PROBABILITY = 1.0):\n    # input image - is a batch of images of size [n,dim,dim,3] not a single image of [dim,dim,3]\n    # output - a batch of images with mixup applied\n    CLASSES = 5\n    \n    imgs = []; labs = []\n    for j in range(len(image)):\n        # DO MIXUP WITH PROBABILITY DEFINED ABOVE\n        P = tf.cast( tf.random.uniform([],0,1)<=PROBABILITY, tf.float32)\n                   \n        # CHOOSE RANDOM\n        k = tf.cast( tf.random.uniform([],0,len(image)),tf.int32)\n        a = tf.random.uniform([],0,1)*P # this is beta dist with alpha=1.0\n                    \n        # MAKE MIXUP IMAGE\n        img1 = image[j,]\n        img2 = image[k,]\n        imgs.append((1-a)*img1 + a*img2)\n                    \n        # MAKE CUTMIX LABEL\n        labs.append((1-a)*label[j] + a*label[k])\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR (maybe use Python typing instead?)\n    image2 = tf.reshape(tf.stack(imgs),(len(image),DIM,DIM,3))\n    label2 = tf.reshape(tf.stack(labs),(len(image),CLASSES))\n    return image2,label2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**FMix** Augmentation","metadata":{}},{"cell_type":"code","source":"sys.path.insert(0, \"/kaggle/input/pyutils\")\nfrom fmix_utils import sample_mask\n\ndef FMix(image, label, DIM,  alpha=1, decay_power=3, max_soft=0.0, reformulate=False):\n    lam, mask = sample_mask(alpha, decay_power,(DIM, DIM), max_soft, reformulate)\n    index = tf.constant(np.random.permutation(int(image.shape[0])))\n    mask  = np.expand_dims(mask, -1)\n    \n    # samples \n    image1 = image * mask\n    image2 = tf.gather(image, index) * (1 - mask)\n    image3 = image1 + image2\n\n    # labels\n    label1 = label * lam \n    label2 = tf.gather(label, index) * (1 - lam)\n    label3 = label1 + label2 \n    return image3, label3","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cassava Samples Generator\n\nHere is a custom data generator. It's a standalone for training, validation and also inference. The image resizing and normalizing is done in the augmentation part, however. The randomness of choosing **cutmix** or **mixup** is arbitrary. ","metadata":{}},{"cell_type":"code","source":"class CassavaGenerator(tf.keras.utils.Sequence):\n    def __init__(self, img_path, data, batch_size, random_state, \n                 dim, shuffle=True, transform=None, \n                 use_mixup=False, use_cutmix=False,\n                 use_fmix=False, is_train=False):\n        self.dim  = dim\n        self.data = data\n        self.random_state = random_state\n        self.shuffle  = shuffle\n        self.img_path = img_path\n        self.is_train = is_train\n        self.augment  = transform\n        self.use_cutmix = use_cutmix\n        self.use_mixup  = use_mixup\n        self.use_fmix   = use_fmix \n        self.batch_size = batch_size\n        self.list_idx   = self.data.index.values\n        self.label = pd.get_dummies(self.data['label'], \n                                    columns = ['label']) if self.is_train else np.nan\n        self.on_epoch_end()\n        \n    def __len__(self):\n        return int(np.floor(len(self.list_idx) / self.batch_size))\n    \n    def __getitem__(self, index):\n        batch_idx = self.indices[index*self.batch_size:(index+1)*self.batch_size]\n        idx = [self.list_idx[k] for k in batch_idx]\n        \n        Data   = np.empty((self.batch_size, *self.dim))\n        Target = np.empty((self.batch_size, 5), dtype = np.float32)\n\n        for i, k in enumerate(idx):\n            # load the image file using cv2\n            image = cv2.imread(self.img_path + self.data['image_id'][k])\n            image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)\n            \n            res = self.augment(image=image)\n            image = res['image']\n            \n            # assign \n            if self.is_train:\n                Data[i,] =  image\n                Target[i,] = self.label.iloc[k,].values\n            else:\n                Data[i,] =  image \n                \n        if np.random.rand() > 0.2 and self.use_cutmix:\n            Data, Target = CutMix(Data, Target, self.dim[0])\n        elif np.random.rand() > 0.2 and self.use_mixup:\n            Data, Target = MixUp(Data, Target, self.dim[0]) \n        elif np.random.rand() > 0.1 and self.use_fmix:\n            Data, Target = FMix(Data, Target, self.dim[0])\n\n        return Data, Target if self.is_train else Data\n    \n    def on_epoch_end(self):\n        self.indices = np.arange(len(self.list_idx))\n        if self.shuffle:\n            np.random.seed(self.random_state)\n            np.random.shuffle(self.indices)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sanity Checks**","metadata":{}},{"cell_type":"code","source":"check_gens = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                              df, 20, 1234, (128, 128, 3),\n                              shuffle = True, is_train = True, \n                              use_mixup = False, use_cutmix = False, \n                              use_fmix = False,transform = albu_transforms_train(128))\n\nprint('Only Albumentation')\nplot_imgs(check_gens, 2, 4)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**With CutMix** + `np.random.rand() > 0.7`","metadata":{}},{"cell_type":"code","source":"check_gens = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                              df, 20,1234,(128, 128, 3),\n                              shuffle = True, is_train = True, \n                              use_mixup = False, use_cutmix = True, \n                              use_fmix = False, transform = albu_transforms_train(128))\n\nplot_imgs(check_gens, 2, 4)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**With MixUp** + `np.random.rand() > 0.4`","metadata":{}},{"cell_type":"code","source":"check_gens = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                              df, 20, 1234, (128, 128, 3),\n                              shuffle = True, is_train = True, \n                              use_mixup = True, use_cutmix = False,\n                              use_fmix = False, transform = albu_transforms_train(128))\n\nplot_imgs(check_gens, 2, 4)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**With FMix** + `np.random.rand() > 0.1`","metadata":{}},{"cell_type":"code","source":"check_gens = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                              df, 20, 1234, (128, 128, 3),\n                              shuffle = True, is_train = True, \n                              use_mixup = False, use_cutmix = False,\n                              use_fmix = True, transform = albu_transforms_train(128))\n\nplot_imgs(check_gens, 2, 4)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Train Config**\n\nA convenient way to experiment with different fold. ","metadata":{}},{"cell_type":"code","source":"class TrainConfig(BaseConfig):\n    NUM_CLASS    = 5\n    EPOCH        = 25\n    TTA          = 3 \n    FOLDS        = 3 \n    VERBOSITY    = 1\n    WORKERS      = 2\n    LABEL_SMOOTH = 0\n    MULTIPROCESS = False  # Didn't work as expected \n    CLS_BLN_WGS  = False  # Didn't work as expected \n\n    # randomly applied\n    MixUp  = False\n    CutMix = False\n    FMix   = False\n    \n    # Learning Rate for each fold\n    LR_RATE = {\n        '0' : 1e-3,  # learning rate in fold 0\n        '1' : 1e-3,  # learning rate in fold 1\n        '2' : 1e-3,  # learning rate in fold 2\n        '3' : 1e-3,  # learning rate in fold 3\n        '4' : 1e-3   # learning rate in fold 4\n    }\n    \n    # Batch Size for each fold \n    BATCH_SIZE = {\n        '0' : 64,  # batch size in fold 0\n        '1' : 86,  # batch size in fold 1 \n        '2' : 86,  # batch size in fold 2\n        '3' : 86,  # batch size in fold 3\n        '4' : 86   # batch size in fold 4\n    }\n    \n    # Base Networks for each fold\n    IMG_SIZE = {\n        '0': 260, # size of the image in fold 0\n        '1': 260, # size of the image in fold 1\n        '2': 260, # size of the image in fold 2\n        '3': 260, # size of the image in fold 3\n        '4': 260  # size of the image in fold 4\n    } \n    \n    BASE_NETS = {\n        '0' : [efn.EfficientNetB3, \n               '../input/efficientnet-keras-noisystudent-weights-b0b7/efficientnet-b3_noisy-student_notop.h5'],\n        '1' : [efn.EfficientNetB1, \n               '../input/efficientnet-keras-noisystudent-weights-b0b7/efficientnet-b1_noisy-student_notop.h5'],\n        '2' : [efn.EfficientNetB2, \n               '../input/efficientnet-keras-noisystudent-weights-b0b7/efficientnet-b2_noisy-student_notop.h5'],\n        '3' : [efn.EfficientNetB0, \n               '../input/efficientnet-keras-noisystudent-weights-b0b7/efficientnet-b0_noisy-student_notop.h5'],\n        '4' : [efn.EfficientNetB1, \n               '../input/efficientnet-keras-noisystudent-weights-b0b7/efficientnet-b1_noisy-student_notop.h5'],\n    }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Image Modeling\n\n\nLet's build the whole model. Here we're using subclassed api. But this can be easily implemented with functional api too. However, in case you're new to subclassing api and wants to learn, [here](https://www.linkedin.com/pulse/model-sub-classing-custom-training-loop-from-scratch-tensorflow/?trackingId=z7MZlI8kTTeKPINnkak5Ag%3D%3D) are our recent article where we comprehensively demonstrate model subclassing and custom training loop from scratch in `tensorflow 2.x`. ","metadata":{}},{"cell_type":"code","source":"class CassavaClassifier(tf.keras.Model):\n    def __init__(self, dim, base_efnet):\n        super(CassavaClassifier, self).__init__()\n        # Layer of Block\n        self.Base  = base_efnet[0](input_shape=dim,include_top = False, \n                                   weights=base_efnet[1])\n        # Keras Built-in\n        self.GAP = tf.keras.layers.GlobalAveragePooling2D()\n        # Tail\n        self.DENS = tf.keras.layers.Dense(512, activation=tf.nn.relu)\n        self.DROP = tf.keras.layers.Dropout(0.5)\n        self.OUT  = tf.keras.layers.Dense(5, activation='softmax')\n    \n    def call(self, input_tensor, training=False):\n        # Base Inputs\n        x = self.Base(input_tensor)\n        # Global Weighted Average Poolin\n        x = self.GAP(x)\n        x = self.DENS(x)\n        x = self.DROP(x)\n        return self.OUT(x)\n    \n    # AFAIK: The most convenient method to print model.summary() \n    # in suclassed model\n    def build_graph(self):\n        x = tf.keras.layers.Input(shape=(TrainConfig.IMG_SIZE['0'],\n                                         TrainConfig.IMG_SIZE['0'], 3))\n        return tf.keras.Model(inputs=[x], \n                              outputs=self.call(x))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stratified KFolding","metadata":{}},{"cell_type":"code","source":"skf = StratifiedKFold(n_splits=TrainConfig.FOLDS, shuffle=True, random_state=BaseConfig.SEED)\n    \nfor fold, (trn_idx, val_idx) in enumerate(skf.split(np.arange(df.shape[0]), df.label.values)):\n    df.loc[df.iloc[val_idx].index, 'fold'] = fold\n    \nprint('Class distribution per fold.\\n', df.groupby('fold')['label'].value_counts())","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Mechanism","metadata":{}},{"cell_type":"code","source":"# some placeholder for oof \noof_pred = []; oof_tar = []; oof_names = []; oof_folds = []; oof_val = [] \n\n# compute the extact batch size and step, mainly for validation step\ndef count_data_items(length, b_max):\n    batch_size = sorted([int(length/n) for n in range(1, length+1) \\\n                         if length % n == 0 and length/n <= b_max], reverse=True)[0]  \n    steps  = length / batch_size \n    return batch_size, steps\n\n# learing rate sched\ndef get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n  \n        return lr\n\n    return tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Symetric Cross Entropy loss function for dealing with \n# nosisy labels\nclass SymmetricCrossEntropy(tf.losses.Loss):\n    def __init__(self, alpha=0.1, beta=1.0):\n        '''\n        Paper: https://arxiv.org/abs/1908.06112\n        '''\n        super(SymmetricCrossEntropy, self).__init__()\n        self.alpha = alpha\n        self.beta = beta\n\n    def call(self, y_true, y_pred):\n        ce_loss = tf.reduce_mean(-tf.reduce_sum(y_true * \\\n                    tf.math.log(tf.clip_by_value(y_pred, 1e-7, 1.0)), \n                    axis = -1))\n\n        rce_loss = tf.reduce_mean(-tf.reduce_sum(y_pred * \\\n                   tf.math.log(tf.clip_by_value(y_true, 1e-4, 1.0)), \n                   axis = -1))\n\n        return self.alpha*ce_loss + self.beta*rce_loss\n    \n# create an instance\nSCELoss = SymmetricCrossEntropy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nFollowing class (CassavaTrainer) contians \n\n- __init__: initiate basic parameter and compile the model and \n            print out basic training information \n  \n- fold_generator: based on train_index and valid_index, it \n                  will create training fold and validation fold set\n\n- callbacks: Contains necessary callbacks.\n\n- fold_training: Actual training. Also OOF with TTA over the validation set. \n'''\nclass CassavaTrainer:\n    def __init__(self, fold, trn_idx, val_idx):\n        tf.keras.backend.clear_session()\n        gc.collect()\n        \n        self.fold = fold\n        self.trn_idx = trn_idx\n        self.val_idx = val_idx\n        \n        # building the complete model and compile\n        self.model = CassavaClassifier((TrainConfig.IMG_SIZE[str(self.fold)], \n                                        TrainConfig.IMG_SIZE[str(self.fold)], 3),\n                                       TrainConfig.BASE_NETS[str(self.fold)]) \n        self.model.compile(\n            loss      = tf.keras.losses.CategoricalCrossentropy(label_smoothing=TrainConfig.LABEL_SMOOTH),\n            metrics   = tf.keras.metrics.CategoricalAccuracy(),\n            optimizer = tf.keras.optimizers.Adam(learning_rate=TrainConfig.LR_RATE[str(self.fold)]))\n    \n        # print out the model params\n        trainable_count = np.sum([K.count_params(w) \\\n                                  for w in self.model.trainable_weights]) \n        non_trainable_count = np.sum([K.count_params(w) \\\n                                      for w in self.model.non_trainable_weights])\n        \n        print('[INFO]: OOF Fold No. {}'.format(self.fold))\n        print('[INFO]: Model Build Successfully.')\n        print('[INFO]: Model {} - Image Size {} - Batch Size {}'.\\\n              format(TrainConfig.BASE_NETS[str(self.fold)][0].__name__, \n              TrainConfig.IMG_SIZE[str(self.fold)], \n                     TrainConfig.BATCH_SIZE[str(self.fold)]))\n        print('[INFO]: Initial Learning Rate: {} - Class Balancing: {}'.\\\n              format(TrainConfig.LR_RATE[str(self.fold)],\n                     TrainConfig.CLS_BLN_WGS))\n        print('Total params: {:,}'.format(trainable_count + non_trainable_count))\n        print('Trainable params: {:,}'.format(trainable_count))\n        print('Non-trainable params: {:,}'.format(non_trainable_count))\n        \n    \n    def fold_generator(self):\n        # for way one - data generator\n        train_labels = df.iloc[self.trn_idx].reset_index(drop=True) \n        val_labels   = df.iloc[self.val_idx].reset_index(drop=True) \n\n        # training generator  \n        train_generator = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                                           train_labels, TrainConfig.BATCH_SIZE[str(self.fold)],\n                                           BaseConfig.SEED,\n                                           (TrainConfig.IMG_SIZE[str(self.fold)], \n                                            TrainConfig.IMG_SIZE[str(self.fold)], 3),\n                                           shuffle = True, is_train = True, \n                                           use_mixup = TrainConfig.MixUp, \n                                           use_cutmix = TrainConfig.CutMix,\n                                           transform = albu_transforms_train(TrainConfig.IMG_SIZE[str(self.fold)]))\n        \n        # validation generator: no shuffle , not augmentation\n        valid_batch, valid_step = count_data_items(len(val_labels), \n                                                   TrainConfig.BATCH_SIZE[str(self.fold)])\n        val_generator = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                                         val_labels, valid_batch, \n                                         BaseConfig.SEED,\n                                         (TrainConfig.IMG_SIZE[str(self.fold)], \n                                          TrainConfig.IMG_SIZE[str(self.fold)], 3), \n                                         shuffle = False, is_train = True, \n                                         transform = albu_transforms_valid(TrainConfig.IMG_SIZE[str(self.fold)]))\n        \n        return train_generator, val_generator, train_labels, val_labels, valid_step, valid_batch\n\n    def callbacks(self):\n        # model check point, save best weight based on val loss\n        checkpoint = tf.keras.callbacks.ModelCheckpoint('./fold_{}.h5'.format(self.fold),\n                                                        monitor='val_loss',\n                                                        verbose= 0,save_best_only=True,\n                                                        mode= 'min',save_weights_only=True)\n        # save model history information in csv file \n        csv_logger = tf.keras.callbacks.CSVLogger('./history_{}.csv'.format(self.fold))\n        \n        # reduce learning rate based on val loss\n        # if we use LearningRateScheduler, we won't use reduceLROnPlat\n        reduceLROnPlat = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss',\n                                                              factor=0.3, patience=2,\n                                                              verbose=0, mode='auto',\n                                                              epsilon=0.0001, cooldown=1, \n                                                              min_lr=0.00001)\n        return [checkpoint,\n                csv_logger,\n                reduceLROnPlat]\n    \n\n    def fold_training(self):\n        # call each fold set\n        train_generator, val_generator, train_labels, \\\n        val_labels, valid_step, valid_batch = self.fold_generator()\n\n        # invoke callbacks functions\n        steps_per_epoch  = np.ceil(float(len(train_labels)) \\\n                                   / float(TrainConfig.BATCH_SIZE[str(self.fold)])) \n        validation_steps = valid_step \n        \n        # try balancing \n        cls_wgts = class_weight.compute_class_weight(\"balanced\",\n                                                  sorted(train_labels.label.unique()), \n                                                  train_labels.label.values)\n        cls_wgts = {i : cls_wgts[i] for i, label \\\n                    in enumerate(sorted(train_labels.label.unique()))}\n        \n        # fit generator\n        history = self.model.fit(\n            train_generator,\n            steps_per_epoch  = steps_per_epoch,\n            validation_data  = val_generator,\n            validation_steps = validation_steps,\n            epochs = TrainConfig.EPOCH, \n            class_weight = None if not TrainConfig.CLS_BLN_WGS else cls_wgts,\n            verbose = TrainConfig.VERBOSITY, callbacks = self.callbacks(), \n            workers = TrainConfig.WORKERS, use_multiprocessing = TrainConfig.MULTIPROCESS\n        )\n    \n        print('[INFO]: Loading Best Model...')\n        self.model.build((None, *(TrainConfig.IMG_SIZE[str(self.fold)],\n                             TrainConfig.IMG_SIZE[str(self.fold)], 3)))\n        self.model.load_weights('./fold_{}.h5'.format(self.fold))\n        \n        # appending / saving \n        oof_names.append(val_labels.image_id.tolist())\n        oof_tar.append(val_labels.label.tolist()) \n        oof_folds.append(np.ones_like(oof_tar[-1], dtype='int8')*self.fold ) \n        oof_val.append(np.max(history.history['val_categorical_accuracy']))\n        \n        print('[INFO]: Predicting TTA over OOF...')\n        tta_preds = []\n        for _ in range(TrainConfig.TTA):\n            # using the tta augmentation on validation set in prediction time \n            tta_val_generator = CassavaGenerator(BaseConfig.TRAIN_IMG_PATH, \n                                                 val_labels, valid_batch,\n                                                 BaseConfig.SEED,\n                                                 (TrainConfig.IMG_SIZE[str(self.fold)], \n                                                  TrainConfig.IMG_SIZE[str(self.fold)], 3), \n                                                 shuffle = False, is_train = True, \n                                                 transform = albu_transforms_inference(TrainConfig.IMG_SIZE[str(self.fold)]))\n            # predict and take mean\n            pred = self.model.predict(tta_val_generator, verbose=0)\n            tta_preds.append(pred/TrainConfig.TTA)\n               \n        oof_pred.append(np.argmax(np.sum(tta_preds, axis=0), axis=1))\n        print('[INFO]: OOF Acc without TTA: {} - with TTA: {}'.\\\n              format(oof_val[-1], accuracy_score(oof_tar[-1], oof_pred[-1])))\n        print('[INFO]: OOF Balance Acc with TTA: ', balanced_accuracy_score(oof_tar[-1], oof_pred[-1]))\n        print('\\n'*2)\n        \n        del self.model, train_generator, val_generator,tta_val_generator, train_labels, val_labels ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calling method to run on all folds\nskf = StratifiedKFold(n_splits=TrainConfig.FOLDS, shuffle=True, random_state=BaseConfig.SEED)\nfor each_fold, (trn_idx, val_idx) in enumerate(skf.split(np.arange(df.shape[0]), df.label.values)):\n    tx = CassavaTrainer(each_fold, trn_idx, val_idx)\n    tx.fold_training()\n    break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Calculating OOF Accuracy\n\nHere overall **OOF** is calculated. In order to ensemble of different models which are trained with the **same configuration**, in that case we will need this. Basically **OOF** is used to determine what are the best weights to blend these models with. To do that, please, check these notebooks:\n\n- [Forward Selection OOF Ensemble](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private)\n- [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble)","metadata":{}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF ACCURACY\noof   = np.concatenate(oof_pred)\ntrue  = np.concatenate(oof_tar)\nnames = np.concatenate(oof_names)\nfolds = np.concatenate(oof_folds)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(\n    image_id = names, \n    target   = true, \n    pred     = oof, \n    fold     = folds\n))\n\nprint('Overall OOF Acc: ', accuracy_score(true, oof))\nprint('Overall OOF Balance Acc: ', balanced_accuracy_score(true, oof))\n\ndf_oof.to_csv('oof.csv', index=False)\ndf_oof.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test Gens and Inference\n\nHere, **TTA** will be applied over each trained model followed by a simple ensemble. The applied augmentation is mentioned in the **Augmentation** section. It's just a random selection though, we need to do experiment to get the most optimized ones. \n\n```\n- Test Time Augmentation (TTA)\n- Ensembling\n```","metadata":{}},{"cell_type":"code","source":"df_test = pd.DataFrame({\n    'image_id': os.listdir(BaseConfig.TEST_IMG_PATH),\n})\ndf_test.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_check_points = sorted(glob('./*.h5'))\nprint(f'Found {len(model_check_points)} Weights Files.')\n\n# container for predictive probabilities\ntarget = []\n\nfor i, each_check_points in enumerate(model_check_points):\n    # define and load weights\n    model = CassavaClassifier((TrainConfig.IMG_SIZE[str(i)], \n                               TrainConfig.IMG_SIZE[str(i)], 3), \n                              TrainConfig.BASE_NETS[str(i)])\n    model.build((None, *(TrainConfig.IMG_SIZE[str(i)],\n                         TrainConfig.IMG_SIZE[str(i)], 3)))\n    model.load_weights(each_check_points)\n    \n    tta_preds = []\n    for _ in range(TrainConfig.TTA):\n        test_batch, test_step = count_data_items(len(df_test), \n                                                 TrainConfig.BATCH_SIZE[str(i)])\n        test_generator = CassavaGenerator(BaseConfig.TEST_IMG_PATH, df_test,\n                                          test_batch, BaseConfig.SEED,\n                                          (TrainConfig.IMG_SIZE[str(i)], \n                                           TrainConfig.IMG_SIZE[str(i)], 3), \n                                          shuffle = False, is_train = False,\n                                          transform = albu_transforms_inference(TrainConfig.IMG_SIZE[str(i)]))\n        # predict and take mean\n        pred = model.predict(test_generator, verbose=1)\n        tta_preds.append(pred/TrainConfig.TTA)\n    \n    tta_preds = np.sum(tta_preds, axis=0)\n    target.append(tta_preds/len(model_check_points))\n\n    del model, test_generator\n    gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test.loc[:, 'label'] = np.argmax(np.sum(target, axis=0), axis=1)\ndf_test.to_csv(\"submission.csv\", index=False)\ndf_test.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}