{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Version Update [4]**\n\nHopefully tha last update. Finally, we've added `Test Time Augmentation` in this version. [qubvel/tta_wrapper](https://github.com/qubvel/tta_wrapper) is used but need to convert it to `tf 2.x` compatible. Below is the `tensorflow 2.x` compatible of `qubvel/tta_wrapper`. \n\n- [TF.Keras TTA Wrapper](https://www.kaggle.com/ipythonx/keras-tta-wrapper)\n\n\n---\n\n**Version Update [3]**\n\nIn this version, couple of things have been changed. Now, I've used merge data set from [Alex](https://www.kaggle.com/shonenkov). Unlike the previous version, I've trained the model on 3 folds on kaggle kernel GPU with **256** resized samples around **5** epochs each fold; and rank the predicted data followed by an average ensemble. Apart from this, I have also changed following:\n\n```\n- Optimizer and Loss Function\n    - RAdam with OoverHead [Ranger]\n    - Binary Crossentropy with label smoothing = 0.05\n- Image Modeling\n    - EfficientNet B3 + Generalized Mean Pool + Dense + BN + Drop + Sigmoid\n```\n\nAnyway, this notebook is pretty long as we include both `image modeling` and `tabular modeling` at a same place. Personally we like to create single notebook for each competition, and try to put everything possible in an organized ways and avoid creating new notebook unless its too necessary. So,sorry about that. But I can ensure you will find this work helpful.\n\n---\n\n**Version Update [2]**\n\nConduct **Tabular Modeling**. A new tabular data learning architecture - [`TabNet`](https://arxiv.org/pdf/1908.07442.pdf) is used. To get the optimal hyper-param, **RandomizedSearchCV** is also used in general.\n\n---\n\n**Version Update [1]**\n\nUsing `Group-Stratify-KFold` instead of `Stratify-KFold`.\n\n---\n\n# Melanoma Classification\n\nHi, <br>\nThis is a baseline starter in `tf.keras` for this **Melanoma Classification** problem. Here `jpeg` samples will be used for modeling the ene-to-end deep learning pipelines. Main approaches are:\n\n```\n- Group-Stratify-KFold\n- 224 Resized Samples\n- Image Augmentation\n    - AugMix\n    - MixUp\n    - In General (CoarseDropout, Fliplr, Flipud, Affine, ... )\n- Optimizer and Loss Function\n    - Adam with OoverHead\n    - Focal Loss\n- Image Modeling\n    - EfficientNet B0 + GAP + Sigmoid\n- Tabular Modeling\n    - TabNet\n- TTA + Ensemble\n```\n\nAccording to the problem space, It's super important to stratify the samples by `patient_groups`. Apart from previous version, this time we'll be using `Stratify by Group`. The code implementation of it for this comp. is taken from [this kernel](https://www.kaggle.com/shonenkov/merge-external-data) of [Alex Shonenkov](https://www.kaggle.com/shonenkov).\n\nAbout the augmentation, [`AugMix`](https://arxiv.org/abs/1912.02781) augmentation is used which implemented based on [**albumentation**](https://github.com/albumentations-team/albumentations) and also some general augmentation methods that is implemented in [**img_aug**](https://imgaug.readthedocs.io/en/latest/) library. The [`MixUp`](https://arxiv.org/pdf/1710.09412.pdf), though it's not used in training, but it can be used. \n\nFor optimizing, `Adram(lr=0.001)` optimizers with the [**LookaHead**](https://en.wikipedia.org/wiki/Lookahead#:~:text=Lookahead%20or%20Look%20Ahead%20may,decide%20which%20rule%20to%20use) mechanism is used. This method improves the learning stability and lowers the variance of the optimizer. And lastly, [Focal loss](https://arxiv.org/abs/1708.02002) is used which is extremely useful for classification when we've highly imbalanced classes. It down-weights well-classified examples and focuses on hard examples.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install pytorch-tabnet\n!pip install tensorflow-addons==0.9.1    \n!pip install git+https://github.com/aleju/imgaug.git\n!pip install -U git+https://github.com/qubvel/efficientnet\n!pip install -U git+https://github.com/albumentations-team/albumentations\n!pip install git+https://github.com/qubvel/classification_models.git","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# basic imports\nimport torch\nfrom pytorch_tabnet.tab_model import TabNetClassifier\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom tqdm import tqdm\nfrom glob import glob\nfrom pylab import rcParams\nimport matplotlib.pyplot as plt\nfrom scipy.stats import rankdata\nimport os, gc, cv2, random, warnings, math, sys\nfrom collections import Counter, defaultdict\n\n# sklearn\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import RandomizedSearchCV, train_test_split\n\n# tf \nimport tensorflow as tf\nimport tensorflow_addons as tfa\nimport efficientnet.tfkeras as efn \nimport tensorflow.keras.layers as L\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.utils import Sequence\nfrom tensorflow.keras.callbacks import (ModelCheckpoint, LearningRateScheduler,\n                                        EarlyStopping, ReduceLROnPlateau, CSVLogger)\n\n# augmentation libs [albumentation, img_aug]\nimport albumentations\nimport imgaug.augmenters as iaa\nfrom PIL import Image, ImageOps, ImageEnhance\nfrom albumentations.augmentations import functional as F\nfrom albumentations.core.transforms_interface import ImageOnlyTransform\n\nwarnings.simplefilter('ignore')\nsys.path.insert(0, \"/kaggle/input/keras-tta-wrapper\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Utility**","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def display(df, path):\n    fig = plt.figure(figsize=(20, 16))\n    for class_id in [0, 1]:\n        for i, (idx, row) in enumerate(df.loc[df['target'] == class_id].sample(4, random_state=101).iterrows()):\n            ax = fig.add_subplot(4, 4, class_id * 4 + i + 1, xticks=[], yticks=[])\n\n            image = cv2.imread(os.path.join(path + '{}.jpg'.format(row['image_name'])))\n            ax.set_title('Label: {}'.format(class_id) )\n            plt.imshow(image)\n    \n    \n# helper function to plot sample from dataloader/generator \ndef plot_imgs(dataset_show, is_train=True):\n    rcParams['figure.figsize'] = 30,20\n    for i in range(2):\n        f, ax = plt.subplots(1,5)\n        for p in range(5):\n            idx = np.random.randint(0, len(dataset_show))\n            if is_train:\n                img, label = dataset_show[idx]\n            else:\n                img = dataset_show[idx]\n            ax[p].grid(False)\n            ax[p].imshow(img[0])\n            ax[p].set_title(idx)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 48\ndim = 256, 256\n\nroot = '../input/melanoma-merged-external-data-512x512-jpeg/'\ndf = pd.read_csv(os.path.join(root, 'marking.csv'))\n\ntrain_images = os.path.join(root, '512x512-dataset-melanoma/512x512-dataset-melanoma/')\ntest_images  = os.path.join(root, '512x512-test/512x512-test/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# for reproducibiity\ndef seed_all(s):\n    random.seed(s)\n    np.random.seed(s)\n    tf.random.set_seed(s)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['PYTHONHASHSEED'] = str(s) \n    \n# seed all\nseed_all(101)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Overral View**","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(df.info())\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# each patient has more than one samples\ndf.rename({\"image_id\": \"image_name\"},axis='columns',inplace =True) \nprint('Unique Image Id  : ', len(df.image_name.unique()))\nprint('Unique Patient Id: ',len(df.patient_id.unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Significant Class Imbalance**","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# significant imbalance\nsns.set_style('darkgrid')\nsns.countplot(df.target)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Displaying Samples**","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display(df, train_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Group Stratified KFolding","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def stratified_group_k_fold(X, y, groups, k, seed=None):\n    \"\"\" https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation \"\"\"\n    labels_num = np.max(y) + 1\n    y_counts_per_group = defaultdict(lambda: np.zeros(labels_num))\n    y_distr = Counter()\n    for label, g in zip(y, groups):\n        y_counts_per_group[g][label] += 1\n        y_distr[label] += 1\n\n    y_counts_per_fold = defaultdict(lambda: np.zeros(labels_num))\n    groups_per_fold = defaultdict(set)\n\n    def eval_y_counts_per_fold(y_counts, fold):\n        y_counts_per_fold[fold] += y_counts\n        std_per_label = []\n        for label in range(labels_num):\n            label_std = np.std([y_counts_per_fold[i][label] / y_distr[label] for i in range(k)])\n            std_per_label.append(label_std)\n        y_counts_per_fold[fold] -= y_counts\n        return np.mean(std_per_label)\n    \n    groups_and_y_counts = list(y_counts_per_group.items())\n    random.Random(seed).shuffle(groups_and_y_counts)\n\n    for g, y_counts in tqdm(sorted(groups_and_y_counts, key=lambda x: -np.std(x[1])), total=len(groups_and_y_counts)):\n        best_fold = None\n        min_eval = None\n        for i in range(k):\n            fold_eval = eval_y_counts_per_fold(y_counts, i)\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = i\n        y_counts_per_fold[best_fold] += y_counts\n        groups_per_fold[best_fold].add(g)\n\n    all_groups = set(groups)\n    for i in range(k):\n        train_groups = all_groups - groups_per_fold[i]\n        test_groups = groups_per_fold[i]\n\n        train_indices = [i for i, g in enumerate(groups) if g in train_groups]\n        test_indices = [i for i, g in enumerate(groups) if g in test_groups]\n\n        yield train_indices, test_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df['patient_id'] = df['patient_id'].fillna(df['image_name'])\ndf['sex'] = df['sex'].fillna('unknown')\ndf['anatom_site_general_challenge'] = df['anatom_site_general_challenge'].fillna('unknown')\ndf['age_approx'] = df['age_approx'].fillna(round(df['age_approx'].mean()))\n\npatient_id_2_count = df[['patient_id', \n                         'image_name']].groupby('patient_id').count()['image_name'].to_dict()\ndf = df.set_index('image_name')\n\ndef get_stratify_group(row):\n    stratify_group = row['sex']\n    stratify_group += f'_{row[\"source\"]}'\n    stratify_group += f'_{row[\"target\"]}'\n    patient_id_count = patient_id_2_count[row[\"patient_id\"]]\n    if patient_id_count > 80:   stratify_group += f'_80'\n    elif patient_id_count > 60: stratify_group += f'_60'\n    elif patient_id_count > 50: stratify_group += f'_50'\n    elif patient_id_count > 30: stratify_group += f'_30'\n    elif patient_id_count > 20: stratify_group += f'_20'\n    elif patient_id_count > 10: stratify_group += f'_10'\n    else: stratify_group += f'_0'\n    return stratify_group\n\ndf['stratify_group'] = df.apply(get_stratify_group, axis=1)\ndf['stratify_group'] = df['stratify_group'].astype('category').cat.codes\ndf.loc[:, 'fold'] = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nskf = stratified_group_k_fold(X=df.index, \n                              y=df['stratify_group'], \n                              groups=df['patient_id'], \n                              k=3, seed=101)\n\nfor fold_number, (train_index, val_index) in enumerate(skf):\n    df.loc[df.iloc[val_index].index, 'fold'] = fold_number","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.reset_index(inplace=True)\ndf.to_csv('innat_df.csv', index=False)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Augmentation\n\n`MixUp` is defined in the dataloader, but never used. I found the same implementation in many notebooks. And couldn't catch the original author to give credit. However, rest of the augmentations will be applied randomly in training. =)\n\n```\n- AugMix\n- MixUp\n- Simple Aug (CoarseDropout, Fliplr, Flipud, Affine)\n```\n\n`AugMix` is used from the [official implementation](https://github.com/google-research/augmix) and is called using `albumentation` library. However, we won't follow official normalization [here](https://github.com/google-research/augmix/blob/master/augment_and_mix.py#L25-L30) function, as we've already used normalization in the data loader. Initially I observed that `AugMix` is much more promising than `MixUp` and `CutMix`. The intuition of this augmentation is really cool. Here is the augmenting flow from official repo.\n\n![](https://raw.githubusercontent.com/google-research/augmix/master/assets/augmix.gif)\n\n### AugMix","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def int_parameter(level, maxval):\n    \"\"\"Helper function to scale `val` between 0 and maxval .\n    Args:\n    level: Level of the operation that will be between [0, `PARAMETER_MAX`].\n    maxval: Maximum value that the operation can have. This will be scaled to\n      level/PARAMETER_MAX.\n    Returns:\n    An int that results from scaling `maxval` according to `level`.\n    \"\"\"\n    return int(level * maxval / 10)\n\ndef float_parameter(level, maxval):\n    \"\"\"Helper function to scale `val` between 0 and maxval.\n    Args:\n    level: Level of the operation that will be between [0, `PARAMETER_MAX`].\n    maxval: Maximum value that the operation can have. This will be scaled to\n      level/PARAMETER_MAX.\n    Returns:\n    A float that results from scaling `maxval` according to `level`.\n    \"\"\"\n    return float(level) * maxval / 10.\n\n\ndef sample_level(n):\n    return np.random.uniform(low=0.1, high=n)\n\n\ndef autocontrast(pil_img, _):\n    return ImageOps.autocontrast(pil_img)\n\n\ndef equalize(pil_img, _):\n    return ImageOps.equalize(pil_img)\n\n\ndef posterize(pil_img, level):\n    level = int_parameter(sample_level(level), 4)\n    return ImageOps.posterize(pil_img, 4 - level)\n\ndef rotate(pil_img, level):\n    degrees = int_parameter(sample_level(level), 30)\n    if np.random.uniform() > 0.5:\n        degrees = -degrees\n    return pil_img.rotate(degrees, resample=Image.BILINEAR)\n\ndef solarize(pil_img, level):\n    level = int_parameter(sample_level(level), 256)\n    return ImageOps.solarize(pil_img, 256 - level)\n\ndef shear_x(pil_img, level):\n    level = float_parameter(sample_level(level), 0.3)\n    if np.random.uniform() > 0.5:\n        level = -level\n    return pil_img.transform(pil_img.size,\n                           Image.AFFINE, (1, level, 0, 0, 1, 0),\n                           resample=Image.BILINEAR)\n\ndef shear_y(pil_img, level):\n    level = float_parameter(sample_level(level), 0.3)\n    if np.random.uniform() > 0.5:\n        level = -level\n    return pil_img.transform(pil_img.size,\n                           Image.AFFINE, (1, 0, 0, level, 1, 0),\n                           resample=Image.BILINEAR)\n\ndef translate_x(pil_img, level):\n    level = int_parameter(sample_level(level), pil_img.size[0] / 3)\n    if np.random.random() > 0.5:\n        level = -level\n    return pil_img.transform(pil_img.size,\n                           Image.AFFINE, (1, 0, level, 0, 1, 0),\n                           resample=Image.BILINEAR)\n\ndef translate_y(pil_img, level):\n    level = int_parameter(sample_level(level), pil_img.size[0] / 3)\n    if np.random.random() > 0.5:\n        level = -level\n    return pil_img.transform(pil_img.size,\n                           Image.AFFINE, (1, 0, 0, 0, 1, level),\n                           resample=Image.BILINEAR)\n\n# operation that overlaps with ImageNet-C's test set\ndef color(pil_img, level):\n    level = float_parameter(sample_level(level), 1.8) + 0.1\n    return ImageEnhance.Color(pil_img).enhance(level)\n\n# operation that overlaps with ImageNet-C's test set\ndef contrast(pil_img, level):\n    level = float_parameter(sample_level(level), 1.8) + 0.1\n    return ImageEnhance.Contrast(pil_img).enhance(level)\n\n# operation that overlaps with ImageNet-C's test set\ndef brightness(pil_img, level):\n    level = float_parameter(sample_level(level), 1.8) + 0.1\n    return ImageEnhance.Brightness(pil_img).enhance(level)\n\n# operation that overlaps with ImageNet-C's test set\ndef sharpness(pil_img, level):\n    level = float_parameter(sample_level(level), 1.8) + 0.1\n    return ImageEnhance.Sharpness(pil_img).enhance(level)\n\n\naugmentations = [\n    autocontrast, equalize, posterize, rotate, solarize, shear_x, shear_y,\n    translate_x, translate_y\n]\n\ndef normalize(image):\n    \"\"\"Normalize input image channel-wise to zero mean and unit variance.\"\"\"\n    return image - 127\n\ndef apply_op(image, op, severity):\n    #   image = np.clip(image, 0, 255)\n    pil_img = Image.fromarray(image)  # Convert to PIL.Image\n    pil_img = op(pil_img, severity)\n    return np.asarray(pil_img)\n\ndef augment_and_mix(image, severity=3, width=3, depth=-1, alpha=1.):\n    \"\"\"Perform AugMix augmentations and compute mixture.\n    Args:\n    image: Raw input image as float32 np.ndarray of shape (h, w, c)\n    severity: Severity of underlying augmentation operators (between 1 to 10).\n    width: Width of augmentation chain\n    depth: Depth of augmentation chain. -1 enables stochastic depth uniformly\n      from [1, 3]\n    alpha: Probability coefficient for Beta and Dirichlet distributions.\n    Returns:\n    mixed: Augmented and mixed image.\n    \"\"\"\n    ws = np.float32(\n      np.random.dirichlet([alpha] * width))\n    m = np.float32(np.random.beta(alpha, alpha))\n\n    mix = np.zeros_like(image).astype(np.float32)\n    for i in range(width):\n        image_aug = image.copy()\n        depth = depth if depth > 0 else np.random.randint(1, 4)\n        for _ in range(depth):\n            op = np.random.choice(augmentations)\n            image_aug = apply_op(image_aug, op, severity)\n            \n        # Preprocessing commutes since all coefficients are convex\n        mix += ws[i] * image_aug\n    mixed = (1 - m) * image + m * mix\n    return mixed","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AugMix(ImageOnlyTransform):\n\n    def __init__(self, severity=3, width=3, depth=-1, \n                 alpha=1., always_apply=False, p=0.5):\n        super().__init__(always_apply, p)\n        self.severity = severity\n        self.width = width\n        self.depth = depth\n        self.alpha = alpha\n\n    def apply(self, image, **params):\n        image = augment_and_mix(image,\n            self.severity,\n            self.width,\n            self.depth,\n            self.alpha)\n        return image\n    \n    \n# augmix augmentation using albumentation\nalbu_transforms_train = albumentations.Compose([\n    AugMix(severity=3, width=3, alpha=1., p=1.),\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Simple Augmentation\n\n[`img_aug`](https://github.com/aleju/imgaug) which is another popular augmentation library. It's used to apply some very basic augmentation methods such as: `CoarseDropout, Fliplr, Flipud, and Affine`. The `GridMask` augmentation can be applied easily but `CoarseDropout` simply do the same job. =)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# general augmentation methods using img_aug library\niaa_train_transform = iaa.Sequential([\n    iaa.OneOf([ ## rotate\n        iaa.Affine(rotate=0),\n        iaa.Affine(rotate=90),\n        iaa.Affine(rotate=180),\n        iaa.Affine(rotate=270),\n    ]),\n    iaa.CoarseDropout((0.0, 0.05), size_percent=(0.02, 0.25)),\n    iaa.Fliplr(0.5),\n    iaa.Flipud(0.5),\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Melanoma Samples Generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class SIMelanomaGenerate(Sequence):\n    def __init__(self, data, batch_size, \n                 dim, mixcutup=False, shuffle=False, \n                 is_train=False, transform=False):\n        '''initiate params\n        data      : dataframe\n        batch_size: batch size for training\n        dim       : image resolution\n        mixcutup  : True for \"mixup\" augmentation \n        shuffle   : shuffling the data set\n        is_train  : false for test set. \n        transform : Augmentaiton on Train set (AugMix, img_aug*)\n        '''\n        self.dim        = dim\n        self.data       = data\n        self.shuffle    = shuffle\n        self.is_train   = is_train\n        self.mix        = mixcutup\n        self.transform  = transform\n        self.batch_size = batch_size\n        self.label      = self.data['target'] if self.is_train else np.nan\n        self.list_idx   = data.index.values\n        self.indices    = np.arange(len(self.list_idx))\n        self.on_epoch_end()\n\n    def __len__(self):\n        return int(np.ceil(len(self.data) / float(self.batch_size)))\n\n    def __getitem__(self, index):\n        batch_idx = self.indices[index*self.batch_size:(index+1)*self.batch_size]\n        idx = [self.list_idx[k] for k in batch_idx]\n        \n        # placeholder\n        Data   = np.empty((self.batch_size, *self.dim, 3))\n        Target = np.empty((self.batch_size), dtype = np.float32)\n        \n        for i, k in enumerate(idx):\n            # load the image file using cv2\n            if self.is_train:\n                image = cv2.imread(train_images + self.data['image_name'][k] + '.jpg',\n                                  cv2.IMREAD_COLOR) \n            else:\n                image = cv2.imread(test_images + self.data['image_name'][k] + '.jpg',\n                                  cv2.IMREAD_COLOR)\n                \n            # resize and scaling \n            image = cv2.resize(image, self.dim)\n            \n            # all about transformation \n            if self.transform:\n                if np.random.rand() < 0.9:\n                    # image augmentation using \"img_aug\"\n                    image = iaa_train_transform.augment_image(image)\n                else:\n                    # image augmentation using \"albumentation\"\n                    res   = albu_transforms_train(image=image)\n                    image = res['image'].astype(np.float32)\n                \n            # image scaling\n            image = image.astype(np.float32)/255.0 \n        \n            # pass training set or simply test samples \n            if self.is_train:\n                Data[i,:, :, :] =  image\n                Target[i] = self.label.loc[k]\n            else:\n                Data[i,:, :, :] =  image\n                \n            # mix_up augmenation\n            if self.mix:\n                Data, Target = self.mix_up(Data, Target)\n                \n        return Data, Target if self.is_train else Data\n    \n    def on_epoch_end(self):\n        if self.shuffle:\n            np.random.shuffle(self.indices)\n    \n    @staticmethod\n    def mix_up(self, x, y):\n        lam = np.random.beta(0.2, 0.4)\n        ori_index = np.arange(int(len(x)))\n        index_array = np.arange(int(len(x)))\n        np.random.shuffle(index_array)        \n        \n        mixed_x = lam * x[ori_index] + (1 - lam) * x[index_array]\n        mixed_y = lam * y[ori_index] + (1 - lam) * y[index_array]\n        \n        return mixed_x, mixed_y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Sample Fold Generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def fold_generator(fold):\n    # for way one - data generator\n    train_labels = df[df.fold != fold].reset_index(drop=True)\n    val_labels = df[df.fold == fold].reset_index(drop=True)\n\n    # training generator\n    train_generator = SIMelanomaGenerate(train_labels, batch_size, \n                                         dim, mixcutup=False, shuffle = True, \n                                         is_train = True, transform = True)\n\n    # validation generator: no shuffle , not augmentation\n    val_generator = SIMelanomaGenerate(val_labels, batch_size, dim, \n                                       mixcutup=False, shuffle = False, \n                                       is_train = True, transform = None)\n\n    return train_generator, val_generator, train_labels, val_labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Image Modeling","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Generalized mean pool - GeM\ngm_exp = tf.Variable(3.0, dtype = tf.float32)\ndef GeM2d(X):\n    pool = (tf.reduce_mean(tf.abs(X**(gm_exp)), \n                        axis = [1, 2], \n                        keepdims = False) + 1.e-7)**(1./gm_exp)\n    return pool\n\ndef Net(input_dim):\n    input = L.Input(input_dim)\n    efnet = efn.EfficientNetB3(weights='noisy-student',\n                               include_top = False, \n                               input_tensor = input)\n    \n    # GeM\n    lambda_layer = L.Lambda(GeM2d) \n    lambda_layer.trainable_weights.extend([gm_exp])\n    features     = lambda_layer(efnet.output)\n    \n    # tails\n    features     = L.Dense(512, activation='relu',name='relu_act') (features)\n    features     = L.Dropout(0.5)(features)\n    classifier   = L.Dense(1, activation='sigmoid',name='predictions') (features)\n    \n    model        = Model(efnet.input, classifier)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Optimizer and Loss Functions\n\n**Optimizer**: `Adam` is primarily used with the **lookahead mechanism**. The mechanism is proposed by Michael R. Zhang et.al in the paper [Lookahead Optimizer: k steps forward, 1 step back](https://arxiv.org/abs/1907.08610v2). The optimizer iteratively updates two sets of weights: the search directions for weights are chosen by the inner optimizer, while the **slow weights** are updated each k steps based on the directions of the **fast weights** and the two sets of weights are synchronized. This method improves the learning stability and lowers the variance of its inner optimizer.\n\n**Loss**: And for the loss function, Focal loss is used which was first introduced in the [RetinaNet](https://arxiv.org/pdf/1708.02002.pdf) paper. Focal loss is extremely useful for classification when we have highly imbalanced classes. It down-weights well-classified examples and focuses on hard examples. The loss value is much high for a sample which is misclassified by the classifier as compared to the loss value corresponding to a well-classified example.\n\n---\n\n**[Update]:** In this version, I've used Ranger `(RAdam + Lookahead)` and `BCE` with `laabel smoothing`. Pleae re-check the last commit at the very top. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Optimizer\nradam  = tfa.optimizers.RectifiedAdam(lr=0.001)\nranger = tfa.optimizers.Lookahead(radam, \n                                  sync_period=6, \n                                  slow_step_size=0.5)\n\n# Loss\ndef focal_loss(alpha=0.25,gamma=2.0):\n    def focal_crossentropy(y_true, y_pred):\n        bce    = K.binary_crossentropy(y_true, y_pred)\n        y_pred = K.clip(y_pred, K.epsilon(), 1.- K.epsilon())\n        p_t    = (y_true*y_pred) + ((1-y_true)*(1-y_pred))\n        alpha_factor      = y_true*alpha + ((1-alpha)*(1-y_true))\n        modulating_factor = K.pow((1-p_t), gamma)\n        return K.mean(alpha_factor*modulating_factor*bce, axis=-1)\n    return focal_crossentropy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Callbacks Functions**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def Call_Back(each_fold):\n    # model check point\n    checkpoint = ModelCheckpoint('../working/fold_{}.h5'.format(each_fold), \n                                 monitor='val_loss', \n                                 verbose= 0,save_best_only=True, \n                                 mode= 'min',save_weights_only=True)\n    \n    csv_logger = CSVLogger('../working/history_{}.csv'.format(each_fold))\n    \n    reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss',\n                                   factor=0.3, patience=2,\n                                   verbose=1, mode='auto',\n                                   epsilon=0.0001, cooldown=1, min_lr=0.00001)\n    \n    return [checkpoint, csv_logger,reduceLROnPlat]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Iterative Folds Training**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def folds_training(each_fold):\n    # clean space\n    tf.keras.backend.clear_session()\n    gc.collect()\n\n    # call each fold set\n    print('\\nFold No. {}'.format(each_fold))\n    train_generator, val_generator, train_labels, val_labels = fold_generator(each_fold)\n     \n    # Train set fold\n    print('Train Generator: \\n', train_labels.target.value_counts())\n    plot_imgs(train_generator)\n    \n    # Valid set fold\n    print('Valid Generator: \\n', val_labels.target.value_counts())\n    plot_imgs(val_generator) \n    \n    # building the complete model and compile\n    model = Net(input_dim=(*dim,3))\n    model.compile(\n        optimizer = ranger,\n        loss      = [tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05)],\n        metrics   = [tf.keras.metrics.AUC()]\n    )\n    \n    # print out the model params\n    if each_fold == 0: \n        trainable_count = np.sum([K.count_params(w) for w in model.trainable_weights])\n        non_trainable_count = np.sum([K.count_params(w) for w in model.non_trainable_weights])\n        print('Total params: {:,}'.format(trainable_count + non_trainable_count))\n        print('Trainable params: {:,}'.format(trainable_count))\n        print('Non-trainable params: {:,}'.format(non_trainable_count))\n        \n    # invoke callbacks functions\n    callbacks = Call_Back(each_fold)\n    steps_per_epoch = np.ceil(float(len(train_labels)) / float(batch_size))\n    validation_steps = np.ceil(float(len(val_labels)) / float(batch_size))\n\n    # fit generator\n    train_history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=steps_per_epoch,\n        validation_data=val_generator,\n        validation_steps=validation_steps,\n        epochs=5, verbose=1,\n        callbacks=callbacks\n    )\n    \n    del model\n\n# calling method to run on all folds\n[folds_training(each_fold) for each_fold in range(len(df.fold.value_counts()))] ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Gens with TTA and Submission ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = pd.DataFrame({\n    'image_name': os.listdir(test_images)\n})\n\ndf_test['image_name'] = df_test['image_name'].str.split('.').str[0]\nprint(df_test.shape)\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"from tta_wrapper import tta_classification \n\n# calling test generator\nbatch_size = 1\nmodel_check_points = sorted(glob('../working/*.h5'))\n\ntest_generator = SIMelanomaGenerate(df_test, batch_size, dim, \n                                    shuffle   = False, \n                                    is_train  = False, \n                                    transform = None)\n\nfor each_check_points in model_check_points:\n    # define and load weights\n    model  = Net(input_dim=(*dim,3))\n    model.load_weights(each_check_points)\n    \n    # test time augmentation: horizontal flip, vertical flip, rotate etc\n    tta_model = tta_classification(model, h_flip=True, v_flip=True,\n                                   rotation=(90,270), h_shift=(-5, 5), \n                                   merge='mean')\n    \n    # predict and take mean\n    df_test[each_check_points.split('/')[-1]] = tta_model.predict(test_generator,\n                                                              steps=np.ceil(float(len(df_test)) / float(batch_size)),\n                                                              verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# rank the predicted data and average ensemble \ndf_test['target'] = (rankdata(df_test[\"fold_0.h5\"].astype(float).values) + \n                     rankdata(df_test[\"fold_1.h5\"].astype(float).values) +\n                     rankdata(df_test[\"fold_2.h5\"].astype(float).values))\n\ndf_test = df_test[['image_name', 'target']]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"df_test.to_csv('img_submission.csv', index=False)\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tabular Modeling\n\nFor modeling the meta information, a new deep tabular data learning architecture, `TabNet` is used. We've used an open-source **PyTorch** implementation of [`TabNet`](https://github.com/dreamquark-ai/tabnet) model. However, we won't coduct any intensive feature engineering on the meta data, but a baseline demonstration.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"root = '../input/siim-isic-melanoma-classification/'\n\ntrain = df.copy()\ntest  = pd.read_csv(os.path.join(root , 'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('train set:')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('test set')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pre-processing, same as train set (while making group-stratify-kfold)\ntest['sex'] = test['sex'].fillna('unknown')\ntest['age_approx'] = test['age_approx'].fillna(round(test['age_approx'].mean()))\ntest['anatom_site_general_challenge'] = test['anatom_site_general_challenge'].fillna('unknown')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Columns Items\\n')\nprint('train set cols\\n', train.columns.tolist())\nprint('\\ntest set cols\\n',test.columns.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\n\n# feat: sex\ntrain.sex = le.fit_transform(train.sex.astype('str'))\ntest.sex  = le.transform(test.sex.astype('str'))\n\n# feat: anatom site general challenge\ntrain.anatom_site_general_challenge = le.fit_transform(train.anatom_site_general_challenge.astype('str'))\ntest.anatom_site_general_challenge  = le.transform(test.anatom_site_general_challenge.astype('str'))\n\n# apporx age\ntrain.age_approx = le.fit_transform(train['age_approx'].astype('str'))\ntest.age_approx  = le.transform(test['age_approx'].astype('str'))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('train set:')\ntrain = train[['sex', 'age_approx',\n               'anatom_site_general_challenge',\n               'target', 'fold']]\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('test set')\n\ntest = test[['sex','age_approx','anatom_site_general_challenge']]\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Hyper-Param Searching: RandomizedSearchCV\n\nWe've also used **RandomizedSearchCV** to get optimal hyper-param. As its time consuming, we've searched with a small set of combination. Below are the possible combination, feel free to experiment. =)    \n\n```\ngrid = {\n    \"n_a\": [8, 16, 32, 64],\n    \"n_independent\": [1, 2, 3, 4, 5],\n    \"n_shared\": [1, 2, 3, 4, 5], \n    \"n_steps\": [3, 5, 8],\n    \"clip_value\": [1.],\n    \"gamma\": [0.5, 1.3, 1.8, 2.],\n    \"momentum\": [0.1, 0.05, 0.02, 0.005],\n    \"lambda_sparse\": [0.1, 0.01, 0.001],\n    \"lr\": [0.1, 0.02, 0.001, 0.005],\n    \"verbose\": [1]\n}\n```\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if \"Set\" not in train.columns:\n    train[\"Set\"] = np.random.choice(\n        [\"train\", \"valid\"], p=[0.8, 0.2], size=(train.shape[0],)\n    )\n    \ntrain_indices = train[train.Set == \"train\"].index\nvalid_indices = train[train.Set == \"valid\"].index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To tune the hyper-parameter for **TabNet**, we need to make a small wrapper. The fact is, in [`TabNet-PyTorch`](https://github.com/dreamquark-ai/tabnet) implementation, `TabNetClassifier` does not have a `get_params` method for hyperparameter estimation yet.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class TabNetTuner(TabNetClassifier):\n    def fit(self, X, y, *args, **kwargs):\n        \n        self.n_d = self.n_a\n        \n        X_train, X_valid, y_train, y_valid = train_test_split(\n            X, y, test_size=0.2, \n            random_state=0, \n            shuffle=True, \n            stratify=y\n        )\n        \n        return super().fit(\n            X_train,y_train,\n            patience=3,\n            X_valid=X_valid,y_valid=y_valid,\n            num_workers=os.cpu_count(),max_epochs=10, \n            batch_size=2048, virtual_batch_size=512\n        )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define tuner\ntb = TabNetTuner()\n\n# define param\n# list(tb.get_params().keys())\ngrid = {\n    \"n_a\": [16, 32],\n    \"n_independent\": [3, 4, 5],\n    \"n_shared\": [1, 2], \n    \"n_steps\": [3, 5],\n    \"clip_value\": [1.],\n    \"gamma\": [0.5, 2.],\n    \"momentum\": [0.1, 0.005],\n    \"lambda_sparse\": [0.1, 0.01],\n    \"verbose\": [1],\n    'seed':[42]\n}\n\n# define searching object\nrand_search = RandomizedSearchCV(\n    tb, grid,n_iter=5,\n    scoring=\"roc_auc\",n_jobs=1,\n    iid=False,refit=False,\n    cv=[(train_indices, valid_indices)],\n    verbose=1,pre_dispatch=0,\n    random_state=42,\n    return_train_score=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get relevant features\nfeatures = list(set(train.columns.tolist()) - set(['target']) -\n                set([\"Set\"]) - set(['fold']) - set(['stratify_group']))\n\nlabel = ['target']\n\nX = train[features].values\ny = train[label].values.squeeze(1)\n\nprint(features, X.shape)\nprint(label, y.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Start Searching**","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"rand_search.fit(X, y)\nrand_search.best_params_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\n- TabNetClassifier (with best params)\n- Group Stratify-KFold\n```","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tab_net = TabNetClassifier(**rand_search.best_params_)\n\ndef fold_generator(fold):\n    print('Fold Number: ', fold)\n    \n    train_labels = train[train.fold != fold].reset_index(drop=True)\n    val_labels   = train[train.fold == fold].reset_index(drop=True)\n    \n    X_train = train_labels[features].values\n    y_train = train_labels[label].values.squeeze(1)\n    \n    X_val   = val_labels[features].values\n    y_val   = val_labels[label].values.squeeze(1)\n    \n    print(X_train.shape)\n    print(y_train.shape)\n\n    tab_net.fit(X_train,y_train,\n              X_val,y_val,weights=1,\n              max_epochs=10,patience=7, \n              batch_size=2048, virtual_batch_size=512,\n              num_workers=0,drop_last=False)\n    \n    print(\"Validation score: {:<8.5f}\".format(roc_auc_score(y_val,\n                                                            tab_net.predict_proba(X_val)[:,1])))\n    \n    test[fold] = tab_net.predict_proba(test[features].values)[:,1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Calling Folds**","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"[fold_generator(i) for i in range(3)] ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('test set with all folds cols [0:3)')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = pd.read_csv(os.path.join(root , 'sample_submission.csv'))\nsample.target = test.iloc[:, 3:].astype(float).mean(axis=1)\nsample.to_csv('tab_submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\n- [WIP]: GroupStratify KFold : ✔\n- [WIP]: Tabular Modeling : ✔\n- [WIP]: Ensemble +  TTA : ✔\n```","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}