{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\n\nIn this notebook, we illustrate some commonly used augmentations for images in various libraries, and perform inference with both TensorFlow and PyTorch pipelines.\n\nA key augmentation that we will like to illustrate is the needle augmentation, or rather an augmentation that performs an overlay of small artifacts onto the images with a certain probability.\n\nA variant of this approach first appeared in the recent Melanoma Detection competition, created by Roman. I've since adapted it in prior and other competitions such as the Global Wheat Detection competition."},{"metadata":{},"cell_type":"markdown","source":"# Summary of Findings (Using Needle Augmentation)\n\n- PyTorch:\n    - when applied to @yasufuminakama pipeline with ResNeXt50, boosted LB from 0.949 to 0.953 with 5-folds CV, using same hyperparameters\n    - when using 10-folds with same hyperparameters, LB became 0.954\n- TensorFlow-Keras:\n    - when applied to @xhlulu pipeline with EfficientNetB7, LB got boosted from 0.957 to 0.959 using same hyperparameters"},{"metadata":{"papermill":{"duration":0.018987,"end_time":"2020-12-20T14:04:28.720252","exception":false,"start_time":"2020-12-20T14:04:28.701265","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-12-20T14:04:28.759077Z","iopub.status.busy":"2020-12-20T14:04:28.758022Z","iopub.status.idle":"2020-12-20T14:04:40.747551Z","shell.execute_reply":"2020-12-20T14:04:40.748123Z"},"papermill":{"duration":12.012311,"end_time":"2020-12-20T14:04:40.74832","exception":false,"start_time":"2020-12-20T14:04:28.736009","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torchvision\nfrom torchvision import transforms\nimport tensorflow as tf\nimport albumentations as A","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-12-20T14:04:40.79212Z","iopub.status.busy":"2020-12-20T14:04:40.790928Z","iopub.status.idle":"2020-12-20T14:04:40.796909Z","shell.execute_reply":"2020-12-20T14:04:40.796271Z"},"papermill":{"duration":0.030663,"end_time":"2020-12-20T14:04:40.797066","exception":false,"start_time":"2020-12-20T14:04:40.766403","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"DIR = \"../input/ranzcr-clip-catheter-line-classification/\"","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:04:40.84002Z","iopub.status.busy":"2020-12-20T14:04:40.839342Z","iopub.status.idle":"2020-12-20T14:04:40.975917Z","shell.execute_reply":"2020-12-20T14:04:40.976425Z"},"papermill":{"duration":0.162464,"end_time":"2020-12-20T14:04:40.976607","exception":false,"start_time":"2020-12-20T14:04:40.814143","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df_annotations = pd.read_csv(os.path.join(DIR, \"train_annotations.csv\"))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.017597,"end_time":"2020-12-20T14:04:41.011734","exception":false,"start_time":"2020-12-20T14:04:40.994137","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Read Sample Image without Augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:04:41.057145Z","iopub.status.busy":"2020-12-20T14:04:41.056388Z","iopub.status.idle":"2020-12-20T14:04:41.17384Z","shell.execute_reply":"2020-12-20T14:04:41.17437Z"},"papermill":{"duration":0.144205,"end_time":"2020-12-20T14:04:41.174537","exception":false,"start_time":"2020-12-20T14:04:41.030332","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"row = df_annotations.iloc[8]\nimage_path = os.path.join(DIR, \"train\", row[\"StudyInstanceUID\"] + \".jpg\")\nchosen_image = cv2.imread(image_path)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016429,"end_time":"2020-12-20T14:04:41.207043","exception":false,"start_time":"2020-12-20T14:04:41.190614","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Albumentations Augmentations\n\n- Albumentations part adapted from my good friend Hongnan's notebbok in the Global Wheat Detection competition (https://www.kaggle.com/reighns/augmentations-data-cleaning-and-bounding-boxes#Bounding-Boxes-with-Albumentations)\n- Added more augmentations which may be useful\n- Added TensorFlow and Torchvision versions of the augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:04:41.243749Z","iopub.status.busy":"2020-12-20T14:04:41.243022Z","iopub.status.idle":"2020-12-20T14:04:41.255152Z","shell.execute_reply":"2020-12-20T14:04:41.255806Z"},"papermill":{"duration":0.032217,"end_time":"2020-12-20T14:04:41.255974","exception":false,"start_time":"2020-12-20T14:04:41.223757","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"albumentation_list = [A.RandomSunFlare(p=1), \n                      A.RandomFog(p=1), \n                      A.RandomBrightness(p=1),\n                      A.RandomCrop(p=1,height = 512, width = 512), \n                      A.Rotate(p=1, limit=90),\n                      A.RGBShift(p=1), \n                      A.RandomSnow(p=1),\n                      A.HorizontalFlip(p=1), \n                      A.VerticalFlip(p=1), \n                      A.RandomContrast(limit = 0.5,p = 1),\n                      A.HueSaturationValue(p=1,hue_shift_limit=20, sat_shift_limit=30, val_shift_limit=50),\n                      A.Cutout(p=1),\n                      A.Transpose(p=1), \n                      A.JpegCompression(p=1),\n                      A.CoarseDropout(p=1),\n                      A.IAAAdditiveGaussianNoise(loc=0, scale=(2.5500000000000003, 12.75), per_channel=False, p=1),\n                      A.IAAAffine(scale=1.0, translate_percent=None, translate_px=None, rotate=0.0, shear=0.0, order=1, cval=0, mode='reflect', p=1),\n                      A.IAAAffine(rotate=90., p=1),\n                      A.IAAAffine(rotate=180., p=1)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-20T14:04:41.292819Z","iopub.status.busy":"2020-12-20T14:04:41.292064Z","iopub.status.idle":"2020-12-20T14:05:01.665363Z","shell.execute_reply":"2020-12-20T14:05:01.665945Z"},"papermill":{"duration":20.393794,"end_time":"2020-12-20T14:05:01.66615","exception":false,"start_time":"2020-12-20T14:04:41.272356","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_type in albumentation_list:\n    img = aug_type(image = chosen_image)['image']\n    img_matrix_list.append(img)\n\nimg_matrix_list.insert(0,chosen_image)    \n\ntitles_list = [\"Original\",\"RandomSunFlare\",\"RandomFog\",\"RandomBrightness\",\n               \"RandomCrop\",\"Rotate\", \"RGBShift\", \"RandomSnow\",\"HorizontalFlip\", \"VerticalFlip\", \"RandomContrast\",\"HSV\",\n               \"Cutout\",\"Transpose\",\"JpegCompression\",\"CoarseDropout\",\"IAAAdditiveGaussianNoise\",\"IAAAffine\",\"IAAAffineRotate90\",\"IAAAffineRotate180\"]\n\ndef plot_multiple_img(img_matrix_list, title_list, ncols, nrows=5,  main_title=\"\"):\n    fig, myaxes = plt.subplots(figsize=(20, 15), nrows=nrows, ncols=ncols, squeeze=False)\n    fig.suptitle(main_title, fontsize = 30)\n    fig.subplots_adjust(wspace=0.3)\n    fig.subplots_adjust(hspace=0.3)\n    for i, (img, title) in enumerate(zip(img_matrix_list, title_list)):\n        myaxes[i // ncols][i % ncols].imshow(img)\n        myaxes[i // ncols][i % ncols].set_title(title, fontsize=15)\n    plt.show()\n    \nplot_multiple_img(img_matrix_list, titles_list, ncols = 4,main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.035306,"end_time":"2020-12-20T14:05:01.734663","exception":false,"start_time":"2020-12-20T14:05:01.699357","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# OpenCV-Based Custom Augmentations\n\n## Xray Needle Augmentation\n\n- Originally, I implemented this for the recent Global Wheat Detection competition at https://www.kaggle.com/khoongweihao/insect-augmentation-with-efficientdet-d6, inspired by Roman's hair augmentation from the recent Melanoma competition\n- Motivations for this approach:\n    - the intuition was to mimic as close to as possible, actual scenarios in reality that can happen\n    - needles were used as an example. Other small artifacts such as intubation or bullets (due to gunshot wounds) overlayed with the images should suffice. Main idea is to mimic what we can observe in reality, in the form of augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:05:01.822845Z","iopub.status.busy":"2020-12-20T14:05:01.817489Z","iopub.status.idle":"2020-12-20T14:05:01.83938Z","shell.execute_reply":"2020-12-20T14:05:01.839997Z"},"papermill":{"duration":0.065236,"end_time":"2020-12-20T14:05:01.840549","exception":false,"start_time":"2020-12-20T14:05:01.775313","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def NeedleAugmentation(image, n_needles=2, dark_needles=False, p=0.5, needle_folder='../input/xray-needle-augmentation'):\n    aug_prob = random.random()\n    if aug_prob < p:\n        height, width, _ = image.shape  # target image width and height\n        needle_images = [im for im in os.listdir(needle_folder) if 'png' in im]\n\n        for _ in range(1, n_needles):\n            needle = cv2.cvtColor(cv2.imread(os.path.join(needle_folder, random.choice(needle_images))), cv2.COLOR_BGR2RGB)\n            needle = cv2.flip(needle, random.choice([-1, 0, 1]))\n            needle = cv2.rotate(needle, random.choice([0, 1, 2]))\n\n            h_height, h_width, _ = needle.shape  # needle image width and height\n            roi_ho = random.randint(0, abs(image.shape[0] - needle.shape[0]))\n            roi_wo = random.randint(0, abs(image.shape[1] - needle.shape[1]))\n            roi = image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width]\n\n            # Creating a mask and inverse mask \n            img2gray = cv2.cvtColor(needle, cv2.COLOR_BGR2GRAY)\n            ret, mask = cv2.threshold(img2gray, 10, 255, cv2.THRESH_BINARY)\n            mask_inv = cv2.bitwise_not(mask)\n\n            # Now black-out the area of needle in ROI\n            img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n\n            # Take only region of insect from insect image.\n            if dark_needles:\n                img_bg = cv2.bitwise_and(roi, roi, mask=mask_inv)\n                needle_fg = cv2.bitwise_and(img_bg, img_bg, mask=mask)\n            else:\n                needle_fg = cv2.bitwise_and(needle, needle, mask=mask)\n\n            # Put needle in ROI and modify the target image\n            dst = cv2.add(img_bg, needle_fg, dtype=cv2.CV_64F)\n\n            image[roi_ho:roi_ho + h_height, roi_wo:roi_wo + h_width] = dst\n\n    return image","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.034583,"end_time":"2020-12-20T14:05:01.906249","exception":false,"start_time":"2020-12-20T14:05:01.871666","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Original Needles"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:05:01.979495Z","iopub.status.busy":"2020-12-20T14:05:01.978501Z","iopub.status.idle":"2020-12-20T14:05:03.195924Z","shell.execute_reply":"2020-12-20T14:05:03.195073Z"},"papermill":{"duration":1.258056,"end_time":"2020-12-20T14:05:03.196105","exception":false,"start_time":"2020-12-20T14:05:01.938049","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\naug_image = NeedleAugmentation(chosen_image, n_needles=3, dark_needles=False, p=1.0)\nplt.imshow(aug_image)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.03211,"end_time":"2020-12-20T14:05:03.263489","exception":false,"start_time":"2020-12-20T14:05:03.231379","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Dark Needles (Just black, dark, sinister as it is on an Xray...)"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:05:03.33696Z","iopub.status.busy":"2020-12-20T14:05:03.335873Z","iopub.status.idle":"2020-12-20T14:05:04.403602Z","shell.execute_reply":"2020-12-20T14:05:04.402948Z"},"papermill":{"duration":1.106903,"end_time":"2020-12-20T14:05:04.40374","exception":false,"start_time":"2020-12-20T14:05:03.296837","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\naug_image = NeedleAugmentation(chosen_image, n_needles=3, dark_needles=True, p=1.0)\nplt.imshow(aug_image)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.035575,"end_time":"2020-12-20T14:05:04.473247","exception":false,"start_time":"2020-12-20T14:05:04.437672","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# PyTorch-Based Augmentations (Torchvision)"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:05:04.553407Z","iopub.status.busy":"2020-12-20T14:05:04.552641Z","iopub.status.idle":"2020-12-20T14:05:04.555778Z","shell.execute_reply":"2020-12-20T14:05:04.555028Z"},"papermill":{"duration":0.046283,"end_time":"2020-12-20T14:05:04.555915","exception":false,"start_time":"2020-12-20T14:05:04.509632","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"torch_trans_list = [transforms.CenterCrop((178, 178)),\n                    transforms.Resize(128),\n                    transforms.RandomRotation(45),\n                    transforms.RandomAffine(35),\n                    transforms.RandomCrop(128),\n                    transforms.RandomHorizontalFlip(p=1),\n                    transforms.RandomPerspective(p=1),\n                    transforms.RandomVerticalFlip(p=1)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-20T14:05:04.634521Z","iopub.status.busy":"2020-12-20T14:05:04.633645Z","iopub.status.idle":"2020-12-20T14:05:14.729529Z","shell.execute_reply":"2020-12-20T14:05:14.730116Z"},"papermill":{"duration":10.139954,"end_time":"2020-12-20T14:05:14.730294","exception":false,"start_time":"2020-12-20T14:05:04.59034","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_type in torch_trans_list:\n    # convert to tensor\n    chosen_image = cv2.imread(image_path)\n    chosen_tensor = transforms.Compose([transforms.ToTensor()])(chosen_image)\n    chosen_tensor = transforms.Compose([aug_type])(chosen_tensor)\n    trans_img = transforms.ToPILImage()(chosen_tensor)\n    img_matrix_list.append(trans_img)\n\nimg_matrix_list.insert(0, chosen_image)    \n\ntitles_list = [\"Original\",\"CenterCrop\",\"Resize\",\"RandomRotation\",\"RandomAffine\",\"RandomCrop\",\"RandomHorizontalFlip\",\"RandomPerspective\",\n               \"RandomVerticalFlip\"]\n\nplot_multiple_img(img_matrix_list, titles_list, ncols = 3, nrows=3, main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.048207,"end_time":"2020-12-20T14:05:14.826219","exception":false,"start_time":"2020-12-20T14:05:14.778012","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# TensorFlow-Based Augmentations"},{"metadata":{"execution":{"iopub.execute_input":"2020-12-20T14:05:14.925981Z","iopub.status.busy":"2020-12-20T14:05:14.925172Z","iopub.status.idle":"2020-12-20T14:05:16.233672Z","shell.execute_reply":"2020-12-20T14:05:16.23296Z"},"papermill":{"duration":1.360088,"end_time":"2020-12-20T14:05:16.233797","exception":false,"start_time":"2020-12-20T14:05:14.873709","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"chosen_image = cv2.imread(image_path)\n\ntf_trans_list = [\n    tf.image.rot90(chosen_image, k=1), # 90 degrees counter-clockwise\n    tf.image.rot90(chosen_image, k=2), # 180 degrees counter-clockwise\n    tf.image.rot90(chosen_image, k=3), # 270 degrees counter-clockwise\n    tf.image.random_brightness(chosen_image, 0.5), \n    tf.image.random_contrast(chosen_image, 0.2, 0.5), \n    tf.image.random_flip_left_right(chosen_image, seed=42),\n    tf.image.random_flip_up_down(chosen_image, seed=42),\n    tf.image.random_hue(chosen_image, 0.5),\n    tf.image.random_jpeg_quality(chosen_image, 35, 50), \n    tf.image.random_saturation(chosen_image, 5, 10), \n    tf.image.transpose(chosen_image),\n]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"execution":{"iopub.execute_input":"2020-12-20T14:05:16.341308Z","iopub.status.busy":"2020-12-20T14:05:16.340265Z","iopub.status.idle":"2020-12-20T14:05:26.081379Z","shell.execute_reply":"2020-12-20T14:05:26.081929Z"},"papermill":{"duration":9.803288,"end_time":"2020-12-20T14:05:26.082089","exception":false,"start_time":"2020-12-20T14:05:16.278801","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"img_matrix_list = []\nbboxes_list = []\nfor aug_image in tf_trans_list:\n    img_matrix_list.append(aug_image)\n\nimg_matrix_list.insert(0, chosen_image)    \n\ntitles_list = [\"Original\",\"Rotate90\",\"Rotate180\",\"Rotate270\",\"RandomBrightness\",\"RandomContrast\",\"RandomLeftRightFlip\",\"RandomUpDownFlip\",\n               \"RandomHue\",\"RandomJPEGQuality\",\"RandomSaturation\",\"Transpose\"]\n\nplot_multiple_img(img_matrix_list, titles_list, ncols = 3, nrows=4, main_title=\"Different Types of Augmentations with Albumentations\")","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.055681,"end_time":"2020-12-20T14:05:26.196813","exception":false,"start_time":"2020-12-20T14:05:26.141132","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Model Pipelines with Needle Augmentations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# select which to inference from\nINFER_MODE = 'TF' # or 'TORCH'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## PyTorch\n\n- An example inference using needle augmentation built upon @yasufuminakama inference pipeline at https://www.kaggle.com/yasufuminakama/ranzcr-resnext50-32x4d-starter-inference\n- 10 folds were used, LB 0.954 (original baseline by yasufuminakama had LB 0.949 with 5-folds, no needle aug)\n- can't seem to perform inference here as it is, having out of memory issues"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/timm-package/timm-0.1.26-py3-none-any.whl","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"if INFER_MODE == 'TORCH':\n    import os\n    import sys\n    sys.path.append('../input/pytorch-image-models/pytorch-image-models-master')\n\n    MODEL_DIR = '../input/ranzcr-pytorch-weights/'\n    OUTPUT_DIR = './'\n    if not os.path.exists(OUTPUT_DIR):\n        os.makedirs(OUTPUT_DIR)\n\n    TEST_PATH = '../input/ranzcr-clip-catheter-line-classification/test'","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.057212,"end_time":"2020-12-20T14:05:26.314165","exception":false,"start_time":"2020-12-20T14:05:26.256953","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class CFG:\n    debug=False\n    num_workers=4\n    model_name='resnext50_32x4d'\n    size=600\n    batch_size=64\n    seed=42\n    target_size=11\n    target_cols=['ETT - Abnormal', 'ETT - Borderline', 'ETT - Normal',\n                 'NGT - Abnormal', 'NGT - Borderline', 'NGT - Incompletely Imaged', 'NGT - Normal', \n                 'CVC - Abnormal', 'CVC - Borderline', 'CVC - Normal',\n                 'Swan Ganz Catheter Present']\n    n_fold=10\n    trn_fold=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"if INFER_MODE == 'TORCH':\n    import math\n    import time\n    import random\n    import shutil\n    from pathlib import Path\n    from contextlib import contextmanager\n    from collections import defaultdict, Counter\n\n    import scipy as sp\n    import numpy as np\n    import pandas as pd\n\n    from sklearn import preprocessing\n    from sklearn.metrics import roc_auc_score\n    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold\n\n    from tqdm.auto import tqdm\n    from functools import partial\n\n    import cv2\n    from PIL import Image\n\n    from matplotlib import pyplot as plt\n\n    import torch\n    import torch.nn as nn\n    import torch.nn.functional as F\n    from torch.optim import Adam, SGD\n    import torchvision.models as models\n    from torch.nn.parameter import Parameter\n    from torch.utils.data import DataLoader, Dataset\n    from torch.optim.lr_scheduler import CosineAnnealingWarmRestarts, CosineAnnealingLR, ReduceLROnPlateau\n\n    from albumentations import (\n        Compose, OneOf, Normalize, Resize, RandomResizedCrop, RandomCrop, HorizontalFlip, VerticalFlip, \n        RandomBrightness, RandomContrast, RandomBrightnessContrast, Rotate, ShiftScaleRotate, Cutout, \n        IAAAdditiveGaussianNoise, Transpose\n        )\n    from albumentations.pytorch import ToTensorV2\n    from albumentations import ImageOnlyTransform\n\n    import timm\n\n    from torch.cuda.amp import autocast, GradScaler\n\n    import warnings \n    warnings.filterwarnings('ignore')\n\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n\n    def get_score(y_true, y_pred):\n        scores = []\n        for i in range(y_true.shape[1]):\n            score = roc_auc_score(y_true[:,i], y_pred[:,i])\n            scores.append(score)\n        avg_score = np.mean(scores)\n        return avg_score, scores\n\n\n    def get_result(result_df):\n        preds = result_df[[f'pred_{c}' for c in CFG.target_cols]].values\n        labels = result_df[CFG.target_cols].values\n        score, scores = get_score(labels, preds)\n        LOGGER.info(f'Score: {score:<.4f}  Scores: {np.round(scores, decimals=4)}')\n\n\n    @contextmanager\n    def timer(name):\n        t0 = time.time()\n        LOGGER.info(f'[{name}] start')\n        yield\n        LOGGER.info(f'[{name}] done in {time.time() - t0:.0f} s.')\n\n\n    def init_logger(log_file=OUTPUT_DIR+'inference.log'):\n        from logging import getLogger, INFO, FileHandler,  Formatter,  StreamHandler\n        logger = getLogger(__name__)\n        logger.setLevel(INFO)\n        handler1 = StreamHandler()\n        handler1.setFormatter(Formatter(\"%(message)s\"))\n        handler2 = FileHandler(filename=log_file)\n        handler2.setFormatter(Formatter(\"%(message)s\"))\n        logger.addHandler(handler1)\n        logger.addHandler(handler2)\n        return logger\n\n    LOGGER = init_logger()\n\n    def seed_torch(seed=42):\n        random.seed(seed)\n        os.environ['PYTHONHASHSEED'] = str(seed)\n        np.random.seed(seed)\n        torch.manual_seed(seed)\n        torch.cuda.manual_seed(seed)\n        torch.backends.cudnn.deterministic = True\n\n    seed_torch(seed=CFG.seed)\n\n    train = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/train.csv')\n    folds = train.copy()\n    Fold = GroupKFold(n_splits=CFG.n_fold)\n    groups = folds['PatientID'].values\n    for n, (train_index, val_index) in enumerate(Fold.split(folds, folds[CFG.target_cols], groups)):\n        folds.loc[val_index, 'fold'] = int(n)\n    folds['fold'] = folds['fold'].astype(int)\n    display(folds.groupby('fold').size())\n\n    test = pd.read_csv('../input/ranzcr-clip-catheter-line-classification/sample_submission.csv')\n\n    if CFG.debug:\n        test = test.head()\n\n    class TestDataset(Dataset):\n        def __init__(self, df, transform=None):\n            self.df = df\n            self.file_names = df['StudyInstanceUID'].values\n            self.transform = transform\n\n        def __len__(self):\n            return len(self.df)\n\n        def __getitem__(self, idx):\n            file_name = self.file_names[idx]\n            file_path = f'{TEST_PATH}/{file_name}.jpg'\n            image = cv2.imread(file_path)\n            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n            if self.transform:\n                augmented = self.transform(image=image)\n                image = augmented['image']\n            return image\n\n    def get_transforms(*, data):\n\n        if data == 'train':\n            return Compose([\n                Resize(CFG.size, CFG.size),\n                Normalize(\n                    mean=[0.485, 0.456, 0.406],\n                    std=[0.229, 0.224, 0.225],\n                ),\n                ToTensorV2(),\n            ])\n\n        elif data == 'valid':\n            return Compose([\n                Resize(CFG.size, CFG.size),\n                Normalize(\n                    mean=[0.485, 0.456, 0.406],\n                    std=[0.229, 0.224, 0.225],\n                ),\n                ToTensorV2(),\n            ])\n\n    class CustomResNext(nn.Module):\n        def __init__(self, model_name='resnext50_32x4d', pretrained=False):\n            super().__init__()\n            self.model = timm.create_model(model_name, pretrained=pretrained)\n            n_features = self.model.fc.in_features\n            self.model.fc = nn.Linear(n_features, CFG.target_size)\n\n        def forward(self, x):\n            x = self.model(x)\n            return x\n\n    def inference(model, states, test_loader, device):\n        model.to(device)\n        tk0 = tqdm(enumerate(test_loader), total=len(test_loader))\n        probs = []\n        for i, (images) in tk0:\n            images = images.to(device)\n            avg_preds = []\n            for state in states:\n                model.load_state_dict(state['model'])\n                model.eval()\n                with torch.no_grad():\n                    y_preds = model(images)\n                avg_preds.append(y_preds.sigmoid().to('cpu').numpy())\n            avg_preds = np.mean(avg_preds, axis=0)\n            probs.append(avg_preds)\n        probs = np.concatenate(probs)\n        return probs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if INFER_MODE == 'TORCH':\n    model = CustomResNext(CFG.model_name, pretrained=False)\n    states = [torch.load(MODEL_DIR+f'needle_more_augs_10folds_pretrained_{CFG.model_name}_fold{fold}_best.pth') for fold in CFG.trn_fold]\n    test_dataset = TestDataset(test, transform=get_transforms(data='valid'))\n    test_loader = DataLoader(test_dataset, batch_size=CFG.batch_size, shuffle=False, \n                             num_workers=CFG.num_workers, pin_memory=True)\n    predictions = inference(model, states, test_loader, device)\n\n    test[CFG.target_cols] = predictions\n    test[['StudyInstanceUID'] + CFG.target_cols].to_csv(OUTPUT_DIR+'submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TensorFlow-Keras\n\n- an example inference pipeline with needle augmentation using @xhlulu's notebook from https://www.kaggle.com/xhlulu/ranzcr-efficientnet-submission\n- hyperparameters in training were kept constant, LB boosted from 0.957 to 0.959 (maybe just lucky, as its not seeded)"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install /kaggle/input/kerasapplications -q\n!pip install /kaggle/input/efficientnet-keras-source-code/ -q --no-deps","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if INFER_MODE == 'TF':\n    import os\n    import efficientnet.tfkeras as efn\n    import numpy as np\n    import pandas as pd\n    import tensorflow as tf\n    \n    def auto_select_accelerator():\n        try:\n            tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"Running on TPU:\", tpu.master())\n        except ValueError:\n            strategy = tf.distribute.get_strategy()\n        print(f\"Running on {strategy.num_replicas_in_sync} replicas\")\n\n        return strategy\n\n\n    def build_decoder(with_labels=True, target_size=(300, 300), ext='jpg'):\n        def decode(path):\n            file_bytes = tf.io.read_file(path)\n            if ext == 'png':\n                img = tf.image.decode_png(file_bytes, channels=3)\n            elif ext in ['jpg', 'jpeg']:\n                img = tf.image.decode_jpeg(file_bytes, channels=3)\n            else:\n                raise ValueError(\"Image extension not supported\")\n\n            img = tf.cast(img, tf.float32) / 255.0\n            img = tf.image.resize(img, target_size)\n\n            return img\n\n        def decode_with_labels(path, label):\n            return decode(path), label\n\n        return decode_with_labels if with_labels else decode\n\n\n    def build_augmenter(with_labels=True):\n        def augment(img):\n            # insert needle augmentation here\n            img = NeedleAugmentation(img, n_needles=2, dark_needles=False, p=0.5)\n            img = tf.image.random_flip_left_right(img)\n            img = tf.image.random_flip_up_down(img)\n            return img\n\n        def augment_with_labels(img, label):\n            return augment(img), label\n\n        return augment_with_labels if with_labels else augment\n\n\n    def build_dataset(paths, labels=None, bsize=32, cache=True,\n                      decode_fn=None, augment_fn=None,\n                      augment=True, repeat=True, shuffle=1024, \n                      cache_dir=\"\"):\n        if cache_dir != \"\" and cache is True:\n            os.makedirs(cache_dir, exist_ok=True)\n\n        if decode_fn is None:\n            decode_fn = build_decoder(labels is not None)\n\n        if augment_fn is None:\n            augment_fn = build_augmenter(labels is not None)\n\n        AUTO = tf.data.experimental.AUTOTUNE\n        slices = paths if labels is None else (paths, labels)\n\n        dset = tf.data.Dataset.from_tensor_slices(slices)\n        dset = dset.map(decode_fn, num_parallel_calls=AUTO)\n        dset = dset.cache(cache_dir) if cache else dset\n        dset = dset.map(augment_fn, num_parallel_calls=AUTO) if augment else dset\n        dset = dset.repeat() if repeat else dset\n        dset = dset.shuffle(shuffle) if shuffle else dset\n        dset = dset.batch(bsize).prefetch(AUTO)\n\n        return dset\n    \n    COMPETITION_NAME = \"ranzcr-clip-catheter-line-classification\"\n    strategy = auto_select_accelerator()\n    BATCH_SIZE = strategy.num_replicas_in_sync * 16\n    \n    IMSIZE = (224, 240, 260, 300, 380, 456, 528, 600)\n\n    load_dir = f\"/kaggle/input/{COMPETITION_NAME}/\"\n    sub_df = pd.read_csv(load_dir + 'sample_submission.csv')\n    test_paths = load_dir + \"test/\" + sub_df['StudyInstanceUID'] + '.jpg'\n\n    # Get the multi-labels\n    label_cols = sub_df.columns[1:]\n\n    test_decoder = build_decoder(with_labels=False, target_size=(IMSIZE[7], IMSIZE[7]))\n    dtest = build_dataset(\n        test_paths, bsize=BATCH_SIZE, repeat=False, \n        shuffle=False, augment=False, cache=False,\n        decode_fn=test_decoder\n    )\n    \n    with strategy.scope():\n        model = tf.keras.models.load_model(\n            '../input/ranzcr-tpu-weights/efficientnetb7-tpu-needle-aug-resume-train.h5'\n        )\n\n    model.summary()\n    \n    sub_df[label_cols] = model.predict(dtest, verbose=1)\n    sub_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.0567,"end_time":"2020-12-20T14:05:26.427573","exception":false,"start_time":"2020-12-20T14:05:26.370873","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Useful References\n\n- https://www.tensorflow.org/api_docs/python/tf/image/\n- https://albumentations.ai/docs/api_reference/augmentations/transforms/\n- https://pytorch.org/docs/stable/torchvision/transforms.html\n\n# Some Remarks\n\n- the needle augmentation implementation still remains buggy with the TensorFlow-Keras pipeline\n- Torch pipeline seemed to work perfectly fine, but TF pipeline does raise exceptions once in a while on some seeds\n- using insect augmentation (bees instead of needles) in the recent cassava competition showed that it helped stabilize the difference between public and private LB scores\n\n## Do let me know if you have any comments and recommendations going forward! All the best!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}