{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# [SIIM-FISABIO-RSNA COVID-19 Detection](https://www.kaggle.com/c/siim-covid19-detection)\n> Identify and localize COVID-19 abnormalities on chest radiographs\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/26680/logos/header.png)","metadata":{}},{"cell_type":"markdown","source":"# Overview:\n* Basic idea was to use **classification** model for **Study-Level** & **detection** model for **Image-Level**,\n\n# Notebooks:\n\n#### Study-Level:\n* **train**: [SIIM-COVID-19: Study-Level [train] TPU🩺](https://www.kaggle.com/awsaf49/siim-covid-19-study-level-train-tpu/)\n* **infer**: [SIIM-COVID-19: Study-Level [infer]🩺](https://www.kaggle.com/awsaf49/siim-covid-19-study-level-infer) [LB: **0.360**]\n* **data**: [SIIM-COVID-19: 512x512 tfrec Data](https://www.kaggle.com/awsaf49/siim-covid-19-512x512-tfrec-data)\n\n#### Image-Level:\n* **train**: [SIIM-COVID-19: YOLOv5 Image-Level [train]](https://www.kaggle.com/awsaf49/siim-covid-19-yolov5-image-level-train)\n* **infer**: [SIIM-COVID-19: YOLOv5 Image-Level [infer]](https://www.kaggle.com/awsaf49/siim-covid-19-yolov5-image-level-infer) **placeholder**, seems someting is wrong with `image-level` data, gives very small score `0.051`.\n\n# Dataset:\n\n#### JPEG\n* [1024x1024](https://www.kaggle.com/awsaf49/siimcovid19-1024-jpg-image-dataset)\n* [512x512](https://www.kaggle.com/awsaf49/siimcovid19-512-jpg-image-dataset)\n* [256x256](https://www.kaggle.com/awsaf49/siimcovid19-256-jpg-image-dataset)\n\n#### TFRECORD\n* [1024x1024](https://www.kaggle.com/awsaf49/siimcovid19-1024x1024-tfrec-dataset)\n* [512x512](https://www.kaggle.com/awsaf49/siimcovid19-512x512-tfrec-dataset)\n* [256x256](https://www.kaggle.com/awsaf49/siimcovid19-256x256-tfrec-dataset)","metadata":{}},{"cell_type":"markdown","source":"# Version Info","metadata":{}},{"cell_type":"markdown","source":"<!-- # Notebook for Dataset\n* [SETI-BL: 256x256 tfrec Data](https://www.kaggle.com/awsaf49/seti-bl-256x256-tfrec-data)\n\n# Datasets\n* [128x128](https://www.kaggle.com/awsaf49/setibl-128x128-tfrec-dataset)\n* [256x256](https://www.kaggle.com/awsaf49/setibl-256x256-tfrec-dataset)\n* [384x384](https://www.kaggle.com/awsaf49/setibl-384x384-tfrec-dataset) -->","metadata":{}},{"cell_type":"markdown","source":"# Initialize Environment","metadata":{}},{"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:05.11184Z","iopub.execute_input":"2021-05-28T07:43:05.112234Z","iopub.status.idle":"2021-05-28T07:43:13.859132Z","shell.execute_reply.started":"2021-05-28T07:43:05.11215Z","shell.execute_reply":"2021-05-28T07:43:13.85785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd, numpy as np, random,os, shutil\nfrom glob import glob\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nimport tensorflow_addons as tfa\nprint('tf:',tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:13.861019Z","iopub.execute_input":"2021-05-28T07:43:13.861328Z","iopub.status.idle":"2021-05-28T07:43:21.126764Z","shell.execute_reply.started":"2021-05-28T07:43:13.861294Z","shell.execute_reply":"2021-05-28T07:43:21.125711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration","metadata":{}},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE      = 0\nDISPLAY_PLOT = True\n\nDEVICE = \"TPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 2, 5, 10\nFOLDS = 5\n\n# WHICH IMAGE SIZES TO LOAD EACH FOLD\n# CHOOSE 128, 192, 256, 384, 512, 512 \nIMG_SIZES = [[512, 512]]*FOLDS\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [16]*FOLDS\nEPOCHS      = [12]*FOLDS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [7]*FOLDS\n\n# Augmentations\nAUGMENT   = True\nTRANSFORM = True\n\n# transformations\nROT_    = 0.0\nSHR_    = 2.0\nHZOOM_  = 8.0\nWZOOM_  = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0\n\n# Dropout\nPROBABILITY = 0.75\nCT          = 8\nSZ          = 0.08\n\n#bri, contrast\nsat  = (0.7, 1.3)\ncont = (0.8, 1.2)\nbri  =  0.1\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/FOLDS]*FOLDS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 1","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-05-28T07:43:21.128451Z","iopub.execute_input":"2021-05-28T07:43:21.12877Z","iopub.status.idle":"2021-05-28T07:43:21.13723Z","shell.execute_reply.started":"2021-05-28T07:43:21.12874Z","shell.execute_reply":"2021-05-28T07:43:21.135893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reproducibility\nNot very helpful for **TPU** ","metadata":{}},{"cell_type":"code","source":"def seeding(SEED):\n    np.random.seed(SEED)\n    random.seed(SEED)\n    os.environ['PYTHONHASHSEED'] = str(SEED)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = str(SEED)\n    tf.random.set_seed(SEED)\n    print('seeding done!!!')\nseeding(SEED)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:21.13938Z","iopub.execute_input":"2021-05-28T07:43:21.139742Z","iopub.status.idle":"2021-05-28T07:43:21.152389Z","shell.execute_reply.started":"2021-05-28T07:43:21.139711Z","shell.execute_reply":"2021-05-28T07:43:21.151321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TPU Configs","metadata":{}},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:21.155547Z","iopub.execute_input":"2021-05-28T07:43:21.155956Z","iopub.status.idle":"2021-05-28T07:43:27.108166Z","shell.execute_reply.started":"2021-05-28T07:43:21.155909Z","shell.execute_reply":"2021-05-28T07:43:27.107182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 1: Preprocess","metadata":{}},{"cell_type":"code","source":"GCS_PATH = [None]*FOLDS\nfor i,k in enumerate(IMG_SIZES):\n    GCS_PATH[i] = KaggleDatasets().get_gcs_path('siimcovid19-%ix%i-tfrec-dataset'%(k[0],k[1]))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))\nnum_train_files = len(files_train)\nnum_test_files  = len(files_test)\nprint('train_files:',num_train_files)\nprint('test_files:',num_test_files)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:27.109242Z","iopub.execute_input":"2021-05-28T07:43:27.109541Z","iopub.status.idle":"2021-05-28T07:43:29.200862Z","shell.execute_reply.started":"2021-05-28T07:43:27.109495Z","shell.execute_reply":"2021-05-28T07:43:29.199541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Light EDA","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/siim-covid19-yolov5-2class-labels/meta.csv')\ntrain_df.head(2)","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:29.202396Z","iopub.execute_input":"2021-05-28T07:43:29.202878Z","iopub.status.idle":"2021-05-28T07:43:29.370889Z","shell.execute_reply.started":"2021-05-28T07:43:29.20283Z","shell.execute_reply":"2021-05-28T07:43:29.369817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_names = train_df.class_name.tolist(); class_labels = train_df.class_label.tolist()\nname2label = dict(set(zip(class_names, class_labels )))\nlabel2name = {v:k for k, v in name2label.items()}\nname2label","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:29.374322Z","iopub.execute_input":"2021-05-28T07:43:29.374772Z","iopub.status.idle":"2021-05-28T07:43:29.384956Z","shell.execute_reply.started":"2021-05-28T07:43:29.374735Z","shell.execute_reply":"2021-05-28T07:43:29.383829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Class Distribution\n ","metadata":{}},{"cell_type":"code","source":"from plotly.offline import init_notebook_mode, iplot, plot\nimport plotly as py\nimport plotly.express as px\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\n\ninfo = train_df.class_name.value_counts()\nfig = go.Figure(data=[\n    go.Bar(name=name2label[info.index.tolist()[idx]], \n           y=[info[idx]],\n           x=[info.index.tolist()[idx]],\n           text = str(info[idx]),\n           orientation='v',\n           textposition='outside',) for idx in range(len(info))\n])\n# Change the bar mode\nfig.update_layout(\n                  width=800,\n                  height=600,\n                  title=f'Class Distribution',\n                  yaxis_title='Number of Images',\n                  xaxis_title='Class Name',)\niplot(fig)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:29.387126Z","iopub.execute_input":"2021-05-28T07:43:29.387512Z","iopub.status.idle":"2021-05-28T07:43:31.684777Z","shell.execute_reply.started":"2021-05-28T07:43:29.387431Z","shell.execute_reply":"2021-05-28T07:43:31.683691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Data Augmentation\nUsed simple augmentations, some of them may hurt the model.\n* RandomFlip (Left-Right)\n* No Rotation\n* RandomBrightness\n* RndomContrast\n* Shear\n* Zoom\n* Coarsee Dropout/Cutout\n\nAs this is not typical **image** data rather **signal** typical augmantation for **image** may do some damage","metadata":{}},{"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=IMG_SIZES[0]):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    \n    # fixed for non-square image thanks to Chris Deotte\n    \n    if DIM[0]!=DIM[1]:\n        pad = (DIM[0]-DIM[1])//2\n        image = tf.pad(image, [[0, 0], [pad, pad+1],[0, 0]])\n        \n    NEW_DIM = DIM[0]\n    \n    XDIM = NEW_DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(NEW_DIM//2, -NEW_DIM//2,-1), NEW_DIM)\n    y   = tf.tile(tf.range(-NEW_DIM//2, NEW_DIM//2), [NEW_DIM])\n    z   = tf.ones([NEW_DIM*NEW_DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -NEW_DIM//2+XDIM+1, NEW_DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([NEW_DIM//2-idx2[0,], NEW_DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n    \n    if DIM[0]!=DIM[1]:\n        image = tf.reshape(d,[NEW_DIM, NEW_DIM,3])\n        image = image[:, pad:DIM[1]+pad,:]\n    image = tf.reshape(image, [*DIM, 3])\n        \n    return image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:31.688087Z","iopub.execute_input":"2021-05-28T07:43:31.688403Z","iopub.status.idle":"2021-05-28T07:43:31.710284Z","shell.execute_reply.started":"2021-05-28T07:43:31.68837Z","shell.execute_reply":"2021-05-28T07:43:31.708483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dropout\ncheck this [notebook](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169721) by chris for more explanation on **Coarse Dropout and Cutout**\n\n![](http://playagricola.com/Kaggle/drop-7-24.jpg)\n\nCoarse Dropout and Cutout augmentation are techniques to prevent **overfitting** and encourage generalization. They randomly remove rectangles from training images. By removing portions of the images, we challenge our models to pay attention to the entire image because it never knows what part of the image will be present. (This is similar and different to dropout layer within a **CNN**).\n\n* Cutout is the technique of removing 1 large rectangle of random size\n* Coarse dropout is the technique of removing many small rectanges of similar size.\n\nBy changing the parameters below, we can have either coarse dropout or cutout. (For cutout, you'll need to add tf.random.uniform for random size. I leave this as an exercise for the reader.","metadata":{}},{"cell_type":"code","source":"def dropout(image,DIM=IMG_SIZES[0], PROBABILITY = 0.6, CT = 5, SZ = 0.1):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image with CT squares of side size SZ*DIM removed\n    \n    # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n    P = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (P==0)|(CT==0)|(SZ==0): \n        return image\n    \n    for k in range(CT):\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM[1]),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM[0]),tf.int32)\n        # COMPUTE SQUARE \n        WIDTH = tf.cast( SZ*min(DIM),tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM[0],y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM[1],x+WIDTH//2)\n        # DROPOUT IMAGE\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3], dtype = image.dtype) \n        three = image[ya:yb,xb:DIM[1],:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM[0],:,:]],axis=0) \n\n    image = tf.reshape(image,[*DIM,3])\n    return image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:31.711758Z","iopub.execute_input":"2021-05-28T07:43:31.712132Z","iopub.status.idle":"2021-05-28T07:43:31.726698Z","shell.execute_reply.started":"2021-05-28T07:43:31.712103Z","shell.execute_reply":"2021-05-28T07:43:31.725272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Reading TFRecord Data","metadata":{}},{"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_id'                     : tf.io.FixedLenFeature([], tf.string),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], tf.one_hot(example['target'], 4)\n\n\ndef read_unlabeled_tfrecord(example, return_image_id):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_id'                     : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_id'] if return_image_id else 0\n\n \ndef prepare_image(img, augment=True, dim=IMG_SIZES[0]):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32)\n    img = img/255.0\n    \n    if augment:\n        img = transform(img,DIM=dim) if TRANSFORM else img\n        img = tf.image.random_flip_left_right(img)\n        #img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, sat[0], sat[1])\n        img = tf.image.random_contrast(img, cont[0], cont[1])\n        img = tf.image.random_brightness(img, bri)     \n                      \n    img = tf.reshape(img, [*dim, 3])\n            \n    return img\n\ndef count_data_items(fileids):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(fileid).group(1)) \n         for fileid in fileids]\n    return np.sum(n)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:31.72812Z","iopub.execute_input":"2021-05-28T07:43:31.728448Z","iopub.status.idle":"2021-05-28T07:43:31.743423Z","shell.execute_reply.started":"2021-05-28T07:43:31.728419Z","shell.execute_reply":"2021-05-28T07:43:31.742244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pipeline","metadata":{}},{"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_ids=True, batch_size=16, dim=IMG_SIZES[0]):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*2, seed=SEED)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_ids), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgid_or_label: (prepare_image(img, augment=augment, dim=dim), \n                                               imgid_or_label), \n                num_parallel_calls=AUTO)\n    if labeled and augment:\n        ds = ds.map(lambda img, label: (dropout(img, DIM=dim, PROBABILITY = PROBABILITY, CT = CT, SZ = SZ), label),\n                    num_parallel_calls=AUTO)\n    \n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:31.745453Z","iopub.execute_input":"2021-05-28T07:43:31.745797Z","iopub.status.idle":"2021-05-28T07:43:31.760384Z","shell.execute_reply.started":"2021-05-28T07:43:31.745758Z","shell.execute_reply":"2021-05-28T07:43:31.759374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization","metadata":{}},{"cell_type":"code","source":"def display_batch(batch, size=2):\n    imgs, tars = batch\n    plt.figure(figsize=(size*5, 5))\n    for img_idx in range(size):\n        plt.subplot(1, size, img_idx+1)\n        plt.title(f'class: {label2name[tars[img_idx].numpy().argmax()]}', fontsize=15)\n        plt.imshow(imgs[img_idx,:, :, :])\n        plt.xticks([])\n        plt.yticks([])\n    plt.tight_layout()\n    plt.show() ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:31.761824Z","iopub.execute_input":"2021-05-28T07:43:31.762279Z","iopub.status.idle":"2021-05-28T07:43:31.775507Z","shell.execute_reply.started":"2021-05-28T07:43:31.76224Z","shell.execute_reply":"2021-05-28T07:43:31.774627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fold = 0\nds = get_dataset(files_train, augment=True, shuffle=False, repeat=True,labeled=True,return_image_ids=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold])\nds = ds.unbatch().batch(20)\nbatch = next(iter(ds))\ndisplay_batch(batch, 5);","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:31.777077Z","iopub.execute_input":"2021-05-28T07:43:31.777703Z","iopub.status.idle":"2021-05-28T07:43:41.229743Z","shell.execute_reply.started":"2021-05-28T07:43:31.777656Z","shell.execute_reply":"2021-05-28T07:43:41.228678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 3: Build Model\n>Though there have been some amazing development, **EFficientNet** always come up with a decent score.\n\n![EffNet](https://1.bp.blogspot.com/-MQO5qKuTT8c/XpdE8_IwpsI/AAAAAAAAFtg/mSjhF2ws5FYxwcHN6h9_l5DqYzQlNYJwwCLcBGAsYHQ/s1600/image1.png)\n\nYou can try other models like, \n* Vision Transformer (ViT)\n* ResNet\n* InceptionNet\n* XceptionNet\n\nUnfortunately, I couldn't find any completed implementation of **NF-Net**, so couldn't include it here.","metadata":{}},{"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6, efn.EfficientNetB7]\n\ndef build_model(dim=IMG_SIZES[0], ef=0):\n    inp = tf.keras.layers.Input(shape=(*dim,3))\n    base = EFNS[ef](input_shape=(*dim,3),weights='imagenet',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = tf.keras.layers.Dense(64, activation = 'relu')(x)\n    x = tf.keras.layers.Dense(4,activation='softmax')(x)\n    model = tf.keras.Model(inputs=inp,outputs=x)\n    opt = tf.keras.optimizers.Adam(learning_rate=0.001)\n    loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.01)\n    auc = tf.keras.metrics.AUC(curve='ROC',\n                               multi_label=True)\n    acc = tf.keras.metrics.CategoricalAccuracy()\n    f1  = tfa.metrics.F1Score(num_classes=4,average='macro',threshold=None)\n    model.compile(optimizer=opt,loss=loss,metrics=[auc, acc, f1])\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-05-28T07:43:41.231145Z","iopub.execute_input":"2021-05-28T07:43:41.231445Z","iopub.status.idle":"2021-05-28T07:43:41.240673Z","shell.execute_reply.started":"2021-05-28T07:43:41.231416Z","shell.execute_reply":"2021-05-28T07:43:41.239905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Train Schedule\nThis is a common train schedule for transfer learning. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","metadata":{}},{"cell_type":"code","source":"def get_lr_callback(batch_size=8, plot=False):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n    if plot:\n        plt.figure(figsize=(10,5))\n        plt.plot(np.arange(EPOCHS[0]), [lrfn(epoch) for epoch in np.arange(EPOCHS[0])], marker='o')\n        plt.xlabel('epoch'); plt.ylabel('learnig rate')\n        plt.title('Learning Rate Scheduler')\n        plt.show()\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback\n\n_=get_lr_callback(BATCH_SIZES[0], plot=True )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:41.242135Z","iopub.execute_input":"2021-05-28T07:43:41.242667Z","iopub.status.idle":"2021-05-28T07:43:41.426465Z","shell.execute_reply.started":"2021-05-28T07:43:41.242632Z","shell.execute_reply":"2021-05-28T07:43:41.42581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train Model\n* Our model will be trained for the number of `FOLDS` and `EPOCHS` you chose in the configuration above. Each fold the model with lowest validation `AUC` will be saved and used to predict OOF and test. \n* Adjust the variables `VERBOSE` and `DISPLOY_PLOT` below to determine what output you want displayed. The variable `VERBOSE=1 or 2` will display the training and validation loss and auc for each epoch as text. The variable `DISPLAY_PLOT` shows this information as a plot. ","metadata":{}},{"cell_type":"code","source":"skf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\noof_pred = []; oof_tar = []; oof_val = []; oof_f1 = []; oof_ids = []; oof_folds = [] \n#preds = np.zeros((count_data_items(files_test),1))\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(num_train_files))):\n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n    np.random.shuffle(files_train);\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size: (%i, %i) | model: %s | batch_size %i'%\n          (IMG_SIZES[fold][0],IMG_SIZES[fold][1],EFNS[EFF_NETS[fold]].__name__,BATCH_SIZES[fold]*REPLICAS))\n    train_images = count_data_items(files_train)\n    val_images   = count_data_items(files_valid)\n    print('#### Training: %i | Validation: %i'%(train_images, val_images))\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n    print('#'*25)   \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_auc', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='max', save_freq='epoch')\n   \n    # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, augment=AUGMENT, shuffle=True, repeat=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold]), \n        epochs=EPOCHS[fold], \n        callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n        steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n                repeat=False,dim=IMG_SIZES[fold]), \n        #class_weight = {0:1,1:2},\n        verbose=VERBOSE\n    )\n    \n    # Loading best model for inference\n    print('Loading best model...')\n    model.load_weights('fold-%i.h5'%fold)  \n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_ids=False,augment=AUGMENT if TTA>1 else False,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*2)\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/2/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:TTA*ct_valid,] \n    oof_pred.append( np.mean(pred.reshape((ct_valid,4,TTA),order='F'),axis=-1) )                 \n    \n    # GET OOF TARGETS AND idS\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True, return_image_ids=True)\n    oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n    oof_folds.append( np.ones_like(oof_tar[-1],dtype='int8')*fold )\n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_ids=True)\n    oof_ids.append( np.array([img_id.numpy().decode(\"utf-8\") for img, img_id in iter(ds.unbatch())]))\n    \n#     # PREDICT TEST USING TTA\n#     print('Predicting Test with TTA...')\n#     ds_test = get_dataset(files_test,labeled=False,return_image_ids=False,augment=AUGMENT,\n#             repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*2)\n#     ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZES[fold]/2/REPLICAS\n#     pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n#     preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n    \n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1], average='macro')\n    oof_val.append(np.max( history.history['val_auc'] ))\n    print('#### FOLD %i OOF AUC without TTA = %.3f, with TTA = %.3f'%(fold+1,oof_val[-1],auc))\n    \n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt.plot(np.arange(len(history.history['auc'])),history.history['auc'],'-o',label='Train auc',color='#ff7f0e')\n        plt.plot(np.arange(len(history.history['auc'])),history.history['val_auc'],'-o',label='Val auc',color='#1f77b4')\n        x = np.argmax( history.history['val_auc'] ); y = np.max( history.history['val_auc'] )\n        xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max auc\\n%.2f'%y,size=14)\n        plt.ylabel('auc',size=14); plt.xlabel('Epoch',size=14)\n        plt.legend(loc=2)\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(len(history.history['auc'])),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(len(history.history['auc'])),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n        plt.ylabel('Loss',size=14)\n        plt.title('FOLD %i - Image Size (%i, %i), %s'%\n                (fold+1,IMG_SIZES[fold][0],IMG_SIZES[fold][1],EFNS[EFF_NETS[fold]].__name__),size=18)\n        plt.legend(loc=3)\n        plt.savefig(f'fig{fold}.png')\n        plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-28T07:43:41.427615Z","iopub.execute_input":"2021-05-28T07:43:41.428068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Calculate OOF AUC\nThe **OOF** (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the **OOF** to determine what are the best weights to blend your models with. Choose weights that maximize **OOF** `CV` score when used to blend **OOF**. Then use those same weights to blend your test predictions.\n\n**Remember**,\n* Don't do blending just to climb **LB**, because most of the time it ends up getting overfitted.\n* Try improving the **CV** by blending different model and you can keep an eye on the **LB**.\n* As only`20%` data will be used for calculating **LB** score, relying on **CV** should be a safe option","metadata":{}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof = np.concatenate(oof_pred); true = np.concatenate(oof_tar);\nids = np.concatenate(oof_ids); folds = np.concatenate(oof_folds)\nauc = roc_auc_score(true,oof, average='macro')\nprint('Overall OOF AUC with TTA = %.3f'%auc)\n\n# SAVE OOF TO DISK\ncolumns = ['image_id', 'fold']+[f'{idx}_true' for idx in range(4)] + [f'{idx}_pred' for idx in range(4)]\ndf_oof = pd.DataFrame(np.concatenate([ids[:,None], folds[:, 0:1], true, oof], axis=1), columns=columns)\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Post process\nThere are ways to modify predictions based on patient information to increase **CV**-**LB**. You can experiment with that here on your **OOF**.","metadata":{}},{"cell_type":"markdown","source":"# Submit To Kaggle","metadata":{}},{"cell_type":"code","source":"# ds = get_dataset(files_test, augment=False, repeat=False, dim=IMG_SIZES[fold],\n#                  labeled=False, return_image_ids=True)\n\n# image_ids = np.array([img_id.numpy().decode(\"utf-8\") \n#                         for img, img_id in iter(ds.unbatch())])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission = pd.DataFrame({'id':image_ids, 'target':preds[:,0]})\n# submission = submission.sort_values('id') \n# submission.to_csv('submission.csv', index=False)\n# submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction Distribution","metadata":{}},{"cell_type":"code","source":"# plt.figure(figsize=(10,5))\n# plt.hist(submission.target,bins=100);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reference\n* Please Check this amazing notebook, [Triple Stratified KFold with TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) by [Chris Deotte](https://www.kaggle.com/cdeotte)\n","metadata":{}},{"cell_type":"markdown","source":"# Please **Upvote** if you find this helpful 👽","metadata":{}}]}