{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Using test time augmentation for confidence prediction\n\nTest time augmentation is a method where the final prediction for a given image is found by considering multiple augmentations of that image and taking an average. This approach can also be used to generate a confidence because the predictions from augmentation have some spread, and by taking a measure of that spread we can compute a confidence. This allows us to use powerful image processing techniques and transfer learning to generate predictions for this dataset.\n\nIn this notebook I predict the slope of decline and take the arithmetic mean of the predictions from augmentation to predict the FVC score and the standard deviation of these as a measure of the confidence. From what I have managed to achieve so far it looks like the method could work well, but it requires more work and I don't have any more time to spend on it. Maybe somebody else thinks this is worth pursuing and can finish it off!\n\nThis notebook is modified from [this](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords) kernel by [@cdeotte](https://www.kaggle.com/cdeotte)."},{"metadata":{},"cell_type":"markdown","source":"# Initialize Environment"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nimport pickle\n\nfrom sklearn.preprocessing import StandardScaler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration\nIn order to be a proper cross validation with a meaningful overall CV score (aligned with LB score), **you need to choose the same** `EFF_NETS` and `DROP` parameters **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds. \n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different kfold split.\n* FOLDS - number of folds.\n* IMG_SIZES - is a Python list of length FOLDS. This is redundant in this notebook.\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* EFF_NETS - is a list of length FOLDS. These are the EfficientNets to use each fold. The number refers to the B. So a number of `0` refers to EfficientNetB0, and `1` refers to EfficientNetB1, etc.\n* TTA - test time augmentation. Each test image is randomly augmented and predicted TTA times and the average prediction is used. TTA is also applied to OOF during validation."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 3, 5, OR 15 \nFOLDS = 5\n\n# DIMENSION OF THE SLICES\nIMG_SIZES = [512]*FOLDS\n\n# FILE TO TAKE IMAGES FROM\n# image_file = 'osicallscanssimple'\n# image_file = 'scannormalised'\nimage_file = 'osic-scans-tfrecords-512'\n\n# CUTOUT AUGMENTATION PARAMETERS\nDROP_FREQ = [0.75]*FOLDS\nDROP_CT = [20]*FOLDS\nDROP_SIZE = [0.2]*FOLDS\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [32]*FOLDS #TPU\n# BATCH_SIZES = [8]*FOLDS # GPU\nEPOCHS = [12]*FOLDS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [4]*FOLDS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 11","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Preprocess\nThe preprocessing has already been done and saved to TFRecords. These TFRecords are created [here](https://www.kaggle.com/samklein/three-channel-images-normalised)"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = [KaggleDatasets().get_gcs_path(image_file)]*FOLDS;\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_im = []; file_extension = []\nfor file in files_train:\n    n_im+=[int(file.split('-')[-1].split('.')[0])]\n    file_extension += ['/'+file.split('/')[-1]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD TRAIN META DATA\n# This file contains predictions for slopes and intercepts that were generated somewhere else, this is something that I \n# suspect I have done very poorly and is one place that needs investigating, another option is to have the network predict\n# the outcome from bayesian programming and that is something I have attempted in another kernel.\nmeta = pd.read_csv('../input/clean-data/train')\ntry: meta.drop('Unnamed: 0',inplace=True,axis=1)\nexcept: pass\nencoded = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encode Meta Data\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\n\n# One hot encode the meta data and scale targets.\nif not encoded:\n    meta['Sex'] = (meta['Sex']=='Male').astype('int')\n    meta['Smoked'] = (meta['SmokingStatus']=='Ex-smoker').astype('int')\n    meta['CurrentlySmokes'] = (meta['SmokingStatus']=='Currently smokes').astype('int')\n    meta.drop(['SmokingStatus'],axis=1,inplace=True)\n    \n    # Scale the meta data\n    num_enc = StandardScaler()\n    num_cols_to_scale = ['FVC','Percent','Weeks','Age']\n    meta[num_cols_to_scale] = num_enc.fit_transform(meta[num_cols_to_scale])\n\n    # Scale the targets\n    slope_enc = MinMaxScaler(feature_range=(1, 3))\n    meta['slope'] = slope_enc.fit_transform(meta[['slope']])\n#     intercept_enc = MinMaxScaler(feature_range=(1, 5))\n#     meta['intercept'] = intercept_enc.fit_transform(meta[['intercept']])\n    \n    encoded = True\n\n# Look at the preprocessed meta data\nmeta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Doing things this way leaves me the freedom to change labels or how I preprocess meta data without having to worry about\n# regenerating tfrecords.\nnames = meta['Patient'].unique()\ndata = meta.drop(['Patient','intercept','slope'],axis=1)\ntarget = meta['slope']\n\n# Make a dictionary to access the meta data for each patient, also a dictionary to access the correct intercept.\nacc = []; int_dict = {}\ncin=0\nfor pid in names:\n    locs = np.where(meta['Patient']==pid)[0]\n    if cin > locs[0]:\n        print('The dataframe is not ordered')\n    cin=locs[-1]\n    acc += [[locs[0],locs[-1]]]\n    int_dict[pid] = meta['intercept'].loc[locs].iloc[0]\n\n#Make a lookup table for the data\nwith strategy.scope():\n    get_index = tf.lookup.StaticHashTable(\n      tf.lookup.KeyValueTensorInitializer(names, np.arange( len(names) )), -1\n    )\n    meta_access = tf.constant(np.array(acc))\n    meta_tensor = tf.constant(data)\n    target_tensor = tf.constant(target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Data Augmentation\nThis notebook uses rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. This notebook also uses horizontal flip, hue, saturation, contrast, brightness augmentation similar to AgentAuers' notebook.\n\nThe code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work."},{"metadata":{"trusted":true},"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image with CT squares of side size SZ*DIM removed\n    \n    # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n    P = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (P==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        # COMPUTE SQUARE \n        WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # DROPOUT IMAGE\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n    image = tf.reshape(image,[DIM,DIM,3])\n    return image\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Methods for gathering data "},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.random.set_seed(5)\n\ndef prep_meta(example):\n    query = get_index.lookup(example['image_name'])\n    ind_range = tf.gather(meta_access, query)\n    indx = tf.random.uniform([1], minval=ind_range[0], maxval=ind_range[1], dtype=tf.dtypes.int64)\n    meta_data = tf.gather(meta_tensor, indx)\n    target = tf.gather(target_tensor, indx)\n    return tf.squeeze(meta_data), tf.squeeze(target)\n\ndef read_labeled_tfrecord(example):\n\n    tfrec_format = {\n      'image': tf.io.FixedLenFeature([], tf.string),\n      'image_name': tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n\n    meta_data, target = prep_meta(example)\n    return (example['image'],meta_data), target\n\n\n# The unlabelled data is the test set and we cannot read this in with tfrecord\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n      'image': tf.io.FixedLenFeature([], tf.string),\n      'image_name': tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    meta_data, _ = prep_meta(example)\n    return (example['image'],meta_data), example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(data, augment=True, dim=256, droprate=0, dropct=0, dropsize=0):   \n    img=data[0]; meta = data[1]\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img,DIM=dim)\n        if (droprate!=0)&(dropct!=0)&(dropsize!=0): \n            img = dropout(img, DIM=dim, PROBABILITY=droprate, CT=dropct, SZ=dropsize)\n        img = tf.image.random_flip_left_right(img)\n        #img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n                      \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return (img,meta)\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                batch_size=16, dim=512, labeled = True, return_image_names=False,\n                droprate=0, dropct=0, dropsize=0):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(lambda example: read_labeled_tfrecord(example),\n                    num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, dim=dim,\n                                                            droprate=droprate, dropct=dropct, dropsize=dropsize), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n\n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Build Model\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras import layers as L\nimport tensorflow_addons as tfa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\ndef build_model(dim=128, ef=0):\n    inp = tf.keras.layers.Input(shape=(dim,dim,3))\n    base = EFNS[ef](input_shape=(dim,dim,3),weights='noisy-student',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = L.Dense(124, activation='relu')(x)\n    meta_dim = data.shape[1]\n    meta_inp = tf.keras.layers.Input(shape=(meta_dim,))\n    meta = tf.keras.layers.GaussianNoise(0.2)(meta_inp)\n    xm = L.concatenate((x,meta))\n    xm = L.Dropout(0.2)(xm)\n    xm = L.Dense(124, activation='relu')(xm)\n    xm = L.Dropout(0.2)(xm)\n    xm = L.Dense(1, activation='relu')(xm)\n    model = tf.keras.Model(inputs=(inp, meta_inp), outputs=xm)\n    opt = tfa.optimizers.AdamW(learning_rate=0.001,weight_decay=0.0001)\n    loss = tf.keras.losses.MeanAbsoluteError()\n#     loss = tf.keras.losses.MeanSquaredError()\n#     loss = tf.keras.losses.MeanAbsolutePercentageError()\n    model.compile(optimizer=opt,loss=loss)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> # Step 4: Metric scoring\nWe need special ways to make predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate the predictions, confidence and metric for samples.\nuntrans_meta = pd.read_csv('../input/clean-data/train')\n\ndef metric(estimate,confidence,samples):\n    sig = np.where(confidence<70, 70, confidence)\n    abs_diff = np.abs(estimate-samples)\n    delta = np.where(abs_diff>1000,1000,abs_diff)\n    return np.mean(-2**(1/2) * delta / sig - np.log(2**(1/2) * sig))\n\ndef get_metric(names,preds,confidence):\n    score=[]\n    for i,pid in enumerate(names):\n        mx = untrans_meta['Patient']==pid\n        pdata = untrans_meta.loc[mx]\n        inds = pdata['Weeks']; FVCs = pdata['FVC']\n        score += [metric(preds[i][inds],confidence[i][inds],FVCs)]\n    return score\n\ndef predict_full(ids,oof_slope,oof_intercept):\n    oof_slope = np.array(oof_slope); oof_intercept = np.array(oof_intercept)\n    weeks = np.array(range(-12,134))\n    names = np.unique(ids)\n    preds = np.zeros((len(names),len(weeks)));confidence = np.zeros((len(names),len(weeks)))\n    for i,pid in enumerate(names):\n        inds = np.where(ids==pid)\n        slopes = oof_slope[inds].ravel()\n        intercepts = oof_intercept[inds].ravel()\n        preds_tta = np.outer(slopes,weeks) + intercepts[:,np.newaxis]\n        preds[i] = np.mean(preds_tta,axis=0)\n        confidence[i] = np.std(preds_tta,axis=0)\n    return names, preds, confidence","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5: Train Schedule\nThis is a common train schedule for transfer learning. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Adjust the variables `VERBOSE` and `DISPLOY_PLOT` below to determine what output you want displayed. The variable `VERBOSE=1 or 2` will display the training and validation loss and auc for each epoch as text. The variable `DISPLAY_PLOT` shows this information as a plot. \n\nCurrently the early stopping is commented out, this is because the dataset that we have is tiny and we do not want to overfit to the validation data. Once a working method is found the optimal number of epochs can be found by calculating the metric score after every epoch for all folds and then taking the epoch that has the best total metric score when considering all of the folds (epoch per epoch). This has NOT been implemented."},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 0\nDISPLAY_PLOT = True\n\nskf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\noof_tar_slope = []; oof_tar_intercept = []; oof_val = []; oof_names = []; oof_folds = []; oof_score = []\nnimages = count_data_items(file_extension)\noof_slope_pred = np.zeros((nimages,TTA)); oof_intercept_pred = np.zeros((nimages,TTA)); fcnt = 0\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(len(file_extension)))):\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n          (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n        \n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + file_extension[i] for i in idxT])\n        \n    np.random.shuffle(files_train); print('#'*25)\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + file_extension[i] for i in idxV])\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n   \n#     # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n                   droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold]), \n        epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])],\n#         epochs=EPOCHS[fold], callbacks = [sv],\n        steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n                repeat=False,dim=IMG_SIZES[fold]),\n        verbose=VERBOSE\n    )\n    \n#     print('Loading best model...')\n#     model.load_weights('fold-%i.h5'%fold)\n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4,\n            droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold])\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n    pred_model = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:TTA*ct_valid,]\n    oof_slope_pred[fcnt:fcnt+ct_valid] = slope_enc.inverse_transform(pred_model).reshape((ct_valid,TTA),order='F')           \n    \n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True,shuffle=False, return_image_names=True)\n    tslope=[];tint=[]\n    for img, target in iter(ds_valid.unbatch()):\n        tslope.append(target.numpy()); \n    oof_tar_slope.append( slope_enc.inverse_transform(np.array(tslope).reshape(-1, 1)) ); \n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_names=True)\n    nms = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])\n    for i,nm in enumerate(nms):\n        tint.append(int_dict[nm])\n        oof_intercept_pred[fcnt+i] = np.array([int_dict[nm]]*TTA)\n    oof_tar_intercept.append( np.array(tint) )\n    oof_names.append( nms )\n    oof_folds.append( np.ones(nms.shape[0],dtype='int8')*fold )\n    \n    # REPORT RESULTS\n    ids, full_preds, confidence = predict_full(nms,oof_slope_pred[fcnt:fcnt+ct_valid],\n                                               oof_intercept_pred[fcnt:fcnt+ct_valid])\n    scores = get_metric(ids, full_preds, confidence)\n    oof_score.append( scores )\n    oof_val.append(np.max( history.history['val_loss'] ))\n    print('#### FOLD %i OOF Loss = %.3f, Metric score of %.2f'%(fold+1,oof_val[-1],np.mean(scores)))\n    fcnt+=ct_valid\n    \n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        plt.ylabel('Loss',size=14)\n        plt.title('Image Size %i, EfficientNet B%i'%\n                (IMG_SIZES[fold],EFF_NETS[fold]),size=18)\n        plt.legend(loc=3)\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tscores = np.concatenate(oof_score)\nplt.hist(tscores)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Calculate OOF AUC\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\nnames = np.concatenate(oof_names); folds = np.concatenate(oof_folds);\nprint('Overall OOF Score = %.3f'%np.mean(tscores))\n\n# SAVE OOF TO DISK\noof_inf = pd.DataFrame(dict(\n    image_name = names, fold=folds))\n\npreds_df = pd.DataFrame(np.concatenate((oof_slope_pred[fcnt:(fcnt+1)*ct_valid],oof_intercept_pred[fcnt:(fcnt+1)*ct_valid]),axis=1),\n                        index=list(range(len(oof_slope_pred[fcnt:(fcnt+1)*ct_valid]))),\n                        columns=['slope' + str(i) for i in range(TTA)]+['intercept' + str(i) for i in range(TTA)])\n\ndf_oof = pd.concat((oof_inf,preds_df),axis=1)\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> # Step 6: Post-Processing\nHandling the submission of the predictions from this approach is something that I will have to leave to others."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}