{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Initialize Environment"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nimport pickle\n\nfrom sklearn.preprocessing import StandardScaler","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration\nIn order to be a proper cross validation with a meaningful overall CV score (aligned with LB score), **you need to choose the same** `IMG_SIZES`, and `EFF_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds. \n* DEVICE - is GPU or TPU\n* SEED - a different seed produces a different triple stratified kfold split.\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* IMG_SIZES - is a Python list of length FOLDS. These are the image sizes to use each fold\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* EFF_NETS - is a list of length FOLDS. These are the EfficientNets to use each fold. The number refers to the B. So a number of `0` refers to EfficientNetB0, and `1` refers to EfficientNetB1, etc.\n* TTA - test time augmentation. Each test image is randomly augmented and predicted TTA times and the average prediction is used. TTA is also applied to OOF during validation."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42\n\n# NUMBER OF FOLDS. USE 3, 5, OR 15 \nFOLDS = 5\n\n# DIMENSION OF THE SLICES\nIMG_SIZES = [512]*FOLDS\n\n# FILE TO TAKE IMAGES FROM\n# image_file = 'osicallscanssimple'\n# image_file = 'scannormalised'\nimage_file = 'osic-scans-tfrecords-512'\n\n# CUTOUT AUGMENTATION PARAMETERS\nDROP_FREQ = [0.75]*FOLDS\nDROP_CT = [20]*FOLDS\nDROP_SIZE = [0.2]*FOLDS\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [32]*FOLDS #TPU\n# BATCH_SIZES = [8]*FOLDS # GPU\nEPOCHS = [12]*FOLDS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [3]*FOLDS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 11","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Preprocess\nPreprocess has already been done and saved to TFRecords. Here we choose which size to load. We can use either 128x128, 192x192, 256x256, 384x384, 512x512, 768x768 by changing the `IMG_SIZES` variable in the preceeding code section. These TFRecords are discussed [here][1]. The advantage of using different input sizes is discussed [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = [KaggleDatasets().get_gcs_path(image_file)]*FOLDS;\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_im = []; file_extension = []\nfor file in files_train:\n    n_im+=[int(file.split('-')[-1].split('.')[0])]\n    file_extension += ['/'+file.split('/')[-1]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The fits come from [this](https://www.kaggle.com/samklein/probabilistic-fits) notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"# LOAD TRAIN META DATA\n# It is a small issue that one patient is dropped from this but not from the fits.\nmeta = pd.read_csv('../input/clean-data/train')\ntry: meta.drop('Unnamed: 0',inplace=True,axis=1)\nexcept: pass\nencoded = False\n\nfit_names = np.load('../input/probabilistic-fits-studentt/names.npy')\nfit_samples = np.load('../input/probabilistic-fits-studentt/samples.npy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encode Meta Data\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\n\n# One hot encode the meta data\n# Should you give current smokers the label of smoked as well? Because they used to smoke, and they still do.\nif not encoded:\n    meta['Sex'] = (meta['Sex']=='Male').astype('int')\n    meta['Smoked'] = (meta['SmokingStatus']=='Ex-smoker').astype('int')\n    meta['CurrentlySmokes'] = (meta['SmokingStatus']=='Currently smokes').astype('int')\n    meta.drop(['SmokingStatus'],axis=1,inplace=True)\n    \n    # Scale the meta data\n    num_enc = StandardScaler()\n    num_cols_to_scale = ['FVC','Percent','Weeks','Age']\n    meta[num_cols_to_scale] = num_enc.fit_transform(meta[num_cols_to_scale])\n\n    # Scale the targets\n    slope_enc = MinMaxScaler(feature_range=(1, 10))\n    scaled_slope = slope_enc.fit_transform(fit_samples[0::2].ravel().reshape(-1, 1))\n    intercept_enc = MinMaxScaler(feature_range=(1, 20))\n    scaled_intercept = intercept_enc.fit_transform(fit_samples[1::2].ravel().reshape(-1, 1))\n    \n    encoded = True\n\n# Look at the preprocessed meta data\nmeta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# One thing that is clear is that there is an imbalance in the targets, some kind of sample weighting or upsampling\n# is probably required to address this.\nfig, ax = plt.subplots(1,2,figsize=(14,5))\nax[0].hist(fit_samples[0::2].ravel())\nax[0].set_title('Slope')\nax[1].hist(fit_samples[1::2].ravel())\nax[1].set_title('Intercept')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Doing things this way leaves me the freedom to change labels or how I preprocess without having to worry about\n# regenerating tfrecords.\nnames = meta['Patient'].unique()\ndata = meta.drop(['Patient','intercept','slope'],axis=1)\ntarget = meta[['slope','intercept']]\n\nacc = []; ordered_slope = np.zeros(scaled_slope.shape); ordered_intercept = np.zeros(scaled_intercept.shape)\ncin=0\nfor i,pid in enumerate(names):\n    locs = np.where(meta['Patient']==pid)[0]\n    if cin > locs[0]:\n        print('The dataframe is not ordered')\n    cin=locs[-1]\n    acc += [[locs[0],locs[-1]]]\n    ordered_slope[i] = scaled_slope[np.where(fit_names==pid)]\n    ordered_intercept[i] = scaled_intercept[np.where(fit_names==pid)]\n\n# The number of samples drawn for the intercept and slope.\nnsamples = ordered_intercept.shape[1]\n\n#Make a lookup table for the data\nwith strategy.scope():\n    get_index = tf.lookup.StaticHashTable(\n      tf.lookup.KeyValueTensorInitializer(names, np.arange( len(names) )), -1\n    )\n    meta_access = tf.constant(np.array(acc))\n    meta_tensor = tf.constant(data)\n    target_slope = tf.constant(ordered_slope)\n    target_intercept = tf.constant(ordered_intercept)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Data Augmentation\nThis notebook uses rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. This notebook also uses horizontal flip, hue, saturation, contrast, brightness augmentation similar to last years winner and also similar to AgentAuers' notebook.\n\nAdditionally we can decide to use external data by changing the variables `INC2019` and `INC2018` in the preceeding code section. These variables respectively indicate whether to load last year 2019 data and/or year 2018 + 2017 data. These datasets are discussed [here][3]\n\nConsider experimenting with different augmenation and/or external data. The code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work.\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910"},{"metadata":{"trusted":true},"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image with CT squares of side size SZ*DIM removed\n    \n    # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n    P = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (P==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        # COMPUTE SQUARE \n        WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # DROPOUT IMAGE\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n    image = tf.reshape(image,[DIM,DIM,3])\n    return image\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Methods for gathering data "},{"metadata":{"trusted":true},"cell_type":"code","source":"tf.random.set_seed(5)\n\ndef prep_meta(example):\n    query = get_index.lookup(example['image_name'])\n    ind_range = tf.gather(meta_access, query)\n    indx = tf.random.uniform([1], minval=ind_range[0], maxval=ind_range[1], dtype=tf.dtypes.int64)\n    ## Uncomment the below to only take the first date during training\n#     indx = ind_range[0]\n    meta_data = tf.gather(meta_tensor, indx)\n    indtarget = tf.random.uniform([1], minval=0, maxval=nsamples, dtype=tf.dtypes.int64)\n    ts = tf.squeeze(tf.gather(target_slope[query], indtarget))\n    ti = tf.squeeze(tf.gather(target_intercept[query], indtarget))\n    return tf.squeeze(meta_data), tf.stack([ts,ti],axis=-1)\n\ndef read_labeled_tfrecord(example):\n\n    tfrec_format = {\n      'image': tf.io.FixedLenFeature([], tf.string),\n      'image_name': tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n\n    meta_data, target = prep_meta(example)\n    return (example['image'],meta_data), target\n\n\n# The unlabelled data is the test set and we cannot read this in with tfrecord\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n      'image': tf.io.FixedLenFeature([], tf.string),\n      'image_name': tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    meta_data, _ = prep_meta(example)\n    return (example['image'],meta_data), example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(data, augment=True, dim=256, droprate=0, dropct=0, dropsize=0):   \n    img=data[0]; meta = data[1]\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img,DIM=dim)\n        if (droprate!=0)&(dropct!=0)&(dropsize!=0): \n            img = dropout(img, DIM=dim, PROBABILITY=droprate, CT=dropct, SZ=dropsize)\n        img = tf.image.random_flip_left_right(img)\n        #img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n                      \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return (img,meta)\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                batch_size=16, dim=512, labeled = True, return_image_names=False,\n                droprate=0, dropct=0, dropsize=0):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(lambda example: read_labeled_tfrecord(example),\n                    num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, dim=dim,\n                                                            droprate=droprate, dropct=dropct, dropsize=dropsize), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n\n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fold=0\n# files_debug = files_train = tf.io.gfile.glob([GCS_PATH[fold] + file_extension[i] for i in [0]])\n\n# # ds_valid = get_dataset(files_valid,augment=False,shuffle=False,repeat=False,dim=IMG_SIZES[fold])\n# ds_train = get_dataset(files_debug, augment=True, shuffle=True, repeat=True,\n#                 dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold], labeled=True,\n#                    droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold])\n# for img, target in iter(ds_train.unbatch()):\n#         print(target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Build Model\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers. Also consider inputing meta features into your CNN."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras import layers as L\nimport tensorflow_addons as tfa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\ndef build_model(dim=128, ef=0):\n    inp = tf.keras.layers.Input(shape=(dim,dim,3))\n    base = EFNS[ef](input_shape=(dim,dim,3),weights='noisy-student',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    fdim = 128\n    x = L.Dense(fdim, activation='relu')(x)\n    meta_dim = data.shape[1]\n    meta_inp = tf.keras.layers.Input(shape=(meta_dim,))\n    xm = L.concatenate((x,meta_inp))\n    xm = L.Dropout(0.2)(xm)\n    xm = L.Dense(fdim, activation='relu')(xm)\n    xm = L.Dropout(0.1)(xm)\n    xm = L.Dense(2, activation='relu')(xm)\n    model = tf.keras.Model(inputs=(inp, meta_inp), outputs=xm)\n#     step = tf.Variable(0, trainable=False)\n#     schedule = tf.optimizers.schedules.PiecewiseConstantDecay(\n#         [2, 5, 7], [1e-2, 1e-1, 1e-2, 1e-3])\n#     # lr and wd can be a function or a tensor\n#     lr = 1e-2 * schedule(step)\n#     wd = lambda: 1e-2 * schedule(step)\n#     opt = tfa.optimizers.AdamW(learning_rate=0.001, weight_decay=wd)\n    opt = tfa.optimizers.AdamW(learning_rate=0.001,weight_decay=0.001)\n#     loss = tf.keras.losses.MeanAbsoluteError()\n    loss = tf.keras.losses.MeanSquaredError()\n#     loss = tf.keras.losses.MeanAbsolutePercentageError()\n    model.compile(optimizer=opt,loss=loss)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Metric scoring\nWe need special ways to make predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate the predictions, confidence and metric for samples.\n# Should be done using tensors directly\nuntrans_meta = pd.read_csv('../input/clean-data/train')\n\ndef metric(estimate,confidence,samples):\n    sig = np.where(confidence<70, 70, confidence)\n    abs_diff = np.abs(estimate-samples)\n    delta = np.where(abs_diff>1000,1000,abs_diff)\n    return np.mean(-2**(1/2) * delta / sig - np.log(2**(1/2) * sig))\n\ndef get_metric(names,preds,confidence):\n    score=[]\n    for i,pid in enumerate(names):\n        mx = untrans_meta['Patient']==pid\n        pdata = untrans_meta.loc[mx]\n        inds = pdata['Weeks']; FVCs = pdata['FVC']\n        score += [metric(preds[i][inds],confidence[i][inds],FVCs)]\n    return score\n\ndef predict_full(ids,oof_slope,oof_intercept):\n    oof_slope = np.array(oof_slope); oof_intercept = np.array(oof_intercept)\n    weeks = np.array(range(-12,134))\n    names = np.unique(ids)\n    preds = np.zeros((len(names),len(weeks)));confidence = np.zeros((len(names),len(weeks)))\n    for i,pid in enumerate(names):\n        inds = np.where(ids==pid)\n        slopes = oof_slope[inds].ravel()\n        intercepts = oof_intercept[inds].ravel()\n        preds_tta = np.outer(slopes,weeks) + intercepts[:,np.newaxis]\n        preds[i] = np.mean(preds_tta,axis=0)\n        confidence[i] = np.std(preds_tta,axis=0)\n    return names, preds, confidence","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4: Train Schedule\nThis is a common train schedule for transfer learning. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model\nOur model will be trained for the number of FOLDS and EPOCHS you chose in the configuration above. Each fold the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variables `VERBOSE` and `DISPLOY_PLOT` below to determine what output you want displayed. The variable `VERBOSE=1 or 2` will display the training and validation loss and auc for each epoch as text. The variable `DISPLAY_PLOT` shows this information as a plot. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 0\nDISPLAY_PLOT = True\n\nskf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\n# oof_tar_slope = []; oof_tar_intercept = []; \noof_val = []; oof_names = []; oof_folds = []; oof_score = []\nnimages = count_data_items(file_extension)\noof_slope_pred = np.zeros((nimages,TTA)); oof_intercept_pred = np.zeros((nimages,TTA)); fcnt = 0\n# oof_tta_save = []; test_tta_save = []\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(len(file_extension)))):\n    \n    if fold > 0:\n        break\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n          (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n        \n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + file_extension[i] for i in idxT])\n        \n    np.random.shuffle(files_train); print('#'*25)\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + file_extension[i] for i in idxV])\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n   \n#     # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n                   droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold]), \n        epochs=EPOCHS[fold], \n        callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])],\n#         epochs=EPOCHS[fold], callbacks = [sv],\n        steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n                repeat=False,dim=IMG_SIZES[fold]),\n        verbose=VERBOSE\n    )\n    \n#     print('Loading best model...')\n#     model.load_weights('fold-%i.h5'%fold)\n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4,\n            droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold])\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:TTA*ct_valid,]\n#     pred = target_enc.inverse_transform(pred_model) # Transform back to the proper scale\n    oof_slope_pred[fcnt:fcnt+ct_valid] = slope_enc.inverse_transform(pred[:,0].reshape(-1, 1)).reshape((ct_valid,TTA),order='F')\n    oof_intercept_pred[fcnt:fcnt+ct_valid] = intercept_enc.inverse_transform(pred[:,1].reshape(-1, 1)).reshape((ct_valid,TTA),order='F')             \n    \n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True,shuffle=False, return_image_names=True)\n#     tslope=[];tint=[]\n#     for img, target in iter(ds_valid.unbatch()):\n#         tslope.append(target.numpy()[0]); tint.append(target.numpy()[1])\n#     oof_tar_slope.append( slope_enc.inverse_transform(np.array(tslope).reshape(-1, 1)) ); \n#     oof_tar_intercept.append( intercept_enc.inverse_transform(np.array(tint).reshape(-1, 1)) )\n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_names=True)\n    nms = np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())])\n    oof_names.append( nms )\n    oof_folds.append( np.ones(nms.shape[0],dtype='int8')*fold )\n    \n    # REPORT RESULTS\n    ids, full_preds, confidence = predict_full(nms,oof_slope_pred[fcnt:fcnt+ct_valid],\n                                               oof_intercept_pred[fcnt:fcnt+ct_valid])\n    scores = get_metric(ids, full_preds, confidence)\n    oof_score.append( scores )\n    oof_val.append(np.max( history.history['val_loss'] ))\n    print('#### FOLD %i OOF Loss = %.3f, Metric score of %.2f'%(fold+1,oof_val[-1],np.mean(scores)))\n    fcnt+=ct_valid\n    \n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        plt.ylabel('Loss',size=14)\n        plt.title('Image Size %i, EfficientNet B%i'%\n                (IMG_SIZES[fold],EFF_NETS[fold]),size=18)\n        plt.legend(loc=3)\n        plt.show()  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tscores = np.concatenate(oof_score)\nplt.hist(tscores)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Calculate OOF AUC\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\n# true_slope = np.concatenate(oof_tar_slope); true_intercept = np.concatenate(oof_tar_intercept)\nnames = np.concatenate(oof_names); folds = np.concatenate(oof_folds);\nprint('Overall OOF Score = %.3f'%np.mean(tscores))\n\n# SAVE OOF TO DISK\noof_inf = pd.DataFrame(dict(\n    image_name = names, fold=folds))\n\npreds_df = pd.DataFrame(np.concatenate((oof_slope_pred[fcnt:(fcnt+1)*ct_valid],oof_intercept_pred[fcnt:(fcnt+1)*ct_valid]),axis=1),\n                        index=list(range(len(oof_slope_pred[fcnt:(fcnt+1)*ct_valid]))),\n                        columns=['slope' + str(i) for i in range(TTA)]+['intercept' + str(i) for i in range(TTA)])\n\ndf_oof = pd.concat((oof_inf,preds_df),axis=1)\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5: Post process\nI hope somebody pursues this."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}