{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":20270,"databundleVersionId":1222630,"sourceType":"competition"},{"sourceId":1322485,"sourceType":"datasetVersion","datasetId":758225},{"sourceId":1322494,"sourceType":"datasetVersion","datasetId":688574},{"sourceId":1322517,"sourceType":"datasetVersion","datasetId":689329},{"sourceId":1322552,"sourceType":"datasetVersion","datasetId":688719},{"sourceId":1322612,"sourceType":"datasetVersion","datasetId":689578},{"sourceId":1324333,"sourceType":"datasetVersion","datasetId":762256},{"sourceId":1324349,"sourceType":"datasetVersion","datasetId":762108},{"sourceId":1324366,"sourceType":"datasetVersion","datasetId":762176},{"sourceId":1324385,"sourceType":"datasetVersion","datasetId":762138},{"sourceId":1324412,"sourceType":"datasetVersion","datasetId":762168},{"sourceId":1324430,"sourceType":"datasetVersion","datasetId":768238},{"sourceId":1324483,"sourceType":"datasetVersion","datasetId":758409},{"sourceId":1339671,"sourceType":"datasetVersion","datasetId":758489},{"sourceId":1362814,"sourceType":"datasetVersion","datasetId":791650},{"sourceId":1362826,"sourceType":"datasetVersion","datasetId":791647},{"sourceId":1362834,"sourceType":"datasetVersion","datasetId":791645},{"sourceId":1362843,"sourceType":"datasetVersion","datasetId":791643},{"sourceId":1362855,"sourceType":"datasetVersion","datasetId":791638},{"sourceId":1362877,"sourceType":"datasetVersion","datasetId":791635}],"dockerImageVersionId":29981,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How To Upsample and Coarse Dropout with TFRecords\nIn this notebook, we present a template for performing multiple experiments on the same KFold fold. Afterward we can compare validation scores. Or we can repeat the same experiment over and over to assess validation score variability.\n\nWe will explore **upsampling** and **coarse dropout** with TFRecords. When using a dataloader it would be a simple job of returning more minority samples and using Albumentations for augmentation. With TFRecords, it is different. We must write more TF code and make more TFRecords.\n\n## Coarse dropout\n* Coarse dropout is a data augmentation technique to prevent your model from overfitting. We randomly remove squares from training images. (Discussion [here][1]).\n![dropout](http://playagricola.com/Kaggle/drop-7-24.jpg)\n\n## Upsampling \n* Upsampling (oversampling) is a technique to help your model learn the minority class by adding more examples of the minority. Alternatively, we can downsample (undersample) but then we would have less training data. (Discussion [here][2]).\n![dropout](http://playagricola.com/Kaggle/up-7-24.png)\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169721\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169139","metadata":{}},{"cell_type":"markdown","source":"# Initialize Environment","metadata":{}},{"cell_type":"code","source":"!/opt/conda/bin/python3.7 -m pip install --upgrade pip","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null\n\nimport pandas as pd, numpy as np, gc\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configuration\nFirst we choose a number of `EXPERIMENTS`. Then this notebook will perform that many experiments on the same KFold fold. Afterward we can compare validation scores. Or you can repeat the same experiment many times and assess validation score variability. If you have read my previous notebook [here][1], you are familar with most of these configuration variables. We will list the new parameters first:\n\n### New Variables\n* EXPERIMENTS - number of experiments to perform\n* FNUMBER - which of the KFolds to repeatedly perform experiments on. \n* M1 - is a list of length EXPERIMENTS. For each experiment, choose 0, 1, 2, 3, etc. Determines how many additional copies of malignant images from this years comp data to add\n* M2 - is a list. Adds copies of malignant images from ISIC archive that are not in 2020, 2019, 2018, 2017 comp data\n* M3 - is a list. Adds copies of malignant images from 2019 comp data. They have been filtered to include the ones that look like 2020 data\n* M4 - is a list. Adds copies of malignant images from 2018 2017 data.\n* DROP_FREQ - a list of floats between 0 and 1. Determines proportion of train images to apply coarse dropout to\n* DROP_CT - a list of ints. How many squares to remove from train images when applying dropout. (Note that if you use CT>16 with FREQ=1, it may slow down training when using small image resolutions like 128x128 if your CPU is slow).\n* DROP_SIZE - a list of floats between 0 and 1. The size of square side equals `IMG_SIZE * DROP_SIZE`\n* INFER_TEST - whether to predict test images each experiment\n\n### Old Variables\n* DEVICE - is GPU or TPU\n* SEED and FOLDS - a different seed produces a different validation hold out set.\n* IMG_SIZES - is a list of length EXPERIMENTS. These are the image sizes to use each experiment\n* INC2019 - is a list of 0s and 1s. This includes the new half of the 2019 competition data. The second half of the 2019 data is the comp data from 2018 plus 2017\n* INC2018 - is a list of 0s and 1s. This includes the second half of the 2019 competition data which is the comp data from 2018 plus 2017\n* BATCH_SIZES - is a list of length EXPERIMENTS. These are batch sizes for each experiment. For maximum speed, it is best to use the largest batch size your GPU or TPU allows.\n* EPOCHS - is a list of length EXPERIMENTS. These are maximum epochs. Note that each experiment, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* EFF_NETS - is a list of length EXPERIMENTS. These are the EfficientNets to use each experiment. The number refers to the B. So a number of `0` refers to EfficientNetB0, and `1` refers to EfficientNetB1, etc.\n* WGTS - this should be `1/EXPERIMENTS` for each experiment. This is the weight when ensembling the experiments to predict the test set.\n* TTA - test time augmentation. Each validation image is randomly augmented and predicted TTA times and the average prediction is used. TTA is also applied to test images during test prediction.\n\n[1]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords","metadata":{}},{"cell_type":"code","source":"DEVICE = \"GPU\" #or \"TPU\"\n\n# USE DIFFERENT SEED FOR DIFFERENT STRATIFIED KFOLD\nSEED = 42; FOLDS = 5\n\n# WHICH FOLD TO PERFORM EXPERIMENTS ON\nFNUMBER = 3; EXPERIMENTS = 3\n\n# WHICH IMAGE SIZES TO LOAD EACH FOLD\n# CHOOSE 128, 192, 256, 384, 512, 768 \nIMG_SIZES = [128]*EXPERIMENTS\n\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2019 = [0]*EXPERIMENTS\nINC2018 = [1]*EXPERIMENTS\n\n# UPSAMPLE MALIGNANT COUNT TIMES\nM1 = [1,1,1] #2020 malig\nM2 = [1,1,1] #ISIC malig\nM3 = [1,1,1] #2019 good malig\nM4 = [0,0,1] #2018 2017 malig\n\n# COARSE DROPOUT\nDROP_FREQ = [0.7,0.75,0.8] # between 0 and 1\nDROP_CT = [5,8,8] # may slow training if CT>16\nDROP_SIZE = [0.03,0.08,0.11] # between 0 and 1\n\n# BATCH SIZE AND EPOCHS\nBATCH_SIZES = [32]*EXPERIMENTS\nEPOCHS = [12]*EXPERIMENTS\n\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [3,5,6]#[0]*EXPERIMENTS\n\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/EXPERIMENTS]*EXPERIMENTS\n\n# TEST TIME AUGMENTATION STEPS\nTTA = 20\nINFER_TEST = True","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 1: Preprocess\nPreprocess has already been done and saved to TFRecords. Here we choose which size to load. We can use either 128x128, 192x192, 256x256, 384x384, 512x512, 768x768 by changing the `IMG_SIZES` variable in the preceeding code section. These TFRecords are discussed [here][1]. The advantage of using different input sizes is discussed [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147","metadata":{}},{"cell_type":"code","source":"GCS_PATH = [None]*FOLDS; GCS_PATH2 = [None]*FOLDS; GCS_PATH3 = [None]*FOLDS\nfor i,k in enumerate(IMG_SIZES[:FOLDS]):\n    GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n    GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\n    GCS_PATH3[i] = KaggleDatasets().get_gcs_path('malignant-v2-%ix%i'%(k,k))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 2: Data Augmentation\nBelow is TensorFlow code to perform coarse dropout data augmentation on `tf.data.Dataset()`. (Also below is code to perform rotation, sheer, zoom, shift, and color adjustments). Rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. \n\nAdditionally we can decide to use external data by changing the variables `INC2019` and `INC2018` in the preceeding code section. These variables respectively indicate whether to load last year 2019 data and/or year 2018 + 2017 data. These datasets are discussed [here][3]\n\nConsider experimenting with different augmenation and/or external data. The code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work.\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910","metadata":{}},{"cell_type":"code","source":"ROT_ = 180.0; SHR_ = 2.0\nHZOOM_ = 8.0; WZOOM_ = 8.0\nHSHIFT_ = 8.0; WSHIFT_ = 8.0\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image with CT squares of side size SZ*DIM removed\n    \n    # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n    P = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (P==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        # COMPUTE SQUARE \n        WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # DROPOUT IMAGE\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n    image = tf.reshape(image,[DIM,DIM,3])\n    return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name=True):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, augment=True, dim=256, droprate=0, dropct=0, dropsize=0):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img,DIM=dim)\n        if (droprate!=0)&(dropct!=0)&(dropsize!=0): \n            img = dropout(img, DIM=dim, PROBABILITY=droprate, CT=dropct, SZ=dropsize)\n        img = tf.image.random_flip_left_right(img)\n        #img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n                      \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True, batch_size=16, dim=256,\n                droprate=0, dropct=0, dropsize=0):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*2) #if too large causes OOM in GPU CPU\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (\n                prepare_image(img, augment=augment, dim=dim, \n                              droprate=droprate, dropct=dropct, dropsize=dropsize), \n                imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Display Data Augmentation\nBy displaying our data augmentation, this allows us to adjust the parameters until it display to our liking. We will apply lots of dropout to challenge our model and encourage generalization to unseen images. (Code for displaying augmentation is taken from AgentAuers' notebook [here][1]).\n\n[1]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once","metadata":{}},{"cell_type":"code","source":"import PIL, cv2\n\ndef show_dataset(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n    for idx, data in enumerate(iter(ds)):\n        img, target_or_imgid = data\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n        nn = target_or_imgid.numpy().decode(\"utf-8\")\n\n    display(mosaic)\n    return nn","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH9 =  '../input/jpeg-melanoma-128x128/train/'\nfiles_train = tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')\n\n# DROPOUT parameters to display\nRATE = 0.75; CT = 8; SIZE = 0.2\n\n# LOAD DATA AND APPLY AUGMENTATIONS\nds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO).shuffle(1024)\nds = ds.take(1).cache().repeat()\n\nds = ds.map(read_unlabeled_tfrecord, num_parallel_calls=AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, augment=True, dim=IMG_SIZES[0],\n            droprate = RATE, dropct = CT, dropsize = SIZE\n            ), target), num_parallel_calls=AUTO)\nds = ds.take(12*5); ds = ds.prefetch(AUTO)\n\n# DISPLAY IMAGE WITH AND WITHOUT AUGMENTATIONS\nprint('WITH DROPOUT AUGMENTATION - dropout_freq=%.2f count=%i size=%.3f'%(RATE,CT,SIZE))\nname = show_dataset(128, 8, 2, ds)\nimg = cv2.imread(PATH9+name+'.jpg')\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\nplt.imshow(img)\n\nprint('WITHOUT AUGMENTATION - Original Image')\nplt.title('%s'%name,size=16); plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Build Model\nThis is a common model architecute. Consider experimenting with different backbones, custom heads, losses, and optimizers. Also consider inputing meta features into your CNN.","metadata":{}},{"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\ndef build_model(dim=128, ef=0):\n    inp = tf.keras.layers.Input(shape=(dim,dim,3))\n    base = EFNS[ef](input_shape=(dim,dim,3),weights='noisy-student',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.Model(inputs=inp,outputs=x, name='aNetwork')\n    opt = tf.keras.optimizers.Adam(learning_rate=0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=0.05) \n    model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Train Schedule\nThis is a common train schedule for transfer learning. The learning rate starts near zero, then increases to a maximum, then decays over time. Consider changing the schedule and/or learning rates. Note how the learning rate max is larger with larger batches sizes. This is a good practice to follow.","metadata":{}},{"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * REPLICAS * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Experiments\nIn this notebook we run 3 experiments.\n* Experiment 1 - baseline\n* Experiment 2 - add dropout\n* Experiment 3 - add dropout and upsample\n\nThis note book will repeatedly run experiments on the same KFold fold. Each experiment will be trained for the number of EPOCHS you chose in the configuration above. Each experiment the model with lowest validation loss will be saved and used to predict OOF and test. Adjust the variables `VERBOSE` and `DISPLOY_PLOT` below to determine what output you want displayed. The variable `VERBOSE=1 or 2` will display the training and validation loss and auc for each epoch as text. The variable `DISPLAY_PLOT` shows this information as a plot. ","metadata":{}},{"cell_type":"code","source":"skf = KFold(n_splits=FOLDS,shuffle=True,random_state=SEED)\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    if fold==(FNUMBER-1):\n        idxTT = idxT; idxVV = idxV\n        print('### Using fold',fold,'for experiments')\n    print('Fold',fold,'has TRAIN:',idxT,'VALID:',idxV)\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USE VERBOSE=0 for silent, VERBOSE=1 for interactive, VERBOSE=2 for commit\nVERBOSE = 1\nDISPLAY_PLOT = True\n\noof_pred = []; oof_tar = []; oof_val = []; oof_names = []; oof_folds = [] \npreds = np.zeros((count_data_items(files_test),1))\n\nfor fold in range(EXPERIMENTS):\n    # REPEAT SAME FOLD OVER AND OVER\n    idxT = idxTT\n    idxV = idxVV\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### EXPERIMENT',fold+1)\n    print('#### Image Size %i with EfficientNet B%i, batch_size %i, dropout_freq=%.2f count=%i size=%.3f'%\n          (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS,\n           DROP_FREQ[fold],DROP_CT[fold],DROP_SIZE[fold]))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n    if INC2019[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+1])\n        print('#### Using 2019 external data')\n    if INC2018[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2])\n        print('#### Using 2018+2017 external data')\n    for k in range(M1[fold]):\n        files_train += tf.io.gfile.glob([GCS_PATH3[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n        print('#### Upsample MALIG-1 data (2020 comp)')\n    for k in range(M2[fold]):\n        files_train += tf.io.gfile.glob([GCS_PATH3[fold] + '/train%.2i*.tfrec'%x for x in idxT+15])\n        print('#### Upsample MALIG-2 data (ISIC website)')\n    for k in range(M3[fold]):\n        files_train += tf.io.gfile.glob([GCS_PATH3[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+1+30])\n        print('#### Upsample MALIG-3 data (2019 comp)')\n    for k in range(M4[fold]):\n        files_train += tf.io.gfile.glob([GCS_PATH3[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+30])\n        print('#### Upsample MALIG-4 data (2018 2017 comp)')\n    np.random.shuffle(files_train); print('#'*25)\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n   \n    # TRAIN\n    print('Training...')\n    history = model.fit(\n        get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n                dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold],\n                droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold]), \n        epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n        steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n        validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n                repeat=False,dim=IMG_SIZES[fold]), #class_weight = {0:1,1:2},\n        verbose=VERBOSE\n    )\n    \n    print('Loading best model...')\n    model.load_weights('fold-%i.h5'%fold)\n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*2,\n            droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold])\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/2/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=VERBOSE)[:TTA*ct_valid,] \n    oof_pred.append( np.mean(pred.reshape((ct_valid,TTA),order='F'),axis=1) )                 \n    \n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True, return_image_names=True)\n    oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n    oof_folds.append( np.ones_like(oof_tar[-1],dtype='int8')*fold )\n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_names=True)\n    oof_names.append( np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n    \n    # PREDICT TEST USING TTA\n    if INFER_TEST:\n        print('Predicting Test with TTA...')\n        ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*2,\n            droprate=DROP_FREQ[fold], dropct=DROP_CT[fold], dropsize=DROP_SIZE[fold])\n        ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZES[fold]/2/REPLICAS\n        pred = model.predict(ds_test,steps=STEPS,verbose=VERBOSE)[:TTA*ct_test,] \n        preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n    \n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    oof_val.append(np.max( history.history['val_auc'] ))\n    print('#### EXPERIMENT %i OOF AUC without TTA = %.3f, with TTA = %.3f'%(fold+1,oof_val[-1],auc))\n    \n    # PLOT TRAINING\n    if DISPLAY_PLOT:\n        plt.figure(figsize=(15,5))\n        plt.plot(np.arange(EPOCHS[fold]),history.history['auc'],'-o',label='Train AUC',color='#ff7f0e')\n        plt.plot(np.arange(EPOCHS[fold]),history.history['val_auc'],'-o',label='Val AUC',color='#1f77b4')\n        x = np.argmax( history.history['val_auc'] ); y = np.max( history.history['val_auc'] )\n        xdist = plt.xlim()[1] - plt.xlim()[0]; ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#1f77b4'); plt.text(x-0.03*xdist,y-0.13*ydist,'max auc\\n%.2f'%y,size=14)\n        plt.ylabel('AUC',size=14); plt.xlabel('Epoch',size=14)\n        plt.legend(loc=2)\n        plt2 = plt.gca().twinx()\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['loss'],'-o',label='Train Loss',color='#2ca02c')\n        plt2.plot(np.arange(EPOCHS[fold]),history.history['val_loss'],'-o',label='Val Loss',color='#d62728')\n        x = np.argmin( history.history['val_loss'] ); y = np.min( history.history['val_loss'] )\n        ydist = plt.ylim()[1] - plt.ylim()[0]\n        plt.scatter(x,y,s=200,color='#d62728'); plt.text(x-0.03*xdist,y+0.05*ydist,'min loss',size=14)\n        plt.ylabel('Loss',size=14)\n        plt.title('EXPERIMENT %i - Image Size %i, EfficientNet B%i, inc2019=%i, inc2018=%i, M1=%i, M2=%i, M3=%i, M4=%i\\n\\\n        batch_size %i, dropout_freq=%.2f count=%i size=%.3f'%\n                (fold+1,IMG_SIZES[fold],EFF_NETS[fold],INC2019[fold],INC2018[fold],M1[fold],M2[fold],M3[fold],\n                 M4[fold],BATCH_SIZES[fold]*REPLICAS,DROP_FREQ[fold],DROP_CT[fold],DROP_SIZE[fold]),size=18)\n        plt.legend(loc=3)\n        plt.show()  \n        \n    del model; z = gc.collect()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save OOF Preds\nThe OOF (out of fold) predictions are saved to disk. If you wish to ensemble multiple models, use the OOF to determine what are the best weights to blend your models with. Choose weights that maximize OOF CV score when used to blend OOF. Then use those same weights to blend your test predictions.","metadata":{}},{"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof = np.concatenate(oof_pred); true = np.concatenate(oof_tar);\nnames = np.concatenate(oof_names); folds = np.concatenate(oof_folds)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(\n    image_name = names, target=true, pred = oof, fold=folds))\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit To Kaggle\nIf we chose to predict by setting `INFER_TEST = True` above, then we create `submission.csv` here.","metadata":{}},{"cell_type":"code","source":"if INFER_TEST:\n    ds = get_dataset(files_test, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                 labeled=False, return_image_names=True)\n\n    image_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])\n\n    submission = pd.DataFrame(dict(image_name=image_names, target=preds[:,0]))\n    submission = submission.sort_values('image_name') \n    submission.to_csv('first.csv', index=False)\n    submission.head()\n\n    plt.hist(submission.target,bins=100)\n    plt.show()\nelse:\n    print('We did not predict test set')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]}]}