{"cells":[{"metadata":{"id":"2sK4N8oEpxrB"},"cell_type":"markdown","source":"# Melanoma Classification with TPUs\n\n**This notebook is largely just a repurposing of [my kernel](https://www.kaggle.com/tuckerarrants/kfold-efficientnet-cutmix-mixup-tta) for the [Flowers](https://www.kaggle.com/c/tpu-getting-started) competition, adapted for binary classification and with competition specific metrics. I also added course dropout augmentation, taken from Chris Deotte's [starter](https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout); GridMask, taken from [here](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow); and  removed CutMix/MixUp since it is hard to implement in this competition (more [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160784)). I tried to use AugMix, but was not having much sucess with it**\n\n**Update: there was quite a leaderboard shakeup in this competition; moral of the story: trust your CV scores! Congrats to everyone that participated and the winning team `All Data Are Ext` comprised of [Qishen Ha](https://www.kaggle.com/haqishen), [Bo](https://www.kaggle.com/boliu0), and [Gary](https://www.kaggle.com/garybios). I managed to squeeze out a silver medal by blending together predictions from different backbones, image sizes, augmentations, and post processing techniques, all of which can be attained by tweaking this notebook**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"id":"ot-sd-nvpxrB","outputId":"87645e76-1c3b-4ecd-8a45-c8c61226d9db"},"cell_type":"code","source":"#the basics\nfrom matplotlib import pyplot as plt\nimport math, os, re, gc, cv2, random\nimport numpy as np, pandas as pd, seaborn as sns\n\n#deep learning basics\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\n\n#get current TensorFlow version fo\nprint(\"Currently using Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"id":"OoU1g5QwpxrF"},"cell_type":"markdown","source":"# I. Configuration\n\n**To take advantage of TPUs, we have to do some extra work. For the uninitiated, [this](http://www.tensorflow.org/guide/tpu) is an excellent place to start. We start by checking to see if TensorFlow is using a TPU or not - if it isn't, we set the 'strategy' to its default, which works on CPU and a single GPU, though we will definitely need to use the TPU for the current parameter setups of this notebook**","execution_count":null},{"metadata":{"trusted":true,"id":"EkGzSMH4pxrF","outputId":"aab2f9b1-db18-46ed-ea72-6934dff8c53a"},"cell_type":"code","source":"#choose device\nDEVICE = 'TPU'\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"xpYJbPsFpxrM"},"cell_type":"code","source":"#for reproducibility\nSEED = 34 \n\n#select image size             \nIMAGE_SIZE = [256, 256]               \n\n#how many training samples we want going to TPUs \nBATCH_SIZE = 32 * REPLICAS \n\n#how long are we training for?\nEPOCHS = 15\n\n#how many folds we will use to train our model on\nFOLDS = 5\n\n#upsample with external melanoma images (Deotte's)?\nUPSAMPLE = False\n\n#how many TTA steps to apply\n#set to 0 for no TTA\nTTA = 15   \n\n#to see training progress or not\nVERBOSE = 2","execution_count":null,"outputs":[]},{"metadata":{"id":"pYU48G_gpxrI"},"cell_type":"markdown","source":"**TPUs read data directly from Google Cloud Storage (GCS), so we actually need to copy our dataset to a GCS 'bucket' that is near or 'co-located' with the TPU. The below chunk of code accomplishes this using the handy kaggle_datasets. We will be using [Chris Deotte](https://www.kaggle.com/cdeotte)'s TFRecords for both this year's images and also for upsampling images:** ","execution_count":null},{"metadata":{"trusted":true,"id":"6TBv62tNpxrI"},"cell_type":"code","source":"#get GCS path for melanoma classification data set\nfrom kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path(f'melanoma-{IMAGE_SIZE[0]}x{IMAGE_SIZE[1]}')\nGCS_PATH2 = KaggleDatasets().get_gcs_path(f'malignant-v2-{IMAGE_SIZE[0]}x{IMAGE_SIZE[1]}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"PZ9xKjB6pxrK","outputId":"10ca9ede-016a-4e6c-b115-d1488e0a887f"},"cell_type":"code","source":"#can use this path in Google Colabs if you want to run notebook on TPU there instead\nprint(GCS_PATH)\nprint(GCS_PATH2)","execution_count":null,"outputs":[]},{"metadata":{"id":"xTlwnCYTpxrR"},"cell_type":"markdown","source":"**Now we need to create some functions that allow us to extract information from these `TFRecords`. We will create functions that read the image and label from the `TFRecords`. For more about this, see [here](http://www.tensorflow.org/tutorials/load_data/tfrecord)**\n\n**We can also perform some easy augmentations to be used during training and also for test time augmentation. For a quick reference on using `tf.image` to perform image augmentation, see [this](http://www.tensorflow.org/tutorials/images/data_augmentation)**\n\n**To achieve peak performance, we can use a pipeline that 'prefetches' data for the next step before the current step has finished using `tf.data`. You can learn more [here](http://www.tensorflow.org/guide/data_performance)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name = True):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, dim, course_drop = False, grid_mask = False, mat_aug = False, all_aug = False,\n                  droprate = .7, dropct = 8, dropsize = .2):\n    \n    #decode image\n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    #apply course dropout augmentation\n    if course_drop:\n        \n        #flipping augmentations - can apply before or after course dropout function\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_flip_up_down(img)\n        \n        #some other easy transformations we can apply\n        #img = tf.image.random_hue(img, 0.01)\n        #img = tf.image.random_saturation(img, 0.7, 1.3)\n        #img = tf.image.random_contrast(img, 0.8, 1.2)\n        #img = tf.image.random_brightness(img, 0.1)\n        \n        if (droprate!=0)&(dropct!=0)&(dropsize!=0): \n            img = dropout(img, DIM = dim, PROBABILITY = droprate, CT = dropct, SZ = dropsize)\n    \n    #apply shear/shift/zoom augmentation\n    if mat_aug: \n        \n        #flipping augmentations - can apply before or after transform function\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_flip_up_down(img)\n        \n        #some other easy transformations we can apply\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        #img = tf.image.random_contrast(img, 0.8, 1.2)\n        #img = tf.image.random_brightness(img, 0.1)\n        \n        img = transform(img,DIM=dim)\n     \n    #apply grid mask augmentation\n    if grid_mask:\n        \n        #flipping augmentations - can apply before or after grid mask function\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_flip_up_down(img)\n        \n        #some other easy transformations we can apply\n        #img = tf.image.random_hue(img, 0.01)\n        #img = tf.image.random_saturation(img, 0.7, 1.3)\n        #img = tf.image.random_contrast(img, 0.8, 1.2)\n        #img = tf.image.random_brightness(img, 0.1)\\\n        \n        img = apply_grid_mask(img, (dim, dim, 3))\n        \n    #apply combination of all the above\n    if all_aug:\n        \n        #flipping augmentations - can apply before or after grid mask function\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_flip_up_down(img)\n        \n        #some other easy transformations we can apply\n        #img = tf.image.random_hue(img, 0.01)\n        #img = tf.image.random_saturation(img, 0.7, 1.3)\n        #img = tf.image.random_contrast(img, 0.8, 1.2)\n        #img = tf.image.random_brightness(img, 0.1)\\\n        \n        img = apply_all_aug(img, DIM = dim)\n\n    #reshape so TPU knows size of image                 \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"In751JnwpxrU","outputId":"b29b637e-4f9a-4064-c862-75ab4619b2f8"},"cell_type":"code","source":"#define pre fetching strategy\nAUTO = tf.data.experimental.AUTOTUNE\n\n#use tf.io.gfile.glob to find our training and test files from GCS bucket\ntrain_files = np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec'))\ntest_files  = np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec'))\n    \n#show item counts\nNUM_TRAINING_IMAGES = int( count_data_items(train_files) * (FOLDS-1.)/FOLDS )\nNUM_VALIDATION_IMAGES = int( count_data_items(train_files) * (1./FOLDS) )\nNUM_TEST_IMAGES = count_data_items(test_files)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\n\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"n6RnBbJLpxrX"},"cell_type":"code","source":"def get_dataset(files, course_drop = False, grid_mask = False, mat_aug = False, all_aug = False, shuffle = False,\n                repeat = False, labeled = True, return_image_names = True, batch_size = BATCH_SIZE, dim = IMAGE_SIZE[0]):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(2048)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)   \n        \n    ds = ds.map(lambda img, imgname_or_label: (\n                prepare_image(img, course_drop = course_drop, grid_mask = grid_mask ,mat_aug = mat_aug,\n                              all_aug = all_aug, dim=dim,), imgname_or_label), num_parallel_calls = AUTO)\n    \n    ds = ds.batch(batch_size)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{"id":"PTV3XNBmpxrd"},"cell_type":"markdown","source":"# II. Visualization\n\n**Now that we have dealt with all the configuring required to use TPUs, we can extract our images from the TPU and finally get a look at our data:**","execution_count":null},{"metadata":{"trusted":true,"id":"AK_VvHWkpxrd"},"cell_type":"code","source":"#define classes for labeling purposes\nclasses = ['benign', 'malignant']","execution_count":null,"outputs":[]},{"metadata":{"id":"Zav_SHtapxrf"},"cell_type":"markdown","source":"**Define some helper functions to plot our melanoma images:**","execution_count":null},{"metadata":{"trusted":true,"id":"foPjMRkhpxrf"},"cell_type":"code","source":"#numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    #binary strings are image IDs\n    if numpy_labels.dtype == object:\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    #If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return classes[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(classes[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                classes[correct_label] if not correct else ''), correct\n\ndef display_one_image(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    #auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    #size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    #display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else classes[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_image(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #get optimal spacing\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"LSQ4KUiDpxrh"},"cell_type":"code","source":"#first look at training dataset\ntraining_dataset = get_dataset(train_files, labeled = True, course_drop = False, all_aug = False,\n                               grid_mask = False, mat_aug = False, shuffle = True, repeat = True)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"L57gjrovpxrj"},"cell_type":"code","source":"#first look at test dataset\ntest_dataset = get_dataset(test_files, labeled = False, course_drop = False, grid_mask = False,\n                           mat_aug = False, all_aug = False, shuffle = True, repeat = False)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"jriUtSxvpxrl","outputId":"7f577c30-55d3-431a-db9b-687952a0a4af"},"cell_type":"code","source":"#view batch of images from train\ndisplay_batch_of_images(next(train_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0sL0aKTipxrm","outputId":"2a6c6b67-73ce-443d-8cbb-6014c4604299"},"cell_type":"code","source":"#view batch of images from test\ndisplay_batch_of_images(next(test_batch))\n#you can run this cell again and it will load a new batch","execution_count":null,"outputs":[]},{"metadata":{"id":"vE65QNftpxro"},"cell_type":"markdown","source":"# III. Augmentation\n\n**Note: the following augmentation implementations are taken from (4X Kaggle Grandmaster) [Chris Deotte](https://www.kaggle.com/cdeotte)'s notebook, which can be found [here](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/data) and [here](https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout)**\n\n### Rotation, Shift, Zoom, Shear","execution_count":null},{"metadata":{"trusted":true,"id":"dId3DSXjpxrp"},"cell_type":"code","source":"ROT_ = 180.0; SHR_ = 2.0\nHZOOM_ = 8.0; WZOOM_ = 8.0\nHSHIFT_ = 8.0; WSHIFT_ = 8.0\n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    \n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"G58W7nFopxrq"},"cell_type":"code","source":"def transform(image, DIM=256):    \n    \n    XDIM = DIM%2 \n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n              \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#view augmentation\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(train_files, labeled = True, mat_aug = True,\n                           course_drop = False, shuffle = True, repeat = True)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Coarse Dropout\n\n**Using MixUp/CutMix for this problem is a bit tricky as there are only two classes and only about 1.8% of samples are malignant. You can read more about the challenges [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160784) as explained by [Roman](https://www.kaggle.com/nroman)**\n\n**We can instead resort to coarse dropout augmentation for online augmentation. Note that we have an option here: do we apply `transform` for augmentation or `dropout` and for what type of augmentation? You can easily experiment with either (or a combination of both) by changing the parameters of the `get_dataset` function earlier defined:**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 1, CT = 8, SZ = 0.2):\n    \n    prob = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (prob==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        WIDTH = tf.cast( SZ*DIM,tf.int32) * prob\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    image = tf.reshape(image,[DIM,DIM,3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#view what course dropout looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(train_files, labeled = True,\n                           course_drop = True, shuffle = True, repeat = True)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## GridMask\n\n**Now we can explore GridMask, which is essentially a less randomized Course Dropout The below code is taken from [this notebook](https://www.kaggle.com/xiejialun/gridmask-data-augmentation-with-tensorflow)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AugParams = {\n    'd1' : 100,\n    'd2': 160,\n    'rotate' : 45,\n    'ratio' : 0.4\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def transform2(image, inv_mat, image_shape):\n\n    h, w, c = image_shape\n    cx, cy = w//2, h//2\n\n    new_xs = tf.repeat( tf.range(-cx, cx, 1), h)\n    new_ys = tf.tile( tf.range(-cy, cy, 1), [w])\n    new_zs = tf.ones([h*w], dtype=tf.int32)\n\n    old_coords = tf.matmul(inv_mat, tf.cast(tf.stack([new_xs, new_ys, new_zs]), tf.float32))\n    old_coords_x, old_coords_y = tf.round(old_coords[0, :] + w//2), tf.round(old_coords[1, :] + h//2)\n\n    clip_mask_x = tf.logical_or(old_coords_x<0, old_coords_x>w-1)\n    clip_mask_y = tf.logical_or(old_coords_y<0, old_coords_y>h-1)\n    clip_mask = tf.logical_or(clip_mask_x, clip_mask_y)\n\n    old_coords_x = tf.boolean_mask(old_coords_x, tf.logical_not(clip_mask))\n    old_coords_y = tf.boolean_mask(old_coords_y, tf.logical_not(clip_mask))\n    new_coords_x = tf.boolean_mask(new_xs+cx, tf.logical_not(clip_mask))\n    new_coords_y = tf.boolean_mask(new_ys+cy, tf.logical_not(clip_mask))\n\n    old_coords = tf.cast(tf.stack([old_coords_y, old_coords_x]), tf.int32)\n    new_coords = tf.cast(tf.stack([new_coords_y, new_coords_x]), tf.int64)\n    rotated_image_values = tf.gather_nd(image, tf.transpose(old_coords))\n    rotated_image_channel = list()\n    for i in range(c):\n        vals = rotated_image_values[:,i]\n        sparse_channel = tf.SparseTensor(tf.transpose(new_coords), vals, [h, w])\n        rotated_image_channel.append(tf.sparse.to_dense(sparse_channel, default_value=0, validate_indices=False))\n\n    return tf.transpose(tf.stack(rotated_image_channel), [1,2,0])\n\ndef random_rotate(image, angle, image_shape):\n\n    def get_rotation_mat_inv(angle):\n        #transform to radian\n        angle = math.pi * angle / 180\n\n        cos_val = tf.math.cos(angle)\n        sin_val = tf.math.sin(angle)\n        one = tf.constant([1], tf.float32)\n        zero = tf.constant([0], tf.float32)\n\n        rot_mat_inv = tf.concat([cos_val, sin_val, zero,\n                                     -sin_val, cos_val, zero,\n                                     zero, zero, one], axis=0)\n        rot_mat_inv = tf.reshape(rot_mat_inv, [3,3])\n\n        return rot_mat_inv\n    angle = float(angle) * tf.random.normal([1],dtype='float32')\n    rot_mat_inv = get_rotation_mat_inv(angle)\n    return transform2(image, rot_mat_inv, image_shape)\n\n\ndef GridMask(image_height, image_width, d1, d2, rotate_angle=1, ratio=0.5):\n\n    h, w = image_height, image_width\n    hh = int(np.ceil(np.sqrt(h*h+w*w)))\n    hh = hh+1 if hh%2==1 else hh\n    d = tf.random.uniform(shape=[], minval=d1, maxval=d2, dtype=tf.int32)\n    l = tf.cast(tf.cast(d,tf.float32)*ratio+0.5, tf.int32)\n\n    st_h = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n    st_w = tf.random.uniform(shape=[], minval=0, maxval=d, dtype=tf.int32)\n\n    y_ranges = tf.range(-1 * d + st_h, -1 * d + st_h + l)\n    x_ranges = tf.range(-1 * d + st_w, -1 * d + st_w + l)\n\n    for i in range(0, hh//d+1):\n        s1 = i * d + st_h\n        s2 = i * d + st_w\n        y_ranges = tf.concat([y_ranges, tf.range(s1,s1+l)], axis=0)\n        x_ranges = tf.concat([x_ranges, tf.range(s2,s2+l)], axis=0)\n\n    x_clip_mask = tf.logical_or(x_ranges <0 , x_ranges > hh-1)\n    y_clip_mask = tf.logical_or(y_ranges <0 , y_ranges > hh-1)\n    clip_mask = tf.logical_or(x_clip_mask, y_clip_mask)\n\n    x_ranges = tf.boolean_mask(x_ranges, tf.logical_not(clip_mask))\n    y_ranges = tf.boolean_mask(y_ranges, tf.logical_not(clip_mask))\n\n    hh_ranges = tf.tile(tf.range(0,hh), [tf.cast(tf.reduce_sum(tf.ones_like(x_ranges)), tf.int32)])\n    x_ranges = tf.repeat(x_ranges, hh)\n    y_ranges = tf.repeat(y_ranges, hh)\n\n    y_hh_indices = tf.transpose(tf.stack([y_ranges, hh_ranges]))\n    x_hh_indices = tf.transpose(tf.stack([hh_ranges, x_ranges]))\n\n    y_mask_sparse = tf.SparseTensor(tf.cast(y_hh_indices, tf.int64),  tf.zeros_like(y_ranges), [hh, hh])\n    y_mask = tf.sparse.to_dense(y_mask_sparse, 1, False)\n\n    x_mask_sparse = tf.SparseTensor(tf.cast(x_hh_indices, tf.int64), tf.zeros_like(x_ranges), [hh, hh])\n    x_mask = tf.sparse.to_dense(x_mask_sparse, 1, False)\n\n    mask = tf.expand_dims( tf.clip_by_value(x_mask + y_mask, 0, 1), axis=-1)\n\n    mask = random_rotate(mask, rotate_angle, [hh, hh, 1])\n    mask = tf.image.crop_to_bounding_box(mask, (hh-h)//2, (hh-w)//2, image_height, image_width)\n\n    return mask\n\ndef apply_grid_mask(image, image_shape = IMAGE_SIZE):\n    mask = GridMask(image_shape[0],\n                    image_shape[1],\n                    AugParams['d1'],\n                    AugParams['d2'],\n                    AugParams['rotate'],\n                    AugParams['ratio'])\n    \n    if image_shape[-1] == 3:\n        mask = tf.concat([mask, mask, mask], axis=-1)\n\n    return tf.cast(image * tf.cast(mask, tf.float32), tf.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#view what grid mask looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(train_files, labeled = True, grid_mask = True, \n                           course_drop = False, shuffle = True, repeat = True)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## All The Above\n\n**And now we will create a function to apply all the above augmentations with certain probabilities. The current set up delivers a 25/25/25/25 split between rotation/shear/shift/zoom, course dropout, grid mask, and no augmentation. The `no_grid` parameter shifts this to a 50/25/25 split between rotation/shear/shift/zoom, course dropout, and no augmentation**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def apply_all_aug(img, no_grid = False, DIM = IMAGE_SIZE[0]):\n    \n    if tf.random.uniform([],0,1) > .5:\n        if tf.random.uniform([],0,1) > .5:\n            img = transform(img)\n            \n        #apply droupout  \n        else:\n            img = dropout(img, PROBABILITY = 1)\n            \n    else:\n        if not no_grid:\n        #apply grid mask\n            if tf.random.uniform([],0,1) > .5:\n                img = apply_grid_mask(img)\n                \n            else:\n                #do nothing\n                img = img\n                \n        else:\n        #apply transform again \n            if tf.random.uniform([],0,1) > .5:\n                img = transform(img)\n                \n            else:\n                #do nothing\n                img = img\n            \n    return tf.reshape(img,[DIM,DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#view what augmentation combination looks like\nrow = 4; col = 4;\nrow = min(row,BATCH_SIZE//col)\nall_elements = get_dataset(train_files, labeled = True, grid_mask = False, all_aug = True,  \n                           course_drop = False, shuffle = True, repeat = True)\n\nfor (img,label) in all_elements:\n    plt.figure(figsize=(15,15))\n    for j in range(16):\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(img[j,])\n    plt.show()\n    break","execution_count":null,"outputs":[]},{"metadata":{"id":"AiKoUtptpxr3"},"cell_type":"markdown","source":"# IV. Model Training\n\n**Now, it will take far too much time for us to train a model ourselves to learn the optimal weights for classifying our images, so we will instead import a model that has already been pre-trained on ImageNet: a large labeled dataset of real-world images**\n\n**We will be importing several popular pre-trained models. For a list of all the pre-trained models that can be imported with `tf.keras`, see [here](https://keras.io/api/applications/)**","execution_count":null},{"metadata":{"trusted":true,"id":"lC9KziLUpxr3","outputId":"15629c08-7081-4eec-b3d2-e27f25378db8"},"cell_type":"code","source":"#define epoch parameters\nSTEPS_PER_EPOCH = count_data_items(train_files) // BATCH_SIZE\n\n#define learning rate parameters\nLR_START = 5e-6\nLR_MAX = 5e-6 * 8\nLR_MIN = 1e-5\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_DECAY = .8\n\n#define ramp up and decay\ndef lr_schedule(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_schedule, verbose = True)\n\n#visualize learning rate schedule\nrng = [i for i in range(EPOCHS)]\ny = [lr_schedule(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"id":"P6HzJ-4Apxr5"},"cell_type":"markdown","source":"**If you decide to use an EfficientNet model for your final model, you need to install something as it is not yet supported by `keras.applications`. There is another weight option for EffNets to consider that outperforms Imagenet weights called 'Noisy Student' that you can read about [here](https://arxiv.org/abs/1911.04252). For more on EffNets in general, read [this](https://arxiv.org/pdf/1905.11946.pdf)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#import DenseNet201, Xception, InceptionV3, and InceptionResNetV2\nfrom tensorflow.keras.applications import DenseNet201, Xception, InceptionV3, InceptionResNetV2\n\n#requirements to use EfficientNet(s)\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"KBCwXOhTpxr5"},"cell_type":"code","source":"#helper function to create our model\ndef get_DenseNet201():\n    with strategy.scope():\n        dnet = DenseNet201(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        dnet.trainable = True\n        model = tf.keras.Sequential([\n            dnet,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n\n#create Xception model\ndef get_Xception():\n    with strategy.scope():\n        xception = Xception(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        xception.trainable = True\n        model = tf.keras.Sequential([\n            xception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n#create Inception model\ndef get_InceptionV3():\n    with strategy.scope():\n        inception = InceptionV3(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception.trainable = True\n        model = tf.keras.Sequential([\n            inception,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n\n#create InceptionResNet model\ndef get_InceptionResNetV2():\n    with strategy.scope():\n        inception_res = InceptionResNetV2(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'imagenet',\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        inception_res.trainable = True\n        model = tf.keras.Sequential([\n            inception_res,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n\n#create EfficientNetB4 model\ndef get_EfficientNetB4():\n    with strategy.scope():\n        efficient = efn.EfficientNetB4(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n#create EfficientNetB5 model\ndef get_EfficientNetB5():\n    with strategy.scope():\n        efficient = efn.EfficientNetB5(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt,loss = loss,metrics = ['AUC'])\n    return model\n\n#create EfficientNetB6 model\ndef get_EfficientNetB6():\n    with strategy.scope():\n        efficient = efn.EfficientNetB6(\n            input_shape = (IMAGE_SIZE[0], IMAGE_SIZE[1], 3),\n            weights = 'noisy-student', #or imagenet\n            include_top = False\n        )\n        #make trainable so we can fine-tune\n        efficient.trainable = True\n        model = tf.keras.Sequential([\n            efficient,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            #tf.keras.layers.Dropout(0.2),\n            tf.keras.layers.Dense(1, activation = 'sigmoid')\n        ])\n    opt = tf.keras.optimizers.Adam(learning_rate = 0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing = 0.05) \n    model.compile(optimizer = opt, loss = loss,metrics = ['AUC'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**And now we train on 3 three folds. We will predict 'out-of-fold' so that we do not need to save model/model weights from each fold, only the predictions. If we were to try to store each model from the folds in memory and then predict with each model, we would get a memory error. So instead of this:**","execution_count":null},{"metadata":{"trusted":true,"id":"NFw2A8cnpxr8","outputId":"11c21215-a5fa-4384-bdeb-dc800dbf7828"},"cell_type":"code","source":"from sklearn.model_selection import KFold\n#train and cross validate in folds\n\nhistories = []\npreds = np.zeros((count_data_items(test_files),1))\n\n#early_stopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 3)\nkfold = KFold(FOLDS, shuffle = True, random_state = SEED)\n\nfor f,(train_index,val_index) in enumerate(kfold.split(np.arange(15))):\n    \n    #show fold info\n    if DEVICE=='TPU':\n        #hack to clear TPU memory\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('FOLD',f+1); print('#'*25)\n    \n    #create our file paths\n    train_files = tf.io.gfile.glob([GCS_PATH+ '/train%.2i*.tfrec'%x for x in train_index])\n    \n    #now we upsample if we want\n    if UPSAMPLE:\n        train_files += tf.io.gfile.glob([GCS_PATH2 + '/train%.2i*.tfrec'%x for x in train_index])\n    \n    #shuffle for more randomness\n    np.random.shuffle(train_files)\n    \n    #grab validation and test files\n    val_files = tf.io.gfile.glob([GCS_PATH + '/train%.2i*.tfrec'%x for x in val_index])\n    test_files = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))\n    \n    #convert files to datasets\n    train_ds = get_dataset(train_files,labeled = True, return_image_names = False, all_aug = True,\n                           mat_aug = False, course_drop = False, repeat = True, shuffle = True)\n\n    val_ds = get_dataset(val_files, mat_aug = False, course_drop = False, all_aug = False, grid_mask = False,\n                         repeat = False, shuffle = False, labeled = True, return_image_names = False)\n    \n    #define save callback based on loss\n    sv = tf.keras.callbacks.ModelCheckpoint(\n                            f'fold-{f}.h5', monitor = 'val_loss', verbose = 0, save_best_only = True,\n                            save_weights_only = True, mode = 'min', save_freq = 'epoch')\n   \n    #and go!\n    print('Training...'); print('')\n    model = get_EfficientNetB4()\n    history = model.fit(train_ds, validation_data = val_ds, callbacks = [sv, lr_callback],\n                        verbose = VERBOSE, steps_per_epoch = STEPS_PER_EPOCH, epochs = EPOCHS\n    )\n    \n    histories.append(history)\n    print('Loading best model...')\n    model.load_weights(f'fold-{f}.h5')\n    \n    if TTA > 0:\n        ds_test = get_dataset(test_files,labeled = False,return_image_names = False,mat_aug = True,\n                              course_drop = False, repeat = True,shuffle = False)\n        ct_test = count_data_items(test_files); STEPS = TTA * ct_test/BATCH_SIZE\n        pred = model.predict(ds_test,steps = STEPS,verbose = VERBOSE)[:TTA*ct_test,] \n        preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order = 'F'),axis = 1) * 1 / FOLDS\n      \n    else:\n        ds_test = get_dataset(test_files,labeled = False,return_image_names = False, mat_aug = False,\n                              course_drop = False, repeat = False, shuffle = False)   \n        pred = model.predict(ds_test,verbose = VERBOSE)\n        preds += pred * 1 / FOLDS\n     \n    #so we don't hit memory limits\n    del model; z = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"cWj5G2vXpxsI"},"cell_type":"code","source":"#dummy test dataset to grab image names\nds = get_dataset(test_files, course_drop = False, repeat = False, mat_aug = False,\n                 dim = IMAGE_SIZE[0], labeled = False, return_image_names = True)\n\n#snag the image IDs\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])\n\n#save to disk for submission\nsubmission = pd.DataFrame(dict(image_name = image_names, target = preds[:,0]))\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index = False)\n\n#sanity check\nsubmission.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#another sanity check\nprint(f\"Predicted {len(submission[submission['target'] > .5])} images with melanoma\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#define function to visualize learning curves\ndef plot_learning_curves(histories): \n    fig, ax = plt.subplots(1, 2, figsize = (20, 10))\n    \n    #plot accuracies\n    for i in range(0, 3):\n        ax[0].plot(histories[i].history['auc'], color = 'C0')\n        ax[0].plot(histories[i].history['val_auc'], color = 'C1')\n\n    #plot losses\n    for i in range(0, 3):\n        ax[1].plot(histories[i].history['loss'], color = 'C0')\n        ax[1].plot(histories[i].history['val_loss'], color = 'C1')\n\n    #fix legend\n    ax[0].legend(['train', 'validation'], loc = 'upper left')\n    ax[1].legend(['train', 'validation'], loc = 'upper right')\n    \n    #set master titles\n    fig.suptitle(\"Model Performance\", fontsize=14)\n    \n    #label axis\n    for i in range(0,2):\n        ax[0].set_ylabel('Accuracy')\n        ax[0].set_xlabel('Epoch')\n        ax[1].set_ylabel('Loss')\n        ax[1].set_xlabel('Epoch')\n\n    return plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_learning_curves(histories)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#okay last sanity check\nfig, ax = plt.subplots(figsize = (15, 7))\nsns.kdeplot(submission.target, shade = True)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}