{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Triple Stratified KFold CV with TFRecords\nThis is a simple starter notebook for Kaggle's Melanoma Comp showing triple stratifed KFold with TFRecords. Triple stratified KFold is explained [here][2]. There are many configuration variables below to allow you to experiment. Use either GPU or TPU. You can control which size images are loaded, which efficientNets are used, and whether external data is used. You can experiment with different data augmentation, model architecture, loss, optimizers, and learning schedules. The TFRecords contain meta data, so you can input that into your CNN too. \n\n**NOTE:** this notebook lets you run a different experiment in each fold if you want to run lots of experiments. (Then it is like running multiple holdout validaiton experiments and then the overall CV score is meaningless cause LB will be much higher when the multiple experiments are ensembled to predict test). **If you want a proper CV with a reliable CV score you need to choose the same configuration for each fold.**\n\nThis notebook follows the 5 step process presented in my \"How to compete with GPUs Workshop\" [here][1]. Some code sections have been reused from AgentAuers' great notebook [here][3]\n\n[1]: https://www.kaggle.com/cdeotte/how-to-compete-with-gpus-workshop\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526\n[3]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Initialize Environment","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet >> /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf, re, math\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration\nIn order to be a proper cross validation with a meaningful CV score (which will correlate with LB score), **you need to choose the same** `IMG_SIZES`, `INC2019`, `INC2018`, and `EFF_NETS` **for each fold**. If your goal is to just run lots of experiments, then you can choose to have a different experiment in each fold. Then each fold is like a holdout validation experiment. When you find a configuration you like, you can use that configuration for all folds. \n* DEVICE - is GPU or TPU\n* FOLDS - number of folds. Best set to 3, 5, or 15 but can be any number between 2 and 15\n* IMG_SIZES - is a Python list of length FOLDS. These are the image sizes to use each fold\n* INC2019 - This includes the new half of the 2019 competition data. The second half of the 2019 data is the comp data from 2018 plus 2017\n* INC2018 - This includes the second half of the 2019 competition data which is the comp data from 2018 plus 2017\n* BATCH_SIZES - is a list of length FOLDS. These are batch sizes for each fold\n* EPOCHS - is a list of length FOLDS. These are maximum epochs. Note that each fold, the best epoch model is saved and used. So if epochs is too large, it won't matter.\n* EFF_NETS - is a list of length FOLDS. These are the EfficientNets to use each fold. The number refers to the B. So a number of `0` refers to EfficientNetB0, and `1` refers to EfficientNetB1, etc.\n* WGTS - this should be `1/FOLDS` for each fold. This is the weight when ensembling the folds to predict the test set. If you want a weird ensemble, you can use different weights.\n* TTA - test time augmentation. Each test image is randomly augmented and predicted TTA times and the average prediction is used. TTA is also applied to OOF during validation.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# DEVICE = \"TPU\" #or \"GPU\"\n\n# train = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\n# test  = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n# sub   = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\n\n# FOLDS = 8\n# # WHICH IMAGE SIZES TO LOAD EACH FOLD\n# # CHOOSE 128, 192, 256, 384, 512, 768 \n# IMG_SIZES = [128, 192, 256, 384, 384, 512, 512, 768]\n# # INCLUDE OLD COMP DATA? YES=1 NO=0\n# INC2019 = [1,0,1,1,0,1,0,0]\n# INC2018 = [0,1,1,0,1,1,1,0]\n# BATCH_SIZES = [64,64,64,64,64,64,32,32]   #[64,64,64]\n# EPOCHS = [20, 20, 20, 15, 15, 15, 12, 12]   #[10, 10, 10]\n# # WHICH EFFICIENTNET B? TO USE\n# EFF_NETS = [0,1,2,3,4,5,6,7]\n# # WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\n# WGTS = [1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS]#[1/FOLDS,1/FOLDS,1/FOLDS]\n# # TEST TIME AUGMENTATION STEPS\n# TTA = 8","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# DEVICE = \"TPU\" #or \"GPU\"\n\n# train = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\n# test  = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n# sub   = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\n\n# FOLDS = 15\n# # WHICH IMAGE SIZES TO LOAD EACH FOLD\n# # CHOOSE 128, 192, 256, 384, 512, 768 \n# IMG_SIZES = [128, 192, 256, 256, 256, 256, 384, 384, 384, 512, 512, 512, 768, 768, 768]\n# # INCLUDE OLD COMP DATA? YES=1 NO=0\n# INC2019 = [1,0,1,1,0,1,0,0, 1,0,1,1,0,1,0]\n# INC2018 = [0,1,1,0,1,1,1,0, 0,1,1,0,1,1,1]\n# BATCH_SIZES = [64,64,64,64,64,64,64,64, 64,64,64,64,32,32,32]   #[64,64,64]\n# EPOCHS = [20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 25, 25, 25, 25, 25]   #[10, 10, 10]\n# # WHICH EFFICIENTNET B? TO USE\n# EFF_NETS = [0,0,1,1,2,3,3,4, 4,5,5,6,6,7, 7]\n# # WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\n# WGTS = [1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,\n#        1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS,1/FOLDS]#[1/FOLDS,1/FOLDS,1/FOLDS]\n# # TEST TIME AUGMENTATION STEPS\n# TTA = 15","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DEVICE = \"TPU\" #or \"GPU\"\n\ntrain = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\ntest  = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\nsub   = pd.read_csv('../input/siim-isic-melanoma-classification/sample_submission.csv')\n\nFOLDS = 8\n# WHICH IMAGE SIZES TO LOAD EACH FOLD\n# CHOOSE 128, 192, 256, 384, 512, 768 \nIMG_SIZES = [128, 192, 256, 256, 256, 384, 384, 512]\n# INCLUDE OLD COMP DATA? YES=1 NO=0\nINC2019 = [1,0,1,1,0,1,0,1]\nINC2018 = [0,1,1,0,1,1,1,1]\nBATCH_SIZES = [64,64,64,64,64,64,32,32]   #[64,64,64]\nEPOCHS = [20, 20, 20, 20, 20, 20, 20, 20]   #[10, 10, 10]\n# WHICH EFFICIENTNET B? TO USE\nEFF_NETS = [0,1,2,2,3,3,3,4]\n# WEIGHTS FOR FOLD MODELS WHEN PREDICTING TEST\nWGTS = [1/FOLDS]*FOLDS#[1/FOLDS,1/FOLDS,1/FOLDS]\n# TEST TIME AUGMENTATION STEPS\nTTA = 15","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            tf.config.experimental_connect_to_cluster(tpu)\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\nAUTO     = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Preprocess\nPreprocess has already been done and saved to TFRecords. Here we choose which size to load. We can use either 128x128, 192x192, 256x256, 384x384, 512x512, 768x768 by changing the `IMG_SIZES` variable in the preceeding code section. These TFRecords are discussed [here][1]. The advantage of using different input sizes is discussed [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = [None]*FOLDS; GCS_PATH2 = [None]*FOLDS\nfor i,k in enumerate(IMG_SIZES):\n    GCS_PATH[i] = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))\n    GCS_PATH2[i] = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k))\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[0] + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Data Augmentation\nThis notebook uses rotation, sheer, zoom, shift augmentation first shown in this notebook [here][1] and successfully used in Melanoma comp by AgentAuers [here][2]. This notebook also uses horizontal flip, hue, saturation, contrast, brightness augmentation similar to last years winner and also similar to AgentAuers' notebook.\n\nAdditionally we can decide to use external data by changing the variables `INC2019` and `INC2018` in the preceeding code section. These variables respectively indicate whether to load last year 2019 data and/or year 2018 + 2017 data. These datasets are discussed [here][3]\n\nThe code to load TFRecords is taken from AgentAuers' notebook [here][2]. Thank you AgentAuers, this is great work.\n\n[1]: https://www.kaggle.com/cdeotte/rotation-augmentation-gpu-tpu-0-96\n[2]: https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['target']\n\n\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, tfrec_format)\n    return example['image'], example['image_name'] if return_image_name else 0\n\n \ndef prepare_image(img, augment=True, dim=256):    \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    if augment:\n        img = transform(img,DIM=dim)\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n                      \n    img = tf.reshape(img, [dim,dim, 3])\n            \n    return img\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_dataset(files, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True, batch_size=16, dim=256):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, dim=dim), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ds = ds.batch(batch_size * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Build Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"EFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\ndef build_model(dim=128, ef=0):\n    inp = tf.keras.layers.Input(shape=(dim,dim,3))\n    base = EFNS[ef](input_shape=(dim,dim,3),weights='imagenet',include_top=False)\n    x = base(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n    model = tf.keras.Model(inputs=inp,outputs=x)\n    opt = tf.keras.optimizers.Adam(learning_rate=0.001)\n    loss = tf.keras.losses.BinaryCrossentropy(label_smoothing=0.05) \n    model.compile(optimizer=opt,loss=loss,metrics=['AUC'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4: Train Schedule","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.000020 * REPLICAS * batch_size/16\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = KFold(n_splits=FOLDS,shuffle=True,random_state=42)\noof_pred = []; oof_tar = []; oof_val = []; oof_names = [] \npreds = np.zeros((count_data_items(files_test),1))\n\nfor fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    \n    # DISPLAY FOLD INFO\n    if DEVICE=='TPU':\n        if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n    print('#'*25); print('#### FOLD',fold+1)\n    print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n          (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n    # CREATE TRAIN AND VALIDATION SUBSETS\n    files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n    if INC2019[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+1])\n        print('#### Using 2019 external data')\n    if INC2018[fold]:\n        files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2])\n        print('#### Using 2018+2017 external data')\n    np.random.shuffle(files_train); print('#'*25)\n    files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n    files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n    # BUILD MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n    # SAVE BEST MODEL EACH FOLD\n    sv = tf.keras.callbacks.ModelCheckpoint(\n        'fold-%i.h5'%fold, monitor='val_loss', verbose=1, save_best_only=True,\n        save_weights_only=True, mode='min', save_freq='epoch')\n   \n    # TRAIN\n#     history = model.fit(\n#         get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n#                 dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold]), \n#         epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n#         steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n#         validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n#                 repeat=False,dim=IMG_SIZES[fold]), #class_weight = {0:1,1:2},\n#         verbose=1\n#     )\n    \n    print('Loading model...')\n    #model.load_weights('fold-%i.h5'%fold)\n    \n    model.load_weights('../input/melanoma-weights/fold-%i.h5'%fold)\n    \n    # PREDICT OOF USING TTA\n    print('Predicting OOF with TTA...')\n    ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n    pred = model.predict(ds_valid,steps=STEPS,verbose=1)[:TTA*ct_valid,] \n    oof_pred.append( np.mean(pred.reshape((ct_valid,TTA),order='F'),axis=1) )                 \n    #oof_pred.append(model.predict(get_dataset(files_valid,dim=IMG_SIZES[fold]),verbose=1))\n    \n    # GET OOF TARGETS AND NAMES\n    ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n            labeled=True, return_image_names=True)\n    oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n    ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                labeled=False, return_image_names=True)\n    oof_names.append( np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n    \n    # PREDICT TEST USING TTA\n    print('Predicting Test with TTA...')\n    ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n            repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n    ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZES[fold]/4/REPLICAS\n    pred = model.predict(ds_test,steps=STEPS,verbose=1)[:TTA*ct_test,] \n    preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n    \n    # REPORT RESULTS\n    auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n    #oof_val.append(np.max( history.history['val_auc'] ))\n    \n    \n    #print('#### FOLD %i OOF AUC with TTA = %.3f, without TTA = %.3f'%(fold+1,auc,oof_val[-1]))\n    \n    print('#### FOLD %i OOF AUC with TTA = %.3f, without TTA = --'%(fold+1,auc))\n    print()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# skf = KFold(n_splits=FOLDS,shuffle=True,random_state=42)\n# oof_pred = []; oof_tar = []; oof_val = []; oof_names = [] \n# preds = np.zeros((count_data_items(files_test),1))\n\n# for fold,(idxT,idxV) in enumerate(skf.split(np.arange(15))):\n    \n#     # DISPLAY FOLD INFO\n#     if DEVICE=='TPU':\n#         if tpu: tf.tpu.experimental.initialize_tpu_system(tpu)\n#     print('#'*25); print('#### FOLD',fold+1)\n#     print('#### Image Size %i with EfficientNet B%i and batch_size %i'%\n#           (IMG_SIZES[fold],EFF_NETS[fold],BATCH_SIZES[fold]*REPLICAS))\n    \n#     # CREATE TRAIN AND VALIDATION SUBSETS\n#     files_train = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxT])\n#     if INC2019[fold]:\n#         files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2+1])\n#         print('#### Using 2019 external data')\n#     if INC2018[fold]:\n#         files_train += tf.io.gfile.glob([GCS_PATH2[fold] + '/train%.2i*.tfrec'%x for x in idxT*2])\n#         print('#### Using 2018+2017 external data')\n#     np.random.shuffle(files_train); print('#'*25)\n#     files_valid = tf.io.gfile.glob([GCS_PATH[fold] + '/train%.2i*.tfrec'%x for x in idxV])\n#     files_test = np.sort(np.array(tf.io.gfile.glob(GCS_PATH[fold] + '/test*.tfrec')))\n    \n#     # BUILD MODEL\n#     K.clear_session()\n#     with strategy.scope():\n#         model = build_model(dim=IMG_SIZES[fold],ef=EFF_NETS[fold])\n        \n#     # SAVE BEST MODEL EACH FOLD\n#     sv = tf.keras.callbacks.ModelCheckpoint(\n#         'fold-%i.h5'%fold, monitor='val_loss', verbose=1, save_best_only=True,\n#         save_weights_only=True, mode='min', save_freq='epoch')\n   \n#     # TRAIN\n#     history = model.fit(\n#         get_dataset(files_train, augment=True, shuffle=True, repeat=True,\n#                 dim=IMG_SIZES[fold], batch_size = BATCH_SIZES[fold]), \n#         epochs=EPOCHS[fold], callbacks = [sv,get_lr_callback(BATCH_SIZES[fold])], \n#         steps_per_epoch=count_data_items(files_train)/BATCH_SIZES[fold]//REPLICAS,\n#         validation_data=get_dataset(files_valid,augment=False,shuffle=False,\n#                 repeat=False,dim=IMG_SIZES[fold]), #class_weight = {0:1,1:2},\n#         verbose=1\n#     )\n    \n#     print('Loading model...')\n#     model.load_weights('fold-%i.h5'%fold)\n    \n#     # PREDICT OOF USING TTA\n#     print('Predicting OOF with TTA...')\n#     ds_valid = get_dataset(files_valid,labeled=False,return_image_names=False,augment=True,\n#             repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n#     ct_valid = count_data_items(files_valid); STEPS = TTA * ct_valid/BATCH_SIZES[fold]/4/REPLICAS\n#     pred = model.predict(ds_valid,steps=STEPS,verbose=1)[:TTA*ct_valid,] \n#     oof_pred.append( np.mean(pred.reshape((ct_valid,TTA),order='F'),axis=1) )                 \n#     #oof_pred.append(model.predict(get_dataset(files_valid,dim=IMG_SIZES[fold]),verbose=1))\n    \n#     # GET OOF TARGETS AND NAMES\n#     ds_valid = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n#             labeled=True, return_image_names=True)\n#     oof_tar.append( np.array([target.numpy() for img, target in iter(ds_valid.unbatch())]) )\n#     ds = get_dataset(files_valid, augment=False, repeat=False, dim=IMG_SIZES[fold],\n#                 labeled=False, return_image_names=True)\n#     oof_names.append( np.array([img_name.numpy().decode(\"utf-8\") for img, img_name in iter(ds.unbatch())]))\n    \n#     # PREDICT TEST USING TTA\n#     print('Predicting Test with TTA...')\n#     ds_test = get_dataset(files_test,labeled=False,return_image_names=False,augment=True,\n#             repeat=True,shuffle=False,dim=IMG_SIZES[fold],batch_size=BATCH_SIZES[fold]*4)\n#     ct_test = count_data_items(files_test); STEPS = TTA * ct_test/BATCH_SIZES[fold]/4/REPLICAS\n#     pred = model.predict(ds_test,steps=STEPS,verbose=1)[:TTA*ct_test,] \n#     preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n    \n#     # REPORT RESULTS\n#     auc = roc_auc_score(oof_tar[-1],oof_pred[-1])\n#     oof_val.append(np.max( history.history['val_auc'] ))\n#     print('#### FOLD %i OOF AUC with TTA = %.3f, without TTA = %.3f'%(fold+1,auc,oof_val[-1]))\n#     print()    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Calculate OOF AUC","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE OVERALL OOF AUC\noof = np.concatenate(oof_pred)\nnames = np.concatenate(oof_names)\ntrue = np.concatenate(oof_tar)\nauc = roc_auc_score(true,oof)\nprint('Overall OOF AUC with TTA = %.3f'%auc)\n\n# SAVE OOF TO DISK\ndf_oof = pd.DataFrame(dict(image_name = names, pred = oof, target=true))\ndf_oof.to_csv('oof.csv',index=False)\ndf_oof.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5: Post process\nThere are ways to modify predictions based on patient information to increase CV LB. You can experiment with that here on your OOF.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Submit To Kaggle","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = get_dataset(files_test, augment=False, repeat=False, dim=IMG_SIZES[fold],\n                 labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(dict(image_name=image_names, target=preds[:,0]))\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(submission.target,bins=100)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}