{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ** Melanoma - Skin Cancer Detection - EfficientNets Tensorflow 2**\n\nMelanoma is the most serious type of skin cancer, develops in the cells (melanocytes) that produce melanin, early detection is key for recovery.\n\n- Set up TPU\n- Data augmentations in generator - random rotation, random shear, random zoom, random shift\n- Callbacks for learning rate scheduler & model checkpoint saving\n- Binary cross entropy loss & AUC metric tracking\n- Train in full EfficientNets B0 to B7 family use as ensembles\n- Graphing tools for loss and auc metric by epoch for all models\n- Build a decision tree classifier to ensemble EffNet family and take straight average of all EffNet Family\n- Use these two ensembling techniques for inference with test time augmentations\n\nCore code inspiration - https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### **Params/Configs Specification**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"## device setting\nDEVICE = \"TPU\"\n\n## list of B0 to B7 architectures you want to train - provide as list of integers between 0,1,2,3,4,5,6,7\nbuild_effnet_arch = [0, 1, 2, 3, 4, 5, 6, 7]\n\n## configurations\nCFG = dict(\n    \n    ## data sizing\n    batch_size        =  16,\n    read_size         = 256, \n    crop_size         = 250, \n    ## input size of the image to the network\n    net_size          = 248,\n    \n    ## learning rate schedule\n    LR_START          =   0.000003,\n    LR_MAX            =   0.000020,\n    LR_MIN            =   0.000001,\n    LR_RAMPUP_EPOCHS  =   5,\n    LR_SUSTAIN_EPOCHS =   0,\n    LR_EXP_DECAY      =   0.8,\n    epochs            =  15,\n    \n    ## augmentations - rotation, shear, zoom - horizontal & width, shifting - horizontal & width\n    rot               = 180.0,\n    shr               =   1.5,\n    hzoom             =   6.0,\n    wzoom             =   6.0,\n    hshift            =   6.0,\n    wshift            =   6.0,\n\n    \n    ## optimizier \n    optimizer         = 'adam',\n    label_smooth_fac  =   0.05,\n    \n    ## test time augmentation\n    tta_steps         =  25    \n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Install EfficientNet**","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Import Required Libraries**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## regular py imports\nimport os, random, re, math, time\n## set seed\nrandom.seed(a=42)\nimport numpy as np\nimport pandas as pd\n\n## tensorflow imports\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport efficientnet.tfkeras as efn\n\n## import for images - display \nimport PIL\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n## import for pulling in the datasets right inside kaggle\nfrom kaggle_datasets import KaggleDatasets\n\n## import for progress bar\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Read the data**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## list files in kaggle dataset\nBASEPATH = \"../input/siim-isic-melanoma-classification\"\nos.listdir(BASEPATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## read in the train/test data\ndf_train = pd.read_csv(os.path.join(BASEPATH, 'train.csv'))\ndf_test  = pd.read_csv(os.path.join(BASEPATH, 'test.csv'))\ndf_sub   = pd.read_csv(os.path.join(BASEPATH, 'sample_submission.csv'))\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_train.shape)\nprint(df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## get google cloud storage path for melanoma images\nGCS_PATH    = KaggleDatasets().get_gcs_path('melanoma-256x256')\nprint(GCS_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## read in a list of files named like train*.tfrec\n## convert list to array\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **TPU Configuration**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## remember to turn on the accelerator switch to TPU v3-8\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        ## for distributed execution for tf to communicated w clister cluster management systems\n        ## info of the tpu devices\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\n    if tpu:\n        try:\n            print(\"initializing  TPU ...\")\n            ## config for tpu\n            \n            ## connect to clister\n            tf.config.experimental_connect_to_cluster(tpu)\n            ## initialize tpu devices -- pass cluster resolver from above\n            tf.tpu.experimental.initialize_tpu_system(tpu)\n            ## Synchronous training on TPUs -- takes tpu cluster resolver\n            strategy = tf.distribute.experimental.TPUStrategy(tpu)\n            print(\"TPU initialized\")\n        except _:\n            print(\"failed to initialize TPU\")\n    else:\n        DEVICE = \"GPU\"\n\nif DEVICE != \"TPU\":\n    print(\"Using default strategy for CPU and single GPU\")\n    ## execution strategy\n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\":\n    ## list of gpus availble to the host at runtime\n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n    \n\n    \n## auto sharding for data pipelines?\nAUTO     = tf.data.experimental.AUTOTUNE\n\n## print number of devices accessible\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## augmentations and trandformations -- replace with ImageDataGenerator\n## ImageDataGenerator breaks on TPUs, keep checking the open request below\n## https://github.com/tensorflow/tensorflow/issues/34346 \n\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    ## returns filter of rotation, sheared, zoomed, shifted (each transform tyrned on or off) to be convolved?\n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\n\ndef transform(image, cfg):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    \n    DIM = cfg[\"read_size\"]\n    \n    ## dim to trim\n    XDIM = DIM%2 #fix for size 331\n    \n    ## random number from a normal dist for randomizing transforms\n    rot = cfg['rot'] * tf.random.normal([1], dtype='float32')\n    shr = cfg['shr'] * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['hzoom']\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / cfg['wzoom']\n    h_shift = cfg['hshift'] * tf.random.normal([1], dtype='float32') \n    w_shift = cfg['wshift'] * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## define the serialized data formats - tfrecords\ndef read_labeled_tfrecord(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n        'patient_id'                   : tf.io.FixedLenFeature([], tf.int64),\n        'sex'                          : tf.io.FixedLenFeature([], tf.int64),\n        'age_approx'                   : tf.io.FixedLenFeature([], tf.int64),\n        'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n        'diagnosis'                    : tf.io.FixedLenFeature([], tf.int64),\n        'target'                       : tf.io.FixedLenFeature([], tf.int64)\n    } \n    ## parse each example passed\n    example = tf.io.parse_single_example(example, tfrec_format)\n    ## return image and target only not contextual info\n    return example['image'], example['target']\n\n\n## test image read in\ndef read_unlabeled_tfrecord(example, return_image_name):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'image_name'                   : tf.io.FixedLenFeature([], tf.string),\n    }\n    ## parse\n    example = tf.io.parse_single_example(example, tfrec_format)\n    ## read out\n    return example['image'], example['image_name'] if return_image_name else 0\n\n\ndef prepare_image(img, cfg=None, augment=True):\n    ##\n    img = tf.image.decode_jpeg(img, channels=3)\n    ## resize \n    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    ## normalize\n    img = tf.cast(img, tf.float32) / 255.0\n    \n    ## transorm the image if augment is set\n    if augment:\n        img = transform(img, cfg)\n        img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n        img = tf.image.random_flip_left_right(img)\n        img = tf.image.random_hue(img, 0.01)\n        img = tf.image.random_saturation(img, 0.7, 1.3)\n        img = tf.image.random_contrast(img, 0.8, 1.2)\n        img = tf.image.random_brightness(img, 0.1)\n\n    else:\n        img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n                                   \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    ## return the image \n    return img\n\n## not sure\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) \n         for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## create data pipes\ndef get_dataset(files, cfg, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True):\n    \n    ## data pipeline\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ## Caches the elements in this dataset.\n    ds = ds.cache()\n    \n    ## repeat this data\n    if repeat:\n        ds = ds.repeat()\n    \n    ## shuffle randomly the elements of the dataset\n    ## is 1024*8 the buffer size?\n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        ## outputs need to be produced in deterministic order?\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), \n                                               imgname_or_label), \n                num_parallel_calls=AUTO)\n    \n    ## batches\n    ds = ds.batch(cfg['batch_size'] * REPLICAS)\n    ## prefetch elements from this dataset\n    ds = ds.prefetch(AUTO)\n    return ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Test Input Pipeline**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_dataset(thumb_size, cols, rows, ds):\n    ## just a tool to show images in train\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n   \n    for idx, data in enumerate(iter(ds)):\n        img, target_or_imgid = data\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n\n    display(mosaic)\n    \nds = get_dataset(files_train, CFG).unbatch().take(12*5)   \nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Image Augmentation**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## show image augmentation on a particular image\nds = tf.data.TFRecordDataset(files_train, num_parallel_reads=AUTO)\nds = ds.take(1).cache().repeat()\nds = ds.map(read_labeled_tfrecord, num_parallel_calls=AUTO)\nds = ds.map(lambda img, target: (prepare_image(img, cfg=CFG, augment=True), target), \n            num_parallel_calls=AUTO)\nds = ds.take(12*5)\nds = ds.prefetch(AUTO)\n\nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Test Data Images**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## pull out test set images\nds = get_dataset(files_test, CFG, labeled=False).unbatch().take(12*5)   \nshow_dataset(64, 12, 5, ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CALLBACKS...\n\n- Learning Rate scheduler \n- Checkpoint model save after every epoch - added during model fitting","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## set a callback for lr scheduling\ndef get_lr_callback(cfg):\n    lr_start   = cfg['LR_START']\n    lr_max     = cfg['LR_MAX'] * strategy.num_replicas_in_sync\n    lr_min     = cfg['LR_MIN']\n    lr_ramp_ep = cfg['LR_RAMPUP_EPOCHS']\n    lr_sus_ep  = cfg['LR_SUSTAIN_EPOCHS']\n    lr_decay   = cfg['LR_EXP_DECAY']\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n            \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Build Model - helper**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model(cfg, model, Netname):\n    \n    model_input = tf.keras.Input(shape=(cfg['net_size'], cfg['net_size'], 3), name='imgIn')\n    dummy = tf.keras.layers.Lambda(lambda x:x)(model_input)    \n    outputs = []    \n\n    ## get the attributes of efn - effNet\n    constructor = getattr(efn, model)\n    ## remove top, use imagenet weights\n    x = constructor(include_top=False, weights='imagenet', \n                    input_shape=(cfg['net_size'], cfg['net_size'], 3), \n                    pooling='avg')(dummy)\n\n    ## add a dense layer\n    x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n    outputs.append(x)\n    \n    ## create model\n    model = tf.keras.Model(model_input, outputs, name=Netname)\n    model.summary()\n    \n    return model\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Compile Model - helper**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def compile_new_model(cfg, model, Netname):    \n    with strategy.scope():\n        model = get_model(cfg, model, Netname)\n        ## loss -- maybe add cost to miss classifying class 1\n        losses = tf.keras.losses.BinaryCrossentropy(label_smoothing = cfg['label_smooth_fac'])\n        ## compile\n        ## add f1/f2 score\n        model.compile(\n            optimizer = cfg['optimizer'],\n            loss      = losses,\n            metrics   = [tf.keras.metrics.AUC(name='auc')])\n        \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Fit Model**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## directory for model checkpoints\n! mkdir 'model_checkpoints' ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_train     = get_dataset(files_train, CFG, augment=True, shuffle=True, repeat=True)\nds_train     = ds_train.map(lambda img, label: (img, tuple([label])))\n\nsteps_train  = count_data_items(files_train) / (CFG['batch_size'] * REPLICAS)\n\n\nselect_EffNet_Family = []\nselect_EffNet_Family_history = []\nprint(\"\\n Selected Following EffNets for Training\")\nprint([(str('EfficientNetB') + str(i)) for i in build_effnet_arch])\n\n\n## when saving after each epoch - select a few epochs to keep\nkeep_epoch = [15] \nall_epochs = [i for i in range(1,16)]\nrm_list = [str(i).zfill(2) for i in all_epochs if i not in keep_epoch]\n\n\n\nfor arch in build_effnet_arch:\n    eff_arch = str('EfficientNetB') + str(arch)\n    print('\\n******************',eff_arch,'*******************\\n')\n    model = compile_new_model(CFG, eff_arch, eff_arch)\n    ## add a checkpoint callback - save model after every epoch for picking final model\n    ## much more useful when using validation set - monitor validation loss and keep best\n    checkpoint_filename= str(eff_arch)+str('-weights.{epoch:02d}.hdf5')\n    model_checkpoint_callback  =  tf.keras.callbacks.ModelCheckpoint(filepath=os.path.join('model_checkpoints', checkpoint_filename)\n                                                                     , save_weights_only=True\n                                                                     )\n    \n    print((\"\\n Begin Training \"+eff_arch))\n    history = model.fit(ds_train\n                        , verbose = 1\n                        , steps_per_epoch  = steps_train\n                        , epochs           = CFG['epochs']\n                        , callbacks        = [get_lr_callback(CFG)\n                                              ,model_checkpoint_callback])\n    print(\"Done Training \", eff_arch, end = \"\\n\")\n    \n\n    ## checkpoint clean up - Only keep epoch 15, ideally would save multiple like 8,14, 18\n    for i in rm_list:\n        rm_file = str(eff_arch)+str('-weights.')+str(i)+str('.hdf5')\n        os.remove(os.path.join('model_checkpoints', rm_file))\n    print('Checkpoint CleanUp')\n    ##\n\n    ## put model in the family list\n    select_EffNet_Family.append(model)\n    select_EffNet_Family_history.append(history)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## checking checkpoint saves\nos.listdir('model_checkpoints')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## uncomment to download each models weights so you dont have to re-run the training everytime..\n#from IPython.display import FileLink\n#FileLink(r'model_checkpoints/EfficientNetB7-weights.15.hdf5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## loss graphing\ntrain_loss = {}\ni = 0\nfor arch in build_effnet_arch:\n    eff_arch = str('EfficientNetB') + str(arch)\n    train_loss[eff_arch] = select_EffNet_Family_history[i].history['loss']\n\ntrain_loss_df = pd.DataFrame(train_loss)\n## graphing...\ntrain_loss_df.plot.line(title=\"training_loss_per_epoch\", figsize=(18,8))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## loss graphing\ntrain_auc = {}\ni = 0\nfor arch in build_effnet_arch:\n    eff_arch = str('EfficientNetB') + str(arch)\n    train_auc[eff_arch] = select_EffNet_Family_history[i].history['auc']\n\ntrain_auc_df = pd.DataFrame(train_auc)\n## graphing...\ntrain_auc_df.plot.line(title=\"training_auc_per_epoch\", figsize=(18,8))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Weighing EffNet Family for Prediction\n\n- find weights using tree based classifier for weighing all the trained EffNets to get final prediction \n- strategy\n    - find predictions of all effnets on train data\n    - train a not-so-deep tree (grid seach depth?)\n    - use tree on predictions of effnets on test set for final prediction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"CFG['batch_size'] = 16\ncnt_train   = count_data_items(files_train)\nsteps      = cnt_train / (CFG['batch_size'] * REPLICAS)\nds_trainAug = get_dataset(files_train, CFG, augment=False, repeat=False, \n                         labeled=True, return_image_names=True)\n\ni = 0\n## data dict of train predictions\ntrain_preds = {}\nfor arch in build_effnet_arch:\n    eff_arch = str('EfficientNetB') + str(arch)\n    print('\\nPredicting train samples using ',eff_arch)\n    pred = select_EffNet_Family[i].predict(ds_trainAug, verbose=1, steps=steps)\n    print('Test Predictions shape',pred.shape, end = '\\n')\n    pred = np.stack(pred)\n    pred = pred[:,:cnt_train]\n    pred = pred[:df_train.shape[0]]\n    pred = pred.reshape(-1)\n    train_preds[eff_arch] = pred\n    i=i+1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## getting labels\nds = get_dataset(files_train, CFG, augment=False, repeat=False, \n                 labeled=True, return_image_names=True)\n\nimage_labels = np.array([img_label\n                        for img, img_label in iter(ds.unbatch())])\n\n## add to the train_preds dict\ntrain_preds['act_target'] = image_labels\n\n## create a dataframe of train predictions from each model and target \ntrain_model_pred = pd.DataFrame(train_preds)\ntrain_model_pred.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## save\ntrain_model_pred.to_csv(f'intermediate_results.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## build the decision tree classifier on the ensembles predictions to get the final weighted target prediction\n## KISS - small trees - play around for better perf\nfrom sklearn import tree\n\nensemble_model = tree.DecisionTreeClassifier()\nx_ls = [str('EfficientNetB') + str(arch) for arch in build_effnet_arch]\nX_train = train_model_pred.loc[:,x_ls]\ny_train = train_model_pred.loc[:,'act_target']\nensemble_model.fit(X_train, y_train)\ny_predict = ensemble_model.predict(X_train)\n\n## auc \nfrom sklearn import metrics\nfpr, tpr, thresholds = metrics.roc_curve(y_train, y_predict)\nprint(metrics.auc(fpr, tpr))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## vizuvalize the tree\nfig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (20,20), dpi=300)\n\ncn=['No_Melanoma', 'Yes_Melanoma']\ntree.plot_tree(ensemble_model\n               , feature_names = x_ls\n               , class_names=cn\n               , filled = True)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TestSet Prediction\n\n- use the tree classifier above to combine predictions from individual EffNet model\n- Straight average the predictions from all the models ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"CFG['batch_size'] = 256\n\ncnt_test   = count_data_items(files_test)\nsteps      = cnt_test / (CFG['batch_size'] * REPLICAS) * CFG['tta_steps']\nds_testAug = get_dataset(files_test, CFG, augment=True, repeat=True, \n                         labeled=False, return_image_names=False)\n\n\ni = 0\ntest_preds = {}\nfor arch in build_effnet_arch:\n    eff_arch = str('EfficientNetB') + str(arch)\n    print('\\nPredicting test samples using ',eff_arch)\n    pred = select_EffNet_Family[i].predict(ds_testAug, verbose=1, steps=steps)\n    print('Test Predictions (tta) shape',pred.shape)\n    pred = np.stack(pred)\n    pred = pred[:,:cnt_test* CFG['tta_steps']]\n    pred = pred[:df_test.shape[0]*CFG['tta_steps']]\n    pred = np.stack(np.split(pred, CFG['tta_steps']),axis=1)\n    pred = np.mean(pred, axis=1)\n    pred = pred.reshape(-1)\n    test_preds[eff_arch] = pred\n    i=i+1\n\ntest_ensemble_pred = pd.DataFrame(test_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## save intermediate\ntest_ensemble_pred.to_csv(f'intermediate_test_results.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## use tree classifier from above to combine test predictions\ntree_predict = ensemble_model.predict_proba(test_ensemble_pred)[:,1]\nprint(tree_predict.shape)\n\n## straight average test predictions\npreds = test_ensemble_pred.sum(axis=1)/test_ensemble_pred.shape[1]\nprint(preds.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Submission**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"## create the image names\nds = get_dataset(files_test, CFG, augment=False, repeat=False, \n                 labeled=False, return_image_names=True)\n\nimage_names = np.array([img_name.numpy().decode(\"utf-8\") \n                        for img, img_name in iter(ds.unbatch())])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## preds for decision tree\n\nsubmission = pd.DataFrame(dict(\n    image_name = image_names,\n    target     = tree_predict))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv(f'submission_1.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## preds for straight averaging\n\nsubmission = pd.DataFrame(dict(\n    image_name = image_names,\n    target     = preds))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv(f'submission_2.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ensemble_pred.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## just use B7 predictions - all nets were tracking eachothers performance on test from the graphs above \nsubmission = pd.DataFrame(dict(\n    image_name = image_names,\n    target     = test_ensemble_pred['EfficientNetB0']))\n\nsubmission = submission.sort_values('image_name') \nsubmission.to_csv(f'submission_10.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sources/Acknowlegements:\n\n- AgentAuers's notebook : https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once\n- Kaggle Documentation \n- EffNets google blog\n\n## Next Steps:\n- Add past years data \n- Add eval mode - validation set \n- loss function - penalize false negetives more?","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}