{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Pipeline for dual input CNN\n#### This is a simple notebook which is made modular so that one can pick up pieces of code and use to their own requirement. \n\n#### The code is made such that you can try out a bunch of different things while changing only a few lines of code!\n\n#### You can try different ensembles from here to boost your scores.\n\nYou can control which size images are loaded, which efficientNets are used, and whether external data is used. You can experiment with different data augmentation, model architecture, loss, optimizers, and learning schedules. The TFRecords contain meta data, so you can input that into your CNN too.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Kaggle's SIIM-ISIC Melanoma Classification\nIn this competition, we need to identify melanoma in images of skin lesions. Full description [here][1]. This is a very challenging image classification task as seen by looking at the sample images below. Can you recognize the differences between images? Below are example of skin images with and without melanoma.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification\n\n\n\n\nSpecial thanks to @cdeotte for the motivation behind this notebook. I have implemented many of his ideas ! ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import cv2, pandas as pd, matplotlib.pyplot as plt\ntrain = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\nprint('Examples WITH Melanoma')\nimgs = train.loc[train.target==1].sample(10).image_name.values\nplt.figure(figsize=(10,4))\nfor i,k in enumerate(imgs):\n    img = cv2.imread('../input/jpeg-melanoma-128x128/train/%s.jpg'%k)\n    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n    plt.subplot(2,5,i+1); plt.axis('off')\n    plt.imshow(img)\nplt.show()\nprint('Examples WITHOUT Melanoma')\nimgs = train.loc[train.target==0].sample(10).image_name.values\nplt.figure(figsize=(10,4))\nfor i,k in enumerate(imgs):\n    img = cv2.imread('../input/jpeg-melanoma-128x128/train/%s.jpg'%k)\n    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n    plt.subplot(2,5,i+1); plt.axis('off')\n    plt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The purpose of this notebook\n\n*   Here , we write code that can train a model on both **image + metadata**. This helps the model get more information about the image.\n\n*   Such ***dual-input models*** try to give an \"Idea\" to the model about what is it looking at.(Ex. a scan of melanoma on the palm will be completely different from that of the head of a person)\n\n \n*   We also use ***test-time-augmentation*** to improve model performance.\n*   Coarse Dropout has also been added to prevent any overfitting.\n*   Notebook is clearly divided into segments for easy understanding.\n* \n*  If you are a beginner, some things may not come to you on the first read. Understand it one at a time and keep googling if you cant understand something.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. Setup environment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"id":"aH1svRGNTBtD","trusted":true},"cell_type":"code","source":"colab=0\nshow_files=0\ntstamp=0\n\n\n\nif colab:\n    from google.colab import drive\n    drive.mount('/content/gdrive')\n\nif (not colab)&show_files:\n    import os\n    for dirname, _, filenames in os.walk('/kaggle/input'):\n        for filename in filenames:\n            print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output\n\n### Loading libraries\n\n!pip install -q efficientnet\n\nimport math\nimport pytz\nimport random\nimport numpy as np\nimport pandas as pd\nimport math, re, os, gc\nimport tensorflow as tf\nfrom pathlib import Path\nfrom datetime import datetime\nfrom scipy.stats import rankdata\nimport efficientnet.tfkeras as efn\nfrom matplotlib import pyplot as plt\nfrom sklearn.utils import class_weight\nfrom sklearn.metrics import roc_auc_score\n\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\n\nif not colab:\n    from kaggle_datasets import KaggleDatasets\n\n### Loading data\n\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Input","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## C O N F I G U R A T I O N","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Configuration\nNAME='EffNB6_512'\nNFOLDS=5\nNBEST=1 # the number of best models to use for predictions, can set as 1 for simplicity\nSEED=311 # random seed\nef = 3   # Version of efficientNetB? to use\n\n\n#For Coarse dropout \nDROPOUT = True # Whether to use coarse dropout technique or not\ndroprate=0.5 # Between 0 and 1\ndropct= 4 # May slow training if CT>16\ndropsize=0.2 # between 0 and 1\n\n\n\nTTA = 3 # Test Time Augmentation Steps\n\n\nEPOCHS = 10\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [384,384]\n\nANATOM = 1 # Whether or not to use anatom_site_general_challenge feature in metadata\n\n\ndim = IMAGE_SIZE[0] #image dimensions\nDIM = dim\n# IMAGE_SIZE = [256 , 256]\n\n# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n\n# GCS_PATH = [None]*FOLDS; GCS_PATH2 = [None]*FOLDS\n# for i,k in enumerate(IMG_SIZES):\nk = IMAGE_SIZE[0]\n\n#Competition data\nGCS_PATH = KaggleDatasets().get_gcs_path('melanoma-%ix%i'%(k,k))  \n\n#External Data from 2019 competition \nGCS_PATH2 = KaggleDatasets().get_gcs_path('isic2019-%ix%i'%(k,k)) \nGCS_PATH3 = KaggleDatasets().get_gcs_path('malignant-v2-%ix%i'%(k,k))\n \n\nUSE_EXT_DATA = True # use ext data or not in training\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nfiles_test  = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## About Coarse dropout technique\n* Coarse dropout is a data augmentation technique to prevent your model from overfitting. We randomly remove squares from training images. (Discussion [here][1]).\n![dropout](http://playagricola.com/Kaggle/drop-7-24.jpg)\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# files_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def append_path(pre):\n    return np.vectorize(lambda file: os.path.join(GCS_DS_PATH, pre, file))\n\nsub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\n\ntrain = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\n\nimport seaborn as sns\nsns.countplot(train['target'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ### CREATING TRAIN AND VALID","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nALL_TRAIN=tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nALL_TRAIN1 = tf.io.gfile.glob(GCS_PATH2 + '/train*.tfrec')\nALL_TRAIN2 = tf.io.gfile.glob(GCS_PATH3 + '/train*.tfrec')\nALL_TRAIN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## generating Validation files\ntot_train = len(ALL_TRAIN)\nskip = tot_train//NFOLDS\nVAL_FNAMES={}\nx = 0\n\n\nfor n in range(1, NFOLDS+1):\n    VAL_FNAMES[f\"fold_{n}\"] = ALL_TRAIN[x : x+skip]\n    x += skip\n\nif USE_EXT_DATA :    \n    ALL_TRAIN += ALL_TRAIN1\n    ALL_TRAIN += ALL_TRAIN2\n\n    \n### Train files    \nTRAIN_FNAMES={f'fold_{i}': list(set(ALL_TRAIN)-set(VAL_FNAMES[f'fold_{i}']))\n              for i in range(1, NFOLDS+1)}\n\n\n### Test files\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_FNAMES","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint(f\"total folds : {len(TRAIN_FNAMES)} Total per fold : {len(TRAIN_FNAMES['fold_1'])}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To enable CV","execution_count":null},{"metadata":{"id":"9Yb5aY0cuglL","trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    tf.random.set_seed(seed)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","id":"mCiMLPNDuglP","trusted":true},"cell_type":"code","source":"\nif colab:\n    PATH=Path('/content/gdrive/My Drive/kaggle/input/siim-isic-melanoma-classification/') \n    train=pd.read_csv(PATH/'train.csv.zip')\nelse:\n    PATH=Path('/kaggle/input/siim-isic-melanoma-classification/')\n    train=pd.read_csv(PATH/'train.csv')\n\ntest=pd.read_csv(PATH/'test.csv')\nsub=pd.read_csv(PATH/'sample_submission.csv')\n\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"id":"zwrju15xuglg"},"cell_type":"markdown","source":"# 3. TPU-friendly functions\n","execution_count":null},{"metadata":{"id":"L1cjq7Jkuglh","outputId":"5dfbbab1-5f2c-4b0b-b917-94393d0e00f8","trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment \n    # variable is set. On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"id":"AU7BuL2Pugmo","trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, \n    # i.e. test10-687.tfrec = 687 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    \n    return np.sum(n)\n\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    # convert image to floats in [0, 1] range\n    image = tf.cast(image, tf.float32) / 255.0 \n    # explicit size needed for TPU\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    \n    return image\n\ndef read_labeled_tfrecord(example) :\n    LABELED_TFREC_FORMAT = {\n      'image':  tf.io.FixedLenFeature([], tf.string),\n      'image_name':tf.io.FixedLenFeature([], tf.string),\n      'patient_id': tf.io.FixedLenFeature([], tf.int64),\n      'sex': tf.io.FixedLenFeature([], tf.int64),\n      'age_approx': tf.io.FixedLenFeature([], tf.int64),\n      'anatom_site_general_challenge': tf.io.FixedLenFeature ([], tf.int64),\n#       'diagnosis': tf.io.FixedLenFeature([], tf.int64),\n      'target': tf.io.FixedLenFeature([], tf.int64)\n#        'target': tf.io.FixedLenFeature([], tf.int64) \n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['target'], tf.int32)\n    data = {}\n    dumpp = {}\n    data['age_approx'] = tf.cast(example['age_approx'], tf.int32)\n    data['sex'] = tf.cast(example['sex'], tf.int32)\n    data['image_name'] = tf.cast(example['image_name'], tf.string)\n    if ANATOM :\n        dumpp['anatom_site_general_challenge'] = tf.cast(tf.one_hot(example['anatom_site_general_challenge'], 7), tf.int32)\n        for i in range(7) : \n            data[f'anatom_site_general_challenge{i}'] = dumpp['anatom_site_general_challenge'][i]\n\n#         data['anatom_site_general_challenge'] = tf.cast(example['anatom_site_general_challenge'], tf.int64)\n#     label_s = tf.cast(example['target'], tf.string)\n\n    return image, label ,data , label # returns a dataset of (image, label) pairs\n\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n      'image':  tf.io.FixedLenFeature([], tf.string),\n      'image_name':tf.io.FixedLenFeature([], tf.string),\n      'patient_id': tf.io.FixedLenFeature([], tf.int64),\n      'sex': tf.io.FixedLenFeature([], tf.int64),\n      'age_approx': tf.io.FixedLenFeature([], tf.int64),\n      'anatom_site_general_challenge': tf.io.FixedLenFeature ([], tf.int64),\n#       'diagnosis': tf.io.FixedLenFeature([], tf.int64),\n#       'target': tf.io.FixedLenFeature([], tf.int64)\n  }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n#     label = tf.cast(example['target'], tf.int32)\n    data = {}\n    dumpp = {}\n    data['age_approx'] = tf.cast(example['age_approx'], tf.int32)\n    data['sex'] = tf.cast(example['sex'], tf.int32)\n    data['image_name'] = tf.cast(example['image_name'], tf.string)\n    if ANATOM :\n        dumpp['anatom_site_general_challenge'] = tf.cast(tf.one_hot(example['anatom_site_general_challenge'], 7), tf.int32)\n        for i in range(7) : \n            data[f'anatom_site_general_challenge{i}'] = dumpp['anatom_site_general_challenge'][i]\n#         data['0'] , data['1'] , data['2'] , data['3'] , data['4'] , data['5'] , data['6'] ,data['6'] , data[''] = tf.reshape(data, [-1, 9])  \n#         data['anatom_site_general_challenge'] = tf.cast(example['anatom_site_general_challenge'], tf.int32)\n\n#     data['anatom_site_general_challenge'] = tf.cast(tf.one_hot(example['anatom_site_general_challenge'], 7),\n#                                                     tf.int32)\n#     data['diagnosis'] = tf.cast(example['sex'], tf.int32)\n    \n    return image , data # returns a dataset of (image, label) pairs\n\n\n\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files \n    # at once and disregarding data order. Order does not matter since we will \n    # be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        # disable order, increase speed\n        ignore_order.experimental_deterministic = False\n\n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order)\n    # returns a dataset of (image, label) pairs if labeled=True \n    # or (image, id) pairs if labeled=False\n    dataset = dataset.map(read_labeled_tfrecord if labeled \n                          else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    \n    return dataset\n\n# def get_test_dataset(ordered=False):\n#     dataset = load_dataset(TEST_FNAMES, labeled=False, ordered=ordered)\n#     dataset = dataset.batch(BATCH_SIZE)\n#     # prefetch next batch while training (autotune prefetch buffer size)\n#     dataset = dataset.prefetch(AUTO)\n    \n#     return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ===============================================","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"And here are some quick examples of the test data:","execution_count":null},{"metadata":{"_kg_hide-input":true,"id":"s19U5UZhugnD","outputId":"19415e06-2afc-4ecb-8543-2c5ccb51ed5f","trusted":true},"cell_type":"code","source":"# %%time\n# TEST_FNAMES = TEST_FILENAMES\n# test_dataset = get_test_dataset()\n\n# print(\"Examples of the test data:\")\n# for image, data in test_dataset.take(1):\n#     print(\"The image batch size:\", image.numpy().shape)\n#     print(\"Ages, 5 examples:\", data['age_approx'].numpy()[:5])\n# #     print(\"Age (scaled), 5 examples:\", data['age_scaled'].numpy()[:5])\n# #     print(\"Height, 5 examples:\", data['height'].numpy()[:5])\n# #     print(\"Width, 5 examples:\", data['width'].numpy()[:5])","execution_count":null,"outputs":[]},{"metadata":{"id":"bXM39S6zugnG"},"cell_type":"markdown","source":"### Visualization utilities","execution_count":null},{"metadata":{"_kg_hide-input":true,"id":"E656VW7cugnH","trusted":true},"cell_type":"code","source":"# # numpy and matplotlib defaults\n# np.set_printoptions(threshold=15, linewidth=80)\n\n# def batch_to_numpy_images_and_labels(databatch):\n#     if len(databatch)==4:\n#         images, labels, _, _ = databatch\n#         numpy_images = images.numpy()\n#         numpy_labels = labels.numpy()\n#     else:\n#         images, _ = databatch\n#         numpy_images = images.numpy()\n#         numpy_labels = [None for _ in enumerate(numpy_images)]\n\n#     # If no labels, only image IDs, return None for labels (this is the case for test data)\n#     return numpy_images, numpy_labels\n\n# def title_from_label_and_target(label, correct_label):\n#     if correct_label is None:\n#         return CLASSES[label], True\n#     correct = (label == correct_label)\n#     return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" \n#                                 if not correct else '', \n#                                 CLASSES[correct_label] if not correct else ''), correct\n\n# def display_one_image(image, title, subplot, red=False, titlesize=16):\n#     plt.subplot(*subplot)\n#     plt.axis('off')\n#     plt.imshow(image)\n# #     try : \n# #         if len(title) > 0:\n# #             plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), \n# #                       color='red' if red else 'black', fontdict={'verticalalignment':'center'}, \n# #                       pad=int(titlesize/1.5)\n# #                      )\n# #     except :\n# #         plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), \n# #                   color='red' if red else 'black', fontdict={'verticalalignment':'center'}, \n# #                   pad=int(titlesize/1.5)\n# #                  )\n#     return (subplot[0], subplot[1], subplot[2]+1)\n\n# def display_batch_of_images(databatch, predictions=None):\n#     \"\"\"This will work with:\n#     display_batch_of_images(images)\n#     display_batch_of_images(images, predictions)\n#     display_batch_of_images((images, labels))\n#     display_batch_of_images((images, labels), predictions)\n#     \"\"\"\n#     # data\n#     images, labels = batch_to_numpy_images_and_labels(databatch)\n#     if labels is None:\n#         labels = [None for _ in enumerate(images)]\n        \n#     # auto-squaring: this will drop data that does  \n#     # not fit into square or square-ish rectangle\n#     rows = int(math.sqrt(len(images)))\n#     cols = len(images)//rows\n        \n#     # size and spacing\n#     FIGSIZE = 4.0\n#     SPACING = 0.1\n#     subplot=(rows,cols,1)\n#     if rows < cols:\n#         plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n#     else:\n#         plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n#     # display\n#     for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n#         title = '' if label is None else CLASSES[label]\n#         correct = True\n#         if predictions is not None:\n#             title, correct = title_from_label_and_target(predictions[i], label)\n#         # magic formula tested to work from 1x1 to 10x10 images\n#         dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3\n#         subplot = display_one_image(image, title, subplot, \n#                                      not correct, titlesize=dynamic_titlesize)\n    \n#     #layout\n#     plt.tight_layout()\n#     if label is None and predictions is None:\n#         plt.subplots_adjust(wspace=0, hspace=0)\n#     else:\n#         plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n#     plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def dropout(image, DIM=256, PROBABILITY = 0.75, CT = 8, SZ = 0.2):\n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image with CT squares of side size SZ*DIM removed\n    \n    # DO DROPOUT WITH PROBABILITY DEFINED ABOVE\n    P = tf.cast( tf.random.uniform([],0,1)<PROBABILITY, tf.int32)\n    if (P==0)|(CT==0)|(SZ==0): return image\n    \n    for k in range(CT):\n        # CHOOSE RANDOM LOCATION\n        x = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        y = tf.cast( tf.random.uniform([],0,DIM),tf.int32)\n        # COMPUTE SQUARE \n        WIDTH = tf.cast( SZ*DIM,tf.int32) * P\n        ya = tf.math.maximum(0,y-WIDTH//2)\n        yb = tf.math.minimum(DIM,y+WIDTH//2)\n        xa = tf.math.maximum(0,x-WIDTH//2)\n        xb = tf.math.minimum(DIM,x+WIDTH//2)\n        # DROPOUT IMAGE\n        one = image[ya:yb,0:xa,:]\n        two = tf.zeros([yb-ya,xb-xa,3]) \n        three = image[ya:yb,xb:DIM,:]\n        middle = tf.concat([one,two,three],axis=1)\n        image = tf.concat([image[0:ya,:,:],middle,image[yb:DIM,:,:]],axis=0)\n            \n    # RESHAPE HACK SO TPU COMPILER KNOWS SHAPE OF OUTPUT TENSOR \n    image = tf.reshape(image,[DIM,DIM,3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ROT_ = 180.0\nSHR_ = 2.0\nHZOOM_ = 8.0\nWZOOM_ = 8.0\nHSHIFT_ = 8.0\nWSHIFT_ = 8.0\nimport tensorflow.keras.backend as K\ndef get_mat(rotation, shear, height_zoom, width_zoom, height_shift, width_shift):\n    # returns 3x3 transformmatrix which transforms indicies\n        \n    # CONVERT DEGREES TO RADIANS\n    rotation = math.pi * rotation / 180.\n    shear    = math.pi * shear    / 180.\n\n    def get_3x3_mat(lst):\n        return tf.reshape(tf.concat([lst],axis=0), [3,3])\n    \n    # ROTATION MATRIX\n    c1   = tf.math.cos(rotation)\n    s1   = tf.math.sin(rotation)\n    one  = tf.constant([1],dtype='float32')\n    zero = tf.constant([0],dtype='float32')\n    \n    rotation_matrix = get_3x3_mat([c1,   s1,   zero, \n                                   -s1,  c1,   zero, \n                                   zero, zero, one])    \n    # SHEAR MATRIX\n    c2 = tf.math.cos(shear)\n    s2 = tf.math.sin(shear)    \n    \n    shear_matrix = get_3x3_mat([one,  s2,   zero, \n                                zero, c2,   zero, \n                                zero, zero, one])        \n    # ZOOM MATRIX\n    zoom_matrix = get_3x3_mat([one/height_zoom, zero,           zero, \n                               zero,            one/width_zoom, zero, \n                               zero,            zero,           one])    \n    # SHIFT MATRIX\n    shift_matrix = get_3x3_mat([one,  zero, height_shift, \n                                zero, one,  width_shift, \n                                zero, zero, one])\n    \n    return K.dot(K.dot(rotation_matrix, shear_matrix), \n                 K.dot(zoom_matrix,     shift_matrix))\n\n\ndef transform(image, DIM=256):    \n    # input image - is one image of size [dim,dim,3] not a batch of [b,dim,dim,3]\n    # output - image randomly rotated, sheared, zoomed, and shifted\n    XDIM = DIM%2 #fix for size 331\n    \n    rot = ROT_ * tf.random.normal([1], dtype='float32')\n    shr = SHR_ * tf.random.normal([1], dtype='float32') \n    h_zoom = 1.0 + tf.random.normal([1], dtype='float32') / HZOOM_\n    w_zoom = 1.0 + tf.random.normal([1], dtype='float32') / WZOOM_\n    h_shift = HSHIFT_ * tf.random.normal([1], dtype='float32') \n    w_shift = WSHIFT_ * tf.random.normal([1], dtype='float32') \n\n    # GET TRANSFORMATION MATRIX\n    m = get_mat(rot,shr,h_zoom,w_zoom,h_shift,w_shift) \n\n    # LIST DESTINATION PIXEL INDICES\n    x   = tf.repeat(tf.range(DIM//2, -DIM//2,-1), DIM)\n    y   = tf.tile(tf.range(-DIM//2, DIM//2), [DIM])\n    z   = tf.ones([DIM*DIM], dtype='int32')\n    idx = tf.stack( [x,y,z] )\n    \n    # ROTATE DESTINATION PIXELS ONTO ORIGIN PIXELS\n    idx2 = K.dot(m, tf.cast(idx, dtype='float32'))\n    idx2 = K.cast(idx2, dtype='int32')\n    idx2 = K.clip(idx2, -DIM//2+XDIM+1, DIM//2)\n    \n    # FIND ORIGIN PIXEL VALUES           \n    idx3 = tf.stack([DIM//2-idx2[0,], DIM//2-1+idx2[1,]])\n    d    = tf.gather_nd(image, tf.transpose(idx3))\n        \n    return tf.reshape(d,[DIM, DIM,3])\n\n\n\ndef data_augment(data, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement \n    # in the next function (below), this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    img = data['inp1']\n    img = transform(img,DIM=dim)\n    if DROPOUT :\n        img = dropout(img, DIM=dim, PROBABILITY=droprate, CT=dropct, SZ=dropsize)\n    img = tf.image.random_flip_left_right(img)\n    #img = tf.image.random_hue(img, 0.01)\n    img = tf.image.random_saturation(img, 0.7, 1.3)\n    img = tf.image.random_contrast(img, 0.8, 1.2)\n    img = tf.image.random_brightness(img, 0.1)\n    img = tf.reshape(img, [dim,dim, 3])\n    data['inp1'] = img\n    \n#     data['inp1'] = tf.image.random_flip_left_right(data['inp1'])\n#     data['inp1'] = tf.image.random_flip_up_down(data['inp1'])\n    #image = tf.image.random_saturation(image, 0, 2)\n    \n    return data, label\n\ndef data_augment_test(data , label = 0):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement \n    # in the next function (below), this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    img = data['inp1']\n    img = transform(img,DIM=dim)\n    if DROPOUT :\n        img = dropout(img, DIM=dim, PROBABILITY=droprate, CT=dropct, SZ=dropsize)\n    img = tf.image.random_flip_left_right(img)\n    #img = tf.image.random_hue(img, 0.01)\n    img = tf.image.random_saturation(img, 0.7, 1.3)\n    img = tf.image.random_contrast(img, 0.8, 1.2)\n    img = tf.image.random_brightness(img, 0.1)\n    img = tf.reshape(img, [dim,dim, 3])\n    data['inp1'] = img\n    \n#     data['inp1'] = tf.image.random_flip_left_right(data['inp1'])\n#     data['inp1'] = tf.image.random_flip_up_down(data['inp1'])\n    #image = tf.image.random_saturation(image, 0, 2)\n    \n    return data , 0","execution_count":null,"outputs":[]},{"metadata":{"id":"NLTxUNInugnW"},"cell_type":"markdown","source":"### Dataset visualizations","execution_count":null},{"metadata":{"id":"kkHviwL7ugnm","trusted":true},"cell_type":"code","source":"# # Peek at test data\n# test_dataset = test_dataset.unbatch().batch(20)\n# test_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"id":"qGqKfLhlugno","outputId":"dd202ce2-f546-4052-80e9-21c81b86be03","trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\n# display_batch_of_images(next(test_batch))","execution_count":null,"outputs":[]},{"metadata":{"id":"RSmIate_ugnr","outputId":"797ccf18-b1f9-4c62-a768-f3ba4aac0570","trusted":true},"cell_type":"code","source":"# del test_dataset, test_batch\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"id":"1nmPLSlmugnu"},"cell_type":"markdown","source":"# 4. MODEL setup\n","execution_count":null},{"metadata":{"id":"DHBNxb07ugnu","outputId":"0b689531-e141-4ad5-8a3c-205720db037a","trusted":true},"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.000005#0.00001\nLR_MAX = 0.00000725 * strategy.num_replicas_in_sync\nLR_MIN = 0.000005\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 4\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"id":"OaaJgH9Vugnw","outputId":"f76aa200-572f-4a41-d91b-a94b5133fa2b","trusted":true},"cell_type":"code","source":"train['target'].values\n\nclass_weights = class_weight.compute_class_weight(class_weight='balanced',\n                                                  classes=np.unique(train['target'].values),\n                                                  y=train['target'].values,\n                                                 )\n\nclass_weights = {i : class_weights[i] for i in range(len(class_weights))}\n\nprint(class_weights)","execution_count":null,"outputs":[]},{"metadata":{"id":"cAWBwBEUTTuk","outputId":"d7d9d4f1-f899-44e6-be18-8791f820cf01","trusted":true},"cell_type":"code","source":"if ANATOM :\n    tab_feats=[\n          'sex',\n          'age_approx',\n        'anatom_site_general_challenge0',\n        'anatom_site_general_challenge1',\n        'anatom_site_general_challenge2',\n        'anatom_site_general_challenge3',\n        'anatom_site_general_challenge4',\n        'anatom_site_general_challenge5',\n        'anatom_site_general_challenge6'\n    ]\nelse :\n    tab_feats=[\n          'sex',\n          'age_approx',\n    ]\n\nN_TAB_FEATS=len(tab_feats) \n\nprint(f\"The number of tabular features is {N_TAB_FEATS}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"EuHZF_t5ugnz","outputId":"73711688-be72-466c-8054-9fef0128887c","trusted":true},"cell_type":"code","source":"%time\n\nEFNS = [efn.EfficientNetB0, efn.EfficientNetB1, efn.EfficientNetB2, efn.EfficientNetB3, \n        efn.EfficientNetB4, efn.EfficientNetB5, efn.EfficientNetB6]\n\ndef get_model():\n    with strategy.scope():\n        pretrained_model = EFNS[ef](input_shape=(*IMAGE_SIZE, 3),\n                                              weights='imagenet',\n                                              include_top=False\n                                             )\n        # False = transfer learning, True = fine-tuning\n        pretrained_model.trainable = True#False \n\n        inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE, 3), name='inp1')\n        inp2 = tf.keras.layers.Input(shape=(N_TAB_FEATS), name='inp2')\n        \n        # BUILD MODEL HERE\n        \n        x=pretrained_model(inp1)\n        x=tf.keras.layers.GlobalAveragePooling2D()(x)\n        x=tf.keras.layers.Dense(512, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(256, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(128, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        \n        y=tf.keras.layers.Dense(100, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(inp2)\n        \n        concat=tf.keras.layers.concatenate([y, x])\n        \n        output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(concat)\n        \n        model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=[output])\n    \n        model.compile(\n        optimizer='adam',\n        loss = 'binary_crossentropy',\n        metrics=[tf.keras.metrics.AUC()],\n        )\n        \n        return model","execution_count":null,"outputs":[]},{"metadata":{"id":"G8QfbjTtugn1","trusted":true},"cell_type":"code","source":"# %%time\n\n# model=get_model()\n\n# model.summary()\n\n# del model\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"id":"CSBEsfb4s3bW","trusted":true},"cell_type":"code","source":"if colab:\n    \n    SAVE_FOLDER=NAME\n    \n    if tstamp:\n        time_zone = pytz.timezone('America/Chicago')\n        current_datetime = datetime.now(time_zone)\n        ts=current_datetime.strftime(\"%m%d%H%M%S\")\n        SAVE_FOLDER+='_'+ts\n        \n    SAVE_FOLDER=PATH/SAVE_FOLDER\n    if not os.path.exists(SAVE_FOLDER):\n        os.mkdir(SAVE_FOLDER)\n\nelse:\n    SAVE_FOLDER=Path('/kaggle/working')","execution_count":null,"outputs":[]},{"metadata":{"id":"cr2t2NC3jr_G","trusted":true},"cell_type":"code","source":"class save_best_n(tf.keras.callbacks.Callback):\n    def __init__(self, fn, model):\n        self.fn = fn\n        self.model = model\n\n    def on_epoch_end(self, epoch, logs=None):\n        \n        if (epoch>0):\n            score=logs.get(\"val_auc\")\n        else:\n            score=-1\n      \n        if (score > best_score[fold_num].min()):\n          \n            idx_min=np.argmin(best_score[fold_num])\n\n            best_score[fold_num][idx_min]=score\n            best_epoch[fold_num][idx_min]=epoch+1\n\n            path_best_model=f'best_model_fold_{self.fn}_{idx_min}.hdf5'\n            self.model.save(SAVE_FOLDER/path_best_model)\n            ############# WARNING: ##################################\n            # Make sure you have enough space to store your models. \n            # Remember that Kaggle allows you save not more than 5 Gb\n            # to disk. It should not be a problem for EfficientNet B0 \n            # or B3 but it is not going to work for B7. I am saving my\n            # models to Google Drive where I have plenty of space.","execution_count":null,"outputs":[]},{"metadata":{"id":"Q5k46VXBugn3"},"cell_type":"markdown","source":"# 5. Model Train\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def setup_input(image, label, data, label_name):\n    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    \n    tabular=tf.stack(tab_data)\n    \n    return {'inp1': image, 'inp2':  tabular}, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augment","execution_count":null},{"metadata":{"id":"tzQ9BfxiZCGo","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"id":"tXww0kHNugn7","trusted":true},"cell_type":"code","source":"def get_training_dataset(dataset):\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    # the training dataset must repeat for several epochs\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    #dataset = dataset.repeat()\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"id":"lRfOGnqCugn9","trusted":true},"cell_type":"code","source":"def get_validation_dataset(dataset):\n#     return get_training_dataset(dataset)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model = get_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = load_dataset(TRAIN_FNAMES['fold_1'])\ntrain_dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ************************\n## 5.1 Fitting model\n## ************************","execution_count":null},{"metadata":{"_kg_hide-input":false,"id":"7fbGd3r4ugn_","outputId":"955d46f0-71b7-403f-c3db-5ec161f30242","trusted":true},"cell_type":"code","source":"debug=0\n# EPOCHS = 10\nhistories = []\n\nbest_epoch={fn: np.zeros(NBEST) for fn in range(1, NFOLDS+1)}\nbest_score={fn: np.zeros(NBEST) for fn in range(1, NFOLDS+1)}\n\nfor fold_num in range(1, NFOLDS+1):\n# for fold_num in range(1, 2):    \n    tf.keras.backend.clear_session()\n    # clear tpu memory (otherwise can run into Resource Exhausted Error)\n    # see https://www.kaggle.com/c/flower-classification-with-tpus/discussion/131045\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    print(\"=\"*50)\n    print(f\"Starting fold {fold_num} out of {NFOLDS}...\")\n    \n    files_trn=TRAIN_FNAMES[f\"fold_{fold_num}\"]\n    files_val=VAL_FNAMES[f\"fold_{fold_num}\"]\n    \n    if debug:\n#         TTA = 2\n        NUMS = 2\n        files_trn=files_trn[0:2]\n        files_val=files_val[0:2]\n        EPOCHS=2\n       \n    train_dataset = load_dataset(files_trn)\n    train_dataset = train_dataset.map(setup_input, num_parallel_calls=AUTO)\n    \n    val_dataset = load_dataset(files_val, ordered = True)\n    val_dataset = val_dataset.map(setup_input, num_parallel_calls=AUTO)\n    \n    model = get_model()\n    \n    STEPS_PER_EPOCH = count_data_items(files_trn) // BATCH_SIZE\n#     STEPS_PER_EPOCH *= 1.5 \n    \n    print(f'STEPS_PER_EPOCH = {STEPS_PER_EPOCH}')\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n    \n    history = model.fit(get_training_dataset(train_dataset), \n                        steps_per_epoch=STEPS_PER_EPOCH, \n                        epochs=EPOCHS, \n                        callbacks=[lr_callback,\n                                   save_best_n(fold_num, model),\n                                   ],\n                        validation_data=get_validation_dataset(val_dataset),\n                        class_weight=class_weights,\n                        verbose=2,\n                       )\n    \n    idx_sorted=np.argsort(best_score[fold_num])\n    best_score[fold_num]=np.array(best_score[fold_num])[idx_sorted]\n    best_epoch[fold_num]=np.array(best_epoch[fold_num])[idx_sorted]\n\n    print(f\"\\nFold {fold_num} is finished. The best epochs: {[int(best_epoch[fold_num][i]) for i in range(len(best_epoch[fold_num]))]}\")\n    print(f\"The corresponding scores: {[round(best_score[fold_num][i], 5) for i in range(len(best_epoch[fold_num]))]}\")\n\n    histories.append(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2 Visualization of training progress","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train.to_csv('train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history.history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.save(\"2inp.h5\")\n","execution_count":null,"outputs":[]},{"metadata":{"id":"hEmYCG7vugoA","trusted":true},"cell_type":"code","source":"def display_training_curves(fold_num, data):\n\n    plt.figure(figsize=(10,5), facecolor='#F0F0F0')\n\n    epochs=np.arange(1, EPOCHS+1)\n\n    # AUC\n    plt.plot(epochs, data['auc'], label='training auc', color='red')\n    plt.plot(epochs, data['val_auc'], label='validation auc', color='orange')\n\n    # Loss\n    plt.plot(epochs, data['loss'], label='training loss', color='blue')    \n    plt.plot(epochs, data['val_loss'], label='validation loss', color='green')\n\n    # Best\n    ls=['dotted', 'dashed', 'dashdot', 'solid'] # don't use more than 4 best epochs \n                                                # or make proper adjustments!\n    for i in range(NBEST):\n        plt.axvline(best_epoch[fold_num][i], 0, \n                    best_score[fold_num][i], linestyle=ls[i], \n                    color='black', label=f'AUC {best_score[fold_num][i]:.5f}')\n    \n    plt.title(f\"Fold {fold_num}. The best epochs: {[int(best_epoch[fold_num][i]) for i in range(len(best_epoch[fold_num]))]}; the best AUC's: {[round(best_score[fold_num][i], 5) for i in range(len(best_epoch[fold_num]))]}.\", \n              fontsize='14')\n    plt.ylabel('Loss/AUC', fontsize='12')\n    plt.xlabel('Epoch', fontsize='12')\n    plt.ylim((0, 1))\n    plt.legend(loc='lower left')\n    plt.tight_layout()\n    plt.show()\n\n# for fn in range(1, NFOLDS+1):\n#     display_training_curves(fn, data=histories[fn-1].history)\n\n### Predictions\n\ndef setup_test_image(image, data):    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    tabular=tf.stack(tab_data)\n\n    return {'inp1': image, 'inp2': tabular}\n\ndef setup_test_name(image, data):\n    return data['image_name']\n\ndef get_test_dataset(dataset):\n    dataset = dataset.map(data_augment_test, num_parallel_calls=AUTO)\n\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset\n    \ndef get_test_ds_name(dataset) :\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset\n\ndef average_predictions(X, fn):\n    \n    y_probas=[]\n    \n    for idx in range(NBEST):\n        \n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        strategy = tf.distribute.experimental.TPUStrategy(tpu)\n#         gc.collect()\n\n        print(f\"Predicting: fold {fn}, model {idx+1} out of {NBEST}...\")\n\n        with strategy.scope():\n            path_best_model=f'best_model_fold_{fn}_{idx}.hdf5'\n            model=tf.keras.models.load_model(SAVE_FOLDER/path_best_model)\n\n        y=model.predict(X)\n        y = rankdata(y)/len(y)\n        y_probas.append(y)\n    \n    y_probas=np.average(y_probas, axis=0)\n\n    return y_probas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. PREDICTIONS","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Select TTA here","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lsti = []\nTEST_FNAMES = TEST_FILENAMES\nfor i in range(TTA):\n    lstj = []\n    N_TEST_IMGS = count_data_items(TEST_FNAMES)\n    # N_TEST_IMGS = count_data_items(TEST_FNAMES)\n    preds = pd.DataFrame({'image_name': np.zeros(len(test)), 'target': np.zeros(len(test))})\n\n    test_ds = load_dataset(TEST_FNAMES, labeled=False, ordered=True)\n    test_images_ds = test_ds.map(setup_test_image, num_parallel_calls=AUTO)\n\n    test_images_ds = get_test_dataset(test_images_ds)\n    test_ds = get_test_ds_name(test_ds)\n\n    test_ids_ds = test_ds.map(setup_test_name, num_parallel_calls=AUTO).unbatch()\n    sub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\n\n    preds['image_name'] = next(iter(test_ids_ds.batch(N_TEST_IMGS))).numpy().astype('U')\n#     lstj = np.average([average_predictions(test_images_ds, fn) for fn in range(1, 3)], axis = 0)\n\n    lstj = np.average([average_predictions(test_images_ds, fn) for fn in range(1, NFOLDS + 1)], axis = 0)\n    lsti.append(lstj)\n    \n    del test_images_ds , test_ds , test_ids_ds\n    gc.collect()\n    \nlsti\n# y=model.predict(test_images_ds)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a = np.array(lsti)\na","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds['target'] = a.mean(axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds","execution_count":null,"outputs":[]},{"metadata":{"id":"bEKgTC8VugoL","trusted":true},"cell_type":"code","source":"%%time\n# preds = []\n# for i in range(TTA):\n    \n\ndel sub['target']\nsub = sub.merge(preds, on='image_name')\nsub.head()\n\nsub.to_csv('effB1_512_5x15epochs_3Stratif.csv', index=False)\n\n# preds = pd.DataFrame({'image_name': np.zeros(len(test)), 'target': np.zeros(len(test))})\n\n# test_ds = load_dataset(TEST_FNAMES, labeled=False, ordered=True)\n# test_images_ds = test_ds.map(setup_test_image, num_parallel_calls=AUTO)\n\n# test_images_ds = get_test_dataset(test_images_ds)\n# test_ds = get_test_dataset(test_ds)\n\n# test_ids_ds = test_ds.map(setup_test_name, num_parallel_calls=AUTO).unbatch()\n# test_images_ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}