{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook, I will be levaraging the power of TPU to train multiple models and then ensemling the scores from all the invidual models. I will be trying to explain each step along the way. The public score of this ensemble is 0.9071.\n\nReferences\n\nhttps://www.kaggle.com/khoongweihao/siim-isic-multiple-model-training-stacking\nhttps://www.kaggle.com/niteshx2/melanoma-beginner-tpu-efficientnet\nhttps://www.kaggle.com/ragnar123/rank-then-blend","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Installing EfficienntNet to be used Later","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"# Installing tensorflow 2 to fix the TPU error 'Socket closed'\n!pip install tensorflow==2.2-rc1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Importing Required Libraries","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\n\nimport numpy as np\nimport pandas as pd\nimport math\n\nfrom matplotlib import pyplot as plt\n\nfrom sklearn import metrics\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n\nimport efficientnet.tfkeras as efn\n\nfrom kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TPU Configuration","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"We will be using TPU for training the models. TPU which stand for Tensor Processing Units are best suited to train neural networks much faster than GPUs.\n\nBelow is the boilerplate code to detect the TPU on network and defining the TPU strategy","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing for TPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# For tf.dataset\nAUTO = tf.data.experimental.AUTOTUNE\n\n# GCS Data access path for this dataset\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\n\n# Configuration\nEPOCHS = 5\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\nIMAGE_SIZE = [1024, 1024]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading the datasets\n\nsub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')\ntrain = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Reading filenames for TPU\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/train*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/tfrecords/test*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(TRAINING_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are total 16 training filenames for TPU. Each file name consists of mutiple files as we know that there are total 33126 training images.\n\nLets split the training file names into training and validation sets in a ratio of 80-20.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"VALIDATION_FILENAMES = TRAINING_FILENAMES[int(0.8*len(TRAINING_FILENAMES)):]\nTRAINING_FILENAMES = TRAINING_FILENAMES[:int(0.8*len(TRAINING_FILENAMES))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of training file names : {} and number of validation file names :{}\".format(len(TRAINING_FILENAMES),len(VALIDATION_FILENAMES)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" **Functions for TPU Read**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to normalize and reshape the images\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to read labeled training records\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        #\"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n        \"target\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    #label = tf.cast(example['class'], tf.int32)\n    label = tf.cast(example['target'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to read unlabeled test records\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['image_name']\n    return image, idnum # returns a dataset of image(s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to load dataset\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to apply augumentation on training images\n\ndef data_augment(image, label):\n    \n    image = tf.image.random_flip_left_right(image)\n    return image, label  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to get Training dataset\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to get Validation dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to get test dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As discussed earlier, the number of data items is written in the name of .tfrec files, i.e. flowers00-230.tfrec = 230 data items. Below function sums all the image names present in the tfrec files","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Getting the total numnber of training, validation and test images.\n# Defining the number of steps in each epoch\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\n\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images {} unlabeled test images'.format(\n    NUM_TRAINING_IMAGES,NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will be using a function to use variable learning rate. Learning rate will be reduced as the model goes through the training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.00001, lr_max=0.000075, \n               lr_min=0.000001, lr_rampup_epochs=20, \n               lr_sustain_epochs=0, lr_exp_decay=.8):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) * lr_exp_decay**(epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n        return lr\n    \n    return lrfn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Defining the Models","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Model 1\n\nwith strategy.scope():\n        model1 = tf.keras.Sequential([\n            efn.EfficientNetB7(\n                input_shape=(*IMAGE_SIZE, 3),\n                weights='imagenet',\n                include_top=False\n            ),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n        model1.compile(\n            optimizer='adam',\n            loss = 'binary_crossentropy',\n            metrics=[tf.keras.metrics.AUC()]\n        )\n        model1.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Model 2\n\nwith strategy.scope():\n        model2 = tf.keras.Sequential([\n            efn.EfficientNetB0(\n                input_shape=(*IMAGE_SIZE, 3),\n                weights='imagenet',\n                include_top=False\n            ),\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n        model2.compile(\n            optimizer='adam',\n            loss = 'binary_crossentropy',\n            metrics=[tf.keras.metrics.AUC()]\n        )\n        model2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.applications import DenseNet201\n\n# Model 3\n\nwith strategy.scope():\n        dnet201 = DenseNet201(\n            input_shape=(*IMAGE_SIZE, 3),\n            weights='imagenet',\n            include_top=False\n        )\n        dnet201.trainable = True\n\n        model3 = tf.keras.Sequential([\n            dnet201,\n            L.GlobalAveragePooling2D(),\n            L.Dense(1, activation='sigmoid')\n        ])\n        model3.compile(\n            optimizer='adam',\n            loss = 'binary_crossentropy',\n            metrics=[tf.keras.metrics.AUC()]\n        )\n        model3.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"lrfn = build_lrfn()\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = get_training_dataset()\nvalid_dataset = get_validation_dataset()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history1 = model1.fit(\n        train_dataset,\n        epochs=EPOCHS,\n        callbacks=[lr_schedule],\n        steps_per_epoch=STEPS_PER_EPOCH,\n        validation_data=valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create copies for each model if you want to\nsub1 = sub.copy()\n\n# Getting predictions on test data\ntest_ds = get_test_dataset(ordered=True)\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model1.predict(test_images_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generating submission file\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n\npred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del sub1['target']\nsub1 = sub1.merge(pred_df, on='image_name')\nsub1.to_csv('submission_efficientnetb7.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history2 = model2.fit(\n        train_dataset,\n        epochs=EPOCHS,\n        callbacks=[lr_schedule],\n        steps_per_epoch=STEPS_PER_EPOCH,\n        validation_data=valid_dataset)\n\nmodel1.save(\"efficientnetb0.h5\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create copies for each model if you want to\nsub2 = sub.copy()\n\n# Getting predictions on test data\n#test_ds = get_test_dataset(ordered=True)\nprint('Computing predictions...')\n#test_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model2.predict(test_images_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generating submission file\n\nprint('Generating submission.csv file...')\n#test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n#test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n\npred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del sub2['target']\nsub2 = sub2.merge(pred_df, on='image_name')\nsub2.to_csv('submission_efficientnetb0.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history3 = model3.fit(\n        train_dataset,\n        epochs=EPOCHS,\n        callbacks=[lr_schedule],\n        steps_per_epoch=STEPS_PER_EPOCH,\n        validation_data=valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create copies for each model if you want to\nsub3 = sub.copy()\n\n# Getting predictions on test data\n#test_ds = get_test_dataset(ordered=True)\nprint('Computing predictions...')\n#test_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model3.predict(test_images_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generating submission file\n\nprint('Generating submission.csv file...')\n#test_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\n#test_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\n\npred_df = pd.DataFrame({'image_name': test_ids, 'target': np.concatenate(probabilities)})\npred_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del sub3['target']\nsub3 = sub3.merge(pred_df, on='image_name')\nsub3.to_csv('submission_Densenet201.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble with Blending and Submission","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here I will ensemble the predictions from all 3 models, plus one on mine earlier submission using VGG16 (notebook link: https://www.kaggle.com/sawans/baseline-with-vgg16) to get the final output. More weightage will be given to the model having higher public score. \n\nSince all the preditions have different distributions, its not recommebed to ensemble them using simple weighted average. Instead we should rank each individual prediction first, and them blend them using weighted average.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_efficientnetB7 = pd.read_csv('../input/second-ensemble/submission_efficientnetb7.csv')\nsub_efficientnetB0 = pd.read_csv('../input/second-ensemble/submission_efficientnetb0.csv')\nsub_densenet201 = pd.read_csv('../input/second-ensemble/submission_Densenet201.csv')\nsub_vgg16 = pd.read_csv('../input/second-ensemble/submission_vgg16_Complete.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will rank each prediction and then divide by its maximum value so that we have predictions between 0 and 1","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def rank_data(sub):\n    sub['target'] = sub['target'].rank() / sub['target'].rank().max()\n    return sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_efficientnetB7 = rank_data(sub_efficientnetB7)\nsub_efficientnetB0 = rank_data(sub_efficientnetB0)\nsub_densenet201 = rank_data(sub_densenet201)\nsub_vgg16 = rank_data(sub_vgg16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_efficientnetB7.columns = ['image_name', 'target1']\nsub_efficientnetB0.columns = ['image_name', 'target2']\nsub_densenet201.columns = ['image_name', 'target3']\nsub_vgg16.columns = ['image_name', 'target4']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f_sub = sub_efficientnetB7.merge(sub_efficientnetB0, on = 'image_name').merge(sub_densenet201, on = 'image_name').merge(sub_vgg16, on = 'image_name')\n\nf_sub['target'] = f_sub['target1'] * 0.4 + f_sub['target2'] * 0.4 + f_sub['target3'] * 0.02 + f_sub['target4'] * 0.02","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f_sub = f_sub[['image_name', 'target']]\nf_sub.to_csv('blend_sub_2.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}