{"cells":[{"metadata":{},"cell_type":"markdown","source":"## The purpose of this notebook\n\nThe motivation and the main idea of this notebook is outlined in details in [discussion topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395).\n\n**Version 9**: model EfficientNet B0, the notebook scores 0.914 on the public LB. The corresponding CV score is around 0.903.\n\n**Version 10**: we experiment with the Shades of Gray algorithm using the same EfficientNet B0 model. See [this public kernel](https://www.kaggle.com/apacheco/shades-of-gray-color-constancy) and [this discussion topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161719) for more details. The result does not seem to be very encouraging (the CV drops to 0.897)\n\n**Version 13**: we use the original images (no Shades of Gray pre-processing) and upgrade our model to EfficientNet B1. The number of epochs is reduced from 20 to 17 to meet the 3 hour limit requirement.\n\n**Version 20**: we demonstrate how to implement the Advanced Hair Augmentation in TensorFlow using the same approach as before. The idea was suggested by [Roman](https://www.kaggle.com/nroman) in [this discussion topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/159176). The TensorFlow implementation is discussed by me [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/163909) and demonstrated in this [public notebook](https://www.kaggle.com/graf10a/siim-advanced-hair-augmentation-in-tensorflow) of mine. My prior experiments on Kaggle and Colab show that adding this augmentation increases the training time. To stay within the 3 hour limit we return to the EfficientNet B0 model and decrease the number of epochs to 10. So, don't expect a great score -- the purpose of the notebook is just to demonstate the implementation technique. I beleive that the training time can be optimized but this is the task we will save for the future. The images are pre-processed with the Shades of Gray algorithm (let's give it another chance -- it might surprise us!). See [this public kernel](https://www.kaggle.com/apacheco/shades-of-gray-color-constancy), [this discussion topic](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161719), and **Version 10** of this notebook for more information about the Shades of Gray algorithm.\n\n**Version 23**: fixed some minor typos. Added a section where we are blending our predictions with similar predictions generated by ealier versions of this kernel.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Preliminaries","execution_count":null},{"metadata":{"id":"aH1svRGNTBtD","trusted":true},"cell_type":"code","source":"colab=0\nshow_files=0\ntstamp=0","execution_count":null,"outputs":[]},{"metadata":{"id":"xdymLEOA0D1e","outputId":"4233734e-1ab6-41e4-9540-131f8918cbc0","trusted":true},"cell_type":"code","source":"if colab:\n    from google.colab import drive\n    drive.mount('/content/gdrive')","execution_count":null,"outputs":[]},{"metadata":{"id":"OZ7JpuAhugk9","trusted":true},"cell_type":"code","source":"if (not colab)&show_files:\n    import os\n    for dirname, _, filenames in os.walk('/kaggle/input'):\n        for filename in filenames:\n            print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output","execution_count":null,"outputs":[]},{"metadata":{"id":"YmXVe_bBuglC"},"cell_type":"markdown","source":"## Loading libraries","execution_count":null},{"metadata":{"id":"8T0OVkF6uglD","trusted":true},"cell_type":"code","source":"!pip install -q efficientnet","execution_count":null,"outputs":[]},{"metadata":{"id":"wkB3rarVuglH","outputId":"903039f5-bb33-4c1e-ee38-7755fa3fd8c2","trusted":true},"cell_type":"code","source":"import math\nimport pytz\nimport random\nimport numpy as np\nimport pandas as pd\nimport math, re, os, gc\nimport tensorflow as tf\nfrom pathlib import Path\nfrom datetime import datetime\nfrom scipy.stats import rankdata\nimport efficientnet.tfkeras as efn\nfrom matplotlib import pyplot as plt\nfrom sklearn.utils import class_weight\nfrom sklearn.metrics import roc_auc_score\n\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE\n\nif not colab:\n    from kaggle_datasets import KaggleDatasets","execution_count":null,"outputs":[]},{"metadata":{"id":"iwZD-He0uglL"},"cell_type":"markdown","source":"## Loading data","execution_count":null},{"metadata":{"id":"9Yb5aY0cuglL","trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    np.random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    tf.random.set_seed(seed)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","id":"mCiMLPNDuglP","trusted":true},"cell_type":"code","source":"NAME='EffNB0_512'\nNFOLDS=5\nNBEST=2 # the number of best models to use for predictions\nSEED=311\nn_max=7     # the maximum number of hairs to augment\n\nif colab:\n    PATH=Path('/content/gdrive/My Drive/kaggle/input/siim-isic-melanoma-classification/') \n    train=pd.read_csv(PATH/'train.csv.zip')\nelse:\n    PATH=Path('/kaggle/input/siim-isic-melanoma-classification/')\n    train=pd.read_csv(PATH/'train.csv')\n\ntest=pd.read_csv(PATH/'test.csv')\nsub=pd.read_csv(PATH/'sample_submission.csv')\n\nseed_everything(SEED)","execution_count":null,"outputs":[]},{"metadata":{"id":"dv2uvwDpuglS","outputId":"7d75725a-c18b-4e87-afb0-8e866a7a79d0","trusted":true},"cell_type":"code","source":"print(f\"The shape of the training set is {train.shape}.\")\nprint(f\"The shape of the testing set is {test.shape}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"kf3d5LckuglW","outputId":"7ef85529-9edb-4613-f3bc-854dcbc8436c","trusted":true},"cell_type":"code","source":"print(f\"The columns in `train`:\\n {list(train.columns)}.\\n\")\nprint(f\"The columns in `test`:\\n {list(test.columns)}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"NVEHLo8sugla","outputId":"05929128-d667-4b9f-f1ca-c6a97cc7cb9a","trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"hzHasYqVugld","outputId":"1a904964-cff5-4612-cb51-e7d463e4303c","trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"zwrju15xuglg"},"cell_type":"markdown","source":"## TPU or GPU detection","execution_count":null},{"metadata":{"id":"L1cjq7Jkuglh","outputId":"5dfbbab1-5f2c-4b0b-b917-94393d0e00f8","trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment \n    # variable is set. On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"id":"4JHxDRriuglk"},"cell_type":"markdown","source":"## Data access\n\nTPUs read data directly from Google Cloud Storage (GCS). If you are running this notebook on Kaggle, use `KaggleDatasets().get_gcs_path(dataset_name)` to determine the addresses of GCS buckets holding data for a given dataset. The name of the dataset is the name of the directory it is mounted in. Use `!ls /kaggle/input/` to list attached datasets.\n\nAlso, please note that the `KaggleDatasets()` utility does not work on Colab. So, if you are running this notebook on Colab copy and paste the addresses of all GCS buckets from your Kaggle notebook (note that these addresses change periodically, so you will have to update them every once in a while).","execution_count":null},{"metadata":{"id":"Q-C8b_Tnuglo","outputId":"3879f586-7939-435c-ea11-3c5d2ed0dee9","trusted":true},"cell_type":"code","source":"GCS_PATH={}\n\nif colab:\n    # WARNING: Update the GCS bucket addresses!\n    GCS_PATH['train']='gs://kds-4169ce1e5a624d4b5bd866480682658f2063b18c5cf995cd3316a7e9'\n    GCS_PATH['test']='gs://kds-147ff1400c195e2f8c7b01492e04fd2769e0305e9ef6c01afc9c5870'\n    GCS_PATH['hairs']='gs://kds-9e567a6e6d1255950cf43b26db187228f633ac7ec6bb6dcf70c1b511'\nelse:\n    GCS_PATH['train']=KaggleDatasets().get_gcs_path('siim-tfrec-cc-512-train')\n    GCS_PATH['test']=KaggleDatasets().get_gcs_path('siim-tfrec-cc-512-test')\n    # Roman's images of hairs\n    GCS_PATH['hairs']=KaggleDatasets().get_gcs_path('melanoma-hairs')\n\nprint(GCS_PATH['train'])\nprint(GCS_PATH['test'])\nprint(GCS_PATH['hairs'])","execution_count":null,"outputs":[]},{"metadata":{"id":"XfuTfS_7ugmI"},"cell_type":"markdown","source":"## Configuration","execution_count":null},{"metadata":{"id":"UvXY4_PgXfJm","outputId":"822fa778-217a-4162-f513-15df511c24f7","trusted":true},"cell_type":"code","source":"%%time\n\nIMAGE_SIZE = [512, 512] # At this size, a GPU will run out of memory. Use the TPU.\n                          # For GPU training, please select 224 x 224 px image size.\nEPOCHS=10\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\n\nCLASSES = ['benign', 'malignant']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading and storing the .tfrec file names for train, validation, and test sets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ALL_TRAIN=tf.io.gfile.glob(GCS_PATH['train'] + '/*.tfrec')\n\nVAL_FNAMES={}\nfor fn in range(1, NFOLDS+1):\n    VAL_FNAMES[f\"fold_{fn}\"]=[path for path in ALL_TRAIN if f\"fold_{fn}\" in path]    \n    print(\"Fold\", f'{fn}:', len(VAL_FNAMES[f'fold_{fn}']), \"elements in total.\")\n    \nTRAIN_FNAMES={f'fold_{i}': list(set(ALL_TRAIN)-set(VAL_FNAMES[f'fold_{i}']))\n              for i in range(1, NFOLDS+1)}\n\nTEST_FNAMES = tf.io.gfile.glob(GCS_PATH['test'] + '/*.tfrec')\n\n# Roman's images of hairs\nhair_images=tf.io.gfile.glob(GCS_PATH['hairs'] + '/*.png')","execution_count":null,"outputs":[]},{"metadata":{"id":"Gl_RBvu0ugmM","outputId":"87d166f4-ba7d-4128-e977-65a2aab46e03","trusted":true},"cell_type":"code","source":"len(ALL_TRAIN), len(TEST_FNAMES), len(TRAIN_FNAMES), len(VAL_FNAMES), len(hair_images)","execution_count":null,"outputs":[]},{"metadata":{"id":"fpgFfNZmugmf","outputId":"392c8db5-09c7-4e9c-ed6b-04dee7ff8730","trusted":true},"cell_type":"code","source":"VAL_FNAMES","execution_count":null,"outputs":[]},{"metadata":{"id":"--7mrGiYugmi","outputId":"1d33cdcc-a049-454f-c136-a2eec5ebb0b8","trusted":true},"cell_type":"code","source":"TRAIN_FNAMES","execution_count":null,"outputs":[]},{"metadata":{"id":"ulkbgDxOugml","outputId":"a9e2fff4-c7b7-4c95-fa8e-376b251f13ab","trusted":true},"cell_type":"code","source":"ALL_TRAIN","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hair_images","execution_count":null,"outputs":[]},{"metadata":{"id":"AU7BuL2Pugmo","trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, \n    # i.e. test10-687.tfrec = 687 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    \n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"id":"W1xTKXjqugmq","outputId":"95ab1fdd-126d-44b5-9e5c-b2db6630b7ad","trusted":true},"cell_type":"code","source":"%%time\n\nN_TRAIN_IMGS = {f'fold_{i}': count_data_items(TRAIN_FNAMES[f'fold_{i}'])\n                for i in range(1, NFOLDS+1)}\n\nN_VAL_IMGS = {f'fold_{i}': count_data_items(VAL_FNAMES[f'fold_{i}'])\n              for i in range(1, NFOLDS+1)}\n\nN_TEST_IMGS = count_data_items(TEST_FNAMES)\n\nSTEPS_PER_EPOCH = {f'fold_{i}': N_TRAIN_IMGS[f'fold_{i}'] // BATCH_SIZE\n                   for i in range(1, NFOLDS+1)}\n\nprint(\"=\"*75)\n\nprint(f\"The number of unlabeled test image is {N_TEST_IMGS}. It is common for all folds.\")\n\nfor i in range(1, NFOLDS+1):\n    print(\"=\"*75)\n    print(f\"Fold {i}: {N_TRAIN_IMGS[f'fold_{i}']} training and {N_VAL_IMGS[f'fold_{i}']} validation images.\")\nprint(\"=\"*75)","execution_count":null,"outputs":[]},{"metadata":{"id":"j4Kua-WHugmt"},"cell_type":"markdown","source":"## Datasets\n\nBelow are the functions that we will be using to read and process the data from the `.tfrec` files. ","execution_count":null},{"metadata":{"id":"0UyL2B1Pugmt","trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    # convert image to floats in [0, 1] range\n    image = tf.cast(image, tf.float32) / 255.0 \n    # explicit size needed for TPU\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{"id":"Xj4KFcQ2ugmw","trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        # tf.string means bytestring\n        # shape [] means single element\n        ################################\n        # bytestring features\n        \"image\": tf.io.FixedLenFeature([], tf.string), \n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"patient_id\": tf.io.FixedLenFeature([], tf.string),\n        \"benign_malignant\": tf.io.FixedLenFeature([], tf.string),\n        # integer features\n        \"age\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex_female\": tf.io.FixedLenFeature([], tf.int64),        \n        \"sex_male\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex_unknown\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_head/neck\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_lower extremity\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_oral/genital\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_palms/soles\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_torso\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_unknown\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_upper extremity\": tf.io.FixedLenFeature([], tf.int64),\n        \"height\": tf.io.FixedLenFeature([], tf.int64),\n        \"width\": tf.io.FixedLenFeature([], tf.int64),\n        \"target\": tf.io.FixedLenFeature([], tf.int64), \n        # float features\n        \"age_scaled\": tf.io.FixedLenFeature([], tf.float32),\n    }\n\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    # image data\n    image = decode_image(example['image']) \n    data={}\n    # bytestring features\n    data['image_name']=image_name=tf.cast(example['image_name'], tf.string)\n    data['patient_id']=tf.cast(example['patient_id'], tf.string)\n    # integer features\n    data['age']=tf.cast(example['age'], tf.int32)\n    data['sex_female']=tf.cast(example['sex_female'], tf.int32)\n    data['sex_male']=tf.cast(example['sex_male'], tf.int32)\n    data['sex_unknown']=tf.cast(example['sex_unknown'], tf.int32)\n    data['site_head/neck']=tf.cast(example['site_head/neck'], tf.int32)\n    data['site_lower extremity']=tf.cast(example['site_lower extremity'], tf.int32)\n    data['site_oral/genital']=tf.cast(example['site_oral/genital'], tf.int32)\n    data['site_palms/soles']=tf.cast(example['site_palms/soles'], tf.int32)\n    data['site_torso']=tf.cast(example['site_torso'], tf.int32)\n    data['site_unknown']=tf.cast(example['site_unknown'], tf.int32)\n    data['site_upper extremity']=tf.cast(example['site_upper extremity'], tf.int32)\n#     data['height']=tf.cast(example['height'], tf.int32)\n#     data['width']=tf.cast(example['width'], tf.int32)\n    # float features\n    data['age_scaled']=tf.cast(example['age_scaled'], tf.float32)\n    # target (integer)\n    label=tf.cast(example['target'], tf.int32)\n     # target (string)\n    label_name=tf.cast(example['benign_malignant'], tf.string)\n\n    return image, label, data, label_name","execution_count":null,"outputs":[]},{"metadata":{"id":"LNQ3UpQfugmz","trusted":true},"cell_type":"code","source":"def read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        # tf.string means bytestring\n        # shape [] means single element\n        ################################\n        # bytestring features\n        \"image\": tf.io.FixedLenFeature([], tf.string), \n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"patient_id\": tf.io.FixedLenFeature([], tf.string),\n        # integer features\n        \"age\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex_female\": tf.io.FixedLenFeature([], tf.int64),        \n        \"sex_male\": tf.io.FixedLenFeature([], tf.int64),\n        \"sex_unknown\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_head/neck\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_lower extremity\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_oral/genital\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_palms/soles\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_torso\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_unknown\": tf.io.FixedLenFeature([], tf.int64),\n        \"site_upper extremity\": tf.io.FixedLenFeature([], tf.int64),\n        \"height\": tf.io.FixedLenFeature([], tf.int64),\n        \"width\": tf.io.FixedLenFeature([], tf.int64), \n        # float features\n        \"age_scaled\": tf.io.FixedLenFeature([], tf.float32),\n    }\n\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    # image data\n    image = decode_image(example['image']) \n    data={}\n    # bytestring features\n    data['image_name']=image_name=tf.cast(example['image_name'], tf.string)\n    data['patient_id']=tf.cast(example['patient_id'], tf.string)\n    # integer features\n    data['age']=tf.cast(example['age'], tf.int32)\n    data['sex_female']=tf.cast(example['sex_female'], tf.int32)\n    data['sex_male']=tf.cast(example['sex_male'], tf.int32)\n    data['sex_unknown']=tf.cast(example['sex_unknown'], tf.int32)\n    data['site_head/neck']=tf.cast(example['site_head/neck'], tf.int32)\n    data['site_lower extremity']=tf.cast(example['site_lower extremity'], tf.int32)\n    data['site_oral/genital']=tf.cast(example['site_oral/genital'], tf.int32)\n    data['site_palms/soles']=tf.cast(example['site_palms/soles'], tf.int32)\n    data['site_torso']=tf.cast(example['site_torso'], tf.int32)\n    data['site_unknown']=tf.cast(example['site_unknown'], tf.int32)\n    data['site_upper extremity']=tf.cast(example['site_upper extremity'], tf.int32)\n#     data['height']=tf.cast(example['height'], tf.int32)\n#     data['width']=tf.cast(example['width'], tf.int32)\n    # float features\n    data['age_scaled']=tf.cast(example['age_scaled'], tf.float32)\n\n    return image, data","execution_count":null,"outputs":[]},{"metadata":{"id":"hqcM3BK5ugm2","trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files \n    # at once and disregarding data order. Order does not matter since we will \n    # be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        # disable order, increase speed\n        ignore_order.experimental_deterministic = False\n\n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.with_options(ignore_order)\n    # returns a dataset of (image, label) pairs if labeled=True \n    # or (image, id) pairs if labeled=False\n    dataset = dataset.map(read_labeled_tfrecord if labeled \n                          else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a quick look at some training and validation examples:","execution_count":null},{"metadata":{"id":"u5QMwJq7ugm5","outputId":"1b534ba9-fc16-47ac-dddf-3123b895e0a0","trusted":true},"cell_type":"code","source":"%%time\n\ntraining_dataset = load_dataset(TRAIN_FNAMES['fold_2'])\n\nprint(\"Example of the training data:\")\nfor image, label, data, label_name in training_dataset.take(1):\n    print(\"The image batch size:\", image.numpy().shape)\n    print(\"Label:\", label.numpy())\n    print(\"Label name:\", label_name.numpy())\n    print(\"Age:\", data['age'].numpy())\n    print(\"Age (scaled):\", data['age_scaled'].numpy())","execution_count":null,"outputs":[]},{"metadata":{"id":"xLFNbXcaugm8","outputId":"2c13e096-4cdb-432d-a3a9-8da4950cfb33","trusted":true},"cell_type":"code","source":"%%time\n\nvalidation_dataset = load_dataset(VAL_FNAMES['fold_2'])\n\nprint(\"Examples of the validation data:\")\nfor image, label, data, label_name in validation_dataset.take(1):\n    print(\"The image batch size:\", image.numpy().shape)\n    print(\"Label:\", label.numpy())\n    print(\"Label name:\", label_name.numpy())\n    print(\"Age:\", data['age'].numpy())\n    print(\"Age (scaled):\", data['age_scaled'].numpy())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is a function for generagting a test dataset.","execution_count":null},{"metadata":{"id":"pvbWYWXcugnA","trusted":true},"cell_type":"code","source":"def get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FNAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And here are some quick examples of the test data:","execution_count":null},{"metadata":{"id":"s19U5UZhugnD","outputId":"19415e06-2afc-4ecb-8543-2c5ccb51ed5f","trusted":true},"cell_type":"code","source":"%%time\n\ntest_dataset = get_test_dataset()\n\nprint(\"Examples of the test data:\")\nfor image, data in test_dataset.take(1):\n    print(\"The image batch size:\", image.numpy().shape)\n    print(\"Ages, 5 examples:\", data['age'].numpy()[:5])\n    print(\"Age (scaled), 5 examples:\", data['age_scaled'].numpy()[:5])\n#     print(\"Height, 5 examples:\", data['height'].numpy()[:5])\n#     print(\"Width, 5 examples:\", data['width'].numpy()[:5])","execution_count":null,"outputs":[]},{"metadata":{"id":"1nmPLSlmugnu"},"cell_type":"markdown","source":"## Model","execution_count":null},{"metadata":{"id":"DHBNxb07ugnu","outputId":"0b689531-e141-4ad5-8a3c-205720db037a","trusted":true},"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.000005#0.00001\nLR_MAX = 0.00000725 * strategy.num_replicas_in_sync\nLR_MIN = 0.000005\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 4\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{"id":"OaaJgH9Vugnw","outputId":"f76aa200-572f-4a41-d91b-a94b5133fa2b","trusted":true},"cell_type":"code","source":"train['target'].values\nclass_weights = class_weight.compute_class_weight(class_weight='balanced',\n                                                  classes=np.unique(train['target'].values),\n                                                  y=train['target'].values,\n                                                 )\n\nclass_weights = {i : class_weights[i] for i in range(len(class_weights))}\n\nprint(class_weights)","execution_count":null,"outputs":[]},{"metadata":{"id":"cAWBwBEUTTuk","outputId":"d7d9d4f1-f899-44e6-be18-8791f820cf01","trusted":true},"cell_type":"code","source":"tab_feats=['age_scaled',\n           'sex_female', \n           'sex_male', \n           'sex_unknown', \n           'site_head/neck', \n           'site_lower extremity', \n           'site_oral/genital',\n           'site_palms/soles',\n           'site_torso',\n           'site_unknown',\n           'site_upper extremity',\n#            'height',\n#            'width',\n          ]\n\nN_TAB_FEATS=len(tab_feats)\n\nprint(f\"The number of tabular features is {N_TAB_FEATS}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"EuHZF_t5ugnz","outputId":"73711688-be72-466c-8054-9fef0128887c","trusted":true},"cell_type":"code","source":"%time\n\ndef get_model():\n    with strategy.scope():\n        pretrained_model = efn.EfficientNetB0(input_shape=(*IMAGE_SIZE, 3),\n                                              weights='imagenet',\n                                              include_top=False\n                                             )\n        # False = transfer learning, True = fine-tuning\n        pretrained_model.trainable = True#False \n\n        inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE, 3), name='inp1')\n        inp2 = tf.keras.layers.Input(shape=(N_TAB_FEATS), name='inp2')\n        \n        # BUILD MODEL HERE\n        \n        x=pretrained_model(inp1)\n        x=tf.keras.layers.GlobalAveragePooling2D()(x)\n        x=tf.keras.layers.Dense(512, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(256, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(128, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        x=tf.keras.layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(x)\n        x=tf.keras.layers.Dropout(0.2)(x)\n        \n        y=tf.keras.layers.Dense(100, \n                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n                                activation='relu')(inp2)\n        \n        concat=tf.keras.layers.concatenate([y, x])\n        \n        output = tf.keras.layers.Dense(1, activation='sigmoid', name='output')(concat)\n        \n        model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=[output])\n    \n        model.compile(\n        optimizer='adam',\n        loss = 'binary_crossentropy',\n        metrics=[tf.keras.metrics.AUC()],\n        )\n        \n        return model","execution_count":null,"outputs":[]},{"metadata":{"id":"G8QfbjTtugn1","trusted":true},"cell_type":"code","source":"%%time\n\nmodel=get_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"id":"jefX_l-7jyl1","trusted":true},"cell_type":"code","source":"del model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"id":"CSBEsfb4s3bW","trusted":true},"cell_type":"code","source":"if colab:\n    \n    SAVE_FOLDER=NAME\n    \n    if tstamp:\n        time_zone = pytz.timezone('America/Chicago')\n        current_datetime = datetime.now(time_zone)\n        ts=current_datetime.strftime(\"%m%d%H%M%S\")\n        SAVE_FOLDER+='_'+ts\n        \n    SAVE_FOLDER=PATH/SAVE_FOLDER\n    if not os.path.exists(SAVE_FOLDER):\n        os.mkdir(SAVE_FOLDER)\n\nelse:\n    SAVE_FOLDER=Path('/kaggle/working')","execution_count":null,"outputs":[]},{"metadata":{"id":"cr2t2NC3jr_G","trusted":true},"cell_type":"code","source":"class save_best_n(tf.keras.callbacks.Callback):\n    def __init__(self, fn, model):\n        self.fn = fn\n        self.model = model\n\n    def on_epoch_end(self, epoch, logs=None):\n        \n        if (epoch>0):\n            score=logs.get(\"val_auc\")\n        else:\n            score=-1\n      \n        if (score > best_score[fold_num].min()):\n          \n            idx_min=np.argmin(best_score[fold_num])\n\n            best_score[fold_num][idx_min]=score\n            best_epoch[fold_num][idx_min]=epoch+1\n\n            path_best_model=f'best_model_fold_{self.fn}_{idx_min}.hdf5'\n            self.model.save(SAVE_FOLDER/path_best_model)\n            ############# WARNING: ##################################\n            # Make sure you have enough space to store your models. \n            # Remember that Kaggle allows you save not more than 5 Gb\n            # to disk. It should not be a problem for EfficientNet B0 \n            # or B3 but it is not going to work for B7. I am saving my\n            # models to Google Drive where I have plenty of space.","execution_count":null,"outputs":[]},{"metadata":{"id":"Q5k46VXBugn3"},"cell_type":"markdown","source":"## Setting up the train/validation data pipeline with augmentation","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def setup_input(image, label, data, label_name):\n    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    \n    tabular=tf.stack(tab_data)\n    \n    return {'inp1': image, 'inp2':  tabular}, label","execution_count":null,"outputs":[]},{"metadata":{"id":"tzQ9BfxiZCGo","trusted":true},"cell_type":"code","source":"def data_augment(data, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement \n    # in the next function (below), this happens essentially for free on TPU. \n    # Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(data['inp1'])\n    image = tf.image.random_flip_up_down(image)\n    \n    return {'inp1': image, 'inp2':  data['inp2']}, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hair_images_tf=tf.convert_to_tensor(hair_images)\nscale=tf.cast(IMAGE_SIZE[0]/256, dtype=tf.int32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def hair_aug(data, label):\n    # Copy the input image, so it won't be changed\n    img=tf.identity(data['inp1']) \n    # Randomly choose the number of hairs to augment (up to n_max)\n    n_hairs = tf.random.uniform(shape=[], maxval=tf.constant(n_max)+1, \n                                dtype=tf.int32)\n    \n    im_height=tf.shape(img)[1]\n    im_width=tf.shape(img)[0]\n    \n    if n_hairs == 0:\n        return data, label\n\n    for _ in tf.range(n_hairs):\n\n        # Read a random hair image\n        i=tf.random.uniform(shape=[], maxval=tf.shape(hair_images_tf)[0], \n                            dtype=tf.int32)\n        fname=hair_images_tf[i]\n\n        bits = tf.io.read_file(fname)\n        hair = tf.image.decode_jpeg(bits)\n        \n        # Rescale the hair image to the right size (256 -- original size)\n        new_width=scale*tf.shape(hair)[1]\n        new_height=scale*tf.shape(hair)[0]\n        hair = tf.image.resize(hair, [new_height, new_width])\n\n        \n        # Random flips of the hair image\n        hair = tf.image.random_flip_left_right(hair)\n        hair = tf.image.random_flip_up_down(hair)\n        # Random number of 90 degree rotations\n        n_rot=tf.random.uniform(shape=[], maxval=4,\n                                dtype=tf.int32)\n        hair = tf.image.rot90(hair, k=n_rot)\n        \n        h_height=tf.shape(hair)[0]\n        h_width=tf.shape(hair)[1]\n        \n        roi_h0 = tf.random.uniform(shape=[], maxval=im_height - h_height + 1, \n                                    dtype=tf.int32)\n        roi_w0 = tf.random.uniform(shape=[], maxval=im_width - h_width + 1, \n                                    dtype=tf.int32)\n\n\n        roi = img[roi_h0:(roi_h0 + h_height), roi_w0:(roi_w0 + h_width)]  \n\n        # Convert the hair image to grayscale \n        # (slice to remove the trainsparency channel)\n        hair2gray = tf.image.rgb_to_grayscale(hair[:, :, :3])\n\n        mask=hair2gray>10\n\n        img_bg = tf.multiply(roi, tf.cast(tf.image.grayscale_to_rgb(~mask),\n                                          dtype=tf.float32))\n        hair_fg = tf.multiply(tf.cast(hair[:, :, :3], dtype=tf.int32),\n                              tf.cast(tf.image.grayscale_to_rgb(mask), \n                                      dtype=tf.int32\n                                      )\n                             )\n\n        dst = tf.add(img_bg, tf.cast(hair_fg, dtype=tf.float32)/255)\n\n        paddings = tf.stack([\n            [roi_h0, im_height-(roi_h0 + h_height)], \n            [roi_w0, im_width-(roi_w0 + h_width)],\n            [0, 0]\n        ])\n\n        # Pad dst with zeros to make it the same shape as image.\n        dst_padded=tf.pad(dst, paddings, \"CONSTANT\")\n        # Create a boolean mask with zeros at the pixels of\n        # the augmentation segment and ones everywhere else\n        mask_img=tf.pad(tf.ones_like(dst), paddings, \"CONSTANT\")\n        mask_img=~tf.cast(mask_img, dtype=tf.bool)\n        # Make a hole in the original image at the location\n        # of the augmentation segment\n        img_hole=tf.multiply(img, tf.cast(mask_img, dtype=tf.float32))\n        # Inserting the augmentation segment in place of the hole\n        img=tf.add(img_hole, dst_padded)\n        \n    return {'inp1': img, 'inp2':  data['inp2']}, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_training_dataset(dataset):\n    # horizontal and vertical random flips\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    # advanced hair augmentation\n    dataset = dataset.map(hair_aug, num_parallel_calls=AUTO)\n    # the training dataset must repeat for several epochs\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(512)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"id":"lRfOGnqCugn9","trusted":true},"cell_type":"code","source":"def get_validation_dataset(dataset):\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"id":"bXM39S6zugnG"},"cell_type":"markdown","source":"## Visualization utilities","execution_count":null},{"metadata":{"id":"E656VW7cugnH","trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)","execution_count":null,"outputs":[]},{"metadata":{"id":"uq3v8VzlugnJ","trusted":true},"cell_type":"code","source":"def batch_to_numpy_images_and_labels(databatch, ds='train'):\n    if ds=='train':\n        data, labels = databatch\n        numpy_images = data['inp1'].numpy()\n        numpy_labels = labels.numpy()\n    else:\n        data = databatch\n        numpy_images = data['inp1'].numpy()\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels","execution_count":null,"outputs":[]},{"metadata":{"id":"o8Ry9wnTugnN","trusted":true},"cell_type":"code","source":"def title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" \n                                if not correct else '', \n                                CLASSES[correct_label] if not correct else ''), correct","execution_count":null,"outputs":[]},{"metadata":{"id":"sX6JyMKIugnP","trusted":true},"cell_type":"code","source":"def display_one_image(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), \n                  color='red' if red else 'black', fontdict={'verticalalignment':'center'}, \n                  pad=int(titlesize/1.5)\n                 )\n    return (subplot[0], subplot[1], subplot[2]+1)","execution_count":null,"outputs":[]},{"metadata":{"id":"gLnRL6T8ugnS","trusted":true},"cell_type":"code","source":"def display_batch_of_images(databatch, predictions=None, ds='train'):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch, ds=ds)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does  \n    # not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        # magic formula tested to work from 1x1 to 10x10 images\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3\n        subplot = display_one_image(image, title, subplot, \n                                     not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset visualizations","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at training data\n\ntraining_dataset = training_dataset.map(setup_input, num_parallel_calls=AUTO)   \ntraining_dataset = get_training_dataset(training_dataset)\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del training_dataset, train_batch\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Peek at validation data\n\nvalidation_dataset = validation_dataset.map(setup_input, num_parallel_calls=AUTO)\nvalidation_dataset = get_validation_dataset(validation_dataset)\nvalidation_dataset = validation_dataset.unbatch().batch(20)\nvalidation_batch = iter(validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(validation_batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del validation_dataset, validation_batch\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def setup_test_image(image, data):    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    tabular=tf.stack(tab_data)\n\n    return {'inp1': image, 'inp2': tabular}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_test_dataset(dataset):\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# peer at test data\n\ntest_dataset = load_dataset(TEST_FNAMES, labeled=False, ordered=True)\ntest_dataset = test_dataset.map(setup_test_image, num_parallel_calls=AUTO)\ntest_dataset = get_test_dataset(test_dataset)\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(test_batch), ds='test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del test_dataset, test_batch\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training","execution_count":null},{"metadata":{"id":"7fbGd3r4ugn_","outputId":"955d46f0-71b7-403f-c3db-5ec161f30242","trusted":true},"cell_type":"code","source":"%%time\n\ndebug=0\n    \nhistories = []\n\nbest_epoch={fn: np.zeros(NBEST) for fn in range(1, NFOLDS+1)}\nbest_score={fn: np.zeros(NBEST) for fn in range(1, NFOLDS+1)}\n\nfor fold_num in range(1, NFOLDS+1):\n    \n    tf.keras.backend.clear_session()\n    # clear tpu memory (otherwise can run into Resource Exhausted Error)\n    # see https://www.kaggle.com/c/flower-classification-with-tpus/discussion/131045\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    print(\"=\"*50)\n    print(f\"Starting fold {fold_num} out of {NFOLDS}...\")\n    \n    files_trn=TRAIN_FNAMES[f\"fold_{fold_num}\"]\n    files_val=VAL_FNAMES[f\"fold_{fold_num}\"]\n    \n    if debug:\n        files_trn=files_trn[0:2]\n        files_val=files_val[0:2]\n        EPOCHS=3\n       \n    train_dataset = load_dataset(files_trn)\n    train_dataset = train_dataset.map(setup_input, num_parallel_calls=AUTO)\n    \n    val_dataset = load_dataset(files_val, ordered = True)\n    val_dataset = val_dataset.map(setup_input, num_parallel_calls=AUTO)\n    \n    model = get_model()\n    \n    STEPS_PER_EPOCH = count_data_items(files_trn) // BATCH_SIZE\n    \n    print(f'STEPS_PER_EPOCH = {STEPS_PER_EPOCH}')\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n    \n    history = model.fit(get_training_dataset(train_dataset), \n                        steps_per_epoch=STEPS_PER_EPOCH, \n                        epochs=EPOCHS, \n                        callbacks=[lr_callback,\n                                   save_best_n(fold_num, model),\n                                   ],\n                        validation_data=get_validation_dataset(val_dataset),\n                        class_weight=class_weights,\n                        verbose=2,\n                       )\n    \n    idx_sorted=np.argsort(best_score[fold_num])\n    best_score[fold_num]=np.array(best_score[fold_num])[idx_sorted]\n    best_epoch[fold_num]=np.array(best_epoch[fold_num])[idx_sorted]\n\n    print(f\"\\nFold {fold_num} is finished. The best epochs: {[int(best_epoch[fold_num][i]) for i in range(len(best_epoch[fold_num]))]}\")\n    print(f\"The corresponding scores: {[round(best_score[fold_num][i], 5) for i in range(len(best_epoch[fold_num]))]}\")\n\n    histories.append(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualization of training progress","execution_count":null},{"metadata":{"id":"hEmYCG7vugoA","trusted":true},"cell_type":"code","source":"def display_training_curves(fold_num, data):\n\n    plt.figure(figsize=(10,5), facecolor='#F0F0F0')\n\n    epochs=np.arange(1, EPOCHS+1)\n\n    # AUC\n    plt.plot(epochs, data['auc'], label='training auc', color='red')\n    plt.plot(epochs, data['val_auc'], label='validation auc', color='orange')\n\n    # Loss\n    plt.plot(epochs, data['loss'], label='training loss', color='blue')    \n    plt.plot(epochs, data['val_loss'], label='validation loss', color='green')\n\n    # Best\n    ls=['dotted', 'dashed', 'dashdot', 'solid'] # don't use more than 4 best epochs \n                                                # or make proper adjustments!\n    for i in range(NBEST):\n        plt.axvline(best_epoch[fold_num][i], 0, \n                    best_score[fold_num][i], linestyle=ls[i], \n                    color='black', label=f'AUC {best_score[fold_num][i]:.5f}')\n    \n    plt.title(f\"Fold {fold_num}. The best epochs: {[int(best_epoch[fold_num][i]) for i in range(len(best_epoch[fold_num]))]}; the best AUC's: {[round(best_score[fold_num][i], 5) for i in range(len(best_epoch[fold_num]))]}.\", \n              fontsize='14')\n    plt.ylabel('Loss/AUC', fontsize='12')\n    plt.xlabel('Epoch', fontsize='12')\n    plt.ylim((0, 1))\n    plt.legend(loc='lower left')\n    plt.tight_layout()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"WicxNtrlugoE","trusted":true},"cell_type":"code","source":"for fn in range(1, NFOLDS+1):\n    display_training_curves(fn, data=histories[fn-1].history)","execution_count":null,"outputs":[]},{"metadata":{"id":"ZVU0MJFkugoH"},"cell_type":"markdown","source":"## Predictions","execution_count":null},{"metadata":{"id":"OdB4KIVDugoI","trusted":true},"cell_type":"code","source":"def setup_test_image(image, data):    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    tabular=tf.stack(tab_data)\n\n    return {'inp1': image, 'inp2': tabular}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def setup_test_name(image, data):\n    return data['image_name']","execution_count":null,"outputs":[]},{"metadata":{"id":"Us-Jjd4FaLyz","trusted":true},"cell_type":"code","source":"def average_predictions(X, fn):\n    \n    y_probas=[]\n    \n    for idx in range(NBEST):\n        \n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        strategy = tf.distribute.experimental.TPUStrategy(tpu)\n        gc.collect()\n\n        print(f\"Predicting: fold {fn}, model {idx+1} out of {NBEST}...\")\n\n        with strategy.scope():\n            path_best_model=f'best_model_fold_{fn}_{idx}.hdf5'\n            model=tf.keras.models.load_model(SAVE_FOLDER/path_best_model)\n\n        y=model.predict(X)\n        y = rankdata(y)/len(y)\n        y_probas.append(y)\n    \n    y_probas=np.average(y_probas, axis=0)\n\n    return y_probas","execution_count":null,"outputs":[]},{"metadata":{"id":"bEKgTC8VugoL","trusted":true},"cell_type":"code","source":"%%time\n\npreds = pd.DataFrame({'image_name': np.zeros(len(test)), 'target': np.zeros(len(test))})\n\ntest_ds = load_dataset(TEST_FNAMES, labeled=False, ordered=True)\ntest_images_ds = test_ds.map(setup_test_image, num_parallel_calls=AUTO)\n\ntest_images_ds = get_test_dataset(test_images_ds)\ntest_ds = get_test_dataset(test_ds)\n\ntest_ids_ds = test_ds.map(setup_test_name, num_parallel_calls=AUTO).unbatch()\n\npreds['image_name'] = next(iter(test_ids_ds.batch(N_TEST_IMGS))).numpy().astype('U')\npreds['target'] = np.average([average_predictions(test_images_ds, fn) for fn in range(1, NFOLDS+1)], axis = 0)","execution_count":null,"outputs":[]},{"metadata":{"id":"xqyDWup3ugoN","trusted":true},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"_IZx-PvBugoO","trusted":true},"cell_type":"code","source":"del sub['target']\nsub = sub.merge(preds, on='image_name')\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"FgtkXbHnBc6Q","trusted":true},"cell_type":"code","source":"print(f\"The lengths of the submission file and `test` are {len(sub)} and {len(test)}, respectively.\")\nprint(f\"The number of NA's in the submission file is {sub.isna().sum().sum()}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"pIPm950YQWM-","trusted":true},"cell_type":"code","source":"if colab:\n    OUT_FOLDER=SAVE_FOLDER\nelse:\n    OUT_FOLDER=Path('')\n    \nsub.to_csv(OUT_FOLDER/'submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"id":"OPjhPEEOugoQ"},"cell_type":"markdown","source":"## Out of fold predictions","execution_count":null},{"metadata":{"id":"2Z8lVheYugoQ","trusted":true},"cell_type":"code","source":"def setup_val_image(image, label, data, label_name):\n    \n    tab_data=[tf.cast(data[tfeat], dtype=tf.float32) for tfeat in tab_feats]\n    \n    tabular=tf.stack(tab_data)\n\n    return {'inp1': image, 'inp2': tabular}","execution_count":null,"outputs":[]},{"metadata":{"id":"RvfCedm3ugoS","trusted":true},"cell_type":"code","source":"def setup_val_name(image, label, data, label_name):\n    return data['image_name']","execution_count":null,"outputs":[]},{"metadata":{"id":"S8M7knAjugoU","trusted":true},"cell_type":"code","source":"def setup_val_label(image, label, data, label_name):\n    return label","execution_count":null,"outputs":[]},{"metadata":{"id":"sYA-hj_dugoW","trusted":true},"cell_type":"code","source":"oof= pd.DataFrame({'image_name': train['image_name'].values})\n\nauc=[]\noof_all_folds=[]\n\nfor fold_num in range(1, NFOLDS+1):\n    \n    tf.keras.backend.clear_session()\n    \n    print(\"=\"*50)\n    print(f\"Starting fold {fold_num}...\")    \n\n    print(f\"The best epochs: {[int(best_epoch[fold_num][i]) for i in range(len(best_epoch[fold_num]))]}\")\n    print(f\"The corresponding scores: {[round(best_score[fold_num][i], 5) for i in range(len(best_epoch[fold_num]))]}\")\n\n    files_val=VAL_FNAMES[f\"fold_{fold_num}\"]\n    \n    if debug:\n        files_val=files_val[0:2]\n    \n    val_ds = load_dataset(files_val, ordered = True)\n    val_images_ds = val_ds.map(setup_val_image,\n                               num_parallel_calls=AUTO)\n    val_images_ds = get_validation_dataset(val_images_ds)\n        \n    val_ds = get_validation_dataset(val_ds)\n\n    val_label_ds = val_ds.map(setup_val_label,\n                              num_parallel_calls=AUTO).unbatch()\n    val_ids_ds = val_ds.map(setup_val_name,\n                            num_parallel_calls=AUTO).unbatch()\n    \n    n_val_fold = count_data_items(files_val)\n    \n    print(f'The # of validation files = {n_val_fold}')    \n    \n    oof_fold= pd.DataFrame()\n    \n    oof_fold['image_name'] = next(iter(val_ids_ds.batch(n_val_fold))).numpy().astype('U')\n\n    oof_fold['target'] = average_predictions(val_images_ds, fold_num)\n\n    oof_all_folds.append(oof_fold)\n\n    y_true = next(iter(val_label_ds.batch(n_val_fold))).numpy()\n\n    auc_fold=roc_auc_score(y_true, oof_fold['target'].values)\n    auc.append(auc_fold)\n    \n    print(f\"Fold {fold_num} is done! ROC AUC = {auc_fold:.5f}\")\n\noof=oof.merge(pd.concat(oof_all_folds), \n              on='image_name', \n              how='left').reset_index(drop=True)\n\nif debug:\n    oof=oof.fillna(0)\n\nauc=np.array(auc)\nauc_av=auc.mean()\nauc_std=auc.std()\n\nprint(f\"ROC AUC = {auc_av:.5f}, STDEV = {auc_std:.5f} (average across the folds)\")\n\nauc_oof=roc_auc_score(train['target'].values, oof['target'].values)\n\nprint(f\"ROC AUC = {auc_oof:.5f} (out of folds)\")","execution_count":null,"outputs":[]},{"metadata":{"id":"hVpz3eS_ChnY","trusted":true},"cell_type":"code","source":"print(f\"The lengths of the oof data frame and `train` are {len(oof)} and {len(train)}, respectively.\")\nprint(f\"The number of NA's in the oof data frame is {oof.isna().sum().sum()}.\")","execution_count":null,"outputs":[]},{"metadata":{"id":"4CAr5WvyNoDc","trusted":true},"cell_type":"code","source":"oof.to_csv(OUT_FOLDER/'oof.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Blending \n\nLet's blend our predictions with those from earlier versionos of this notebook. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# # This is included here for debuging\n\n# if colab:\n#     OUT_FOLDER=SAVE_FOLDER\n# else:\n#     OUT_FOLDER=Path('')\n\n# auc=5\n    \n# sub=pd.read_csv('/kaggle/input/siim-benchmark-submission/submission_v20.csv')\n# oof=pd.read_csv('/kaggle/input/siim-benchmark-submission/oof_v20.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_v9=pd.read_csv('/kaggle/input/siim-benchmark-submission/submission_v9.csv')['target'].values\nsub_v10=pd.read_csv('/kaggle/input/siim-benchmark-submission/submission_v10.csv')['target'].values\nsub_v13=pd.read_csv('/kaggle/input/siim-benchmark-submission/submission_v10.csv')['target'].values\n# sub_v20=pd.read_csv('/kaggle/input/siim-benchmark-submission/submission_v20.csv')['target'].values\n\nsub_v9=rankdata(sub_v9)/len(sub_v9)\nsub_v10=rankdata(sub_v10)/len(sub_v10)\nsub_v13=rankdata(sub_v13)/len(sub_v13)\n# sub_v20=rankdata(sub_v20)/len(sub_v20)\n\noof_v9=pd.read_csv('/kaggle/input/siim-benchmark-submission/oof_v9.csv')['target'].values\noof_v10=pd.read_csv('/kaggle/input/siim-benchmark-submission/oof_v10.csv')['target'].values\noof_v13=pd.read_csv('/kaggle/input/siim-benchmark-submission/oof_v13.csv')['target'].values\n# oof_v20=pd.read_csv('/kaggle/input/siim-benchmark-submission/oof_v20.csv')['target'].values\n\noof_v9=rankdata(oof_v9)/len(oof_v9)\noof_v10=rankdata(oof_v10)/len(oof_v10)\noof_v13=rankdata(oof_v13)/len(oof_v13)\n# oof_v20=rankdata(oof_v20)/len(oof_v20)\n\n\npreds_blend=1/2*1/3*(sub_v9+\n                     sub_v10+\n#                    sub_v20+\n                     sub['target'].values\n                    )+1/2*sub_v13\n\noof_blend=1/2*1/3*(oof_v9+\n                   oof_v10+\n#                    oof_v20+\n                   oof['target'].values\n                  )+1/2*oof_v13\n\nauc_blend=roc_auc_score(train['target'].values, oof_blend)\nprint(f\"ROC AUC score of the blend = {auc_blend:.5f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Saving the result of the blend","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_blend=sub.copy()\nsub_blend['target']=preds_blend\nsub_blend.to_csv(OUT_FOLDER/'submission_blend.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sanity check\n\nFinally, we will compute the correlation coefficient between our predictions for the test set and a similar predictions taken from **Version 9** of this Kaggle notebook that scored 0.914 on the public leader board (model -- EfficientNet B0). This is just our sanity check.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"corr=np.corrcoef(rankdata(sub_v9)/len(sub_v9), \n                 rankdata(sub['target'].values)/len(sub))\n\nprint(f\"The correlation with the benchmark submission: \\n {corr}.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ROC curve","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"del auc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_curve, auc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_ROC(oof, model_title, fs):\n    \n    fpr, tpr, thresholds = roc_curve(train['target'].values, oof)\n    roc_auc = auc(fpr, tpr)\n    \n    plt.title(f'ROC: {model_title}', fontsize=fs)\n    plt.plot(fpr, tpr, 'b', label = 'AUC = %0.5f' % roc_auc)\n    plt.legend(loc = 'lower right')\n    plt.plot([0, 1], [0, 1],'r--')\n    plt.xlim([0, 1])\n    plt.ylim([0, 1])\n    plt.ylabel('True Positive Rate', fontsize=fs)\n    plt.xlabel('False Positive Rate', fontsize=fs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10, 5))\n\nplt.subplot(1, 2, 1)\nplot_ROC(oof['target'].values, 'single model', 14)\n\nplt.subplot(1, 2, 2)\nplot_ROC(oof_blend, 'blend', 14)\n\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}