{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1 style=\"display:inline;\">Healthcare Deep Learning with TensorFlow</h1>\n<br>\n<h2 style=\"display:inline;\">Spiro Ganas, MS</h2>\n\n\n<a href=\"https://www.kaggle.com/spiroganas/healthcare-deep-learning-table-of-contents\">Table of Contents</a> \n<br>\n<h2>Chapter 8 - Medical Image Analysis</h2>\n\n\n<strong>This tutorial will explain how to use a TPU and TRFrecords to train a classification model on medical images.</strong>\n\nAn interactive version of the notebook is available on Kaggle at:<br>\nhttps://www.kaggle.com/spiroganas/healthcare-deep-learning-chapter-8\n<hr>\n\nKaggle's [RANZCR CLiP - Catheter and Line Position Challenge](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification) provides data in the [TFRecord format](https://www.tensorflow.org/tutorials/load_data/tfrecord).  TensorFlow can quickly read data stored in the TFRecord format.  This is especially important when the model is being trained on fast GPUs or [TPUs](https://cloud.google.com/tpu) (where IO is frequently the training bottleneck).\n\nThis notebook shows how to to use TFRecords, TensorFlow and a TPU to train a medical image classification model. \n\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Determine if you are using a TPU or GPU/CPU\n\nThis section sets up your TPU or GPU, and sets some flags so the rest of the code can be optimized for the accelerator you are using.\n\nSetting up the TPU needs to be the first step in the program."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(\"Tensorflow version \" + tf.__version__)\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\n    print(\"Using the TPU :-)\")\n    print(\"Number of accelerators: \", strategy.num_replicas_in_sync)\n    USE_TPU = True\n    USE_GCS = True\n    \n    # If using a TPU, turn off eager execution\n    tf.config.run_functions_eagerly(False)\n\nexcept:\n    USE_TPU = False\n    USE_GCS = False\n\n\n\n\nif not USE_TPU:\n    # Set up the GPU\n    strategy = tf.distribute.get_strategy() # default strategy that works on CPU and single GPU\n    #strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy() # for clusters of multi-GPU machines\n    print(\"Using the GPU...\")\n    # View information about the GPU\n    gpu_info = !nvidia-smi\n    gpu_info = '\\n'.join(gpu_info)\n    if gpu_info.find('failed') >= 0:\n        print(\"No GPU enabled!\")\n        print('Select the Runtime > \"Change runtime type\" menu to enable a GPU accelerator, ')\n        print('and then re-execute this cell.')\n    else:\n        print(gpu_info)\n        print()\n        print(\"------------------------------------------------------------------------\")\n        print()\n\n\n\n# This lets you save to the google drive when running a TPU\nsave_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Constants used to train the model\n\n\nDATASET_IMAGE_SIZE = 800\n\nIMAGE_SIZE = 240 #512, 750\n\nLEARNING_RATE = 0.001      # use 0.00001 if training the unfrozen model, 0.001 is the default\nNUMBER_OF_EPOCHS = 17\nSTEPS_PER_EPOCH = None\nBATCH_SIZE=64\nUSE_CUSTOM_LOSS_FUNCTION = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Import libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"from io import BytesIO\nfrom imageio import imread\nimport os\nimport random\nimport csv\nimport glob\nimport datetime\n\n# clear out anything leftover from the last run\ntf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Find the Data\n\nGPUs can access data on your local drives, but TPUs can only read data from Google Cloud Storage (GCS).\nKaggle has created a helper function that will let you read the files from GCS.\n\nWe are also using the TFRecord version of the data.  The TFRecord format was designed to maximize the input/output (IO) speed for your model. \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_GCS:\n    # If you are using a TPU, the input data needs to be stored on Google Cloud Storage\n    from kaggle_datasets import KaggleDatasets  # Helper function from Kaggle that gives you access to a GCS bucket holding the competition data\n    training_data_folder = KaggleDatasets().get_gcs_path(\"ranzcr-clip-catheter-line-classification\") + \"/train_tfrecords/\"\n    !gsutil ls $training_data_folder\nelse:\n    # GPUs can read data that is stored locally or on GCS,\n    # but local storage should be faster.\n    training_data_folder = '/kaggle/input/ranzcr-clip-catheter-line-classification/train_tfrecords/'\nprint(training_data_folder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4:  Decrease precision to improve speed and memory utilization\n\nTPUs (and some GPUs) can use smaller datatypes (e.g. fewer decimal points) to speed up the calculations while reducing the amount of memory used.  This (usually) won't impact the accuracy of your model."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Speed up training by enabling mixed precision data types\n#https://www.tensorflow.org/guide/mixed_precision\nif False:\n\n    if not USE_TPU:\n        policy = tf.keras.mixed_precision.Policy('mixed_float16')\n        \n        tf.keras.mixed_precision.set_global_policy(policy)\n\n        print('Compute dtype: %s' % policy.compute_dtype)\n        print('Variable dtype: %s' % policy.variable_dtype)\n\n    else:\n        pass\n        # I turned off the bfloat because it makes tf.keras.layers.experimental.preprocessing.RandomRotation was throw an exception\n        #policy = tf.keras.mixed_precision.Policy('mixed_bfloat16')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5: Adjust the batch size\n\nWhen using a TPU, you can usually multiply the batch size by the number of processors on the TPU.\nTPUs also often have a lot of memory, so you can sometimes cache the dataset to speed things up."},{"metadata":{"trusted":true},"cell_type":"code","source":"if USE_TPU:\n    # The TPU has a ton of memory, so it can usually handle dataset caching and 1 batch per TPU core\n    BATCH_SIZE = BATCH_SIZE * strategy.num_replicas_in_sync \n    ENABLE_CACHE = True\nelse:\n    ENABLE_CACHE = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 6: Create a function that loads THE TFRecords into a tf.data.Dataset\n\n\nhttps://www.tensorflow.org/guide/data\n\nTFRecord files can be loaded into a tf.data.Dataset.\nThe dataset's map() method can then be used to preprocess the data.  https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map\n\n\n\n\nDatasets can  can be fed directly into your model's model.fit().  They can also be \"optimized\" to speed up the training process."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_RANZCR_datasets(\n    TRAINING_DATA_FOLDER ,\n    DATASET_SIZE = 30083,  # I looked this up in windows explorer\n    val_size = 0.05,  # value from 0 to 1 representing the percent of the data put in the validation data set\n    ENABLE_CACHE = True,\n    INFINITE_DATASET = False,\n    USE_TPU = True,\n    USE_GCS = False, \n    IMAGE_SIZE = 300,\n    BATCH_SIZE = 64,\n    SHUFFLE_DATASET = True,\n    KEEP_ASPECT_RATIO = True  # keeps the ratio or height to width, by padding with zeros where required\n    ):\n\n    print(USE_GCS)\n\n    # If these files can't be read, grant read/write to your gmail address on the google cloud storage bucket\n    if USE_GCS:\n        train_filenames = tf.io.gfile.glob(TRAINING_DATA_FOLDER + '*.tfrec')\n        print(\"train filenames\", train_filenames)\n    else:\n        train_filenames = glob.glob(TRAINING_DATA_FOLDER + '*.tfrec')\n\n\n\n    train_size = int((1.0-val_size) * DATASET_SIZE)\n    val_size = int(val_size * DATASET_SIZE)\n\n    # Create a training and a validation datasets\n    full_dataset = tf.data.TFRecordDataset(\n        train_filenames, num_parallel_reads=tf.data.experimental.AUTOTUNE\n    )\n\n    full_dataset = full_dataset.shuffle(buffer_size=31000, seed=37)\n\n    train_dataset = full_dataset.take(train_size)\n    val_dataset = full_dataset.skip(train_size).take(val_size)\n\n\n    # You can enable this if you want to see what a raw record looks like\n    # This can help you set up your feature dictionary\n    if False:\n        for raw_record in train_dataset.take(1):\n            example = tf.train.Example()\n            example.ParseFromString(raw_record.numpy())\n            print(example)\n    \n    \n    \n    \n    #print(\"Size of Training Dataset: \", len(list(train_dataset)))\n    #print(\"Size of Validation Dataset: \", len(list(val_dataset)))\n\n    if ENABLE_CACHE and not USE_TPU:\n      train_dataset = train_dataset.cache()\n      val_dataset = val_dataset.cache()\n\n\n\n\n\n\n\n\n\n\n\n\n    # The feature dictionary should describe the data stored in the TFRecord For more details, \n    # see: https://www.tensorflow.org/tutorials/load_data/tfrecord#read_the_tfrecord_file\n    feature_dictionary = {\n        \"StudyInstanceUID\": tf.io.FixedLenFeature([], tf.string),\n        \"ETT - Abnormal\": tf.io.FixedLenFeature([], tf.int64),\n        \"ETT - Borderline\": tf.io.FixedLenFeature([], tf.int64),\n        \"ETT - Normal\": tf.io.FixedLenFeature([], tf.int64),\n        \"NGT - Abnormal\": tf.io.FixedLenFeature([], tf.int64),\n        \"NGT - Borderline\": tf.io.FixedLenFeature([], tf.int64),\n        \"NGT - Incompletely Imaged\": tf.io.FixedLenFeature([], tf.int64),\n        \"NGT - Normal\": tf.io.FixedLenFeature([], tf.int64),\n        \"CVC - Abnormal\": tf.io.FixedLenFeature([], tf.int64),\n        \"CVC - Borderline\": tf.io.FixedLenFeature([], tf.int64),\n        \"CVC - Normal\": tf.io.FixedLenFeature([], tf.int64),\n        \"Swan Ganz Catheter Present\": tf.io.FixedLenFeature([], tf.int64),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n    }\n\n\n    # We need to parse the TFRecords into feature, label pairs.\n    # The \"features\" are the data you are using to make your prediction (i.e. the medical image)\n    # The \"labels\" are what you are trying to predict.\n    # We want to transform the TFRecord feature (a string of bytes) back into an image.  \n    # Then we want to decode the jpeg image data into a 3-color-channel array, and change it's size.\n    # For our labels, we just want to turn them into one long list of zeros and ones.\n    \n    # Define the first parsing functions that will turn the TFRecord back into an array and a label\n    def _parse_function(example, feature_dictionary=feature_dictionary):\n        # Parse the input `tf.train.Example` proto using the feature_dictionary.\n        # Create a description of the features.\n        parsed_example = tf.io.parse_example(example, feature_dictionary)\n        return parsed_example\n\n\n    train_dataset = train_dataset.map(\n        _parse_function, num_parallel_calls=tf.data.experimental.AUTOTUNE\n    )\n    val_dataset = val_dataset.map(\n        _parse_function, num_parallel_calls=tf.data.experimental.AUTOTUNE\n    )\n\n\n\n\n    # Define the second parsing functions that will turn the TFRecord back into an array and a label\n    def generate_training_example(example):\n        new_image_size = (IMAGE_SIZE, IMAGE_SIZE)\n\n        # Convert the image to an ndarray, resize it and convert it to RGB color\n        # These are the settings most commonly required by base models used in transfer learning.\n\n\n        features = tf.image.decode_jpeg(example[\"image\"], channels=3)  \n\n\n        if KEEP_ASPECT_RATIO:\n            features = tf.image.resize_with_pad(image=features,\n                                                target_height=IMAGE_SIZE,\n                                                target_width=IMAGE_SIZE,\n                                                method=tf.image.ResizeMethod.BILINEAR,\n                                                antialias=False\n                                                )\n        else:\n            features = tf.image.resize(features, size=new_image_size)\n\n\n\n\n\n\n        labels = [  # Edit this to add whatever labels you want your model to predict\n            example[\"ETT - Abnormal\"],\n            example[\"ETT - Borderline\"],\n            example[\"ETT - Normal\"],\n            example[\"NGT - Abnormal\"],\n            example[\"NGT - Borderline\"],\n            example[\"NGT - Incompletely Imaged\"],\n            example[\"NGT - Normal\"],\n            example[\"CVC - Abnormal\"],\n            example[\"CVC - Borderline\"],\n            example[\"CVC - Normal\"],\n            example[\"Swan Ganz Catheter Present\"],\n        ]\n\n        features = tf.cast(features, tf.float32)\n        labels = tf.cast(labels, tf.float32)\n\n\n        return features, labels\n\n\n    train_dataset = train_dataset.map(\n        generate_training_example, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    val_dataset = val_dataset.map(\n        generate_training_example, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n\n\n\n\n#    if ENABLE_CACHE and USE_TPU:\n#        # TPU stores cache in RAM\n#        train_dataset = train_dataset.cache()\n#        val_dataset = val_dataset.cache()\n#    else:\n#        # GPU has less memory, so we store the cache on the disk\n#        train_dataset = train_dataset.cache(\"/content/spiro_dataset_cache_train/\")\n#        val_dataset = val_dataset.cache(\"/content/spiro_dataset_cache_val/\")\n        \n    if ENABLE_CACHE and USE_TPU:\n        train_dataset = train_dataset.cache()\n        val_dataset = val_dataset.cache()\n\n\n    # I don't know if shuffling in required.  It might cause RAM to run out or pre-processing to go very slow\n    if SHUFFLE_DATASET:\n        train_dataset = train_dataset.shuffle(buffer_size=256, seed=37)\n\n\n\n    if INFINITE_DATASET:\n        train_dataset = train_dataset.repeat()\n\n\n    # Set the batch size before applying the data augmentation\n    # drop_remainder=True prevents a smaller, last batch.  smaller batches could cause training issues on the TPU (i.e. cause the loss to go to NaN)\n    train_dataset = train_dataset.batch(BATCH_SIZE)   #, drop_remainder=True)\n    val_dataset = val_dataset.batch(BATCH_SIZE)\n\n\n\n\n#    # Apply data augmentation to the training data set\n#    if DATA_AUGMENTATION:\n#        data_augmentation = tf.keras.Sequential([\n#        tf.keras.Input(shape=(IMAGE_SIZE, IMAGE_SIZE, 3)),                        \n#        # Randomly pick about 83% of the area of the image (1/(1.1^2))                                       \n#        #tf.keras.layers.experimental.preprocessing.Resizing(height=int(1.1*IMAGE_SIZE), width=int(1.1*IMAGE_SIZE+100), interpolation='bilinear'),\n#        #tf.keras.layers.experimental.preprocessing.RandomCrop(height=IMAGE_SIZE, width=IMAGE_SIZE),\n#\n#        tf.keras.layers.experimental.preprocessing.RandomContrast(factor=0.1 ),\n#\n#        tf.keras.layers.experimental.preprocessing.RandomFlip(),\n#        tf.keras.layers.experimental.preprocessing.RandomRotation(factor=(-0.2, 0.2), fill_mode='constant'),\n#        ], name=\"Data_Augmentation\")\n#        \n#        train_dataset = train_dataset.map(lambda x, y: (data_augmentation(x, training=True), y), num_parallel_calls=tf.data.experimental.AUTOTUNE)\n\n\n    # Speeds up the pipeline\n    train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    val_dataset = val_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n\n\n    return train_dataset, val_dataset\n\n\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3:  Create training and validation data sets"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(training_data_folder)\ntrain_dataset, val_dataset = create_RANZCR_datasets( TRAINING_DATA_FOLDER=training_data_folder,\n                                                     DATASET_SIZE = 30083,  # I looked this up in windows explorer\n                                                     val_size = 0.10,  # value from 0 to 1 representing the percent of the data put in the validation data set\n                                                     IMAGE_SIZE=DATASET_IMAGE_SIZE,\n                                                     ENABLE_CACHE = True,\n                                                     INFINITE_DATASET = False,\n                                                     USE_TPU = USE_TPU,\n                                                     USE_GCS = USE_GCS,\n                                                     BATCH_SIZE = BATCH_SIZE,\n                                                     SHUFFLE_DATASET = True,\n                                                     )\n\n\n\n\n\nprint(train_dataset.take(1))\nprint('-------------------')\nfor X in train_dataset.take(2):\n    print(X[0][0])\n    print('-------------------')\n    print(X[1][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# List of all the Keras.applications models\n# Comment out the ones you don't want to train\n\n# This sets the weights for customized_efficientnet to avoid the exploding gradient problemt\ninitializer = tf.keras.initializers.HeNormal(seed=42)\n\n\n\n    # base model name, Layer to start unfreezing at (None means frozen model, 0 mean train all layers), Keras Model instance, a preprocessing function\nKeras_Models = [\n#                [\"DenseNet121\", 0, tf.keras.applications.DenseNet121, tf.keras.applications.densenet.preprocess_input, {'weights':'imagenet', 'include_top':False, 'default_size' : 224,}],\n#                [\"DenseNet169\", 0, tf.keras.applications.DenseNet169, tf.keras.applications.densenet.preprocess_input, {'weights':'imagenet', 'include_top':False, 'default_size' : 224,}],\n#                [\"DenseNet201\", 0, tf.keras.applications.DenseNet201, tf.keras.applications.densenet.preprocess_input, {'weights':'imagenet', 'include_top':False, 'default_size' : 224,}],                \n#                [\"EfficientNetB0\", 0, tf.keras.applications.EfficientNetB0, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 224,}],\n                [\"EfficientNetB1\", 0, tf.keras.applications.EfficientNetB1, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 240,}],\n#                [\"EfficientNetB2\", 0, tf.keras.applications.EfficientNetB2, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 260,}],\n#                [\"EfficientNetB3\", 0, tf.keras.applications.EfficientNetB3, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 300,}],\n#                [\"EfficientNetB4\", 0, tf.keras.applications.EfficientNetB4, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 380,}],\n#                [\"EfficientNetB5\", 0, tf.keras.applications.EfficientNetB5, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 456,}],\n#                [\"EfficientNetB6\", 0, tf.keras.applications.EfficientNetB6, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 528,}],\n#                [\"EfficientNetB7\", 0, tf.keras.applications.EfficientNetB7, tf.keras.applications.efficientnet.preprocess_input, {'weights':'imagenet', 'include_top':False,'drop_connect_rate':0.7, 'default_size' : 600,}],         \n#                [\"InceptionResNetV2\", 0, tf.keras.applications.InceptionResNetV2, tf.keras.applications.inception_resnet_v2.preprocess_input, {'weights':'imagenet', 'include_top':False,}], \n#                [\"InceptionV3\", 0, tf.keras.applications.InceptionV3, tf.keras.applications.inception_v3.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"MobileNet\", 0, tf.keras.applications.MobileNet, tf.keras.applications.mobilenet.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"MobileNetV2\", 0, tf.keras.applications.MobileNetV2, tf.keras.applications.mobilenet_v2.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"MobileNetV3Large\", 0, tf.keras.applications.MobileNetV3Large, tf.keras.applications.mobilenet_v3.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"MobileNetV3Small\", 0, tf.keras.applications.MobileNetV3Small, tf.keras.applications.mobilenet_v3.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"NASNetLarge\", 0, tf.keras.applications.NASNetLarge, tf.keras.applications.nasnet.preprocess_input, {'weights':'imagenet', 'include_top':False, 'default_size' : 331,}],\n#                [\"NASNetMobile\", 0, tf.keras.applications.NASNetMobile, tf.keras.applications.nasnet.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet50\", 0, tf.keras.applications.ResNet50, tf.keras.applications.resnet.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet101\", 0, tf.keras.applications.ResNet101, tf.keras.applications.resnet.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet152\", 0, tf.keras.applications.ResNet152, tf.keras.applications.resnet.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet50V2\", 0, tf.keras.applications.ResNet50V2, tf.keras.applications.resnet_v2.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet101V2\", 0, tf.keras.applications.ResNet101V2, tf.keras.applications.resnet_v2.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"ResNet152V2\", 0, tf.keras.applications.ResNet152V2, tf.keras.applications.resnet_v2.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"VGG16\", 0, tf.keras.applications.VGG16, tf.keras.applications.vgg16.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"VGG19\", 0, tf.keras.applications.VGG19, tf.keras.applications.vgg19.preprocess_input, {'weights':'imagenet', 'include_top':False,}],\n#                [\"Xception\", 0, tf.keras.applications.Xception, tf.keras.applications.xception.preprocess_input, {'weights':'imagenet', 'include_top':False,}],                                                                                              \n               ]\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(base_model_name, Start_unfreezing_at_layer, base_model, preprocessing_function, model_arguments, Save_Folder, IMAGE_SIZE):\n    \"\"\"Input is the name of a model, output is a dictionary with the trained modesl name and score\"\"\"\n\n\n\n    # Keras manages a global state, which it uses to implement the \n    # Functional model-building API and to uniquify autogenerated layer \n    # names. If you are creating many models in a loop, this global state \n    # will  consume an increasing amount of memory over time, and you may \n    # want to  clear it. Calling clear_session() releases the global state:  \n    # this helps avoid clutter from old models and layers, especially when  \n    # memory is limited.\n    #\n    # With `clear_session()` called at the beginning,\n    # Keras starts with a blank state at each iteration\n    # and memory consumption is constant over time.\n    # https://www.tensorflow.org/api_docs/python/tf/keras/backend/clear_session\n    tf.keras.backend.clear_session()\n\n    print(\"Base Model Name: \", base_model_name)\n    print(\"Start Unfreezing at Layer: \", Start_unfreezing_at_layer)\n    print(\"Base Model: \", base_model)\n    print(\"Preprocessing Function: \", preprocessing_function)\n    print(\"Batch Size: \", BATCH_SIZE)\n\n    \n#    with strategy.scope():\n    tf.keras.backend.clear_session()\n\n    n_labels = 11\n    #auc = tf.keras.metrics.AUC(multi_label=True)   \n\n\n    # if the model has a default size, use that.  otherwise use the IMAGE_SIZE set at the top\n    if model_arguments.get('default_size'):\n        model_image_size = model_arguments.get('default_size')\n        del model_arguments['default_size']\n    else:\n        model_image_size = IMAGE_SIZE\n\n\n\n\n\n    # Function for decaying the learning rate.  Need to be inside the strategy.scope()\n    # https://stackoverflow.com/questions/56542778/what-has-to-be-inside-tf-distribute-strategy-scope\n    lr_reducer = tf.keras.callbacks.ReduceLROnPlateau(\n        monitor=\"val_auc\", factor=0.2, patience=3, min_lr=1e-9, mode='max',verbose=1)\n\n\n\n\n    def learning_rate_schedule(epoch, lrate):\n        if epoch<5:\n            lrate = 0.01\n        elif epoch<10:\n            lrate = 0.01\n        else:\n            lrate = 0.00001\n        return lrate\n\n    learning_rate_schedule_callback = tf.keras.callbacks.LearningRateScheduler(learning_rate_schedule)\n\n\n\n\n\n    EarlyStopping_callback = tf.keras.callbacks.EarlyStopping(\n        monitor='val_auc', \n        min_delta=0.0001, \n        patience=10, \n        verbose=1,\n        mode='max', \n        baseline=None, \n        restore_best_weights=True\n    )\n\n\n\n\n\n#        ModelCheckpoint_folder = CHECKPOINTS_FOLDER +base_model_name+\"_ImageSize_\" + str(model_image_size) + \"/Checkpoint_epoch_{epoch}_auc_{val_auc}.hdf5\"  \n#        print(\"Checkpoint files: \", ModelCheckpoint_folder)      \n#        ModelCheckpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n#                                            filepath = ModelCheckpoint_folder,            \n#                                            monitor='val_auc', \n#                                            verbose=1, \n#                                            save_best_only=True,\n#                                            save_weights_only=False, \n#                                            mode='max', \n#                                            save_freq='epoch',\n#                                            options=save_locally\n#        )\n\n\n\n\n\n    # https://blog.tensorflow.org/2020/04/introducing-new-tensorflow-profiler.html\n#        tensorboard_callback = tf.keras.callbacks.TensorBoard(  log_dir = LOG_DIR,\n#                                                                profile_batch = '2,7')\n\n\n\n\n\n\n#        inputs = tf.keras.layers.Input([IMAGE_SIZE, IMAGE_SIZE, 3])\n   # inputs = tf.cast(inputs, tf.float32)\n#        x = tf.keras.applications.mobilenet.preprocess_input(x)\n    if True:\n        # This is my functional code that uses the model from the list above\n        inputs = tf.keras.Input(shape=(DATASET_IMAGE_SIZE, DATASET_IMAGE_SIZE, 3))\n\n\n\n\n        resized_inputs = tf.keras.layers.experimental.preprocessing.Resizing(height=model_image_size, width=model_image_size, interpolation='bilinear', name='resize_the_inputs')(inputs)\n        preprocessed_inputs = preprocessing_function(resized_inputs)\n\n        # within-model data augmentation\n        #preprocessed_inputs = tf.keras.layers.experimental.preprocessing.RandomFlip()(preprocessed_inputs)\n        preprocessed_inputs = tf.keras.layers.experimental.preprocessing.RandomRotation(factor=(-0.2, 0.2), fill_mode='constant')(preprocessed_inputs)\n        preprocessed_inputs = tf.keras.layers.experimental.preprocessing.RandomContrast(factor=0.2 )(preprocessed_inputs)\n\n\n\n\n        # Create an instance of the Base Model\n        base_model=base_model( **model_arguments)\n\n\n        if Start_unfreezing_at_layer is None:\n            # Freeze the base model\n            for layer in base_model.layers:\n                layer.trainable = False\n        else:\n            # finetuning, unfreeze the top layer of the model\n            # If you change the model or want to change the layers that are unfrozen,\n            # change the variable Start_unfreezing_at_layer.\n            for i, layer in enumerate(base_model.layers):\n                if i>=Start_unfreezing_at_layer:\n                    layer.trainable=True\n                else:\n                    layer.trainable=False\n\n\n\n\n        x = base_model(preprocessed_inputs)\n\n\n        if base_model_name==\"EfficientNet_Custom_with_top\" or base_model_name==\"NFNetF0\":\n            outputs = x\n        else:\n            # Add my own top\n            # Convert features of shape `base_model.output_shape[1:]` to vectors\n            x = tf.keras.layers.GlobalAveragePooling2D()(x)\n            # A Dense classifier with a single unit (binary classification)\n            outputs = tf.keras.layers.Dense(n_labels, activation='sigmoid')(x)\n\n        model = tf.keras.Model([inputs], [outputs], name=base_model_name)\n\n\n    else:\n        #I've confirmed that this model is identical to the functional model with Start_unfreezing_at_layer=0\n        # Testing the model from:\n        # https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950\n        # https://www.kaggle.com/xhlulu/ranzcr-efficientnet-gpu-starter-train-submit\n        # B2\t260\tVal_AUC:0.9206\t \tGPU\tImageNet\t\n        model = tf.keras.Sequential([\n            tf.keras.layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),                         \n            tf.keras.applications.EfficientNetB3(\n                input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),\n                weights='imagenet',\n                include_top=False,\n                drop_connect_rate=0.5),\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(n_labels, activation='sigmoid')\n            ])\n\n    USE_CUSTOM_LOSS_FUNCTION = False\n    if USE_CUSTOM_LOSS_FUNCTION:\n        from custom_loss_function import custom_loss_fn\n        print(\"Using a custom loss function from an external module!\")\n        Loss_Function = custom_loss_fn\n    else:\n        Loss_Function = 'binary_crossentropy'\n\n\n    LOAD_WEIGHTS_FROM_CHECKPOINT = None\n    if LOAD_WEIGHTS_FROM_CHECKPOINT:\n        # Loads the weights\n        model.load_weights(LOAD_WEIGHTS_FROM_CHECKPOINT)\n\n    # use gradient clipping to prevent loss from going to nan\n    # Good default values are clipnorm=1.0 or clipvalue=0.5.\n    Custom_Optimizer = tf.keras.optimizers.Nadam(learning_rate=LEARNING_RATE, clipnorm=1.0)\n    #Custom_Optimizer = SGD_AGC(lr=LEARNING_RATE)\n\n    model.compile(\n        optimizer=Custom_Optimizer,\n        loss=Loss_Function,\n        metrics=[tf.keras.metrics.AUC(multi_label=True)]\n        )\n\n    print(model.summary())\n\n\n\n    history = model.fit(\n        train_dataset, \n        epochs=NUMBER_OF_EPOCHS,\n        steps_per_epoch=STEPS_PER_EPOCH,\n        callbacks=[\n                    lr_reducer,\n                    #learning_rate_schedule_callback,\n                    EarlyStopping_callback,\n                #ModelCheckpoint_callback,\n                   # tensorboard_callback,\n               ],\n        validation_data=val_dataset,\n    )\n\n\n\n    scores = model.evaluate(val_dataset)\n    print(\"Loss: \", scores[0])\n    print(\"Multi-label AUC: \", scores[1])\n\n\n#        SAVED_MODEL_FOLDER = \"/kaggle/working/\"\n#        filename = (SAVED_MODEL_FOLDER +\n#                    base_model_name + \n#                    \"_ImageSize_\" + str(model_image_size) + \n#                    \"_Start_unfreezing_at_layer_\" + str(Start_unfreezing_at_layer) + \n#                    \"_Epochs_\" +str(NUMBER_OF_EPOCHS) + \n#                    \"_Multi_label_AUC_\" + str(round(scores[1],5)) + \n#                    \".h5\")\n#        model.save(filename, save_format=\"h5\", options=save_locally)\n#        print(\"Saved the model to: \" + filename)\n\n\n\n\n    return [ {  #\"filename\":  filename,\n                \"Base Model\": base_model_name,\n                \"Start unfreezing at layer\": str(Start_unfreezing_at_layer),\n                \"Image Size\": model_image_size,\n                \"Initial Learning Rate\": LEARNING_RATE,\n                \"Batch Size\": BATCH_SIZE,\n                \"Loss\": scores[0], \n                \"Multi-label AUC\": scores[1],         \n                }]\n\n\n\n\n\ndef Train_and_Save_Models(Keras_Models, Save_Folder, IMAGE_SIZE):\n    \"\"\"Takes in a list of save filenames and models.  Trains the models and then saves them\"\"\"\n    \n\n    #for_testing\n\n\n    results = []\n    for base_model_name, Start_unfreezing_at_layer, base_model, preprocessing_function, model_arguments in Keras_Models:\n\n        results.append(\n                        train_model(base_model_name, Start_unfreezing_at_layer, base_model, preprocessing_function, model_arguments, Save_Folder=\"/content/drive/MyDrive/ML_DataSets/Catheter_Dataset/Saved_Model/\", IMAGE_SIZE=IMAGE_SIZE)\n                        )\n\n    return results\n\ntry:\n    with strategy.scope():\n        Results = Train_and_Save_Models(Keras_Models, Save_Folder=\"/kaggle/working/\", IMAGE_SIZE=IMAGE_SIZE)\nexcept IndexError:\n    pass  # There was a \"pop from empty list\" error in \"tensorflow/python/distribute/distribution_strategy_context.py\" that I'm ignoring\n\nprint(\"Results:  \")\nimport pprint\npprint.pprint(Results)\nprint(\"-----------------------------------------\")\nprint()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}