{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## <font color = \"seagreen\">Abstraction</font>\n\n[**About Competition**]: In this competition, we’ll detect the presence and position of **catheters** and **lines** on chest x-rays. We need to use machine learning to train and test our model on **40,000** images to categorize a tube that is poorly placed.\n\n[**About Data**]: This is a code-only competition so there is a hidden test set (approximately 4x larger, with ~14k images) as well. And here is list of the given files information for this competition. \n\n    - train.csv - contains image IDs, binary labels, and patient IDs.\n    - sample_submission.csv - a sample submission file in the correct format\n    - test - test images\n    - train - training images\n    - train_annotations.csv - segmentation annotations (additional information for competitors).\n\nHere are the essential information from the meta data (`train.csv`). \n\n- `StudyInstanceUID` - unique ID for each image\n- `ETT - Abnormal` - endotracheal tube placement abnormal\n- `ETT - Borderline` - endotracheal tube placement borderline abnormal\n- `ETT - Normal` - endotracheal tube placement normal\n- `NGT - Abnormal` - nasogastric tube placement abnormal\n- `NGT - Borderline` - nasogastric tube placement borderline abnormal\n- `NGT - Incompletely Imaged` - nasogastric tube placement inconclusive due to imaging\n- `NGT - Normal` - nasogastric tube placement borderline normal\n- `CVC - Abnormal` - central venous catheter placement abnormal\n- `CVC - Borderline` - central venous catheter placement borderline abnormal\n- `CVC - Normal` - central venous catheter placement normal\n- `Swan Ganz Catheter Present`\n- `PatientID` - unique ID for each patient in the dataset\n\nIt's a **multi-label** classification problem. Our target labels are from `ETT - Abnormal` to `Swan Ganz Catheter Present`. The submissions are evaluated on **Area Under the ROC** curve between the predicted probability and the observed target. To calculate the final score, **AUC** is calculated for each of the **11** labels, then averaged. The score is then the average of the individual **AUCs** of each predicted column.","metadata":{}},{"cell_type":"markdown","source":"## <font color = \"seagreen\">Content of Notebook</font>\n\nHere is the content of this code examples. It is configured to run on **TPU** as well as **GPU** hardware with [**mixed precision**](https://www.tensorflow.org/guide/mixed_precision) training. It's also configured to work on **[JPEG](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/data)** samples as well as **[TF-Record](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/data)** format - both option are provided. To know about **TPU** and TF-Record, please refer to [this tpu-guidebook](https://www.kaggle.com/docs/tpu), hosted on Kaggle and [tfrecod-example](https://www.tensorflow.org/tutorials/load_data/tfrecord#:~:text=The%20TFRecord%20format%20is%20a,to%20understand%20a%20message%20type.), hosted on TensorFlow.org.\n\nAlong with the above **device set-up** and **data** specific configuration, we've also tried to develop a research idea, refer to the **multi-branch soft attention modules**. We've tried to implement it in `tf.keras` with model [subclassing API](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) and tried to inspect it with **GradCAM** at the end of epoch using **Callback** function. The approach **Multi-Attention** presented in this notebook is a simple research idea. It can provide ideas or encourage. Practically. it may need extensive experiments and careful integration to the imagenet models or custom models. In this notebook, for demonstration purposes, we have simply tried to add it as a custom neck to the pre-trained model. In a result, this gives promising outputs for this task. More theoretical and mathematical informations about it, can be found in the relevant section in this notebook. Here is the summary:\n\n```\n1 Clean Set-Up for Training (on TPU) and Inference (on GPU).\n2 Data Modeling  \n    - Data preprocessing : tf.data API with \n        - JPEG and, \n        - TF-RECORD\n    - Data augmentation : tf.image modules\n3 Hyperparameter Setting:\n    - Warmup Learning Rate Schedule (exponential, cosine, constant, cosine-restart).\n4 Deep Network Modeling\n    - Backbone: EfficientNet  \n    - Neck: Custom Top Layers (Multi-Attention Mechanisms)\n    - Head: Sigmoid Classifier (Multi-label)\n5 Training\n    - GradCAM set up\n    - Model training\n6 Inference\n```","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.cm as cm\nimport matplotlib.pyplot as plt \nfrom kaggle_datasets import KaggleDatasets\n\nimport os, warnings\nfrom sklearn import model_selection\nwarnings.simplefilter('ignore')\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n\nimport tensorflow as tf; print(tf.__version__)\nfrom tensorflow import keras; print(keras.__version__)\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.optimizers import schedules","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-04T15:25:32.205551Z","iopub.execute_input":"2022-04-04T15:25:32.205918Z","iopub.status.idle":"2022-04-04T15:25:32.214800Z","shell.execute_reply.started":"2022-04-04T15:25:32.205879Z","shell.execute_reply":"2022-04-04T15:25:32.213985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Device Config\n\nAs mentioned, we can use the host provided `jpeg` format files or we can use host provided `tfrecord` format files. Also, this notebook is configured to train the model on **TPU** and inference on **GPU**. But feel free to pick and set option which you prefer most. \n\nI've trained the model in **TPU + JPEG** and saved the best weight. But for demonstration purpose, I will train the model again with **TPU + TF-TFRCORD** configs. Please read also the comment that attach with the code in the following cells to understand the whole set up precisely. ","metadata":{}},{"cell_type":"code","source":"# set: 'TF_RECORD' for using .tfrec format \n# set: 'JPEG' for using jpeg format\nFILE_TYPE = 'TF_RECORD' # OPTION: 'JPEG', 'TF_RECORD'    \n\nMIXED_PRECISION = False  # Faster and use Less memory\nXLA_ACCELERATE  = False # XLA: Optimizing Compiler\n\ntry:  # detect TPUs\n    tpu      = tf.distribute.cluster_resolver.TPUClusterResolver.connect()  # TPU detection\n    strategy = tf.distribute.TPUStrategy(tpu)\n    DEVICE   = 'TPU'\nexcept ValueError:  # detect GPUs\n    strategy = tf.distribute.get_strategy() # default strategy operates on CPU and GPU\n    DEVICE   = 'GPU'\n    \nif DEVICE == \"GPU\":\n    physical_devices = tf.config.list_physical_devices('GPU')\n    print(\"Num GPUs Available: \", len(physical_devices))\n    try: \n        tf.config.experimental.set_memory_growth(physical_devices[0], True)\n        assert tf.config.experimental.get_memory_growth(physical_devices[0])\n    except: # Invalid device or cannot modify virtual devices once initialized.\n        pass \n    \nif MIXED_PRECISION:\n    dtype = 'mixed_bfloat16' if DEVICE == \"TPU\" else 'mixed_float16'\n    tf.keras.mixed_precision.set_global_policy(dtype)\n    print('Mixed precision enabled')\n\nif XLA_ACCELERATE:\n    tf.config.optimizer.set_jit(True)\n    print('Accelerated Linear Algebra enabled')\n    \nAUTO  = tf.data.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\n\nprint('REPLICAS           : ', REPLICAS)\nprint('Eager Mode Status  : ', tf.executing_eagerly())\nprint('TF Cuda Built Test : ', tf.test.is_built_with_cuda)\nprint('TF Device Detected : ', 'Running on TPU' if DEVICE == \"TPU\" else tf.test.gpu_device_name())\n\ntry:\n    print('TF System Cuda V.  : ', tf.sysconfig.get_build_info()[\"cuda_version\"])\n    print('TF System CudNN V. : ', tf.sysconfig.get_build_info()[\"cudnn_version\"])\nexcept:\n    pass","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-04-04T15:25:32.264051Z","iopub.execute_input":"2022-04-04T15:25:32.264364Z","iopub.status.idle":"2022-04-04T15:25:37.649213Z","shell.execute_reply.started":"2022-04-04T15:25:32.264334Z","shell.execute_reply":"2022-04-04T15:25:37.648247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Base Config**","metadata":{}},{"cell_type":"code","source":"ROOT_DIR  = \"ranzcr-clip-catheter-line-classification\"\nTRAIN_DF  = pd.read_csv(f\"/kaggle/input/{ROOT_DIR}/\" + 'train.csv')\nSUBMIT    = pd.read_csv(f\"/kaggle/input/{ROOT_DIR}/\" + 'sample_submission.csv')\nCLASS_MAP = pd.read_csv(f\"/kaggle/input/{ROOT_DIR}/\" + 'train_annotations.csv')\nTARGET    = TRAIN_DF[SUBMIT.columns[1:]].values\ndisplay(TRAIN_DF.head())\n\nif DEVICE == \"TPU\":\n    GCS_DS_PATH    = KaggleDatasets().get_gcs_path(ROOT_DIR)\n    if FILE_TYPE == 'TF_RECORD':\n        TRAIN_IMG_PATH = sorted(tf.io.gfile.glob(GCS_DS_PATH + '/train_tfrecords/*.tfrec'))\n        TEST_IMG_PATH  = sorted(tf.io.gfile.glob(GCS_DS_PATH + '/test_tfrecords/*.tfrec'))\n    elif FILE_TYPE == 'JPEG':\n        TRAIN_IMG_PATH = GCS_DS_PATH + \"/train/\" + TRAIN_DF['StudyInstanceUID'] + '.jpg'\n        TEST_IMG_PATH  = GCS_DS_PATH + \"/test/\"  + SUBMIT['StudyInstanceUID']   + '.jpg'\nelse:\n    if FILE_TYPE == 'TF_RECORD':\n        TRAIN_IMG_PATH = sorted(tf.io.gfile.glob(f\"/kaggle/input/{ROOT_DIR}\" + '/train_tfrecords/*.tfrec'))\n        TEST_IMG_PATH  = sorted(tf.io.gfile.glob(f\"/kaggle/input/{ROOT_DIR}\" + '/test_tfrecords/*.tfrec'))\n    elif FILE_TYPE == 'JPEG':\n        TRAIN_IMG_PATH = f\"/kaggle/input/{ROOT_DIR}\" + \"/train/\" + TRAIN_DF['StudyInstanceUID'] + '.jpg'\n        TEST_IMG_PATH  = f\"/kaggle/input/{ROOT_DIR}\" + \"/test/\"  + SUBMIT['StudyInstanceUID']   + '.jpg'","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:37.651507Z","iopub.execute_input":"2022-04-04T15:25:37.651831Z","iopub.status.idle":"2022-04-04T15:25:38.439109Z","shell.execute_reply.started":"2022-04-04T15:25:37.651780Z","shell.execute_reply":"2022-04-04T15:25:38.438138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <font color = \"seagreen\">Param</font>","metadata":{}},{"cell_type":"code","source":"# training params \nEPOCHS       = 20\nVERBOSITY    = 1\nLABEL_SMOOTH = 0.01\n\n# data params \nBATCH_SIZE   = REPLICAS * 8\nIMG_SIZE     = 512\nSEED         = 101\n\n# model params \nBASE_NETS    = tf.keras.applications.EfficientNetB5","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.440940Z","iopub.execute_input":"2022-04-04T15:25:38.441180Z","iopub.status.idle":"2022-04-04T15:25:38.446297Z","shell.execute_reply.started":"2022-04-04T15:25:38.441154Z","shell.execute_reply":"2022-04-04T15:25:38.445465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <font color = \"seagreen\">RANZCR-CLiP Dataloader - tf.data API: JPEG and TFRecord</font> ","metadata":{}},{"cell_type":"code","source":"def get_target_size(target_size):\n    if isinstance(target_size, int):\n        return (target_size, target_size)\n    if isinstance(target_size, list):\n        return target_size\n    raise ValueError('target_size must be an int, or (height, width) but got %r' % target_size)\n\n@tf.function\ndef augment_image(img):\n    img = tf.image.random_flip_left_right(img)\n    img = tf.image.random_flip_up_down(img)\n    return img\n\n@tf.function\ndef read_image(file_bytes, target_size):\n    img = tf.image.decode_png(file_bytes, channels=3)\n    img = tf.image.resize(img, get_target_size(target_size))\n    return img ","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.448401Z","iopub.execute_input":"2022-04-04T15:25:38.448680Z","iopub.status.idle":"2022-04-04T15:25:38.459867Z","shell.execute_reply.started":"2022-04-04T15:25:38.448645Z","shell.execute_reply":"2022-04-04T15:25:38.459132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**JPEG Builder --- Option 1**","metadata":{}},{"cell_type":"code","source":"# ensure the image format \ndef get_target_size(target_size):\n    if isinstance(target_size, int):\n        return (target_size, target_size)\n    if isinstance(target_size, tuple):\n        return target_size\n    raise ValueError('target_size must be an int, or (height, width) but got %r' % image_size)\n\n# read image and preprocessing \ndef build_decoder(with_labels=True, target_size=(256, 256)):\n    def decode(path):\n        file_bytes = tf.io.read_file(path)\n        return read_image(file_bytes, target_size=target_size)\n    def decode_with_labels(path, label):\n        return decode(path), label\n    return decode_with_labels if with_labels else decode\n\n# data augmentation \ndef build_augmenter(with_labels=True):\n    def augment_with_labels(img, label):\n        return augment_image(img), label\n    return augment_with_labels if with_labels else augment_image","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-04-04T15:25:38.461173Z","iopub.execute_input":"2022-04-04T15:25:38.461454Z","iopub.status.idle":"2022-04-04T15:25:38.478239Z","shell.execute_reply.started":"2022-04-04T15:25:38.461424Z","shell.execute_reply":"2022-04-04T15:25:38.477355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# bind all the previous cell functions\ndef jpeg_loader(paths,\n                labels     = None, \n                bsize      = 32, \n                cache      = True,\n                decode_fn  = None, \n                augment_fn = None,\n                augment    = True, \n                repeat     = True, \n                shuffle    = 1024, \n                cache_dir  = \"\"):\n    if cache_dir != \"\" and cache is True:\n        os.makedirs(cache_dir, exist_ok=True)\n    \n    if decode_fn is None:\n        decode_fn = build_decoder(labels is not None)\n    \n    if augment_fn is None:\n        augment_fn = build_augmenter(labels is not None)\n    \n    slices = paths if labels is None else (paths, labels)\n    dataset = tf.data.Dataset.from_tensor_slices(slices)\n    dataset = dataset.map(decode_fn, num_parallel_calls=AUTO)\n    dataset = dataset.cache(cache_dir) if cache else dset\n    dataset = dataset.map(augment_fn, num_parallel_calls=AUTO) if augment else dataset\n    dataset = dataset.repeat() if repeat else dset\n    dataset = dataset.shuffle(8 * bsize) if shuffle else dset\n    dataset = dataset.batch(bsize).prefetch(AUTO)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.479893Z","iopub.execute_input":"2022-04-04T15:25:38.480361Z","iopub.status.idle":"2022-04-04T15:25:38.493246Z","shell.execute_reply.started":"2022-04-04T15:25:38.480310Z","shell.execute_reply":"2022-04-04T15:25:38.492383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**TFRecord Builder --- Option 2**","metadata":{}},{"cell_type":"code","source":"def fn_read_tfrecords(\n    tfrecords, \n    tfrecords_schema, \n    apply_augment = False, \n    target_size   = 256,\n    inference     = False\n):\n    read_tfrecord = tf.io.parse_single_example(tfrecords, tfrecords_schema)\n    x_train = read_image(read_tfrecord['image'], target_size=target_size)\n\n    if apply_augment:\n        x_train = augment_image(x_train)\n    \n    if not inference:\n        y_train = tf.stack([\n            read_tfrecord['ETT - Abnormal'],\n            read_tfrecord['ETT - Borderline'],\n            read_tfrecord['ETT - Normal'],\n            read_tfrecord[\"NGT - Abnormal\"],\n            read_tfrecord['NGT - Borderline'],\n            read_tfrecord['NGT - Incompletely Imaged'],\n            read_tfrecord['NGT - Normal'],\n            read_tfrecord['CVC - Abnormal'],\n            read_tfrecord['CVC - Borderline'],\n            read_tfrecord['CVC - Normal'],\n            read_tfrecord['Swan Ganz Catheter Present']\n        ], axis=-1)\n        \n        return x_train, y_train\n    else:\n        return x_train","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.494692Z","iopub.execute_input":"2022-04-04T15:25:38.495034Z","iopub.status.idle":"2022-04-04T15:25:38.510109Z","shell.execute_reply.started":"2022-04-04T15:25:38.494994Z","shell.execute_reply":"2022-04-04T15:25:38.509130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def tfrecords_loader(\n    files_path,\n    tfschemes    = None,\n    shuffle      = True,\n    cache        = True,\n    repeat       = True,\n    augment      = False,\n    ignore_order = True,\n    inference    = False\n):\n    dataset = tf.data.TFRecordDataset(\n        files_path, num_parallel_reads=AUTO\n    ) \n    \n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False # disable order, increase speed\n    dataset = dataset.with_options(ignore_order) if ignore_order else dataset\n    \n    dataset = dataset.map(\n        lambda x: fn_read_tfrecords(x, \n                                    tfrecords_schema,\n                                    apply_augment = augment, \n                                    target_size   = IMG_SIZE,\n                                    inference     = inference)\n    )\n    \n    dataset = dataset.repeat() if repeat else dataset\n    dataset = dataset.shuffle(\n        8 * BATCH_SIZE\n    )\n    dataset = dataset.batch(\n        BATCH_SIZE\n    )\n    dataset = dataset.prefetch(\n        buffer_size=AUTO\n    )\n    return dataset\n\n\ntfrecords_schema = {\n    'StudyInstanceUID'           : tf.io.FixedLenFeature([], tf.string),\n    'image'                      : tf.io.FixedLenFeature([], tf.string),\n    'ETT - Abnormal'             : tf.io.FixedLenFeature([], tf.int64),\n    'ETT - Borderline'           : tf.io.FixedLenFeature([], tf.int64),\n    'ETT - Normal'               : tf.io.FixedLenFeature([], tf.int64),\n    \"NGT - Abnormal\"             : tf.io.FixedLenFeature([], tf.int64),\n    'NGT - Borderline'           : tf.io.FixedLenFeature([], tf.int64),\n    'NGT - Incompletely Imaged'  : tf.io.FixedLenFeature([], tf.int64),\n    'NGT - Normal'               : tf.io.FixedLenFeature([], tf.int64),\n    'CVC - Abnormal'             : tf.io.FixedLenFeature([], tf.int64),\n    'CVC - Borderline'           : tf.io.FixedLenFeature([], tf.int64),\n    'CVC - Normal'               : tf.io.FixedLenFeature([], tf.int64),\n    'Swan Ganz Catheter Present' : tf.io.FixedLenFeature([], tf.int64)\n}","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.511616Z","iopub.execute_input":"2022-04-04T15:25:38.511985Z","iopub.status.idle":"2022-04-04T15:25:38.528542Z","shell.execute_reply.started":"2022-04-04T15:25:38.511942Z","shell.execute_reply":"2022-04-04T15:25:38.527509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <font color = \"seagreen\">Build Data Set</font>","metadata":{}},{"cell_type":"code","source":"if FILE_TYPE == 'TF_RECORD':\n    TRAIN_SET, VALID_SET = model_selection.train_test_split(TRAIN_IMG_PATH, test_size=0.2, random_state=SEED)\n\n    # The following two line may take a while to compute\n    train_set_len = sum(1 for record in tf.data.TFRecordDataset(TRAIN_SET))\n    valid_set_len = sum(1 for record in tf.data.TFRecordDataset(VALID_SET))\n\n    TRAIN_STEPS_PER_EPOCH = int(np.ceil(train_set_len / float(BATCH_SIZE)))\n    VALID_STEPS_PER_EPOCH = int(np.ceil(valid_set_len / float(BATCH_SIZE)))\n    \n    train_datasets = tfrecords_loader(\n        TRAIN_SET, \n        tfschemes    = tfrecords_schema,\n        shuffle      = True,\n        repeat       = True,\n        augment      = True,\n        ignore_order = False,\n        inference    = False\n    )\n    \n    valid_datasets = tfrecords_loader(\n        VALID_SET,\n        tfschemes    = tfrecords_schema,\n        shuffle      = False,\n        repeat       = False,\n        augment      = False,\n        ignore_order = True,\n        inference    = False\n    )\nelif FILE_TYPE == 'JPEG':\n    (\n        train_paths, valid_paths, \n        train_labels, valid_labels\n    ) = model_selection.train_test_split(TRAIN_IMG_PATH, TARGET, test_size=0.2, random_state=SEED)\n    \n    TRAIN_STEPS_PER_EPOCH = int(np.ceil(train_set_len / float(BATCH_SIZE)))\n    VALID_STEPS_PER_EPOCH = int(np.ceil(train_set_len / float(BATCH_SIZE)))\n    \n    decoder = build_decoder(with_labels=True, target_size=IMG_SIZE)\n    \n    train_datasets = jpeg_loader(\n        train_paths, \n        train_labels, \n        bsize     = BATCH_SIZE, \n        decode_fn = decoder,\n        repeat    = True, \n        shuffle   = True, \n        augment   = True\n    )\n\n    valid_datasets = jpeg_loader(\n        valid_paths, \n        valid_labels, \n        bsize     = BATCH_SIZE, \n        decode_fn = decoder,\n        repeat    = False, \n        shuffle   = False, \n        augment   = False\n    )","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:25:38.529973Z","iopub.execute_input":"2022-04-04T15:25:38.530196Z","iopub.status.idle":"2022-04-04T15:26:12.282005Z","shell.execute_reply.started":"2022-04-04T15:25:38.530171Z","shell.execute_reply":"2022-04-04T15:26:12.281058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train Data Visualization","metadata":{}},{"cell_type":"code","source":"image_batch, label_batch = next(iter(train_datasets))\nprint(image_batch.shape, label_batch.shape)\n\ndef show_batch(image_batch, label_batch=None, title=''):\n    fig = plt.figure(figsize=(15, 15))\n    plt.title(title)\n    plt.yticks([])\n    plt.xticks([])\n    \n    if DEVICE == \"TPU\":\n        xy = int(np.sqrt(image_batch.shape[0]))\n    else:\n        xy = image_batch.shape[0] // 2\n    for n in range(image_batch.shape[0]):\n        ax = fig.add_subplot(xy, xy, n + 1)\n        plt.imshow(image_batch[n] / 255.0)\n        plt.tight_layout()\n        plt.axis(\"off\")\n\nshow_batch(image_batch.numpy(), label_batch=label_batch.numpy(), title='Augmented Training Set')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:26:12.284645Z","iopub.execute_input":"2022-04-04T15:26:12.284927Z","iopub.status.idle":"2022-04-04T15:26:31.392219Z","shell.execute_reply.started":"2022-04-04T15:26:12.284896Z","shell.execute_reply":"2022-04-04T15:26:31.391257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Validation Data Visualization","metadata":{}},{"cell_type":"code","source":"image_batch, label_batch = next(iter(valid_datasets))\nprint(image_batch.shape, label_batch.shape)\nshow_batch(image_batch.numpy(), label_batch=label_batch.numpy(), title='Validation Set (No Augmentation)')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:26:31.393533Z","iopub.execute_input":"2022-04-04T15:26:31.393776Z","iopub.status.idle":"2022-04-04T15:26:49.437921Z","shell.execute_reply.started":"2022-04-04T15:26:31.393748Z","shell.execute_reply":"2022-04-04T15:26:49.436996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Modeling</font>\n\nHere, we've tried to integrate a **Multi-Attention** mechanism on the top of base model. In essence, we've added [**Convolutional Block Attention Module (CBAM)**](https://arxiv.org/abs/1807.06521) and [**DeepMoji**](https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py)'s attention mechanism in parallel and merge them at the end. Additionally, we've modified a bit of the **CBAM** mechanism. Also the end part of **spatial** module of **CBAN** by integrating the **Global Weighted Average Pooling (GWAP)** method, mathematically as follows. The idea of **GWAP** is inspired from [Dr. Kevin](https://www.kaggle.com/kmader)'s great work, [check](https://www.kaggle.com/kmader/attention-on-pretrained-vgg16-for-bone-age).\n\n\n$$ \\text{GWAP}(x, y, d) = \\frac{ \\sum\\limits_{x}\\sum\\limits_{y} \\text{Attention}(x,y,d) \\text{Feature}(x,y,d)} {\\sum\\limits_{x}\\sum\\limits_{y} \\text{Attention}(x,y,d)} $$\n\n[Attention Learning in CV (!ViT) - Details](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203083). ","metadata":{}},{"cell_type":"code","source":"class SpatialAttentionModule(keras.layers.Layer):\n    def __init__(self, kernel_size=3):\n        '''\n        paper: https://arxiv.org/abs/1807.06521\n        code: https://gist.github.com/innat/99888fa8065ecbf3ae2b297e5c10db70\n        '''\n        super().__init__()\n        self.conv1 = keras.layers.Conv2D(64, \n                                            kernel_size=kernel_size, \n                                            use_bias=False, \n                                            kernel_initializer='he_normal',\n                                            strides=1, padding='same', \n                                            activation=tf.nn.relu)\n        self.conv2 = keras.layers.Conv2D(32, kernel_size=kernel_size, \n                                            use_bias=False, \n                                            kernel_initializer='he_normal',\n                                            strides=1, padding='same', \n                                            activation=tf.nn.relu)\n        self.conv3 = keras.layers.Conv2D(16, kernel_size=kernel_size, \n                                            use_bias=False, \n                                            kernel_initializer='he_normal',\n                                            strides=1, padding='same', \n                                            activation=tf.nn.relu)\n        self.conv4 = keras.layers.Conv2D(1, \n                                            kernel_size=(1, 1),  \n                                            use_bias=False,\n                                            kernel_initializer='he_normal',\n                                            strides=1, padding='same', \n                                            activation=tf.math.sigmoid)\n\n    def call(self, inputs):\n        avg_out = tf.reduce_mean(inputs, axis=3)\n        max_out = tf.reduce_max(inputs,  axis=3)\n        x = tf.stack([avg_out, max_out], axis=3) \n        x = self.conv1(x)\n        x = self.conv2(x)\n        x = self.conv3(x)\n        return self.conv4(x)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:26:49.439472Z","iopub.execute_input":"2022-04-04T15:26:49.439761Z","iopub.status.idle":"2022-04-04T15:26:49.454515Z","shell.execute_reply.started":"2022-04-04T15:26:49.439728Z","shell.execute_reply":"2022-04-04T15:26:49.453781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A custom layer\nclass ChannelAttentionModule(keras.layers.Layer):\n    def __init__(self, ratio=8):\n        '''paper: https://arxiv.org/abs/1807.06521\n        code: https://gist.github.com/innat/99888fa8065ecbf3ae2b297e5c10db70\n        '''\n        super(ChannelAttentionModule, self).__init__()\n        self.ratio = ratio\n        self.gapavg = keras.layers.GlobalAveragePooling2D()\n        self.gmpmax = keras.layers.GlobalMaxPooling2D()\n        \n    def build(self, input_shape):\n        self.conv2 = keras.layers.Conv2D(input_shape[-1], \n                                         kernel_size=1,\n                                         strides=1, \n                                         padding='same',\n                                         use_bias=False, \n                                         activation=tf.nn.elu)\n        super(ChannelAttentionModule, self).build(input_shape)\n\n    def call(self, inputs):\n        # compute gap and gmp pooling \n        gapavg = self.gapavg(inputs)\n        gmpmax = self.gmpmax(inputs)\n        gapavg = keras.layers.Reshape((1, 1, gapavg.shape[1]))(gapavg)   \n        gmpmax = keras.layers.Reshape((1, 1, gmpmax.shape[1]))(gmpmax)   \n        # forward passing to the respected layers\n        gapavg_out = self.conv2(gapavg)\n        gmpmax_out = self.conv2(gmpmax)\n        return tf.math.sigmoid(gapavg_out + gmpmax_out)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:30:16.750075Z","iopub.execute_input":"2022-04-04T15:30:16.750400Z","iopub.status.idle":"2022-04-04T15:30:16.761309Z","shell.execute_reply.started":"2022-04-04T15:30:16.750369Z","shell.execute_reply":"2022-04-04T15:30:16.760417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Original Src: https://github.com/bfelbo/DeepMoji/blob/master/deepmoji/attlayer.py\nclass AttentionWeightedAverage2D(keras.layers.Layer):\n    def __init__(self, **kwargs):\n        self.init = keras.initializers.get('uniform')\n        super(AttentionWeightedAverage2D, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        self.input_spec = [layers.InputSpec(ndim=4)]\n        assert len(input_shape) == 4\n        self.W = self.add_weight(shape=(input_shape[3], 1),\n                                 name='{}_W'.format(self.name),\n                                 initializer=self.init)\n        self._trainable_weights = [self.W]\n        super(AttentionWeightedAverage2D, self).build(input_shape)\n\n    def call(self, x):\n        # computes a probability distribution over the timesteps\n        # uses 'max trick' for numerical stability\n        # reshape is done to avoid issue with Tensorflow\n        # and 2-dimensional weights\n        logits  = K.dot(x, self.W)\n        x_shape = K.shape(x)\n        logits  = K.reshape(logits, (x_shape[0], x_shape[1], x_shape[2]))\n        ai      = K.exp(logits - K.max(logits, axis=[1,2], keepdims=True))\n        \n        att_weights    = ai / (K.sum(ai, axis=[1,2], keepdims=True) + K.epsilon())\n        weighted_input = x * K.expand_dims(att_weights)\n        result         = K.sum(weighted_input, axis=[1,2])\n        return result\n\n    def get_output_shape_for(self, input_shape):\n        return self.compute_output_shape(input_shape)\n\n    def compute_output_shape(self, input_shape):\n        output_len = input_shape[3]\n        return (input_shape[0], output_len)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:30:19.802563Z","iopub.execute_input":"2022-04-04T15:30:19.802867Z","iopub.status.idle":"2022-04-04T15:30:19.817096Z","shell.execute_reply.started":"2022-04-04T15:30:19.802822Z","shell.execute_reply":"2022-04-04T15:30:19.816172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Keras Model Sub-Classing</font>\n\nUntil now, we've seen some of the building blocks of the complete model. Now, we will be building the entire model with these blocks. In the following, we've used `keras` model new [subclassing API](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) to build the model. Though it can be easily done with functional API, but we will now choose the subclassing API approach. ","metadata":{}},{"cell_type":"code","source":"class RANZCRClassifier(keras.Model):\n    def __init__(self, dim):\n        super(RANZCRClassifier, self).__init__()\n        # Defining all trainable layers in __init__ / build\n        self.base  = BASE_NETS(\n            input_shape=(IMG_SIZE, IMG_SIZE, 3),\n            weights='imagenet' if DEVICE == \"TPU\" else None, \n            include_top=False\n        )\n        \n        # Keras Built-in\n        self.batch_norm  = layers.BatchNormalization()\n        self.dropout     = layers.Dropout(rate=0.5)\n        \n        # Neck\n        self.can_module   = ChannelAttentionModule()\n        self.san_module_x = SpatialAttentionModule()\n        self.san_module_y = SpatialAttentionModule()\n        self.awn_module   = AttentionWeightedAverage2D()\n        \n        # Head\n        self.dense_layer = layers.Dense(512, activation=tf.nn.relu)\n        self.classifier  = layers.Dense(len(SUBMIT.columns[1:]), activation='sigmoid', dtype=tf.float32)\n    \n    def call(self, input_tensor, training=False):\n        if training is None:\n            training = K.learning_phase()\n            \n        # Base Inputs\n        base_out = self.base(input_tensor)\n\n        # Attention Modules 1\n        # Channel Attention + Spatial Attention \n        canx   = self.can_module(base_out)*base_out\n        spnx   = self.san_module_x(canx)*canx\n        spny   = self.san_module_y(canx)\n\n        # Global Weighted Average Pooling\n        gapx   = layers.GlobalAveragePooling2D()(spnx)\n        wvgx   = layers.GlobalAveragePooling2D()(spny)\n        gapavg = layers.Average()([gapx, wvgx])\n        \n        # Attention Modules 2\n        # Attention Weighted Average (AWG)\n        awgavg = self.awn_module(base_out)\n        # Summation of Attentions\n        attns_adds = layers.Add()([gapavg, awgavg])\n        \n        # Tails\n        x = self.batch_norm(attns_adds)\n        x = self.dense_layer(x)\n        x = self.dropout(x, training=training)\n        x = self.classifier(x)\n        \n        if not training:\n            return x, base_out, canx\n        return x\n        \n    # AFAIK: The most convenient method to print model.summary() in suclassed model\n    def build_graph(self):\n        x = keras.Input(shape=(IMG_SIZE, IMG_SIZE,3))\n        return keras.Model(inputs=[x], outputs=self.call(x))","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:30:22.140288Z","iopub.execute_input":"2022-04-04T15:30:22.140587Z","iopub.status.idle":"2022-04-04T15:30:22.156196Z","shell.execute_reply.started":"2022-04-04T15:30:22.140555Z","shell.execute_reply":"2022-04-04T15:30:22.155348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Build Model and Plot**","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    model = RANZCRClassifier((IMG_SIZE, IMG_SIZE, 3))\n    model.build((None, *(IMG_SIZE, IMG_SIZE, 3)))\n\n\nkeras.utils.plot_model(\n    model.build_graph(), \n    show_shapes      = True, \n    show_layer_names = True, \n    expand_nested    = False,                      \n)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:30:25.288802Z","iopub.execute_input":"2022-04-04T15:30:25.289106Z","iopub.status.idle":"2022-04-04T15:30:58.351829Z","shell.execute_reply.started":"2022-04-04T15:30:25.289077Z","shell.execute_reply":"2022-04-04T15:30:58.350983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Learning Rate Schedule Config</font>\n\n**Learning Rate Schedule. LRS** It's a function that takes an epoch index and current learning rate as inputs and returns a new learning rate as output. There are many types of **LRS**, [see](https://keras.io/api/optimizers/learning_rate_schedules/). For our preference, we will subclass the [schedules.LearningRateSchedule](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules/LearningRateSchedule) API to build a **WarmUp Learning Rate Schedule** of the following **LRS**.\n\n- ExponentialDecay\n- CosineDecay\n- Constant\n- **CosineDecay-Restart** (We will use it)","metadata":{}},{"cell_type":"code","source":"class WarmupLearningRateSchedule(schedules.LearningRateSchedule):\n    \"\"\"Provides a variety of learning rate decay schedules with warm up.\"\"\"\n    def __init__(self,\n                 initial_lr,\n                 steps_per_epoch=None,\n                 lr_decay_type='exponential',\n                 decay_factor=0.97,\n                 decay_epochs=2.4,\n                 total_steps=None,\n                 warmup_epochs=5,\n                 minimal_lr=0):\n        super(WarmupLearningRateSchedule, self).__init__()\n        self.initial_lr      = initial_lr\n        self.steps_per_epoch = steps_per_epoch\n        self.lr_decay_type   = lr_decay_type\n        self.decay_factor    = decay_factor\n        self.decay_epochs    = decay_epochs\n        self.total_steps     = total_steps\n        self.warmup_epochs   = warmup_epochs\n        self.minimal_lr      = minimal_lr\n\n    def __call__(self, step):\n        if self.lr_decay_type == 'exponential':\n            assert self.steps_per_epoch is not None\n            decay_steps = self.steps_per_epoch * self.decay_epochs\n            lr = schedules.ExponentialDecay(self.initial_lr, decay_steps,  self.decay_factor, staircase=True)(step)\n        elif self.lr_decay_type == 'cosine':\n            assert self.total_steps is not None\n            lr = 0.5 * self.initial_lr * (1 + tf.cos(np.pi * tf.cast(step,  tf.float32) / self.total_steps))\n            \n        elif self.lr_decay_type == 'linear':\n            assert self.total_steps is not None\n            lr = (1.0 - tf.cast(step, tf.float32) / self.total_steps) * self.initial_lr\n            \n        elif self.lr_decay_type == 'constant':\n            lr = self.initial_lr\n        \n        elif self.lr_decay_type == 'cosine_restart':\n            decay_steps = self.steps_per_epoch * self.decay_epochs\n            lr = tf.keras.experimental.CosineDecayRestarts(self.initial_lr, decay_steps)(step)\n        else:\n            assert False, 'Unknown lr_decay_type : %s' % self.lr_decay_type\n\n        if self.minimal_lr:\n            lr = tf.math.maximum(lr, self.minimal_lr)\n\n        if self.warmup_epochs:\n            warmup_steps = int(self.warmup_epochs * self.steps_per_epoch)\n            warmup_lr = (\n              self.initial_lr * tf.cast(step, tf.float32) /\n              tf.cast(warmup_steps, tf.float32))\n            lr = tf.cond(step < warmup_steps, lambda: warmup_lr, lambda: lr)\n\n        return lr\n\n    def get_config(self):\n        return {\n            'initial_lr'     : self.initial_lr,\n            'steps_per_epoch': self.steps_per_epoch,\n            'lr_decay_type'  : self.lr_decay_type,\n            'decay_factor'   : self.decay_factor,\n            'decay_epochs'   : self.decay_epochs,\n            'total_steps'    : self.total_steps,\n            'warmup_epochs'  : self.warmup_epochs,\n            'minimal_lr'     : self.minimal_lr,\n        }\n\n\nlr_sched = 'cosine_restart'\nlr_base  = 0.016\nlr_min   = 0\nlr_decay_epoch  = 2.4\nlr_warmup_epoch = 5\nlr_decay_factor = 0.97\n\nscaled_lr     = lr_base * (BATCH_SIZE / 256.0)\nscaled_lr_min = lr_min * (BATCH_SIZE / 256.0)\ntotal_steps   = TRAIN_STEPS_PER_EPOCH * EPOCHS\n\nlearning_rate = WarmupLearningRateSchedule(\n    scaled_lr,\n    steps_per_epoch=TRAIN_STEPS_PER_EPOCH,\n    decay_epochs=lr_decay_epoch,\n    warmup_epochs=lr_warmup_epoch,\n    decay_factor=lr_decay_factor,\n    lr_decay_type=lr_sched,\n    total_steps=total_steps,\n    minimal_lr=scaled_lr_min)\n\nrng = [i for i in range(total_steps)]\nlr_y = [learning_rate(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, lr_y)\nplt.xlabel('Iteration',size=14)\nplt.ylabel('Learning Rate',size=14)","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:31:57.344736Z","iopub.execute_input":"2022-04-04T15:31:57.345574Z","iopub.status.idle":"2022-04-04T15:33:06.150687Z","shell.execute_reply.started":"2022-04-04T15:31:57.345534Z","shell.execute_reply":"2022-04-04T15:33:06.149719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Callbacks</font>\n\n\n**Callback.** A callback is an object that can perform actions at various stages of training. Here we will use some built-in callback object such as `ModelCheckpoint`, `CSVLogger` etc. Along with these built-in callback, we will aslo build a custom callback that will perform computation of the **GradCAM** on some random samples picked from validation sets. We will set an interval to show the **GradCAM** afther the end of the epoch. We'll refer this callback as **GradCAMCallback**. \n\nSee more details about [callback](https://keras.io/api/callbacks/). ","metadata":{}},{"cell_type":"code","source":"class GradCAMCallback(keras.callbacks.Callback):\n    def __init__(self, epoch_interval=None):\n        self.epoch_interval = epoch_interval\n\n    # ref: https://keras.io/examples/vision/grad_cam/\n    def make_gradcam_heatmap(self, img_array, grad_model, pred_index=None):\n        with tf.GradientTape(persistent=True) as tape:\n            preds, base_top, swin_top = grad_model(img_array)\n            if pred_index is None:\n                pred_index = tf.argmax(preds[0])\n            class_channel = preds[:, pred_index]\n            \n        grads = tape.gradient(class_channel, base_top)\n        pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n        base_top  = base_top[0]\n        heatmap_a = base_top @ pooled_grads[..., tf.newaxis]\n        heatmap_a = tf.squeeze(heatmap_a)\n        heatmap_a = tf.maximum(heatmap_a, 0) / tf.math.reduce_max(heatmap_a)\n        heatmap_a = heatmap_a.numpy()\n        \n        grads = tape.gradient(class_channel, swin_top)\n        pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n        swin_top = swin_top[0]\n        heatmap_b = swin_top @ pooled_grads[..., tf.newaxis]\n        heatmap_b = tf.squeeze(heatmap_b)\n        heatmap_b = tf.maximum(heatmap_b, 0) / tf.math.reduce_max(heatmap_b)\n        heatmap_b = heatmap_b.numpy()\n        return heatmap_a, heatmap_b\n\n    def on_epoch_end(self, epoch, logs=None):\n        if epoch and epoch % self.epoch_interval == 0:\n            # pick some samples (i.e here 5) to compute grad-cam and plot\n            image_batch, label_batch = next(iter(valid_datasets))\n            image_batch, label_batch = image_batch[:5], label_batch[:5]\n\n            for sample, label in zip(image_batch, label_batch):\n                # make grad-cam \n                img_array = sample[tf.newaxis, ...] \n                heatmap_a, heatmap_b = self.make_gradcam_heatmap(img_array, model)\n\n                # overaly heatmap and input sample \n                overaly_a = self.save_and_display_gradcam(sample, heatmap_a)\n                overlay_b = self.save_and_display_gradcam(sample, heatmap_b)\n\n                # passing three 2D samples to plot\n                self.plot_stuff(img_array, overaly_a, overlay_b)\n\n\n    # ref: https://keras.io/examples/vision/grad_cam/\n    def save_and_display_gradcam(self, \n                                 img, \n                                 heatmap, \n                                 target=None, \n                                 pred=None,\n                                 cam_path=\"cam.jpg\", \n                                 alpha=0.6, \n                                 plot=None):\n        # Rescale heatmap to a range 0-255\n        heatmap = np.uint8(255 * heatmap)\n\n        # Use jet colormap to colorize heatmap\n        jet = cm.get_cmap(\"jet\") \n\n        # Use RGB values of the colormap\n        jet_colors  = jet(np.arange(256))[:, :3]\n        jet_heatmap = jet_colors[heatmap]\n\n        # Create an image with RGB colorized heatmap\n        # jet_heatmap = keras.utils.array_to_img(jet_heatmap)\n        jet_heatmap = keras.preprocessing.image.array_to_img(jet_heatmap)\n        \n        # resize to input image\n        jet_heatmap = jet_heatmap.resize((img.shape[0], img.shape[1]))\n    \n        # jet_heatmap = keras.utils.img_to_array(jet_heatmap)\n        jet_heatmap = keras.preprocessing.image.img_to_array(jet_heatmap)\n\n        # Superimpose the heatmap on original image\n        superimposed_img = img + jet_heatmap * alpha\n        \n        # superimposed_img = keras.utils.array_to_img(superimposed_img)\n        superimposed_img = keras.preprocessing.image.array_to_img(superimposed_img)\n        return superimposed_img\n        \n    def plot_stuff(self, inputs, features_a, features_b):\n        plt.figure(figsize=(25, 25))\n        \n        plt.subplot(1, 3, 1)\n        plt.axis('off')\n        plt.imshow(tf.squeeze(inputs/255, axis=0))\n        plt.title('Input')\n        \n        plt.subplot(1, 3, 2)\n        plt.axis('off')\n        plt.imshow(features_a)\n        plt.title('BaseModule')\n        \n        plt.subplot(1, 3, 3)\n        plt.axis('off')\n        plt.imshow(features_b)\n        plt.title('HeadModule (CAN)')\n        plt.show() ","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:33:11.951287Z","iopub.execute_input":"2022-04-04T15:33:11.951611Z","iopub.status.idle":"2022-04-04T15:33:11.977831Z","shell.execute_reply.started":"2022-04-04T15:33:11.951577Z","shell.execute_reply":"2022-04-04T15:33:11.976637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_callbacks():\n    # save model checkpoint based on monitored metrics \n    checkpoint = keras.callbacks.ModelCheckpoint('model.h5', \n                                                 save_best_only    = True, \n                                                 save_weights_only = True,\n                                                 monitor  = 'val_auc', \n                                                 mode     = 'max')\n    # stop training safely if nan loss occurs\n    stop_if_nan = keras.callbacks.TerminateOnNaN()\n    \n    # save training logs for post processing or post eda \n    save_log = keras.callbacks.CSVLogger('history.csv')\n    \n    return [checkpoint, stop_if_nan, save_log, GradCAMCallback(epoch_interval=4)]","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:33:11.979814Z","iopub.execute_input":"2022-04-04T15:33:11.980113Z","iopub.status.idle":"2022-04-04T15:33:11.996712Z","shell.execute_reply.started":"2022-04-04T15:33:11.980085Z","shell.execute_reply":"2022-04-04T15:33:11.995637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Compile & Training</font>\n\nIn compile, we will pass `learning_rate` achieved from `WarmupLearningRateSchedule`. Also we will use `label smoothing` technique. And lastly, as it's a mult-label classification task and competition metric is `AUC`, we will set those accordingly. ","metadata":{}},{"cell_type":"code","source":"model.compile(\n    optimizer = keras.optimizers.Adam(learning_rate),\n    loss = keras.losses.BinaryCrossentropy(label_smoothing=LABEL_SMOOTH), \n    metrics = [keras.metrics.AUC(multi_label=True)],\n    steps_per_execution=REPLICAS\n)\n\nif DEVICE == \"TPU\":\n    history = model.fit(\n        train_datasets, \n        epochs    = EPOCHS,\n        verbose   = VERBOSITY,\n        callbacks = get_callbacks(),\n        steps_per_epoch  = TRAIN_STEPS_PER_EPOCH,\n        validation_data  = valid_datasets,\n        validation_steps = VALID_STEPS_PER_EPOCH\n    )\nelse:\n    print('Please use TPU for training.\\nReloading from saved weights for Inference.')\n    model.load_weights('../input/multiattentioncheckwg/model_new.h5')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T15:33:11.998200Z","iopub.execute_input":"2022-04-04T15:33:11.998710Z","iopub.status.idle":"2022-04-04T16:49:49.262557Z","shell.execute_reply.started":"2022-04-04T15:33:11.998678Z","shell.execute_reply":"2022-04-04T16:49:49.261585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <font color = \"seagreen\">Inference</font>","metadata":{}},{"cell_type":"code","source":"if FILE_TYPE != 'TF_RECORD':\n    test_decoder = build_decoder(with_labels=False, target_size=IMG_SIZE)\n    test_dataset = jpeg_loader(\n        TEST_IMG_PATH, \n        bsize     = BATCH_SIZE, \n        repeat    = False, \n        shuffle   = False, \n        augment   = False, \n        cache     = False, \n        decode_fn = test_decoder\n    )\n    num_files = len(TEST_IMG_PATH)\n    pred_step = int(np.ceil(num_files / float(BATCH_SIZE)))\n\nelse:\n    tfrecords_schema = {\n        \"StudyInstanceUID\" : tf.io.FixedLenFeature([], tf.string),\n        \"image\"            : tf.io.FixedLenFeature([], tf.string)\n    }\n    test_dataset = tfrecords_loader(\n            TEST_IMG_PATH, \n            tfschemes   = tfrecords_schema,\n            shuffle     = False,\n            repeat      = False,\n            augment     = False,\n            ignore_order= False,\n            inference   = True\n        )\n    num_files = sum(1 for record in tf.data.TFRecordDataset(TEST_IMG_PATH))\n    pred_step = int(np.ceil(num_files / float(BATCH_SIZE / REPLICAS)))\n    \n    \nimage_batch = next(iter(test_dataset))\nprint(image_batch.shape)\nshow_batch(image_batch.numpy(), title='Test Sets')","metadata":{"execution":{"iopub.status.busy":"2022-04-04T16:49:49.264942Z","iopub.execute_input":"2022-04-04T16:49:49.265205Z","iopub.status.idle":"2022-04-04T16:50:20.135173Z","shell.execute_reply.started":"2022-04-04T16:49:49.265175Z","shell.execute_reply":"2022-04-04T16:50:20.134208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if DEVICE != 'TPU':\n    '''It's a code competition, to submit, you must disable internet, \n    means, either use GPU or CPU for model inference w/o internet.\n    '''\n    SUBMIT[SUBMIT.columns[1:]] = model.predict(test_dataset, steps=pred_step, verbose=1)\n    SUBMIT.to_csv('submission11.csv', index=False)\n    display(SUBMIT.head())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final Note\n\n1. The best score the **multi-branch soft attention net** achieved is `0.95656` on the test dataset which is without the presence of any sort of ensembling and test time augmentation. With extensive experiment and careful integreation, this can achieve much further better result and able to produce strong visual interpretation.\n2. How to use it on my own dataset?\n    - First, understand the competition task and its data format. And try to relate with yours.\n    - Second, run this notebook successfully on the competition data.\n    - Lastly, replace the dataset with yours.\n3. The competition data also provides some segmentation annotation for potential segmentation modeling. Most of the top solution used it. Check out the following series of notebook, [tt195361](https://www.kaggle.com/tt195361) who reproduces the 1st place solution in `TensorFlow.Keras`. \n    - [Make Mask](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-1-make-masks) - [Segmentation Model](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-2-seg-model)\n    - [Gen Mask](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-3-gen-masks) - [Cls Model](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-4-cls-model)\n    - [Inference](https://www.kaggle.com/tt195361/ranzcr-1st-place-solution-by-tf-5-inference)","metadata":{}}]}