{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Version 11-12 Update**\n\n- Upgrade TF `2.6` from `2.4`.\n- Adjust corresponding issues due to package upgrade.\n- Fix minor bugs and gramatical spelling.\n\n**Version 9-10 Update**\n \n- Added`TF` Augmentation for **3D** samples. \n- Added `Keras` Augmentation for **3D** samples. \n- Added `Volumentations` Augmentation for **3D** samples, which is based on **Albumentation**. Thanks to [ZFTurbo](https://github.com/ZFTurbo/volumentations).\n- Add **Gradient Accumulation** technique in **3D** model training. More description are added to the relevant section on this notebook.\n\n**Note**: For **3D** modeling, instead of `(h, w, depth * channel, 1)`, now we will be following `(h, w, depth, channel)`. For example: previously we have input shape as `(224, 224, 5*4, 1)` but now we've `(224, 224, 5, 4)`. Here, `4` is the number of modality in this dataset and `5` is the number of samples from each modality. \n\nPlease, whom you've already forked the notebook, see the update from version **9**  as it contains some major changes. Also, in the data loader, we've added some basic functionalities to tackle the **full black image** and **less number of slices** regarding issue. Also, from version **9**, we will not update on `2D` modeling training approach; please feel free to check version 8 or lower for that if needed. \n\n**This code is full of commentry, please go through each line of this code example to understand correctly. Further update will be added to this repo: [innat/BrainTumorClassification](https://github.com/innat/BraTS-MGMT-Classification). If it becoems helpful for your projects and research, we would appreciate the [citation](https://github.com/innat/BraTS-MGMT-Classification/blob/main/CITATION.cff).**\n\n\n---\n\n**Version 8 Update**\n\n- Update `tf.data` pipeline in a clean modular way, would be easy to tweak\n- Add random data augmentation on each input modality (slices) using `tf.image.random_*` module \n- Add random `cutout` augmentation. \n- Add `3D` pre-trained modeling setup: `efficientnet-3d` and others. \n- Add `2D` modeling setup either: `efficientnet-2d` and others.\n- Basic training of `3D` and `2D` modeling with shape of (`312, 312, 4, 1`) and (`312, 312, 4`) respectively. \n\n**Note**: The pre-trained weights of `3D` model is taken from [ZFTurbo/efficientnet_3D](https://github.com/ZFTurbo/efficientnet_3D) and [ZFTurbo/classification_models_3D](https://github.com/ZFTurbo/classification_models_3D). The weight files are uploaded and can be found from here ([efnet-3d](https://www.kaggle.com/ipythonx/keras-3d-efficientnet-imagenet-weights-b0b7)) and here ([classification-3d](https://www.kaggle.com/ipythonx/pretrained-3d-model-weights-keras)) respectively. \n\n\n**Version 3 Update**\n\n- Wrap up `keras` sequence generator to `tf.data` API \n- Add data augmentation by using `keras` preprocessing layers \n- Subclass the model for tweaking gradient before update and reg loss \n- Train the model (`InceptionResNet3D`) with shape of (`256, 256, 20, 1`)\n\n\n**Note**: As there are many empty image (`img.mean() == 0`) among four modalities, we found that, the training progress is bit unstable. We need to find out a better and efficient technique for `n` depth 3D model training. Also, we've seen that, among four series or modalities, training with only `T2w` was compratively better. \n\n---\n\n## RSNA-MICCAI Brain Tumor Radiogenomic Classification\n\nIn this baseline example, we will be building a 3D convolutional neural network to predict the status of a genetic biomaker important for brain cancer treatment in the **mpMRI** scans that includes the following four types of images for single study or patient. \n\n```\n- Fluid Attenuated Inversion Recovery (FLAIR)\n- T1-weighted pre-contrast (T1w)\n- T1-weighted post-contrast (T1Gd)\n- T2-weighted (T2)\n```\n\nIn the training data, each study (or each patient id e.g `00000`) belongs to a single label (`train_labels.csv`). Each of this study has further 4 series (`FLAIR`, `T1w`, `T1Gd`, `T2`), and each series has multiple images, in a DICOM format. The file structure is something looks like as follows:\n\n```\nTraining/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n└─── 00001\n│   │ ...\n└─── 00002\n│   │ ...\n```","metadata":{}},{"cell_type":"markdown","source":"## TF.Keras Code","metadata":{}},{"cell_type":"code","source":"!pip install Keras-Applications==1.0.8 -q\n!pip install ../input/keras-3d-model-and-3d-augmentation/efficientnet_3D-1.0.1-py3-none-any.whl -q\n!pip install ../input/keras-3d-model-and-3d-augmentation/classification_models_3D-1.0.2-py3-none-any.whl -q\n!pip install ../input/keras-3d-model-and-3d-augmentation/volumentations_3D-1.0.3-py3-none-any.whl -q","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-13T17:44:09.049318Z","iopub.execute_input":"2022-01-13T17:44:09.049871Z","iopub.status.idle":"2022-01-13T17:44:49.973750Z","shell.execute_reply.started":"2022-01-13T17:44:09.049753Z","shell.execute_reply":"2022-01-13T17:44:49.972755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# general \nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt \nimport os, glob, random, cv2, glob, pydicom, warnings\nfrom sklearn.model_selection import StratifiedKFold\n\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nwarnings.filterwarnings('ignore')\n\n# ML pack\nimport tensorflow as tf; print(tf.__version__)\nfrom tensorflow import keras\n\n# If memory growth is enabled for a PhysicalDevice, the runtime - \n# initialization will not allocate all memory on the device.\nphysical_devices = tf.config.list_physical_devices('GPU')\ntry: \n    tf.config.experimental.set_memory_growth(physical_devices[0], True)\n    assert tf.config.experimental.get_memory_growth(physical_devices[0])\nexcept:\n    # Invalid device or cannot modify virtual devices once initialized.\n    pass \n\n# Params \nAUTO = tf.data.AUTOTUNE\n\n# For reproducible results    \ndef seed_all(s):\n    random.seed(s)\n    np.random.seed(s)\n    tf.random.set_seed(s)\n    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'\n    os.environ['PYTHONHASHSEED'] = str(s) ","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:49.977805Z","iopub.execute_input":"2022-01-13T17:44:49.978502Z","iopub.status.idle":"2022-01-13T17:44:56.216186Z","shell.execute_reply.started":"2022-01-13T17:44:49.978466Z","shell.execute_reply":"2022-01-13T17:44:56.215243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Seed all\nglobal_seed = 101 \nseed_all(global_seed)\n\n# Training Fold\ntotal_fold = 5\nfold = 0  \n\n# After notebook version 8, modeling_in = '2D' is not supported. \nmodeling_in = '3D' # Support:'2D' or '3D' modeling,  \n\n# Augmentation library that can be used for 3D model training.\n# We can use `keras` preprocessing image augmentation library,\n# Or, we can use pure 'tf' image operations for augmentation, \n# Or, we can use 'volumentations' 3rd party library build on albumentation. \naug_lib = 'keras'  # Support, arg: 'tf', 'keras', 'volumentations'\n\n# In this competition, we have four modalities for each study\n# If you want to train slngle modality training, you need to change few code \n# in the data loader. \ninput_modality = [\"FLAIR\", \"T1w\", \"T1wCE\", \"T2w\"] \n\n# Sample input shape: \n# (batch_size, input_height, input_width, input_depth, input_channel) \nbatch_size    = 1   # Number of the batch size\naccum_step    = 3   # Gradient accumulation steps\ninput_height  = 120\ninput_width   = 120\ninput_channel = len(input_modality)  # Total number of channel, e.g. 4 \ninput_depth   = 30 # Total number of slices from each modality, e.g. 30 ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-01-13T17:44:56.218150Z","iopub.execute_input":"2022-01-13T17:44:56.218720Z","iopub.status.idle":"2022-01-13T17:44:56.227115Z","shell.execute_reply.started":"2022-01-13T17:44:56.218664Z","shell.execute_reply":"2022-01-13T17:44:56.226175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"root = '../input/rsna-miccai-brain-tumor-radiogenomic-classification/'\n\ndf = pd.read_csv(f'{root}train_labels.csv') \ntrain_sample_path = f'{root}train'\n\nprint(df.shape)\nprint(df.MGMT_value.value_counts())\n\nprint('\\nno. of the image {} and no. of the unique id {}'.\\\n      format(len(os.listdir(train_sample_path)), df.BraTS21ID.nunique()))\n\nprint('\\ntop 5:')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:56.230159Z","iopub.execute_input":"2022-01-13T17:44:56.230918Z","iopub.status.idle":"2022-01-13T17:44:56.425038Z","shell.execute_reply.started":"2022-01-13T17:44:56.230862Z","shell.execute_reply":"2022-01-13T17:44:56.423931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loader\n\n**Note: Notebook V.9+**: Almost same, except input shape now would be (`None, height, weight, 5, 4`). More details are added at the beginning of the notebook.\n\n**Note: Notebook V.8**: For each study, we will pick **5** middle dicom or image samples at cetain interval from each of the four series (`FLAIR`, `T1w`, `T1Gd`, `T2`) for our 3D model input, e.g `input_shape: (None, height, weights, 5*4, 1)`. So, for picking 5 samples from each series, the depth would be `4 * 5` or `20` for each patient id or study id. However, we probabely need to find some better technique to pick samples from each of these four series though. For now, we're doing kinda random stuff. For example: check the following illustration; for the 3D model the input size would be : `height, width, 12, 1`. Also, as we already notice, there're too much black pixel, we will try to address that too. \n\n\n![quick png](https://user-images.githubusercontent.com/17668390/126070190-4a2d8aed-9dd8-4bee-aa15-f95429553a9c.png)","metadata":{}},{"cell_type":"markdown","source":"**Keras Sequence to TensorFlow tf.Data API**\n\nIt's known that, `tf.data` is more efficient when it comes to data pipelines compare to others, e.g. `keras` sequences data generator. Here we will try to wrap up an **existing `keras` sequence generator** to `tf.data` API using [tf.data.Dataset.from_generator function](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). ","metadata":{}},{"cell_type":"code","source":"skf = StratifiedKFold(n_splits=total_fold, shuffle=True, random_state=global_seed)\n\nfor index, (train_index, val_index) in enumerate(skf.split(X=df.index, \n                                                           y=df.MGMT_value)):\n    df.loc[val_index, 'fold'] = index\n    \nprint('Ground Truth Distribution Fold-Wise..')\nprint(df.groupby(['fold', df.MGMT_value]).size())","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:56.427051Z","iopub.execute_input":"2022-01-13T17:44:56.427778Z","iopub.status.idle":"2022-01-13T17:44:56.453759Z","shell.execute_reply.started":"2022-01-13T17:44:56.427738Z","shell.execute_reply":"2022-01-13T17:44:56.452732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data loader \nclass BrainTumorGenerator(keras.utils.Sequence):\n    def __init__(self, dicom_path, data, is_train=True):\n        self.is_train = is_train # to control training/validation/inference part \n        self.data = data\n        self.dicom_path = dicom_path\n        self.label = self.data['MGMT_value']\n  \n    def __len__(self):\n        return len(self.data['BraTS21ID'])\n    \n    def __getitem__(self, index):\n        patient_ids = f\"{self.dicom_path}/{str(self.data['BraTS21ID'][index]).zfill(5)}/\"\n   \n        # For 2D modeling \n        # NOTE: For 2D model training, see notebook version 8 or less. \n        channel = []\n    \n        # for 3D modeling \n        flair = []\n        t1w   = []\n        t1wce = []\n        t2w   = [] \n        \n        # Iterating over each modality\n        for m, t in enumerate(input_modality):\n            t_paths = sorted(\n                glob.glob(os.path.join(patient_ids, t, \"*\")), \n                key=lambda x: int(x[:-4].split(\"-\")[-1]),\n            )\n            \n            # Pick input_depth times slices -\n            # - from middle range possible \n            strt_idx = (len(t_paths) // 2) - (input_depth // 2)\n            end_idx = (len(t_paths) // 2) + (input_depth // 2)\n            # slicing extracting elements with 1 intervals \n            picked_slices = t_paths[strt_idx:end_idx:1]\n            \n            # Iterating over the picked slices and do some basic processing, \n            # such as removing black borders\n            # and lastly, bind them all in the end.\n            for i in picked_slices:\n                # Reading pixel file from dicom file \n                image = self.read_dicom_xray(i)\n                \n                # It's possible that among picked_slices, there can be some black image, \n                # which is not wanted, so we iterate back to dicom file to get - \n                # - any non-black image otherwise move on with black image \n                j = 0\n                while True:\n                    # if it's a black image, try to pick any random slice of non-black  \n                    # otherwise move on with black image. \n                    if image.mean() == 0:\n                        # do something \n                        image = self.read_dicom_xray(random.choice(t_paths)) \n                        j += 1\n                        if j == 100:\n                            break\n                    else:\n                        break\n                        \n                # Now, we remove black areas; remove black borders from brain image \n                rows = np.where(np.max(image, 0) > 0)[0]\n                cols = np.where(np.max(image, 1) > 0)[0]\n                if rows.size:\n                    image = image[cols[0]: cols[-1] + 1, rows[0]: rows[-1] + 1]\n                else:\n                    image = image[:1, :1]\n           \n                # In 3D modeling, we now add frames / slices of individual modalities \n                if modeling_in == '3D':\n                    if m == 0:\n                        # Adding flair \n                        flair.append(cv2.resize(image, (input_height, input_width)))\n                    elif m == 1:\n                        # Adding t1w\n                        t1w.append(cv2.resize(image, (input_height, input_width)))\n                    elif m == 2:\n                        # Adding t1wce\n                        t1wce.append(cv2.resize(image, (input_height, input_width)))\n                    elif m == 3:\n                        # Adding t2w\n                        t2w.append(cv2.resize(image, (input_height, input_width)))\n                elif modeling_in == '2D':\n                    # Adding all frames at a time \n                    channel.append(cv2.resize(image, (input_height, input_width)))\n                    \n\n        if modeling_in == '3D':\n            # [ATTENTION!!!]\n            # input_shape: (None, h, w, depth, channel)\n            # It's possible that with current data loader set up, \n            # All modalities MAY NOT HAVE SAME number of slices.\n            # In that case, we adopt some workaround.\n            # Right now, we re-append the existing slice with small color variation.\n            # Just to avoid this issue.\n            \n            # for flair \n            while True:\n                if len(flair) < input_depth and flair:\n                    flair.append(cv2.convertScaleAbs(random.choice(flair), alpha=1.2, beta=0))\n                else:\n                    break\n            \n            # for t1w\n            while True:\n                if len(t1w) < input_depth and t1w:\n                    t1w.append(cv2.convertScaleAbs(random.choice(t1w), alpha=1.1, beta=0))\n                else:\n                    break\n\n            # for t1wce\n            while True:\n                if len(t1wce) < input_depth and t1wce:\n                    t1wce.append(cv2.convertScaleAbs(random.choice(t1wce), alpha=1.2, beta=0))\n                else:\n                    break\n\n            # for t2w\n            while True:\n                if len(t2w) < input_depth and t2w:\n                    t2w.append(cv2.convertScaleAbs(random.choice(t2w), alpha=1.1, beta=0))\n                else:\n                    break\n            \n            return np.array((flair, t1w, t1wce, t2w),\n                            dtype=\"object\").T, self.label.iloc[index,]\n            \n    \n        elif modeling_in == '2D':\n            # (None, h, w, channel == depth)\n            return np.array(channel).T, self.label.iloc[index,]\n        \n    # Function to read dicom file \n    def read_dicom_xray(self, path):\n        data = pydicom.read_file(path).pixel_array\n        if data.mean() == 0:\n            # If all black, return data and find non-black if possible.\n            return data \n        data = data - np.min(data)\n        data = data / np.max(data)\n        data = (data * 255).astype(np.uint8)\n        return data","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:56.455514Z","iopub.execute_input":"2022-01-13T17:44:56.456129Z","iopub.status.idle":"2022-01-13T17:44:57.598681Z","shell.execute_reply.started":"2022-01-13T17:44:56.456089Z","shell.execute_reply":"2022-01-13T17:44:57.597459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fold_generator(fold):\n    train_labels = df[df.fold != fold].reset_index(drop=True)\n    val_labels   = df[df.fold == fold].reset_index(drop=True)\n    return (\n        BrainTumorGenerator(train_sample_path, train_labels),\n        BrainTumorGenerator(train_sample_path, val_labels)\n    )\n\n# Get fold set\ntrain_gen, val_gen = fold_generator(fold)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:57.604143Z","iopub.execute_input":"2022-01-13T17:44:57.604496Z","iopub.status.idle":"2022-01-13T17:44:57.626429Z","shell.execute_reply.started":"2022-01-13T17:44:57.604451Z","shell.execute_reply":"2022-01-13T17:44:57.625414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `Keras` Pipelines to `tf.data` Pipelines","metadata":{}},{"cell_type":"code","source":"if modeling_in == '2D':\n    train_data = tf.data.Dataset.from_generator(\n        lambda: map(tuple, train_gen),\n        (tf.float32, tf.float32),\n        (\n            tf.TensorShape([input_height, input_width, input_depth*input_channel]),\n            tf.TensorShape([]),\n        ),\n    )\n\n    val_data = tf.data.Dataset.from_generator(\n        lambda: map(tuple, val_gen),\n        (tf.float32, tf.float32),\n        (\n            tf.TensorShape([input_height, input_width, input_depth*input_channel]),\n            tf.TensorShape([]),\n        ),\n    )\nelse:\n    train_data = tf.data.Dataset.from_generator(\n        lambda: map(tuple, train_gen),\n        (tf.float32, tf.float32),\n        (\n            tf.TensorShape([input_height, input_width, input_depth, input_channel]),\n            tf.TensorShape([]),\n        ),\n    )\n\n    val_data = tf.data.Dataset.from_generator(\n        lambda: map(tuple, val_gen),\n        (tf.float32, tf.float32),\n        (\n            tf.TensorShape([input_height, input_width, input_depth, input_channel]),\n            tf.TensorShape([]),\n        ),\n    )","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:44:57.632431Z","iopub.execute_input":"2022-01-13T17:44:57.634802Z","iopub.status.idle":"2022-01-13T17:45:00.196288Z","shell.execute_reply.started":"2022-01-13T17:44:57.634758Z","shell.execute_reply":"2022-01-13T17:45:00.195425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3D Augmentation\n\nIn order to do data augmentation on 3D samples, we will have option to choose `keras` image preprocessing layers, `tf` image deterministic modules and `volumentations` which is based on well-known albumentation package. Note that, same augmentation of all slices from each modality needs to be ensured.\n\n- In `keras`, we've some built-in [Image augmentation layers](https://keras.io/api/layers/preprocessing_layers/image_augmentation/). We can use it in data loading time or we can integrate them inside the model definition to leverage GPU power. For now, we'll be using them on data loading section. We will also implement and use some custom layers for image augmentation. Note that, we need to ensure that, same augmentation must be applied to each slices of a modality, otherwise it'd be probabely meaningless to use them. FYI, `keras` has a convinient function called [TimeDistributed](https://keras.io/api/layers/recurrent_layers/time_distributed/) which can be applied in such case. \n\n\n- In `tf`, we'll be using [`tf.image.stateless_*`](https://www.tensorflow.org/api_docs/python/tf/image/stateless_random_brightness) modules for deterministic results. So that, we can apply same augmentation on all slices of a modality. \n\n- Lastly, in `volumentations`, originally developed by [hawkey](https://github.com/ashawkey/volumentations) and greatly improved by [ZFTurbo](https://github.com/ZFTurbo/volumentations). Please, appreciate their open-source code if it helps. The same augmentation is handle by itself; less pain. ","metadata":{}},{"cell_type":"markdown","source":"## Keras 3D Augmentation\n\n\nFrom API, we've `RandomFlip`, `RandomRotation`, `RandomCrop` etc layers. Apart from the buil-in layer, we will be building our own image processing layer and use them for image augmentation. For example: `RandomInvert`, `RandomEqualize` and `RandomCutout`.","metadata":{}},{"cell_type":"code","source":"import tensorflow_addons as tfa\nfrom tensorflow.keras import layers\n\n# Custom Layer 1 \nclass RandomInvert(layers.Layer):\n    def __init__(self, prob=0.5, **kwargs):\n        super().__init__(**kwargs)\n        self.prob = prob\n        \n    def call(self, inputs, training=True):\n        tf.random.set_seed(global_seed)\n        if tf.random.uniform([]) < self.prob:\n            return tf.cast(255.0 - inputs, dtype=tf.float32)\n        else: \n            return tf.cast(inputs, dtype=tf.float32)\n        \n    def get_config(self):\n        config = {\n            'prob': self.prob,\n        }\n        base_config = super(RandomInvert, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n    \n    def compute_output_shape(self, input_shape):\n        return input_shape\n    \n# Custom Layer 2 \nclass RandomEqualize(layers.Layer):\n    def __init__(self, prob=0.5, **kwargs):\n        super().__init__(**kwargs)\n        self.prob = prob\n        \n    def call(self, inputs, training=True):\n        tf.random.set_seed(global_seed)\n        if tf.random.uniform([]) < self.prob:\n            return tf.cast(tfa.image.equalize(inputs), dtype=tf.float32)\n        else: \n            return tf.cast(inputs, dtype=tf.float32)\n        \n    def get_config(self):\n        config = {\n            'prob': self.prob,\n        }\n        base_config = super(RandomEqualize, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n    \n    def compute_output_shape(self, input_shape):\n        return input_shape\n        \n# Custom Layer 3\nclass RandomCutout(layers.Layer):\n    def __init__(self, prob=0.5, mask_size=(20, 20), replace=0, **kwargs):\n        super().__init__(**kwargs)\n        self.prob = prob \n        self.replace = replace\n        self.mask_size = mask_size\n        \n    def call(self, inputs, training=True):\n        tf.random.set_seed(global_seed)\n        if tf.random.uniform([]) < self.prob:\n            inputs = tfa.image.random_cutout(inputs, \n                                             mask_size=self.mask_size, \n                                             constant_values=self.replace)  \n            return tf.cast(inputs, dtype=tf.float32)\n        else: \n            return tf.cast(inputs, dtype=tf.float32)\n        \n    def get_config(self):\n        config = {\n            'prob': self.prob,\n            'replace': self.replace,\n            'mask_size': self.mask_size\n        }\n        base_config = super(RandomCutout, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n    \n    def compute_output_shape(self, input_shape):\n        return input_shape","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.198114Z","iopub.execute_input":"2022-01-13T17:45:00.198495Z","iopub.status.idle":"2022-01-13T17:45:00.369093Z","shell.execute_reply.started":"2022-01-13T17:45:00.198439Z","shell.execute_reply":"2022-01-13T17:45:00.368247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed = None\nrand_flip = layers.RandomFlip(\"horizontal_and_vertical\", seed=set_seed)\nrand_tran = layers.RandomTranslation(height_factor=0.1, width_factor=0.2, seed=set_seed)\nrand_rote = layers.RandomRotation(factor=0.01, seed=set_seed)\nrand_cntr = layers.RandomContrast(factor=0.6, seed=set_seed)\nrand_crop = layers.RandomCrop(int(input_height*0.97), int(input_width*0.97), seed=set_seed)\nrand_eqlz = RandomEqualize()\n\ndef keras_augment(image, label):\n    all_modality = tf.reshape(image, [-1, input_height, input_width, input_depth*input_channel])\n    \n    def apply_augment(x):\n        x = rand_eqlz(x)\n        x = rand_flip(x)\n        x = rand_tran(x)\n        x = rand_rote(x)\n        x = rand_cntr(x)\n        x = rand_crop(x)\n        x = layers.Resizing(input_height, input_width)(x)\n        return x \n    \n    aug_images = apply_augment(all_modality)\n    image = tf.reshape(aug_images, \n                       [-1, input_height, input_width, input_depth, input_channel])\n    \n    return image, label","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.373239Z","iopub.execute_input":"2022-01-13T17:45:00.373551Z","iopub.status.idle":"2022-01-13T17:45:00.430522Z","shell.execute_reply.started":"2022-01-13T17:45:00.373520Z","shell.execute_reply":"2022-01-13T17:45:00.429674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if modeling_in == '2D':\n    #  keras augmentation\n    keras_augment = tf.keras.Sequential(\n        [\n            RandomFlip(\"horizontal\", dtype=tf.float32),\n            RandomRotation(factor=0.01, dtype=tf.float32),\n            RandomTranslation(height_factor=0.0, width_factor=0.1, dtype=tf.float32),\n            RandomZoom(height_factor=(-0.1, -0.2), width_factor=(-0.1, -0.2)),\n            RandomInvert(prob=0.1),\n            RandomCutout(prob=0.8, replace=0, \n                         mask_size=(int(input_height * 0.2),\n                                    int(input_width * 0.2))),\n            RandomEqualize(prob=0.5)\n        ],\n        name='keras_augment_layers'\n    )","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.432127Z","iopub.execute_input":"2022-01-13T17:45:00.432463Z","iopub.status.idle":"2022-01-13T17:45:00.441358Z","shell.execute_reply.started":"2022-01-13T17:45:00.432407Z","shell.execute_reply":"2022-01-13T17:45:00.440257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TensorFlow 3D Augmentation\n\nFrom `TF` API, we have very few built-in image processing functions for image augmentation. As we need deterministic results for each slice of a modality, we will be using `tf.image.stateless_*` modules and some few others with conditions. Some of the augmentaions are \n\n```\n- tf.image.stateless_*\n- tf.image.adjust_gama\n- tfa.image.random_cutout\n- Random gaussian noise\n```","metadata":{}},{"cell_type":"code","source":"def tf_image_augmentation(image):  \n    # splitted based on modalites. we have 4 type of modalities. Input shape (h, w, depth, channel==4)\n    if modeling_in == '3D':\n        # input shape: image.shape -> (h, w, input_depth, input_channel)\n        # in such condition, we split based on the number of channel, which is 4 here. \n        # after that the variable will contains 4 splitted tensor \n        # each of which is in a shape of (h, w, input_depth, 1)\n        splitted_modalities = tf.split(tf.cast(image, tf.float32), input_channel, axis=-1) \n    elif modeling_in == '2D':\n        # input shape: image.shape -> (h, w, input_depth * input_channel)\n        # in such condition, we split based on the number of channel, which is 4 here.\n        # after that the variable will contains 4 splitted tensor \n        # each of which is in a shape of (h, w, input_depth)\n        splitted_modalities = tf.split(tf.cast(image, tf.float32), input_channel, axis=-1) \n    \n    \n    # augmented frames for 2d modeling \n    # for 2d modeling we use 1 container to gather all augmented samples from 4 modalites\n    # however, same augmentation is ensured for each modality for one study\n    augment_img = []\n    \n    # augmented frames for 3d modeling \n    # for 3d modeling we use 4 container to gather all augmented samples from 4 modalites\n    # however, same augmentation is ensured for each modality for one study\n    flair_augment_img = []\n    t1w_augment_img   = []\n    t1wce_augment_img = []\n    t2w_augment_img   = []\n    \n    \n    if modeling_in == '3D':\n        # remove the last axis.\n        # input: (h, w, input_depth, 1) : output: (h, w, input_depth)\n        splitted_modalities = [tf.squeeze(i, axis=-1) for i in splitted_modalities] \n    \n    # iterate over each modalities, e.g: flair, t1w, t1wce, t2w\n    for j, modality in enumerate(splitted_modalities):\n        # now splitting each frame from one modality \n        splitted_frames = tf.split(tf.cast(modality, tf.float32), modality.shape[-1], axis=-1)\n        \n        # iterate over each frame to conduct same augmentation on \n        # each frame \n        for i, img in enumerate(splitted_frames):\n            # Given the same seed, they return the same results independent of \n            # how many times they are called.\n            # It's very important to get deterministic augmentation results of each modality. \n            tf.random.set_seed(j)\n            np.random.seed(j)\n            \n            # In tf.image.stateless_random_* , the seed is a Tensor of shape (2,) -\n            # - whose values are any integers.\n            img = tf.image.stateless_random_flip_left_right(img, seed = (j, 2))\n            img = tf.image.stateless_random_flip_up_down(img, seed = (j, 2))\n            img = tf.image.stateless_random_contrast(img, 0.4, 0.8, seed = (j, 2))\n            img = tf.image.stateless_random_brightness(img, 0.3, seed = (j, 2))\n            \n            # For some operation, it requires channel == 3 \n            img = tf.image.stateless_random_saturation(tf.image.grayscale_to_rgb(img), \n                                                       0.9, 1.8, seed = (j, 2))\n            img = tf.image.stateless_random_hue(img, 0.4, seed = (j, 2))\n\n            # For some operation we don't need channel == 3, just 1 is enough \n            img = tf.image.rgb_to_grayscale(img)\n            img = tf.cast(\n                tf.image.stateless_random_jpeg_quality(\n                    tf.cast(img, tf.uint8), \n                    min_jpeg_quality=20, max_jpeg_quality=40, seed = (j, 2)\n                ), tf.float32)\n\n            # Ensuring same augmentation for each modalities \n            if tf.random.uniform((), seed=j) > 0.7:\n                kimg = np.random.choice([1,2,3,4])\n                kgma = np.random.choice([0.7, 0.9, 1.2])\n                \n                img = tf.image.rot90(img, k=kimg) # random rate to any of 90, 180, 270, 360 \n                img = tf.image.adjust_gamma(img, gamma=kgma) # adjust the gamma \n                noise = tf.random.normal(shape=tf.shape(img), mean=0.0, stddev=0.2,\n                                         dtype=tf.float32, seed=j) \n                img = img + noise # additive gaussian noise to image \n\n            # The mask_size should be divisible by 2. \n            if tf.random.uniform((), seed=j) > 0.6:\n                img = tfa.image.random_cutout(tf.expand_dims(img, 0),\n                                              mask_size=(int(input_height * 0.2),\n                                                         int(input_width * 0.2)), \n                                              constant_values=0) \n                img = tf.squeeze(img, axis=0)\n            \n            # Clipping. We'll rescale later. \n            img = tf.clip_by_value(img, 0, 255)\n            \n            # Gathering all frames \n            if modeling_in == '3D':\n                if j == 0: # 1st modality \n                    flair_augment_img.append(img)\n                elif j == 1: # 2nd modality \n                    t1w_augment_img.append(img)\n                elif j == 2: # 3rd modality \n                    t1wce_augment_img.append(img)\n                elif j == 3:  # 4th modality \n                    t2w_augment_img.append(img)\n            elif modeling_in == '2D':\n                augment_img.append(img)\n      \n    \n    if modeling_in == '3D':\n        image = tf.transpose([flair_augment_img, t1w_augment_img, \n                              t1wce_augment_img, t2w_augment_img])\n        image = tf.reshape(image, [input_height, input_width, input_depth, input_channel])\n    elif modeling_in == '2D':\n        image = tf.concat(augment_img, axis=-1)\n        image = tf.reshape(image, [input_height, input_width, input_depth*input_channel])\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.443709Z","iopub.execute_input":"2022-01-13T17:45:00.444376Z","iopub.status.idle":"2022-01-13T17:45:00.471815Z","shell.execute_reply.started":"2022-01-13T17:45:00.444334Z","shell.execute_reply":"2022-01-13T17:45:00.470730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Volumentation 3D\n\nPlease, refer to this [official page](https://github.com/ZFTurbo/volumentations#volumentations-3d), for better description. Here, we will show you how to use them with `tf.data` API pipelines.","metadata":{}},{"cell_type":"code","source":"from functools import partial\nfrom volumentations import *\n\ndef get_augmentation(patch_size):\n    return Compose([\n        Rotate((-5, 5), (0, 0), (0, 0), p=0.5),\n        RandomCropFromBorders(crop_value=0.1, p=0.3),\n        ElasticTransform((0, 0.15), interpolation=2, p=0.5),\n        Resize(patch_size, interpolation=1, always_apply=True, p=1.0),\n        Flip(0, p=0.5),\n        Flip(1, p=0.5),\n        RandomRotate90((0, 1), p=0.6),\n        GaussianNoise(var_limit=(0, 5), p=0.5),\n        RandomGamma(gamma_limit=(0.5, 1.5), p=0.7),\n    ], p=1.0)\n\nvolume3D = get_augmentation((input_height, input_width, input_depth))\n\ndef volume3Dfn(image):    \n    aug_data = volume3D(**{\"image\":image})\n    return tf.cast(aug_data[\"image\"], tf.float32)\n\ndef volumentations_aug(image, label):\n    # Wraps a python function and uses it as a TensorFlow op.\n    aug_img = tf.numpy_function(func=volume3Dfn, \n                                inp=[image], \n                                Tout=tf.float32)\n    aug_img.set_shape((input_height, input_width, input_depth, input_channel))\n    return aug_img, label","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.474006Z","iopub.execute_input":"2022-01-13T17:45:00.474455Z","iopub.status.idle":"2022-01-13T17:45:00.821584Z","shell.execute_reply.started":"2022-01-13T17:45:00.474413Z","shell.execute_reply":"2022-01-13T17:45:00.820681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# `tf.Data` Generator (Master Class)","metadata":{}},{"cell_type":"code","source":"class TFDataGenerator:\n    def __init__(self, \n                 data, \n                 modeling_in, \n                 shuffle, \n                 aug_lib,\n                 batch_size, \n                 rescale):\n        if modeling_in not in ['2D', '3D']:\n            raise ValueError('modeling_in is not set either 2D or 3D')\n        self.data        = data             # data files \n        self.modeling_in = modeling_in      # 2D or 3D \n        self.shuffle     = shuffle          # true for training \n        self.aug_lib     = aug_lib          # type of augmentation library \n        self.batch_size  = batch_size       # batch size number \n        self.rescale     = rescale          # normalize or not \n        \n    # a convinient function to get 2D data set \n    def get_2D_data(self):\n        self.data = self.data.shuffle(buffer_size = self.batch_size * 100) \\\n                                                        if self.shuffle \\\n                                                        else self.data\n        \n        if self.aug_lib == 'tf' and self.shuffle:\n            # augmentaiton using tf.image.stateless_random* functions \n            # applicable for multiple channel (channel > 3 )\n            # same augmentation would be applied to each modalites: flair, t1w, t1wce, t2w\n            # check the tf_image_augmentation code for details \n            self.data = self.data.map(lambda x, y: (tf_image_augmentation(x), y),\n                                      num_parallel_calls=AUTO).batch(self.batch_size,\n                                                                     drop_remainder=self.shuffle)\n        elif self.aug_lib == 'keras' and self.shuffle:\n            # augmentaion using keras image preprocessing layers \n            # applicable for multiple channels (channel > 3)\n            # [known issue]:  same augmentaion would be applied for all modalites \n            # check the tf_image_augmentation code for details  \n            self.data = self.data.batch(self.batch_size, drop_remainder=self.shuffle) \n            self.data = self.data.map(lambda x, y: (keras_augment(x,training=True),\n                                                    y), \n                                      num_parallel_calls=AUTO)\n        elif not self.shuffle:\n            # no shuffle generally assuming no augmentaion too, so not training \n            # for inference and evaluation \n            self.data = self.data.batch(self.batch_size, drop_remainder=self.shuffle) \n            \n            \n        # rescaling the data for faster convergence \n        if self.rescale:    \n            self.data = self.data.map(lambda x, y: (layers.Rescaling(scale=1./255, offset=0.0)(x), y), \n                                      num_parallel_calls=AUTO)\n            \n        # prefetching the data \n        return self.data.prefetch(-1) \n            \n    # a convinient function to get 3D data set \n    def get_3D_data(self):\n        # augmentation on 3D data set\n        # volumentation is based on albumentation, followed by tf and keras \n        if self.aug_lib == 'volumentations' and self.shuffle:\n            self.data = self.data.map(partial(volumentations_aug), num_parallel_calls=AUTO)\n            self.data = self.data.batch(batch_size, drop_remainder=self.shuffle)\n        elif self.aug_lib == 'tf' and self.shuffle:\n            self.data = self.data.map(lambda x, y: (tf_image_augmentation(x), y),num_parallel_calls=AUTO)\n            self.data = self.data.batch(self.batch_size,drop_remainder=self.shuffle)\n        elif self.aug_lib == 'keras' and self.shuffle:\n            self.data = self.data.batch(self.batch_size, drop_remainder=self.shuffle) \n            self.data = self.data.map(lambda x, y: keras_augment(x, y), num_parallel_calls=AUTO)\n        else:\n            # true for evaluation and inference, no augmentation \n            self.data = self.data.batch(self.batch_size, drop_remainder=self.shuffle)\n        \n        # rescaling the data for faster convergence \n        if self.rescale:    \n            self.data = self.data.map(lambda x, y: (layers.Rescaling(scale=1./255, offset=0.0)(x), y), \n                                      num_parallel_calls=AUTO)\n            \n        # prefetching the data \n        return self.data.prefetch(-1) ","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.823351Z","iopub.execute_input":"2022-01-13T17:45:00.823723Z","iopub.status.idle":"2022-01-13T17:45:00.843268Z","shell.execute_reply.started":"2022-01-13T17:45:00.823669Z","shell.execute_reply":"2022-01-13T17:45:00.842317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Get Train Set with **TensorFlow Augmentation**; Let's visualize it.","metadata":{}},{"cell_type":"code","source":"tf_gen = TFDataGenerator(\n    train_data,\n    modeling_in = modeling_in, \n    shuffle     = True,     \n    aug_lib     = 'tf',  \n    batch_size  = batch_size,   \n    rescale     = True\n)  \n                   \nif modeling_in == '2D':\n    train_generator = tf_gen.get_2D_data()\nelif modeling_in == '3D':\n    train_generator = tf_gen.get_3D_data()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:00.845038Z","iopub.execute_input":"2022-01-13T17:45:00.846209Z","iopub.status.idle":"2022-01-13T17:45:42.400867Z","shell.execute_reply.started":"2022-01-13T17:45:00.846162Z","shell.execute_reply":"2022-01-13T17:45:42.399660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(train_generator))\nprint(x.shape, y.shape, x.numpy().max(), y.numpy().min())  \n\nif modeling_in == '3D':\n    for j in range(input_channel):\n        plt.figure(figsize=(30, 20))\n        for i in range(input_depth):\n            plt.subplot(8, 8, i + 1)\n            plt.imshow(x[0 ,:, :, i, j], cmap='gray')\n            plt.axis(\"off\")\n            plt.title(y[0].numpy())\n        plt.show()\nelif modeling_in == '2D':\n    plt.figure(figsize=(30, 20))\n    for i in range(input_depth*input_channel):\n        plt.subplot(5, 8, i + 1)\n        plt.imshow(x[5 ,:, :, i], cmap='gray')\n        plt.axis(\"off\")\n        plt.title(y[j].numpy())","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:45:42.402296Z","iopub.execute_input":"2022-01-13T17:45:42.402630Z","iopub.status.idle":"2022-01-13T17:46:47.806853Z","shell.execute_reply.started":"2022-01-13T17:45:42.402588Z","shell.execute_reply":"2022-01-13T17:46:47.805839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Get Train Set with **Volumentations Augmentation**; Let's visualize it.","metadata":{}},{"cell_type":"code","source":"tf_gen = TFDataGenerator(\n    train_data,\n    modeling_in = modeling_in, \n    shuffle     = True,     \n    aug_lib     = 'volumentations',\n    batch_size  = batch_size,  \n    rescale     = True\n)   \n                   \nif modeling_in == '2D':\n    train_generator = tf_gen.get_2D_data()\nelif modeling_in == '3D':\n    train_generator = tf_gen.get_3D_data()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:46:47.808506Z","iopub.execute_input":"2022-01-13T17:46:47.809129Z","iopub.status.idle":"2022-01-13T17:46:47.956027Z","shell.execute_reply.started":"2022-01-13T17:46:47.809085Z","shell.execute_reply":"2022-01-13T17:46:47.955106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(train_generator))\nprint(x.shape, y.shape, x.numpy().max(), y.numpy().min())  \n\nif modeling_in == '3D':\n    for j in range(input_channel):\n        plt.figure(figsize=(30, 20))\n        for i in range(input_depth):\n            plt.subplot(8, 8, i + 1)\n            plt.imshow(x[0 ,:, :, i, j], cmap='gray')\n            plt.axis(\"off\")\n            plt.title(y[0].numpy())\n        plt.show()\nelif modeling_in == '2D':\n    plt.figure(figsize=(30, 20))\n    for i in range(input_depth*input_channel):\n        plt.subplot(5, 8, i + 1)\n        plt.imshow(x[5 ,:, :, i], cmap='gray')\n        plt.axis(\"off\")\n        plt.title(y[j].numpy())","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:46:47.958219Z","iopub.execute_input":"2022-01-13T17:46:47.958726Z","iopub.status.idle":"2022-01-13T17:47:02.792950Z","shell.execute_reply.started":"2022-01-13T17:46:47.958674Z","shell.execute_reply":"2022-01-13T17:47:02.792169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Get Train Set with **Keras Augmentation**; Let's visualize it.","metadata":{}},{"cell_type":"code","source":"tf_gen = TFDataGenerator(\n    train_data,\n    modeling_in = modeling_in, \n    shuffle     = True,     \n    aug_lib     = 'keras', \n    batch_size  = batch_size,   \n    rescale     = True\n)   \n                   \nif modeling_in == '2D':\n    train_generator = tf_gen.get_2D_data()\nelif modeling_in == '3D':\n    train_generator = tf_gen.get_3D_data()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:47:02.794347Z","iopub.execute_input":"2022-01-13T17:47:02.794758Z","iopub.status.idle":"2022-01-13T17:47:10.804409Z","shell.execute_reply.started":"2022-01-13T17:47:02.794721Z","shell.execute_reply":"2022-01-13T17:47:10.803441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(train_generator))\nprint(x.shape, y.shape, x.numpy().max(), y.numpy().min())  \n\nif modeling_in == '3D':\n    for j in range(input_channel):\n        plt.figure(figsize=(30, 20))\n        for i in range(input_depth):\n            plt.subplot(8, 8, i + 1)\n            plt.imshow(x[0 ,:, :, i, j], cmap='gray')\n            plt.axis(\"off\")\n            plt.title(y[0].numpy())\n        plt.show()\nelif modeling_in == '2D':\n    plt.figure(figsize=(30, 20))\n    for i in range(input_depth*input_channel):\n        plt.subplot(5, 8, i + 1)\n        plt.imshow(x[5 ,:, :, i], cmap='gray')\n        plt.axis(\"off\")\n        plt.title(y[j].numpy())","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:47:10.806159Z","iopub.execute_input":"2022-01-13T17:47:10.806813Z","iopub.status.idle":"2022-01-13T17:47:23.603877Z","shell.execute_reply.started":"2022-01-13T17:47:10.806745Z","shell.execute_reply":"2022-01-13T17:47:23.599679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Get Validation Set**","metadata":{}},{"cell_type":"code","source":"tf_gen = TFDataGenerator(\n    val_data,\n    modeling_in = modeling_in,  \n    shuffle     = False,     \n    aug_lib     = None,    \n    batch_size  = batch_size,   \n    rescale     = True          \n) \n\nif modeling_in == '2D':\n    valid_generator = tf_gen.get_2D_data()\nelif modeling_in == '3D':\n    valid_generator = tf_gen.get_3D_data()","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:47:23.605737Z","iopub.execute_input":"2022-01-13T17:47:23.606305Z","iopub.status.idle":"2022-01-13T17:47:23.628596Z","shell.execute_reply.started":"2022-01-13T17:47:23.606247Z","shell.execute_reply":"2022-01-13T17:47:23.627540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(valid_generator))\nprint(x.shape, y.shape, x.numpy().max(), y.numpy().min()) \n\nif modeling_in == '3D':\n    for j in range(input_channel):\n        plt.figure(figsize=(30, 20))\n        for i in range(input_depth):\n            plt.subplot(8, 8, i + 1)\n            plt.imshow(x[0 ,:, :, i, j], cmap='gray')\n            plt.axis(\"off\")\n            plt.title(y[0].numpy())\n        plt.show()\nelif modeling_in == '2D':\n    plt.figure(figsize=(30, 20))\n    for i in range(input_depth*input_channel):\n        plt.subplot(5, 8, i + 1)\n        plt.imshow(x[0 ,:, :, i], cmap='gray')\n        plt.axis(\"off\")\n        plt.title(y[0].numpy())","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:47:23.630196Z","iopub.execute_input":"2022-01-13T17:47:23.630609Z","iopub.status.idle":"2022-01-13T17:47:35.299161Z","shell.execute_reply.started":"2022-01-13T17:47:23.630568Z","shell.execute_reply":"2022-01-13T17:47:35.298349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A. Modeling: 2D (Pre-trained Weights)\n\n**UPDATE**: After version 8, 2D modeling is not supported. Please, check version 8 or older if needed.","metadata":{}},{"cell_type":"markdown","source":"---\n\n# B. 3D Model : Pre-trained Weights\n\nThe pre-trained weights of `3D` model is taken from [ZFTurbo/efficientnet_3D](https://github.com/ZFTurbo/efficientnet_3D) and [ZFTurbo/classification_models_3D](https://github.com/ZFTurbo/classification_models_3D). The weight files are uploaded and can be found from here ([efnet-3d](https://www.kaggle.com/ipythonx/keras-3d-efficientnet-imagenet-weights-b0b7)) and here ([classification-3d](https://www.kaggle.com/ipythonx/pretrained-3d-model-weights-keras)) respectively. It contains \n\n```\nEfficientNet B0 - B7 \nResNet [18, 34, 50, 101, 152]\nResNeXt [50, 101]\nSE-ResNet [18, 34, 50, 101, 152]\nSE-ResNeXt [50, 101]\nSE-Net [154]\nDenseNet [121, 169, 201]\nMobileNet\nMobileNet v2\n````","metadata":{}},{"cell_type":"markdown","source":"## `TF.Keras` Model Subclassing: BrainTumorModel3D\n\nIn **3D model** training, we need to use lower batch size for expensive computation cost which is somewhat not wanted. So, we will be trying to use **Gradient Accumulation (GA)** technique to over come this issue. In `TF.Keras`, we can customize the popular `.fit` method easily to implement such various techniques and able to control every training details. If you're new to `keras` model subclassing and custom scratch training, you're welcome to check this [medium documents](https://towardsdatascience.com/model-sub-classing-and-custom-training-loop-from-scratch-in-tensorflow-2-cc1d4f10fb4e). \n\n**Acknowledgement**: I was trying to implement **GA** by customizing the `fit` method but faced few issue and this guy, named [Mr.For Example](https://stackoverflow.com/a/66524901/9215780) on StackOverflow helped me to achieve this. Kudos to him.","metadata":{}},{"cell_type":"code","source":"class BrainTumorModel3D(keras.Model):\n    def __init__(self, \n                 model,           # Sequential or Functional or Subclass Model \n                 n_gradients=1,   # e.g total_batch_size = batch_size * n_gradients\n                 *args, **kwargs):\n        super(BrainTumorModel3D, self).__init__(*args, **kwargs)\n        self.model = model\n        self.n_gradients = tf.constant(n_gradients, dtype=tf.int32)\n        self.n_acum_step = tf.Variable(0, dtype=tf.int32, trainable=False)\n        self.gradient_accumulation = [tf.Variable(tf.zeros_like(v, dtype=tf.float32), \n                                                  trainable=False) \n                                      for v in self.model.trainable_variables]\n\n    # The training step, forward and backward propagation \n    def train_step(self, data):\n        # Adding 1 to num_acum_step till n_gradients and start GA\n        self.n_acum_step.assign_add(1)\n        # Unpack the data \n        images, labels = data\n\n        # Open a GradientTape to record the operations run\n        # during the forward pass, which enables auto-differentiation.\n        with tf.GradientTape() as tape:\n            # Run the forward pass of the layer or model .\n            # The operations that the layer applies\n            # to its inputs are going to be recorded\n            # on the GradientTape.\n            predictions = self.model(images, training=True)\n            # Compute the loss value for this minibatch.\n            loss = self.compiled_loss(labels, predictions)\n        \n        # Compute batch gradients\n        gradients = tape.gradient(loss, self.model.trainable_variables)\n        \n        # Accumulating the batch gradients\n        for i in range(len(self.gradient_accumulation)):\n            self.gradient_accumulation[i].assign_add(gradients[i])\n \n        # If n_acum_step reach the n_gradients then we apply accumulated gradients -\n        # - to update the variables otherwise do nothing\n        tf.cond(tf.equal(self.n_acum_step, self.n_gradients),\n                self.apply_accu_gradients, lambda: None)\n\n        # update metrics\n        self.compiled_metrics.update_state(labels, predictions)\n        return {m.name: m.result() for m in self.metrics}\n    \n    # Function for applying Gradient Accum. \n    def apply_accu_gradients(self):\n        # Apply accumulated gradients\n        self.optimizer.apply_gradients(zip(self.gradient_accumulation, \n                                           self.model.trainable_variables))\n\n        # Reset\n        self.n_acum_step.assign(0)\n        for i in range(len(self.gradient_accumulation)):\n            self.gradient_accumulation[i].assign(\n                tf.zeros_like(self.model.trainable_variables[i],  dtype=tf.float32)\n            )\n\n    # The test step for evaluation and inference \n    def test_step(self, data):\n        # Unpack the data \n        images, labels = data\n        \n        # Run model on inference mode \n        predictions = self.model(images, training=False)\n        \n        # Compute the loss value for this minibatch.\n        loss = self.compiled_loss(labels, predictions)\n        \n        # Update metrics\n        self.compiled_metrics.update_state(labels,  predictions)\n        return {m.name: m.result() for m in self.metrics}\n    \n    # A call funciton needs to be implemented \n    def call(self, inputs, *args, **kwargs):\n        return self.model(inputs)\n    \n    # A custom l2 regularization loss for model to tackle overfit \n    def reg_l2_loss(self, weight_decay = 1e-5):\n        return weight_decay * tf.add_n([\n            tf.nn.l2_loss(v)\n            for v in self.model.trainable_variables\n        ])","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:47:35.301002Z","iopub.execute_input":"2022-01-13T17:47:35.301643Z","iopub.status.idle":"2022-01-13T17:47:35.324883Z","shell.execute_reply.started":"2022-01-13T17:47:35.301603Z","shell.execute_reply":"2022-01-13T17:47:35.323995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let first define the **3D** model with well-know **Functional API** and after that we will pass it to the **Subclass API** model. Note, you can pass any model, whether it's **2D** or any custom model, plug-n-play now.","metadata":{}},{"cell_type":"code","source":"import efficientnet_3D.tfkeras as efn \nfrom classification_models_3D.tfkeras import Classifiers  \n\n# build models \ninput_tensor = keras.Input((input_height, input_width, \n                            input_depth, input_channel), name='input3D')\nmapping3feat = keras.layers.Conv3D(3, (3,3,3), \n                                   strides=(1, 1, 1), \n                                   padding='same', \n                                   use_bias=True)(input_tensor)\n\nmobilenetv2, _ = Classifiers.get('mobilenetv2')\nfeat_ext = mobilenetv2(input_shape=(input_height, input_width,input_depth, 3), \n                       include_top=False, weights='imagenet')\n\noutput = feat_ext(mapping3feat)\noutput = keras.layers.GlobalAveragePooling3D(keepdims=False)(output)\noutput = keras.layers.Dense(1, activation='sigmoid')(output)\nmodel  = keras.Model(input_tensor, output)\nmodel.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-01-13T17:47:35.326152Z","iopub.execute_input":"2022-01-13T17:47:35.327204Z","iopub.status.idle":"2022-01-13T17:48:59.894055Z","shell.execute_reply.started":"2022-01-13T17:47:35.327125Z","shell.execute_reply":"2022-01-13T17:48:59.893069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Compiling the Model**","metadata":{}},{"cell_type":"code","source":"# Passing Functional Model on Subclass API class. \n# In order to get some leverage, i.e. Gradient Accumulation. \nkeras.backend.clear_session()\nmodel3D = BrainTumorModel3D(model, n_gradients = batch_size*accum_step)\n\n# compiling \nmodel3D.compile(\n    loss=tfa.losses.SigmoidFocalCrossEntropy(reduction=tf.keras.losses.Reduction.SUM),\n    optimizer=keras.optimizers.Ftrl(),\n    metrics=[keras.metrics.AUC(), keras.metrics.BinaryAccuracy(name='acc')],\n)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:48:59.895867Z","iopub.execute_input":"2022-01-13T17:48:59.896171Z","iopub.status.idle":"2022-01-13T17:49:00.018823Z","shell.execute_reply.started":"2022-01-13T17:48:59.896127Z","shell.execute_reply":"2022-01-13T17:49:00.017867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom Callback\n\nWe will write a custom callback in `keras` to save the model weights based on some target metrics. ","metadata":{}},{"cell_type":"code","source":"# Define callbacks.\nclass CustomModelCheckpoint(keras.callbacks.Callback):\n    def on_train_begin(self, logs=None):\n        self.val_loss = []\n        self.val_auc  = []\n        self.val_acc  = []\n    def on_epoch_end(self, epoch, logs=None):\n        current_val_loss = logs.get(\"val_loss\")\n        current_val_auc  = logs.get('val_auc')\n        current_val_acc  = logs.get('val_acc')\n        self.val_loss.append(current_val_loss)\n        self.val_auc.append(current_val_auc)\n        self.val_acc.append(current_val_acc)\n        \n        # save based on lowest validation loss \n        if current_val_loss <= min(self.val_loss):\n            print('Find lowest val_loss. Saving model weight.')\n            self.model.save_weights('model_at_val_loss.h5') \n            \n        # save based on highest validation auc \n        if current_val_auc >= max(self.val_auc):\n            print('Find highest val_auc. Saving model weight.')\n            self.model.save_weights('model_at_val_auc.h5') \n        \n        # save based on highest validation acc\n        if current_val_acc >= max(self.val_acc):\n            print('Find highest val_acc. Saving model weight.')\n            self.model.save_weights('model_at_val_acc.h5') \n\n# A custom lr sched \ndef get_lr_callback(batch_size=8):\n    lr_start   = 0.000005\n    lr_max     = 0.00000125 * batch_size\n    lr_min     = 0.000001\n    lr_ramp_ep = 5\n    lr_sus_ep  = 0\n    lr_decay   = 0.8\n   \n    def lrfn(epoch):\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n            \n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n    \n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n            \n        return lr\n\n    return keras.callbacks.LearningRateScheduler(lrfn, verbose=True)","metadata":{"execution":{"iopub.status.busy":"2022-01-13T17:49:00.020612Z","iopub.execute_input":"2022-01-13T17:49:00.020906Z","iopub.status.idle":"2022-01-13T17:49:00.033972Z","shell.execute_reply.started":"2022-01-13T17:49:00.020857Z","shell.execute_reply":"2022-01-13T17:49:00.032794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# epoch params\nepochs = 5\n\n# Running 100 training and 50 validation steps,\n# remove `.take` when training on the full dataset\nmodel3D.fit(\n    train_generator.take(100),\n    epochs=epochs,\n    validation_data=valid_generator.take(50),\n    callbacks=[CustomModelCheckpoint(), \n               tf.keras.callbacks.CSVLogger('history.csv'),\n               get_lr_callback(batch_size)], \n    verbose=1\n)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-01-13T17:49:00.035754Z","iopub.execute_input":"2022-01-13T17:49:00.036069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final Note\n\n- I've made public repository to store and mange the above codes. You can find it [here](https://github.com/innat/BraTS-MGMT-Classification). I'll try to improve it when time permists and of course PRs or Suggestions are warmly welcomed. Some possible todo list are as follows\n    - Optimize Input Pipeline \n    - Add TPU and TFRecord Support \n    - Add 3D-Grad-CAM \n    - Integrate Semantic Segmentation Modeling (task 1, [read more](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/253488))\n- Following notebooks are worth reading. It provides some great insight. \n    - [Connecting voxel spaces](https://www.kaggle.com/boojum/connecting-voxel-spaces)\n    - [Manual VOI LUT on MR Images](https://www.kaggle.com/davidbroberts/manual-voi-lut-on-mr-images)\n    - [Determining MR Image Planes](https://www.kaggle.com/davidbroberts/determining-mr-image-planes)\n    - [Normalized Voxels: Align Planes and Crop](https://www.kaggle.com/ren4yu/normalized-voxels-align-planes-and-crop)\n    - [Brain Tumor - EDA with Animations and Modeling](https://www.kaggle.com/ihelon/brain-tumor-eda-with-animations-and-modeling)\n    - [RSNA-MICCAI BTRC: Understanding The Data](https://www.kaggle.com/arnabs007/part-1-rsna-miccai-btrc-understanding-the-data)\n    - [[Train] Brain Tumor as Video Classification + W&B](https://www.kaggle.com/ayuraj/train-brain-tumor-as-video-classification-w-b)\n    - [RSNA Brain Tumor-Tensorflow TPU TFRecords Train](https://www.kaggle.com/kavehshahhosseini/rsna-brain-tumor-tensorflow-tpu-tfrecords-train)\n- After training with **k-fold**, if you begin to start ensembling, then this may help: [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble/notebook)\n    - Scipy Optimization (**L-BFGS-B**)\n    - Bayesian Optimization\n---\n\n> - **[WIP]**: Training : ✔\n> - **[WIP]**: Inference : ✔ [Check.](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/280107)","metadata":{}},{"cell_type":"markdown","source":"# Miscellaneous \n\nRest of the cell currently on the **Void** zone. These were being used at the early of this competition. But I didn't remove them in case if it helps others.\n\n---\n\n# C. Modeling (InceptionResNet-3D) : No Pre-trained Weights\n\nWe couldn't find any well recognized 3D pretrained weights yet, so here we will build a shallow 3D network. \n\n---\n\n**Version 8 Update**: We won't train `InceptionResNet3D` now, as we have pre-trained 3D models to work with.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf \nfrom tensorflow.keras.layers import * \nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.utils import plot_model\n\nclass Conv3DBatchNorm(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size, padding, strides):\n        super(Conv3DBatchNorm, self).__init__()\n        '''ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # parameters \n        self.nb_filters = nb_filters\n        self.kernel_size = kernel_size \n        self.padding = padding \n        self.strides = strides \n\n        # layers\n        self.conv = tf.keras.layers.Conv3D(self.nb_filters, self.kernel_size, \n                                           self.strides, self.padding)\n        self.bn   = tf.keras.layers.BatchNormalization()\n        \n    def call(self, input_tensor, training=False):\n        x = self.conv(input_tensor)\n        x = self.bn(x, training=training)\n        return tf.nn.relu(x)\n    \n    def get_config(self):\n        return {\n            'nb_filters': self.nb_filters,\n            'kernel_size': self.kernel_size,\n            'padding': self.padding,\n            'strides': self.strides\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Inception3DModule(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size1x1, kernel_size3x3):\n        super(Inception3DModule, self).__init__()\n        '''\n        ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # params \n        self.nb_filters = nb_filters\n        self.kernel_size1x1 = kernel_size1x1\n        self.kernel_size3x3 = kernel_size3x3\n\n        # layers \n        self.conv1 = Conv3DBatchNorm(self.nb_filters, \n                                     kernel_size=self.kernel_size1x1,\n                                     strides=1, padding='same')\n        self.conv2 = Conv3DBatchNorm(self.nb_filters, \n                                     kernel_size=self.kernel_size3x3, \n                                     strides=1, padding='same')\n        self.cat   = tf.keras.layers.Concatenate()\n\n    def call(self, input_tensor, training=False):\n        x_1x1 = self.conv1(input_tensor)\n        x_3x3 = self.conv2(input_tensor)\n        x = self.cat([x_1x1, x_3x3])\n        return tf.nn.relu(x) \n\n    def get_config(self):\n        return {\n            'nb_filters': self.nb_filters,\n            'kernel_size1x1': self.kernel_size1x1,\n            'kernel_size3x3': self.kernel_size3x3\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Identity3DBlock(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size, padding, strides, shortcut = False):\n        super(Identity3DBlock, self).__init__()\n        '''\n        ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # params \n        self.shortcut = shortcut \n        self.nb_filters = nb_filters \n        self.kernel_size = kernel_size \n        self.padding = padding \n        self.strides = strides \n        \n        # layers \n        self.conv1 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.conv2 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.conv3 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.inception = Inception3DModule(self.nb_filters, \n                                           kernel_size1x1 = (1,1,1),\n                                           kernel_size3x3 = (3,3,3))\n    \n    def call(self, input_tensor, training=False):\n        x = self.conv1(input_tensor)\n        x = self.conv2(x)\n        if self.shortcut:\n            srtct = self.inception(input_tensor)\n            srtct = self.conv3(srtct)\n            x = Dropout(0.3)(x)\n            x = Add()([x, srtct])\n            return tf.nn.relu(x)\n        else:\n            x = Add()([x, input_tensor])\n            return tf.nn.relu(x)\n        \n    def get_config(self):\n        return {\n            'shortcut': self.shortcut,\n            'nb_filters': self.nb_filters,\n            'kernel_size': self.kernel_size,\n            'padding': self.padding,\n            'strides': self.strides,\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def InceptionResNet3D(width=128, height=128, depth=32, num_cls=1):\n    inpt = Input((width, height, depth, 1), name='input3D')\n    x = ZeroPadding3D((1, 1, 1))(inpt)\n\n    # conv3d + relu + maxplo3d \n    x = Conv3DBatchNorm(nb_filters=16, kernel_size=(3, 3, 3), \n                        strides=1, padding='valid')(x)\n    x = MaxPooling3D(pool_size=(2, 2, 2), strides=1)(x)\n\n    # inception_resnet_block_ab\n    x = Identity3DBlock(nb_filters=32, kernel_size=(3, 3, 3), \n                        padding='same', strides=1, shortcut=True)(x)\n    x = Identity3DBlock(nb_filters=32, kernel_size=(3, 3, 3), \n                        padding='same', strides=1)(x)\n\n    x = AveragePooling3D(pool_size=(2, 2, 2))(x)\n    x = GlobalAveragePooling3D()(tf.nn.relu(x))\n    x = Dense(num_cls, activation='sigmoid')(x)\n\n    model = Model(inputs=inpt, outputs=x)\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.backend.clear_session()\nmodel = InceptionResNet3D(width=input_width, height=input_height, \n                          depth=input_depth, num_cls=1)\nprint(model.summary())","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}