{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Version 8 Update**\n\n- Update `tf.data` pipeline in a clean modular way, would be easy to tweak\n- Add random data augmentation on each input slice using `tf.image.random_*` module \n- Add random `cutout` augmentation. \n- Add `3D` pre-trained modeling setup: `efficientnet-3d` and others. \n- Add `2D` modeling setup either: `efficientnet-2d` and others.\n- Basic training of `3D` and `2D` modeling with shape of (`312, 312, 4, 1`) and (`312, 312, 4`) respectively. \n\n\n**Note**: The pre-trained weights of `3D` model is taken from [ZFTurbo/efficientnet_3D](https://github.com/ZFTurbo/efficientnet_3D) and [ZFTurbo/classification_models_3D](https://github.com/ZFTurbo/classification_models_3D). The weight files are uploaded and can be found from here ([efnet-3d](https://www.kaggle.com/ipythonx/keras-3d-efficientnet-imagenet-weights-b0b7)) and here ([classification-3d](https://www.kaggle.com/ipythonx/pretrained-3d-model-weights-keras)) respectively. \n\n\n**Version 3 Update**\n\n- Wrap up `keras` sequence generator to `tf.data` API \n- Add data augmentation by using `keras` preprocessing layers \n- Subclass the model for tweaking gradient before update and reg loss \n- Train the model (`InceptionResNet3D`) with shape of (`256, 256, 20, 1`)\n\n\n**Note**: As there are many empty image (`img.mean() == 0`) among four modalities, we found that, the training progress is bit unstable. We need to find out a better and efficient technique for `n` depth 3D model training. Also, we've seen that, among four series or modalities, training with only `T2w` was compratively better. \n\n---\n\n## RSNA-MICCAI Brain Tumor Radiogenomic Classification\n\nIn this baseline example, we will be building a 3D convolutional neural network to predict the status of a genetic biomaker important for brain cancer treatment in the **mpMRI** scans that includes the following four types of images for single study or patient. \n\n```\n- Fluid Attenuated Inversion Recovery (FLAIR)\n- T1-weighted pre-contrast (T1w)\n- T1-weighted post-contrast (T1Gd)\n- T2-weighted (T2)\n```\n\nIn the training data, each study (or each patient id e.g `00000`) belongs to a single label (`train_labels.csv`). Each of this study has further 4 series (`FLAIR`, `T1w`, `T1Gd`, `T2`), and each series has multiple images, in a DICOM format. The file structure is something looks like as follows:\n\n```\nTraining/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n└─── 00001\n│   │ ...\n└─── 00002\n│   │ ...\n```","metadata":{}},{"cell_type":"markdown","source":"## Simple EDA","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt \nimport os, glob, random, cv2, glob, pydicom\nfrom sklearn.model_selection import StratifiedKFold","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-07-23T22:36:45.561445Z","iopub.execute_input":"2021-07-23T22:36:45.561971Z","iopub.status.idle":"2021-07-23T22:36:46.756283Z","shell.execute_reply.started":"2021-07-23T22:36:45.561856Z","shell.execute_reply":"2021-07-23T22:36:46.755408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv'); print(df.shape)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:46.757808Z","iopub.execute_input":"2021-07-23T22:36:46.758168Z","iopub.status.idle":"2021-07-23T22:36:46.790415Z","shell.execute_reply.started":"2021-07-23T22:36:46.75813Z","shell.execute_reply":"2021-07-23T22:36:46.78953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.MGMT_value.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:46.792118Z","iopub.execute_input":"2021-07-23T22:36:46.792441Z","iopub.status.idle":"2021-07-23T22:36:46.803094Z","shell.execute_reply.started":"2021-07-23T22:36:46.792406Z","shell.execute_reply":"2021-07-23T22:36:46.802014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sample_path = '../input/rsna-miccai-brain-tumor-radiogenomic-classification/train'\nlen(os.listdir(train_sample_path)), df.BraTS21ID.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:46.805132Z","iopub.execute_input":"2021-07-23T22:36:46.805623Z","iopub.status.idle":"2021-07-23T22:36:46.855391Z","shell.execute_reply.started":"2021-07-23T22:36:46.805584Z","shell.execute_reply":"2021-07-23T22:36:46.854455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Visualization**","metadata":{}},{"cell_type":"code","source":"def read_dicom_xray(path):\n    data = pydicom.read_file(path).pixel_array\n    data = data - np.min(data)\n    data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:46.85662Z","iopub.execute_input":"2021-07-23T22:36:46.856983Z","iopub.status.idle":"2021-07-23T22:36:46.864624Z","shell.execute_reply.started":"2021-07-23T22:36:46.856947Z","shell.execute_reply":"2021-07-23T22:36:46.863848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# following function took from: https://www.kaggle.com/ihelon/brain-tumor-eda-with-animations-and-modeling?scriptVersionId=68202876&cellId=11\ndef visualize_sample(\n    brats21id, \n    mgmt_value,\n    slice_i,\n    types=(\"FLAIR\", \"T1w\", \"T1wCE\", \"T2w\")\n):\n    plt.figure(figsize=(16, 5))\n    patient_path = os.path.join(\n        train_sample_path, \n        str(brats21id).zfill(5),\n    )\n    for i, t in enumerate(types, 1):\n        t_paths = sorted(\n            glob.glob(os.path.join(patient_path, t, \"*\")), \n            key=lambda x: int(x[:-4].split(\"-\")[-1]),\n        )\n        data = read_dicom_xray(t_paths[int(len(t_paths) * slice_i)])\n        plt.subplot(1, 4, i)\n        plt.imshow(data, cmap=\"gray\")\n        plt.title(f\"{t}\", fontsize=16)\n        plt.axis(\"off\")\n\n    plt.suptitle(f\"MGMT_value: {mgmt_value}\", fontsize=16)\n    plt.show()\n    \n    \nfor i in random.sample(range(df.shape[0]), 2):\n    visualize_sample(df.iloc[i][\"BraTS21ID\"], df.iloc[i][\"MGMT_value\"], slice_i=0.5)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:46.866027Z","iopub.execute_input":"2021-07-23T22:36:46.866403Z","iopub.status.idle":"2021-07-23T22:36:47.904614Z","shell.execute_reply.started":"2021-07-23T22:36:46.866367Z","shell.execute_reply":"2021-07-23T22:36:47.903808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loader\n\n**Note**: For each study, we will pick **5** middle dicom or image samples at cetain interval from each of the four series (`FLAIR`, `T1w`, `T1Gd`, `T2`) for our 3D model input, e.g `input_shape: (None, height, weights, 5*4, 1)`. So, for picking 5 samples from each series, the depth would be `4 * 5` or `20` for each patient id or study id. However, we probabely need to find some better technique to pick samples from each of these four series though. For now, we're doing kinda random stuff. For example: check the following illustration; for the 3D model the input size would be : `height, width, 12, 1`. Also, as we already notice, there're too much black pixel, we will try to address that too. \n\n\n![quick png](https://user-images.githubusercontent.com/17668390/126070190-4a2d8aed-9dd8-4bee-aa15-f95429553a9c.png)\n\n\n","metadata":{}},{"cell_type":"markdown","source":"**Keras Sequence to TensorFlow tf.Data API**\n\nIt's known that, `tf.data` is more efficient when it comes to data pipelines compare to others, e.g. `keras` sequences data generator. Here we will try to wrap up an **existing `keras` sequence generator** to `tf.data` API using [tf.data.Dataset.from_generator function](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). ","metadata":{}},{"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\nfor index, (train_index, val_index) in enumerate(skf.split(X=df.index, \n                                                           y=df.MGMT_value)):\n    df.loc[val_index, 'fold'] = index\n    \nprint(df.groupby(['fold', df.MGMT_value]).size())","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:47.905901Z","iopub.execute_input":"2021-07-23T22:36:47.906372Z","iopub.status.idle":"2021-07-23T22:36:47.925119Z","shell.execute_reply.started":"2021-07-23T22:36:47.906332Z","shell.execute_reply":"2021-07-23T22:36:47.924057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ml pack\nimport tensorflow as tf \n\n# params \nAUTO = tf.data.AUTOTUNE\ninput_height = 312\ninput_width = 312\ninput_depth = 4\nbatch_size = 3\nfold = 0\n\n# data loader \nclass BrainTumorGenerator(tf.keras.utils.Sequence):\n    def __init__(self, dicom_path, data, is_train=True):\n        self.is_train = is_train # to control training/validation/inference part         \n        self.data = data\n        self.dicom_path = dicom_path\n        self.label = self.data['MGMT_value']\n  \n    def __len__(self):\n        return self.data['BraTS21ID'].shape[0]\n    \n    def __getitem__(self, index):\n        patient_ids = f\"{self.dicom_path}/{str(self.data['BraTS21ID'][index]).zfill(5)}/\"\n   \n        channel = []\n        for t in (\"FLAIR\", \"T1w\", \"T1wCE\", \"T2w\"): \n            t_paths = sorted(\n                glob.glob(os.path.join(patient_ids, t, \"*\")), \n                key=lambda x: int(x[:-4].split(\"-\")[-1]),\n            )\n            \n            # pick 15 slices \n            K = 15\n            # computing strt, and end index \n            strt_idx = (len(t_paths) // 2) - (K // 2)\n            end_idx = (len(t_paths) // 2) + (K // 2)\n            # slicing extracting elements with 3 intervals \n            r = t_paths[strt_idx + 3: end_idx + 3: 3]\n    \n            # removing black borders \n            # and add multi-modal features maps / channel depth\n            threshold = 0\n            for i in r:\n                image = self.read_dicom_xray(i)\n                temp_image = image\n                \n                rows = np.where(np.max(temp_image, 0) > threshold)[0]\n                if rows.size:\n                    cols = np.where(np.max(temp_image, 1) > threshold)[0]\n                    image = image[cols[0]: cols[-1] + 1, rows[0]: rows[-1] + 1]\n                else:\n                    image = image[:1, :1]\n                \n                channel.append(cv2.resize(image, (input_height, input_width)))\n                break # remove it for r-times frames for each series\n                    \n        if self.is_train:\n            return np.array(channel).T, self.label.iloc[index,]\n        else:\n            return np.array(channel).T\n    \n    def read_dicom_xray(self, path):\n        data = pydicom.read_file(path).pixel_array\n        data = data - np.min(data)\n        data = data / np.max(data)\n        data = (data * 255).astype(np.uint8)\n        return data","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:47.928424Z","iopub.execute_input":"2021-07-23T22:36:47.928777Z","iopub.status.idle":"2021-07-23T22:36:51.800044Z","shell.execute_reply.started":"2021-07-23T22:36:47.928743Z","shell.execute_reply":"2021-07-23T22:36:51.799192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fold_generator(fold):\n    # for way one - data generator\n    train_labels = df[df.fold != fold].reset_index(drop=True)\n    val_labels = df[df.fold == fold].reset_index(drop=True)\n    \n    return (\n        BrainTumorGenerator(train_sample_path, train_labels),\n        BrainTumorGenerator(train_sample_path, val_labels)\n    )\n\n# first fold \ntrain_gen, val_gen = fold_generator(fold)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:51.801947Z","iopub.execute_input":"2021-07-23T22:36:51.802294Z","iopub.status.idle":"2021-07-23T22:36:51.823852Z","shell.execute_reply.started":"2021-07-23T22:36:51.802258Z","shell.execute_reply":"2021-07-23T22:36:51.823115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## `tf.data` Pipelines","metadata":{}},{"cell_type":"code","source":"# augmentations for train set\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.layers.experimental.preprocessing import (RandomFlip,\n                                                                RandomRotation, \n                                                                RandomTranslation)\n\n#  keras augmentation layers \naugmentation_layers = tf.keras.Sequential(\n    [\n        RandomRotation(factor=0.01),\n        RandomTranslation(height_factor=0.0, width_factor=0.1),\n    ],\n    name='keras_augment_layers'\n)\n\n\n# manual preprocessing with more augmentation \ndef preprocessing_image(img, augment=True):   \n    img = tf.cast(img, tf.float32) / 255.0\n\n    # only true for train set \n    if augment:\n        # augment each slices \n        # todo: integrate better technique \n        splitted_img = tf.split(img, input_depth, axis=-1)\n\n        augment_img = []\n        for each_img in splitted_img:\n            img = tf.repeat(each_img, repeats=3, axis=-1)\n            img = tf.image.random_flip_left_right(img)\n            img = tf.image.random_saturation(img, 0.9, 1.3)\n            img = tf.image.random_contrast(img, 0.8, 1.2)\n            img = tf.image.random_brightness(img, 0.2)\n            img, _, _ = tf.split(img, 3, axis=-1)\n            img = tfa.image.random_cutout(tf.expand_dims(img, 0),\n                                          mask_size=(20, 20), \n                                          constant_values=0)\n            augment_img.append(img)\n            \n        img = tf.concat(augment_img, axis=-1)\n    img = tf.reshape(img, [input_height, input_width, input_depth])\n    return img","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:51.825097Z","iopub.execute_input":"2021-07-23T22:36:51.825416Z","iopub.status.idle":"2021-07-23T22:36:53.949796Z","shell.execute_reply.started":"2021-07-23T22:36:51.825383Z","shell.execute_reply":"2021-07-23T22:36:53.948964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_data_generator(data, is_train=False, modeling_in='3D',\n                       shuffle=True, augment=False, \n                       repeat=True, batch_size=32):\n    if repeat: \n        data = data.repeat()\n    \n    if shuffle:\n        data = data.shuffle(batch_size * 10)\n        \n    data = data.map(lambda x, y: (preprocessing_image(x, augment), y),\n                    num_parallel_calls=AUTO)\n    data = data.batch(batch_size, drop_remainder=is_train)\n    \n    if shuffle:\n        data = data.map(lambda x, y: (augmentation_layers(x), y), \n                        num_parallel_calls=AUTO) \n    \n    if modeling_in == '3D':\n        data = data.map(lambda x, y: (tf.expand_dims(x, axis=-1), y),\n                        num_parallel_calls=AUTO)\n        data = data.prefetch(AUTO)\n        return data \n    elif modeling_in == '2D':\n        data = data.prefetch(AUTO)\n        return data \n    else:\n        raise ValueError('volume is not set either 2D or 3D')","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:53.951024Z","iopub.execute_input":"2021-07-23T22:36:53.951341Z","iopub.status.idle":"2021-07-23T22:36:53.960819Z","shell.execute_reply.started":"2021-07-23T22:36:53.951308Z","shell.execute_reply":"2021-07-23T22:36:53.959006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train Set (2D)","metadata":{}},{"cell_type":"code","source":"# wrapping sequence generator to tf.data API \ntrain_data = tf.data.Dataset.from_generator(\n    lambda: map(tuple, train_gen),\n    (tf.float32, tf.float32),\n    (\n        tf.TensorShape([input_height, input_width, input_depth]),\n        tf.TensorShape([]),\n    ),\n)\n\n# generate train sets \ntrain_generator = get_data_generator(train_data, \n                                     is_train=True, repeat=False, \n                                     shuffle=True, modeling_in='2D', \n                                     augment=True, batch_size=batch_size)\n\n\n# visualization \nx, y = next(iter(train_generator))\nprint(x.shape, y.shape)  \nplt.figure(figsize=(35, 15))\nfor i in range(input_depth):\n    plt.subplot(1, input_depth, i + 1)\n    plt.imshow(x[1 ,:, :, i], cmap=\"gray\")\n    plt.axis(\"off\")\n    plt.title(y[1].numpy())","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:36:53.962421Z","iopub.execute_input":"2021-07-23T22:36:53.96291Z","iopub.status.idle":"2021-07-23T22:37:01.30327Z","shell.execute_reply.started":"2021-07-23T22:36:53.962873Z","shell.execute_reply":"2021-07-23T22:37:01.302236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Validation Set (2D)","metadata":{}},{"cell_type":"code","source":"# wrapping sequence generator to tf.data API \nval_data = tf.data.Dataset.from_generator(\n    lambda: map(tuple, val_gen),\n    (tf.float32, tf.float32),\n    (\n        tf.TensorShape([input_height, input_width, input_depth]),\n        tf.TensorShape([]),\n    ),\n)\n\n# generate validation sets \nvalid_generator = get_data_generator(val_data, is_train=True, \n                                     shuffle=False, repeat=False,\n                                     modeling_in='2D', augment=False, \n                                     batch_size=batch_size)\n\n# visualization \nx, y = next(iter(valid_generator))\nprint(x.shape, y.shape)  \nplt.figure(figsize=(35, 15))\nfor i in range(input_depth):\n    plt.subplot(1, input_depth, i + 1)\n    plt.imshow(x[0 ,:, :, i], cmap=\"gray\")\n    plt.axis(\"off\")\n    plt.title(y[0].numpy())","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:37:01.304839Z","iopub.execute_input":"2021-07-23T22:37:01.305176Z","iopub.status.idle":"2021-07-23T22:37:02.953496Z","shell.execute_reply.started":"2021-07-23T22:37:01.305141Z","shell.execute_reply":"2021-07-23T22:37:02.952563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A. Modeling: 2D (Pre-trained Weights)","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import Input, Model \nfrom tensorflow.keras.layers import Conv2D, GlobalAveragePooling2D, Dense\nfrom tensorflow.keras.applications import *\n\npre_wg = '../input/keras-pretrained-imagenet-weights/densenet121_imagenet_1000_no_top.h5'\n\ninput_dim = (input_height, input_width, input_depth)\ninput_tensor = Input(input_dim, name='input2d')\nefnet = DenseNet121(weights=pre_wg, \n                       include_top = False, \n                       input_shape=(input_height, input_width, 3))\nmapping3feat = Conv2D(3, (3, 3), padding='same', use_bias=False)(input_tensor)\n\noutput = efnet(mapping3feat)\noutput = GlobalAveragePooling2D()(output)\noutput = Dense(1, activation='sigmoid')(output)\n\ntf.keras.backend.clear_session()\nmodel = Model(input_tensor, output)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:37:02.955039Z","iopub.execute_input":"2021-07-23T22:37:02.955388Z","iopub.status.idle":"2021-07-23T22:37:07.147661Z","shell.execute_reply.started":"2021-07-23T22:37:02.955351Z","shell.execute_reply":"2021-07-23T22:37:07.146879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow import keras \nfrom tensorflow.keras.optimizers import Adam, SGD, RMSprop\nfrom tensorflow_addons.optimizers import RectifiedAdam, Lookahead\n\n# compiling \nmodel.compile(\n    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n    optimizer=Adam(learning_rate=1e-3),\n    metrics=[tf.keras.metrics.AUC(), \n             tf.keras.metrics.BinaryAccuracy(name='bacc')],\n)\n\n# define callbacks.\ncheckpoint_cb = keras.callbacks.ModelCheckpoint(\n    \"model.h5\", monitor='val_auc', \n    mode='max', save_best_only=True\n)\n\n\n# fitting the model \nepochs = 5\nmodel.fit(\n    train_generator, \n    epochs=epochs,\n    validation_data=valid_generator, \n    callbacks=[checkpoint_cb], verbose=2\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T22:37:07.148928Z","iopub.execute_input":"2021-07-23T22:37:07.149255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n## Train Set (3D)","metadata":{}},{"cell_type":"code","source":"# wrapping sequence generator to tf.data API \ntrain_data = tf.data.Dataset.from_generator(\n    lambda: map(tuple, train_gen),\n    (tf.float32, tf.float32),\n    (\n        tf.TensorShape([input_height, input_width, input_depth]),\n        tf.TensorShape([]),\n    ),\n)\n\n# generate train sets \ntrain_generator = get_data_generator(train_data, \n                                     is_train=True, repeat=False, \n                                     shuffle=True, modeling_in='3D', \n                                     augment=True, batch_size=batch_size)\n\n\n# visualization \nfor j, (x, y) in enumerate(train_generator):\n    print(x.shape, y.shape)\n    plt.figure(figsize=(30, 7))\n    for i in range(input_depth):\n        plt.subplot(1, input_depth, i + 1)\n        plt.imshow(x[0 ,:, :, i], cmap=\"gray\")\n        plt.title(y[0].numpy())\n        plt.axis(\"off\")\n    plt.show()\n    if j == 2: break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Validation Set (3D)","metadata":{}},{"cell_type":"code","source":"# wrapping sequence generator to tf.data API \nval_data = tf.data.Dataset.from_generator(\n    lambda: map(tuple, val_gen),\n    (tf.float32, tf.float32),\n    (\n        tf.TensorShape([input_height, input_width, input_depth]),\n        tf.TensorShape([]),\n    ),\n)\n\n# generate validation sets \nvalid_generator = get_data_generator(val_data, is_train=True, \n                                     shuffle=False, repeat=False,\n                                     modeling_in='3D', augment=False, \n                                     batch_size=batch_size)\n\n\n# visualization \nfor j, (x, y) in enumerate(valid_generator):\n    print(x.shape, y.shape)\n    plt.figure(figsize=(30, 7))\n    for i in range(input_depth):\n        plt.subplot(1, input_depth, i + 1)\n        plt.imshow(x[0 ,:, :, i], cmap=\"gray\")\n        plt.title(y[0].numpy())\n        plt.axis(\"off\")\n    plt.show()\n    if j == 2: break","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# B. Modeling (InceptionResNet-3D) : No Pre-trained Weights\n\nWe couldn't find any well recognized 3D pretrained weights yet, so here we will build a shallow 3D network. \n\n---\n\n**Version 8 Update**: We won't train `InceptionResNet3D` now, as we have pre-trained 3D models to work with.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf \nfrom tensorflow.keras.layers import * \nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.utils import plot_model\n\nclass Conv3DBatchNorm(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size, padding, strides):\n        super(Conv3DBatchNorm, self).__init__()\n        '''\n        ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # parameters \n        self.nb_filters = nb_filters\n        self.kernel_size = kernel_size \n        self.padding = padding \n        self.strides = strides \n\n        # layers\n        self.conv = tf.keras.layers.Conv3D(self.nb_filters, self.kernel_size, \n                                           self.strides, self.padding)\n        self.bn   = tf.keras.layers.BatchNormalization()\n        \n    def call(self, input_tensor, training=False):\n        x = self.conv(input_tensor)\n        x = self.bn(x, training=training)\n        return tf.nn.relu(x)\n    \n    def get_config(self):\n        return {\n            'nb_filters': self.nb_filters,\n            'kernel_size': self.kernel_size,\n            'padding': self.padding,\n            'strides': self.strides\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Inception3DModule(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size1x1, kernel_size3x3):\n        super(Inception3DModule, self).__init__()\n        '''\n        ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # params \n        self.nb_filters = nb_filters\n        self.kernel_size1x1 = kernel_size1x1\n        self.kernel_size3x3 = kernel_size3x3\n\n        # layers \n        self.conv1 = Conv3DBatchNorm(self.nb_filters, kernel_size=self.kernel_size1x1,\n                                     strides=1, padding='same')\n        self.conv2 = Conv3DBatchNorm(self.nb_filters, kernel_size=self.kernel_size3x3, \n                                     strides=1, padding='same')\n        self.cat   = tf.keras.layers.Concatenate()\n\n    def call(self, input_tensor, training=False):\n        x_1x1 = self.conv1(input_tensor)\n        x_3x3 = self.conv2(input_tensor)\n        x = self.cat([x_1x1, x_3x3])\n        return tf.nn.relu(x) \n\n    def get_config(self):\n        return {\n            'nb_filters': self.nb_filters,\n            'kernel_size1x1': self.kernel_size1x1,\n            'kernel_size3x3': self.kernel_size3x3\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Identity3DBlock(tf.keras.layers.Layer):\n    def __init__(self, nb_filters, kernel_size, padding, strides, shortcut = False):\n        super(Identity3DBlock, self).__init__()\n        '''\n        ref: https://gist.github.com/innat/1bc1307044a10a66a63eae9a71294117\n        '''\n        # params \n        self.shortcut = shortcut \n        self.nb_filters = nb_filters \n        self.kernel_size = kernel_size \n        self.padding = padding \n        self.strides = strides \n        \n        # layers \n        self.conv1 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.conv2 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.conv3 = Conv3DBatchNorm(self.nb_filters, self.kernel_size, \n                                     self.padding, self.strides)\n        self.inception = Inception3DModule(self.nb_filters, \n                                           kernel_size1x1 = (1,1,1),\n                                           kernel_size3x3 = (3,3,3))\n    \n    def call(self, input_tensor, training=False):\n        x = self.conv1(input_tensor)\n        x = self.conv2(x)\n        if self.shortcut:\n            srtct = self.inception(input_tensor)\n            srtct = self.conv3(srtct)\n            x = Dropout(0.3)(x)\n            x = Add()([x, srtct])\n            return tf.nn.relu(x)\n        else:\n            x = Add()([x, input_tensor])\n            return tf.nn.relu(x)\n        \n    def get_config(self):\n        return {\n            'shortcut': self.shortcut,\n            'nb_filters': self.nb_filters,\n            'kernel_size': self.kernel_size,\n            'padding': self.padding,\n            'strides': self.strides,\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def InceptionResNet3D(width=128, height=128, depth=32, num_cls=1):\n    inpt = Input((width, height, depth, 1), name='input3D')\n    x = ZeroPadding3D((1, 1, 1))(inpt)\n\n    # conv3d + relu + maxplo3d \n    x = Conv3DBatchNorm(nb_filters=16, kernel_size=(3, 3, 3), \n                        strides=1, padding='valid')(x)\n    x = MaxPooling3D(pool_size=(2, 2, 2), strides=1)(x)\n\n    # inception_resnet_block_ab\n    x = Identity3DBlock(nb_filters=32, kernel_size=(3, 3, 3), \n                        padding='same', strides=1, shortcut=True)(x)\n    x = Identity3DBlock(nb_filters=32, kernel_size=(3, 3, 3), \n                        padding='same', strides=1)(x)\n\n    x = AveragePooling3D(pool_size=(2, 2, 2))(x)\n    x = GlobalAveragePooling3D()(tf.nn.relu(x))\n    x = Dense(num_cls, activation='sigmoid')(x)\n\n    model = Model(inputs=inpt, outputs=x)\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.backend.clear_session()\nmodel = InceptionResNet3D(width=input_width, height=input_height, \n                          depth=input_depth, num_cls=1)\nprint(model.summary())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# C. 3D Model : Pre-trained Weights\n\nThe pre-trained weights of `3D` model is taken from [ZFTurbo/efficientnet_3D](https://github.com/ZFTurbo/efficientnet_3D) and [ZFTurbo/classification_models_3D](https://github.com/ZFTurbo/classification_models_3D). The weight files are uploaded and can be found from here ([efnet-3d](https://www.kaggle.com/ipythonx/keras-3d-efficientnet-imagenet-weights-b0b7)) and here ([classification-3d](https://www.kaggle.com/ipythonx/pretrained-3d-model-weights-keras)) respectively. It contains \n\n```\nEfficientNet B0 - B7 \nResNet [18, 34, 50, 101, 152]\nResNeXt [50, 101]\nSE-ResNet [18, 34, 50, 101, 152]\nSE-ResNeXt [50, 101]\nSE-Net [154]\nDenseNet [121, 169, 201]\nMobileNet\nMobileNet v2\n````","metadata":{}},{"cell_type":"code","source":"!pip install /kaggle/input/kerasapplications -q\n!pip install /kaggle/input/pretrained-3d-model-weights-keras/classification_model3d_keras/classification_models_3D-1.0.2-py3-none-any.whl\n!pip install /kaggle/input/keras-3d-efficientnet-imagenet-weights-b0b7/efficientnet3d_keras/efficientnet_3D-1.0.1-py3-none-any.whl","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subclassing: BrainTumorModel3D","metadata":{}},{"cell_type":"code","source":"class BrainTumorModel3D(tf.keras.Model):\n    def __init__(self, model):\n        super(BrainTumorModel3D, self).__init__()\n        self.model = model\n    \n    def train_step(self, data):\n        images, labels = data\n\n        with tf.GradientTape() as tape:\n            predictions = self.model(images, training=True)\n            loss = self.compiled_loss(labels, predictions)\n            \n        gradients = tape.gradient(loss, self.model.trainable_variables)\n        self.optimizer.apply_gradients(zip(gradients, self.model.trainable_variables))\n        self.compiled_metrics.update_state(labels, predictions)\n        return {m.name: m.result() for m in self.metrics}\n\n    def test_step(self, data):\n        images, labels = data\n        predictions = self.model(images, training=False)\n        loss = self.compiled_loss(labels, predictions, \n                                  regularization_losses=[self.reg_l2_loss()])\n        self.compiled_metrics.update_state(labels, predictions)\n        return {m.name: m.result() for m in self.metrics}\n\n    def call(self, inputs, *args, **kwargs):\n        return self.model(inputs)\n    \n    def reg_l2_loss(self, weight_decay = 1e-5):\n        return weight_decay * tf.add_n([\n            tf.nn.l2_loss(v)\n            for v in self.model.trainable_variables\n        ])\n    \n    def build_graph(self):\n        x = Input(shape=((input_height, input_width, input_depth,1)))\n        return Model(inputs=[x], outputs=self.call(x))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import efficientnet_3D.tfkeras as efn # include_top = false \nfrom classification_models_3D.tfkeras import Classifiers # include_top = true \n\npre_wg = '../input/keras-3d-efficientnet-imagenet-weights-b0b7/efficientnet3d_keras/efficientnet-b0_inp_channel_3_tch_0_top_False.h5'\n\n# build models \ninput_tensor = Input((input_height, input_width, \n                      input_depth, 1), name='input3D')\nmapping3feat = Conv3D(3, (3,3,3), strides=(1, 1, 1), \n                                      padding='same', \n                                      use_bias=False)(input_tensor)\n\nefnet = efn.EfficientNetB0(input_shape=(input_height, \n                                        input_width, \n                                        input_depth, 3), \n                           weights=pre_wg)\n\noutput = efnet(mapping3feat)\noutput = GlobalAveragePooling3D()(output)\noutput = Dense(1, activation='sigmoid')(output)\n\nmodel = Model(input_tensor, output)\nprint(model.summary())\nplot_model(model, show_shapes=True, expand_nested=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# subclassing the model in order to tweak the gradient if needed (todo)\nmodel3D = BrainTumorModel3D(model)\n\n# compiling \nmodel3D.compile(\n    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n    optimizer=Adam(learning_rate=1e-3),\n    metrics=[tf.keras.metrics.AUC(), \n             tf.keras.metrics.BinaryAccuracy(name='bacc')],\n)\n\n# Define callbacks.\n# todo: experiment and add more \ncheckpoint_cb = keras.callbacks.ModelCheckpoint(\n    \"model.h5\", monitor='val_auc', \n    mode='max', save_best_only=True\n)\n\n\n# Train the model\nepochs = 5\nmodel3D.fit(\n    train_generator,\n    epochs=epochs,\n    validation_data=valid_generator,\n    callbacks=[checkpoint_cb], verbose=2\n)","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```\n- [WIP]: Training : ✔\n- [WIP]: Inference\n```","metadata":{}}]}