{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](https://i.ibb.co/tbW6pxF/cover2-01.jpg)\n\n

[TPU] RSNA Keras 3D CNN Voxel Classifier Train

\n\nOverview\n
\n
\n ✔️ Fast Training with Large Voxel Batch Sizes on TPU
\n ✔️ TPU Mixed Precision
\n ✔️ Custom Keras Training Loop
\n ✔️ 3D Augmentations on TPU
\n   ✔️ Flip
\n   ✔️ Gamma
\n   ✔️ Brightness
\n   ✔️ Contrast
\n   ✔️ Cutout
\n   ✔️ Rotate
\n   ✔️ Random Resized Crop
\n   ✔️ Blur
\n ✔️ Experiment Tracking & Interactive Visualizations with Weights & Biases
\n\n
\n\nThis competition presents a unique challenge of predicting the status of a MGMT (O[6]-methylguanine-DNA methyltransferase genetic biomarker, important for the success of brain cancer treatment as a favorable prognostic factor and a strong predictor of responsiveness to chemotherapy.\n
\n
\nWith this challenge, we could potentially build solutions that can minimize the number of exploratory invasive procedures for a diagnosis, to target and refine the treatments required, by making use of MRI imaging techiniques.\n
\n
\nMultiples exploratory surgeries for the characterization of tumour genetic biomarkers is both detrimental and may delay the right treatment upto several weeks, which is cutting short the average span of 5 years for people diagnosed with brain cancer.\n
\n
\nLet's look at how we can train 3D MRI scan voxels on TPUs. The following processing were performed to create the voxels:
\n\n ✔️ Align Scans Across the Volume & Crop
\n ✔️ Filter Out Slices with Less Information
\n ✔️ CLAHE Contrast Enhancement & Normalization Across the 3D Volume
\n ✔️ Resampled to size 64 x 256 x 256 for Optimal Training\n
\n
\n\nAugmentations are applied by preserving the proper geometric alignment between planes and we make sure to uniformly apply the same operation to every slice. Please refer to the dataset generation and augmentation notebook for more details.\n
\n
\nTPUs complete training at a quicker pace and enables fitting more batches into memory comapared to the small volumes and batches that can be fit on GPUs. This is even more useful as entire volumes need to be fit into memory, rather than single image samples.\n
\n
\nWe also make use of mixed precision to make the training run faster and consume less memory, by using 16-bit floating-point calculations wherever it's appropriate.\n
\n
\n\n\n

🏷️ Notebook with Dataset Generation & Augmentations

\n\n  [RSNA 3D CLAHE Voxels + TPU 3D Augmentations🌃🚅](https://www.kaggle.com/sreevishnudamodaran/rsna-3d-clahe-voxels-tpu-3d-augmentations)
\n\n

🏷️ Dataset with Processed 64 x 256 x 256 Voxels

\n\n  [RSNA Processed Voxels 64x256x256](https://www.kaggle.com/sreevishnudamodaran/rsna-processed-voxels-64x256x256)
\n\n

🏷️ Dataset with Processed 64 x 256 x 256 Voxels with CLAHE

\n\n  [RSNA Processed Voxels 64x256x256 CLAHE](https://www.kaggle.com/sreevishnudamodaran/rsna-processed-voxels-64x256x256-clahe)\n\n
\n
\n\"Ask
\n \"Ask\n
","metadata":{"execution":{"iopub.status.busy":"2021-08-29T11:05:06.224396Z","iopub.execute_input":"2021-08-29T11:05:06.225028Z","iopub.status.idle":"2021-08-29T11:05:06.330191Z","shell.execute_reply.started":"2021-08-29T11:05:06.22492Z","shell.execute_reply":"2021-08-29T11:05:06.329163Z"}}},{"cell_type":"markdown","source":"⚗️ Imports & Setup","metadata":{}},{"cell_type":"code","source":"!pip install /kaggle/input/kerasapplications -q\n!pip install /kaggle/input/keras-3d-model-and-3d-augmentation/classification_models_3D-1.0.2-py3-none-any.whl -q","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-12T09:06:10.927981Z","iopub.execute_input":"2021-09-12T09:06:10.928336Z","iopub.status.idle":"2021-09-12T09:06:31.681840Z","shell.execute_reply.started":"2021-09-12T09:06:10.928229Z","shell.execute_reply":"2021-09-12T09:06:31.680949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import itertools\nimport os\nimport random\nimport gc\nimport glob\nimport re\nimport math\nimport time\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport pprint\nimport matplotlib.pylab as plt\nimport matplotlib.ticker as mtick\nimport seaborn as sns\n\nfrom sklearn.model_selection import train_test_split, GroupKFold\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nfrom tensorflow import keras\nimport tensorflow_addons as tfa\nfrom tensorflow.keras.utils import Progbar\n\nimport tensorflow.experimental.numpy as tnp\n\nfrom kaggle_datasets import KaggleDatasets\nfrom kaggle_secrets import UserSecretsClient\nfrom IPython.display import IFrame\nfrom IPython.core.display import display, HTML\nimport imageio","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-09-12T09:06:31.683746Z","iopub.execute_input":"2021-09-12T09:06:31.683978Z","iopub.status.idle":"2021-09-12T09:06:38.600683Z","shell.execute_reply.started":"2021-09-12T09:06:31.683949Z","shell.execute_reply":"2021-09-12T09:06:38.599761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Config:\n seed = 42\n job = 1\n num_classes = 1\n # dataset = '/kaggle/input/rsna-processed-voxels-64x256x256' # No CLAHE\n dataset = '/kaggle/input/rsna-processed-voxels-64x256x256-clahe'\n input_dims = (64, 256, 256)\n voxel_dtype = tf.uint8\n scan_types = ['FLAIR', 'T1w', 'T1wCE', 'T2w']\n batch_size = 32\n kfold = 5\n n_epochs = 15\n lr = 0.00005\n fp16= True\n label_smoothing = 0.1\n wandb_project = 'RSNA-train-public'\n \ncfg = Config()","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:38.602201Z","iopub.execute_input":"2021-09-12T09:06:38.602545Z","iopub.status.idle":"2021-09-12T09:06:38.609457Z","shell.execute_reply.started":"2021-09-12T09:06:38.602506Z","shell.execute_reply":"2021-09-12T09:06:38.608169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(SEED):\n os.environ['PYTHONHASHSEED']=str(SEED)\n random.seed(SEED)\n np.random.seed(SEED)\n tf.random.set_seed(SEED)\n os.environ['TF_CUDNN_DETERMINISTIC'] = str(SEED)\n\nseed_everything(cfg.seed)","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:38.613201Z","iopub.execute_input":"2021-09-12T09:06:38.613631Z","iopub.status.idle":"2021-09-12T09:06:38.624322Z","shell.execute_reply.started":"2021-09-12T09:06:38.613591Z","shell.execute_reply":"2021-09-12T09:06:38.623554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"split = 'train'\ntrain_voxels = sorted(glob.glob(f\"{cfg.dataset}/voxels/*/*.npy\"))\n\ndf_train = pd.DataFrame(train_voxels, columns=['voxel_paths'])\ndf_train['BraTS21ID'] = df_train.voxel_paths.map(lambda path:path.split('/')[-1].strip('.npy'))\ndf_train['scan_type'] = df_train.voxel_paths.map(lambda path:path.split('/')[-2])\ndf_train.sample(3)","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:38.625807Z","iopub.execute_input":"2021-09-12T09:06:38.626039Z","iopub.status.idle":"2021-09-12T09:06:40.233595Z","shell.execute_reply.started":"2021-09-12T09:06:38.626015Z","shell.execute_reply":"2021-09-12T09:06:40.232904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_labels = pd.read_csv('../input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv',\n dtype={'BraTS21ID':np.object,\n 'MGMT_value':np.int32})\ndf_train = df_train.set_index('BraTS21ID').join(df_train_labels.set_index('BraTS21ID'), on='BraTS21ID', how='left')\ndf_train = df_train.reset_index()\ndf_train.to_csv(\"/kaggle/working/train_df_meta.csv\")\ndf_train.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:40.234860Z","iopub.execute_input":"2021-09-12T09:06:40.235068Z","iopub.status.idle":"2021-09-12T09:06:40.290020Z","shell.execute_reply.started":"2021-09-12T09:06:40.235044Z","shell.execute_reply":"2021-09-12T09:06:40.289146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"K-fold Split","metadata":{}},{"cell_type":"code","source":"scan_type = 'FLAIR'\n\ndf_kfold = df_train.drop_duplicates().reset_index(drop=True)\n\nkfold = cfg.kfold\ndf_kfold['fold'] = -1\ngroup_kfold = GroupKFold(n_splits = kfold)\n\nfor fold, (train_index, val_index) in enumerate(group_kfold.split(df_kfold[['BraTS21ID',\n 'scan_type',\n 'voxel_paths']],\n df_kfold['MGMT_value'],\n groups=df_kfold.BraTS21ID.tolist())):\n df_kfold.loc[val_index, 'fold'] = fold\n \ndisplay(df_kfold.head(3))\ndf_kfold.to_csv(\"/kaggle/working/df_kfold.csv\")\n\nfor fold in range(cfg.kfold):\n train_paths = df_kfold[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] != fold)].voxel_paths.unique()\n valid_paths = df_kfold[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] == fold)].voxel_paths.unique()\n\n print(f\"Fold:{fold}\\nSplit Counts\\nTrain Voxels:\\t\\t{len(train_paths)}\\nVal Voxels:\\t\\t{len(valid_paths)}\",\n end='\\n\\n')","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:40.291333Z","iopub.execute_input":"2021-09-12T09:06:40.291559Z","iopub.status.idle":"2021-09-12T09:06:40.359033Z","shell.execute_reply.started":"2021-09-12T09:06:40.291531Z","shell.execute_reply":"2021-09-12T09:06:40.358083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create Dataloader with TPU Augmentations","metadata":{}},{"cell_type":"code","source":"def get_npy_header_cnt(sample_voxel, dtype=tf.uint8):\n img = np.load(sample_voxel)\n tf_img = tf.io.read_file(sample_voxel)\n tf_img = tf.io.decode_raw(tf_img, tf.uint8)\n return tf_img.shape[0]-int(np.prod(img.shape))\n\nnpy_header_size = get_npy_header_cnt(df_kfold.voxel_paths[0],\n cfg.voxel_dtype)\nnpy_header_size","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:40.360520Z","iopub.execute_input":"2021-09-12T09:06:40.360777Z","iopub.status.idle":"2021-09-12T09:06:40.538008Z","shell.execute_reply.started":"2021-09-12T09:06:40.360740Z","shell.execute_reply":"2021-09-12T09:06:40.537201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"voxel_dtype = cfg.voxel_dtype\nseed = cfg.seed\n\nFLIP = 0.5 # @params: probability\nCONTRAST = (0.7, 1.3, 0.5) # @params: (minval, maxval, probability)\nBRIGHTNESS = (0.2, 0.4) # @params: (delta, probability)\nGAMMA = (0.8, 1.2, 0.5) # @params: (minval, maxval, probability)\nROTATE = (20, 0.5) # @params: (maxangle, probability)\nRANDOM_CROP = (180, 180, 0.4) # @params: (min_width, min_height, probability)\nCUTOUT = ((15, 15), 15, 0.4) # @params: ((mask_dim0, mask_dim1), max_num_holes, probability)\nBLUR = ([5, 5], 10, 0.4) # @params: ((filter_dim0, filter_dim1), sigma, probability)\n\ndef build_decoder(with_labels=True, target_size=(64, 256, 256), ext='npy'):\n def decode(path):\n if ext == 'npy':\n voxel = tf.io.read_file(path)\n voxel = tf.io.decode_raw(voxel, voxel_dtype)\n voxel = voxel[npy_header_size:]\n else:\n raise ValueError(\"voxel extension not supported\")\n \n voxel = tf.cast(voxel, tf.float32)/255.0\n voxel = tf.reshape(voxel, target_size)\n voxel = tf.expand_dims(voxel, axis=-1)\n\n return voxel\n \n def decode_with_labels(path, label):\n return decode(path), label\n \n return decode_with_labels if with_labels else decode\n\ndef random_rotate3D(voxel, limit=90, p=0.5):\n if tf.random.uniform(()) < p:\n angle = tf.random.uniform((), minval=-limit, maxval=limit,\n dtype=tf.int32)\n voxel = tfa.image.rotate(voxel, tf.cast(angle,\n tf.float32)*(math.pi/180),\n interpolation='nearest',\n fill_mode='constant',\n fill_value=0.0)\n return voxel\n\ndef random_resized_crop3D(voxel, min_width, min_height, p=0.5):\n if tf.random.uniform(()) < p:\n voxel_shape = voxel.shape\n assert voxel_shape[1] >= min_height\n assert voxel_shape[2] >= min_width\n \n width = tf.random.uniform((), minval=min_width,\n maxval=voxel_shape[2],\n dtype=tf.int32)\n height = tf.random.uniform((), minval=min_height,\n maxval=voxel_shape[1],\n dtype=tf.int32)\n x = tf.random.uniform((), minval=0,\n maxval=voxel_shape[2] - width,\n dtype=tf.int32)\n y = tf.random.uniform((), minval=0,\n maxval=voxel_shape[1] - height,\n dtype=tf.int32)\n voxel = voxel[:, y:y+height, x:x+width, :]\n voxel = tf.image.resize(voxel,\n voxel_shape[1:3],\n method='lanczos5')\n return voxel\n\ndef random_cutout3D(voxel, mask_shape=(10, 10), num_holes=20, p=0.5):\n if tf.random.uniform(()) < p:\n voxel_shape = voxel.shape\n assert voxel_shape[1] >= mask_shape[0]\n assert voxel_shape[2] >= mask_shape[1]\n\n holes = tf.random.uniform((), minval=1, maxval=num_holes,\n dtype=tf.int32)\n mask_size = tf.constant([mask_shape[0], mask_shape[1]])\n mask = tf.Variable((lambda : tf.ones(voxel_shape)),\n trainable=False)\n \n for i in tf.range(holes):\n x = tf.random.uniform((), minval=0,\n maxval=voxel_shape[2],\n dtype=tf.int32)\n y = tf.random.uniform((), minval=0,\n maxval=voxel_shape[1],\n dtype=tf.int32)\n mask_endx = tf.add(x, mask_size[1])\n mask_endy = tf.add(y, mask_size[0])\n mask[:, x:mask_endx,\n y:mask_endy, :].assign(tf.zeros_like(mask[:, x:mask_endx,\n y:mask_endy, :]))\n voxel = tf.multiply(voxel, mask)\n mask.assign(tf.ones(voxel_shape))\n return voxel\n\ndef _get_gaussian_kernel(sigma, filter_shape):\n x = tf.range(-filter_shape // 2 + 1, filter_shape // 2 + 1)\n x = tf.cast(x ** 2, sigma.dtype)\n x = tf.nn.softmax(-x / (2.0 * (sigma ** 2)))\n return x\n\ndef random_gaussian_blur3D(voxel, filter_shape=[5, 5], max_sigma=3, p=0.5):\n if tf.random.uniform(()) < p:\n sigma = tf.random.uniform((), minval=3, maxval=max_sigma,\n dtype=tf.int32)\n filter_shape = tf.constant(filter_shape)\n channels = voxel.shape[-1]\n sigma = tf.cast(sigma, voxel.dtype)\n gaussian_kernel_x = _get_gaussian_kernel(sigma,\n filter_shape[1])\n gaussian_kernel_x = gaussian_kernel_x[tf.newaxis, :]\n gaussian_kernel_y = _get_gaussian_kernel(sigma,\n filter_shape[0]) \n gaussian_kernel_y = gaussian_kernel_y[:, tf.newaxis]\n gaussian_kernel_2d = tf.matmul(gaussian_kernel_y,\n gaussian_kernel_x)\n gaussian_kernel_2d = gaussian_kernel_2d[:, :, tf.newaxis,\n tf.newaxis]\n gaussian_kernel_2d = tf.tile(gaussian_kernel_2d,\n tf.constant([1, 1, channels, 1]))\n voxel = tf.nn.depthwise_conv2d(input=voxel,\n filter=gaussian_kernel_2d,\n strides=(1, 1, 1, 1),\n padding=\"SAME\",\n )\n voxel = tf.cast(voxel, voxel.dtype)\n return voxel\n\ndef build_augmenter(with_labels=True):\n '''\n Performing tranformations with the same seed\n to ensure the same tranformation is applied to every voxel slice.\n ''' \n def augment(voxel):\n aug_seed = tf.random.uniform((2,), minval=1, maxval=9999, dtype=tf.int32)\n if tf.random.uniform(()) < FLIP:\n if tf.random.uniform(()) < 0.5:\n voxel = tf.image.flip_up_down(voxel)\n else:\n voxel = tf.image.flip_left_right(voxel)\n \n if tf.random.uniform(()) < BRIGHTNESS[1]:\n voxel = tf.image.adjust_brightness(\n voxel, tf.random.uniform((), minval=0.0,\n maxval=BRIGHTNESS[0],\n seed=seed))\n if tf.random.uniform(()) < CONTRAST[2]:\n voxel = tf.image.adjust_contrast(\n voxel, tf.random.uniform((), minval=CONTRAST[0],\n maxval=CONTRAST[1],\n seed=seed))\n if tf.random.uniform(()) < GAMMA[2]:\n voxel = tf.image.adjust_gamma(\n voxel, tf.random.uniform((), minval=GAMMA[0],\n maxval=GAMMA[1],\n seed=seed))\n voxel = random_rotate3D(voxel, limit=ROTATE[0],\n p=ROTATE[1])\n voxel = random_resized_crop3D(voxel, RANDOM_CROP[0],\n RANDOM_CROP[1],\n p=RANDOM_CROP[2])\n voxel = random_cutout3D(voxel, mask_shape=CUTOUT[0],\n num_holes=CUTOUT[1],\n p=CUTOUT[2])\n voxel = random_gaussian_blur3D(voxel,\n filter_shape=BLUR[0],\n max_sigma=BLUR[1],\n p=BLUR[2])\n\n # Remove nans in place of black pixels in some imgs.\n # Anyone knows the reason for the nans?\n voxel = tf.where(tf.math.is_nan(voxel),\n tf.zeros_like(voxel), voxel)\n voxel = tnp.maximum(tnp.array([0.]), voxel)\n voxel = tnp.minimum(tnp.array([1.]), voxel)\n voxel = tf.cast(voxel, tf.float32)\n return voxel\n \n def augment_with_labels(voxel, label):\n return augment(voxel), label\n \n return augment_with_labels if with_labels else augment\n\ndef build_dataset(paths, labels=None, bsize=32, cache=True,\n decode_fn=None, augment_fn=None,\n augment=True, repeat=True, shuffle=128,\n seed=None, cache_dir=\"\"):\n \n if cache_dir != \"\" and cache is True:\n os.makedirs(cache_dir, exist_ok=True)\n \n if decode_fn is None:\n decode_fn = build_decoder(labels is not None)\n \n if augment_fn is None:\n augment_fn = build_augmenter(labels is not None)\n \n AUTO = tf.data.experimental.AUTOTUNE\n slices = paths if labels is None else (paths, labels)\n dset = tf.data.Dataset.from_tensor_slices(slices)\n dset = dset.map(decode_fn, num_parallel_calls=AUTO)\n dset = dset.cache(cache_dir) if cache else ds\n \n ## Map the functions to perform Augmentations\n dset = dset.map(augment_fn, num_parallel_calls=AUTO) if augment else dset\n dset = dset.repeat() if repeat else dset\n dset = dset.shuffle(shuffle, seed=seed) if shuffle else dset\n dset = dset.batch(bsize, drop_remainder=True).prefetch(AUTO)\n# dset = dset.batch(bsize).prefetch(AUTO)\n return dset","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:40.539586Z","iopub.execute_input":"2021-09-12T09:06:40.539814Z","iopub.status.idle":"2021-09-12T09:06:40.759201Z","shell.execute_reply.started":"2021-09-12T09:06:40.539788Z","shell.execute_reply":"2021-09-12T09:06:40.758438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scan_type = 'FLAIR'\nlabel_cols = 'MGMT_value'\ntrain_paths = df_kfold.loc[(df_kfold.scan_type==scan_type) & (df_kfold['fold'] != fold), :].voxel_paths.values\ntrain_labels = df_kfold.loc[(df_kfold.scan_type==scan_type) & (df_kfold['fold'] != fold), :].MGMT_value.values\n\ndecoder = build_decoder(with_labels=True, target_size=cfg.input_dims, ext='npy')\n\ntrain_dataset = build_dataset(\n train_paths, train_labels, bsize=32, decode_fn=decoder,\n augment=True, repeat=False, shuffle=False\n)\n\ntrain_voxels, _ = next(iter(train_dataset))\n\nsqrt = math.ceil(math.sqrt(train_voxels.shape[0]))\nfig = plt.figure(figsize=(10, 10))\n\nfor idx, voxel in enumerate(train_voxels):\n ax = fig.add_subplot(int(sqrt), int(sqrt), idx+1)\n ax.imshow(voxel.numpy()[voxel.shape[0]//2], cmap='gray')\n ax.axes.xaxis.set_visible(False)\n ax.axes.yaxis.set_visible(False)\nfig.suptitle(\"Middle Slice of Each Voxel\", fontsize=22)\nfig.tight_layout()\nplt.savefig(f'samples.png')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:06:40.761837Z","iopub.execute_input":"2021-09-12T09:06:40.762337Z","iopub.status.idle":"2021-09-12T09:07:06.841027Z","shell.execute_reply.started":"2021-09-12T09:06:40.762300Z","shell.execute_reply":"2021-09-12T09:07:06.840376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interactive Visualizations with Weights & Biases","metadata":{}},{"cell_type":"code","source":"!pip install wandb --upgrade -q\n\nimport wandb\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\nkey = user_secrets.get_secret(\"wandb_key\")\nwandb.login(key=key)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-12T09:07:06.842208Z","iopub.execute_input":"2021-09-12T09:07:06.842617Z","iopub.status.idle":"2021-09-12T09:07:16.865465Z","shell.execute_reply.started":"2021-09-12T09:07:06.842588Z","shell.execute_reply":"2021-09-12T09:07:16.864589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run = wandb.init(project='RSNA-public', job_type='augmentations-viz',\n name='dataloader-samples')\nos.makedirs('gifs/', exist_ok=True)\n\nfor i, voxel in tqdm(enumerate(train_voxels), total=train_voxels.shape[0]):\n gifs = (voxel*255.).numpy().clip(0, 255).astype('uint8')\n imageio.mimsave(f'gifs/voxel_{i}.gif', gifs) \n\nwandb.log({'images': [wandb.Image(f'gifs/voxel_{i}.gif') for i in range(train_voxels.shape[0])]})\nrun.finish()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-12T11:23:57.308624Z","iopub.execute_input":"2021-09-12T11:23:57.309001Z","iopub.status.idle":"2021-09-12T11:24:47.305148Z","shell.execute_reply.started":"2021-09-12T11:23:57.308966Z","shell.execute_reply":"2021-09-12T11:24:47.304359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(HTML(f'View the Full Dashboard Here 🔎'))\nprint()\ndisplay(HTML(f\"
\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-12T11:44:32.040766Z","iopub.execute_input":"2021-09-12T11:44:32.041073Z","iopub.status.idle":"2021-09-12T11:44:32.053700Z","shell.execute_reply.started":"2021-09-12T11:44:32.041039Z","shell.execute_reply":"2021-09-12T11:44:32.052740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Learning Rate Schedule","metadata":{}},{"cell_type":"code","source":"START_LR = 1e-6\nMAX_LR = cfg.lr\nMIN_LR = 1e-8\nLR_RAMP = 3\nLR_SUSTAIN = 5\nLR_DECAY = 0.90\n\ndef lrfn(epoch):\n if LR_RAMP > 0 and epoch < LR_RAMP:\n lr = (MAX_LR-START_LR)/(LR_RAMP*1.0)*epoch+START_LR\n elif epoch < LR_RAMP+LR_SUSTAIN:\n lr = MAX_LR\n else: # exponential decay from MAX_LR to MIN_LR\n lr = (MAX_LR-MIN_LR)*LR_DECAY**(epoch-LR_RAMP-LR_SUSTAIN)+MIN_LR\n return lr\n \n@tf.function\ndef lrfn_tffun(epoch):\n return lrfn(epoch)\n\nfig = plt.figure(figsize=(14, 5))\nax = fig.add_subplot(111)\nrng = [i for i in range(cfg.n_epochs)]\nplt.plot(rng, [lrfn(x) for x in rng],\n marker='o')\nplt.ticklabel_format(axis=\"y\", style=\"plain\")\nax.yaxis.set_major_formatter(mtick.FormatStrFormatter('%.1e'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:08:06.867546Z","iopub.execute_input":"2021-09-12T09:08:06.867785Z","iopub.status.idle":"2021-09-12T09:08:07.072537Z","shell.execute_reply.started":"2021-09-12T09:08:06.867760Z","shell.execute_reply":"2021-09-12T09:08:07.071570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Setup Mixed Precision","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import mixed_precision\n\nif cfg.fp16:\n # Using 'mixed_bfloat16' compatible with TPUs\n mixed_precision.set_global_policy('mixed_bfloat16')\n\n print('Mixed precision enabled')","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:08:07.073993Z","iopub.execute_input":"2021-09-12T09:08:07.074266Z","iopub.status.idle":"2021-09-12T09:08:07.079703Z","shell.execute_reply.started":"2021-09-12T09:08:07.074225Z","shell.execute_reply":"2021-09-12T09:08:07.078897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define Model Architecure\n\nWe use the models from the awesome [classification_models_3D](https://github.com/ZFTurbo/classification_models_3D) library by [@ZFTurbo](https://www.kaggle.com/zfturbo)\n\n* If mixed precision is enabled, the output layer of the model must be cast into float32 type for numerical stability\n\n
\n\n#### Supported Nets:\n\n\n\n \n \n \n \n \n \n
   
resnet18seresnet34senet154densenet169
resnet34seresnet50resnext50densenet201
resnet50seresnet101resnext101inceptionresnetv2
resnet101seresnet152vgg16inceptionv3
resnet152seresnext50vgg19mobilenet
seresnet18seresnext101densenet121mobilenetv2
\n\n
","metadata":{}},{"cell_type":"code","source":"from classification_models_3D.tfkeras import Classifiers\n\nmodel_arch = 'seresnet50'\n\ndef create_model(input_shape, num_classes):\n inputs = tf.keras.layers.Input((*input_shape, 1), name='inputs')\n x = tf.keras.layers.Conv3D(3, (3, 3, 3), strides=(1, 1, 1), \n padding='same', use_bias=True)(inputs)\n \n net, preprocess_input = Classifiers.get(model_arch)\n x = net(input_shape=(*input_shape, 3), include_top=False,\n weights='imagenet')(x)\n x = tf.keras.layers.GlobalAveragePooling3D()(x)\n x = tf.keras.layers.Dropout(rate=0.5)(x)\n \n # Cast output to float32 for numerical stability\n outputs = tf.keras.layers.Dense(num_classes, activation='sigmoid',\n dtype='float32')(x)\n model = tf.keras.Model(inputs, outputs)\n print(model.summary())\n \n return model\n\n# model = create_model(cfg.input_dims, 1)","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:11:55.830017Z","iopub.execute_input":"2021-09-12T09:11:55.830881Z","iopub.status.idle":"2021-09-12T09:11:56.317039Z","shell.execute_reply.started":"2021-09-12T09:11:55.830840Z","shell.execute_reply":"2021-09-12T09:11:56.316157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"🧪 Train with Custom Training Loop","metadata":{}},{"cell_type":"code","source":"def auto_select_accelerator():\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n print(\"Running on TPU:\", tpu.master())\n except ValueError:\n strategy = tf.distribute.get_strategy()\n print(f\"Running on {strategy.num_replicas_in_sync} replicas\")\n \n return strategy","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:11:57.187894Z","iopub.execute_input":"2021-09-12T09:11:57.188178Z","iopub.status.idle":"2021-09-12T09:11:57.194335Z","shell.execute_reply.started":"2021-09-12T09:11:57.188147Z","shell.execute_reply":"2021-09-12T09:11:57.193623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fold = 0\nDISPLAY_PLOT = True\n\noof_aucs = dict()\n\nfor scan_type in cfg.scan_types:\n print(\"\\nScan Type:\", scan_type)\n # Define TPU strategy and clear TPU\n strategy = auto_select_accelerator()\n \n user_secrets = UserSecretsClient()\n user_credential = user_secrets.get_gcloud_credential()\n user_secrets.set_tensorflow_credential(user_credential)\n \n GCS_DS_PATH = KaggleDatasets().get_gcs_path(cfg.dataset.split('/')[-1])\n print(\"GCS_DS_PATH\", GCS_DS_PATH)\n\n train_paths = df_kfold.loc[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] != fold), :].voxel_paths.map( \\\n lambda path: os.path.join(GCS_DS_PATH+'/' , *path.split('/')[4:])).values\n valid_paths = df_kfold.loc[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] == fold), :].voxel_paths.map( \\\n lambda path: os.path.join(GCS_DS_PATH+'/' , *path.split('/')[4:])).values\n \n train_labels = df_kfold.loc[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] != fold), :].MGMT_value.values.reshape(-1,1)\n valid_labels = df_kfold.loc[(df_kfold.scan_type==scan_type) & \\\n (df_kfold['fold'] == fold), :].MGMT_value.values.reshape(-1,1)\n\n # Converting global config class object to a dictionary to log using Wandb\n config_dict = dict(vars(Config))\n config_dict = {k:(v if type(v)==int else str(v)) for (k,v) in config_dict.items() if '__' not in k}\n config_dict['fold'] = fold\n config_dict['model_arch'] = model_arch\n config_dict['job_name'] = f\"{config_dict['model_arch']}_fold{fold}_{scan_type}_job{config_dict['job']}\"\n print(\"Train Job:\", config_dict['job_name'], \"\\nConfig\")\n pprint.pprint(config_dict)\n\n run = wandb.init(project=cfg.wandb_project, name=config_dict['job_name'],\n config=config_dict)\n\n decoder = build_decoder(with_labels=True, target_size=cfg.input_dims, ext='npy')\n\n train_dataset = build_dataset(\n train_paths, train_labels, bsize=cfg.batch_size,\n decode_fn=decoder, augment=True, shuffle=256\n )\n \n val_batch_size = 8\n valid_dataset = build_dataset(\n valid_paths, valid_labels, bsize=val_batch_size,\n decode_fn=decoder, repeat=False, shuffle=False, augment=False\n )\n\n train_steps_per_epoch = train_paths.shape[0]//cfg.batch_size\n valid_steps_per_epoch = valid_paths.shape[0]//val_batch_size\n print()\n\n with strategy.scope():\n model = create_model(cfg.input_dims, cfg.num_classes)\n\n # Custom learning rate schedule\n class LRSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n def __call__(self, step):\n return lrfn_tffun(epoch=step//train_steps_per_epoch)\n\n optimizer = tf.keras.optimizers.Adam(learning_rate=LRSchedule())\n# optimizer = tf.keras.optimizers.Adam(learning_rate=cfg.lr)\n\n train_accuracy = tf.keras.metrics.BinaryAccuracy(name=\"acc\")\n train_precision = tf.keras.metrics.Precision(name='precision')\n train_recall = tf.keras.metrics.Recall(name='recall')\n train_roc_auc = tf.keras.metrics.AUC(curve='ROC', name=\"auc\")\n train_map = tf.keras.metrics.AUC(curve='PR', name=\"map\")\n\n valid_accuracy = tf.keras.metrics.BinaryAccuracy(name=\"val_acc\")\n valid_precision = tf.keras.metrics.Precision(name='precision')\n valid_recall = tf.keras.metrics.Recall(name='recall')\n valid_roc_auc = tf.keras.metrics.AUC(curve='ROC', name=\"val_auc\")\n valid_map = tf.keras.metrics.AUC(curve='PR', name=\"val_map\")\n \n # Apply label smoothing to training\n loss_fn = lambda labels, probabilities: tf.reduce_mean(\n tf.keras.losses.binary_crossentropy(\n labels,\n probabilities,\n label_smoothing=cfg.label_smoothing))\n \n val_loss_fn = lambda labels, probabilities: tf.reduce_mean(\n tf.keras.losses.binary_crossentropy(labels,\n probabilities))\n \n @tf.function\n def train_step(images, labels):\n with tf.GradientTape() as tape:\n probabilities = model(images, training=True)\n loss = loss_fn(labels, probabilities)\n grads = tape.gradient(loss, model.trainable_variables)\n optimizer.apply_gradients(zip(grads, model.trainable_variables))\n train_accuracy.update_state(labels, probabilities)\n train_precision.update_state(labels, probabilities)\n train_recall.update_state(labels, probabilities)\n train_roc_auc.update_state(labels, probabilities)\n train_map.update_state(labels, probabilities)\n return loss\n\n @tf.function\n def valid_step(images, labels):\n probabilities = model(images, training=False)\n loss = val_loss_fn(labels, probabilities)\n valid_accuracy.update_state(labels, probabilities)\n valid_precision.update_state(labels, probabilities)\n valid_recall.update_state(labels, probabilities)\n valid_roc_auc.update_state(labels, probabilities)\n valid_map.update_state(labels, probabilities)\n return loss\n\n # Distribute the datset according to the strategy\n train_dist_ds = strategy.experimental_distribute_dataset(train_dataset)\n valid_dist_ds = strategy.experimental_distribute_dataset(valid_dataset)\n \n bar_stateful_metrics = [\"lr\", \"loss\", \"acc\", \"auc\",\n \"precision\", \"recall\", \"map\",\n \"val_loss\", \"val_acc\",\n \"val_precision\", \"val_recall\",\n \"val_auc\", \"val_map\"]\n train_prog_bar = Progbar(train_steps_per_epoch, width=60,\n stateful_metrics=bar_stateful_metrics)\n\n epoch = 0\n train_losses=[]\n history = []\n start_time = epoch_start_time = time.time()\n best_metric = 0.0\n metric = 'val_auc'\n \n for step, (voxels, labels) in enumerate(train_dist_ds):\n loss = strategy.run(train_step, args=(voxels, labels))\n loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, loss, axis=None)\n train_losses.append(loss)\n \n train_prog_bar.add(1,\n values=[(\"lr\",lrfn(epoch)),\n (\"loss\", loss.numpy()),\n ('acc',train_accuracy.result().numpy()),\n ('precision',train_precision.result().numpy()),\n ('recall',train_recall.result().numpy()),\n ('auc',train_roc_auc.result().numpy()),\n ('map',train_map.result().numpy())])\n \n if ((step+1) // train_steps_per_epoch) > epoch:\n valid_prog_bar = Progbar(valid_steps_per_epoch, width=60,\n stateful_metrics=[\"lr\", \"loss\", \"acc\",\n \"auc\", \"precision\",\n \"recall\", \"map\", \"val_acc\",\n \"val_precision\",\n \"val_recall\", \"val_auc\",\n \"val_map\"])\n \n valid_loss = []\n for voxel, label in valid_dist_ds:\n batch_loss = strategy.run(valid_step, args=(voxel, label)) # just one batch\n batch_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN,\n batch_loss, axis=None)\n valid_loss.append(batch_loss.numpy())\n \n valid_prog_bar.add(\n 1,\n values=[(\"lr\",lrfn(epoch)),\n ('acc',train_accuracy.result().numpy()),\n ('precision',train_precision.result().numpy()),\n ('recall',train_recall.result().numpy()),\n ('auc',train_roc_auc.result().numpy()),\n ('map',train_map.result().numpy()),\n ('val_loss',batch_loss.numpy()),\n ('val_acc',valid_accuracy.result().numpy()),\n ('val_precision',valid_precision.result().numpy()),\n ('val_recall',valid_recall.result().numpy()),\n ('val_auc',valid_roc_auc.result().numpy()),\n ('val_map',valid_map.result().numpy())\n ])\n \n valid_loss = np.mean(valid_loss)\n \n epoch = (step+1) // train_steps_per_epoch\n epoch_time = time.time() - epoch_start_time\n print(f\"\\nEpoch: {epoch}/{cfg.n_epochs} Done.\")\n print(\"-\"*60)\n \n val_log_dict = {\n 'epoch': epoch,\n 'lr': lrfn(epoch),\n 'time': epoch_time,\n 'loss': loss.numpy(),\n 'acc': train_accuracy.result().numpy(),\n 'precision': train_precision.result().numpy(),\n 'recall': train_recall.result().numpy(),\n 'auc': train_roc_auc.result().numpy(),\n 'map': train_map.result().numpy(),\n 'val_loss': round(valid_loss, 4),\n 'val_acc': valid_accuracy.result().numpy(),\n 'val_precision': valid_precision.result().numpy(),\n 'val_recall': valid_recall.result().numpy(),\n 'val_auc': valid_roc_auc.result().numpy(),\n 'val_map': valid_map.result().numpy(),\n }\n\n # Save the best model\n if val_log_dict[metric]>best_metric:\n best_metric = val_log_dict[metric]\n model.save(f'model_fold{fold}_{scan_type}.h5')\n wandb.save(f'model_fold{fold}_{scan_type}.h5')\n\n val_log_dict['best_val_auc'] = best_metric\n wandb.log(val_log_dict, step=step)\n \n history.append(val_log_dict)\n epoch_start_time = time.time()\n train_accuracy.reset_states()\n train_roc_auc.reset_states()\n train_map.reset_states()\n valid_accuracy.reset_states()\n valid_roc_auc.reset_states()\n valid_map.reset_states()\n\n if epoch >= cfg.n_epochs:\n break\n \n print()\n train_prog_bar = Progbar(train_steps_per_epoch, width=60,\n stateful_metrics=bar_stateful_metrics)\n else:\n log_dict = {\n 'epoch': epoch,\n 'lr': lrfn(epoch),\n 'loss': loss.numpy(),\n 'acc': train_accuracy.result().numpy(),\n 'precision': train_precision.result().numpy(),\n 'recall': train_recall.result().numpy(),\n 'auc': train_roc_auc.result().numpy(),\n 'map': train_map.result().numpy(),\n }\n wandb.log(log_dict, step=step)\n\n training_time = time.time() - start_time\n print(\"\\nTotal Training Time: {:0.1f}s\".format(training_time))\n \n history_df = pd.DataFrame(history)\n history_df.to_csv(f'history{fold}.csv')\n\n del model, decoder, train_dataset, valid_dataset\n gc.collect()\n\n oof_aucs[scan_type] = float(np.max(history_df['val_auc']))\n print(\"oof_aucs\", oof_aucs)\n \n run.finish()\n \n # Plot Training History\n if DISPLAY_PLOT:\n print (\"\\n\\n\")\n plt.figure(figsize=(15,5))\n plt.plot(np.arange(len(history_df['auc'])), history_df['auc'],\n '-o', label='Train auc', color='#ff7f0e')\n plt.plot(np.arange(len(history_df['val_auc'])), history_df['val_auc'],\n '-o', label='Val auc', color='#1f77b4')\n x = np.argmax(history_df['val_auc'])\n y = np.max(history_df['val_auc'])\n xdist = plt.xlim()[1] - plt.xlim()[0]\n ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x,y,s=200, color='#1f77b4')\n plt.text(x-0.03*xdist, y-0.13*ydist, 'Max AUC\\n%.2f'%y, size=10)\n plt.ylabel('auc', size=14)\n plt.xlabel('Epoch', size=14)\n plt.legend(loc=2)\n \n plt2 = plt.gca().twinx()\n plt2.plot(np.arange(len(history_df['loss'])),\n history_df['loss'],'-o', label='Train Loss', color='#f98b88')\n plt2.plot(np.arange(len(history_df['val_loss'])),\n history_df['val_loss'],'-o', label='Val Loss', color='#3c1361')\n x = np.argmin(history_df['val_loss'])\n y = np.min(history_df['val_loss'])\n ydist = plt.ylim()[1] - plt.ylim()[0]\n plt.scatter(x, y, s=200, color='#3c1361')\n plt.text(x-0.03*xdist, y+0.05*ydist,'Min Loss', size=10)\n plt.ylabel('Loss', size=14)\n plt.title(f\"{config_dict['job_name']} Size - {cfg.input_dims}\")\n plt.legend(loc=3)\n plt.savefig(f'fig{fold}_{scan_type}.png')\n plt.show() ","metadata":{"execution":{"iopub.status.busy":"2021-09-12T09:11:59.212382Z","iopub.execute_input":"2021-09-12T09:11:59.213104Z","iopub.status.idle":"2021-09-12T10:07:26.637591Z","shell.execute_reply.started":"2021-09-12T09:11:59.213064Z","shell.execute_reply":"2021-09-12T10:07:26.636548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(HTML(f'Last Run Stats - View the Full Dashboard Here 🔎'))\nprint()\ndisplay(HTML(\"
\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-12T11:46:03.661484Z","iopub.execute_input":"2021-09-12T11:46:03.662312Z","iopub.status.idle":"2021-09-12T11:46:03.674016Z","shell.execute_reply.started":"2021-09-12T11:46:03.662253Z","shell.execute_reply":"2021-09-12T11:46:03.672969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!find ./ -name \"*.gif\" -type f -delete","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-09-12T11:47:08.468451Z","iopub.execute_input":"2021-09-12T11:47:08.468734Z","iopub.status.idle":"2021-09-12T11:47:09.292877Z","shell.execute_reply.started":"2021-09-12T11:47:08.468707Z","shell.execute_reply":"2021-09-12T11:47:09.291776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

HAVE A GREAT DAY !

\n\n

Let me know if you have any suggestions!

","metadata":{}}]}