{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n\n\n
\n\n---\n\n
\n\n

EXPLORATORY DATA ANALSYSIS AND SOLUTIONING

TOWARDS BRAIN TUMOR RADIOGENOMIC CLASSIFICATION

\n\n
\n \n
CREATED BY: DARIEN SCHETTLER

\n\n---\n\n
\n\n
\n
πŸ›‘πŸ›‘πŸ›‘   CAUTION: THIS NOTEBOOK IS A WORK IN PROGRESS   πŸ›‘πŸ›‘πŸ›‘

\n
\n\n
\n\n\n---\n
","metadata":{"papermill":{"duration":0.036465,"end_time":"2021-06-04T20:52:51.125208","exception":false,"start_time":"2021-06-04T20:52:51.088743","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

TABLE OF CONTENTS

\n\n---\n\n

0    IMPORTS

\n\n---\n\n

1    BACKGROUND INFORMATION

\n\n---\n\n

2    SETUP

\n\n---\n\n

3    HELPER FUNCTIONS

\n\n---\n","metadata":{"papermill":{"duration":0.033613,"end_time":"2021-06-04T20:52:51.194111","exception":false,"start_time":"2021-06-04T20:52:51.160498","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS

","metadata":{"papermill":{"duration":0.03344,"end_time":"2021-06-04T20:52:51.261152","exception":false,"start_time":"2021-06-04T20:52:51.227712","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(\"\\n... PIP/APT INSTALLS STARTING ...\")\n!cp /kaggle/input/gdcm-conda-install/gdcm.tar .\n!tar -xvzf gdcm.tar\n!conda install --offline ./gdcm/gdcm-2.8.9-py37h71b2a6d_0.tar.bz2\n!rm -rf ./gdcm*\n!pip install efficientnet-3D\n!pip install git+https://github.com/qubvel/classification_models.git\n!pip install -U --pre efficientnet\n\n# !conda install -c conda-forge gdcm -y\n!pip install pandarallel\nprint(\"... PIP/APT INSTALLS COMPLETE ...\\n\")\n\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Machine Learning and Data Science Imports\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('max_columns', 100);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport efficientnet_3D.tfkeras as efn\n\n# Other Competition Related Imports\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom pandarallel import pandarallel; pandarallel.initialize();\n\n# Built In Imports\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom glob import glob\nimport warnings\nimport requests\nimport imageio\nimport IPython\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport scipy\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport matplotlib.patches as patches\n\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nimport plotly\nimport PIL\nimport cv2\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")\n \n\nprint(\"\\n... SEEDING FOR DETERMINISTIC BEHAVIOUR ...\")\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\nseed_it_all()\nprint(\"... SEEDING COMPLETE ...\\n\\n\")\n\nprint(\"\\n... SETTING PRESETS STARTING...\")\nFIG_FONT = dict(family=\"Helvetica, Arial\", size=14, color=\"#7f7f7f\")\nlabel_map_int_2_str = {0: \"MGMT_NEGATIVE\", 1: \"MGMT_POSITIVE\"}\nlabel_map_str_2_int = {\"MGMT_NEGATIVE\":0, \"MGMT_POSITIVE\":1}\nprint(\"... SETTING PRESETS COMPLETE...\\n\\n\")","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":51.261012,"end_time":"2021-06-04T20:53:42.555772","exception":false,"start_time":"2021-06-04T20:52:51.29476","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-28T15:55:08.544227Z","iopub.execute_input":"2021-07-28T15:55:08.544618Z","iopub.status.idle":"2021-07-28T15:56:13.795095Z","shell.execute_reply.started":"2021-07-28T15:55:08.544543Z","shell.execute_reply":"2021-07-28T15:56:13.794227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  BACKGROUND INFORMATION

","metadata":{"papermill":{"duration":0.040472,"end_time":"2021-06-04T20:53:42.636927","exception":false,"start_time":"2021-06-04T20:53:42.596455","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

1.1 OVERVIEW

\n\n---\n\nCOMPETITION DESCRIPTION\n\nA malignant tumor in the brain is a life-threatening condition. Known as glioblastoma, it's both the most common form of brain cancer in adults and the one with the worst prognosis, with median survival being less than a year. The presence of a specific genetic sequence in the tumor known as MGMT promoter methylation has been shown to be a favorable prognostic factor and a strong predictor of responsiveness to chemotherapy.\n\nCurrently, genetic analysis of cancer requires surgery to extract a tissue sample. Then it can take several weeks to determine the genetic characterization of the tumor. Depending upon the results and type of initial therapy chosen, a subsequent surgery may be necessary. If an accurate method to predict the genetics of the cancer through imaging (i.e., radiogenomics) alone could be developed, this would potentially minimize the number of surgeries and refine the type of therapy required.\n\nThe Radiological Society of North America (RSNA) has teamed up with the Medical Image Computing and Computer Assisted Intervention Society (the MICCAI Society) to improve diagnosis and treatment planning for patients with glioblastoma. In this competition you will predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to train and test your model to detect for the presence of MGMT promoter methylation.\n\nIf successful, you'll help brain cancer patients receive less invasive diagnoses and treatments. The introduction of new and customized treatment strategies before surgery has the potential to improve the management, survival, and prospects of patients with brain cancer.\n\n**Secondary Description From UPenn**\n> The participants are called to use the provided mpMRI data to extract imaging/radiomic features that they consider appropriate, and analyze them through machine learning algorithms, in an attempt to predict the MGMT promoter methylation status. The participants do not need to be limited to volumetric parameters, but can also consider intensity, morphologic, histogram-based, and textural features, as well as spatial information, deep learning features, and glioma diffusion properties extracted from glioma growth models.\n\n> Note that participants will be evaluated for the predicted MGMT status of the subjects indicated in the accompanying spreadsheet.\n\n
\n\n
\n
πŸ€”πŸ€”πŸ€”   MY INTERPRETATION OF THIS COMPETITION   πŸ€”πŸ€”πŸ€”


In this competition we are tasked with identifying/predicting the genetic subtype (genetic subtype = a group of tumors that is enriched for genetic aberrations in a set of subtype predictor genes) of glioblastoma (glioblastoma = brain tumor)

HUH?!?!?

All this means is that we are taking in dicom images containing 3D representations (slices) of a patients brain which we will process with computer vision algorithms to allow us to perform binary classification. The binary classification is to identify if, within the patients image data, MGMT promoter methylation is present. The diagram below shows a simple, if slightly inaccurate, representation of what is required.


\n
\n\n---\n\n
\n\nSUBMISSION EVALUATION/RESTRICTIONS AND FILE FORMAT\n\n**Submission Evaluation**\n* Submissions are evaluated on the [area under the ROC curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) between the predicted probability and the observed target.\n\n
\n\n**Submission Restrictions**\n* **THIS IS A KERNELS ONLY COMPETITION**\n * Submissions to this competition must be made through Notebooks. \n * In order for the \"Submit\" button to be active after a commit, the following conditions must be met:\n * *CPU Notebook <= 9 hours run-time*\n * *GPU Notebook <= 9 hours run-time*\n * *Internet access disabled*\n * *Freely & publicly available external data is allowed, including pre-trained models*\n * *Submission file must be named `submission.csv`*\n\n
\n\n**Submission File Format**\n* For each **`BraTS21ID`** in the test set, you must predict a probability for the target **`MGMT_value`**. The file should contain a header and have the following format:\n>```\n>BraTS21ID,MGMT_value\n>00001,0.5\n>00013,0.999\n>00015,0.1\n>etc.\n>```\n\n
\n\n---\n\n
\n\nCOMPETITION TIMELINE\n\n* **July 13, 2021** - Start Date.\n* **October 8, 2021** - Entry Deadline. You must accept the competition rules before this date in order to compete.\n* **October 8, 2021** - Team Merger Deadline. This is the last day participants may join or merge teams.\n* **October 15, 2021** - Final Submission Deadline.\n* **October 25, 2021** - Winners’ Requirements Deadline. This is the deadline for winners to submit to the host/Kaggle their training code, video, method description.\n\n> All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\n\n","metadata":{"papermill":{"duration":0.041492,"end_time":"2021-06-04T20:53:42.719208","exception":false,"start_time":"2021-06-04T20:53:42.677716","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

1.2 DATA DESCRIPTION

\n\n---\n\n
\n\nDATA SPLITS/COHORTS\n\nThe competition data is defined by three cohorts: **Training**, **Validation (Public)**, and **Testing (Private)**. \n* The **β€œTraining”** and the **β€œValidation”** cohorts are provided to the participants\n* The **β€œTesting”** cohort is kept hidden at all times, during and after the competition\n\nThese 3 cohorts are structured as follows: \n* Each independent case has a **dedicated folder identified by a five-digit number**.\n* Within each of these **β€œcase”** folders, there are four sub-folders\n * Each of these **\"case\"** subfolders corresponds to each of the structural **m**ulti-**p**arametric **MRI** (**mpMRI**) scans, in **DICOM** format. \n * The exact mpMRI scans included are:\n * Fluid Attenuated Inversion Recovery (FLAIR)\n * T1-weighted pre-contrast (T1w)\n * T1-weighted post-contrast (T1Gd)\n * T2-weighted (T2)\n\nExact folder structure:\n\n```\nTraining/Validation/Testing\nβ”‚\n└─── 00000\nβ”‚ β”‚\nβ”‚ └─── FLAIR\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T1w\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T1wCE\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T2w\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ .....\nβ”‚ \n└─── 00001\nβ”‚ β”‚ ...\nβ”‚ \nβ”‚ ... \nβ”‚ \n└─── 00002\nβ”‚ β”‚ ...\n\n```\n\nFILES\n\n**`train/`** \n- folder containing the training files, with each top-level folder representing a subject\n\n**`train_labels.csv`** \n- file containing the target MGMT_value for each subject in the training data (e.g. the presence of MGMT promoter methylation)\n\n**`test/`** \n- the test files, which use the same structure as train/; your task is to predict the MGMT_value for each subject in the test data. NOTE: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set\n\n**`sample_submission.csv`** \n- a sample submission file in the correct format","metadata":{"papermill":{"duration":0.041685,"end_time":"2021-06-04T20:53:42.803402","exception":false,"start_time":"2021-06-04T20:53:42.761717","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n\n\n

2  SETUP

","metadata":{"papermill":{"duration":0.04017,"end_time":"2021-06-04T20:53:42.971481","exception":false,"start_time":"2021-06-04T20:53:42.931311","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ROOT_DIR = \"/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification\"\n\nTRAIN_DIR = os.path.join(ROOT_DIR, \"train\")\nTEST_DIR = os.path.join(ROOT_DIR, \"test\")\n\nSS_CSV = os.path.join(ROOT_DIR, \"sample_submission.csv\")\nTRAIN_CSV = os.path.join(ROOT_DIR, \"train_labels.csv\")\n\ntrain_df = pd.read_csv(TRAIN_CSV)\ntrain_df[\"path_to_flair_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"FLAIR\"))\ntrain_df[\"flair_image_count\"] = train_df.path_to_flair_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t1w_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T1w\"))\ntrain_df[\"t1w_image_count\"] = train_df.path_to_t1w_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t1wce_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T1wCE\"))\ntrain_df[\"t1wce_image_count\"] = train_df.path_to_t1wce_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t2w_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T2w\"))\ntrain_df[\"t2w_image_count\"] = train_df.path_to_t2w_dir.progress_apply(lambda x: len(os.listdir(x)))\n \nss_df = pd.read_csv(SS_CSV)\nss_df[\"path_to_flair_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"FLAIR\"))\nss_df[\"flair_image_count\"] = ss_df.path_to_flair_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t1w_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T1w\"))\nss_df[\"t1w_image_count\"] = ss_df.path_to_t1w_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t1wce_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T1wCE\"))\nss_df[\"t1wce_image_count\"] = ss_df.path_to_t1wce_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t2w_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T2w\"))\nss_df[\"t2w_image_count\"] = ss_df.path_to_t2w_dir.progress_apply(lambda x: len(os.listdir(x)))\n\nprint(\"\\n\\nTRAIN DATAFRAME\\n\")\ndisplay(train_df.head())\n\nprint(\"\\n\\n\\nSAMPLE SUBMISSION DATAFRAME\\n\")\ndisplay(ss_df.head())\n\nprint(\"\\nCOUNTS PER SCAN TYPE\\n\")\nfor c_name in [c for c in train_df.columns if \"count\" in c]:\n print(c_name, print(train_df[c_name].sum()))","metadata":{"papermill":{"duration":507.188703,"end_time":"2021-06-04T21:02:10.201009","exception":false,"start_time":"2021-06-04T20:53:43.012306","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-27T16:28:09.252611Z","iopub.execute_input":"2021-07-27T16:28:09.253004Z","iopub.status.idle":"2021-07-27T16:29:03.284936Z","shell.execute_reply.started":"2021-07-27T16:28:09.252964Z","shell.execute_reply":"2021-07-27T16:29:03.28387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

3  HELPER FUNCTIONS

","metadata":{"papermill":{"duration":0.053621,"end_time":"2021-06-04T21:02:10.308317","exception":false,"start_time":"2021-06-04T21:02:10.254696","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_list_of_dcm_paths(dir_path):\n return sorted([os.path.join(dir_path, f_name) for f_name in os.listdir(dir_path)], key=lambda x: int(x.rsplit(\"-\", 1)[1].split(\".\", 1)[0]))\n\ndef dicom2array(path, voi_lut=True, fix_monochrome=True):\n \"\"\" Convert dicom file to numpy array \n \n Args:\n path (str): Path to the dicom file to be converted\n voi_lut (bool): Whether or not VOI LUT is available\n fix_monochrome (bool): Whether or not to apply monochrome fix\n \n Returns:\n Numpy array of the respective dicom file \n \n \"\"\"\n # Use the pydicom library to read the dicom file\n dicom = pydicom.read_file(path)\n \n # VOI LUT (if available by DICOM device) is used to \n # transform raw DICOM data to \"human-friendly\" view\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n \n # The XRAY may look inverted\n # - If we want to fix this we can\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n \n # Normalize the image array and return\n data = (data-np.min(data))/(np.max(data)-np.min(data))\n return data\n\ndef create_animation(np_arr):\n fig = plt.figure(figsize=(5, 5))\n plt.axis('off')\n im = plt.imshow(np_arr[..., 0], cmap=\"bone\")\n plt.title(\"mpMRI Animation\", fontweight=\"bold\")\n def animate_func(i):\n im.set_array(np_arr[..., i])\n return [im]\n plt.close()\n return animation.FuncAnimation(fig, animate_func, frames = np_arr.shape[-1], interval = 1000//24)\n\n\ndef create_4scan_animation(flair_arr, t1w_arr, t1wce_arr, t2w_arr):\n \n def animate_func(i):\n flair_im.set_array(flair_arr[..., i])\n t1w_im.set_array(t1w_arr[..., i])\n t1wce_im.set_array(t1wce_arr[..., i])\n t2w_im.set_array(t2w_arr[..., i])\n return [flair_im, t1w_im, t1wce_im, t2w_im]\n \n fig = plt.figure(figsize=(18, 5))\n \n #\n plt.subplot(1,4,1)\n plt.axis('off')\n flair_im = plt.imshow(flair_arr[..., 0], cmap=\"bone\")\n plt.title(\"FLAIR Animation\", fontweight=\"bold\")\n \n #\n plt.subplot(1,4,2)\n plt.axis('off')\n t1w_im = plt.imshow(t1w_arr[..., 0], cmap=\"bone\")\n plt.title(\"T1w Animation\", fontweight=\"bold\")\n \n #\n plt.subplot(1,4,3)\n plt.axis('off')\n t1wce_im = plt.imshow(t1wce_arr[..., 0], cmap=\"bone\")\n plt.title(\"T1wCE Animation\", fontweight=\"bold\")\n\n \n #\n plt.subplot(1,4,4)\n plt.axis('off')\n t2w_im = plt.imshow(t2w_arr[..., 0], cmap=\"bone\")\n plt.title(\"T2w Animation\", fontweight=\"bold\")\n \n plt.close()\n \n return animation.FuncAnimation(fig, animate_func, frames = flair_arr.shape[-1], interval = 1000//16)\n\n\ndef get_dicom_meta(row, attrs):\n dcm_file = pydicom.read_file(row.dcm_path)\n for val in attrs:\n row[val] = dcm_file.get(val, None)\n return row\n\n\ndef load_npz(np_file_path, is_tf=False):\n if is_tf:\n return np.load(np_file_path.numpy().decode())[\"arr_0\"] \n else:\n return np.load(np_file_path)[\"arr_0\"] \n \n \ndef flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:03.286967Z","iopub.execute_input":"2021-07-27T16:29:03.287349Z","iopub.status.idle":"2021-07-27T16:29:03.31173Z","shell.execute_reply.started":"2021-07-27T16:29:03.287308Z","shell.execute_reply":"2021-07-27T16:29:03.310689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

4  DICOM UNDERSTANDING

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

4.0 SET EXAMPLE PATIENT DATA

\n\n---\n\n
\n","metadata":{}},{"cell_type":"code","source":"EX_NUMBER = 0\nEX_SLICE = 0\nEX_PATIENT = train_df.iloc[EX_NUMBER]\n\nALL_DICOM_PATHS = glob(os.path.join(TRAIN_DIR, \"**/*.dcm\"), recursive=True)\nFLAIR_DCM_PATHS = get_list_of_dcm_paths(EX_PATIENT[\"path_to_flair_dir\"])\nT1W_DCM_PATHS = get_list_of_dcm_paths(EX_PATIENT[\"path_to_t1w_dir\"])\nT1WCE_DCM_PATHS = get_list_of_dcm_paths(EX_PATIENT[\"path_to_t1wce_dir\"])\nT2W_DCM_PATHS = get_list_of_dcm_paths(EX_PATIENT[\"path_to_t2w_dir\"])\n\nFLAIR_EXAMPLE_DCM_1 = FLAIR_DCM_PATHS[EX_SLICE]\nFLAIR_EXAMPLE_DCM_2 = FLAIR_DCM_PATHS[EX_SLICE+1]\nT1W_EXAMPLE_DCM_1 = T1W_DCM_PATHS[EX_SLICE]\nT1W_EXAMPLE_DCM_2 = T1W_DCM_PATHS[EX_SLICE+1]\nT1WCE_EXAMPLE_DCM_1 = T1WCE_DCM_PATHS[EX_SLICE]\nT1WCE_EXAMPLE_DCM_2 = T1WCE_DCM_PATHS[EX_SLICE+1]\nT2W_EXAMPLE_DCM_1 = T2W_DCM_PATHS[EX_SLICE]\nT2W_EXAMPLE_DCM_2 = T2W_DCM_PATHS[EX_SLICE+1]","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:03.313975Z","iopub.execute_input":"2021-07-27T16:29:03.314569Z","iopub.status.idle":"2021-07-27T16:29:07.946618Z","shell.execute_reply.started":"2021-07-27T16:29:03.314507Z","shell.execute_reply":"2021-07-27T16:29:07.945625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.1 BASIC STRUCTURE OF FLAIR DICOM

\n\n---\n\n
\n","metadata":{}},{"cell_type":"code","source":"flair_ex_dcm_1 = pydicom.read_file(FLAIR_EXAMPLE_DCM_1)\nflair_ex_dcm_2 = pydicom.read_file(FLAIR_EXAMPLE_DCM_2)\nFLAIR_ATTRS = flair_ex_dcm_1.dir()\n\nprint(\"FLAIR ATTRIBUTES:\")\nfor k in FLAIR_ATTRS: print(f\"\\t--> {k:<30}( Same Across Dicoms = {str(flair_ex_dcm_1.get(k)==flair_ex_dcm_2.get(k)):<5} )\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:07.948045Z","iopub.execute_input":"2021-07-27T16:29:07.948431Z","iopub.status.idle":"2021-07-27T16:29:07.990827Z","shell.execute_reply.started":"2021-07-27T16:29:07.948393Z","shell.execute_reply":"2021-07-27T16:29:07.99005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.2 BASIC STRUCTURE OF T1W DICOM

\n\n---\n\n
\n","metadata":{}},{"cell_type":"code","source":"t1w_ex_dcm_1 = pydicom.read_file(T1W_EXAMPLE_DCM_1)\nt1w_ex_dcm_2 = pydicom.read_file(T1W_EXAMPLE_DCM_2)\nT1W_ATTRS = t1w_ex_dcm_1.dir()\n\nprint(\"T1W ATTRIBUTES:\")\nfor k in T1W_ATTRS: print(f\"\\t--> {k:<30}( Same Across Two Dicoms = {str(t1w_ex_dcm_1.get(k)==t1w_ex_dcm_2.get(k)):<5} )\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:07.992002Z","iopub.execute_input":"2021-07-27T16:29:07.99232Z","iopub.status.idle":"2021-07-27T16:29:08.028714Z","shell.execute_reply.started":"2021-07-27T16:29:07.992285Z","shell.execute_reply":"2021-07-27T16:29:08.027872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.3 BASIC STRUCTURE OF T1WCE DICOM

\n\n---\n\n
\n\nThis dicom has an additional two fields that the other dicoms don't have:\n* **`InPlanePhaseEncodingDirection`**\n* **`InStackPositionNumber`**","metadata":{}},{"cell_type":"code","source":"t1wce_ex_dcm_1 = pydicom.read_file(T1WCE_EXAMPLE_DCM_1)\nt1wce_ex_dcm_2 = pydicom.read_file(T1WCE_EXAMPLE_DCM_2)\nT1WCE_ATTRS = t1wce_ex_dcm_1.dir()\n\nprint(\"T1WCE ATTRIBUTES:\")\nfor k in T1WCE_ATTRS: print(f\"\\t--> {k:<30}( Same Across Two Dicoms = {str(t1wce_ex_dcm_1.get(k)==t1wce_ex_dcm_2.get(k)):<5} )\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:08.031846Z","iopub.execute_input":"2021-07-27T16:29:08.03212Z","iopub.status.idle":"2021-07-27T16:29:08.062162Z","shell.execute_reply.started":"2021-07-27T16:29:08.032093Z","shell.execute_reply":"2021-07-27T16:29:08.061451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.4 BASIC STRUCTURE OF T2W DICOM

\n\n---\n\n
\n","metadata":{}},{"cell_type":"code","source":"t2w_ex_dcm_1 = pydicom.read_file(T2W_EXAMPLE_DCM_1)\nt2w_ex_dcm_2 = pydicom.read_file(T2W_EXAMPLE_DCM_2)\nT2W_ATTRS = t2w_ex_dcm_1.dir()\n\nprint(\"T2W ATTRIBUTES:\")\nfor k in T2W_ATTRS: print(f\"\\t--> {k:<30}( Same Across Two Dicoms = {str(t2w_ex_dcm_1.get(k)==t2w_ex_dcm_2.get(k)):<5} )\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:08.064Z","iopub.execute_input":"2021-07-27T16:29:08.064312Z","iopub.status.idle":"2021-07-27T16:29:08.093455Z","shell.execute_reply.started":"2021-07-27T16:29:08.064276Z","shell.execute_reply":"2021-07-27T16:29:08.092729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.5 CAPTURE ALL OPTIONS AND VALUE COUNTS FOR DICOM

\n\n---\n\n
\n\nWe will not be capturing dicom metadata for UIDs or PixelData (this would cause OOM issues).\n\n
\n
⏱⏱⏱   NOTE: THIS CELL WILL TAKE ~15 MINUTES TO RUN   ⏱⏱⏱

\n
\n","metadata":{}},{"cell_type":"code","source":"t1 = time.time()\n\nDESIRED_ATTRS = [x for x in T1WCE_ATTRS if (x!=\"PixelData\" and \"UID\" not in x)]\ndicom_df = pd.DataFrame({\"dcm_path\":ALL_DICOM_PATHS})\ndicom_df = dicom_df.reindex(columns=([\"dcm_path\",]+DESIRED_ATTRS))\n\nprint(f\"\\n ... Processing Started...\")\n\n# Regular method\n# - for 10,000 dicom files the regular takes 94.639 seconds\n# - processing will take (350000/10000)*94.639 ~= 3325 seconds ~= 56 minutes (actual is 71 minutes)\n\n##### dicom_df = dicom_df.progress_apply(lambda x: get_dicom_meta(x, DESIRED_ATTRS), axis=1)\n\n# Use Multiprocessing (Pandarallel)\n# - no progress bar but approximately 3.75 times faster\n# - for 10,000 dicom files pandarallel takes 25.529 seconds\n# - processing will take (350000/10000)*25.529 ~= 893 seconds ~= 15 minutes (actual is 19 minutes)\ndicom_df = dicom_df.parallel_apply(lambda x: get_dicom_meta(x, DESIRED_ATTRS), axis=1)\n\nprint(f\"... Processing Completed in >>>>> {time.time()-t1} <<<<< Seconds ...\\n\")\n\ndisplay(dicom_df)","metadata":{"execution":{"iopub.status.busy":"2021-07-27T18:00:36.103113Z","iopub.execute_input":"2021-07-27T18:00:36.103452Z","iopub.status.idle":"2021-07-27T18:32:07.968211Z","shell.execute_reply.started":"2021-07-27T18:00:36.10342Z","shell.execute_reply":"2021-07-27T18:32:07.967415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.6 PERFORM ANALYSIS ON EACH COLUMN

\n\n---\n\nWe will first identify all of the possible values (including NaN) for a given attribute as well as their respective occurence counts.","metadata":{}},{"cell_type":"code","source":"for attr in DESIRED_ATTRS:\n print(f\"\\n\\nAttribute Name: {attr} ({dicom_df[attr].astype(str).nunique()} Different Values)\\n\")\n for i, (k,v) in enumerate(dicom_df[attr].astype(str).value_counts(dropna=False).items()): \n print(f\"\\t{repr(k):<50} --> Occurs {v} Times In Training Data\")\n if i == 10:\n print(\"\\n\\t--------- TRUNCATED ------ ENDING EARLY ------ TRUNCATED ---------\")\n break","metadata":{"execution":{"iopub.status.busy":"2021-07-27T18:32:07.969929Z","iopub.execute_input":"2021-07-27T18:32:07.970276Z","iopub.status.idle":"2021-07-27T18:32:47.152001Z","shell.execute_reply.started":"2021-07-27T18:32:07.97024Z","shell.execute_reply":"2021-07-27T18:32:47.151146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

5  LOAD 3D REPRESENTATIONS OF EXAMPLES

\n\nI have gone ahead and created 3D numpy arrays containing the images interpolated to a fixed size (512,512,128) or (128,128,32). Please see this notebook for details on how this was completed.\n\n
\n\n**NOTE SOME IDS DO NOT POSSESS RESPECTIVE SCANS WITH NON-ZERO VALUES**\n* FLAIR --> ['00709.npz', '00109.npz']\n* T1w --> ['00123.npz']\n* T1wCE --> []\n* T2w --> ['00123.npz']","metadata":{}},{"cell_type":"markdown","source":"
\n\n

5.1 SETUP FILE PATHS

\n\n---","metadata":{}},{"cell_type":"code","source":"# train_ids = [f\"{x:>05}.npz\" for x in os.listdir(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train\")]\n\n# print([x for x in train_ids if x not in os.listdir(FLAIR_NPY_DIR)])\n# print([x for x in train_ids if x not in os.listdir(T1w_NPY_DIR)])\n# print([x for x in train_ids if x not in os.listdir(T1wCE_NPY_DIR)])\n# print([x for x in train_ids if x not in os.listdir(T2w_NPY_DIR)])\n\n# total_sum = 0\n# IDX = \"00026\"\n# TYPE = \"T2w\"\n# for i in range(1000):\n# try:\n# total_sum+=pydicom.read_file([os.path.join(f\"/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train/{IDX}/{TYPE}\", x) for x in os.listdir(f\"/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train/{IDX}/{TYPE}\")][i]).pixel_array.max()\n# except:\n# break\n \n# print(total_sum)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_NPY_DIR = \"/kaggle/input/create-3d-npz-rsna-radiogenomic-classification/train\"\n# TRAIN_NPY_DIR = \"/kaggle/input/rsna-radiogenomic-3dnumpy-512x512x128/train\"\nTEST_NPY_DIR = \"/kaggle/input/create-3d-npz-rsna-radiogenomic-classification/test\"\n\nTRAIN_FLAIR_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"FLAIR\")\nTRAIN_T1w_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T1w\")\nTRAIN_T1wCE_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T1wCE\")\nTRAIN_T2w_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T2w\")\n\nTEST_FLAIR_NPY_DIR = os.path.join(TEST_NPY_DIR, \"FLAIR\")\nTEST_T1w_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T1w\")\nTEST_T1wCE_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T1wCE\")\nTEST_T2w_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T2w\")\n\n\nprint(\"\\n... FILE COUNTS ...\\n\")\nprint(f\"\\t--> The number of FLAIR numpy files is: {len(os.listdir(TRAIN_FLAIR_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T1w numpy files is: {len(os.listdir(TRAIN_T1w_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T1wCE numpy files is: {len(os.listdir(TRAIN_T1wCE_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T2w numpy files is: {len(os.listdir(TRAIN_T2w_NPY_DIR))} ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:08.096099Z","iopub.execute_input":"2021-07-27T16:29:08.096342Z","iopub.status.idle":"2021-07-27T16:29:09.062727Z","shell.execute_reply.started":"2021-07-27T16:29:08.096318Z","shell.execute_reply":"2021-07-27T16:29:09.06199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.2 UPDATE THE TRAIN DATAFRAME

\n\n---","metadata":{}},{"cell_type":"code","source":"train_df[\"path_to_flair_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_FLAIR_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t1w_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T1w_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t1wce_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T1wCE_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t2w_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T2w_NPY_DIR, f\"{x:>05}.npz\"))\n\nss_df[\"path_to_flair_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_FLAIR_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t1w_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T1w_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t1wce_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T1wCE_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t2w_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T2w_NPY_DIR, f\"{x:>05}.npz\"))\n\ndisplay(train_df.head())\ndisplay(ss_df.head())","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:09.065361Z","iopub.execute_input":"2021-07-27T16:29:09.065728Z","iopub.status.idle":"2021-07-27T16:29:09.120643Z","shell.execute_reply.started":"2021-07-27T16:29:09.065694Z","shell.execute_reply":"2021-07-27T16:29:09.119721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

5.3 VISUALIZE A RANDOM SCAN

\n\n---","metadata":{}},{"cell_type":"code","source":"RANDOM_IMG_IDX = int(random.random()*len(train_df))\nprint(f\"\\n... Visualize Single Flair Scan From Disk - IDX={RANDOM_IMG_IDX} ...\\n\")\ndisplay(create_animation(load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX])))\n\nRANDOM_IMG_IDX = int(random.random()*len(train_df))\nprint(f\"\\n... Visualize Set of mpMRI Scans From Disk - IDX={RANDOM_IMG_IDX} ...\\n\")\ndisplay(create_4scan_animation(\n load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t1w_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t1wce_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t2w_npz[RANDOM_IMG_IDX]),\n))\n\nprint(train_df.path_to_flair_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t1w_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t1wce_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t2w_npz[RANDOM_IMG_IDX])","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:09.121862Z","iopub.execute_input":"2021-07-27T16:29:09.122387Z","iopub.status.idle":"2021-07-27T16:29:14.00872Z","shell.execute_reply.started":"2021-07-27T16:29:09.122348Z","shell.execute_reply":"2021-07-27T16:29:14.007795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def get_3d_effnet(dropout=0.5):\n# \"\"\"Build a 3D convolutional neural network model.\"\"\"\n\n# inputs = tf.keras.layers.Input(shape=INPUT_SHAPE)\n# pp_inputs = tf.keras.applications.efficientnet.preprocess_input(inputs)\n# x = efn.EfficientNetB0(input_shape=INPUT_SHAPE, pooling=\"avg\")(pp_inputs)\n \n# x = tf.keras.layers.Dropout(dropout)(x)\n# x = tf.keras.layers.Dense(units=256, activation=\"relu\")(x)\n\n# x = tf.keras.layers.Dropout(dropout/1.25)(x)\n# outputs = tf.keras.layers.Dense(units=1, activation=\"sigmoid\")(x)\n\n# # Define the model.\n# model = tf.keras.Model(inputs, outputs, name=\"3d_mpmri_cnn\")\n# return model\n\n# model = get_3d_effnet()\n# print(model.predict(tf.repeat(load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX]).reshape(1, *INPUT_SHAPE[:-1], 1), axis=-1, repeats=3)))\n","metadata":{"jupyter":{"source_hidden":true}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

6  MODELLING

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

6.1 CREATE THE MODELS

\n\n---\n\n
\n\nOne for each type of scan\n\n
","metadata":{}},{"cell_type":"code","source":"INPUT_SHAPE = (128,128,32,1)\nBATCH_SIZE = 8\n\ndef get_3d_model(width=128, height=128, depth=32, dropout=0.3):\n \"\"\"Build a 3D convolutional neural network model.\"\"\"\n\n inputs = tf.keras.layers.Input(shape=(width, height, depth, 1))\n\n # print(x.shape)\n # x1 = tf.keras.layers.Conv3D(filters=64, kernel_size=1, activation=None, padding=\"same\")(inputs)\n # x1 = tf.keras.layers.MaxPool3D(pool_size=2)(x1)\n # x1 = tf.keras.layers.BatchNormalization()(x1)\n \n x = tf.keras.layers.Conv3D(filters=64, kernel_size=3, activation=\"relu\")(inputs)\n x = tf.keras.layers.MaxPool3D(pool_size=2)(x)\n x = tf.keras.layers.BatchNormalization()(x)\n \n # x3 = tf.keras.layers.Conv3D(filters=64, kernel_size=5, activation=None, padding=\"same\")(inputs)\n # x3 = tf.keras.layers.MaxPool3D(pool_size=2)(x3)\n # x3 = tf.keras.layers.BatchNormalization()(x3)\n # x = tf.keras.layers.Add()([x1,x2,x3])\n # x = tf.keras.layers.Activation(\"relu\")(x)\n \n x = tf.keras.layers.Conv3D(filters=64, kernel_size=3, activation=\"relu\")(x)\n x = tf.keras.layers.MaxPool3D(pool_size=2)(x)\n x = tf.keras.layers.BatchNormalization()(x)\n\n # print(x.shape)\n x = tf.keras.layers.Conv3D(filters=128, kernel_size=3, activation=\"relu\")(x)\n # x = tf.keras.layers.MaxPool3D(pool_size=2)(x)\n x = tf.keras.layers.BatchNormalization()(x)\n\n # print(x.shape)\n x = tf.keras.layers.Conv3D(filters=256, kernel_size=3, activation=\"relu\")(x)\n # x = tf.keras.layers.MaxPool3D(pool_size=2)(x)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Dropout(dropout/2)(x)\n \n # print(x.shape)\n x = tf.keras.layers.GlobalAveragePooling3D()(x)\n x = tf.keras.layers.Dense(units=128, activation=\"relu\")(x)\n x = tf.keras.layers.Dropout(dropout)(x)\n\n # print(x.shape)\n outputs = tf.keras.layers.Dense(units=1, activation=\"sigmoid\")(x)\n\n # Define the model.\n model = tf.keras.Model(inputs, outputs, name=\"3d_mpmri_cnn\")\n return model\n\n\n# Build model.\nDROPOUT = 0.666\nflair_model = get_3d_model(width=INPUT_SHAPE[0], height=INPUT_SHAPE[1], depth=INPUT_SHAPE[2], dropout=DROPOUT)\nt1w_model = get_3d_model(width=INPUT_SHAPE[0], height=INPUT_SHAPE[1], depth=INPUT_SHAPE[2], dropout=DROPOUT)\nt1wce_model = get_3d_model(width=INPUT_SHAPE[0], height=INPUT_SHAPE[1], depth=INPUT_SHAPE[2], dropout=DROPOUT)\nt2w_model = get_3d_model(width=INPUT_SHAPE[0], height=INPUT_SHAPE[1], depth=INPUT_SHAPE[2], dropout=DROPOUT)\nmodels = {\"flair\":flair_model, \"t1w\":t1w_model, \"t1wce\":t1wce_model, \"t2w\":t2w_model}\n\nprint(\"\\n... EXAMPLE PREDICTION ...\\n\")\nprint(flair_model.predict(load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX]).reshape(1, *INPUT_SHAPE)))\n\nprint(\"\\n\\n... MODEL SUMMARY ...\\n\")\nflair_model.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:35.704965Z","iopub.execute_input":"2021-07-27T16:29:35.705299Z","iopub.status.idle":"2021-07-27T16:29:41.846713Z","shell.execute_reply.started":"2021-07-27T16:29:35.705264Z","shell.execute_reply":"2021-07-27T16:29:41.845812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.2 CREATE THE DATASETS

\n\n---","metadata":{}},{"cell_type":"code","source":"# This is dataframe wrangling\nUSE_COL = \"all\" # one of [`all`, `path_to_t1w_npz`, `path_to_t1wce_npz`, `path_to_flair_npz`, `path_to_t2w_npz`]\nif USE_COL is \"all\":\n USE_COLS = [\"path_to_t1w_npz\", \"path_to_t1wce_npz\", \"path_to_flair_npz\", \"path_to_t2w_npz\"]\nelse:\n USE_COLS = [USE_COL,]\n\n# Initialize dictionaries\ndataset_storage = {}\n\nfor use_col in USE_COLS:\n col_identifier = use_col.split(\"_\")[2]\n dataset_storage[col_identifier] = {}\n \n dataset_storage[col_identifier][\"train_ds_df\"] = train_df[train_df[use_col].apply(lambda x: True if os.path.isfile(x) else False)][[\"MGMT_value\", use_col]]\n dataset_storage[col_identifier][\"train_lbl_list\"] = dataset_storage[col_identifier][\"train_ds_df\"].MGMT_value.to_list()\n dataset_storage[col_identifier][\"train_npz_file_list\"] = dataset_storage[col_identifier][\"train_ds_df\"][use_col].to_list()\n\n # fix this.......\n dataset_storage[col_identifier][\"test_ds_df\"] = ss_df[ss_df[use_col].apply(lambda x: True if os.path.isfile(x) else False)][[\"BraTS21ID\", use_col]]\n dataset_storage[col_identifier][\"test_id_list\"] = dataset_storage[col_identifier][\"test_ds_df\"].BraTS21ID.to_list()\n dataset_storage[col_identifier][\"test_npz_file_list\"] = dataset_storage[col_identifier][\"test_ds_df\"][use_col].to_list()\n\n # This is for splitting\n dataset_storage[col_identifier][\"N_EX\"] = len(dataset_storage[col_identifier][\"train_lbl_list\"])\n dataset_storage[col_identifier][\"VAL_FRAC\"] = 0.1\n dataset_storage[col_identifier][\"N_TRAIN\"] = int(dataset_storage[col_identifier][\"N_EX\"]*(1-dataset_storage[col_identifier][\"VAL_FRAC\"]))\n dataset_storage[col_identifier][\"N_VAL\"] = dataset_storage[col_identifier][\"N_EX\"]-dataset_storage[col_identifier][\"N_TRAIN\"]\n dataset_storage[col_identifier][\"RANDOM_INDICES\"] = random.sample(range(dataset_storage[col_identifier][\"N_EX\"]), dataset_storage[col_identifier][\"N_EX\"])\n dataset_storage[col_identifier][\"TRAIN_INDICES\"] = dataset_storage[col_identifier][\"RANDOM_INDICES\"][:dataset_storage[col_identifier][\"N_TRAIN\"]]\n dataset_storage[col_identifier][\"VAL_INDICES\"] = dataset_storage[col_identifier][\"RANDOM_INDICES\"][dataset_storage[col_identifier][\"N_TRAIN\"]:]\n\n # TF Data Integration\n dataset_storage[col_identifier][\"lbl_train_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_lbl_list\"])[dataset_storage[col_identifier][\"TRAIN_INDICES\"]])\n dataset_storage[col_identifier][\"npz_file_train_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_npz_file_list\"])[dataset_storage[col_identifier][\"TRAIN_INDICES\"]])\n dataset_storage[col_identifier][\"raw_train_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_train_ds\"], dataset_storage[col_identifier][\"lbl_train_ds\"]))\n\n dataset_storage[col_identifier][\"lbl_val_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_lbl_list\"])[dataset_storage[col_identifier][\"VAL_INDICES\"]])\n dataset_storage[col_identifier][\"npz_file_val_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_npz_file_list\"])[dataset_storage[col_identifier][\"VAL_INDICES\"]])\n dataset_storage[col_identifier][\"raw_val_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_val_ds\"], dataset_storage[col_identifier][\"lbl_val_ds\"]))\n\n dataset_storage[col_identifier][\"id_test_ds\"] = tf.data.Dataset.from_tensor_slices(dataset_storage[col_identifier][\"test_id_list\"])\n dataset_storage[col_identifier][\"npz_file_test_ds\"] = tf.data.Dataset.from_tensor_slices(dataset_storage[col_identifier][\"test_npz_file_list\"])\n dataset_storage[col_identifier][\"raw_test_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_test_ds\"], dataset_storage[col_identifier][\"id_test_ds\"]))\n\n if INPUT_SHAPE[-1]==3:\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"raw_train_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"train_ds\"].shuffle(BATCH_SIZE*5).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"raw_val_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"val_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"raw_test_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.int32)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"test_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n else:\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"raw_train_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"train_ds\"].shuffle(BATCH_SIZE*5).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"raw_val_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"val_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"raw_test_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.int32)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"test_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\nfor k, v in dataset_storage.items():\n print(f\"\\n\\n... SCAN TYPE: {k.upper()}\")\n print(\"\\t-->\", v[\"train_ds\"])\n print(\"\\t-->\", v[\"val_ds\"])\n print(\"\\t-->\", v[\"test_ds\"])\nprint(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T18:45:05.230699Z","iopub.execute_input":"2021-07-27T18:45:05.231053Z","iopub.status.idle":"2021-07-27T18:45:07.921266Z","shell.execute_reply.started":"2021-07-27T18:45:05.231021Z","shell.execute_reply":"2021-07-27T18:45:07.920485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.3 VISUALIZE A FEW EXAMPLES

\n\n---","metadata":{}},{"cell_type":"code","source":"def plot_batch(dataset):\n r_idx = int(random.random()*BATCH_SIZE)\n \n img, lbl = next(iter(dataset))\n \n img = tf.squeeze(img[r_idx])\n lbl = lbl[r_idx]\n \n plt.figure(figsize=(20,10.5), )\n plt.suptitle(f\"LABEL = {label_map_int_2_str[lbl.numpy()]} ({lbl.numpy()})\", fontsize=20, fontweight=\"bold\")\n for i in range(img.shape[-1]):\n plt.subplot(4,8,i+1)\n plt.imshow(img[..., i], cmap=\"gray\")\n plt.axis(False)\n \n plt.tight_layout(h_pad=0.025, w_pad=0.025)\n # plt.subplots_adjust(wspace=0.001, hspace=0.001)\n plt.show()\n \n\nfor k, v in dataset_storage.items():\n print(\"\\n\\n\\n\\n\\n\\n\"+\"-\"*35)\n print(\"-\"*35)\n print(f\"\\t SCAN TYPE: {k.upper()}\")\n print(\"-\"*35)\n print(\"-\"*35,\"\\n\\n\")\n \n print(f\"\\n\\n... DEMO TRAIN {k.upper()} EXAMPLE ...\\n\\n\")\n plot_batch(v[\"train_ds\"])\n \n print(f\"\\n\\n... DEMO VALIDATION {k.upper()} EXAMPLE ...\\n\\n\")\n plot_batch(v[\"val_ds\"])\n\nprint(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:29:48.055409Z","iopub.execute_input":"2021-07-27T16:29:48.055719Z","iopub.status.idle":"2021-07-27T16:30:04.73318Z","shell.execute_reply.started":"2021-07-27T16:29:48.055691Z","shell.execute_reply":"2021-07-27T16:30:04.73233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.4 LEARNING RATE FUNCTION/PLOT

\n\n---","metadata":{}},{"cell_type":"code","source":"# Learning Rate\nMAX_STEPS = 20000\nINIT_LR = 0.00015\nDECAY_RATE = 0.98\nDECAY_STEPS = int((len(train_df)/BATCH_SIZE)*5) # 5 epochs = 1 decay\nlr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(\n INIT_LR, decay_steps=DECAY_STEPS, decay_rate=DECAY_RATE, staircase=True\n)\n\n# Plot Learning Rate\nplt.figure(figsize=(10,10))\nplt.plot(range(0, MAX_STEPS, int(len(train_df)/BATCH_SIZE)), [lr_schedule(i) for i in range(0, MAX_STEPS, int(len(train_df)/BATCH_SIZE))], linewidth=3)\nplt.vlines(range(int(len(train_df)/BATCH_SIZE*5), MAX_STEPS, int(len(train_df)/BATCH_SIZE*5)), ymin=0, ymax=INIT_LR, linestyles=\"dotted\", colors='gray')\nplt.vlines(range(int(len(train_df)/BATCH_SIZE*50), MAX_STEPS, int(len(train_df)/BATCH_SIZE*50)), ymin=0, ymax=INIT_LR, colors='r')\nplt.title(\"Learning Rate\\nDotted Lines At Every 5 Epochs (~350 steps)\\nSolid Red Lines At Every 50 Epochs\", fontweight=\"bold\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-28T15:58:47.687504Z","iopub.execute_input":"2021-07-28T15:58:47.687862Z","iopub.status.idle":"2021-07-28T15:58:48.124604Z","shell.execute_reply.started":"2021-07-28T15:58:47.687830Z","shell.execute_reply":"2021-07-28T15:58:48.122780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.5 TRAINING LOOP

\n\n---","metadata":{}},{"cell_type":"code","source":"# Compile model\nN_EPOCHS=250\nPATIENCE=50\n\nhistories = {}\nfor model_style, model in models.items():\n print(\"\\n\\n\\n\\n\\n\\n\\n\\n\")\n print(\"-\"*80)\n print(\"-\"*20+\"\\t\"+f\"{model_style.upper()} TRAINING LOOP\"+\"\\t\"+\"-\"*20)\n print(\"-\"*80)\n print(\"\\n\\n\\n\")\n \n # Get datasets\n train_ds = dataset_storage[model_style][\"train_ds\"]\n val_ds = dataset_storage[model_style][\"val_ds\"]\n \n lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(\n INIT_LR, decay_steps=DECAY_STEPS, decay_rate=DECAY_RATE, staircase=True\n )\n\n model.compile(\n loss=\"binary_crossentropy\",\n optimizer=tf.keras.optimizers.Adam(learning_rate=lr_schedule),\n metrics=[\"acc\", tf.keras.metrics.AUC(name=\"auc\")],\n )\n\n # Define callbacks.\n CKPT_DIR = os.path.join(\"/kaggle/working/model_files\", model_style)\n if not os.path.isdir(CKPT_DIR): \n os.makedirs(CKPT_DIR, exist_ok=True)\n checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\n os.path.join(CKPT_DIR, \"epoch_{epoch:02d}--val_loss_{val_loss:.2f}--val_auc_{val_auc:.2f}\"), \n save_best_only=True, monitor=\"val_auc\", mode=\"max\",\n )\n early_stopping_cb = tf.keras.callbacks.EarlyStopping(monitor=\"val_auc\", patience=PATIENCE, restore_best_weights=True, mode=\"max\")\n \n # Train the model, doing validation at the end of each epoch\n history = model.fit(\n train_ds,\n validation_data=val_ds,\n epochs=N_EPOCHS,\n callbacks=[checkpoint_cb, early_stopping_cb],\n )\n histories[model_style] = history","metadata":{"execution":{"iopub.status.busy":"2021-07-27T16:32:17.270225Z","iopub.execute_input":"2021-07-27T16:32:17.270579Z","iopub.status.idle":"2021-07-27T17:52:37.445652Z","shell.execute_reply.started":"2021-07-27T16:32:17.270538Z","shell.execute_reply":"2021-07-27T17:52:37.444827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.6 PLOT MODEL TRAINING

\n\n---\n\nhTBD","metadata":{}},{"cell_type":"code","source":"histories","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.7 MODEL PREDICTIONS

\n\n---","metadata":{}},{"cell_type":"code","source":"model_preds_dfs = {}\nfor model_style, model in tqdm(models.items(), total=4):\n model_preds_dfs[model_style] = pd.DataFrame()\n model_preds_dfs[model_style][\"BraTS21ID\"] = flatten_l_o_l([x.tolist() for x in list(dataset_storage[model_style][\"test_ds\"].map(lambda x,y: y).as_numpy_iterator())])\n model_preds_dfs[model_style][model_style+\"_MGMT_value\"] = tf.squeeze(model.predict(dataset_storage[model_style][\"test_ds\"].map(lambda x,y: x)))\n \nss_df = ss_df[[\"BraTS21ID\"]]\nfor style, style_df in model_preds_dfs.items():\n ss_df = pd.merge(ss_df, style_df, on=\"BraTS21ID\", how=\"left\")\nss_df[[\"BraTS21ID\", \"flair_MGMT_value\", \"t1w_MGMT_value\", \"t1wce_MGMT_value\", \"t2w_MGMT_value\"]].to_csv(\"submission.csv\", index=False)\ndisplay(ss_df)","metadata":{"execution":{"iopub.status.busy":"2021-07-27T18:55:59.626369Z","iopub.execute_input":"2021-07-27T18:55:59.626756Z","iopub.status.idle":"2021-07-27T18:55:59.644908Z","shell.execute_reply.started":"2021-07-27T18:55:59.626721Z","shell.execute_reply":"2021-07-27T18:55:59.64383Z"},"trusted":true},"execution_count":null,"outputs":[]}]}