{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n\n\n
\n\n---\n\n
\n\n

CREATION OF 3D NUMPY ARRAYS

TOWARDS BRAIN TUMOR CLASSIFICATION

\n\n
\n \n
CREATED BY: DARIEN SCHETTLER

\n\n---\n\n
\n\n
\n
πŸ›‘πŸ›‘πŸ›‘   CAUTION: THIS NOTEBOOK IS A WORK IN PROGRESS   πŸ›‘πŸ›‘πŸ›‘

\n
\n\n
\n\n\n---\n
","metadata":{"papermill":{"duration":0.036465,"end_time":"2021-06-04T20:52:51.125208","exception":false,"start_time":"2021-06-04T20:52:51.088743","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

TABLE OF CONTENTS

\n\n---\n\n

0    IMPORTS

\n\n---\n\n

1    BACKGROUND INFORMATION

\n\n---\n\n

2    SETUP

\n\n---\n\n

3    HELPER FUNCTIONS

\n\n---","metadata":{"papermill":{"duration":0.033613,"end_time":"2021-06-04T20:52:51.194111","exception":false,"start_time":"2021-06-04T20:52:51.160498","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS

","metadata":{"papermill":{"duration":0.03344,"end_time":"2021-06-04T20:52:51.261152","exception":false,"start_time":"2021-06-04T20:52:51.227712","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(\"\\n... PIP/APT INSTALLS STARTING ...\")\n!cp /kaggle/input/gdcm-conda-install/gdcm.tar .\n!tar -xvzf gdcm.tar\n!conda install --offline ./gdcm/gdcm-2.8.9-py37h71b2a6d_0.tar.bz2\n!rm -rf ./gdcm*\n\n# !conda install -c conda-forge gdcm -y\n!pip install pandarallel\nprint(\"... PIP/APT INSTALLS COMPLETE ...\\n\")\n\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n# Machine Learning and Data Science Imports\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('max_columns', 100);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\n\n# Other Competition Related Imports\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nfrom pandarallel import pandarallel; pandarallel.initialize();\n\n# Built In Imports\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom glob import glob\nimport warnings\nimport requests\nimport imageio\nimport IPython\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport scipy\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport matplotlib.patches as patches\n\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nimport plotly\nimport PIL\nimport cv2\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")\n \n\nprint(\"\\n... SEEDING FOR DETERMINISTIC BEHAVIOUR ...\")\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\nseed_it_all()\nprint(\"... SEEDING COMPLETE ...\\n\\n\")\n\nprint(\"\\n... SETTING PRESETS STARTING...\")\nFIG_FONT = dict(family=\"Helvetica, Arial\", size=14, color=\"#7f7f7f\")\nprint(\"... SETTING PRESETS COMPLETE...\\n\\n\")","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":51.261012,"end_time":"2021-06-04T20:53:42.555772","exception":false,"start_time":"2021-06-04T20:52:51.29476","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-28T16:05:27.207072Z","iopub.execute_input":"2021-07-28T16:05:27.207633Z","iopub.status.idle":"2021-07-28T16:06:17.143665Z","shell.execute_reply.started":"2021-07-28T16:05:27.207502Z","shell.execute_reply":"2021-07-28T16:06:17.142603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  BACKGROUND INFORMATION

","metadata":{"papermill":{"duration":0.040472,"end_time":"2021-06-04T20:53:42.636927","exception":false,"start_time":"2021-06-04T20:53:42.596455","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

1.1 OVERVIEW

\n\n---\n\nCOMPETITION DESCRIPTION\n\nA malignant tumor in the brain is a life-threatening condition. Known as glioblastoma, it's both the most common form of brain cancer in adults and the one with the worst prognosis, with median survival being less than a year. The presence of a specific genetic sequence in the tumor known as MGMT promoter methylation has been shown to be a favorable prognostic factor and a strong predictor of responsiveness to chemotherapy.\n\nCurrently, genetic analysis of cancer requires surgery to extract a tissue sample. Then it can take several weeks to determine the genetic characterization of the tumor. Depending upon the results and type of initial therapy chosen, a subsequent surgery may be necessary. If an accurate method to predict the genetics of the cancer through imaging (i.e., radiogenomics) alone could be developed, this would potentially minimize the number of surgeries and refine the type of therapy required.\n\nThe Radiological Society of North America (RSNA) has teamed up with the Medical Image Computing and Computer Assisted Intervention Society (the MICCAI Society) to improve diagnosis and treatment planning for patients with glioblastoma. In this competition you will predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to train and test your model to detect for the presence of MGMT promoter methylation.\n\nIf successful, you'll help brain cancer patients receive less invasive diagnoses and treatments. The introduction of new and customized treatment strategies before surgery has the potential to improve the management, survival, and prospects of patients with brain cancer.\n\n**Secondary Description From UPenn**\n> The participants are called to use the provided mpMRI data to extract imaging/radiomic features that they consider appropriate, and analyze them through machine learning algorithms, in an attempt to predict the MGMT promoter methylation status. The participants do not need to be limited to volumetric parameters, but can also consider intensity, morphologic, histogram-based, and textural features, as well as spatial information, deep learning features, and glioma diffusion properties extracted from glioma growth models.\n\n> Note that participants will be evaluated for the predicted MGMT status of the subjects indicated in the accompanying spreadsheet.\n\n
\n\n
\n
πŸ€”πŸ€”πŸ€”   MY INTERPRETATION OF THIS COMPETITION   πŸ€”πŸ€”πŸ€”


In this competition we are tasked with identifying/predicting the genetic subtype (genetic subtype = a group of tumors that is enriched for genetic aberrations in a set of subtype predictor genes) of glioblastoma (glioblastoma = brain tumor)

HUH?!?!?

All this means is that we are taking in dicom images containing 3D representations (slices) of a patients brain which we will process with computer vision algorithms to allow us to perform binary classification. The binary classification is to identify if, within the patients image data, MGMT promoter methylation is present. The diagram below shows a simple, if slightly inaccurate, representation of what is required.


\n
\n\n---\n\n
\n\nSUBMISSION EVALUATION/RESTRICTIONS AND FILE FORMAT\n\n**Submission Evaluation**\n* Submissions are evaluated on the [area under the ROC curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) between the predicted probability and the observed target.\n\n
\n\n**Submission Restrictions**\n* **THIS IS A KERNELS ONLY COMPETITION**\n * Submissions to this competition must be made through Notebooks. \n * In order for the \"Submit\" button to be active after a commit, the following conditions must be met:\n * *CPU Notebook <= 9 hours run-time*\n * *GPU Notebook <= 9 hours run-time*\n * *Internet access disabled*\n * *Freely & publicly available external data is allowed, including pre-trained models*\n * *Submission file must be named `submission.csv`*\n\n
\n\n**Submission File Format**\n* For each **`BraTS21ID`** in the test set, you must predict a probability for the target **`MGMT_value`**. The file should contain a header and have the following format:\n>```\n>BraTS21ID,MGMT_value\n>00001,0.5\n>00013,0.999\n>00015,0.1\n>etc.\n>```\n\n
\n\n---\n\n
\n\nCOMPETITION TIMELINE\n\n* **July 13, 2021** - Start Date.\n* **October 8, 2021** - Entry Deadline. You must accept the competition rules before this date in order to compete.\n* **October 8, 2021** - Team Merger Deadline. This is the last day participants may join or merge teams.\n* **October 15, 2021** - Final Submission Deadline.\n* **October 25, 2021** - Winners’ Requirements Deadline. This is the deadline for winners to submit to the host/Kaggle their training code, video, method description.\n\n> All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.\n\n","metadata":{"papermill":{"duration":0.041492,"end_time":"2021-06-04T20:53:42.719208","exception":false,"start_time":"2021-06-04T20:53:42.677716","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n

1.2 DATA DESCRIPTION

\n\n---\n\n
\n\nDATA SPLITS/COHORTS\n\nThe competition data is defined by three cohorts: **Training**, **Validation (Public)**, and **Testing (Private)**. \n* The **β€œTraining”** and the **β€œValidation”** cohorts are provided to the participants\n* The **β€œTesting”** cohort is kept hidden at all times, during and after the competition\n\nThese 3 cohorts are structured as follows: \n* Each independent case has a **dedicated folder identified by a five-digit number**.\n* Within each of these **β€œcase”** folders, there are four sub-folders\n * Each of these **\"case\"** subfolders corresponds to each of the structural **m**ulti-**p**arametric **MRI** (**mpMRI**) scans, in **DICOM** format. \n * The exact mpMRI scans included are:\n * Fluid Attenuated Inversion Recovery (FLAIR)\n * T1-weighted pre-contrast (T1w)\n * T1-weighted post-contrast (T1Gd)\n * T2-weighted (T2)\n\nExact folder structure:\n\n```\nTraining/Validation/Testing\nβ”‚\n└─── 00000\nβ”‚ β”‚\nβ”‚ └─── FLAIR\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T1w\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T1wCE\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ ...\nβ”‚ β”‚ \nβ”‚ └─── T2w\nβ”‚ β”‚ β”‚ Image-1.dcm\nβ”‚ β”‚ β”‚ Image-2.dcm\nβ”‚ β”‚ β”‚ .....\nβ”‚ \n└─── 00001\nβ”‚ β”‚ ...\nβ”‚ \nβ”‚ ... \nβ”‚ \n└─── 00002\nβ”‚ β”‚ ...\n\n```\n\nFILES\n\n**`train/`** \n- folder containing the training files, with each top-level folder representing a subject\n\n**`train_labels.csv`** \n- file containing the target MGMT_value for each subject in the training data (e.g. the presence of MGMT promoter methylation)\n\n**`test/`** \n- the test files, which use the same structure as train/; your task is to predict the MGMT_value for each subject in the test data. NOTE: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set\n\n**`sample_submission.csv`** \n- a sample submission file in the correct format","metadata":{"papermill":{"duration":0.041685,"end_time":"2021-06-04T20:53:42.803402","exception":false,"start_time":"2021-06-04T20:53:42.761717","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"
\n\n\n\n

2  SETUP

","metadata":{"papermill":{"duration":0.04017,"end_time":"2021-06-04T20:53:42.971481","exception":false,"start_time":"2021-06-04T20:53:42.931311","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ROOT_DIR = \"/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification\"\n\nTRAIN_DIR = os.path.join(ROOT_DIR, \"train\")\nTEST_DIR = os.path.join(ROOT_DIR, \"test\")\n\nSS_CSV = os.path.join(ROOT_DIR, \"sample_submission.csv\")\nTRAIN_CSV = os.path.join(ROOT_DIR, \"train_labels.csv\")\n\ntrain_df = pd.read_csv(TRAIN_CSV)\ntrain_df[\"path_to_flair_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"FLAIR\"))\ntrain_df[\"flair_image_count\"] = train_df.path_to_flair_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t1w_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T1w\"))\ntrain_df[\"t1w_image_count\"] = train_df.path_to_t1w_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t1wce_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T1wCE\"))\ntrain_df[\"t1wce_image_count\"] = train_df.path_to_t1wce_dir.progress_apply(lambda x: len(os.listdir(x)))\ntrain_df[\"path_to_t2w_dir\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_DIR, f\"{x:>05}\", \"T2w\"))\ntrain_df[\"t2w_image_count\"] = train_df.path_to_t2w_dir.progress_apply(lambda x: len(os.listdir(x)))\n \nss_df = pd.read_csv(SS_CSV)\nss_df[\"path_to_flair_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"FLAIR\"))\nss_df[\"flair_image_count\"] = ss_df.path_to_flair_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t1w_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T1w\"))\nss_df[\"t1w_image_count\"] = ss_df.path_to_t1w_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t1wce_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T1wCE\"))\nss_df[\"t1wce_image_count\"] = ss_df.path_to_t1wce_dir.progress_apply(lambda x: len(os.listdir(x)))\nss_df[\"path_to_t2w_dir\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_DIR, f\"{x:>05}\", \"T2w\"))\nss_df[\"t2w_image_count\"] = ss_df.path_to_t2w_dir.progress_apply(lambda x: len(os.listdir(x)))\n\nprint(\"\\n\\nTRAIN DATAFRAME\\n\")\ndisplay(train_df.head())\n\nprint(\"\\n\\n\\nSAMPLE SUBMISSION DATAFRAME\\n\")\ndisplay(ss_df.head())","metadata":{"papermill":{"duration":507.188703,"end_time":"2021-06-04T21:02:10.201009","exception":false,"start_time":"2021-06-04T20:53:43.012306","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-07-28T16:06:17.145178Z","iopub.execute_input":"2021-07-28T16:06:17.145452Z","iopub.status.idle":"2021-07-28T16:07:21.352324Z","shell.execute_reply.started":"2021-07-28T16:06:17.145426Z","shell.execute_reply":"2021-07-28T16:07:21.351167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

3  HELPER FUNCTIONS

","metadata":{"papermill":{"duration":0.053621,"end_time":"2021-06-04T21:02:10.308317","exception":false,"start_time":"2021-06-04T21:02:10.254696","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_list_of_dcm_paths(dir_path):\n return sorted([os.path.join(dir_path, f_name) for f_name in os.listdir(dir_path)], key=lambda x: int(x.rsplit(\"-\", 1)[1].split(\".\", 1)[0]))\n\ndef dicom2array(path, voi_lut=True, fix_monochrome=True):\n \"\"\" Convert dicom file to numpy array \n \n Args:\n path (str): Path to the dicom file to be converted\n voi_lut (bool): Whether or not VOI LUT is available\n fix_monochrome (bool): Whether or not to apply monochrome fix\n \n Returns:\n Numpy array of the respective dicom file \n \n \"\"\"\n # Use the pydicom library to read the dicom file\n dicom = pydicom.read_file(path)\n \n # VOI LUT (if available by DICOM device) is used to \n # transform raw DICOM data to \"human-friendly\" view\n if voi_lut:\n data = apply_voi_lut(dicom.pixel_array, dicom)\n else:\n data = dicom.pixel_array\n \n # The XRAY may look inverted\n # - If we want to fix this we can\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n \n # Normalize the image array and return\n data = (data-np.min(data))/(np.max(data)-np.min(data))\n return data\n\ndef create_animation(ims):\n fig = plt.figure(figsize=(4, 4))\n plt.axis('off')\n im = plt.imshow(ims[..., 0], cmap=\"bone\")\n def animate_func(i):\n im.set_array(ims[..., i])\n return [im]\n plt.close()\n return animation.FuncAnimation(fig, animate_func, frames = ims.shape[-1], interval = 1000//24)\n\ndef get_dicom_meta(row, attrs):\n dcm_file = pydicom.read_file(row.dcm_path)\n for val in attrs:\n row[val] = dcm_file.get(val, None)\n return row\n\ndef load_npz(np_file_path, is_tf=False):\n if is_tf:\n return np.load(np_file_path.numpy().decode())[\"arr_0\"] \n else:\n return np.load(np_file_path)[\"arr_0\"] \n \n \ndef flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\n\ndef create_4scan_animation(flair_arr, t1w_arr, t1wce_arr, t2w_arr):\n \n def animate_func(i):\n flair_im.set_array(flair_arr[..., i])\n t1w_im.set_array(t1w_arr[..., i])\n t1wce_im.set_array(t1wce_arr[..., i])\n t2w_im.set_array(t2w_arr[..., i])\n return [flair_im, t1w_im, t1wce_im, t2w_im]\n \n fig = plt.figure(figsize=(18, 5))\n \n #\n plt.subplot(1,4,1)\n plt.axis('off')\n flair_im = plt.imshow(flair_arr[..., 0], cmap=\"bone\")\n plt.title(\"FLAIR Animation\", fontweight=\"bold\")\n \n #\n plt.subplot(1,4,2)\n plt.axis('off')\n t1w_im = plt.imshow(t1w_arr[..., 0], cmap=\"bone\")\n plt.title(\"T1w Animation\", fontweight=\"bold\")\n \n #\n plt.subplot(1,4,3)\n plt.axis('off')\n t1wce_im = plt.imshow(t1wce_arr[..., 0], cmap=\"bone\")\n plt.title(\"T1wCE Animation\", fontweight=\"bold\")\n\n \n #\n plt.subplot(1,4,4)\n plt.axis('off')\n t2w_im = plt.imshow(t2w_arr[..., 0], cmap=\"bone\")\n plt.title(\"T2w Animation\", fontweight=\"bold\")\n \n plt.close()\n \n return animation.FuncAnimation(fig, animate_func, frames = flair_arr.shape[-1], interval = 1000//16)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:09:21.987341Z","iopub.execute_input":"2021-07-28T16:09:21.987870Z","iopub.status.idle":"2021-07-28T16:09:22.008040Z","shell.execute_reply.started":"2021-07-28T16:09:21.987839Z","shell.execute_reply":"2021-07-28T16:09:22.006872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

4  CREATE/LOAD 3D REPRESENTATIONS OF EXAMPLES

","metadata":{}},{"cell_type":"code","source":"# ############################# #\n# SMALL = ( 128, 128, 32 ) #\n# MEDIUM = ( 256, 256, 64 ) #\n# LARGE = ( 512, 512, 128 ) #\n# ############################# #\nINPUT_SHAPE = RESIZE_TO = ( 128, 128, 32 )\nDEMO_FLAIR = train_df.path_to_flair_dir.iloc[0]\n# ############################# #\n\nDEMO_FLAIR","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:10:20.927021Z","iopub.execute_input":"2021-07-28T16:10:20.927425Z","iopub.status.idle":"2021-07-28T16:10:20.935176Z","shell.execute_reply.started":"2021-07-28T16:10:20.927388Z","shell.execute_reply":"2021-07-28T16:10:20.933932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_numpy_arr(path_to_dir, resize_to=(512,512,128), save_to_disk=False, output_dir=\"/kaggle/working\"):\n\n # Get paths to all dicom files for a given brain 🧠 \n dicom_paths = get_list_of_dcm_paths(path_to_dir)\n\n # Get ref file\n ref_dicom = pydicom.read_file(dicom_paths[0])\n\n # Load dimensions based on the number of rows, columns, and slices (along the Z axis)\n original_img_dims = (int(ref_dicom.Rows), int(ref_dicom.Columns), len(dicom_paths))\n\n # Load spacing values (in mm)\n px_spacing = (float(ref_dicom.PixelSpacing[0]), float(ref_dicom.PixelSpacing[1]), float(ref_dicom.SliceThickness))\n\n # The array is sized based on dicom information gathered above\n np_arr_list = []\n\n # loop through all the DICOM files\n print(f\"\\n... Creating Numpy Array ...\\n\")\n for i, dcm_file in tqdm(enumerate(dicom_paths), total=len(dicom_paths)):\n\n # read the file\n dcm_slice = pydicom.read_file(dcm_file)\n \n # store the raw image data\n slice_arr = dcm_slice.pixel_array\n \n if slice_arr.max() == 0:\n continue\n else:\n slice_arr = ((slice_arr/np.max(slice_arr))*255).astype(np.uint8)\n \n # Add to the numpy slice list\n np_arr_list.append(slice_arr)\n \n \n # Stack the numpy slices into a numpy 3d array\n if len(np_arr_list)==0:\n return None\n \n np_arr = np.stack(np_arr_list, axis=-1)\n \n # Interpolate to the correct 3d shape\n print(f\"\\n... Interpoloating Numpy Array Starting - From {original_img_dims} – w/ {np_arr.shape[-1]} Non Empty Slizes – To {resize_to} ...\")\n np_arr = scipy.ndimage.zoom(np_arr, (resize_to[0]/np_arr.shape[0], resize_to[1]/np_arr.shape[1], resize_to[-1]/np_arr.shape[-1]))\n print(f\"... Interpoloating Completed ...\\n\")\n \n # Save to disk or return np array\n if save_to_disk:\n path_stuff = path_to_dir.rsplit(\"/\", 3)[1:]\n output_path = os.path.join(output_dir, path_stuff[0], path_stuff[2], path_stuff[1])\n if not os.path.isdir(output_path.rsplit(\"/\", 1)[0]):\n os.makedirs(output_path.rsplit(\"/\", 1)[0], exist_ok=True)\n \n print(f\"\\n... Writing Numpy Array to Disk Starting - {output_path} ...\")\n np.savez_compressed(output_path, np_arr)\n print(f\"... Writing Numpy Array to Disk Completed ...\\n\")\n \n else:\n return np_arr\n\ndef create_npz_arrays(row, resize_to=(512,512,128), output_dir=\"/kaggle/working\", save_to_disk=True):\n get_numpy_arr(row.path_to_flair_dir, resize_to=resize_to, save_to_disk=save_to_disk, output_dir=output_dir)\n get_numpy_arr(row.path_to_t1w_dir, resize_to=resize_to, save_to_disk=save_to_disk, output_dir=output_dir)\n get_numpy_arr(row.path_to_t1wce_dir, resize_to=resize_to, save_to_disk=save_to_disk, output_dir=output_dir)\n get_numpy_arr(row.path_to_t2w_dir, resize_to=resize_to, save_to_disk=save_to_disk, output_dir=output_dir)\n \ndemo_resized_np_arr = get_numpy_arr(DEMO_FLAIR, save_to_disk=False, resize_to=RESIZE_TO)","metadata":{"execution":{"iopub.status.busy":"2021-07-26T14:20:13.6707Z","iopub.execute_input":"2021-07-26T14:20:13.671433Z","iopub.status.idle":"2021-07-26T14:20:27.041179Z","shell.execute_reply.started":"2021-07-26T14:20:13.671379Z","shell.execute_reply":"2021-07-26T14:20:27.03991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Run to generate all npz files... only run if not already run\ntrain_df.parallel_apply(lambda x: create_npz_arrays(x, resize_to=RESIZE_TO), axis=1)\nss_df.parallel_apply(lambda x: create_npz_arrays(x, resize_to=RESIZE_TO), axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-07-26T14:20:35.830205Z","iopub.execute_input":"2021-07-26T14:20:35.830608Z","iopub.status.idle":"2021-07-26T14:28:15.498792Z","shell.execute_reply.started":"2021-07-26T14:20:35.830574Z","shell.execute_reply":"2021-07-26T14:28:15.480086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... Visualization After Resize From Memory ...\\n\")\ndisplay(create_animation(demo_resized_np_arr))\n\nprint(\"\\n... Visualization From Disk ...\\n\")\ndisplay(create_animation(np.load(\"./train/FLAIR/00000.npz\")[\"arr_0\"]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# a","metadata":{}},{"cell_type":"code","source":"TRAIN_NPY_DIR = \"/kaggle/input/create-3d-npz-rsna-radiogenomic-classification/train\"\n# TRAIN_NPY_DIR = \"/kaggle/input/rsna-radiogenomic-3dnumpy-512x512x128/train\"\nTEST_NPY_DIR = \"/kaggle/input/create-3d-npz-rsna-radiogenomic-classification/test\"\n\nTRAIN_FLAIR_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"FLAIR\")\nTRAIN_T1w_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T1w\")\nTRAIN_T1wCE_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T1wCE\")\nTRAIN_T2w_NPY_DIR = os.path.join(TRAIN_NPY_DIR, \"T2w\")\n\nTEST_FLAIR_NPY_DIR = os.path.join(TEST_NPY_DIR, \"FLAIR\")\nTEST_T1w_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T1w\")\nTEST_T1wCE_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T1wCE\")\nTEST_T2w_NPY_DIR = os.path.join(TEST_NPY_DIR, \"T2w\")\n\n\nprint(\"\\n... FILE COUNTS ...\\n\")\nprint(f\"\\t--> The number of FLAIR numpy files is: {len(os.listdir(TRAIN_FLAIR_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T1w numpy files is: {len(os.listdir(TRAIN_T1w_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T1wCE numpy files is: {len(os.listdir(TRAIN_T1wCE_NPY_DIR))} ...\")\nprint(f\"\\t--> The number of T2w numpy files is: {len(os.listdir(TRAIN_T2w_NPY_DIR))} ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:08:10.705040Z","iopub.execute_input":"2021-07-28T16:08:10.705418Z","iopub.status.idle":"2021-07-28T16:08:10.716784Z","shell.execute_reply.started":"2021-07-28T16:08:10.705376Z","shell.execute_reply":"2021-07-28T16:08:10.715896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"path_to_flair_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_FLAIR_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t1w_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T1w_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t1wce_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T1wCE_NPY_DIR, f\"{x:>05}.npz\"))\ntrain_df[\"path_to_t2w_npz\"] = train_df.BraTS21ID.apply(lambda x: os.path.join(TRAIN_T2w_NPY_DIR, f\"{x:>05}.npz\"))\n\nss_df[\"path_to_flair_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_FLAIR_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t1w_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T1w_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t1wce_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T1wCE_NPY_DIR, f\"{x:>05}.npz\"))\nss_df[\"path_to_t2w_npz\"] = ss_df.BraTS21ID.apply(lambda x: os.path.join(TEST_T2w_NPY_DIR, f\"{x:>05}.npz\"))\n\ndisplay(train_df.head())\ndisplay(ss_df.head())","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:08:29.223733Z","iopub.execute_input":"2021-07-28T16:08:29.224219Z","iopub.status.idle":"2021-07-28T16:08:29.278768Z","shell.execute_reply.started":"2021-07-28T16:08:29.224190Z","shell.execute_reply":"2021-07-28T16:08:29.277820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"RANDOM_IMG_IDX = int(random.random()*len(train_df))\nprint(f\"\\n... Visualize Single Flair Scan From Disk - IDX={RANDOM_IMG_IDX} ...\\n\")\ndisplay(create_animation(load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX])))\n\nRANDOM_IMG_IDX = int(random.random()*len(train_df))\nprint(f\"\\n... Visualize Set of mpMRI Scans From Disk - IDX={RANDOM_IMG_IDX} ...\\n\")\ndisplay(create_4scan_animation(\n load_npz(train_df.path_to_flair_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t1w_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t1wce_npz[RANDOM_IMG_IDX]),\n load_npz(train_df.path_to_t2w_npz[RANDOM_IMG_IDX]),\n))\n\nprint(train_df.path_to_flair_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t1w_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t1wce_npz[RANDOM_IMG_IDX])\nprint(train_df.path_to_t2w_npz[RANDOM_IMG_IDX])","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:09:24.913359Z","iopub.execute_input":"2021-07-28T16:09:24.914116Z","iopub.status.idle":"2021-07-28T16:09:29.906935Z","shell.execute_reply.started":"2021-07-28T16:09:24.914062Z","shell.execute_reply":"2021-07-28T16:09:29.905331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This is dataframe wrangling\nUSE_COL = \"all\" # one of [`all`, `path_to_t1w_npz`, `path_to_t1wce_npz`, `path_to_flair_npz`, `path_to_t2w_npz`]\nif USE_COL is \"all\":\n USE_COLS = [\"path_to_t1w_npz\", \"path_to_t1wce_npz\", \"path_to_flair_npz\", \"path_to_t2w_npz\"]\nelse:\n USE_COLS = [USE_COL,]\n\n# Initialize dictionaries\ndataset_storage = {}\n\nfor use_col in USE_COLS:\n col_identifier = use_col.split(\"_\")[2]\n dataset_storage[col_identifier] = {}\n \n dataset_storage[col_identifier][\"train_ds_df\"] = train_df[train_df[use_col].apply(lambda x: True if os.path.isfile(x) else False)][[\"MGMT_value\", use_col]]\n dataset_storage[col_identifier][\"train_lbl_list\"] = dataset_storage[col_identifier][\"train_ds_df\"].MGMT_value.to_list()\n dataset_storage[col_identifier][\"train_npz_file_list\"] = dataset_storage[col_identifier][\"train_ds_df\"][use_col].to_list()\n\n # fix this.......\n dataset_storage[col_identifier][\"test_ds_df\"] = ss_df[ss_df[use_col].apply(lambda x: True if os.path.isfile(x) else False)][[\"BraTS21ID\", use_col]]\n dataset_storage[col_identifier][\"test_id_list\"] = dataset_storage[col_identifier][\"test_ds_df\"].BraTS21ID.to_list()\n dataset_storage[col_identifier][\"test_npz_file_list\"] = dataset_storage[col_identifier][\"test_ds_df\"][use_col].to_list()\n\n # This is for splitting\n dataset_storage[col_identifier][\"N_EX\"] = len(dataset_storage[col_identifier][\"train_lbl_list\"])\n dataset_storage[col_identifier][\"VAL_FRAC\"] = 0.1\n dataset_storage[col_identifier][\"N_TRAIN\"] = int(dataset_storage[col_identifier][\"N_EX\"]*(1-dataset_storage[col_identifier][\"VAL_FRAC\"]))\n dataset_storage[col_identifier][\"N_VAL\"] = dataset_storage[col_identifier][\"N_EX\"]-dataset_storage[col_identifier][\"N_TRAIN\"]\n dataset_storage[col_identifier][\"RANDOM_INDICES\"] = random.sample(range(dataset_storage[col_identifier][\"N_EX\"]), dataset_storage[col_identifier][\"N_EX\"])\n dataset_storage[col_identifier][\"TRAIN_INDICES\"] = dataset_storage[col_identifier][\"RANDOM_INDICES\"][:dataset_storage[col_identifier][\"N_TRAIN\"]]\n dataset_storage[col_identifier][\"VAL_INDICES\"] = dataset_storage[col_identifier][\"RANDOM_INDICES\"][dataset_storage[col_identifier][\"N_TRAIN\"]:]\n\n # TF Data Integration\n dataset_storage[col_identifier][\"lbl_train_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_lbl_list\"])[dataset_storage[col_identifier][\"TRAIN_INDICES\"]])\n dataset_storage[col_identifier][\"npz_file_train_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_npz_file_list\"])[dataset_storage[col_identifier][\"TRAIN_INDICES\"]])\n dataset_storage[col_identifier][\"raw_train_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_train_ds\"], dataset_storage[col_identifier][\"lbl_train_ds\"]))\n\n dataset_storage[col_identifier][\"lbl_val_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_lbl_list\"])[dataset_storage[col_identifier][\"VAL_INDICES\"]])\n dataset_storage[col_identifier][\"npz_file_val_ds\"] = tf.data.Dataset.from_tensor_slices(np.array(dataset_storage[col_identifier][\"train_npz_file_list\"])[dataset_storage[col_identifier][\"VAL_INDICES\"]])\n dataset_storage[col_identifier][\"raw_val_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_val_ds\"], dataset_storage[col_identifier][\"lbl_val_ds\"]))\n\n dataset_storage[col_identifier][\"id_test_ds\"] = tf.data.Dataset.from_tensor_slices(dataset_storage[col_identifier][\"test_id_list\"])\n dataset_storage[col_identifier][\"npz_file_test_ds\"] = tf.data.Dataset.from_tensor_slices(dataset_storage[col_identifier][\"test_npz_file_list\"])\n dataset_storage[col_identifier][\"raw_test_ds\"] = tf.data.Dataset.zip((dataset_storage[col_identifier][\"npz_file_test_ds\"], dataset_storage[col_identifier][\"id_test_ds\"]))\n\n if INPUT_SHAPE[-1]==3:\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"raw_train_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"train_ds\"].shuffle(BATCH_SIZE*5).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"raw_val_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"val_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"raw_test_ds\"].map(lambda x,y: (tf.repeat(tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), (*INPUT_SHAPE[:-1], 1))/255, axis=-1, repeats=3), tf.cast(y, tf.int32)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"test_ds\"].batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n else:\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"raw_train_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"train_ds\"] = dataset_storage[col_identifier][\"train_ds\"]\n\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"raw_val_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.uint8)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"val_ds\"] = dataset_storage[col_identifier][\"val_ds\"]\n\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"raw_test_ds\"].map(lambda x,y: (tf.reshape(tf.py_function(\n load_npz, [x, True], (tf.uint8,)\n ), INPUT_SHAPE)/255, tf.cast(y, tf.int32)), num_parallel_calls=tf.data.AUTOTUNE)\n dataset_storage[col_identifier][\"test_ds\"] = dataset_storage[col_identifier][\"test_ds\"]\n\nfor k, v in dataset_storage.items():\n print(f\"\\n\\n... SCAN TYPE: {k.upper()}\")\n print(\"\\t-->\", v[\"train_ds\"])\n print(\"\\t-->\", v[\"val_ds\"])\n print(\"\\t-->\", v[\"test_ds\"])\nprint(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:11:07.967589Z","iopub.execute_input":"2021-07-28T16:11:07.968188Z","iopub.status.idle":"2021-07-28T16:11:10.313089Z","shell.execute_reply.started":"2021-07-28T16:11:07.968155Z","shell.execute_reply":"2021-07-28T16:11:10.311760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def _bytes_feature(value, is_list=False):\n \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n if isinstance(value, type(tf.constant(0))):\n value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n \n if not is_list:\n value = [value]\n \n return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))\n\ndef _float_feature(value, is_list=False):\n \"\"\"Returns a float_list from a float / double.\"\"\"\n \n if not is_list:\n value = [value]\n \n return tf.train.Feature(float_list=tf.train.FloatList(value=value))\n\ndef _int64_feature(value, is_list=False):\n \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n \n if not is_list:\n value = [value]\n \n return tf.train.Feature(int64_list=tf.train.Int64List(value=value))\n\ndef serialize_raw(scan, other_val, rescale_up=True, is_test=False):\n \"\"\"\n Creates a tf.Example message ready to be written to a file from 4 features.\n\n Args:\n scan (TBD): TBD\n other_val (int): TBD\n rescale_up (bool): tbd\n\n Returns:\n A tf.Example Message ready to be written to file\n \"\"\"\n \n # Create a dictionary mapping the feature name to the \n # tf.Example-compatible data type.\n if rescale_up:\n feature = {'scan': _bytes_feature(tf.io.serialize_tensor(tf.cast(scan*255, tf.uint8)), is_list=False)}\n else:\n feature = {'scan': _bytes_feature(tf.io.serialize_tensor(scan), is_list=False)}\n \n if not is_test:\n feature[\"label\"] = _int64_feature(other_val, is_list=False)\n else:\n feature[\"brats_id\"] = _int64_feature(other_val, is_list=False)\n \n # Create a Features message using tf.train.Example.\n example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n return example_proto.SerializeToString()\n\n# x = tf.io.serialize_tensor(load_npz(train_df.path_to_flair_npz[0]))\n# (tf.io.parse_tensor(x, out_type=tf.uint8)!=load_npz(train_df.path_to_flair_npz[0])).numpy().sum()","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:29:15.162477Z","iopub.execute_input":"2021-07-28T16:29:15.162848Z","iopub.status.idle":"2021-07-28T16:29:15.174214Z","shell.execute_reply.started":"2021-07-28T16:29:15.162820Z","shell.execute_reply":"2021-07-28T16:29:15.173211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def write_tfrecords(ds, n_ex, n_ex_per_rec=135, serialize_fn=serialize_raw, out_dir=\"/kaggle/working/flair\", ds_type=\"train\"):\n \"\"\"\"\"\"\n n_recs = int(np.ceil(n_ex/n_ex_per_rec))\n \n # Make dataset iterable\n ds = iter(ds)\n \n out_dir = os.path.join(out_dir, ds_type)\n # Create folder\n if not os.path.isdir(out_dir):\n os.makedirs(out_dir, exist_ok=True)\n \n # Create tfrecords\n for i in tqdm(range(n_recs), total=n_recs):\n print(f\"\\n... Writing {ds_type.title()} TFRecord {i+1} of {n_recs} ...\\n\")\n tfrec_path = os.path.join(out_dir, f\"{ds_type}__{(i+1):02}_{n_recs:02}.tfrec\")\n \n # This makes the tfrecord\n with tf.io.TFRecordWriter(tfrec_path) as writer:\n for ex in tqdm(range(n_ex_per_rec), total=n_ex_per_rec):\n try:\n example = serialize_fn(*next(ds), is_test=ds_type==\"test\")\n writer.write(example)\n except:\n break\n\nfor scan_type in [\"flair\", \"t1w\", \"t1wce\", \"t2w\"]:\n write_tfrecords(dataset_storage[scan_type][\"train_ds\"], dataset_storage[scan_type][\"N_TRAIN\"], out_dir=f\"/kaggle/working/{scan_type}\", ds_type=\"train\")\n write_tfrecords(dataset_storage[scan_type][\"val_ds\"], dataset_storage[scan_type][\"N_VAL\"], out_dir=f\"/kaggle/working/{scan_type}\", ds_type=\"val\")\n write_tfrecords(dataset_storage[scan_type][\"test_ds\"], len(dataset_storage[scan_type][\"test_ds_df\"]), out_dir=f\"/kaggle/working/{scan_type}\", ds_type=\"test\")","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:34:05.155928Z","iopub.execute_input":"2021-07-28T16:34:05.156305Z","iopub.status.idle":"2021-07-28T16:34:22.710058Z","shell.execute_reply.started":"2021-07-28T16:34:05.156276Z","shell.execute_reply":"2021-07-28T16:34:22.709074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... CREATE TFRECORD RAW DATASETS STARTING ...\\n\")\n\nFLAIR_TRAIN_TFREC_PATHS = glob(\"/kaggle/working/flair/train/**/*.tfrec\", recursive=True)\nT1W_TRAIN_TFREC_PATHS = glob(\"/kaggle/working/t1w/train/**/*.tfrec\", recursive=True)\nT1WCE_TRAIN_TFREC_PATHS = glob(\"/kaggle/working/t1wce/train/**/*.tfrec\", recursive=True)\nT2W_TRAIN_TFREC_PATHS = glob(\"/kaggle/working/t2w/train/**/*.tfrec\", recursive=True)\n\nFLAIR_VAL_TFREC_PATHS = glob(\"/kaggle/working/flair/val/**/*.tfrec\", recursive=True)\nT1W_VAL_TFREC_PATHS = glob(\"/kaggle/working/t1w/val/**/*.tfrec\", recursive=True)\nT1WCE_VAL_TFREC_PATHS = glob(\"/kaggle/working/t1wce/val/**/*.tfrec\", recursive=True)\nT2W_VAL_TFREC_PATHS = glob(\"/kaggle/working/t2w/val/**/*.tfrec\", recursive=True)\n\nFLAIR_TEST_TFREC_PATHS = glob(\"/kaggle/working/flair/test/**/*.tfrec\", recursive=True)\nT1W_TEST_TFREC_PATHS = glob(\"/kaggle/working/t1w/test/**/*.tfrec\", recursive=True)\nT1WCE_TEST_TFREC_PATHS = glob(\"/kaggle/working/t1wce/test/**/*.tfrec\", recursive=True)\nT2W_TEST_TFREC_PATHS = glob(\"/kaggle/working/t2w/test/**/*.tfrec\", recursive=True)\n\n# Create tf.data.Dataset from filepaths for conversion later\nraw_flair_train_ds = tf.data.TFRecordDataset(FLAIR_TRAIN_TFREC_PATHS, num_parallel_reads=None)\nraw_t1w_val_ds = tf.data.TFRecordDataset(T1W_TRAIN_TFREC_PATHS, num_parallel_reads=None)\nraw_t1wce_test_ds = tf.data.TFRecordDataset(T1WCE_TRAIN_TFREC_PATHS, num_parallel_reads=None)\nraw_t2w_train_ds = tf.data.TFRecordDataset(T2W_TRAIN_TFREC_PATHS, num_parallel_reads=None)\n\nraw_flair_val_ds = tf.data.TFRecordDataset(FLAIR_VAL_TFREC_PATHS, num_parallel_reads=None)\nraw_t1w_test_ds = tf.data.TFRecordDataset(T1W_VAL_TFREC_PATHS, num_parallel_reads=None)\nraw_t1wce_train_ds = tf.data.TFRecordDataset(T1WCE_VAL_TFREC_PATHS, num_parallel_reads=None)\nraw_t2w_val_ds = tf.data.TFRecordDataset(T2W_VAL_TFREC_PATHS, num_parallel_reads=None)\n\nraw_flair_test_ds = tf.data.TFRecordDataset(FLAIR_TEST_TFREC_PATHS, num_parallel_reads=None)\nraw_t1w_train_ds = tf.data.TFRecordDataset(T1W_TEST_TFREC_PATHS, num_parallel_reads=None)\nraw_t1wce_val_ds = tf.data.TFRecordDataset(T1WCE_TEST_TFREC_PATHS, num_parallel_reads=None)\nraw_t2w_test_ds = tf.data.TFRecordDataset(T2W_TEST_TFREC_PATHS, num_parallel_reads=None)\n\nprint(f\"\\n... THE RAW TF.DATA.TFRECORDDATASET OBJECT:\\n\\t--> {raw_flair_train_ds}\\n\")\n\nprint(\"\\n... CREATE TFRECORD RAW DATASETS COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:38:58.714923Z","iopub.execute_input":"2021-07-28T16:38:58.715391Z","iopub.status.idle":"2021-07-28T16:38:58.808108Z","shell.execute_reply.started":"2021-07-28T16:38:58.715357Z","shell.execute_reply":"2021-07-28T16:38:58.806836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def decode(serialized_example, is_test=False):\n \"\"\" Parses a set of features and label from the given `serialized_example`.\n \n It is used as a map function for `dataset.map`\n\n Args:\n serialized_example (tf.Example): A serialized example containing the\n following features:\n – 'scan'\n – 'brats_id' or 'label'\n is_test (bool, optional): Whether to looks for brats_id or label\n \n Returns:\n A decoded tf.data.Dataset object representing the tfrecord dataset\n \"\"\"\n feature_dict = {\n 'scan': tf.io.FixedLenFeature(shape=[], dtype=tf.string, default_value=''),\n }\n \n if not is_test:\n feature_dict[\"label\"] = tf.io.FixedLenFeature(shape=[], dtype=tf.int64, default_value=0)\n else:\n feature_dict[\"brats_id\"] = tf.io.FixedLenFeature(shape=[], dtype=tf.int64, default_value=0)\n \n # Define a parser\n features = tf.io.parse_single_example(serialized_example, features=feature_dict)\n \n # Decode the tf.string\n scan = tf.reshape(tf.io.parse_tensor(features['scan'], out_type=tf.uint8), INPUT_SHAPE)\n \n # Figure out the correct information to return\n if is_test:\n brats_id = tf.cast(features[\"brats_id\"], tf.int32)\n return scan, brats_id\n else:\n label = tf.cast(features[\"label\"], tf.uint8)\n return scan, label","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:45:05.216516Z","iopub.execute_input":"2021-07-28T16:45:05.217023Z","iopub.status.idle":"2021-07-28T16:45:05.225446Z","shell.execute_reply.started":"2021-07-28T16:45:05.216989Z","shell.execute_reply":"2021-07-28T16:45:05.224380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... DECODING RAW FLAIR TFRECORD DATASETS STARTING ...\\n\")\n\n# Decode the tfrecords completely –– decode is our `_parse_function` (from recipe above)\nflair_train_ds = raw_flair_train_ds.map(lambda x: decode(x, is_test=False))\nflair_val_ds = raw_flair_val_ds.map(lambda x: decode(x, is_test=False))\nflair_test_ds = raw_flair_test_ds.map(lambda x: decode(x, is_test=True))\n\nprint(f\"\\n... THE DECODED TF.DATA.TFRECORDDATASET OBJECT:\" \\\n f\"\\n\\t--> ((image), (brats_id or label))\" \\\n f\"\\n\\t--> {flair_train_ds}\\n\")\n\nprint(\"\\n... 2 EXAMPLES OF IMAGES AND LABELS AFTER DECODING ...\")\nfor i, (scan, label) in enumerate(flair_train_ds.take(2)):\n print(f\"\\nSCAN SHAPE : {scan.shape}\")\n print(f\"SCAN LABEL : {label.numpy()}\\n\")\n display(create_animation(scan))\n\nprint(\"\\n... DECODING RAW FLAIR TFRECORD DATASETS COMPLETED ...\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:45:14.343333Z","iopub.execute_input":"2021-07-28T16:45:14.343737Z","iopub.status.idle":"2021-07-28T16:45:16.275179Z","shell.execute_reply.started":"2021-07-28T16:45:14.343700Z","shell.execute_reply":"2021-07-28T16:45:16.274146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... DECODING RAW T1W TFRECORD DATASETS STARTING ...\\n\")\n\n# Decode the tfrecords completely –– decode is our `_parse_function` (from recipe above)\nt1w_train_ds = raw_t1w_train_ds.map(lambda x: decode(x, is_test=False))\nt1w_val_ds = raw_t1w_val_ds.map(lambda x: decode(x, is_test=False))\nt1w_test_ds = raw_t1w_test_ds.map(lambda x: decode(x, is_test=True))\n\nprint(f\"\\n... THE DECODED TF.DATA.TFRECORDDATASET OBJECT:\" \\\n f\"\\n\\t--> ((image), (brats_id or label))\" \\\n f\"\\n\\t--> {t1w_train_ds}\\n\")\n\nprint(\"\\n... 2 EXAMPLES OF IMAGES AND LABELS AFTER DECODING ...\")\nfor i, (scan, label) in enumerate(t1w_train_ds.take(2)):\n print(f\"\\nSCAN SHAPE : {scan.shape}\")\n print(f\"SCAN LABEL : {label.numpy()}\\n\")\n display(create_animation(scan))\n\nprint(\"\\n... DECODING RAW T1W TFRECORD DATASETS COMPLETED ...\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:46:10.472818Z","iopub.execute_input":"2021-07-28T16:46:10.473206Z","iopub.status.idle":"2021-07-28T16:46:12.350996Z","shell.execute_reply.started":"2021-07-28T16:46:10.473174Z","shell.execute_reply":"2021-07-28T16:46:12.350031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... DECODING RAW T1WCE TFRECORD DATASETS STARTING ...\\n\")\n\n# Decode the tfrecords completely –– decode is our `_parse_function` (from recipe above)\nt1wce_train_ds = raw_t1wce_train_ds.map(lambda x: decode(x, is_test=False))\nt1wce_val_ds = raw_t1wce_val_ds.map(lambda x: decode(x, is_test=False))\nt1wce_test_ds = raw_t1wce_test_ds.map(lambda x: decode(x, is_test=True))\n\nprint(f\"\\n... THE DECODED TF.DATA.TFRECORDDATASET OBJECT:\" \\\n f\"\\n\\t--> ((image), (brats_id or label))\" \\\n f\"\\n\\t--> {t1wce_train_ds}\\n\")\n\nprint(\"\\n... 2 EXAMPLES OF IMAGES AND LABELS AFTER DECODING ...\")\nfor i, (scan, label) in enumerate(t1wce_train_ds.take(2)):\n print(f\"\\nSCAN SHAPE : {scan.shape}\")\n print(f\"SCAN LABEL : {label.numpy()}\\n\")\n display(create_animation(scan))\n\nprint(\"\\n... DECODING RAW T1WCE TFRECORD DATASETS COMPLETED ...\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:47:15.016481Z","iopub.execute_input":"2021-07-28T16:47:15.016862Z","iopub.status.idle":"2021-07-28T16:47:16.858844Z","shell.execute_reply.started":"2021-07-28T16:47:15.016821Z","shell.execute_reply":"2021-07-28T16:47:16.857919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... DECODING RAW T2W TFRECORD DATASETS STARTING ...\\n\")\n\n# Decode the tfrecords completely –– decode is our `_parse_function` (from recipe above)\nt2w_train_ds = raw_t2w_train_ds.map(lambda x: decode(x, is_test=False))\nt2w_val_ds = raw_t2w_val_ds.map(lambda x: decode(x, is_test=False))\nt2w_test_ds = raw_t2w_test_ds.map(lambda x: decode(x, is_test=True))\n\nprint(f\"\\n... THE DECODED TF.DATA.TFRECORDDATASET OBJECT:\" \\\n f\"\\n\\t--> ((image), (brats_id or label))\" \\\n f\"\\n\\t--> {t2w_train_ds}\\n\")\n\nprint(\"\\n... 2 EXAMPLES OF IMAGES AND LABELS AFTER DECODING ...\")\nfor i, (scan, label) in enumerate(t2w_train_ds.take(2)):\n print(f\"\\nSCAN SHAPE : {scan.shape}\")\n print(f\"SCAN LABEL : {label.numpy()}\\n\")\n display(create_animation(scan))\n\nprint(\"\\n... DECODING RAW T2W TFRECORD DATASETS COMPLETED ...\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:46:55.215116Z","iopub.execute_input":"2021-07-28T16:46:55.215687Z","iopub.status.idle":"2021-07-28T16:46:56.975243Z","shell.execute_reply.started":"2021-07-28T16:46:55.215623Z","shell.execute_reply":"2021-07-28T16:46:56.974255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}