{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CONVERT DICOM TO AXIAL PNG 2D SLICES,
k-FOLD CV, FILTER BASED ON IMAGE PIXEL STATISTICS\n## ๐Ÿง  RSNA-MICCAI Brain Tumor Radiogenomic Classification ๐Ÿง \n\n\n**Data Preprocessing** for [RSNA-MICCAI Brain Tumor Radiogenomic Classification](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification) challenge with steps on how to **filter** useful images, **create map of chosen images**, **make k-fold Train/Val splits**, **useful insights on Image Pixel Statistics** and **convert DICOM to PNG records** of the MRI scans.\n\n---------------------------------------\n\n**Credit:**\nNotebook based on \n1. this wonderful solution [Connecting voxel spaces](https://www.kaggle.com/boojum/connecting-voxel-spaces), \n2. my EDA notebook [\n๐Ÿง  Brain Radiogenomics Advanced EDA](https://www.kaggle.com/smoschou55/brain-radiogenomics-advanced-eda) \n\nand some inspiration from the following notebooks: \n1. [\n๐Ÿง Brain Tumor๐Ÿง  - EDA with Animations and Modeling](https://www.kaggle.com/ihelon/brain-tumor-eda-with-animations-and-modeling), \n2. [DICOM to PNG dataset (128 GB -> 5.2 GB) ๐ŸŽจ๐Ÿ”ฅ](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/253000) and \n3. [๐Ÿง Fast DICOM--> PNG full DATA+ {Download DATA} โœ…](https://www.kaggle.com/anasshnn/fast-dicom-png-full-data-download-data).\n\n","metadata":{}},{"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/29653/logos/header.png)","metadata":{}},{"cell_type":"markdown","source":"\n\n
\n

Quick Navigation

\n\n* [1. Overview](#1)\n* [2. Helper Functions](#2)\n* [3. Create DataFrame with Image Filepaths](#10)\n* [4. Compute Image Data Stats](#15)\n* [5. Filter Based on Theshold and Save PNGs](#20)\n* [6. Train / Val Split](#30)\n* [7. Convert to Voxel Space of Choice](#40)","metadata":{}},{"cell_type":"markdown","source":"\n

1. Overview

","metadata":{}},{"cell_type":"code","source":"!pip install pandarallel\n\nimport os\nimport shutil\nimport ast\nimport json\nimport glob\nimport random\nimport collections\nimport gc\n\nimport numpy as np\nimport pandas as pd\n\n# Visualization\nimport nibabel as nib\nimport matplotlib.pyplot as plt\nimport SimpleITK as sitk\nimport matplotlib.image as mpimg\nimport seaborn as sns; sns.set();\n\nimport imageio # save to PNG images\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport cv2\n\nfrom tqdm.notebook import tqdm; tqdm.pandas(); # get nice bar\n\nfrom pandarallel import pandarallel; pandarallel.initialize(); \n\n# Seed for reproducability\nseed = 1234\nnp.random.seed(seed)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:19.024514Z","iopub.execute_input":"2021-08-23T11:49:19.024875Z","iopub.status.idle":"2021-08-23T11:49:28.389232Z","shell.execute_reply.started":"2021-08-23T11:49:19.024800Z","shell.execute_reply":"2021-08-23T11:49:28.388443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Files\n**train/** - folder containing the training files, with each top-level folder representing a subject \n**train_labels.csv** - file containing the target MGMT_value for each subject in the training data (e.g. the presence of MGMT promoter methylation) \n**test/** - the test files, which use the same structure as train/; your task is to predict the MGMT_value for each subject in the test data. NOTE: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set \n**sample_submission.csv** - a sample submission file in the correct format","metadata":{}},{"cell_type":"markdown","source":"The exact mpMRI scans included are:\n\n- Fluid Attenuated Inversion Recovery (FLAIR)\n- T1-weighted pre-contrast (T1w)\n- T1-weighted post-contrast (T1Gd)\n- T2-weighted (T2)","metadata":{}},{"cell_type":"markdown","source":"Exact folder structure:\n\n\n```\nTraining/Validation/Testing\nโ”‚\nโ””โ”€โ”€โ”€ 00000\nโ”‚ โ”‚\nโ”‚ โ””โ”€โ”€โ”€ FLAIR\nโ”‚ โ”‚ โ”‚ Image-1.dcm\nโ”‚ โ”‚ โ”‚ Image-2.dcm\nโ”‚ โ”‚ โ”‚ ...\nโ”‚ โ”‚ \nโ”‚ โ””โ”€โ”€โ”€ T1w\nโ”‚ โ”‚ โ”‚ Image-1.dcm\nโ”‚ โ”‚ โ”‚ Image-2.dcm\nโ”‚ โ”‚ โ”‚ ...\nโ”‚ โ”‚ \nโ”‚ โ””โ”€โ”€โ”€ T1wCE\nโ”‚ โ”‚ โ”‚ Image-1.dcm\nโ”‚ โ”‚ โ”‚ Image-2.dcm\nโ”‚ โ”‚ โ”‚ ...\nโ”‚ โ”‚ \nโ”‚ โ””โ”€โ”€โ”€ T2w\nโ”‚ โ”‚ โ”‚ Image-1.dcm\nโ”‚ โ”‚ โ”‚ Image-2.dcm\nโ”‚ โ”‚ โ”‚ .....\nโ”‚ \nโ””โ”€โ”€โ”€ 00001\nโ”‚ โ”‚ ...\nโ”‚ \nโ”‚ ... \nโ”‚ \nโ””โ”€โ”€โ”€ 00002\nโ”‚ โ”‚ ...\n```","metadata":{}},{"cell_type":"markdown","source":"DICOMยฎ โ€” [Digital Imaging and Communications in Medicine](https://www.dicomstandard.org/about-home) โ€” is the international standard for medical images and related information. It defines the formats for medical images that can be exchanged with the data and quality necessary for clinical use. With hundreds of thousands of medical imaging devices in use, DICOMยฎ is one of the most widely deployed healthcare messaging Standards in the world.","metadata":{}},{"cell_type":"markdown","source":"\n

2. Helper Functions

","metadata":{"execution":{"iopub.status.busy":"2021-07-14T06:41:32.077425Z","iopub.execute_input":"2021-07-14T06:41:32.077767Z","iopub.status.idle":"2021-07-14T06:41:32.0845Z","shell.execute_reply.started":"2021-07-14T06:41:32.077737Z","shell.execute_reply":"2021-07-14T06:41:32.082683Z"}}},{"cell_type":"code","source":"# Paths \nKAGGLE_DIR = '/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/'\nIMG_PATH_TRAIN = KAGGLE_DIR + 'train/'\nIMG_PATH_TEST = KAGGLE_DIR + 'test/'\nTRAIN_CSV_PATH = KAGGLE_DIR + 'train_labels.csv'\nTEST_CSV_PATH = KAGGLE_DIR + 'sample_submission.csv'","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.392077Z","iopub.execute_input":"2021-08-23T11:49:28.392327Z","iopub.status.idle":"2021-08-23T11:49:28.398270Z","shell.execute_reply.started":"2021-08-23T11:49:28.392301Z","shell.execute_reply":"2021-08-23T11:49:28.397483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# All filenames for train and test images\ntrain_images = os.listdir(IMG_PATH_TRAIN)\ntest_images = os.listdir(IMG_PATH_TEST)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.400060Z","iopub.execute_input":"2021-08-23T11:49:28.400654Z","iopub.status.idle":"2021-08-23T11:49:28.450149Z","shell.execute_reply.started":"2021-08-23T11:49:28.400619Z","shell.execute_reply":"2021-08-23T11:49:28.449461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### For more details in pixel arrays see : [Working with Pixel Data](https://pydicom.github.io/pydicom/stable/old/working_with_pixel_data.html)","metadata":{}},{"cell_type":"code","source":"def load_dicom(path):\n # read file\n dicom = pydicom.read_file(path)\n # get pixel data into a useful format. \n data = dicom.pixel_array\n # transform data into black and white scale / grayscale\n data = data - np.min(data)\n if np.max(data) != 0:\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n return data\n\ndef save_png_to_disk(data, path, png_master_dir):\n # SAVE PNG TO DISK \n image_name=path.split('/')[4:][-1].split('.')[0] \n png_image_path=png_master_dir+'/'+'/'.join(path.split('/')[4:-1])+'/'+image_name+'.png' \n imageio.imsave(png_image_path,data)\n\ndef convert_dicom_to_png(path, png_master_dir, resize = None):\n dicom = pydicom.read_file(path)\n data = apply_voi_lut(dicom.pixel_array, dicom)\n # If Resize == True, Resize Image to Specified Resolution\n if resize:\n data = cv2.resize(data, resize)\n # Transform Data as Necessary \n if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n data = data - np.min(data)\n if np.max(data) != 0:\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n # SAVE PNG TO DISK \n save_png_to_disk(data, path, png_master_dir)\n return data\n\ndef image_stats(path):\n dicom = pydicom.read_file(path)\n data = apply_voi_lut(dicom.pixel_array, dicom)\n \n # Transform Data as Necessary \n if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n \n data = data - np.min(data)\n if np.max(data) != 0:\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n \n # Compute and Return Image Stats: min, max, mean, std, 25th-, 50th-, and 75th percentile\n return np.min(data), \\\n np.max(data), \\\n np.mean(data), \\\n np.std(data), \\\n np.percentile(data, 25), \\\n np.percentile(data, 50), \\\n np.percentile(data, 75)\n\ndef is_valid_slice(path, threshold=0):\n dicom = pydicom.read_file(path)\n data = apply_voi_lut(dicom.pixel_array, dicom)\n \n # Transform Data as Necessary \n if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = np.amax(data) - data\n \n data = data - np.min(data)\n if np.max(data) != 0:\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n \n # REMOVE FILEPATH FROM IMAGE REGISTRY IF IMAGE HAS LOW INFO VALUE\n if np.mean(data) <= threshold:\n return False\n else:\n return True","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.451609Z","iopub.execute_input":"2021-08-23T11:49:28.451943Z","iopub.status.idle":"2021-08-23T11:49:28.553889Z","shell.execute_reply.started":"2021-08-23T11:49:28.451909Z","shell.execute_reply":"2021-08-23T11:49:28.552895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

3. Create DataFrame with Image Filepaths

","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— **Uncomment Below to Repeat the entire IO process from sratch** โ—โ—โ—","metadata":{}},{"cell_type":"markdown","source":"**\\train**","metadata":{}},{"cell_type":"code","source":"# f = []\n# for (dirpath, dirnames, filenames) in os.walk(IMG_PATH_TRAIN):\n# f.extend(os.path.join(dirpath, x) for x in filenames)\n \n# train_file_paths_df = pd.DataFrame({'file_paths': f})\n# train_file_paths_df['directory'] = IMG_PATH_TRAIN\n# train_file_paths_df['dataset'] = train_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[4]\n# train_file_paths_df['patient_id'] = train_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[5]\n# train_file_paths_df['scan_type'] = train_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[6]\n# train_file_paths_df['file'] = train_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[7]\n# display(train_file_paths_df.head(2))\n# train_file_paths_df.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.555206Z","iopub.execute_input":"2021-08-23T11:49:28.555564Z","iopub.status.idle":"2021-08-23T11:49:28.566347Z","shell.execute_reply.started":"2021-08-23T11:49:28.555523Z","shell.execute_reply":"2021-08-23T11:49:28.565512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**\\test**","metadata":{}},{"cell_type":"code","source":"# f = []\n# for (dirpath, dirnames, filenames) in os.walk(IMG_PATH_TEST):\n# f.extend(os.path.join(dirpath, x) for x in filenames)\n \n# test_file_paths_df = pd.DataFrame({'file_paths': f})\n# test_file_paths_df['directory'] = IMG_PATH_TEST\n# test_file_paths_df['dataset'] = test_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[4]\n# test_file_paths_df['patient_id'] = test_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[5]\n# test_file_paths_df['scan_type'] = test_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[6]\n# test_file_paths_df['file'] = test_file_paths_df['file_paths'].str.split(\"/\", n = 7, expand = True)[7]\n# display(test_file_paths_df.head(2))\n# test_file_paths_df.shape[0]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.569624Z","iopub.execute_input":"2021-08-23T11:49:28.569958Z","iopub.status.idle":"2021-08-23T11:49:28.577467Z","shell.execute_reply.started":"2021-08-23T11:49:28.569931Z","shell.execute_reply":"2021-08-23T11:49:28.576694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Exclude 3 problematic cases](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262046)\n\n**BraTSIDs**\n* 00109\n* 00123\n* 00709","metadata":{}},{"cell_type":"code","source":"# train_df = train_file_paths_df.copy()\n# test_df = test_file_paths_df.copy()\n\n# train_df = train_df[(train_df.patient_id != \"00109\") & \n# (train_df.patient_id != \"00123\") &\n# (train_df.patient_id != \"00709\")]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.578772Z","iopub.execute_input":"2021-08-23T11:49:28.579162Z","iopub.status.idle":"2021-08-23T11:49:28.586500Z","shell.execute_reply.started":"2021-08-23T11:49:28.579126Z","shell.execute_reply":"2021-08-23T11:49:28.585757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Save to *.csv file","metadata":{}},{"cell_type":"code","source":"# train_df.to_csv('train_filepaths_rsna.csv', index=False)\n# test_df.to_csv('test_filepaths_rsna.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.589049Z","iopub.execute_input":"2021-08-23T11:49:28.589447Z","iopub.status.idle":"2021-08-23T11:49:28.594748Z","shell.execute_reply.started":"2021-08-23T11:49:28.589405Z","shell.execute_reply":"2021-08-23T11:49:28.593901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

4. Compute Image Data Stats

","metadata":{}},{"cell_type":"markdown","source":"# Create Master DataFrame (and save into *.csv - file) with Pixel Stats per Image.\nThis only needs to take place once and then can be used a number of times to remove images using any type of stats criterion from the path files dataframe before creating a new PNG dataset.","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— **Uncomment Below to Repeat the entire IO process from sratch** โ—โ—โ—","metadata":{}},{"cell_type":"markdown","source":"**/train**","metadata":{}},{"cell_type":"code","source":"# stdf = train_df.copy()\n# stdf[\"stats\"] = stdf[\"file_paths\"].parallel_apply(lambda x: image_stats(x))\n# stdf[[\"min_px\", \"max_px\", \"mean_px\", \"std_px\", \"q1\", \"q2\", \"q3\"]] = pd.DataFrame(stdf[\"stats\"].tolist(), index=stdf.index)\n# stdf = stdf.drop(['stats'], axis=1)\n# stdf","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.597818Z","iopub.execute_input":"2021-08-23T11:49:28.598163Z","iopub.status.idle":"2021-08-23T11:49:28.603149Z","shell.execute_reply.started":"2021-08-23T11:49:28.598138Z","shell.execute_reply":"2021-08-23T11:49:28.602303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# stdf.to_csv('stats_train_file_paths_df.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.605022Z","iopub.execute_input":"2021-08-23T11:49:28.605362Z","iopub.status.idle":"2021-08-23T11:49:28.613806Z","shell.execute_reply.started":"2021-08-23T11:49:28.605329Z","shell.execute_reply":"2021-08-23T11:49:28.612947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**/test**","metadata":{}},{"cell_type":"code","source":"# ssdf = test_df.copy()\n# ssdf[\"stats\"] = ssdf[\"file_paths\"].parallel_apply(lambda x: image_stats(x))\n# ssdf[[\"min_px\", \"max_px\", \"mean_px\", \"std_px\", \"q1\", \"q2\", \"q3\"]] = pd.DataFrame(ssdf[\"stats\"].tolist(), index=ssdf.index)\n# ssdf = ssdf.drop(['stats'], axis=1)\n# ssdf","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.615002Z","iopub.execute_input":"2021-08-23T11:49:28.615530Z","iopub.status.idle":"2021-08-23T11:49:28.621715Z","shell.execute_reply.started":"2021-08-23T11:49:28.615497Z","shell.execute_reply":"2021-08-23T11:49:28.620928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ssdf.to_csv('stats_test_file_paths_df.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.624038Z","iopub.execute_input":"2021-08-23T11:49:28.624346Z","iopub.status.idle":"2021-08-23T11:49:28.630136Z","shell.execute_reply.started":"2021-08-23T11:49:28.624322Z","shell.execute_reply":"2021-08-23T11:49:28.629338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read pre-saved *csv file","metadata":{}},{"cell_type":"markdown","source":"If you don't want to uncomment code-blocks above and rerun the entire process.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/stats_train_file_paths_df.csv')\ntest_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/stats_test_file_paths_df.csv')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:28.632196Z","iopub.execute_input":"2021-08-23T11:49:28.632507Z","iopub.status.idle":"2021-08-23T11:49:30.914839Z","shell.execute_reply.started":"2021-08-23T11:49:28.632473Z","shell.execute_reply":"2021-08-23T11:49:30.913998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize Overall Pixel Statistics to get Insights on Images","metadata":{}},{"cell_type":"markdown","source":"Number of unique values","metadata":{}},{"cell_type":"markdown","source":"**\\train**","metadata":{}},{"cell_type":"code","source":"stats_cols = []\nnum_unique = []\n\nfor col in train_df:\n# print(\"* For attribute '{}' , there are [ {} ] unique values.\".format(col,\n# len(train_meta_df[col].unique())))\n stats_cols.append(col)\n num_unique.append(len(train_df[col].unique()))\n \ntrain_df_stats = pd.DataFrame(\n {'col_name': stats_cols,\n 'value_count': num_unique,\n 'nan_count': train_df.isna().sum()\n })\n\ntrain_df_stats = train_df_stats.sort_values(by=['value_count'], ascending=False).reset_index(drop=True)\ntrain_df_stats","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:30.916017Z","iopub.execute_input":"2021-08-23T11:49:30.916352Z","iopub.status.idle":"2021-08-23T11:49:31.402466Z","shell.execute_reply.started":"2021-08-23T11:49:30.916319Z","shell.execute_reply":"2021-08-23T11:49:31.401525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**\\test**","metadata":{}},{"cell_type":"code","source":"stats_cols = []\nnum_unique = []\n\nfor col in test_df:\n# print(\"* For attribute '{}' , there are [ {} ] unique values.\".format(col,\n# len(train_meta_df[col].unique())))\n stats_cols.append(col)\n num_unique.append(len(test_df[col].unique()))\n \ntest_df_stats = pd.DataFrame(\n {'col_name': stats_cols,\n 'value_count': num_unique,\n 'nan_count': test_df.isna().sum()\n })\n\ntest_df_stats = test_df_stats.sort_values(by=['value_count'], ascending=False).reset_index(drop=True)\ntest_df_stats","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:31.403748Z","iopub.execute_input":"2021-08-23T11:49:31.404090Z","iopub.status.idle":"2021-08-23T11:49:31.488496Z","shell.execute_reply.started":"2021-08-23T11:49:31.404055Z","shell.execute_reply":"2021-08-23T11:49:31.487578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some Useful Histograms","metadata":{}},{"cell_type":"markdown","source":"**\\train**","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 3, figsize=(25, 8), sharey=True)\nfig.suptitle('Train Dataset Pixel Distributions')\n\nsns.histplot(ax=axes[0], data = train_df[['mean_px', 'std_px']], bins=50, alpha=0.5,)\naxes[0].set_title(\"mean, std\")\nsns.histplot(ax=axes[1], data = train_df[['min_px', 'max_px']], bins=50, alpha=0.5,)\naxes[1].set_title(\"min, max\")\nsns.histplot(ax=axes[2], data= train_df[['q1', 'q2', 'q3']], bins=50, alpha=0.5,)\naxes[2].set_title(\"q1, q2, q3\")\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:31.489899Z","iopub.execute_input":"2021-08-23T11:49:31.490247Z","iopub.status.idle":"2021-08-23T11:49:36.864989Z","shell.execute_reply.started":"2021-08-23T11:49:31.490212Z","shell.execute_reply":"2021-08-23T11:49:36.864158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are many empty images.","metadata":{}},{"cell_type":"markdown","source":"**\\test**","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 3, figsize=(25, 8), sharey=True)\nfig.suptitle('Test Dataset Pixel Distributions')\n\nsns.histplot(ax=axes[0], data = test_df[['mean_px', 'std_px']], bins=50, alpha=0.5,)\naxes[0].set_title(\"mean, std\")\nsns.histplot(ax=axes[1], data = test_df[['min_px', 'max_px']], bins=50, alpha=0.5,)\naxes[1].set_title(\"min, max\")\nsns.histplot(ax=axes[2], data= test_df[['q1', 'q2', 'q3']], bins=50, alpha=0.5,)\naxes[2].set_title(\"q1, q2, q3\")\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:36.866244Z","iopub.execute_input":"2021-08-23T11:49:36.866746Z","iopub.status.idle":"2021-08-23T11:49:38.683595Z","shell.execute_reply.started":"2021-08-23T11:49:36.866706Z","shell.execute_reply":"2021-08-23T11:49:38.682719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Similar Distributions as for Train dataset.","metadata":{}},{"cell_type":"markdown","source":"\n

5. Filter Based on Thershold and Save PNGs

","metadata":{}},{"cell_type":"markdown","source":"# Remove Paths to Images based on some Critetia\ne.g. remove images from path files df that have np.mean(data) < 10","metadata":{}},{"cell_type":"code","source":"threshold = 50\ntruncated_train_df = train_df[train_df.mean_px >= threshold].reset_index(drop=True)\ntruncated_test_df = test_df[test_df.mean_px >= threshold].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:38.684835Z","iopub.execute_input":"2021-08-23T11:49:38.685330Z","iopub.status.idle":"2021-08-23T11:49:38.697635Z","shell.execute_reply.started":"2021-08-23T11:49:38.685290Z","shell.execute_reply":"2021-08-23T11:49:38.696688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Directory Tree to Host PNGs","metadata":{}},{"cell_type":"code","source":"def create_png_dir_tree(PNG_MSTR_DIR, train_df, test_df):\n #delete old folder (if you run the code twice)\n shutil.rmtree(PNG_MSTR_DIR, ignore_errors=True)\n \n # create the main PNG folder with the test and train \n png_test_path=PNG_MSTR_DIR + '/test/'\n png_train_path=PNG_MSTR_DIR + '/train/'\n \n os.makedirs(PNG_MSTR_DIR)\n os.makedirs(png_train_path)\n os.makedirs(png_test_path)\n print('\\t\\t\\t DONE DIR TREE')\n\n # floders creation \n for trfold in set(train_df.patient_id):\n os.mkdir(png_train_path+str(trfold).zfill(5))\n for mp in set(train_df.scan_type):\n os.mkdir(png_train_path+str(trfold).zfill(5)+'/'+str(mp))\n print('\\t\\t\\t DONE CREATING TRAIN DIR')\n\n for trfold in set(test_df.patient_id): \n os.mkdir(png_test_path+str(trfold).zfill(5))\n for mp in set(test_df.scan_type):\n os.mkdir(png_test_path+str(trfold).zfill(5)+'/'+str(mp))\n print('\\t\\t\\t DONE CREATING TEST DIR')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:38.698850Z","iopub.execute_input":"2021-08-23T11:49:38.699204Z","iopub.status.idle":"2021-08-23T11:49:38.706711Z","shell.execute_reply.started":"2021-08-23T11:49:38.699148Z","shell.execute_reply":"2021-08-23T11:49:38.705658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SAVE PNG IMAGES INTO WORKING DIR","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— IF YOU WANT TO SAVE PNG FILES ABOVE THRESHOLD UNCOMMENT CELL BELOW โ—โ—โ—","metadata":{}},{"cell_type":"code","source":"# # Create directory\n# PNG_ROOT_DIR = 'png_dataset_threshold_' + str(threshold)\n# create_png_dir_tree(PNG_ROOT_DIR, truncated_train_df, truncated_test_df)\n\n# print('\\t\\t\\t Start Saving TRAIN Images')\n# # Convert DICOM to PNG and Save to Disk\n# truncated_train_df[\"file_paths\"].parallel_apply(lambda x: convert_dicom_to_png(x, PNG_ROOT_DIR, (256, 256)));\n# print('\\t\\t\\t Finished Saving TRAIN Images')\n\n# print('\\t\\t\\t Start Saving TEST Images')\n# truncated_test_df[\"file_paths\"].parallel_apply(lambda x: convert_dicom_to_png(x, PNG_ROOT_DIR, (256, 256)));\n# print('\\t\\t\\t Finished Saving TEST Images')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:38.708102Z","iopub.execute_input":"2021-08-23T11:49:38.708706Z","iopub.status.idle":"2021-08-23T11:49:38.718626Z","shell.execute_reply.started":"2021-08-23T11:49:38.708670Z","shell.execute_reply":"2021-08-23T11:49:38.717977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize PNG images","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— IF YOU WANT TO VISUALIZE n - RANDOM SAVED PNG FILES UNCOMMENT CELL BELOW โ—โ—โ—","metadata":{}},{"cell_type":"code","source":"# n = 5\n# j = 0\n# plt.figure(figsize=(18, 10))\n# for i in random.sample(range(truncated_train_df.shape[0]), n):\n# j +=1\n# img = mpimg.imread(PNG_ROOT_DIR + '/train/' + \n# str(truncated_train_df.patient_id.iloc[i]).zfill(5) +\"/\"+\n# str(truncated_train_df.scan_type.iloc[i]) +\"/\"+\n# str(truncated_train_df.file.iloc[i].split(\".\")[0])+'.png') \n# print(img.shape)\n# plt.subplot(1, n, j)\n# imgplot = plt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:38.721374Z","iopub.execute_input":"2021-08-23T11:49:38.721654Z","iopub.status.idle":"2021-08-23T11:49:38.727485Z","shell.execute_reply.started":"2021-08-23T11:49:38.721627Z","shell.execute_reply":"2021-08-23T11:49:38.726567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n* Images have different sizes, so we used CV2 to resize to the desired resolution, in this case 256 x 256.\n\n* We might also want to convert all images to the same plane of reference, most probably into axial plane (most images are already into axial plane).","metadata":{}},{"cell_type":"markdown","source":"\n

6. Train / Val Splits

","metadata":{}},{"cell_type":"markdown","source":"# Train / Val k -fold Cross Validation Splits","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/train_filepaths_rsna.csv')\ntest_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/test_filepaths_rsna.csv')\n\ntrain_lbl_df = pd.read_csv('/kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:38.728712Z","iopub.execute_input":"2021-08-23T11:49:38.729117Z","iopub.status.idle":"2021-08-23T11:49:40.545267Z","shell.execute_reply.started":"2021-08-23T11:49:38.729082Z","shell.execute_reply":"2021-08-23T11:49:40.544477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Exclude 3 problematic cases](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/262046)\n\n**BraTSIDs**\n* 109\n* 123\n* 709","metadata":{}},{"cell_type":"code","source":"train_lbl_df = train_lbl_df[(train_lbl_df.BraTS21ID != 109) & \n (train_lbl_df.BraTS21ID != 123) &\n (train_lbl_df.BraTS21ID != 709)].reset_index(drop = True)\ntrain_lbl_df","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:40.546484Z","iopub.execute_input":"2021-08-23T11:49:40.546819Z","iopub.status.idle":"2021-08-23T11:49:40.562240Z","shell.execute_reply.started":"2021-08-23T11:49:40.546787Z","shell.execute_reply":"2021-08-23T11:49:40.561359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5 - fold Stratified Cross Validation on Target Bindary Values MGMT_value","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\n\nskf = StratifiedKFold(n_splits=5)\n\nprint('Class Ratio:',sum(train_lbl_df['MGMT_value'])/len(train_lbl_df['MGMT_value']))\n\ntarget = train_lbl_df.loc[:,'MGMT_value']\n\nfold_no = 1\ntrain_fold_dict = {}\nval_fold_dict = {}\nfor train_index, test_index in skf.split(train_lbl_df, target):\n train = train_lbl_df.loc[train_index,:]\n val = train_lbl_df.loc[test_index,:]\n train_fold_dict['train_fold_'+str(fold_no)] = train.set_index('BraTS21ID')['MGMT_value'].to_dict()\n val_fold_dict['val_fold_'+str(fold_no)] = val.set_index('BraTS21ID')['MGMT_value'].to_dict()\n print('Fold',str(fold_no),'Class Ratio:',sum(val['MGMT_value'])/len(val['MGMT_value']),\n ',\\t len train, val, sum:',len(train), len(val), len(train)+len(val))\n fold_no += 1","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:40.563524Z","iopub.execute_input":"2021-08-23T11:49:40.563868Z","iopub.status.idle":"2021-08-23T11:49:40.828686Z","shell.execute_reply.started":"2021-08-23T11:49:40.563834Z","shell.execute_reply":"2021-08-23T11:49:40.827112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**train/ folds**","metadata":{}},{"cell_type":"code","source":"train_df_1 = train_df.copy()\ntrain_fold_df_1 = train_fold_dict['train_fold_1']\ntrain_df_1['MGMT_value'] = train_df_1['patient_id'].map(train_fold_df_1)\ntrain_df_1 = train_df_1.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\ntrain_df_2 = train_df.copy()\ntrain_fold_df_2 = train_fold_dict['train_fold_2']\ntrain_df_2['MGMT_value'] = train_df_2['patient_id'].map(train_fold_df_2)\ntrain_df_2 = train_df_2.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\ntrain_df_3 = train_df.copy()\ntrain_fold_df_3 = train_fold_dict['train_fold_3']\ntrain_df_3['MGMT_value'] = train_df_3['patient_id'].map(train_fold_df_3)\ntrain_df_3 = train_df_3.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\ntrain_df_4 = train_df.copy()\ntrain_fold_df_4 = train_fold_dict['train_fold_4']\ntrain_df_4['MGMT_value'] = train_df_4['patient_id'].map(train_fold_df_4)\ntrain_df_4 = train_df_4.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\ntrain_df_5 = train_df.copy()\ntrain_fold_df_5 = train_fold_dict['train_fold_5']\ntrain_df_5['MGMT_value'] = train_df_5['patient_id'].map(train_fold_df_5)\ntrain_df_5 = train_df_5.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:40.830037Z","iopub.execute_input":"2021-08-23T11:49:40.830398Z","iopub.status.idle":"2021-08-23T11:49:42.119444Z","shell.execute_reply.started":"2021-08-23T11:49:40.830360Z","shell.execute_reply":"2021-08-23T11:49:42.118559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**val/ folds**","metadata":{}},{"cell_type":"code","source":"val_df_1 = train_df.copy()\nval_fold_df_1 = val_fold_dict['val_fold_1']\nval_df_1['MGMT_value'] = val_df_1['patient_id'].map(val_fold_df_1)\nval_df_1 = val_df_1.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\nval_df_2 = train_df.copy()\nval_fold_df_2 = val_fold_dict['val_fold_2']\nval_df_2['MGMT_value'] = val_df_2['patient_id'].map(val_fold_df_2)\nval_df_2 = val_df_2.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\nval_df_3 = train_df.copy()\nval_fold_df_3 = val_fold_dict['val_fold_3']\nval_df_3['MGMT_value'] = val_df_3['patient_id'].map(val_fold_df_3)\nval_df_3 = val_df_3.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\nval_df_4 = train_df.copy()\nval_fold_df_4 = val_fold_dict['val_fold_4']\nval_df_4['MGMT_value'] = val_df_4['patient_id'].map(val_fold_df_4)\nval_df_4 = val_df_4.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\nval_df_5 = train_df.copy()\nval_fold_df_5 = val_fold_dict['val_fold_5']\nval_df_5['MGMT_value'] = val_df_5['patient_id'].map(val_fold_df_5)\nval_df_5 = val_df_5.dropna(axis=0).groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:42.120720Z","iopub.execute_input":"2021-08-23T11:49:42.121213Z","iopub.status.idle":"2021-08-23T11:49:43.090966Z","shell.execute_reply.started":"2021-08-23T11:49:42.121177Z","shell.execute_reply":"2021-08-23T11:49:43.090141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Check sizes make sense**","metadata":{}},{"cell_type":"code","source":"print('Train, ', 'Test, ', 'Sum')\nprint(len(set(list(train_df_1.patient_id))), len(set(list(val_df_1.patient_id))), len(set(list(train_df_1.patient_id)))+len(set(list(val_df_1.patient_id))))\nprint(len(set(list(train_df_2.patient_id))), len(set(list(val_df_2.patient_id))), len(set(list(train_df_2.patient_id)))+len(set(list(val_df_2.patient_id))))\nprint(len(set(list(train_df_3.patient_id))), len(set(list(val_df_3.patient_id))), len(set(list(train_df_3.patient_id)))+len(set(list(val_df_3.patient_id))))\nprint(len(set(list(train_df_4.patient_id))), len(set(list(val_df_4.patient_id))), len(set(list(train_df_4.patient_id)))+len(set(list(val_df_4.patient_id))))\nprint(len(set(list(train_df_5.patient_id))), len(set(list(val_df_5.patient_id))), len(set(list(train_df_5.patient_id)))+len(set(list(val_df_5.patient_id))))","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:43.095674Z","iopub.execute_input":"2021-08-23T11:49:43.095941Z","iopub.status.idle":"2021-08-23T11:49:43.114007Z","shell.execute_reply.started":"2021-08-23T11:49:43.095915Z","shell.execute_reply":"2021-08-23T11:49:43.112873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

7. Convert to Voxel Space of Choice

","metadata":{}},{"cell_type":"markdown","source":"Based on this fantastic notebook: [\nConnecting voxel spaces](https://www.kaggle.com/boojum/connecting-voxel-spaces)","metadata":{}},{"cell_type":"markdown","source":"# Steps\n* As we saw in the [\n๐Ÿง  Advanced EDA - Brain Tumor Data ๐Ÿง ](https://www.kaggle.com/smoschou55/advanced-eda-brain-tumor-data) each patient has all images in a single modality in the same plane of reference, e.g. all FLAIR coronal, all T1w and T1wCE axial, and all T2w sagittal.\n\n* Thus, we just need to \n 1. Determine the orientation of each modality per patient (we need the summary table from EDA)\n 2. Choose a reference example modality in terms of orientation and resolution\n 3. Use Simple ITK Python package to convert one 3D volume (collection of all images in a modality per patient) into the orientation and resolution of the reference example 3D volume (e.g. Axial T1w Volume of (256, 256, 32)).\n 4. Finally, we can save 2D SLICES into DICOM or PNG formats. \n 5. OPTIONALLY: Determine which **IMAGES** need to be removed (np.mean(data) < threshold) and only store those images that have important information in them. E.g. save each image in the X-range of the 3D volume (image sequences) into PNG **IF and ONLY IF** np.mean(data) >= threshold.","metadata":{}},{"cell_type":"markdown","source":"## 1. Determine the orientation of each modality per patient","metadata":{}},{"cell_type":"code","source":"# Read Metadata train and test dfs\ntrain_meta_df = pd.read_csv('/kaggle/input/stage0-metadata-rsna/stage_0_train_with_metadata.csv')\ntest_meta_df = pd.read_csv('/kaggle/input/stage0-metadata-rsna/stage_0_test_with_metadata.csv')\n\ndef get_image_plane(data):\n '''\n Returns the MRI's plane from the dicom data.\n \n '''\n x1,y1,_,x2,y2,_ = [round(j) for j in ast.literal_eval(data.ImageOrientationPatient)]\n cords = [x1,y1,x2,y2]\n\n if cords == [1,0,0,0]:\n return 'coronal'\n if cords == [1,0,0,1]:\n return 'axial'\n if cords == [0,1,0,0]:\n return 'sagittal'\n\ntrain_meta_df['Orientation'] = train_meta_df.apply(get_image_plane, axis=1)\n\ntest_meta_df['Orientation'] = test_meta_df.apply(get_image_plane, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:49:43.116110Z","iopub.execute_input":"2021-08-23T11:49:43.116614Z","iopub.status.idle":"2021-08-23T11:50:03.975700Z","shell.execute_reply.started":"2021-08-23T11:49:43.116575Z","shell.execute_reply":"2021-08-23T11:50:03.974790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dftr = train_meta_df.groupby(['PatientID', 'Orientation', 'SeriesDescription']).size().reset_index(name='count') \ndftr","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:03.977075Z","iopub.execute_input":"2021-08-23T11:50:03.977441Z","iopub.status.idle":"2021-08-23T11:50:04.050854Z","shell.execute_reply.started":"2021-08-23T11:50:03.977388Z","shell.execute_reply":"2021-08-23T11:50:04.049899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"NOTE: Here we have not removed yet the 3 patients with problematic datasets, thus 585 * 4 = 2340.","metadata":{}},{"cell_type":"markdown","source":"## 2. Choose a reference example modality in terms of orientation and resolution","metadata":{}},{"cell_type":"code","source":"dftr2 = train_meta_df[(train_meta_df.Rows == 256) & \n (train_meta_df.Columns == 256) &\n (train_meta_df.Orientation == \"axial\") &\n (train_meta_df.SeriesDescription == \"T1w\")].groupby(['PatientID', 'Orientation', 'SeriesDescription']).size().reset_index(name='count') \ndftr2.loc[(dftr2['count'] < 50) & (dftr2['count'] >15)].reset_index(drop = True)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:04.052237Z","iopub.execute_input":"2021-08-23T11:50:04.052597Z","iopub.status.idle":"2021-08-23T11:50:04.377053Z","shell.execute_reply.started":"2021-08-23T11:50:04.052560Z","shell.execute_reply":"2021-08-23T11:50:04.376103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dftr[dftr.PatientID == 143]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:04.378498Z","iopub.execute_input":"2021-08-23T11:50:04.378844Z","iopub.status.idle":"2021-08-23T11:50:04.390803Z","shell.execute_reply.started":"2021-08-23T11:50:04.378808Z","shell.execute_reply":"2021-08-23T11:50:04.389870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### In Conclusion:\n* There are 20 examples (20 patients) with T1w 3D volumes of resolution 256 x 256 and between 15 and 50 slices and \"Axial\" orientation.\n\n* We chose PatientID = 143 with T1w 256x256 and 36 slices as the rederence 3D volume.\n\n* Next we will convert all other modalities for all patients into the same voxel space as the 3D volume of PatientID = 143. \n\n* For that we will need a dataframe with all the modalities and their associated file - paths (about 2328 = 582 * 4).","metadata":{}},{"cell_type":"markdown","source":"### DataFrame of all modalities with their associated file-paths","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/train_filepaths_rsna.csv')\ntest_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/test_filepaths_rsna.csv')\n\ndir_train_df = train_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\ndir_test_df = test_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:04.392263Z","iopub.execute_input":"2021-08-23T11:50:04.392623Z","iopub.status.idle":"2021-08-23T11:50:05.355688Z","shell.execute_reply.started":"2021-08-23T11:50:04.392589Z","shell.execute_reply":"2021-08-23T11:50:05.354819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(dir_train_df.head(2))\ndisplay(dir_test_df.head(2))","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.356999Z","iopub.execute_input":"2021-08-23T11:50:05.357345Z","iopub.status.idle":"2021-08-23T11:50:05.375202Z","shell.execute_reply.started":"2021-08-23T11:50:05.357311Z","shell.execute_reply":"2021-08-23T11:50:05.374284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dir_train_df[dir_train_df.patient_id == 143]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.376665Z","iopub.execute_input":"2021-08-23T11:50:05.377015Z","iopub.status.idle":"2021-08-23T11:50:05.388469Z","shell.execute_reply.started":"2021-08-23T11:50:05.376981Z","shell.execute_reply":"2021-08-23T11:50:05.387490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Index 369","metadata":{}},{"cell_type":"code","source":"dir_train_df.iloc[369]","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.389879Z","iopub.execute_input":"2021-08-23T11:50:05.390396Z","iopub.status.idle":"2021-08-23T11:50:05.398705Z","shell.execute_reply.started":"2021-08-23T11:50:05.390359Z","shell.execute_reply":"2021-08-23T11:50:05.397590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3 + 4. Convert to Voxel Space of Choice with Simple ITK and Save","metadata":{}},{"cell_type":"markdown","source":"### Helper Functions to : Resample, Normalize and Convert to Voxel Space of Interest","metadata":{}},{"cell_type":"code","source":"def resample(image, ref_image):\n\n resampler = sitk.ResampleImageFilter()\n resampler.SetReferenceImage(ref_image)\n resampler.SetInterpolator(sitk.sitkLinear)\n \n resampler.SetTransform(sitk.AffineTransform(image.GetDimension()))\n\n resampler.SetOutputSpacing(ref_image.GetSpacing())\n\n resampler.SetSize(ref_image.GetSize())\n\n resampler.SetOutputDirection(ref_image.GetDirection())\n\n resampler.SetOutputOrigin(ref_image.GetOrigin())\n\n resampler.SetDefaultPixelValue(image.GetPixelIDValue())\n\n resamped_image = resampler.Execute(image)\n \n return resamped_image","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.400131Z","iopub.execute_input":"2021-08-23T11:50:05.400557Z","iopub.status.idle":"2021-08-23T11:50:05.407891Z","shell.execute_reply.started":"2021-08-23T11:50:05.400522Z","shell.execute_reply":"2021-08-23T11:50:05.406863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def normalize_png(data):\n '''Input: BLock of 2D Images forming a 3D volume of a full brain'''\n data = data - np.min(data)\n if np.max(data) != 0:\n data = data / np.max(data)\n data = (data * 255).astype(np.uint8)\n return data","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.409141Z","iopub.execute_input":"2021-08-23T11:50:05.409682Z","iopub.status.idle":"2021-08-23T11:50:05.417203Z","shell.execute_reply.started":"2021-08-23T11:50:05.409646Z","shell.execute_reply":"2021-08-23T11:50:05.416401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Helper Functions to : Create Directory Tree and Save Converted Voxel Space","metadata":{}},{"cell_type":"code","source":"def create_train_val_dir(PNG_MSTR_DIR, train_df, val_df, fold = None):\n if fold:\n # create the main PNG folder with the val and train \n png_val_path=PNG_MSTR_DIR + '/val_'+str(fold)+'/'\n png_train_path=PNG_MSTR_DIR + '/train_'+str(fold)+'/'\n else:\n png_val_path=PNG_MSTR_DIR + '/val/'\n png_train_path=PNG_MSTR_DIR + '/train/'\n \n os.makedirs(png_train_path)\n os.makedirs(png_val_path)\n print('\\t\\t\\t DONE DIR TREE')\n\n # floders creation \n for trfold in set(train_df.patient_id):\n os.mkdir(png_train_path+str(trfold).zfill(5))\n for mp in set(train_df.scan_type):\n os.mkdir(png_train_path+str(trfold).zfill(5)+'/'+str(mp))\n print('\\t\\t\\t DONE SAVING TRAIN IMAGES')\n\n for trfold in set(val_df.patient_id): \n os.mkdir(png_val_path+str(trfold).zfill(5))\n for mp in set(val_df.scan_type):\n os.mkdir(png_val_path+str(trfold).zfill(5)+'/'+str(mp))\n print('\\t\\t\\t DONE SAVING VAL IMAGES')\n \n \ndef create_test_dir(PNG_MSTR_DIR, test_df):\n png_test_path=PNG_MSTR_DIR + '/test/'\n\n os.makedirs(png_test_path)\n print('\\t\\t\\t DONE DIR TREE')\n\n # floders creation \n for trfold in set(test_df.patient_id):\n os.mkdir(png_test_path+str(trfold).zfill(5))\n for mp in set(test_df.scan_type):\n os.mkdir(png_test_path+str(trfold).zfill(5)+'/'+str(mp))\n print('\\t\\t\\t DONE SAVING TEST IMAGES')\n \ndef convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df, val_df, fold = None, \n is_test=True, save_cv_test = False, save_train_val = True, \n test_df = None, threshold = -1.0):\n# ref_dir = str(dir_train_df.directory.iloc[369])+ \\\n# str(dir_train_df.patient_id.iloc[369]).zfill(5)+ \\\n# '/'+str(dir_train_df.scan_type.iloc[369])\n print('Reference Dir:')\n print(f'{ref_dir}')\n\n reader = sitk.ImageSeriesReader()\n reader.LoadPrivateTagsOn()\n\n filenamesDICOM = reader.GetGDCMSeriesFileNames(f'{ref_dir}')\n reader.SetFileNames(filenamesDICOM)\n ref_sitk = reader.Execute()\n\n if save_train_val:\n if is_test:\n print('\\t STARTED TRAIN CONVERSION and SAVING TO DISK')\n else:\n print('\\t STARTED TRAIN_'+str(fold)+' CONVERSION and SAVING TO DISK')\n for i in tqdm(range(len(train_df))):\n scan_dir = str(train_df.directory.iloc[i])+ \\\n str(train_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(train_df.scan_type.iloc[i])\n\n if is_test:\n output_dir = png_out_dir+'/train/'+ \\\n str(train_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(train_df.scan_type.iloc[i])\n else:\n output_dir = png_out_dir+'/train_'+str(fold)+'/'+ \\\n str(train_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(train_df.scan_type.iloc[i])\n\n filenamesDICOM = reader.GetGDCMSeriesFileNames(f'{scan_dir}')\n reader.SetFileNames(filenamesDICOM)\n scan_sitk = reader.Execute()\n\n scan_resampled = resample(scan_sitk, ref_sitk)\n scan_sitk_array = normalize_png(sitk.GetArrayFromImage(scan_resampled))\n\n for j in range(len(scan_sitk_array[:,0,0])):\n # SAVE PNG TO DISK IF Criterion TRUE \n if np.mean(scan_sitk_array[j,:,:]) <= threshold:\n pass\n else:\n imageio.imsave(output_dir+'/Image-'+str(j)+'.png', scan_sitk_array[j,:,:])\n if is_test:\n print('\\t FINISHED TRAIN CONVERSION and SAVING TO DISK')\n else:\n print('\\t FINISHED TRAIN_'+str(fold)+' CONVERSION and SAVING TO DISK')\n\n if is_test:\n print('\\t STARTED TEST CONVERSION and SAVING TO DISK')\n else:\n print('\\t STARTED VAL_'+str(fold)+' CONVERSION and SAVING TO DISK')\n for i in tqdm(range(len(val_df))):\n scan_dir = str(val_df.directory.iloc[i])+ \\\n str(val_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(val_df.scan_type.iloc[i])\n\n if is_test:\n output_dir = png_out_dir+'/test/'+ \\\n str(val_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(val_df.scan_type.iloc[i])\n else:\n output_dir = png_out_dir+'/val_'+str(fold)+'/'+ \\\n str(val_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(val_df.scan_type.iloc[i])\n\n filenamesDICOM = reader.GetGDCMSeriesFileNames(f'{scan_dir}')\n reader.SetFileNames(filenamesDICOM)\n scan_sitk = reader.Execute()\n\n scan_resampled = resample(scan_sitk, ref_sitk)\n scan_sitk_array = normalize_png(sitk.GetArrayFromImage(scan_resampled))\n\n for j in range(len(scan_sitk_array[:,0,0])):\n # SAVE PNG TO DISK IF Criterion TRUE \n if np.mean(scan_sitk_array[j,:,:]) <= threshold:\n pass\n else:\n imageio.imsave(output_dir+'/Image-'+str(j)+'.png', scan_sitk_array[j,:,:])\n if is_test:\n print('\\t FINISHED TEST CONVERSION and SAVING TO DISK')\n else:\n print('\\t FINISHED VAL_'+str(fold)+' CONVERSION and SAVING TO DISK')\n else:\n print('\\t ONLY SAVE TEST - SKIP TRAIN / VAL')\n \n if save_cv_test:\n print('\\t STARTED TEST CONVERSION and SAVING TO DISK')\n for i in tqdm(range(len(test_df))):\n scan_dir = str(test_df.directory.iloc[i])+ \\\n str(test_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(test_df.scan_type.iloc[i])\n\n output_dir = png_out_dir+'/test/'+ \\\n str(test_df.patient_id.iloc[i]).zfill(5)+ \\\n '/'+str(test_df.scan_type.iloc[i])\n\n filenamesDICOM = reader.GetGDCMSeriesFileNames(f'{scan_dir}')\n reader.SetFileNames(filenamesDICOM)\n scan_sitk = reader.Execute()\n\n scan_resampled = resample(scan_sitk, ref_sitk)\n scan_sitk_array = normalize_png(sitk.GetArrayFromImage(scan_resampled))\n\n for j in range(len(scan_sitk_array[:,0,0])):\n # SAVE PNG TO DISK \n imageio.imsave(output_dir+'/Image-'+str(j)+'.png', scan_sitk_array[j,:,:])\n print('\\t FINISHED TEST CONVERSION and SAVING TO DISK')","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.418514Z","iopub.execute_input":"2021-08-23T11:50:05.418854Z","iopub.status.idle":"2021-08-23T11:50:05.448047Z","shell.execute_reply.started":"2021-08-23T11:50:05.418821Z","shell.execute_reply":"2021-08-23T11:50:05.447217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Create k-fold Train / Val Directories, Convert and Save PNGs**","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— SAVE A FEW EXAMPLES TO DEMONSTRATE โ—โ—โ— ","metadata":{}},{"cell_type":"code","source":"create_k_fold_cv_ds = True\ncreate_test_ds = False\nfold = 1\n\nif create_k_fold_cv_ds:\n # Load summary dataframes\n train_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/train_filepaths_rsna.csv')\n test_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/test_filepaths_rsna.csv')\n\n dir_train_df = train_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n dir_test_df = test_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\n # Create k-fold Train / Val Directories\n png_out_dir = os.path.join('/kaggle/working', 'png_'+str(fold)+'_outof_5_fold_cv')\n if os.path.exists(png_out_dir) and os.path.isdir(png_out_dir):\n shutil.rmtree(png_out_dir)\n os.makedirs(png_out_dir)\n\n create_train_val_dir(png_out_dir, train_df_1, val_df_1, fold = fold)\n# create_train_val_dir(png_out_dir, train_df_2, val_df_2, fold = fold)\n# create_train_val_dir(png_out_dir, train_df_3, val_df_3, fold = fold)\n# create_train_val_dir(png_out_dir, train_df_4, val_df_4, fold = fold)\n# create_train_val_dir(png_out_dir, train_df_5, val_df_5, fold = fold)\n\n # Choose Reference 3D Volume (Collection of Images)\n ref_dir = str(dir_train_df.directory.iloc[369])+ \\\n str(dir_train_df.patient_id.iloc[369]).zfill(5)+ \\\n '/'+str(dir_train_df.scan_type.iloc[369])\n \n if create_test_ds:\n # Create Test Directory\n# test_png_out_dir = os.path.join('/kaggle/working', 'png_test_axial_256x256x36')\n# if os.path.exists(test_png_out_dir) and os.path.isdir(test_png_out_dir):\n# shutil.rmtree(test_png_out_dir)\n# os.makedirs(test_png_out_dir)\n# png_out_dir = test_png_out_dir\n create_test_dir(png_out_dir, dir_test_df)\n convert_train_val_2_voxel_space(ref_dir, test_png_out_dir, train_df_1, val_df_1, fold = fold, \n is_test=False, save_cv_test = True, save_train_val = False, test_df = dir_test_df[:8])\n\n # Convert to Voxel Space of Choice\n convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df_1[:8], val_df_1[:8], fold = fold, \n is_test=False, threshold = 0.0)\n# convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df_2, val_df_2, fold = fold, \n# is_test=False)\n# convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df_3, val_df_3, fold = fold, \n# is_test=False)\n# convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df_4, val_df_4, fold = fold, \n# is_test=False)\n# convert_train_val_2_voxel_space(ref_dir, png_out_dir, train_df_5, val_df_5, fold = fold, \n# is_test=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:05.449213Z","iopub.execute_input":"2021-08-23T11:50:05.449565Z","iopub.status.idle":"2021-08-23T11:50:34.858834Z","shell.execute_reply.started":"2021-08-23T11:50:05.449531Z","shell.execute_reply":"2021-08-23T11:50:34.857881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check that empty slices were removed","metadata":{}},{"cell_type":"code","source":"!ls -alt ./png_1_outof_5_fold_cv/train_1/00183/FLAIR","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:34.860071Z","iopub.execute_input":"2021-08-23T11:50:34.860442Z","iopub.status.idle":"2021-08-23T11:50:35.510665Z","shell.execute_reply.started":"2021-08-23T11:50:34.860390Z","shell.execute_reply":"2021-08-23T11:50:35.509765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed Images 0,1,2 and 32, 33, 34, 35 for threshold = 0 were removed.","metadata":{}},{"cell_type":"markdown","source":"**Create Train / Test Directories, Convert and Save PNGs**","metadata":{}},{"cell_type":"markdown","source":"โ—โ—โ— SAVE A FEW EXAMPLES TO DEMONSTRATE โ—โ—โ— ","metadata":{}},{"cell_type":"code","source":"create_train_test_ds = False\n\nif create_train_test_ds:\n # Load summary dataframes\n train_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/train_filepaths_rsna.csv')\n test_df = pd.read_csv('/kaggle/input/train-test-filepaths-rsna-full/test_filepaths_rsna.csv')\n\n dir_train_df = train_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n dir_test_df = test_df.groupby(['patient_id', 'directory', 'scan_type']).size().reset_index(name='count')\n\n # Create Train / Test Dirs\n png_out_dir = os.path.join('/kaggle/working', 'png_voxel_converted_ds')\n if os.path.exists(png_out_dir) and os.path.isdir(png_out_dir):\n shutil.rmtree(png_out_dir)\n os.makedirs(png_out_dir)\n \n create_png_dir_tree(png_out_dir, dir_train_df, dir_test_df)\n\n # Choose Reference 3D Volume (Collection of Images)\n ref_dir = str(dir_train_df.directory.iloc[369])+ \\\n str(dir_train_df.patient_id.iloc[369]).zfill(5)+ \\\n '/'+str(dir_train_df.scan_type.iloc[369])\n\n # Convert to Voxel Space of Choice and Save\n convert_train_val_2_voxel_space(ref_dir, png_out_dir, dir_train_df[:8], dir_test_df[:8], \n fold = None, is_test=True)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:35.512256Z","iopub.execute_input":"2021-08-23T11:50:35.512632Z","iopub.status.idle":"2021-08-23T11:50:35.521399Z","shell.execute_reply.started":"2021-08-23T11:50:35.512593Z","shell.execute_reply":"2021-08-23T11:50:35.520364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''Check out the writing to disk process.'''\n# !ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00143/T1w/ | wc -l\n# !ls /kaggle/working/png_voxel_converted_ds/train/00000/FLAIR","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:35.522858Z","iopub.execute_input":"2021-08-23T11:50:35.523210Z","iopub.status.idle":"2021-08-23T11:50:35.537966Z","shell.execute_reply.started":"2021-08-23T11:50:35.523169Z","shell.execute_reply":"2021-08-23T11:50:35.537072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Visualize slices of Patient ID 0, which has T1w, T1wCE : Axial, FLAIR : Coronal, and T2w: Sagittal Orientations and can easily see whether the conversion to Axial worked or not.**","metadata":{}},{"cell_type":"code","source":"if create_train_test_ds:\n n = 1\n j = 0\n plt.figure(figsize=(18, 10))\n img = mpimg.imread('/kaggle/working/png_voxel_converted_ds/train/00000/T2w/Image-18.png')\n print(img.shape)\n plt.subplot(1, 4, 1)\n imgplot = plt.imshow(img)\n img = mpimg.imread('/kaggle/working/png_voxel_converted_ds/train/00000/FLAIR/Image-18.png')\n print(img.shape)\n plt.subplot(1, 4, 2)\n imgplot = plt.imshow(img)\n img = mpimg.imread('/kaggle/working/png_voxel_converted_ds/train/00000/T1wCE/Image-18.png')\n print(img.shape)\n plt.subplot(1, 4, 3)\n imgplot = plt.imshow(img)\n img = mpimg.imread('/kaggle/working/png_voxel_converted_ds/train/00000/T1w/Image-18.png')\n print(img.shape)\n plt.subplot(1, 4, 4)\n imgplot = plt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:35.539302Z","iopub.execute_input":"2021-08-23T11:50:35.539712Z","iopub.status.idle":"2021-08-23T11:50:35.547292Z","shell.execute_reply.started":"2021-08-23T11:50:35.539676Z","shell.execute_reply":"2021-08-23T11:50:35.546063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 5. You can remove images with mostly empty pixel values (see Sec 5: Filter Based on Theshold and Save PNGs)","metadata":{}},{"cell_type":"markdown","source":"# Trick to be able to create Dataset from PNG image collections","metadata":{}},{"cell_type":"markdown","source":"Save any time of csv or text file","metadata":{}},{"cell_type":"code","source":"!echo \"This dataset contains PNG files in AXIAL orientation for all patients\" > README.txt","metadata":{"execution":{"iopub.status.busy":"2021-08-23T11:50:35.548662Z","iopub.execute_input":"2021-08-23T11:50:35.549135Z","iopub.status.idle":"2021-08-23T11:50:36.195900Z","shell.execute_reply.started":"2021-08-23T11:50:35.549101Z","shell.execute_reply":"2021-08-23T11:50:36.194835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}