{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EDA 1 for Brain Tumor Classification\n\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport pandas as pd\nimport seaborn as sns\nfrom pathlib import Path\n\nsns.set_context(\"talk\", font_scale=1.4)\nsns.set_style(\"whitegrid\")","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:06.557190Z","iopub.execute_input":"2021-10-17T11:04:06.557680Z","iopub.status.idle":"2021-10-17T11:04:07.486181Z","shell.execute_reply.started":"2021-10-17T11:04:06.557592Z","shell.execute_reply":"2021-10-17T11:04:07.484838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 1: Goal\n\nTo predict the genetics of the cancer through imaging (i.e., radiogenomics) to minimize the number of surgeries and refine the type of therapy required for patients. Specifically, predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to train and test your model to detect for the presence of MGMT promoter methylation. This problem can be treated as a binary classification problem to predict a probability for the target MGMT_value.\n\n* The training data should generalize to the test data but that depends on how the split was done? Each data point is one patient. Does the data represent the whole population, all patients in the world or just NA?\n    * multiple institutions, variety of instruments (paper)\n* Is each entry independent (five-digit number per unique patient) or do we have duplicates, e.g. same patient different time?\n* How was the negative set created? Imagery of random selected people?\n* ***Submission rules***:\n    * According to the submission (button) extra explanation: \"In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.\"\n        * this means the submitted notebook needs to work with the raw dicom files (additionally provided with private testset)\n    * Notebook has to be without internet access.\n\n\n## Metric\n\n* The metric to optimize is the AUC ROC curve.\n* AUC Still prone to class imbalance to some degree.\n* based on MGMT_value=1/0\n","metadata":{}},{"cell_type":"markdown","source":"# Step 2: Gather the data.\n\n* The data was already provided by Kaggle and split in training and test set. How was the split done? Was ensured that same distribution of data in train and test?\n* Is there alternative open data which could help to improve performance?\n* Data Described in paper: https://arxiv.org/pdf/2107.02314.pdf\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 3: Extract the Data\n\nOrganizers suggest to exclude the following training data: [00109, 00123, 00709]","metadata":{}},{"cell_type":"code","source":"! tree /kaggle/input/ -L 2","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:07.488195Z","iopub.execute_input":"2021-10-17T11:04:07.488517Z","iopub.status.idle":"2021-10-17T11:04:08.284045Z","shell.execute_reply.started":"2021-10-17T11:04:07.488475Z","shell.execute_reply":"2021-10-17T11:04:08.283055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train | head -n 5","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:08.285687Z","iopub.execute_input":"2021-10-17T11:04:08.285984Z","iopub.status.idle":"2021-10-17T11:04:09.115091Z","shell.execute_reply.started":"2021-10-17T11:04:08.285951Z","shell.execute_reply":"2021-10-17T11:04:09.114019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:09.117456Z","iopub.execute_input":"2021-10-17T11:04:09.117752Z","iopub.status.idle":"2021-10-17T11:04:09.898654Z","shell.execute_reply.started":"2021-10-17T11:04:09.117716Z","shell.execute_reply":"2021-10-17T11:04:09.897907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Different RMI imagery:\n\n* Fluid Attenuated Inversion Recovery (FLAIR)\n* T1-weighted pre-contrast (T1w)\n* T1-weighted post-contrast (T1Gd)\n* T2-weighted (T2)","metadata":{}},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/FLAIR | head -n 5","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:09.900122Z","iopub.execute_input":"2021-10-17T11:04:09.900378Z","iopub.status.idle":"2021-10-17T11:04:10.777431Z","shell.execute_reply.started":"2021-10-17T11:04:09.900348Z","shell.execute_reply":"2021-10-17T11:04:10.776409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/test","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:10.779154Z","iopub.execute_input":"2021-10-17T11:04:10.780206Z","iopub.status.idle":"2021-10-17T11:04:11.555905Z","shell.execute_reply.started":"2021-10-17T11:04:10.780154Z","shell.execute_reply":"2021-10-17T11:04:11.554848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> * Small test set!\n> * IDs seem unique, also between training and test set","metadata":{}},{"cell_type":"markdown","source":"# Step 4: Meet and Greet the data\n\nTrain - Test Split: 87% train, rest test\n","metadata":{}},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/train | wc -l","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:11.557534Z","iopub.execute_input":"2021-10-17T11:04:11.557832Z","iopub.status.idle":"2021-10-17T11:04:12.331932Z","shell.execute_reply.started":"2021-10-17T11:04:11.557796Z","shell.execute_reply":"2021-10-17T11:04:12.331070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"! ls /kaggle/input/rsna-miccai-brain-tumor-radiogenomic-classification/test | wc -l","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:12.333581Z","iopub.execute_input":"2021-10-17T11:04:12.334291Z","iopub.status.idle":"2021-10-17T11:04:13.110099Z","shell.execute_reply.started":"2021-10-17T11:04:12.334243Z","shell.execute_reply":"2021-10-17T11:04:13.109082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = Path(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification\")","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.111645Z","iopub.execute_input":"2021-10-17T11:04:13.111935Z","iopub.status.idle":"2021-10-17T11:04:13.118161Z","shell.execute_reply.started":"2021-10-17T11:04:13.111902Z","shell.execute_reply":"2021-10-17T11:04:13.117168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = pd.read_csv(data_path/\"train_labels.csv\")\nlabels.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.122277Z","iopub.execute_input":"2021-10-17T11:04:13.122573Z","iopub.status.idle":"2021-10-17T11:04:13.145746Z","shell.execute_reply.started":"2021-10-17T11:04:13.122538Z","shell.execute_reply":"2021-10-17T11:04:13.144782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.147345Z","iopub.execute_input":"2021-10-17T11:04:13.147661Z","iopub.status.idle":"2021-10-17T11:04:13.166451Z","shell.execute_reply.started":"2021-10-17T11:04:13.147618Z","shell.execute_reply":"2021-10-17T11:04:13.165830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels.sample(10, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.167803Z","iopub.execute_input":"2021-10-17T11:04:13.168665Z","iopub.status.idle":"2021-10-17T11:04:13.183516Z","shell.execute_reply.started":"2021-10-17T11:04:13.168624Z","shell.execute_reply":"2021-10-17T11:04:13.182496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels.info()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.184832Z","iopub.execute_input":"2021-10-17T11:04:13.185061Z","iopub.status.idle":"2021-10-17T11:04:13.211668Z","shell.execute_reply.started":"2021-10-17T11:04:13.185036Z","shell.execute_reply":"2021-10-17T11:04:13.210781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Missing Values","metadata":{}},{"cell_type":"code","source":"labels.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.213066Z","iopub.execute_input":"2021-10-17T11:04:13.213289Z","iopub.status.idle":"2021-10-17T11:04:13.221676Z","shell.execute_reply.started":"2021-10-17T11:04:13.213264Z","shell.execute_reply":"2021-10-17T11:04:13.220795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Duplication","metadata":{}},{"cell_type":"code","source":"labels.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.223913Z","iopub.execute_input":"2021-10-17T11:04:13.224341Z","iopub.status.idle":"2021-10-17T11:04:13.237015Z","shell.execute_reply.started":"2021-10-17T11:04:13.224292Z","shell.execute_reply":"2021-10-17T11:04:13.236372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"folder_struct = []\nfor path in (data_path).rglob('*.dcm'):\n    folder_struct.append(str(path).split(\"/\")[3:]+[str(path)])","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:04:13.238060Z","iopub.execute_input":"2021-10-17T11:04:13.238408Z","iopub.status.idle":"2021-10-17T11:05:49.067288Z","shell.execute_reply.started":"2021-10-17T11:04:13.238380Z","shell.execute_reply":"2021-10-17T11:05:49.066243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image Metadata\n\n","metadata":{}},{"cell_type":"code","source":"image_meta = pd.DataFrame(folder_struct, columns=[\"type\", \"id\", \"mri\", \"file\", \"path\"])\nimage_meta.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:49.068726Z","iopub.execute_input":"2021-10-17T11:05:49.069054Z","iopub.status.idle":"2021-10-17T11:05:49.236461Z","shell.execute_reply.started":"2021-10-17T11:05:49.069012Z","shell.execute_reply":"2021-10-17T11:05:49.235679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_meta['img_num'] = image_meta['file'].str.extract(r'Image-(\\d+).dcm').astype(int)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:49.237618Z","iopub.execute_input":"2021-10-17T11:05:49.237905Z","iopub.status.idle":"2021-10-17T11:05:50.617181Z","shell.execute_reply.started":"2021-10-17T11:05:49.237857Z","shell.execute_reply":"2021-10-17T11:05:50.616213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_meta.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:50.618184Z","iopub.execute_input":"2021-10-17T11:05:50.618416Z","iopub.status.idle":"2021-10-17T11:05:50.625193Z","shell.execute_reply.started":"2021-10-17T11:05:50.618390Z","shell.execute_reply":"2021-10-17T11:05:50.624158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_file_counts =image_meta.groupby([\"type\",\"id\", \"mri\"]).count()['file'].reset_index().rename(columns={\"file\": \"counts\"})\nimage_file_counts.head(20)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:50.626322Z","iopub.execute_input":"2021-10-17T11:05:50.626550Z","iopub.status.idle":"2021-10-17T11:05:51.005258Z","shell.execute_reply.started":"2021-10-17T11:05:50.626524Z","shell.execute_reply":"2021-10-17T11:05:51.004387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Number of .dcm files not consistent,maybe due to different methdos in hospitals?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(16,4))\nsns.histplot(data=image_file_counts, x=\"counts\", stat=\"count\", hue='mri', ax=ax)\nax.set_xlabel(\"# of .dcm files per person\")","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:51.006825Z","iopub.execute_input":"2021-10-17T11:05:51.007314Z","iopub.status.idle":"2021-10-17T11:05:51.700061Z","shell.execute_reply.started":"2021-10-17T11:05:51.007271Z","shell.execute_reply":"2021-10-17T11:05:51.699170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_file_counts['counts'].describe()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:51.701644Z","iopub.execute_input":"2021-10-17T11:05:51.702163Z","iopub.status.idle":"2021-10-17T11:05:51.715460Z","shell.execute_reply.started":"2021-10-17T11:05:51.702121Z","shell.execute_reply":"2021-10-17T11:05:51.714412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> At maximum we have 514 images per MRI type.","metadata":{}},{"cell_type":"code","source":"image_file_counts.groupby(\"type\")['id'].nunique(), image_file_counts['id'].shape","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:05:51.716644Z","iopub.execute_input":"2021-10-17T11:05:51.716964Z","iopub.status.idle":"2021-10-17T11:05:51.728160Z","shell.execute_reply.started":"2021-10-17T11:05:51.716923Z","shell.execute_reply":"2021-10-17T11:05:51.727230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Missing Values 2\n\nAre there persons which have a MRI type missing?","metadata":{}},{"cell_type":"code","source":"image_meta_miss = image_meta[['type', 'id', 'mri', 'path']].pivot_table(index=['type', 'id'], columns=[\"mri\"], values=['path'], aggfunc=\n                                                                        lambda x: len(x) if len(x)>0 else np.nan)\nimage_meta_miss.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:26:53.791873Z","iopub.execute_input":"2021-10-17T11:26:53.792218Z","iopub.status.idle":"2021-10-17T11:26:54.188467Z","shell.execute_reply.started":"2021-10-17T11:26:53.792183Z","shell.execute_reply":"2021-10-17T11:26:54.187851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_meta_miss.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:26:58.510390Z","iopub.execute_input":"2021-10-17T11:26:58.511564Z","iopub.status.idle":"2021-10-17T11:26:58.520267Z","shell.execute_reply.started":"2021-10-17T11:26:58.511505Z","shell.execute_reply":"2021-10-17T11:26:58.519478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Train and Test Data all have all MRI types.","metadata":{}},{"cell_type":"markdown","source":"Are there missing labels for some persons?","metadata":{}},{"cell_type":"code","source":"labels['id'] = labels[\"BraTS21ID\"].apply(lambda num: f\"{num:05d}\")\nlabels['label'] = labels['MGMT_value']\nlabels.drop(columns=[\"BraTS21ID\", 'MGMT_value'], inplace=True)\nimage_meta = pd.merge(image_meta, labels, on = \"id\", how=\"left\")","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:30:04.866234Z","iopub.execute_input":"2021-10-17T11:30:04.867208Z","iopub.status.idle":"2021-10-17T11:30:05.031658Z","shell.execute_reply.started":"2021-10-17T11:30:04.867167Z","shell.execute_reply":"2021-10-17T11:30:05.030777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_meta.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:30:30.431216Z","iopub.execute_input":"2021-10-17T11:30:30.431532Z","iopub.status.idle":"2021-10-17T11:30:30.447430Z","shell.execute_reply.started":"2021-10-17T11:30:30.431499Z","shell.execute_reply":"2021-10-17T11:30:30.446562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_meta[image_meta['type'] == \"train\"].isna().sum()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:32:58.784994Z","iopub.execute_input":"2021-10-17T11:32:58.786190Z","iopub.status.idle":"2021-10-17T11:32:59.160851Z","shell.execute_reply.started":"2021-10-17T11:32:58.786122Z","shell.execute_reply":"2021-10-17T11:32:59.159906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> No missing values for label in train data.","metadata":{}},{"cell_type":"markdown","source":"### Look at Specific Person in Detail","metadata":{}},{"cell_type":"code","source":"! pip install pydicom","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:33.533933Z","iopub.execute_input":"2021-10-17T11:33:33.534209Z","iopub.status.idle":"2021-10-17T11:33:41.515255Z","shell.execute_reply.started":"2021-10-17T11:33:33.534180Z","shell.execute_reply":"2021-10-17T11:33:41.514163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pydicom import dcmread\nfrom pydicom.data import get_testdata_file\nfrom pydicom import filereader\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:06:01.014471Z","iopub.execute_input":"2021-10-17T11:06:01.014698Z","iopub.status.idle":"2021-10-17T11:06:01.136204Z","shell.execute_reply.started":"2021-10-17T11:06:01.014669Z","shell.execute_reply":"2021-10-17T11:06:01.135271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_person = image_meta.query(\"id=='00688'\")\nsample_person.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.517329Z","iopub.execute_input":"2021-10-17T11:33:41.517554Z","iopub.status.idle":"2021-10-17T11:33:41.609242Z","shell.execute_reply.started":"2021-10-17T11:33:41.517526Z","shell.execute_reply":"2021-10-17T11:33:41.608318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dcmread(sample_person.at[51473, 'path'])","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.610535Z","iopub.execute_input":"2021-10-17T11:33:41.610797Z","iopub.status.idle":"2021-10-17T11:33:41.632346Z","shell.execute_reply.started":"2021-10-17T11:33:41.610768Z","shell.execute_reply":"2021-10-17T11:33:41.631476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = filereader.dcmread(sample_person.at[51473, 'path'])\nimg = dataset.pixel_array","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.634089Z","iopub.execute_input":"2021-10-17T11:33:41.634324Z","iopub.status.idle":"2021-10-17T11:33:41.642571Z","shell.execute_reply.started":"2021-10-17T11:33:41.634296Z","shell.execute_reply":"2021-10-17T11:33:41.641799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img.shape, img.dtype","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.643879Z","iopub.execute_input":"2021-10-17T11:33:41.644307Z","iopub.status.idle":"2021-10-17T11:33:41.652965Z","shell.execute_reply.started":"2021-10-17T11:33:41.644275Z","shell.execute_reply":"2021-10-17T11:33:41.651890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots()\nax.imshow(img, cmap='gray')\nax.set_axis_off()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.655230Z","iopub.execute_input":"2021-10-17T11:33:41.656652Z","iopub.status.idle":"2021-10-17T11:33:41.839643Z","shell.execute_reply.started":"2021-10-17T11:33:41.656617Z","shell.execute_reply":"2021-10-17T11:33:41.838533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"first_imgs = sample_person.sort_values(['mri', 'img_num']).groupby(\"mri\").first()\nfirst_imgs","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.841156Z","iopub.execute_input":"2021-10-17T11:33:41.841477Z","iopub.status.idle":"2021-10-17T11:33:41.864517Z","shell.execute_reply.started":"2021-10-17T11:33:41.841437Z","shell.execute_reply":"2021-10-17T11:33:41.863805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_img_array(image_path):\n    dataset = filereader.dcmread(image_path)\n    return dataset.pixel_array","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.865490Z","iopub.execute_input":"2021-10-17T11:33:41.866166Z","iopub.status.idle":"2021-10-17T11:33:41.870287Z","shell.execute_reply.started":"2021-10-17T11:33:41.866131Z","shell.execute_reply":"2021-10-17T11:33:41.869520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig,axes = plt.subplots(nrows=2, ncols=2)\naxes= axes.flatten()\nfor i,ax in enumerate(axes):\n    ax.imshow(get_img_array(first_imgs.iloc[i]['path']), cmap='gray')\n    ax.set_axis_off()\n    ax.set_title(first_imgs.index[i])\n    #plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:41.871291Z","iopub.execute_input":"2021-10-17T11:33:41.871841Z","iopub.status.idle":"2021-10-17T11:33:42.385367Z","shell.execute_reply.started":"2021-10-17T11:33:41.871814Z","shell.execute_reply":"2021-10-17T11:33:42.384500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_person.sort_values(['mri', 'img_num']).groupby(\"mri\")['img_num'].apply(lambda lst: lst.to_list())","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:42.387824Z","iopub.execute_input":"2021-10-17T11:33:42.388158Z","iopub.status.idle":"2021-10-17T11:33:42.403396Z","shell.execute_reply.started":"2021-10-17T11:33:42.388121Z","shell.execute_reply":"2021-10-17T11:33:42.402267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compare_img_types(img_num):\n    select_img = sample_person[sample_person['img_num'] == img_num]\n    print(select_img['mri'])\n    \n    fig,axes = plt.subplots(nrows=2, ncols=2, figsize=(16,16))\n    axes= axes.flatten()\n    for i,ax in enumerate(axes):\n        ax.imshow(get_img_array(select_img.iloc[i]['path']), cmap='gray')\n        ax.set_axis_off()\n        ax.set_title(first_imgs.index[i])","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:42.404811Z","iopub.execute_input":"2021-10-17T11:33:42.405046Z","iopub.status.idle":"2021-10-17T11:33:42.414199Z","shell.execute_reply.started":"2021-10-17T11:33:42.405019Z","shell.execute_reply":"2021-10-17T11:33:42.412970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_img_types(150)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:42.415529Z","iopub.execute_input":"2021-10-17T11:33:42.416364Z","iopub.status.idle":"2021-10-17T11:33:43.137515Z","shell.execute_reply.started":"2021-10-17T11:33:42.416321Z","shell.execute_reply":"2021-10-17T11:33:43.136516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"last_imgs = sample_person.sort_values(['mri', 'img_num']).groupby(\"mri\").last()\nlast_imgs","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:43.139243Z","iopub.execute_input":"2021-10-17T11:33:43.139580Z","iopub.status.idle":"2021-10-17T11:33:43.160293Z","shell.execute_reply.started":"2021-10-17T11:33:43.139537Z","shell.execute_reply":"2021-10-17T11:33:43.159660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig,axes = plt.subplots(nrows=2, ncols=2)\naxes= axes.flatten()\nfor i,ax in enumerate(axes):\n    ax.imshow(get_img_array(last_imgs.iloc[i]['path']), cmap='gray')\n    ax.set_axis_off()\n    ax.set_title(first_imgs.index[i])\n    #plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:43.161509Z","iopub.execute_input":"2021-10-17T11:33:43.161771Z","iopub.status.idle":"2021-10-17T11:33:43.658164Z","shell.execute_reply.started":"2021-10-17T11:33:43.161741Z","shell.execute_reply":"2021-10-17T11:33:43.657168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Data Distribution\n\n\n","metadata":{"execution":{"iopub.status.busy":"2021-10-01T18:51:20.245918Z","iopub.execute_input":"2021-10-01T18:51:20.246197Z","iopub.status.idle":"2021-10-01T18:51:20.252822Z","shell.execute_reply.started":"2021-10-01T18:51:20.246163Z","shell.execute_reply":"2021-10-01T18:51:20.25157Z"}}},{"cell_type":"markdown","source":"## Time-series Analyis of Single Person\nCross-setional analysis of brain","metadata":{}},{"cell_type":"code","source":"def compare_img_mri(mri_type, interval=10):\n    \n    select_img = sample_person[sample_person['mri'] == \"FLAIR\"].sort_values('img_num').reset_index()\n    \n    \n    interval_imgs = select_img[::interval]\n    \n    #print(select_img['mri'])\n    \n    fig,axes = plt.subplots(nrows=len(interval_imgs), ncols=6, figsize=(16,26))\n    axes= axes.flatten()\n    for i,ax in enumerate(axes):\n        ax.imshow(get_img_array(select_img.iloc[i]['path']), cmap='gray', aspect='auto')\n        ax.set_axis_off()\n        #ax.set_title(select_img.at[i, 'img_num'])","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:43.659898Z","iopub.execute_input":"2021-10-17T11:33:43.660194Z","iopub.status.idle":"2021-10-17T11:33:43.667657Z","shell.execute_reply.started":"2021-10-17T11:33:43.660163Z","shell.execute_reply":"2021-10-17T11:33:43.666603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_img_mri(mri_type=\"FLAIR\", interval=10)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:43.669212Z","iopub.execute_input":"2021-10-17T11:33:43.669486Z","iopub.status.idle":"2021-10-17T11:33:59.693794Z","shell.execute_reply.started":"2021-10-17T11:33:43.669439Z","shell.execute_reply":"2021-10-17T11:33:59.692880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_img_mri(mri_type=\"T1w\", interval=10)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:33:59.695540Z","iopub.execute_input":"2021-10-17T11:33:59.696069Z","iopub.status.idle":"2021-10-17T11:34:15.687540Z","shell.execute_reply.started":"2021-10-17T11:33:59.696013Z","shell.execute_reply":"2021-10-17T11:34:15.686645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_img_mri(mri_type=\"T1wCE\", interval=10)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:34:15.688813Z","iopub.execute_input":"2021-10-17T11:34:15.689112Z","iopub.status.idle":"2021-10-17T11:34:31.860502Z","shell.execute_reply.started":"2021-10-17T11:34:15.689080Z","shell.execute_reply":"2021-10-17T11:34:31.859808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compare_img_mri(mri_type=\"T2\", interval=10)","metadata":{"execution":{"iopub.status.busy":"2021-10-17T11:34:31.861470Z","iopub.execute_input":"2021-10-17T11:34:31.862410Z","iopub.status.idle":"2021-10-17T11:34:47.922880Z","shell.execute_reply.started":"2021-10-17T11:34:31.862362Z","shell.execute_reply":"2021-10-17T11:34:47.921888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> * types of scans similar between mri types, scan in multiple cross sectional layers\n> * Do we need all layers for predictions or is one shot in the middle enough?\n> * Which image has been used for predictions of the class?\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}