{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EXPLORATORY DATA ANALYSIS (EDA)\n\n**Objective:** predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to train and test your model to detect for the presence of MGMT promoter methylation.\n\n---\n### **Glioblastoma**\n\n[Glioblastoma](https://www.mayoclinic.org/diseases-conditions/glioblastoma/cdc-20350148) (GLB) is an aggresive type of cancer that occurs in the brain or spinalcord. It's formed from astrocytes (specialized glial cells that outnumber neurons by fivefold). Occurs more often in older adults, symptoms include worsening headaches, nausea, vomiting & seizures.\n\n<img src=\"https://www.mayoclinic.org/-/media/kcms/gbs/patient-consumer/images/2019/01/11/10/47/glioblastoma-8col-3802786-002-0.jpg\" width=\"250px\"/> <img src=\"https://media.springernature.com/m685/springer-static/image/art%3A10.1038%2Fs41593-019-0367-6/MediaObjects/41593_2019_367_Fig1_HTML.png\" width=\"350px\"/>\n\n**Diagnosis:** Neurological exam, Imaging tests, Biopsy.\n\n**Treatment:** Surgery to remove it, Radiation therapy, Chemotherapy, Tumor treating fields (TTF) therapy, Targeted drug therapy, Clinical trials & Supportive care.\n\n---\n### **MGMT**\n\nO6-methylguanine-DNA methyl-transferase ([MGMT](https://www.frontiersin.org/articles/10.3389/fonc.2019.01547/full)) it's an enzyme involved in DNA repair. Located in chromosome 10q26.3, this enzyme removes alkyl addicts from the O6 position of Guanine. [Methyl moiety](https://en.wikipedia.org/wiki/Moiety_(chemistry)) of the O6-methylguanine adduct is transferred to the MGMT protein, then it undergoes irreversible inhibition.\n\n<img src=\"https://media.springernature.com/lw785/springer-static/image/prt%3A978-3-540-29623-2%2F1/MediaObjects/978-3-540-29623-2_1_Part_Fig1-2580_HTML.jpg\" width=\"250px\"/>\n\nDefective MGMT results into persistence of the O6-methylguanine adduct, causing base misrepairing and mismatch repair futile. This leads to cell cycle arrest and apoptosis. In cases with **monosomy** of chromosome 10 methylation of the remaning allele completely blocks MGMT-mediated DNA repair.\n\n---\n### **MGMT in Glioma**\n\n[Methylation](https://www.nature.com/scitable/topicpage/the-role-of-methylation-in-gene-expression-1070/) of the MGMT gene promoter has been observed in approx. 50% of grade IV gliomas. MGMT gene [promoter methylation](https://oncologypro.esmo.org/education-library/factsheets-on-biomarkers/mgmt-promoter-methylation-in-glioma) has been investigated as a potential biomarker of sensitivity to alkylating chemotherapy.\n\n<img src=\"https://www.researchgate.net/profile/Walter-Taal/publication/275358605/figure/fig1/AS:418919660703747@1476889638840/Classification-of-glioma-in-types-and-WHO-grades.png\" width=\"500px\"/>\n\nMGMT promotor methylated glioblastoma is [**_likely to show less aggressive image features_**](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7410549/) than unmethylated glioblastoma (Suh, C. _et al._, 2018):\n\n* **Less edema:** [Swelling](https://my.clevelandclinic.org/health/diseases/12564-edema) caused by fluid trapped in the body's tissue, Glioma prognosis may be independently predicted by identifying the degree of peritumoral edema on MRI.\n* **Apparent diffusion coeficient (ADC):** [Measure of the magnitude of diffusion](https://radiopaedia.org/articles/apparent-diffusion-coefficient-1) (of water molecules) within tissue. Commonly calculated using MRI with diffusion-weighted imaging (DWI).\n* **Perfusion:** The process in which [blood is forced to flow](https://www.sciencedirect.com/topics/immunology-and-microbiology/perfusion) through a network of microscopic vessels with biological tissue, allowing exchange of oxygen and other molecules across semipermeable microvascular walls.\n    * GLB exhibits profound intratumoral heterogeneity in perfusion, low perfusion may induce treatment resistance. Therefore, imaging approaches that define low perfusion compartments are crucial for clinical management. \n\n---\n### **Data Description**\nThree structured cohorts: Training (Public), Validation (Public) and Testing (Private during and after competition).\n\nIn this dataset the MGMT promoter status is defined as a binary label: 0 for unmethylated and 1 for methylated.\n\nEach independent case has a dedicated folder idenfied by a 5-digit number, the folder is composed by a 4 sub-folders corresponding to each of the structural multi-parametric MRI ([mpMRI](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5832385/)). Differences in [MRI imaging methods](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/252843) by @dlbsabu:\n* T1-weighted pre-contrast (T1w) \n\nFat is depicted in white and water in black. The shape of the brain can be clearly seen and morphological abnormalities are easy to detect (Atrophy, tumors, etc.).\n* T1-weighted post-contrast (T1Gd)\n\nT1Gd is T1 imaging with contrast medium and is the method that best reflects the location, size, and shape of the mass.\n* T2-weighted (T2)\n\nWater is painted white and lesions appear white, suitable for lesion evaluation.\n* Fluid Attenuated Inversion Recovery (FLAIR)\n\nIn T2, the spinal fluid (water) is white and the lesion is also white, so you have to look for the white in the white, which is difficult to understand. FLAIR can be roughly thought of as T2, in which the water is also black, making it easier to find the lesion.\n\n---\n### **DICOM**\nThe data is given in [DICOM](https://www.dicomstandard.org/about-home) (Digital Imaging and Communications in Medicine) format. It's the international standard for medical images and related information (first publication in 1993). \n\n---\n### **Files**\n* train/ - folder containing the training files, with each top-level folder representing a subject. NOTE: There are some unexpected issues with the following three cases in the training dataset, participants can exclude the cases during training: [00109, 00123, 00709]. We have checked and confirmed that the testing dataset is free from such issues.\n* train_labels.csv - file containing the target MGMT_value for each subject in the training data (e.g. the presence of MGMT promoter methylation).\n* test/ - the test files, which use the same structure as train/; your task is to predict the MGMT_value for each subject in the test data. NOTE: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set.\n* sample_submission.csv - a sample submission file in the correct format\n\n---\n### **Evaluation**\nSubmissions are evaluated on the [area under the ROC curve between](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) the predicted probability and the observed target.\n\nFor each \"BraTS21ID\" in the test set, you must predict a probability for the target MGMT_value. The file should contain a header and have the following format:\n\n    BraTS21ID,MGMT_value\n    00001,0.5\n    00013,0.5\n    00015,0.5\n    etc.","metadata":{}},{"cell_type":"markdown","source":"---\n# **Loading data & visualization**\n\nCode from @ihelon 's notebok:\n\nhttps://www.kaggle.com/ihelon/brain-tumor-eda-with-animations-and-modeling","metadata":{}},{"cell_type":"markdown","source":"---\n### **Import Libraries**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport os\nimport json\nimport glob\nimport random\nimport collections\nimport cv2\n\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\n\nfrom matplotlib import animation, rc\nrc('animation', html='jshtml')","metadata":{"execution":{"iopub.status.busy":"2021-08-30T12:21:39.483619Z","iopub.execute_input":"2021-08-30T12:21:39.484236Z","iopub.status.idle":"2021-08-30T12:21:39.489633Z","shell.execute_reply.started":"2021-08-30T12:21:39.484192Z","shell.execute_reply":"2021-08-30T12:21:39.488922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n### **Loading & Visualization of Data**","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-30T12:21:39.491023Z","iopub.execute_input":"2021-08-30T12:21:39.491544Z","iopub.status.idle":"2021-08-30T12:21:39.540393Z","shell.execute_reply.started":"2021-08-30T12:21:39.491507Z","shell.execute_reply":"2021-08-30T12:21:39.539204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(4, 4))\nsns.countplot(data=train_df, x=\"MGMT_value\");\nprint(train_df['MGMT_value'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2021-08-30T12:21:39.542721Z","iopub.execute_input":"2021-08-30T12:21:39.543212Z","iopub.status.idle":"2021-08-30T12:21:39.705688Z","shell.execute_reply.started":"2021-08-30T12:21:39.543173Z","shell.execute_reply":"2021-08-30T12:21:39.704772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dicom(path):\n    dicom = pydicom.read_file(path)\n    data = dicom.pixel_array\n    data = data - np.min(data)\n    if np.max(data) != 0:\n        data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n\n\n\ndef visualize_sample(\n    brats21id, \n    slice_i,\n    mgmt_value,\n    types=(\"FLAIR\", \"T1w\", \"T1wCE\", \"T2w\")\n):\n    plt.figure(figsize=(10, 3))\n    patient_path = os.path.join(\n        \"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/\", \n        str(brats21id).zfill(5),\n    )\n    for i, t in enumerate(types, 1):\n        t_paths = sorted(\n            glob.glob(os.path.join(patient_path, t, \"*\")), \n            key=lambda x: int(x[:-4].split(\"-\")[-1]),\n        )\n        data = load_dicom(t_paths[int(len(t_paths) * slice_i)])\n        plt.subplot(1, 4, i)\n        plt.imshow(data, cmap=\"gray\")\n        plt.title(f\"{t}\", fontsize=10)\n        plt.axis(\"off\")\n    \n    plt.suptitle(f\"MGMT_value: {mgmt_value}\", fontsize=10)\n    plt.show()\n    ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:26:17.752238Z","iopub.execute_input":"2021-08-30T12:26:17.752659Z","iopub.status.idle":"2021-08-30T12:26:17.763295Z","shell.execute_reply.started":"2021-08-30T12:26:17.752619Z","shell.execute_reply":"2021-08-30T12:26:17.762439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in random.sample(range(train_df.shape[0]), 4):\n    _brats21id = train_df.iloc[i][\"BraTS21ID\"]\n    _mgmt_value = train_df.iloc[i][\"MGMT_value\"]\n    visualize_sample(brats21id=_brats21id, mgmt_value=_mgmt_value, slice_i=0.5)","metadata":{"execution":{"iopub.status.busy":"2021-08-30T12:27:01.857448Z","iopub.execute_input":"2021-08-30T12:27:01.857810Z","iopub.status.idle":"2021-08-30T12:27:03.737662Z","shell.execute_reply.started":"2021-08-30T12:27:01.857763Z","shell.execute_reply":"2021-08-30T12:27:03.736771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## **Animation**","metadata":{}},{"cell_type":"code","source":"def create_animation(ims):\n    fig = plt.figure(figsize=(6, 6))\n    plt.axis('off')\n    im = plt.imshow(ims[0], cmap=\"gray\")\n    plt.close()\n\n    def animate_func(i):\n        im.set_array(ims[i])\n        return im\n\n    return animation.FuncAnimation(fig, animate_func, frames = len(ims), interval = 1000//24)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:49:23.304986Z","iopub.execute_input":"2021-08-30T12:49:23.305364Z","iopub.status.idle":"2021-08-30T12:49:23.311679Z","shell.execute_reply.started":"2021-08-30T12:49:23.305312Z","shell.execute_reply":"2021-08-30T12:49:23.310606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dicom_line(path):\n    t_paths = sorted(\n        glob.glob(os.path.join(path, \"*\")), \n        key=lambda x: int(x[:-4].split(\"-\")[-1]),\n    )\n    images = []\n    for filename in t_paths:\n        data = load_dicom(filename)\n        if data.max() == 0:\n            continue\n        images.append(data)\n        \n    return images","metadata":{"execution":{"iopub.status.busy":"2021-08-30T12:28:14.145142Z","iopub.execute_input":"2021-08-30T12:28:14.145512Z","iopub.status.idle":"2021-08-30T12:28:14.151042Z","shell.execute_reply.started":"2021-08-30T12:28:14.145470Z","shell.execute_reply":"2021-08-30T12:28:14.150056Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Case 00000, FLAIR**\nFluid Attenuated Inversion Recovery (FLAIR)\n\nIn T2, the spinal fluid (water) is white and the lesion is also white, so you have to look for the white in the white, which is difficult to understand. FLAIR can be roughly thought of as T2, in which the water is also black, making it easier to find the lesion.","metadata":{}},{"cell_type":"code","source":"images = load_dicom_line(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/FLAIR\")\ncreate_animation(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:49:26.051667Z","iopub.execute_input":"2021-08-30T12:49:26.052148Z","iopub.status.idle":"2021-08-30T12:49:43.623781Z","shell.execute_reply.started":"2021-08-30T12:49:26.052118Z","shell.execute_reply":"2021-08-30T12:49:43.618731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Case 00000, T1w**\n\nT1-weighted pre-contrast (T1w) \n\nFat is depicted in white and water in black. The shape of the brain can be clearly seen and morphological abnormalities are easy to detect (Atrophy, tumors, etc.).\n","metadata":{}},{"cell_type":"code","source":"images = load_dicom_line(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/T1w\")\ncreate_animation(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:50:57.858268Z","iopub.execute_input":"2021-08-30T12:50:57.858807Z","iopub.status.idle":"2021-08-30T12:50:59.611781Z","shell.execute_reply.started":"2021-08-30T12:50:57.858766Z","shell.execute_reply":"2021-08-30T12:50:59.611105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Case 00000, T1wCE**\n\nT1-weighted post-contrast (T1Gd)\n\nT1Gd is T1 imaging with contrast medium and is the method that best reflects the location, size, and shape of the mass.","metadata":{}},{"cell_type":"code","source":"images = load_dicom_line(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/T1wCE\")\ncreate_animation(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:51:06.593594Z","iopub.execute_input":"2021-08-30T12:51:06.594151Z","iopub.status.idle":"2021-08-30T12:51:12.179324Z","shell.execute_reply.started":"2021-08-30T12:51:06.594065Z","shell.execute_reply":"2021-08-30T12:51:12.177962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Case 00000, T2w**\n\nT2-weighted (T2)\n\nWater is painted white and lesions appear white, suitable for lesion evaluation.","metadata":{}},{"cell_type":"code","source":"images = load_dicom_line(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train/00000/T2w\")\ncreate_animation(images)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-08-30T12:51:17.034805Z","iopub.execute_input":"2021-08-30T12:51:17.035304Z","iopub.status.idle":"2021-08-30T12:51:33.735165Z","shell.execute_reply.started":"2021-08-30T12:51:17.035263Z","shell.execute_reply":"2021-08-30T12:51:33.730915Z"},"trusted":true},"execution_count":null,"outputs":[]}]}