{"cells":[{"metadata":{},"cell_type":"markdown","source":"# A. Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"\n
HuBMAP: Hacking the Kidney - EDA
\n \n# 1. Introduction\n\n### 1.1 What is HuBMAP?\nThe [Human BioMolecular Atlas Program](https://hubmapconsortium.org/what-is-hubmap/) is a major endeavour working to catalyze the development of a framework for mapping the human body at single cell resolution.One component of this overarching goal is to identify medically relevant functional tissue units (FTUs) within whole slide microscopy images of human tissues. Once these FTUs are detected, information on size, shape, variability in number and location within the tissue samples can be used to help build a spatially accurate and semantically explicit model of the human body."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import HTML\nHTML('
')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 What is HuBMAP:Hacking the Kidney Competition?\n- This compeition starts by mapping the human kidney at single cell resolution. We will detect the [functional tissue units](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4126067/)(FTU) accross different tissue preparation pipelines.By this we will implement a successful and robust glomeruli FTU detector.\n- We also have the opportunity to present our findings to a panel of judges of this competition for additional consideration\n\n\n### 1.3 What is FTU?\n- An [FTU](https://pubmed.ncbi.nlm.nih.gov/24103658/) is defined as a “three-dimensional block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block”.Bernard de Bono and his team coined the term “functional tissue unit” in 2013 as: “...a three-dimensional block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block” [(de Bono, 2013)](https://www.ncbi.nlm.nih.gov/pubmed/24103658) One example of an FTU is the glomerulus found in the outer layer of kidney tissue known as the cortex, which in humans has an area of about 800 mm2 and average depth of about 9 mm [(Mounier-Vehier, 2002)](https://doi.org/10.1046/j.1523-1755.2002.00167.x). \n- Glomeruli consist of capillaries that facilitate filtration of waste products out of blood. Normal glomeruli typically range from 100-350 μm in diameter with a roughly spherical shape [(Kannan, 2019)](https://www.kireports.org/article/S2468-0249(19)30155-X/abstract). Refer to Figure 1 for a zoom sequence from the human body to single-cell level for kidney. Figure 2 highlights the cortex region of a tissue sample in green. Glomeruli contain four cell types: Parietal epithelial cells that form Bowman’s capsule, podocytes cover the outer layer of the filtration barrier, fenestrated endothelial cells that are coated with a glycolipid and glycoprotein matrix called glycocalyx that are in direct contact with blood and mesangial cells that occupy the space between the capillary blood vessel loops and are stained by the colorimetric histological stain called Periodic acid-Schiff (PAS) stain [(Vaughan, 2008)](https://doi.org/10.1681/ASN.2007040471). PAS stains polysaccharides (complex sugars like glycogen) such as those found in and around the glomeruli making it a favored stain for delineating them in tissue sections [(Agarwal, 2013)](https://doi.org/10.4103/0971-4065.114462). The periodic acid oxidizes the sugars to expose aldehyde free tips of the broken monosaccharide rings that react with the Schiff reagent to give a magenta color. Figure 3 is a light microscopy image that contains a glomerulus from a subsection of human kidney tissue stained with a PAS. One nucleus (spherical in shape) per cell, containing the genetic material for a cell (its chromosomes composed of nucleic acids) is stained dark bluish purple in the PAS stained images. The magenta stained regions in PAS stained tissue are the stained polysaccharides.\n\n### 1.4 What do we need to do?\n- We will develop segmentation algorithms that identify glomeruli in the PAS stained microscopy data. Yes, external data is allowed and/or pre-trained machine learning models in support of FTU segmentation.\n- The leaderboard of this competition is calculated with approximately 42% of the test data.The final results will be based on the other 58%, so the final standings may be different.\n\n### 1.5 Metric: Dice Coefficient.\n\n\n\n- Image Credits: https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient\n- The evaluation metric of this competition is Dice Coefficient. \nThe Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:\n \n
$\\Large \\frac{2*|X ∩ Y|}{|X|+|Y|}$
\n\nwhere X is the predicted set of pixels and Y is the ground truth.\n\n- Dice coefficient, is a statistical tool which measures the similarity between two sets of data. This index has become arguably the most broadly used tool in the validation of image segmentation algorithms created with AI, but it is a much more general concept which can be applied sets of data for a variety of applications including NLP.\n- Read more about it on the [Evaluation Page](https://www.kaggle.com/c/hubmap-kidney-segmentation/overview/supervised-ml-evaluation)\n\n### 1.6 Relation between Dice Coefficient and Jaccard Score.\n\n
$\\Large D = \\frac{2*|X ∩ Y|}{|X|+|Y|}$
\n \n
$\\Large J = \\frac{|X ∩ Y|}{|X|+|Y|-|X ∩ Y|}$
\n \n
$\\Large D = \\frac{2J}{J+1}$
\n \n
$\\Large J = \\frac{D}{2-D}$
\n\n### 1.7 What has been achieved so far?\n\n### *Automatic glomerular identification and quantification of histological phenotypes using image analysis and machine learning (Sheehan and Korstanje 2018)*\n\nCurrent methods of scoring histological kidney samples, specifically glomeruli, do not allow for collection of quantitative data in a high-throughput and consistent manner. Neither untrained individuals nor computers are presently capable of identifying glomerular features, so expert pathologists must do the identification and score using a categorical matrix, complicating statistical analysis. Critical information regarding overall health and physiology is encoded in these samples. Rapid comprehensive histological scoring could be used, in combination with other physiological measures, to significantly advance renal research. Therefore, we used machine learning to develop a high-throughput method to automatically identify and collect quantitative data from glomeruli. Our method requires minimal human interaction between steps and provides quantifiable data independent of user bias. The method uses free existing software and is usable without extensive image analysis training. Validation of the classifier and feature scores in mice is highlighted in this work and shows the power of applying this method in murine research. Preliminary results indicate that the method can be applied to data sets from different species after training on relevant data, allowing for fast glomerular identification and quantitative measurements of glomerular features. Validation of the classifier and feature scores are highlighted in this work and show the power of applying this method. The resulting data are free from user bias. Continuous data, such that statistical analysis can be performed, allows for more precise and comprehensive interrogation of samples. These data can then be combined with other physiological data to broaden our overall understanding of renal function.\n\n### *Glomerulus Classification and Detection Based on Convolutional Neural Networks (Gallego et al. 2018)*\n\nGlomerulus classification and detection in kidney tissue segments are key processes in nephropathology used for the correct diagnosis of the diseases. In this paper, we deal with the challenge of automating Glomerulus classification and detection from digitized kidney slide segments using a deep learning framework. The proposed method applies Convolutional Neural Networks (CNNs) between two classes: Glomerulus and Non-Glomerulus, to detect the image segments belonging to Glomerulus regions. We configure the CNN with the public pre-trained AlexNet model and adapt it to our system by learning from Glomerulus and Non-Glomerulus regions extracted from training slides. Once the model is trained, labeling is performed by applying the CNN classification to the image blocks under analysis. The results of the method indicate that this technique is suitable for correct Glomerulus detection in Whole Slide Images (WSI), showing robustness while reducing false positive and false negative detections.\n\n### *Region-Based Convolutional Neural Nets for Localization of Glomeruli in Trichrome-Stained Whole Kidney Sections (Bukowy et al. 2018)*\n\nBackground Histologic examination of fixed renal tissue is widely used to assess morphology and the progression of disease. Commonly reported metrics include glomerular number and injury. However, characterization of renal histology is a time-consuming and user-dependent process. To accelerate and improve the process, we have developed a glomerular localization pipeline for trichrome-stained kidney sections using a machine learning image classification algorithm.\n\nMethods We prepared 4-μm slices of kidneys from rats of various genetic backgrounds that were subjected to different experimental protocols and mounted the slices on glass slides. All sections used in this analysis were trichrome stained and imaged in bright field at a minimum resolution of 0.92 μm per pixel. The training and test datasets for the algorithm comprised 74 and 13 whole renal sections, respectively, totaling over 28,000 glomeruli manually localized. Additionally, because this localizer will be ultimately used for automated assessment of glomerular injury, we assessed bias of the localizer for preferentially identifying healthy or damaged glomeruli.\n\nResults Localizer performance achieved an average precision and recall of 96.94% and 96.79%, respectively, on whole kidney sections without evidence of bias for or against glomerular injury or the need for manual preprocessing.\n\nConclusions This study presents a novel and robust application of convolutional neural nets for the localization of glomeruli in healthy and damaged trichrome-stained whole-renal section mounts and lays the groundwork for automated glomerular injury scoring.\n\n### *Segmentation of Glomeruli Within Trichrome Images Using Deep Learning (Kannan et al. 2019)*\n\n\n Introduction The number of glomeruli and glomerulosclerosis evaluated on kidney biopsy slides constitute standard components of a renal pathology report. Prevailing methods for glomerular assessment remain manual, labor intensive, and nonstandardized. We developed a deep learning framework to accurately identify and segment glomeruli from digitized images of human kidney biopsies. Methods Trichrome-stained images (n = 275) from renal biopsies of 171 patients with chronic kidney disease treated at the Boston Medical Center from 2009 to 2012 were analyzed. A sliding window operation was defined to crop each original image to smaller images. Each cropped image was then evaluated by at least 3 experts into 3 categories: (i) no glomerulus, (ii) normal or partially sclerosed (NPS) glomerulus, and (iii) globally sclerosed (GS) glomerulus. This led to identification of 751 unique images representing nonglomerular regions, 611 images with NPS glomeruli, and 134 images with GS glomeruli. A convolutional neural network (CNN) was trained with cropped images as inputs and corresponding labels as output. Using this model, an image processing routine was developed to scan the test images to segment the GS glomeruli. Results The CNN model was able to accurately discriminate nonglomerular images from NPS and GS images (performance on test data: Accuracy: 92.67% ± 2.02% and Kappa: 0.8681 ± 0.0392). The segmentation model that was based on the CNN multilabel classifier accurately marked the GS glomeruli on the test data (Matthews correlation coefficient = 0.628). Conclusion This work demonstrates the power of deep learning for assessing complex histologic structures from digitized human kidney biopsies."},{"metadata":{},"cell_type":"markdown","source":"# 2. Importing the necessary libraries📗 "},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import os\nfrom os import listdir\nimport pandas as pd\nimport numpy as np\nimport glob\nimport tqdm\nfrom typing import Dict\nimport matplotlib.pyplot as plt\nimport pandas_profiling as pdp\nimport json\n%matplotlib inline\nimport shapely.geometry as sg\nimport shapely.ops as so\nimport zipfile\nimport cv2\n\n#plotly\n!pip install chart_studio\nimport plotly.express as px\nimport plotly.offline as pyo\nimport chart_studio.plotly as py\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\nimport cufflinks\ncufflinks.go_offline()\ncufflinks.set_config_file(world_readable=True, theme='pearl')\n\n#seaborn\nimport seaborn as sns\n\n#color\nfrom colorama import Fore, Back, Style\n\n#networkx\nimport networkx as nx\n\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\n\n#tifffile\nfrom PIL import Image\nimport tifffile as tiff\nimport cv2\nfrom tqdm.notebook import tqdm\nimport zipfile\n\n# Suppress warnings \nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Settings for pretty nice plots\nplt.style.use('fivethirtyeight')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Reading the train.csv 📚"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# List files available\nlist(os.listdir(\"../input/hubmap-kidney-segmentation\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"IMAGE_PATH = \"../input/hubmap-kidney-segmentation/\"\n\ntrain_df = pd.read_csv('../input/hubmap-kidney-segmentation/train.csv')\nhubmap_df = pd.read_csv('../input/hubmap-kidney-segmentation/HuBMAP-20-dataset_information.csv')\ntest_df = pd.read_csv('../input/hubmap-kidney-segmentation/sample_submission.csv')\n\nprint(Fore.YELLOW + 'Training data shape: ',Style.RESET_ALL,train_df.shape)\nprint(Fore.YELLOW + 'HubMap data shape: ',Style.RESET_ALL,hubmap_df.shape)\nprint(Fore.YELLOW + 'Test data shape: ',Style.RESET_ALL,test_df.shape)\n\ntrain_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df.groupby(['race']).count()['sex'].to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4.Basic Data Exploration 🏕️ "},{"metadata":{},"cell_type":"markdown","source":"## General Information"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Null values and Data types\nprint(Fore.YELLOW + 'Train Set !!',Style.RESET_ALL)\nprint(train_df.info())\nprint('-------------')\nprint(Fore.BLUE + 'Test Set !!',Style.RESET_ALL)\nprint(test_df.info())\nprint('-------------')\nprint(Fore.GREEN + 'HuBMAP Set !!',Style.RESET_ALL)\nprint(hubmap_df.info())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing values"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 3 missing values in hubmap_df and no missing values in train_df"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"! ls ../input/hubmap-kidney-segmentation/train/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are `16` json files and `8` tiff files"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Total number of Patient in the dataset(train+test)\n\nprint(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['id'].count())\nprint(Fore.BLUE +\"Total Patients in Test set: \",Style.RESET_ALL,test_df['id'].count())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`5` : Patients in Test Set\n\n`8` : Patients in Train Set"},{"metadata":{},"cell_type":"markdown","source":"## Unique Patients(Ids)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(Fore.YELLOW + \"The total patient ids are\",Style.RESET_ALL,f\"{train_df['id'].count()},\", Fore.BLUE + \"from those the unique ids are\", Style.RESET_ALL, f\"{train_df['id'].value_counts().shape[0]}.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_patient_ids = set(train_df['id'].unique())\ntest_patient_ids = set(test_df['id'].unique())\n\ntrain_patient_ids.intersection(test_patient_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see `none` patients in test set that can be found in train set."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"columns = train_df.keys()\ncolumns = list(columns)\nprint(columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Patient Counts"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"train_df['id'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['id'].value_counts().max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In train and test sets, we can see one Patient and it mean Patient id is unique"},{"metadata":{},"cell_type":"markdown","source":"## Number of Patients and Files in Training Images Folder"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"files = folders = 0\nfolder_names = {'a'}\nfolder_names.remove('a')\n\npath = \"../input/hubmap-kidney-segmentation/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n folder_names.add(filenames[j][:9])\n folders = len(folder_names)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{files:,}',Style.RESET_ALL,\"files, \" + Fore.BLUE + f'{folders:,}',Style.RESET_ALL ,'patients/ids')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"files = folders = 0\nfolder_names = {'a'}\nfolder_names.remove('a')\n\npath = \"../input/hubmap-kidney-segmentation/test\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n folder_names.add(filenames[j][:9])\n folders = len(folder_names)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{files:,}',Style.RESET_ALL,\"files, \" + Fore.BLUE + f'{folders:,}',Style.RESET_ALL ,'patients/ids')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## JSON FILES COUNT"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = folders = 0\nfolder_names = {'a'}\nfolder_names.remove('a')\n\npath = \"../input/hubmap-kidney-segmentation/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json':\n folder_names.add(filenames[j][:9])\n folders = len(folder_names)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{folders:,}',Style.RESET_ALL,\" json-files\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = folders = 0\nfolder_names = {'a'}\nfolder_names.remove('a')\n\npath = \"../input/hubmap-kidney-segmentation/test\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json':\n folder_names.add(filenames[j][:9])\n folders = len(folder_names)\n#print(Fore.YELLOW +\"Total Patients in Train set: \",Style.RESET_ALL,train_df['Patient'].count())\nprint(Fore.YELLOW +f'{folders:,}',Style.RESET_ALL,\" json-files\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading Normal json files"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = 0\npath = \"../input/hubmap-kidney-segmentation/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json' and len(filenames[j])<15:\n df = pd.read_json(f'../input/hubmap-kidney-segmentation/train/{filenames[j]}')\n print(Fore.RED + f'{df.shape[0]}',Style.RESET_ALL, \"rows and \"+ Fore.GREEN + f'{df.shape[1]}',Style.RESET_ALL,f\" columns in the {filenames[j][:-5]} json file\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are no normal json files in the test set."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = 0\npath = \"../input/hubmap-kidney-segmentation\"\ni = 0\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n if i==0:\n i += 1\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json' and len(filenames[j])<15:\n df = pd.read_json(f'../input/hubmap-kidney-segmentation/{filenames[j]}')\n print(Fore.RED + f'{df.shape[0]}',Style.RESET_ALL, \"rows and \"+ Fore.GREEN + f'{df.shape[1]}',Style.RESET_ALL,f\" columns in the {filenames[j][:-5]} json file\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading anatomical-structure json files"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = 0\npath = \"../input/hubmap-kidney-segmentation/train\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json' and len(filenames[j])>15:\n df = pd.read_json(f'../input/hubmap-kidney-segmentation/train/{filenames[j]}')\n print(Fore.RED + f'{df.shape[0]}',Style.RESET_ALL, \"rows and \"+ Fore.GREEN + f'{df.shape[1]}',Style.RESET_ALL,f\" columns in the {filenames[j][:-5]} json file\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = 0\npath = \"../input/hubmap-kidney-segmentation/test\"\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json' and len(filenames[j])>15:\n df = pd.read_json(f'../input/hubmap-kidney-segmentation/test/{filenames[j]}')\n print(Fore.RED + f'{df.shape[0]}',Style.RESET_ALL, \"rows and \"+ Fore.GREEN + f'{df.shape[1]}',Style.RESET_ALL,f\" columns in the {filenames[j][:-5]} json file\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"files = 0\npath = \"../input/hubmap-kidney-segmentation\"\ni = 0\n\nfor _, dirnames, filenames in os.walk(path):\n # ^ this idiom means \"we won't be using this value\"\n if i==0:\n i += 1\n files += len(filenames)\n for j in range(files):\n if filenames[j][-4:]=='json' and len(filenames[j])>15:\n df = pd.read_json(f'../input/hubmap-kidney-segmentation/{filenames[j]}')\n print(Fore.RED + f'{df.shape[0]}',Style.RESET_ALL, \"rows and \"+ Fore.GREEN + f'{df.shape[1]}',Style.RESET_ALL,f\" columns in the {filenames[j][:-5]} json file\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see some encoding column over there. This encoding column is RLE-encoded representation of the mask for the objects in the image."},{"metadata":{},"cell_type":"markdown","source":"## What is this RLE Encoding?"},{"metadata":{},"cell_type":"markdown","source":"[Run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding) (RLE) is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs. Consider, for example, simple graphic images such as icons, line drawings, Conway's Game of Life, and animations. It is not useful with files that don't have many runs as it could greatly increase the file size. "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"## Sample code for RLE Encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Perform Run Length Encoding (RLE) data compression algorithm on string\ndef encode(s):\n\n encoding = \"\" \n i = 0\n while i < len(s):\n # count occurrences of character at index i\n count = 1\n\n while i + 1 < len(s) and s[i] == s[i + 1]:\n count = count + 1\n i = i + 1\n encoding += str(count) + s[i]\n i = i + 1\n\n return encoding\ns = \"ABBCCCD\"\nprint(encode(s))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Advantages and Disadvantages Of RLE:"},{"metadata":{},"cell_type":"markdown","source":"This [algorithm](https://www.prepressure.com/library/compression-algorithm/rle) is very easy to implement and does not require much CPU horsepower. RLE compression is only efficient with files that contain lots of repetitive data. These can be text files if they contain lots of spaces for indenting but line-art images that contain large white or black areas are far more suitable. Computer-generated color images (e.g. architectural drawings) can also give fair compression ratios. And this is used to compress Tiff and Pdf files."},{"metadata":{},"cell_type":"markdown","source":"## How Run Length Encoding is used Here"},{"metadata":{},"cell_type":"markdown","source":"In order to reduce the submission file size, we must submit segmentation results using run-length encoding on the pixel values. That is, instead of submitting an exhaustive list of indices for your segmentation, you will submit pairs of values that contain a start position and a run length. E.g. '0 3' implies starting at pixel 0 and running a total of 3 pixels (0,1,2). The competition format requires a space delimited list of pairs. For example, '0 3 10 5' implies pixels 0,1,2, and 10,11,12,13,14 are to be included in the mask. The metric checks that the pairs are sorted, positive, and the decoded pixel values are not duplicated. The pixels are numbered from top to bottom, then left to right: 0 is pixel (0,0), 1 is pixel (1,0), and 2 is pixel (2,0) etc., see Figure 1.\n\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"# 5. Data Exploration in Details 🎠"},{"metadata":{},"cell_type":"markdown","source":"Getting the columns in hubmap_df"},{"metadata":{"trusted":true},"cell_type":"code","source":"hubmap_df[\"split\"] = \"test\"\nhubmap_df.loc[hubmap_df[\"image_file\"].isin(os.listdir(os.path.join(IMAGE_PATH, \"train\"))), \"split\"] = \"train\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hubmap_df.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Parallel Category Diagram"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"parallel_diagram = hubmap_df[['patient_number','age','sex','race','percent_cortex']]\nfig = px.parallel_categories(parallel_diagram, color_continuous_scale=px.colors.sequential.Inferno)\nfig.update_layout(title='Parallel category diagram 1 on hubmap set')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"parallel_diagram = hubmap_df[['patient_number','age','sex','race','percent_medulla']]\nfig = px.parallel_categories(parallel_diagram, color_continuous_scale=px.colors.sequential.Inferno)\nfig.update_layout(title='Parallel category diagram 2 on hubmap set')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def dist(df,column,color):\n sns.distplot(df[column],label=column,color=color)\n plt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the 'race' columns"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['race','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['race','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='blue',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the race column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['race','split'])['race'].count().reset_index(name = 'counts')\ndf['race_split'] = df['race']+'_'+df['split']\nfig = px.pie(df, values='counts', names='race_split', title='Patient Race Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['race','split'])['race'].count().reset_index(name = 'counts')\nplt.figure(figsize=(10,10))\ndist(df,\"counts\",\"red\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"white = hubmap_df[hubmap_df['race']=='White']\nblack = hubmap_df[hubmap_df['race']=='Black or African American']\ncount_white = white['split'].value_counts().reset_index()\ncount_black = black['split'].value_counts().reset_index()\npie_white = go.Pie(labels=count_white['index'],values=count_white['split'],name=\"White\",hole=0.4,domain={'x': [0,0.46]})\npie_black = go.Pie(labels=count_black['index'],values=count_black['split'],name=\"Black or African American\",hole=0.5,domain={'x': [0.52,1]})\nlayout = dict(title = 'Race', font=dict(size=10), legend=dict(orientation=\"h\"),\n annotations = [dict(x=0.2, y=0.5, text='White', showarrow=False, font=dict(size=20)),\n dict(x=0.8, y=0.5, text='Black or African American', showarrow=False, font=dict(size=20)) ])\n\nfig = dict(data=[pie_white, pie_black], layout=layout)\npyo.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`9` : White - `5` : Train, `4` : Test\n\n`4` : Black or African American `3` : Train, `1` : Test"},{"metadata":{},"cell_type":"markdown","source":"### Network Graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (20, 12))\nfor i, t in enumerate(hubmap_df['race'].unique()):\n hubmap_df_type = hubmap_df.loc[hubmap_df['race'] == t]\n bad_weight = list(hubmap_df_type['weight_kilograms'].value_counts(normalize=True)[hubmap_df_type['weight_kilograms'].value_counts(normalize=True) < 0.01].index)\n bad_height = list(hubmap_df_type['height_centimeters'].value_counts(normalize=True)[hubmap_df_type['height_centimeters'].value_counts(normalize=True) < 0.01].index)\n bad_mes = list(set(bad_weight + bad_height))\n hubmap_df_type = hubmap_df_type.loc[(hubmap_df_type['weight_kilograms'].isin(bad_weight) == False) & (hubmap_df_type['height_centimeters'].isin(bad_height) == False)]\n G = nx.from_pandas_edgelist(hubmap_df_type, 'weight_kilograms', 'height_centimeters', ['bmi_kg/m^2'])\n plt.subplot(2, 1, i + 1);\n nx.draw(G, with_labels=True);\n plt.title(f'Network Graph for race {t}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Patient Number Columns"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['patient_number','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['patient_number','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='red',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the patient_number column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['patient_number','split'])['patient_number'].count().reset_index(name = 'counts')\ndf['patient_number_split'] = df['patient_number'].astype('str')+'_'+df['split']\nfig = px.pie(df, values='counts', names='patient_number_split', title='Patient Number Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Network Graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (20, 12))\nfor i, t in enumerate(hubmap_df['patient_number'].unique()):\n hubmap_df_type = hubmap_df.loc[hubmap_df['patient_number'] == t]\n bad_weight = list(hubmap_df_type['weight_kilograms'].value_counts(normalize=True)[hubmap_df_type['weight_kilograms'].value_counts(normalize=True) < 0.01].index)\n bad_height = list(hubmap_df_type['height_centimeters'].value_counts(normalize=True)[hubmap_df_type['height_centimeters'].value_counts(normalize=True) < 0.01].index)\n bad_mes = list(set(bad_weight + bad_height))\n hubmap_df_type = hubmap_df_type.loc[(hubmap_df_type['weight_kilograms'].isin(bad_weight) == False) & (hubmap_df_type['height_centimeters'].isin(bad_height) == False)]\n G = nx.from_pandas_edgelist(hubmap_df_type, 'weight_kilograms', 'height_centimeters', ['bmi_kg/m^2'])\n plt.subplot(5, 2, i + 1);\n nx.draw(G, with_labels=True);\n plt.title(f'Network Graph for patient_number {t}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Ethinicity Column"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['ethnicity','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['ethnicity','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='green',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Ethnicity column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['ethnicity','split'])['ethnicity'].count().reset_index(name = 'counts')\ndf['ethnicity_split'] = df['ethnicity']+'_'+df['split']\nfig = px.pie(df, values='counts', names='ethnicity_split', title='Patient Ethnicity Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the sex columns"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"hubmap_df[['sex','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['sex','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='yellow',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Sex column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['sex','split'])['sex'].count().reset_index(name = 'counts')\ndf['sex_split'] = df['sex']+'_'+df['split']\nfig = px.pie(df, values='counts', names='sex_split', title='Patient Gender Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"male = hubmap_df[hubmap_df['sex']=='Male']\nfemale = hubmap_df[hubmap_df['sex']== 'Female']\ncount_male = male['split'].value_counts().reset_index()\ncount_female = female['split'].value_counts().reset_index()\npie_male = go.Pie(labels=count_male['index'],values=count_male['split'],name=\"Male\",hole=0.4,domain={'x': [0,0.46]})\npie_female = go.Pie(labels=count_female['index'],values=count_female['split'],name=\"Female\",hole=0.5,domain={'x': [0.52,1]})\nlayout = dict(title = 'Sex', font=dict(size=10), legend=dict(orientation=\"h\"),\n annotations = [dict(x=0.2, y=0.5, text='Male', showarrow=False, font=dict(size=20)),\n dict(x=0.8, y=0.5, text='Female', showarrow=False, font=dict(size=20)) ])\n\nfig = dict(data=[pie_male, pie_female], layout=layout)\npyo.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the age columns"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"hubmap_df[['age','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['age','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='orange',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Age column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['age','split'])['age'].count().reset_index(name = 'counts')\ndf['age_split'] = df['age'].astype('str')+'_'+df['split']\nfig = px.pie(df, values='counts', names='age_split', title='Patient Ages Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Laterality Column"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"hubmap_df[['laterality','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['laterality','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='purple',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Laterality column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['laterality','split'])['laterality'].count().reset_index(name = 'counts')\ndf['laterality_split'] = df['laterality']+'_'+df['split']\nfig = px.pie(df, values='counts', names='laterality_split', title='Patient Laterality Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Percent Medulla Column"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"hubmap_df[['percent_medulla','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['percent_medulla','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='pink',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Medulla column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['percent_medulla','split'])['percent_medulla'].count().reset_index(name = 'counts')\ndf['percent_medulla_split'] = df['percent_medulla'].astype('str')+'_'+df['split']\nfig = px.pie(df, values='counts', names='percent_medulla_split', title='Patient Percent_Medulla Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the Percent Cortex column"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"hubmap_df[['percent_cortex','split']].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hubmap_df[['percent_cortex','split']].value_counts().iplot(kind='bar',\n yTitle='Counts', \n linecolor='black', \n opacity=0.7,\n color='cyan',\n theme='pearl',\n bargap=0.5,\n gridcolor='white',\n title='Distribution of the Medulla column in the HuBMAP-20 Set')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['percent_cortex','split'])['percent_cortex'].count().reset_index(name = 'counts')\ndf['percent_cortex_split'] = df['percent_cortex'].astype('str')+'_'+df['split']\nfig = px.pie(df, values='counts', names='percent_cortex_split', title='Patient Percent_Cortex Count')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution Of Age Over Race"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = hubmap_df\nfig = px.violin(df, y='age', x='race', box=True, color='sex', points=\"all\",\n hover_data=hubmap_df.columns)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nsns.kdeplot(df.loc[df['race'] == 'White', 'age'], label = 'White',shade=True)\nsns.kdeplot(df.loc[df['race'] == 'Black or African American', 'age'], label = 'Black or African American',shade=True)\n\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = df['age'], y = df['race'], palette = 'Reds')\nax.set_xlabel(xlabel = 'age', fontsize = 15)\nax.set_ylabel(ylabel = 'race', fontsize = 15)\nax.set_title(label = 'Distribution of age over race', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['age','race'])['age'].count().reset_index(name = 'counts')\ndf['age_race'] = df['age'].astype('str')+'_'+df['race']\nfig = px.pie(df, values='counts', names='age_race', title='Distribution Of Age Over Race')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution of race over Medulla"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = hubmap_df\nfig = px.violin(df, x='race', y='percent_medulla', box=True, color='sex', points=\"all\",\n hover_data=hubmap_df.columns)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\nax = sns.violinplot(x = df['race'], y = df['percent_medulla'], palette = 'Reds')\nax.set_xlabel(xlabel = 'race', fontsize = 15)\nax.set_ylabel(ylabel = 'percent_medulla', fontsize = 15)\nax.set_title(label = 'Distribution of race over medulla', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['race','percent_medulla'])['race'].count().reset_index(name = 'counts')\ndf['race_percent_medulla'] = df['race']+'_'+df['percent_medulla'].astype('str')\nfig = px.pie(df, values='counts', names='race_percent_medulla', title='Distribution Of Race Over Percent Medulla')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distiribution of age over percent medulla"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = hubmap_df\nplt.figure(figsize=(16, 6))\nsns.kdeplot(df.loc[df['percent_medulla'] == 20, 'age'], label = 20,shade=True)\nsns.kdeplot(df.loc[df['percent_medulla'] == 25, 'age'], label = 25,shade=True)\nsns.kdeplot(df.loc[df['percent_medulla'] == 35, 'age'], label = 35,shade=True)\nsns.kdeplot(df.loc[df['percent_medulla'] == 45, 'age'], label = 45,shade=True)\n\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['age','percent_medulla'])['age'].count().reset_index(name = 'counts')\ndf['age_percent_medulla'] = df['age'].astype('str')+'_'+df['percent_medulla'].astype('str')\nfig = px.pie(df, values='counts', names='age_percent_medulla', title='Distribution Of Age Over Percent Medulla')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution of age over percent cortex"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = hubmap_df\nplt.figure(figsize=(16, 6))\nsns.kdeplot(df.loc[df['percent_cortex'] == 80, 'age'], label = 20,shade=True)\nsns.kdeplot(df.loc[df['percent_cortex'] == 75, 'age'], label = 25,shade=True)\nsns.kdeplot(df.loc[df['percent_cortex'] == 65, 'age'], label = 35,shade=True)\nsns.kdeplot(df.loc[df['percent_cortex'] == 55, 'age'], label = 45,shade=True)\n\n\n# Labeling of plot\nplt.xlabel('Age (years)'); plt.ylabel('Density'); plt.title('Distribution of Ages')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['age','percent_cortex'])['age'].count().reset_index(name = 'counts')\ndf['age_percent_cortex'] = df['age'].astype('str')+'_'+df['percent_cortex'].astype('str')\nfig = px.pie(df, values='counts', names='age_percent_cortex', title='Distribution Of Age Over Percent Cortex')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above graphs we infer that the sum of percent medulla and percent cortex is 100. Let's verify it below."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"(hubmap_df['percent_medulla']+hubmap_df['percent_cortex']).unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sex vs Race"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(16, 6))\na = sns.countplot(data=hubmap_df, x='race', hue='sex')\n\nfor p in a.patches:\n a.annotate(format(p.get_height(), ','), \n (p.get_x() + p.get_width() / 2., \n p.get_height()), ha = 'center', va = 'center', \n xytext = (0, 4), textcoords = 'offset points')\n\nplt.title('Sex split by Race', fontsize=16)\nsns.despine(left=True, bottom=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.box(hubmap_df, x=\"sex\", y=\"age\", points=\"all\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df = hubmap_df.groupby(['sex','race'])['sex'].count().reset_index(name = 'counts')\ndf['sex_race'] = df['sex'].astype('str')+'_'+df['race'].astype('str')\nfig = px.pie(df, values='counts', names='sex_race', title='Sex Versus Race')\nfig.update_layout(\n autosize=False,\n width=800,\n height=650,\n margin=dict(\n l=50,\n r=50,\n b=50,\n t=50,\n pad=4\n ),\n paper_bgcolor=\"LightSteelBlue\",\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"male = hubmap_df[hubmap_df['sex']=='Male']\nfemale = hubmap_df[hubmap_df['sex']== 'Female']\ncount_male = male[['race','split']].value_counts().reset_index()\ncount_female = female[['race','split']].value_counts().reset_index()\npie_male = go.Pie(labels=count_male[['race','split']],values=count_male[0],name=\"Male\",hole=0.4,domain={'x': [0,0.46]})\npie_female = go.Pie(labels=count_female[['race','split']],values=count_female[0],name=\"Female\",hole=0.5,domain={'x': [0.52,1]})\nlayout = dict(title = 'Sex_Race', font=dict(size=10), legend=dict(orientation=\"h\"),\n annotations = [dict(x=0.2, y=0.5, text='Male', showarrow=False, font=dict(size=20)),\n dict(x=0.8, y=0.5, text='Female', showarrow=False, font=dict(size=20)) ])\n\nfig = dict(data=[pie_male, pie_female], layout=layout)\npyo.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"male = hubmap_df[hubmap_df['sex']=='Male']\nfemale = hubmap_df[hubmap_df['sex']== 'Female']\ncount_male = male['race'].value_counts().reset_index()\ncount_female = female['race'].value_counts().reset_index()\npie_male = go.Pie(labels=count_male['index'],values=count_male['race'],name=\"Male\",hole=0.4,domain={'x': [0,0.46]})\npie_female = go.Pie(labels=count_female['index'],values=count_female['race'],name=\"Female\",hole=0.5,domain={'x': [0.52,1]})\nlayout = dict(title = 'Sex_Race', font=dict(size=10), legend=dict(orientation=\"h\"),\n annotations = [dict(x=0.2, y=0.5, text='Male', showarrow=False, font=dict(size=20)),\n dict(x=0.8, y=0.5, text='Female', showarrow=False, font=dict(size=20)) ])\n\nfig = dict(data=[pie_male, pie_female], layout=layout)\npyo.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above graphs we can infer that there is no female black in the test set."},{"metadata":{},"cell_type":"markdown","source":"## HeatMap for hubmap"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"corrmat = hubmap_df.corr() \nf, ax = plt.subplots(figsize =(9, 8)) \nsns.heatmap(corrmat, ax = ax, cmap = 'RdYlBu_r', linewidths = 0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Visualising Images : Tiff 🗺️ "},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"print(Fore.YELLOW + 'Train .tiff number of images:',Style.RESET_ALL, len(list(os.listdir('../input/hubmap-kidney-segmentation/train')))/3, '\\n' +\n Fore.BLUE + 'Test .tiff number of images:',Style.RESET_ALL, len(list(os.listdir('../input/hubmap-kidney-segmentation/test')))/2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"IMAGE_PATH = '../input/hubmap-kidney-segmentation'\nos.path.join(IMAGE_PATH, 'train/095bf7a1f.tiff')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"im = tiff.imread(\n os.path.join(IMAGE_PATH, \"train/0486052bb.tiff\")\n)\nplt.figure(figsize=(10, 10))\nplt.imshow(im)\nplt.axis(\"off\")\ndel im","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Credit goes to https://www.kaggle.com/iafoss/256x256-images/data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sz = 256 #the size of tiles\nreduce = 4 #reduce the original images by 4 times \nMASKS = '../input/hubmap-kidney-segmentation/train.csv'\nDATA = '../input/hubmap-kidney-segmentation/train/'\nOUT_TRAIN = 'train.zip'\nOUT_MASKS = 'masks.zip'\ndef enc2mask(encs, shape):\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for m,enc in enumerate(encs):\n if isinstance(enc,np.float) and np.isnan(enc): continue\n s = enc.split()\n for i in range(len(s)//2):\n start = int(s[2*i]) - 1\n length = int(s[2*i+1])\n img[start:start+length] = 1 + m\n return img.reshape(shape).T\n\ndef mask2enc(mask, shape, n=1):\n pixels = mask.T.flatten()\n encs = []\n for i in range(1,n+1):\n p = (pixels == i).astype(np.int8)\n if p.sum() == 0: encs.append(np.nan)\n else:\n p = np.concatenate([[0], p, [0]])\n runs = np.where(p[1:] != p[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n encs.append(' '.join(str(x) for x in runs))\n return encs\ndf_masks = pd.read_csv(MASKS).set_index('id')\nx_tot,x2_tot = [],[]\nwith zipfile.ZipFile(OUT_TRAIN, 'w') as img_out,\\\n zipfile.ZipFile(OUT_MASKS, 'w') as mask_out:\n for index, encs in tqdm(df_masks.iterrows(),total=len(df_masks)):\n #read image and generate the mask\n img = tiff.imread(os.path.join(DATA,index+'.tiff'))\n if len(img.shape) == 5:img = np.transpose(img.squeeze(), (1,2,0))\n mask = enc2mask(encs,(img.shape[1],img.shape[0]))\n\n #add padding to make the image dividable into tiles\n shape = img.shape\n pad0,pad1 = (reduce*sz - shape[0]%(reduce*sz))%(reduce*sz), (reduce*sz - shape[1]%(reduce*sz))%(reduce*sz)\n img = np.pad(img,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2],[0,0]],constant_values=0)\n mask = np.pad(mask,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2]],constant_values=0)\n\n #split image and mask into tiles using the reshape+transpose trick\n img = cv2.resize(img,(img.shape[0]//reduce,img.shape[1]//reduce),interpolation = cv2.INTER_AREA)\n img = img.reshape(img.shape[0]//sz,sz,img.shape[1]//sz,sz,3)\n img = img.transpose(0,2,1,3,4).reshape(-1,sz,sz,3)\n\n mask = cv2.resize(mask,(mask.shape[0]//reduce,mask.shape[1]//reduce),interpolation = cv2.INTER_NEAREST)\n mask = mask.reshape(mask.shape[0]//sz,sz,mask.shape[1]//sz,sz)\n mask = mask.transpose(0,2,1,3).reshape(-1,sz,sz)\n\n #write data\n for i,(im,m) in enumerate(zip(img,mask)):\n if im.sum() == 0: continue\n \n x_tot.append((im/255.0).reshape(-1,3).mean(0))\n x2_tot.append(((im/255.0)**2).reshape(-1,3).mean(0))\n \n im = cv2.imencode('.png',cv2.cvtColor(im, cv2.COLOR_RGB2BGR))[1]\n img_out.writestr(f'{index}_{i}.png', im)\n m = cv2.imencode('.png',m)[1]\n mask_out.writestr(f'{index}_{i}.png', m)\n\n#image stats\nimg_avr = np.array(x_tot).mean(0)\nimg_std = np.sqrt(np.array(x2_tot).mean(0) - img_avr**2)\nprint('mean:',img_avr, ', std:', img_std)\ncolumns, rows = 4,4\nidx0 = 870\nfig=plt.figure(figsize=(columns*4, rows*4))\nwith zipfile.ZipFile(OUT_TRAIN, 'r') as img_arch, \\\n zipfile.ZipFile(OUT_MASKS, 'r') as msk_arch:\n fnames = sorted(img_arch.namelist())[8:]\n for i in range(rows):\n for j in range(columns):\n idx = i+j*columns\n flags = cv2.IMREAD_COLOR\n img = cv2.imdecode(np.frombuffer(img_arch.read(fnames[idx0+idx]), np.uint8), flags)\n flags = cv2.IMREAD_GRAYSCALE\n mask = cv2.imdecode(np.frombuffer(msk_arch.read(fnames[idx0+idx]), np.uint8), flags)\n \n fig.add_subplot(rows, columns, idx+1)\n plt.axis('off')\n plt.imshow(Image.fromarray(img))\n plt.imshow(Image.fromarray(mask), alpha=0.2)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Plot polygon from JSON Files"},{"metadata":{},"cell_type":"markdown","source":"Credits : https://www.kaggle.com/subbuvolvosekar/plot-ploygon-from-json-using-shapely"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"TRAIN_PATH = '/kaggle/input/hubmap-kidney-segmentation/train/'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"file = sorted(os.listdir(TRAIN_PATH))\nfor i in range(0,len(file),3):\n with open(TRAIN_PATH+file[i]) as f:\n data = json.load(f)\n\n r1 = sg.Polygon([tuple(i) for i in data[0]['geometry']['coordinates'][0]])\n r2 = sg.box(0.5,0.5,1.5,1.5)\n r3 = sg.box(4,4,5,5)\n\n new_shape = so.cascaded_union([r1, r2, r3])\n fig, axs = plt.subplots()\n axs.set_aspect('equal', 'datalim')\n\n for geom in new_shape.geoms: \n xs, ys = geom.exterior.xy \n axs.fill(xs, ys, alpha=0.5, fc='r', ec='none')\n\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 9. Pandas Profiling 🌤️"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"profile_hubmap_df = pdp.ProfileReport(hubmap_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"profile_hubmap_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# B.Modelling"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"### We are using **pytorch** implementation of **UNet** Model implemented in **https://github.com/qubvel/segmentation_models.pytorch** and this is getting installed offline."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!mkdir -p /tmp/pip/cache/\n!cp ../input/segmentationmodelspytorch/segmentation_models/efficientnet_pytorch-0.6.3.xyz /tmp/pip/cache/efficientnet_pytorch-0.6.3.tar.gz\n!cp ../input/segmentationmodelspytorch/segmentation_models/pretrainedmodels-0.7.4.xyz /tmp/pip/cache/pretrainedmodels-0.7.4.tar.gz\n!cp ../input/segmentationmodelspytorch/segmentation_models/segmentation-models-pytorch-0.1.2.xyz /tmp/pip/cache/segmentation_models_pytorch-0.1.2.tar.gz\n!cp ../input/segmentationmodelspytorch/segmentation_models/timm-0.1.20-py3-none-any.whl /tmp/pip/cache/\n!cp ../input/segmentationmodelspytorch/segmentation_models/timm-0.2.1-py3-none-any.whl /tmp/pip/cache/\n!pip install --no-index --find-links /tmp/pip/cache/ efficientnet-pytorch\n!pip install --no-index --find-links /tmp/pip/cache/ segmentation-models-pytorch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Necessary Imports"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"from sklearn.model_selection import GroupKFold\nimport torch\nfrom torch import nn\nimport torchvision\nimport cv2\nfrom torchvision import transforms\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler, RandomSampler\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom scipy.ndimage.interpolation import zoom\nimport albumentations as A\nfrom torch.nn import functional as F\n\nimport time\nimport random\nfrom albumentations.pytorch import ToTensorV2\nfrom segmentation_models_pytorch.unet import Unet","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!mkdir -p /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b0-355c32eb.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b1-f1951068.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b2-8bb594d6.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b3-5fb5a3c3.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b4-6ed6700e.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b5-b6417697.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b6-c76e70fd.pth /root/.cache/torch/hub/checkpoints/\n!cp ../input/efficientnet-pytorch-b0-b7/efficientnet-b7-dcc49843.pth /root/.cache/torch/hub/checkpoints/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\nseed_everything(42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"class HuBMAPDataset(Dataset):\n def __init__(self, ids, phase):\n self.ids = ids\n if phase=='train':\n self.transform = get_train_transform()\n else:\n self.transform = get_val_transform()\n \n def __getitem__(self, idx):\n name = self.ids[idx]\n img = cv2.imread(f\"../input/256256-hubmap/train/{name}\").astype(\"float32\")\n img /= 255.\n mask = cv2.imread(f\"../input/256256-hubmap/masks/{name}\")[:,:,0:1]\n\n transformed = self.transform(image=img, mask=mask)\n img = transformed['image']\n mask = transformed['mask']\n img = img.transpose(2,0,1).astype('float32')\n mask = mask.transpose(2,0,1).astype('float32')\n return img, mask\n\n def __len__(self):\n return len(self.ids)\n\n \ndef get_train_transform():\n return A.Compose([\n A.HorizontalFlip(),\n A.OneOf([\n A.RandomContrast(),\n A.RandomGamma(),\n A.RandomBrightness(),\n ], p=0.3),\n A.OneOf([\n A.ElasticTransform(alpha=120, sigma=120 * 0.05, alpha_affine=120 * 0.03),\n A.GridDistortion(),\n A.OpticalDistortion(distort_limit=2, shift_limit=0.5),\n ], p=0.3),\n A.ShiftScaleRotate(p=0.2),\n A.Resize(256,256,always_apply=True),\n ],p=1.)\n\ndef get_val_transform():\n return A.Compose([\n A.Resize(256,256,always_apply=True),\n ],p=1.)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DataLoader"},{"metadata":{"trusted":true},"cell_type":"code","source":"directory_list = os.listdir('../input/256256-hubmap/train')\ndir_df = pd.DataFrame(directory_list, columns=['Image_Paths'])\ndir_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_train_valid_dataloader(df, fold):\n train_ids = df.loc[~df.Folds.isin(fold), \"Image_Paths\"].values\n val_ids = df.loc[df.Folds.isin(fold), \"Image_Paths\"].values\n train_ds = HuBMAPDataset(train_ids, \"train\")\n val_ds = HuBMAPDataset(val_ids, \"val\")\n train_loader = DataLoader(train_ds, batch_size=16, pin_memory=True, shuffle=True, num_workers=4)\n val_loader = DataLoader(val_ds, batch_size=4, pin_memory=True, shuffle=False, num_workers=4)\n return train_loader, val_loader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"class HuBMAP(nn.Module):\n def __init__(self):\n super(HuBMAP, self).__init__()\n self.cnn_model = Unet('efficientnet-b5', encoder_weights=\"imagenet\", classes=1, activation=None)\n #self.cnn_model.decoder.blocks.append(self.cnn_model.decoder.blocks[-1])\n #self.cnn_model.decoder.blocks[-2] = self.cnn_model.decoder.blocks[-3]\n \n def forward(self, imgs):\n img_segs = self.cnn_model(imgs)\n return img_segs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loss Function"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"class DiceLoss(nn.Module):\n def __init__(self, weight=None, size_average=True):\n super(DiceLoss, self).__init__()\n\n def forward(self, inputs, targets, smooth=1):\n \n #comment out if your model contains a sigmoid or equivalent activation layer\n inputs = F.sigmoid(inputs) \n \n #flatten label and prediction tensors\n inputs = inputs.view(-1)\n targets = targets.view(-1)\n \n intersection = (inputs * targets).sum() \n dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth) \n \n return 1 - dice\n \n \n \nclass DiceBCELoss(nn.Module):\n # Formula Given above.\n def __init__(self, weight=None, size_average=True):\n super(DiceBCELoss, self).__init__()\n\n def forward(self, inputs, targets, smooth=1):\n \n #comment out if your model contains a sigmoid or equivalent activation layer\n inputs = F.sigmoid(inputs) \n \n #flatten label and prediction tensors\n inputs = inputs.view(-1)\n targets = targets.view(-1)\n \n intersection = (inputs * targets).sum() \n dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth) \n BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')\n Dice_BCE = BCE + dice_loss\n \n return Dice_BCE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def HuBMAPLoss(images, targets, model, device):\n model.to(device)\n images = images.to(device)\n targets = targets.to(device)\n outputs = model(images)\n criterion = DiceBCELoss()\n loss = criterion(outputs, targets)\n return loss, outputs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_one_epoch(epoch, model, device, optimizer, scheduler, trainloader):\n model.train()\n t = time.time()\n total_loss = 0\n for step, (images, targets) in enumerate(trainloader):\n loss, outputs = HuBMAPLoss(images, targets, model, device)\n loss.backward()\n if ((step+1)%4==0 or (step+1)==len(trainloader)):\n optimizer.step()\n scheduler.step()\n optimizer.zero_grad()\n loss = loss.detach().item()\n total_loss += loss\n if ((step+1)%10==0 or (step+1)==len(trainloader)):\n print(\n f'epoch {epoch} train step {step+1}/{len(trainloader)}, ' + \\\n f'loss: {total_loss/len(trainloader):.4f}, ' + \\\n f'time: {(time.time() - t):.4f}', end= '\\r' if (step + 1) != len(trainloader) else '\\n'\n )\n\n \n \ndef valid_one_epoch(epoch, model, device, optimizer, scheduler, validloader):\n model.eval()\n t = time.time()\n total_loss = 0\n for step, (images, targets) in enumerate(validloader):\n loss, outputs = HuBMAPLoss(images, targets, model, device)\n loss = loss.detach().item()\n total_loss += loss\n if ((step+1)%4==0 or (step+1)==len(validloader)):\n scheduler.step(total_loss/len(validloader))\n if ((step+1)%10==0 or (step+1)==len(validloader)):\n print(\n f'epoch {epoch} valid step {step+1}/{len(validloader)}, ' + \\\n f'loss: {total_loss/len(validloader):.4f}, ' + \\\n f'time: {(time.time() - t):.4f}', end= '\\r' if (step + 1) != len(validloader) else '\\n'\n )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Folds Column"},{"metadata":{"trusted":true},"cell_type":"code","source":"FOLDS = 5\ngkf = GroupKFold(FOLDS)\ndir_df['Folds'] = 0\nfor fold, (tr_idx, val_idx) in enumerate(gkf.split(dir_df, groups=dir_df[dir_df.columns[0]].values)):\n dir_df.loc[val_idx, 'Folds'] = fold","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Real Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold, (tr_idx, val_idx) in enumerate(gkf.split(dir_df, groups=dir_df[dir_df.columns[0]].values)):\n if fold>0:\n break\n trainloader, validloader = prepare_train_valid_dataloader(dir_df, [fold])\n device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n model = HuBMAP().to(device)\n optimizer = Adam(model.parameters(), lr=5e-4)\n scheduler = torch.optim.lr_scheduler.StepLR(optimizer, gamma=0.1, step_size=1)\n #num_epochs = 15\n num_epochs = 1\n for epoch in range(num_epochs):\n train_one_epoch(epoch, model, device, optimizer, scheduler, trainloader)\n with torch.no_grad():\n valid_one_epoch(epoch, model, device, optimizer, scheduler, validloader)\n torch.save(model.state_dict(),f'FOLD-{fold}-model.pth')\n break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since EDA alone took more space while running on gpu..I have made the training and inference for same model in my another [kernel](https://www.kaggle.com/vineeth1999/hubmap-pytorch-efficientunet-offline). Dont forget to see that kernel too. TPU version of my kernel has been written by [Kishan](https://www.kaggle.com/joshi98kishan) which is [here](https://www.kaggle.com/joshi98kishan/training-pytorch-tpu-8-cores)\n\n`GPU` - [vineeth](https://www.kaggle.com/vineeth1999/hubmap-pytorch-efficientunet-offline)\n\n`TPU` - [kishan](https://www.kaggle.com/joshi98kishan/training-pytorch-tpu-8-cores)"},{"metadata":{},"cell_type":"markdown","source":"## Thank you for Reading My kernel."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
\n
"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DiceLoss(nn.Module):\n def __init__(self, weight=None, size_average=True):\n super(DiceLoss, self).__init__()\n\n def forward(self, inputs, targets, smooth=1):\n \n #comment out if your model contains a sigmoid or equivalent activation layer\n inputs = F.sigmoid(inputs) \n \n #flatten label and prediction tensors\n inputs = inputs.view(-1)\n targets = targets.view(-1)\n \n intersection = (inputs * targets).sum() \n dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth) \n \n return 1 - dice\n \n \n \nclass DiceBCELoss(nn.Module):\n # Formula Given above.\n def __init__(self, weight=None, size_average=True):\n super(DiceBCELoss, self).__init__()\n\n def forward(self, inputs, targets, smooth=1):\n \n #comment out if your model contains a sigmoid or equivalent activation layer\n inputs = F.sigmoid(inputs) \n \n #flatten label and prediction tensors\n inputs = inputs.view(-1)\n targets = targets.view(-1)\n \n intersection = (inputs * targets).sum() \n dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth) \n BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')\n Dice_BCE = BCE + dice_loss\n \n return Dice_BCE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def HuBMAPLoss(images, targets, model, device):\n model.to(device)\n images = images.to(device)\n targets = targets.to(device)\n outputs = model(images)\n criterion = DiceBCELoss()\n loss = criterion(outputs, targets)\n return loss, outputs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_one_epoch(epoch, model, device, optimizer, scheduler, trainloader):\n model.train()\n t = time.time()\n total_loss = 0\n for step, (images, targets) in enumerate(trainloader):\n loss, outputs = HuBMAPLoss(images, targets, model, device)\n loss.backward()\n if ((step+1)%4==0 or (step+1)==len(trainloader)):\n optimizer.step()\n scheduler.step()\n optimizer.zero_grad()\n loss = loss.detach().item()\n total_loss += loss\n if ((step+1)%10==0 or (step+1)==len(trainloader)):\n print(\n f'epoch {epoch} train step {step+1}/{len(trainloader)}, ' + \\\n f'loss: {total_loss/len(trainloader):.4f}, ' + \\\n f'time: {(time.time() - t):.4f}', end= '\\r' if (step + 1) != len(trainloader) else '\\n'\n )\n\n \n \ndef valid_one_epoch(epoch, model, device, optimizer, scheduler, validloader):\n model.eval()\n t = time.time()\n total_loss = 0\n for step, (images, targets) in enumerate(validloader):\n loss, outputs = HuBMAPLoss(images, targets, model, device)\n loss = loss.detach().item()\n total_loss += loss\n if ((step+1)%4==0 or (step+1)==len(validloader)):\n scheduler.step(total_loss/len(validloader))\n if ((step+1)%10==0 or (step+1)==len(validloader)):\n print(\n f'epoch {epoch} valid step {step+1}/{len(validloader)}, ' + \\\n f'loss: {total_loss/len(validloader):.4f}, ' + \\\n f'time: {(time.time() - t):.4f}', end= '\\r' if (step + 1) != len(validloader) else '\\n'\n )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Creating Folds Column"},{"metadata":{"trusted":true},"cell_type":"code","source":"FOLDS = 5\ngkf = GroupKFold(FOLDS)\ndir_df['Folds'] = 0\nfor fold, (tr_idx, val_idx) in enumerate(gkf.split(dir_df, groups=dir_df[dir_df.columns[0]].values)):\n dir_df.loc[val_idx, 'Folds'] = fold","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Real Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"for fold, (tr_idx, val_idx) in enumerate(gkf.split(dir_df, groups=dir_df[dir_df.columns[0]].values)):\n if fold>0:\n break\n trainloader, validloader = prepare_train_valid_dataloader(dir_df, [fold])\n device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n model = HuBMAP().to(device)\n optimizer = Adam(model.parameters(), lr=5e-4)\n scheduler = torch.optim.lr_scheduler.StepLR(optimizer, gamma=0.1, step_size=1)\n #num_epochs = 15\n num_epochs = 1\n for epoch in range(num_epochs):\n train_one_epoch(epoch, model, device, optimizer, scheduler, trainloader)\n with torch.no_grad():\n valid_one_epoch(epoch, model, device, optimizer, scheduler, validloader)\n torch.save(model.state_dict(),f'FOLD-{fold}-model.pth')\n break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since EDA alone took more space while running on gpu..I have made the training and inference for same model in my another [kernel](https://www.kaggle.com/vineeth1999/hubmap-pytorch-efficientunet-offline). Dont forget to see that kernel too. TPU version of my kernel has been written by [Kishan](https://www.kaggle.com/joshi98kishan) which is [here](https://www.kaggle.com/joshi98kishan/training-pytorch-tpu-8-cores)\n\n`GPU` - [vineeth](https://www.kaggle.com/vineeth1999/hubmap-pytorch-efficientunet-offline)\n\n`TPU` - [kishan](https://www.kaggle.com/joshi98kishan/training-pytorch-tpu-8-cores)"},{"metadata":{},"cell_type":"markdown","source":"## Thank you for Reading My kernel."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}