\n
If you find this notebook useful, do give us an upvote, it motivates us a lot.
This notebook is still a work in progress. Keep checking for further developments!😊
","metadata":{}},{"cell_type":"markdown","source":"\n
\n\n
0. BACKGROUND INFORMATION ⤒
\n\n# Task description:\n\nIn this competition, we'll **identify and segment functional tissue units (FTUs)** accross **five** human organs:\n\n* Prostate\n* Spleen\n* Lung\n* Kidney\n* Large Intestine\n\nThe challenge in this competition is to build algorithms that **generalize**:\n* across different **organs** and\n* across different **dataset** differences\n\n=> This is a **semantic segmentation** problem.\n\n# Data description:\n\nThey only release public *Human Protein Atlas (HPA)* data for the training dataset. However, they will release private *HPA* data and *Human BioMolecular Atlas Program (HuBMAP)* for their public test set. For the private test set, they only use *HuBMAP* data.\n\n**File Information**:\n1. `train|test.csv`:\n* `id` - The image ID.\n* `organ` - The organ that the biopsy sample was taken from.\n* `data_source` - Whether the image was provided by Hubamp or HPA.\n* `img_height` - The height of the image in pixels.\n* `img_width` - The width of the image in pixels.\n* `pixel_size` - The height/width of a single pixel from this image in micrometers. All HPA images have a pixel size of 0.4 µm. For Hubmap imagery the pixel size is 0.5 µm for kidney, 0.2290 µm for large intestine, 0.7562 µm for lung, 0.4945 µm for spleen, and 6.263 µm for prostate.\n* `tissue_thickness` - The thickness of the biopsy sample in micrometers. All HPA images have a thickness of 4 µm. The Hubmap samples have tissue slice thicknesses 10 µm for kidney, 8 µm for large intestine, 4 µm for spleen, 5 µm for lung, and 5 µm for prostate.\n* `rle` - The target column. A run length encoded copy of the annotations. Provided for the training set only.\n* `age` - The patient's age in years. Provided for the training set only.\n* `sex` - The sex of the patient. Provided for the training set only.\n2. `train_annotations`: provided in the format of **points that define the boundaries of the polygon masks of the FTUs**\n3. `train|test_images`: the images:\n* Expect roughly 550 images in the hidden test set.\n* All images used have at least one FTU.\n* All tissue data used in this competition is from healthy donors that pathologists identified as pathologically unremarkable tissue.\n* HPA details:\n * All HPA images are 3000 x 3000 pixels with a tissue area within the image around 2500 x 2500 pixels.\n * HPA samples were stained with antibodies visualized with 3,3'-diaminobenzidine (DAB) and counterstained with hematoxylin.\n* HuBMAP details:\n * The Hubmap images range in size from 4500x4500 down to 160x160 pixels.\n * HuBMAP images were prepared using Periodic acid-Schiff (PAS)/hematoxylin and eosin (H&E) stains.\n4. `sample_submission.csv`:\n* `id`-the image ID\n* `rle`-a run length encoded mask of the FTUs in the image\n\n# Evaluation metric:\n\nThis competition is evaluated on the mean [Dice coefficient](https://radiopaedia.org/articles/dice-similarity-coefficient#:~:text=The%20Dice%20similarity%20coefficient%2C%20also,between%20two%20sets%20of%20data.). The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:\n\n$$\\frac{2∗|𝑋∩𝑌|}{|𝑋|+|𝑌|}$$\n\nwhere \n* X is the predicted set of pixels and Y is the ground truth. \n* The Dice coefficient is defined to be 1 when both X and Y are empty. \n\n**Note**: metric is to judge the performance of the model, whereas loss function is to optimize the model.\n\nIn this case, our metric is the **mean Dice coefficient**, and we can use different loss functions like \n* **Dice Loss**\n* **Jaccard Loss**\n* **BCE Loss**\n* **Lovasz Loss**\n* **Tversky Loss**\n\nto optimize our models\n\n# Submission file format\n\nTo reduce the submission file size, the metric uses run-length encoding on the pixel values.\n\nInstead of submitting an exhaustive list of indices for our segmentation, we will submit pairs of values that contain a start position and a run length\n\nE.g. '1 3' implies starting at pixel 1 and running a total of 3 pixels (1,2,3).\n\nNote that, at the time of encoding, the mask should be **binary**\n* The masks for all objects in an image are joined into a single large mask\n* The value of 0 should indicate pixels that are not masked\n* The value of 1 will indicate pixels that are masked.\n\n# Methods:\n\n1. Overview:\n * **Run-length encoding (RLE)**: a form of lossless data compression.\n \n Since we already have the RLE masks, we don't need to use the annotations from `.json` file.\n * **Given**: images (`.tiff`), masks in RLE (we need to convert RLE to binary mask before feeding to our models)\n * **Predict**: masks then convert to RLE for submission\n2. Data processing:\n * Resize\n * Normalize\n3. Data augmentation:\n4. Baseline model: \n * UNET\n\n5. Testing model:\n * UNET + pretrained model from previous competition","metadata":{}},{"cell_type":"markdown","source":"
\n\n
1. IMPORTS ⤒
","metadata":{}},{"cell_type":"code","source":"import os \nimport glob\nfrom pathlib import Path\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.auto import tqdm\n\nimport plotly\nfrom plotly import tools\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nimport plotly.offline as pyo\nimport plotly.io as pio\nimport plotly.graph_objects as go\n#pio.templates.default = 'plotly_white'\nsns.set_theme(style=\"dark\")\nimport cv2\nimport tifffile as tiff\n\nimport rasterio\nfrom rasterio.windows import Window\n\nimport warnings\nwarnings.simplefilter(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:27:05.596054Z","iopub.execute_input":"2022-09-14T22:27:05.596533Z","iopub.status.idle":"2022-09-14T22:27:06.092573Z","shell.execute_reply.started":"2022-09-14T22:27:05.596483Z","shell.execute_reply":"2022-09-14T22:27:06.091445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Global config","metadata":{}},{"cell_type":"code","source":"class config:\n BASE_PATH = Path(\"../input/hubmap-organ-segmentation/\")\n TRAIN_CSV_PATH = BASE_PATH / \"train.csv\"\n TRAIN_IMAGES_PATH = BASE_PATH / \"train_images/\"\n TRAIN_ANNOTATIONS_PATH = BASE_PATH / \"train_annotations/\" # Not needed","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:27:37.589127Z","iopub.execute_input":"2022-09-14T22:27:37.590386Z","iopub.status.idle":"2022-09-14T22:27:37.597043Z","shell.execute_reply.started":"2022-09-14T22:27:37.590337Z","shell.execute_reply":"2022-09-14T22:27:37.595463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load datasets","metadata":{}},{"cell_type":"markdown","source":"## Train dataset","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(config.TRAIN_CSV_PATH)\ndata.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-09-14T22:27:40.659810Z","iopub.execute_input":"2022-09-14T22:27:40.660422Z","iopub.status.idle":"2022-09-14T22:27:41.085920Z","shell.execute_reply.started":"2022-09-14T22:27:40.660370Z","shell.execute_reply":"2022-09-14T22:27:41.084331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train images\n\nLoad a single image from `train_images`","metadata":{}},{"cell_type":"code","source":"id_ = 10274\nimg = tiff.imread(str(config.TRAIN_IMAGES_PATH/ f\"{id_}.tiff\"))\nprint(img.shape)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:27:46.936114Z","iopub.execute_input":"2022-09-14T22:27:46.936775Z","iopub.status.idle":"2022-09-14T22:27:47.478243Z","shell.execute_reply.started":"2022-09-14T22:27:46.936718Z","shell.execute_reply":"2022-09-14T22:27:47.476640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To enhance image contrast, we apply a method from the [LAB color space](https://www.xrite.com/blog/lab-color-space)\n\nLAB color space expresses color variations across three channels. One channel for brightness and two channels for color:\n\n* L-channel: representing lightness in the image\n* a-channel: representing change in color between red and green\n* b-channel: representing change in color between yellow and blue\n\nWe perform adaptive histogram equalization on the L-channel, and convert the image back to RGB color space.","metadata":{}},{"cell_type":"code","source":"# https://stackoverflow.com/questions/39308030/how-do-i-increase-the-contrast-of-an-image-in-python-opencv\n# Enhance image contrast\nsample = img.copy()\nlab= cv2.cvtColor(sample, cv2.COLOR_BGR2LAB)\nl_channel, a, b = cv2.split(lab)\n# Applying CLAHE to L-channel\n# feel free to try different values for the limit and grid size:\nclahe = cv2.createCLAHE(clipLimit=5.0, tileGridSize=(12,12))\ncl = clahe.apply(l_channel)\n\n# merge the CLAHE enhanced L-channel with the a and b channel\nlimg = cv2.merge((cl,a,b))\n\n# Converting image from LAB Color model to BGR color space\nenhanced_img = cv2.cvtColor(limg, cv2.COLOR_LAB2BGR)\n\nplt.figure(figsize=(10, 10))\n# Stacking the original image with the enhanced image\nresult = np.hstack((img, enhanced_img))\nplt.title('Image Enhancement\\n [Left]: Original [Right]: Enhanced', fontsize=14)\nplt.imshow(result)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:27:51.139991Z","iopub.execute_input":"2022-09-14T22:27:51.140430Z","iopub.status.idle":"2022-09-14T22:27:54.222902Z","shell.execute_reply.started":"2022-09-14T22:27:51.140393Z","shell.execute_reply":"2022-09-14T22:27:54.221940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot image","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nplt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:27:58.628124Z","iopub.execute_input":"2022-09-14T22:27:58.628524Z","iopub.status.idle":"2022-09-14T22:28:00.158635Z","shell.execute_reply.started":"2022-09-14T22:27:58.628491Z","shell.execute_reply":"2022-09-14T22:28:00.157544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train annotations","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/paulorzp/rle-functions-run-length-encode-decode\ndef mask2rle(img): # encoder\n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels= img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n \ndef rle2mask(mask_rle, shape=(1600,256)): # decoder\n '''\n mask_rle: run-length as string formated (start length)\n shape: (width,height) of array to return \n Returns numpy array, 1 - mask, 0 - background\n\n '''\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:01.841479Z","iopub.execute_input":"2022-09-14T22:28:01.842247Z","iopub.status.idle":"2022-09-14T22:28:02.015445Z","shell.execute_reply.started":"2022-09-14T22:28:01.842206Z","shell.execute_reply":"2022-09-14T22:28:02.012294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask = rle2mask(data[data[\"id\"]==id_][\"rle\"].values[0], (img.shape[1], img.shape[0]))\nmask.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:05.077354Z","iopub.execute_input":"2022-09-14T22:28:05.077967Z","iopub.status.idle":"2022-09-14T22:28:05.106541Z","shell.execute_reply.started":"2022-09-14T22:28:05.077913Z","shell.execute_reply":"2022-09-14T22:28:05.105126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plot mask","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(mask, cmap='seismic', alpha=0.7)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:24.278831Z","iopub.execute_input":"2022-09-14T22:28:24.280047Z","iopub.status.idle":"2022-09-14T22:28:25.626847Z","shell.execute_reply.started":"2022-09-14T22:28:24.279985Z","shell.execute_reply":"2022-09-14T22:28:25.625565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
2. IMAGE + MASK VISUALIZATION ⤒
","metadata":{}},{"cell_type":"markdown","source":"# Overlay image and mask","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(enhanced_img)\nplt.imshow(mask, cmap='seismic', alpha=0.5)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:27.604337Z","iopub.execute_input":"2022-09-14T22:28:27.604773Z","iopub.status.idle":"2022-09-14T22:28:30.241753Z","shell.execute_reply.started":"2022-09-14T22:28:27.604737Z","shell.execute_reply":"2022-09-14T22:28:30.240577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot mask on enhanced image\ndef plot_mask(image, mask, image_id):\n plt.figure(figsize=(10, 10))\n \n lab= cv2.cvtColor(image, cv2.COLOR_BGR2LAB)\n l_channel, a, b = cv2.split(lab)\n # Applying CLAHE to L-channel\n # feel free to try different values for the limit and grid size:\n clahe = cv2.createCLAHE(clipLimit=5.0, tileGridSize=(12,12))\n cl = clahe.apply(l_channel)\n\n # merge the CLAHE enhanced L-channel with the a and b channel\n limg = cv2.merge((cl,a,b))\n\n # Converting image from LAB Color model to BGR color space\n enhanced_img = cv2.cvtColor(limg, cv2.COLOR_LAB2BGR)\n \n \n # subplot(nrows, ncols, index)\n plt.subplot(2, 2, 1)\n plt.imshow(image)\n plt.title(f\"Image {image_id}\", fontsize=14)\n\n plt.subplot(2, 2, 2)\n plt.imshow(enhanced_img)\n plt.title(f\"Enhanced Image {image_id}\", fontsize=14)\n \n plt.subplot(2, 2, 3)\n plt.imshow(mask, cmap=\"seismic\", alpha =0.8)\n plt.title(f\"Mask\", fontsize=14) \n \n plt.subplot(2, 2, 4)\n plt.imshow(enhanced_img)\n plt.imshow(mask, cmap=\"seismic\", alpha=0.5)\n plt.title(f\"Enhanced Image {image_id} + mask\", fontsize=14) \n \n \n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:35.639769Z","iopub.execute_input":"2022-09-14T22:28:35.640244Z","iopub.status.idle":"2022-09-14T22:28:35.651508Z","shell.execute_reply.started":"2022-09-14T22:28:35.640203Z","shell.execute_reply":"2022-09-14T22:28:35.650006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_mask(img, mask, id_)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:37.633957Z","iopub.execute_input":"2022-09-14T22:28:37.634814Z","iopub.status.idle":"2022-09-14T22:28:43.931144Z","shell.execute_reply.started":"2022-09-14T22:28:37.634776Z","shell.execute_reply":"2022-09-14T22:28:43.929937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Batches of images + masks grouped by organs\n\n* Prostate\n* Spleen\n* Lung\n* Kidney\n* Large Intestine","metadata":{}},{"cell_type":"markdown","source":"## Helper functions","metadata":{}},{"cell_type":"code","source":"def read_image_mask(df, img_id):\n image = tiff.imread(str(config.TRAIN_IMAGES_PATH/ f\"{img_id}.tiff\"))\n \n mask = rle2mask(\n df[df[\"id\"] == img_id][\"rle\"].values[0], \n (image.shape[1], image.shape[0])\n )\n return image, mask","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:44.842713Z","iopub.execute_input":"2022-09-14T22:28:44.843268Z","iopub.status.idle":"2022-09-14T22:28:44.851741Z","shell.execute_reply.started":"2022-09-14T22:28:44.843219Z","shell.execute_reply":"2022-09-14T22:28:44.850390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# functions to convert encoding to mask and mask to encoding\ndef enc2mask(encs, shape):\n '''\n Args:\n encs: list of rle masks\n shape: mask shape\n '''\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for m,enc in enumerate(encs):\n if isinstance(enc,np.float32) and np.isnan(enc): continue\n s = enc.split()\n for i in range(len(s)//2):\n start = int(s[2*i]) - 1\n length = int(s[2*i+1])\n img[start:start+length] = 1 + m\n return img.reshape(shape).T","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:28:47.249830Z","iopub.execute_input":"2022-09-14T22:28:47.250385Z","iopub.status.idle":"2022-09-14T22:28:47.261480Z","shell.execute_reply.started":"2022-09-14T22:28:47.250332Z","shell.execute_reply":"2022-09-14T22:28:47.260251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Visualize batch of images for each organ with cropped tissue area\ndef visualize_batch(df, organ, slide):\n count = 0\n fig = plt.figure(figsize=(30,30))\n for index, row in df.iterrows():\n if count == 25: break\n \n if row['organ'] == organ:\n img = rasterio.open(os.path.join(DATA,str(index)+'.tiff'), num_threads='all_cpus')\n mask = enc2mask([row['rle']],(img.shape[1],img.shape[0]))\n \n #read window slide\n img = img.read([1,2,3],window=Window.from_slices(slide, slide)) \n mask = mask[slide[0]:slide[1], slide[0]:slide[1]] \n \n plt.subplot(5,5, count+1)\n plt.imshow(np.transpose(img, (1,2,0)))\n plt.imshow(mask, cmap='seismic', alpha=0.4)\n plt.title(f'{index}', size=40)\n plt.axis('off')\n \n count += 1\n del img, mask\n \n plt.tight_layout() \n plt.suptitle(f'{organ}', size=50, weight=\"bold\", y = 1.1) \n plt.show() ","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:33:32.388269Z","iopub.execute_input":"2022-09-14T22:33:32.389113Z","iopub.status.idle":"2022-09-14T22:33:32.401346Z","shell.execute_reply.started":"2022-09-14T22:33:32.389064Z","shell.execute_reply":"2022-09-14T22:33:32.400179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def image_info(df, img_id):\n organ = df[df['id'] == img_id]['organ'].values[0]\n pixel_size = df.loc[df['id'] == img_id]['pixel_size'].values[0]\n tissue_thickness = df.loc[df['id'] == img_id]['tissue_thickness'].values[0]\n age = df.loc[df['id'] == img_id]['age'].values[0]\n sex = df.loc[df['id'] == img_id]['sex'].values[0]\n image, mask_ = read_image_mask(df, img_id)\n print('\\033[1m' + 'Case study : {}'.format(img_id) + '\\033[0m')\n print('\\n------------------------------\\n')\n print('\\033[1m' + 'General info: \\n' + '\\033[0m')\n print('\\033[1m' + 'Organ: ' '\\033[0m' + f'{organ}') \n print('\\033[1m' + 'Age: ' '\\033[0m' + f'{age}')\n print('\\033[1m' + 'Sex: ' '\\033[0m' + f'{sex}')\n print('\\n------------------------------\\n')\n print('\\033[1m' + 'Image + Mask info: \\n' + '\\033[0m')\n print('\\033[1m' + 'Pixel size: ' '\\033[0m' + f'{pixel_size}')\n print('\\033[1m' + 'Tissue thickness: ' '\\033[0m' + f'{tissue_thickness}')\n print('\\033[1m' + 'Image shape: ' '\\033[0m' + f'{image.shape}')\n print('\\033[1m' + 'Mask shape: ' '\\033[0m' + f'{mask_.shape}')\n plot_mask(image, mask_, img_id)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:29:55.753010Z","iopub.execute_input":"2022-09-14T22:29:55.754494Z","iopub.status.idle":"2022-09-14T22:29:55.766422Z","shell.execute_reply.started":"2022-09-14T22:29:55.754440Z","shell.execute_reply":"2022-09-14T22:29:55.764761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inter- and Intra-Class Variability","metadata":{}},{"cell_type":"code","source":"LI1 = tiff.imread('../input/hubmap-organ-segmentation/train_images/11662.tiff')\nLI2 = tiff.imread('../input/hubmap-organ-segmentation/train_images/11890.tiff')\n\nlung1 = tiff.imread('../input/hubmap-organ-segmentation/train_images/10666.tiff')\nlung2 = tiff.imread('../input/hubmap-organ-segmentation/train_images/10971.tiff')","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:30:06.506603Z","iopub.execute_input":"2022-09-14T22:30:06.507218Z","iopub.status.idle":"2022-09-14T22:30:08.050309Z","shell.execute_reply.started":"2022-09-14T22:30:06.507164Z","shell.execute_reply":"2022-09-14T22:30:08.048937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(25,16))\n \nplt.subplot(1,4,1)\nplt.imshow(LI1)\nplt.axis('off')\n\nplt.subplot(1,4,2)\nplt.imshow(LI2)\nplt.title('Large Intestine', fontsize=30, y=1.1, x=-0.1)\nplt.axis('off')\n\nplt.subplot(1,4,3)\nplt.imshow(lung1)\nplt.axis('off')\n\nplt.subplot(1,4,4)\nplt.imshow(lung2)\nplt.title('Prostate', fontsize=30, y=1.1, x=-0.1)\nplt.axis('off')\n\nplt.tight_layout()\n#plt.savefig('inter_intra_variability.png')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:30:08.052364Z","iopub.execute_input":"2022-09-14T22:30:08.052717Z","iopub.status.idle":"2022-09-14T22:30:13.338020Z","shell.execute_reply.started":"2022-09-14T22:30:08.052686Z","shell.execute_reply":"2022-09-14T22:30:13.336798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prostate (Only male)","metadata":{}},{"cell_type":"markdown","source":"### Case study","metadata":{}},{"cell_type":"code","source":"df_prostate = data.loc[data['organ']=='prostate']\ndf_prostate.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:30:32.355196Z","iopub.execute_input":"2022-09-14T22:30:32.355582Z","iopub.status.idle":"2022-09-14T22:30:32.377939Z","shell.execute_reply.started":"2022-09-14T22:30:32.355551Z","shell.execute_reply":"2022-09-14T22:30:32.376815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(df_prostate, 10044)\nimage_info(df_prostate, 10274)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:30:35.456515Z","iopub.execute_input":"2022-09-14T22:30:35.456960Z","iopub.status.idle":"2022-09-14T22:30:48.717102Z","shell.execute_reply.started":"2022-09-14T22:30:35.456925Z","shell.execute_reply":"2022-09-14T22:30:48.715610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize prostate batch with cropped tissue area\n","metadata":{}},{"cell_type":"code","source":"DATA = '../input/hubmap-organ-segmentation/train_images'","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:34:17.701760Z","iopub.execute_input":"2022-09-14T22:34:17.702174Z","iopub.status.idle":"2022-09-14T22:34:17.707640Z","shell.execute_reply.started":"2022-09-14T22:34:17.702141Z","shell.execute_reply":"2022-09-14T22:34:17.705946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_masks = pd.read_csv('../input/hubmap-organ-segmentation/train.csv')[['id', 'rle','organ']].set_index('id')\ndf_masks.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:31:59.657672Z","iopub.execute_input":"2022-09-14T22:31:59.658099Z","iopub.status.idle":"2022-09-14T22:31:59.814253Z","shell.execute_reply.started":"2022-09-14T22:31:59.658066Z","shell.execute_reply":"2022-09-14T22:31:59.812620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_batch(df_masks, 'prostate', (400,2500))","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:34:20.460484Z","iopub.execute_input":"2022-09-14T22:34:20.460952Z","iopub.status.idle":"2022-09-14T22:34:57.501195Z","shell.execute_reply.started":"2022-09-14T22:34:20.460910Z","shell.execute_reply":"2022-09-14T22:34:57.500136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Spleen","metadata":{}},{"cell_type":"markdown","source":"### Case study","metadata":{}},{"cell_type":"code","source":"df_spleen = data.loc[data['organ']=='spleen']\ndf_spleen.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:34:57.502732Z","iopub.execute_input":"2022-09-14T22:34:57.503802Z","iopub.status.idle":"2022-09-14T22:34:57.529042Z","shell.execute_reply.started":"2022-09-14T22:34:57.503764Z","shell.execute_reply":"2022-09-14T22:34:57.527911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(df_spleen, 2279)\nimage_info(df_spleen, 10610)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:34:57.530732Z","iopub.execute_input":"2022-09-14T22:34:57.531416Z","iopub.status.idle":"2022-09-14T22:35:11.263111Z","shell.execute_reply.started":"2022-09-14T22:34:57.531374Z","shell.execute_reply":"2022-09-14T22:35:11.262042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize spleen batch with cropped tissue area","metadata":{}},{"cell_type":"code","source":"visualize_batch(df_masks, 'spleen', (400,2500))","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:35:20.401079Z","iopub.execute_input":"2022-09-14T22:35:20.401685Z","iopub.status.idle":"2022-09-14T22:35:59.048126Z","shell.execute_reply.started":"2022-09-14T22:35:20.401643Z","shell.execute_reply":"2022-09-14T22:35:59.045928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Lung","metadata":{}},{"cell_type":"markdown","source":"### Case study","metadata":{}},{"cell_type":"code","source":"df_lung = data.loc[data['organ']=='lung']\ndf_lung.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:35:59.051339Z","iopub.execute_input":"2022-09-14T22:35:59.052053Z","iopub.status.idle":"2022-09-14T22:35:59.078493Z","shell.execute_reply.started":"2022-09-14T22:35:59.051981Z","shell.execute_reply":"2022-09-14T22:35:59.077251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(df_lung, 10488)\nimage_info(df_lung, 11629)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:35:59.080132Z","iopub.execute_input":"2022-09-14T22:35:59.080507Z","iopub.status.idle":"2022-09-14T22:36:11.663709Z","shell.execute_reply.started":"2022-09-14T22:35:59.080474Z","shell.execute_reply":"2022-09-14T22:36:11.662642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize lung batch with cropped tissue area","metadata":{}},{"cell_type":"code","source":"visualize_batch(df_masks, 'lung', (400,2500))","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:36:11.666114Z","iopub.execute_input":"2022-09-14T22:36:11.666803Z","iopub.status.idle":"2022-09-14T22:36:48.214061Z","shell.execute_reply.started":"2022-09-14T22:36:11.666736Z","shell.execute_reply":"2022-09-14T22:36:48.210650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Kidney","metadata":{}},{"cell_type":"markdown","source":"### Case study","metadata":{}},{"cell_type":"code","source":"df_kidney = data.loc[data['organ']=='kidney']\ndf_kidney.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:36:48.218194Z","iopub.execute_input":"2022-09-14T22:36:48.221012Z","iopub.status.idle":"2022-09-14T22:36:48.244411Z","shell.execute_reply.started":"2022-09-14T22:36:48.220925Z","shell.execute_reply":"2022-09-14T22:36:48.243512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(df_kidney, 1157)\nimage_info(df_kidney, 11497)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:36:48.245628Z","iopub.execute_input":"2022-09-14T22:36:48.246848Z","iopub.status.idle":"2022-09-14T22:37:00.818744Z","shell.execute_reply.started":"2022-09-14T22:36:48.246791Z","shell.execute_reply":"2022-09-14T22:37:00.817252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize kidney batch with cropped tissue area","metadata":{}},{"cell_type":"code","source":"visualize_batch(df_masks, 'kidney', (400,2500))","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:37:00.820776Z","iopub.execute_input":"2022-09-14T22:37:00.821272Z","iopub.status.idle":"2022-09-14T22:37:39.011045Z","shell.execute_reply.started":"2022-09-14T22:37:00.821226Z","shell.execute_reply":"2022-09-14T22:37:39.008451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Large intestine","metadata":{}},{"cell_type":"markdown","source":"### Case study","metadata":{}},{"cell_type":"code","source":"df_LI = data.loc[data['organ']=='largeintestine']\ndf_LI.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:37:39.015807Z","iopub.execute_input":"2022-09-14T22:37:39.016930Z","iopub.status.idle":"2022-09-14T22:37:39.057673Z","shell.execute_reply.started":"2022-09-14T22:37:39.016839Z","shell.execute_reply":"2022-09-14T22:37:39.056186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_info(df_LI, 1168)\nimage_info(df_LI, 11662)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:37:39.059538Z","iopub.execute_input":"2022-09-14T22:37:39.060912Z","iopub.status.idle":"2022-09-14T22:37:51.659921Z","shell.execute_reply.started":"2022-09-14T22:37:39.060840Z","shell.execute_reply":"2022-09-14T22:37:51.658474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize large intestine batch with cropped tissue area","metadata":{}},{"cell_type":"code","source":"visualize_batch(df_masks, 'largeintestine', (400,2500))","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:37:51.661957Z","iopub.execute_input":"2022-09-14T22:37:51.662417Z","iopub.status.idle":"2022-09-14T22:38:29.122298Z","shell.execute_reply.started":"2022-09-14T22:37:51.662382Z","shell.execute_reply":"2022-09-14T22:38:29.120807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n
","metadata":{}},{"cell_type":"markdown","source":"# Statistical Description","metadata":{}},{"cell_type":"code","source":"def EDA(df):\n \n print('\\033[1m' +'EXPLORATORY DATA ANALYSIS :'+ '\\033[0m\\n')\n print('\\033[1m' + 'Shape of the data (rows, columns):' + '\\033[0m')\n print(df.shape, \n '\\n------------------------------------------------------------------------------------\\n')\n \n print('\\033[1m' + 'All columns from the dataframe :' + '\\033[0m')\n print(df.columns, \n '\\n------------------------------------------------------------------------------------\\n')\n \n print('\\033[1m' + 'Datatypes and Missing values:' + '\\033[0m')\n print(df.info(), \n '\\n------------------------------------------------------------------------------------\\n')\n \n for col in df.columns:\n print('\\033[1m' + 'Unique values in {} :'.format(col) + '\\033[0m',len(data[col].unique()))\n print('\\n------------------------------------------------------------------------------------\\n')\n \n print('\\033[1m' + 'Summary statistics for the data :' + '\\033[0m')\n print(df.describe(include='all'), \n '\\n------------------------------------------------------------------------------------\\n')\n \n \n print('\\033[1m' + 'Memory used by the data :' + '\\033[0m')\n print(df.memory_usage(), \n '\\n------------------------------------------------------------------------------------\\n')\n \n print('\\033[1m' + 'Number of duplicate values :' + '\\033[0m')\n print(df.duplicated().sum())\n \nEDA(data)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:29.125417Z","iopub.execute_input":"2022-09-14T22:38:29.126447Z","iopub.status.idle":"2022-09-14T22:38:29.293410Z","shell.execute_reply.started":"2022-09-14T22:38:29.126391Z","shell.execute_reply":"2022-09-14T22:38:29.291926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data visualization","metadata":{}},{"cell_type":"markdown","source":"## Univariate visualization of categorical variables (sex, organ)\n","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/code/toomuchsauce/mental-health-plotly-interactive-viz\ncolumns = ['organ','sex']\ndf = data[columns]\n\nbuttons = []\ni = 0\nvis = [False] * 4\n\nfor col in df.columns:\n vis[i] = True\n buttons.append({'label' : col,\n 'method' : 'update',\n 'args' : [{'visible' : vis},\n {'title' : col}] })\n i+=1\n vis = [False] * 4\n\nfig = go.Figure()\n\nfor col in df.columns:\n fig.add_trace(go.Pie(\n values = df[col].value_counts(),\n labels = df[col].value_counts().index,\n title = dict(text = 'Distribution of {}'.format(col),\n font = dict(size=18, family = 'monospace'),\n ),\n hole = 0.5,\n hoverinfo='label+percent',))\n\nfig.update_traces(hoverinfo='label+percent',\n textinfo='label+percent',\n textfont_size=12,\n opacity = 0.8,\n showlegend = False,\n marker = dict(colors = sns.color_palette('Pastel1').as_hex(),\n line=dict(color='#000000', width=1)))\n \n\nfig.update_layout(margin=dict(t=0, b=0, l=0, r=0),\n updatemenus = [dict(\n type = 'dropdown',\n x = 1.15,\n y = 0.85,\n showactive = True,\n active = 0,\n buttons = buttons)],\n annotations=[\n dict(text = \"
Choose
Column : \",\n showarrow=False,\n x = 1.06, y = 0.92, yref = \"paper\", align = \"left\")])\n\nfor i in range(1,2):\n fig.data[i].visible = False\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:29.295309Z","iopub.execute_input":"2022-09-14T22:38:29.295742Z","iopub.status.idle":"2022-09-14T22:38:29.534720Z","shell.execute_reply.started":"2022-09-14T22:38:29.295703Z","shell.execute_reply":"2022-09-14T22:38:29.533232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Organ Distribution","metadata":{}},{"cell_type":"code","source":"sns.set_style('darkgrid')\nplt.figure(figsize=(13,6))\ng = sns.countplot(data=data, x='organ')\ng.set(xlabel='Organ', ylabel='Count')\nfor p in g.patches:\n x, w, h = p.get_x(), p.get_width(), p.get_height()\n if h > 0:\n g.text(x + w / 2, h, f'{h}\\n', ha='center', va='center', size=11)\ng.set_title('Organ Distribution', fontsize=20)\n#plt.savefig('organ_distribution.png')","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:29.537848Z","iopub.execute_input":"2022-09-14T22:38:29.538393Z","iopub.status.idle":"2022-09-14T22:38:29.744453Z","shell.execute_reply.started":"2022-09-14T22:38:29.538346Z","shell.execute_reply":"2022-09-14T22:38:29.742988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Density chart with Age and Sex","metadata":{}},{"cell_type":"code","source":"g = sns.FacetGrid(data,hue='sex', height=4, aspect=4)\ng = g.map(sns.kdeplot,'age',shade=True,alpha=0.4) \ng.add_legend()\ng.set(xlim=(20, 84), xticks=np.arange(20,88,4))\ng.fig.suptitle('Density between Age and Sex', y=1.05)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:29.745969Z","iopub.execute_input":"2022-09-14T22:38:29.746333Z","iopub.status.idle":"2022-09-14T22:38:30.326271Z","shell.execute_reply.started":"2022-09-14T22:38:29.746298Z","shell.execute_reply":"2022-09-14T22:38:30.324947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nplt.figure(figsize=(15,8))\ng = sns.histplot(data=data, x=\"age\", hue=\"sex\", multiple=\"dodge\", color='sex', bins='fd')\ng.set(title='Distribution of Age by Gender')\ng.set(xlim=(20, 84), xticks=np.arange(20,88,4))\nfor p in g.patches:\n x, w, h = p.get_x(), p.get_width(), p.get_height()\n if h > 0:\n g.text(x + w / 2, h, f'{h}\\n', ha='center', va='center', size=11)\ng.margins(y=0.07)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:30.328031Z","iopub.execute_input":"2022-09-14T22:38:30.328426Z","iopub.status.idle":"2022-09-14T22:38:30.821216Z","shell.execute_reply.started":"2022-09-14T22:38:30.328389Z","shell.execute_reply":"2022-09-14T22:38:30.820297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Distribution of organ type by gender","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nplt.figure(figsize=(15,8))\ng = sns.histplot(data=data, x=\"sex\", hue=\"organ\", multiple=\"dodge\", color='organ', bins='fd')\ng.set(title='Distribution of Organ Type by Gender')\nfor p in g.patches:\n x, w, h = p.get_x(), p.get_width(), p.get_height()\n if h > 0:\n g.text(x + w / 2, h, f'{h}\\n', ha='center', va='center', size=11)\ng.margins(y=0.07)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:30.822832Z","iopub.execute_input":"2022-09-14T22:38:30.823569Z","iopub.status.idle":"2022-09-14T22:38:31.185666Z","shell.execute_reply.started":"2022-09-14T22:38:30.823528Z","shell.execute_reply":"2022-09-14T22:38:31.184612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Distribution of age by organ type","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nplt.figure(figsize=(15,10))\ng = sns.histplot(data=data, x=\"age\", hue=\"organ\", multiple=\"stack\", color='organ', bins='fd')\ng.set(xlim=(20, 86), xticks=np.arange(20,88,4))\ng.set(title='Distribution of Age by Organ Type')\nfor p in g.patches:\n x, y, w, h = p.get_x(), p.get_y(), p.get_width(), p.get_height()\n if h > 0:\n g.text(x + w / 2, y + h / 2, f'{h:.0f}\\n', ha='center', va='center', size=11)\ng.margins(y=0.07)","metadata":{"execution":{"iopub.status.busy":"2022-09-14T22:38:31.187367Z","iopub.execute_input":"2022-09-14T22:38:31.188067Z","iopub.status.idle":"2022-09-14T22:38:31.880288Z","shell.execute_reply.started":"2022-09-14T22:38:31.188022Z","shell.execute_reply":"2022-09-14T22:38:31.878951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}