{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n
\n\n

đŸ«đŸ«€ EDA – HuBMAP+HPA – Organ Segmentation đŸ«€đŸ«

\n
CREATED BY: DARIEN SCHETTLER
\n\n
\n\n---\n\n
\n\n
\n 🛑   WARNING:

THIS IS A WORK IN PROGRESS
\n
\n\n\n
\n 👏   IF YOU FORK THIS OR FIND THIS HELPFUL   👏

PLEASE UPVOTE!

This was a lot of work for me and while it may seem silly, it makes me feel appreciated when others like my work. 😅\n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n\n

TABLE OF CONTENTS

\n\n---\n\n

0    IMPORTS

\n\n---\n\n

1    BACKGROUND INFORMATION

\n\n---\n\n

2    SETUP

\n\n---\n\n

3    HELPER FUNCTIONS

\n\n---\n\n

4    DATASET EXPLORATION

\n\n---\n\n

5    MODELLING

\n\n---\n\n

9    APPENDIX

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS    ⤒

","metadata":{}},{"cell_type":"code","source":"print(\"\\n... IMPORTS STARTING ...\\n\")\n\nprint(\"\\n\\tVERSION INFORMATION\")\n# Machine Learning and Data Science Imports\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport tensorflow_hub as tfhub; print(f\"\\t\\t– TENSORFLOW HUB VERSION: {tfhub.__version__}\");\nimport tensorflow_addons as tfa; print(f\"\\t\\t– TENSORFLOW ADDONS VERSION: {tfa.__version__}\");\nimport tensorflow_io as tfio; print(f\"\\t\\t– TENSORFLOW I/O VERSION: {tfio.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None;\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\nfrom sklearn.preprocessing import RobustScaler, PolynomialFeatures\nfrom pandarallel import pandarallel; pandarallel.initialize();\nfrom sklearn.model_selection import GroupKFold, StratifiedKFold\nfrom scipy.spatial import cKDTree\n\n# # RAPIDS\n# import cudf, cupy, cuml\n# from cuml.neighbors import NearestNeighbors\n# from cuml.manifold import TSNE, UMAP\n\n# Built In Imports\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom glob import glob\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image, ImageEnhance\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport plotly\nimport PIL\nimport cv2\n\nimport plotly.io as pio\nprint(pio.renderers)\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\n \nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-06T19:42:23.995113Z","iopub.execute_input":"2022-07-06T19:42:23.996170Z","iopub.status.idle":"2022-07-06T19:42:36.264599Z","shell.execute_reply.started":"2022-07-06T19:42:23.996068Z","shell.execute_reply":"2022-07-06T19:42:36.263768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  BACKGROUND INFORMATION    ⤒

\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.1 BASIC COMPETITION INFORMATION

\n\n---\n\n
PRIMARY TASK DESCRIPTION\n\nIn this competition, you’ll identify and segment functional tissue units **(FTUs)** across five human organs.\n* You'll build your model using a dataset of tissue section images, with the best submissions segmenting FTUs as accurately as possible.\n* The five human organs we are segmenting are:\n * **Prostate**\n * **Spleen**\n * **Lung**\n * **Kidney**\n * **Large Intestine**\n\n
BASIC BACKGROUND INFORMATION\n\nWhen you think of “life hacks,” normally you’d imagine productivity techniques. But how about the kind that helps you understand your body at a molecular level? It may be possible! Researchers must first determine the function and relationships among the 37 trillion cells that make up the human body. A better understanding of our cellular composition could help people live healthier, longer lives.\n\nA previous Kaggle competition aimed to annotate cell population neighborhoods that perform an organ’s main physiologic function, also called functional tissue units (FTUs). Manually annotating FTUs (e.g., glomeruli in kidney or alveoli in the lung) is a time-consuming process. In the average kidney, there are over 1 million glomeruli FTUs. While there are existing cell and FTU segmentation methods, we want to push the boundaries by building algorithms that generalize across different organs and are robust across different dataset differences.\n\nThe Human BioMolecular Atlas Program (HuBMAP) is working to create a Human Reference Atlas at the cellular level. Sponsored by the National Institutes of Health (NIH), HuBMAP and Indiana University’s Cyberinfrastructure for Network Science Center (CNS) have partnered with institutions across the globe for this endeavor. A major partner is the Human Protein Atlas (HPA), a Swedish research program aiming to map the protein expression in human cells, tissues, and organs, funded by the Knut and Alice Wallenberg Foundation.\n\n
VISUAL EXPLANATION\n\nTBD\n\n
COMPETITION IMPACT STATEMENT\n\nIf successful, you'll help accelerate the world’s understanding of the relationships between cell and tissue organization. With a better idea of the relationship of cells, researchers will have more insight into the function of cells that impact human health. Further, the Human Reference Atlas constructed by HuBMAP will be freely available for use by researchers and pharmaceutical companies alike, potentially improving and prolonging human life.\n\n
COMPETITION HOST INFORMATION\n\n
PRIMARY SPONSORS\n\n
\n\n
SECONDARY SPONSORS\n* Google\n* Genentech\n* Indiana University\n\n

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.2 COMPETITION EVALUATION

\n\n---\n\n
GENERAL EVALUATION INFORMATION\n\nThis competition is evaluated on the mean Dice Coefficient. \n\nThe Dice Coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:\n\n$$\n\\frac{2 * |X \\cap Y|}{|X| + |Y|}\n$$\n\n* Where $X$ is the predicted set of pixels and $Y$ is the ground truth. \n* The Dice coefficient is defined to be $1$ when both $X$ and $Y$ are empty. \n* The leaderboard score is the mean of the Dice Coefficients for each image in the test set.\n\n
\n\n
SUBMISSION FILE INFORMATION\n\nIn order to reduce the submission file size, our metric uses **run-length encoding** on the pixel values. \n* Instead of submitting an exhaustive list of indices for your segmentation, you will submit pairs of values that contain a start position and a run length\n* E.g. '1 3' implies starting at pixel 1 and running a total of 3 pixels (1,2,3).\n* Note that, at the time of encoding, the mask should be **binary**\n * The masks for all objects in an image are joined into a single large mask\n * The value of **0** should indicate pixels that are not **masked**\n * The value of **1** will indicate pixels that are **masked**.\n\nThe competition format requires a space delimited list of pairs. For example, '1 3 10 5' implies pixels 1,2,3,10,11,12,13,14 are to be included in the mask. The metric checks that the pairs are sorted, positive, and the decoded pixel values are not duplicated. The pixels are numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.\n\nNote that, at the time of encoding, the mask should be binary, meaning the masks for all objects in an image are joined into a single large mask. A value of 0 should indicate pixels that are not masked, and a value of 1 will indicate pixels that are masked.\n\n
\n\nThe file should contain a header and have the following format:\n\n```html\nid,rle\n1,1 1 5 1\n2,1 1\n3,1 1\n4,1 5 2 17\netc.\n```\n\n
IS THIS A CODE COMPETITION?\n\nYES\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.3 DATASET OVERVIEW

\n\n---\n\n
GENERAL INFORMATION\n\nThe goal of this competition is to identify the locations of each functional tissue unit (FTU) in biopsy slides from several different organs.. \n\nThe training **annotations are provided as RLE-encoded masks**, and the images are in **16-bit**, **grayscale**, **PNG format**.\n\nThe underlying data includes imagery from different sources prepared with different protocols at a variety of resolutions, reflecting typical challenges for working with medical data.\n\nThis competition uses data from two different consortia, the Human Reference Atlas (HPA) and Human BioMolecular Atlas Program (HuBMAP). \n\nThe data is sourced as following:\n* The training dataset consists of data from public HPA data\n* the public test set is a combination of private HPA data **and HuBMAP data**\n* the private test set contains **only HuBMAP data**. \n\n
\n\n**Adapting models to function properly when presented with data that was prepared using a different protocol will be one of the core challenges of this competition. While this is expected to make the problem more difficult, developing models that generalize is a key goal of this endeavor.**\n\n
\n\n
FILE INFORMATION\n\n**[`train|test.csv`]**\n\n\n\n
\n\n**[`sample_submission.csv`]**\n\n\n\n
\n\n**[`train|test_images`]**\n \nThe images\n* Expect roughly **550 images in the hidden test set**. \n* All images used have **at least one FTU**.\n* All tissue data used in this competition is **from healthy donors that pathologists identified as pathologically unremarkable tissue**.\n* HPA details:\n * All HPA images are **3000 x 3000 pixels** with a **tissue area within the image around 2500 x 2500 pixels**. \n * HPA samples were stained with antibodies visualized with 3,3'-diaminobenzidine (DAB) and counterstained with hematoxylin. \n* HuBMAP details:\n * The Hubmap images range in size from **4500x4500** down to **160x160 pixels**. \n * HuBMAP images were prepared using Periodic acid-Schiff (PAS)/hematoxylin and eosin (H&E) stains. \n\n
\n\n**[`train_annotations`]**\n\nThe annotations \n* Provided in the format of **points that define the boundaries of the polygon masks of the FTUs**\n\n
\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2  SETUP    ⤒

\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.1 ACCELERATOR DETECTION

\n\n---\n\nIn order to use **`TPU`**, we use **`TPUClusterResolver`** for the initialization which is necessary to connect to the remote cluster and initialize cloud TPUs. Let's go over two important points\n\n1. When using TPU on Kaggle, you don't need to specify arguments for **`TPUClusterResolver`**\n2. However, on **G**oogle **C**ompute **E**ngine (**GCE**), you will need to do the following:\n\n
\n\n```python\n# The name you gave to the TPU to use\nTPU_WORKER = 'my-tpu-name'\n\n# or you can also specify the grpc path directly\n# TPU_WORKER = 'grpc://xxx.xxx.xxx.xxx:8470'\n\n# The zone you chose when you created the TPU to use on GCP.\nZONE = 'us-east1-b'\n\n# The name of the GCP project where you created the TPU to use on GCP.\nPROJECT = 'my-tpu-project'\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=TPU_WORKER, zone=ZONE, project=PROJECT)\n```\n\n
\n 🛑   WARNING:

- Although the Tensorflow documentation says it is the project name that should be provided for the argument `project`, it is actually the Project ID, that you should provide. This can be found on the GCP project dashboard page.
\n
\n\n
\n 📖   REFERENCES:

\n - Guide - Use TPUs
\n - Doc - TPUClusterResolver
\n\n
","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... ACCELERATOR SETUP STARTING ...\\n\")\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n TPU = tf.distribute.cluster_resolver.TPUClusterResolver() \nexcept ValueError:\n TPU = None\n\nif TPU:\n print(f\"\\n... RUNNING ON TPU - {TPU.master()}...\")\n tf.config.experimental_connect_to_cluster(TPU)\n tf.tpu.experimental.initialize_tpu_system(TPU)\n strategy = tf.distribute.experimental.TPUStrategy(TPU)\nelse:\n print(f\"\\n... RUNNING ON CPU/GPU ...\")\n # Yield the default distribution strategy in Tensorflow\n # --> Works on CPU and single GPU.\n strategy = tf.distribute.get_strategy() \n\n# What Is a Replica?\n# --> A single Cloud TPU device consists of FOUR chips, each of which has TWO TPU cores. \n# --> Therefore, for efficient utilization of Cloud TPU, a program should make use of each of the EIGHT (4x2) cores. \n# --> Each replica is essentially a copy of the training graph that is run on each core and \n# trains a mini-batch containing 1/8th of the overall batch size\nN_REPLICAS = strategy.num_replicas_in_sync\n \nprint(f\"... # OF REPLICAS: {N_REPLICAS} ...\\n\")\n\nprint(f\"\\n... ACCELERATOR SETUP COMPLTED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.266073Z","iopub.execute_input":"2022-07-06T19:42:36.266794Z","iopub.status.idle":"2022-07-06T19:42:36.284841Z","shell.execute_reply.started":"2022-07-06T19:42:36.266759Z","shell.execute_reply":"2022-07-06T19:42:36.283952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

2.2 COMPETITION DATA ACCESS

\n\n---\n\nTPUs read data must be read directly from **G**oogle **C**loud **S**torage **(GCS)**. Kaggle provides a utility library – **`KaggleDatasets`** – which has a utility function **`.get_gcs_path`** that will allow us to access the location of our input datasets within **GCS**.

\n\n
\n 📌   TIPS:

- If you have multiple datasets attached to the notebook, you should pass the name of a specific dataset to the `get_gcs_path()` function. In our case, the name of the dataset is the name of the directory the dataset is mounted within.

\n
","metadata":{}},{"cell_type":"code","source":"print(\"\\n... DATA ACCESS SETUP STARTED ...\\n\")\n\nif TPU:\n # Google Cloud Dataset path to training and validation images\n DATA_DIR = KaggleDatasets().get_gcs_path('hubmap-organ-segmentation')\n save_locally = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost')\n load_locally = tf.saved_model.LoadOptions(experimental_io_device='/job:localhost')\nelse:\n # Local path to training and validation images\n DATA_DIR = \"/kaggle/input/hubmap-organ-segmentation\"\n save_locally = None\n load_locally = None\n\nprint(f\"\\n... DATA DIRECTORY PATH IS:\\n\\t--> {DATA_DIR}\")\n\nprint(f\"\\n... IMMEDIATE CONTENTS OF DATA DIRECTORY IS:\")\nfor file in tf.io.gfile.glob(os.path.join(DATA_DIR, \"*\")): print(f\"\\t--> {file}\")\n\nprint(\"\\n\\n... DATA ACCESS SETUP COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.286097Z","iopub.execute_input":"2022-07-06T19:42:36.286812Z","iopub.status.idle":"2022-07-06T19:42:36.323572Z","shell.execute_reply.started":"2022-07-06T19:42:36.286776Z","shell.execute_reply":"2022-07-06T19:42:36.322182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

2.3 LEVERAGING XLA OPTIMIZATIONS

\n\n---\n\n\n**XLA** (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes. **The results are improvements in speed and memory usage**.\n\n
\n\nWhen a TensorFlow program is run, all of the operations are executed individually by the TensorFlow executor. Each TensorFlow operation has a precompiled GPU/TPU kernel implementation that the executor dispatches to.\n\nXLA provides us with an alternative mode of running models: it compiles the TensorFlow graph into a sequence of computation kernels generated specifically for the given model. Because these kernels are unique to the model, they can exploit model-specific information for optimization.

\n\n
\n 🛑   WARNING:

- XLA can not currently compile functions where dimensions are not inferrable: that is, if it's not possible to infer the dimensions of all tensors without running the entire computation
\n
\n\n
\n 📌   NOTE:

- XLA compilation is only applied to code that is compiled into a graph (in TF2 that's only a code inside tf.function).
- The jit_compile API has must-compile semantics, i.e. either the entire function is compiled with XLA, or an errors.InvalidArgumentError exception is thrown)\n
\n\n
\n 📖   REFERENCE:

- XLA: Optimizing Compiler for Machine Learning
\n
","metadata":{}},{"cell_type":"code","source":"print(f\"\\n... XLA OPTIMIZATIONS STARTING ...\\n\")\n\nprint(f\"\\n... CONFIGURE JIT (JUST IN TIME) COMPILATION ...\\n\")\n# enable XLA optmizations (10% speedup when using @tf.function calls)\ntf.config.optimizer.set_jit(True)\n\nprint(f\"\\n... XLA OPTIMIZATIONS COMPLETED ...\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.325260Z","iopub.execute_input":"2022-07-06T19:42:36.325603Z","iopub.status.idle":"2022-07-06T19:42:36.334734Z","shell.execute_reply.started":"2022-07-06T19:42:36.325562Z","shell.execute_reply":"2022-07-06T19:42:36.333733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

2.4 BASIC DATA DEFINITIONS & INITIALIZATIONS

\n\n---\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n... BASIC DATA SETUP STARTING ...\\n\\n\")\n\n# Open the training dataframe and display the initial dataframe\nTRAIN_IMAGES_DIR = os.path.join(DATA_DIR, \"train_images\")\nTRAIN_LABELS_DIR = os.path.join(DATA_DIR, \"train_annotations\")\nTRAIN_CSV = os.path.join(DATA_DIR, \"train.csv\")\ntrain_df = pd.read_csv(TRAIN_CSV)\n\n# Get all training images\nall_train_images = glob(os.path.join(TRAIN_IMAGES_DIR, \"*.tiff\"), recursive=True)\nall_train_labels = glob(os.path.join(TRAIN_LABELS_DIR, \"*.json\"), recursive=True)\n\nprint(\"\\n... ORIGINAL TRAINING DATAFRAME... \\n\")\ndisplay(train_df)\n\nTEST_IMAGES_DIR = os.path.join(DATA_DIR, \"test_images\")\nTEST_CSV = os.path.join(DATA_DIR, \"test.csv\")\ntest_df = pd.read_csv(TEST_CSV)\n\nSS_CSV = os.path.join(DATA_DIR, \"sample_submission.csv\")\nss_df = pd.read_csv(SS_CSV)\n\n# Get all testing images if there are any\nall_test_images = glob(os.path.join(TEST_IMAGES_DIR, \"*.tiff\"), recursive=True)\n\nprint(\"\\n\\n\\n... ORIGINAL SUBMISSION DATAFRAME... \\n\")\ndisplay(ss_df)\n\n# For debugging purposes when the test set hasn't been substituted we will know\nDEBUG=len(ss_df)==0\n\n# if DEBUG:\n# TEST_DIR = TRAIN_DIR\n# all_test_images = all_train_images\n# ss_df = train_df.iloc[:10]\n# ss_df = ss_df[[\"id\", \"class\"]]\n# ss_df[\"predicted\"] = \"\"\n \n# print(\"\\n\\n\\n... DEBUG SUBMISSION DATAFRAME... \\n\")\n# display(ss_df)\n\ndef rgb2hex(rgb_tuple):\n r,g,b = rgb_tuple\n def clamp(x): \n return max(0, min(x, 255))\n return \"#{0:02x}{1:02x}{2:02x}\".format(clamp(r), clamp(g), clamp(b))\nORGANS = ['kidney', 'largeintestine', 'lung', 'prostate', 'spleen']\n_COLOURS = [(230, 0, 73), (11, 180, 255), (80, 233, 145), (230, 216, 0), (155, 25, 245)]\nO2C_MAP = {_o:_c for _o,_c in zip(ORGANS, _COLOURS)}\nO2C_HEX_MAP = {_o:rgb2hex(_c) for _o,_c in O2C_MAP.items()}\n\nprint(f\"\\n\\n\\n... ARE WE DEBUGGING: {DEBUG}... \\n\")\n\nprint(\"\\n... BASIC DATA SETUP FINISHED ...\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.336092Z","iopub.execute_input":"2022-07-06T19:42:36.337123Z","iopub.status.idle":"2022-07-06T19:42:36.801929Z","shell.execute_reply.started":"2022-07-06T19:42:36.337092Z","shell.execute_reply":"2022-07-06T19:42:36.800809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

2.5 UPDATE DATAFRAMES WITH ACCESSIBLE EXTRA INFORMATION

\n\n---\n\nI wrapped the logic in a preprocessing function but also went through step by step so people could validate if they so wished\n\n**NOTE:** \n","metadata":{}},{"cell_type":"code","source":"sex_2_int = {\"Male\":0, \"Female\":1}\nint_2_sex = {v:k for k,v in sex_2_int.items()}\nage_divisor = 100.0\n\ntrain_df[\"sex\"] = train_df[\"sex\"].map(sex_2_int)\ntrain_df[\"age\"] = train_df[\"age\"]/age_divisor\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.803503Z","iopub.execute_input":"2022-07-06T19:42:36.804524Z","iopub.status.idle":"2022-07-06T19:42:36.836162Z","shell.execute_reply.started":"2022-07-06T19:42:36.804482Z","shell.execute_reply":"2022-07-06T19:42:36.834969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_img_map = {int(x[:-5].rsplit(\"/\", 1)[-1]):x for x in all_train_images}\ntrain_lbl_map = {int(x[:-5].rsplit(\"/\", 1)[-1]):x for x in all_train_labels}\ntest_img_map = {int(x[:-5].rsplit(\"/\", 1)[-1]):x for x in all_test_images}\n\ntrain_df.insert(3, \"img_path\", train_df[\"id\"].map(train_img_map))\ntrain_df.insert(4, \"lbl_path\", train_df[\"id\"].map(train_lbl_map))\ntest_df.insert(3, \"img_path\", test_df[\"id\"].map(test_img_map))\n\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.837451Z","iopub.execute_input":"2022-07-06T19:42:36.837760Z","iopub.status.idle":"2022-07-06T19:42:36.871334Z","shell.execute_reply.started":"2022-07-06T19:42:36.837731Z","shell.execute_reply":"2022-07-06T19:42:36.869911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

\n 3  HELPER FUNCTION & CLASSES    ⤒\n

\n\n---","metadata":{}},{"cell_type":"code","source":"# ref: https://www.kaggle.com/paulorzp/run-length-encode-and-decode\n# modified from: https://www.kaggle.com/inversion/run-length-decoding-quick-start\ndef rle_decode(mask_rle, shape, color=1):\n \"\"\" TBD\n \n Args:\n mask_rle (str): run-length as string formated (start length)\n shape (tuple of ints): (height,width) of array to return \n \n Returns: \n Mask (np.array)\n - 1 indicating mask\n - 0 indicating background\n\n \"\"\"\n # Split the string by space, then convert it into a integer array\n s = np.array(mask_rle.split(), dtype=int)\n\n # Every even value is the start, every odd value is the \"run\" length\n starts = s[0::2] - 1\n lengths = s[1::2]\n ends = starts + lengths\n\n # The image image is actually flattened since RLE is a 1D \"run\"\n if len(shape)==3:\n h, w, d = shape\n img = np.zeros((h * w, d), dtype=np.float32)\n else:\n h, w = shape\n img = np.zeros((h * w,), dtype=np.float32)\n\n # The color here is actually just any integer you want!\n for lo, hi in zip(starts, ends):\n img[lo : hi] = color\n \n # Don't forget to change the image back to the original shape\n return img.reshape(shape).T\n\n# https://www.kaggle.com/namgalielei/which-reshape-is-used-in-rle\ndef rle_decode_top_to_bot_first(mask_rle, shape):\n \"\"\" TBD\n \n Args:\n mask_rle (str): run-length as string formated (start length)\n shape (tuple of ints): (height,width) of array to return \n \n Returns:\n Mask (np.array)\n - 1 indicating mask\n - 0 indicating background\n\n \"\"\"\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape((shape[1], shape[0]), order='F').T # Reshape from top -> bottom first\n\n# ref.: https://www.kaggle.com/stainsby/fast-tested-rle\ndef rle_encode(img):\n \"\"\" TBD\n \n Args:\n img (np.array): \n - 1 indicating mask\n - 0 indicating background\n \n Returns: \n run length as string formated\n \"\"\"\n pixels = img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n\ndef flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\ndef load_json_to_dict(json_path):\n \"\"\" tbd \"\"\"\n with open(json_path) as json_file:\n data = json.load(json_file)\n return data\n\ndef tf_decode_tiff(img_path, to_numpy=False, to_rgb=False):\n img = tf.io.read_file(img_path)\n img = tfio.experimental.image.decode_tiff(img)\n \n # Optionals\n if to_rgb: img = tfio.experimental.color.rgba_to_rgb(img)\n if to_numpy: img = img.numpy()\n \n return img","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.873181Z","iopub.execute_input":"2022-07-06T19:42:36.873989Z","iopub.status.idle":"2022-07-06T19:42:36.889292Z","shell.execute_reply.started":"2022-07-06T19:42:36.873940Z","shell.execute_reply":"2022-07-06T19:42:36.888476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

\n 4  DATASET EXPLORATION    ⤒\n

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n

4.0 LOOK AT A SINGLE EXAMPLE PRIOR TO INVESTIGATION

\n\n---\n\nWe simply do this to make sure everything is where it should be and we understand the basics of how to access all the relevant data.\n\nWe will wrap this basic exploration functionality as single function to allow for easy examination of any passed identifier","metadata":{}},{"cell_type":"code","source":"def get_overlay(img_path, organ, rle_str, img_shape, _alpha=0.999, _beta=0.35, _gamma=0):\n _img = tf_decode_tiff(img_path, to_numpy=True, to_rgb=True).astype(np.float32)\n _seg_rgb = (np.stack([rle_decode(rle_str, shape=img_shape, color=1),]*3, axis=-1)*O2C_MAP[organ]).astype(np.float32)\n seg_overlay = cv2.addWeighted(src1=_img, alpha=_alpha, \n src2=_seg_rgb, beta=_beta, gamma=_gamma)\n return seg_overlay/255.\n\ndef examine_id(df, ex_id=None, plot_overlay=True, print_meta=True, plot_original=False, plot_segmentation=False, _figsize=(20,20)):\n \"\"\" Wrapper function to allow for easy visual exploration of an example \"\"\"\n if ex_id is None:\n ex_id = df[\"id\"].sample(1).values[0]\n print(f\"\\n... NO ID GIVEN... RANDOM ID CHOSEN: ID={ex_id} ...\")\n \n print(f\"\\n... ID ({ex_id}) EXPLORATION STARTED ...\\n\\n\")\n demo_ex = df[df[\"id\"]==ex_id].squeeze()\n\n if print_meta:\n print(f\"\\n... WITH id=`{ex_id}` (organ=`{demo_ex['organ']}`) – WE HAVE THE FOLLOWING DEMO EXAMPLE TO WORK FROM ... \\n\\n\")\n display(demo_ex.to_frame())\n\n if plot_original:\n print(f\"\\n\\n... ORIGINAL IMAGE PLOT ...\\n\")\n plt.figure(figsize=_figsize)\n plt.imshow(tf_decode_tiff(demo_ex[\"img_path\"], to_numpy=True, to_rgb=True))\n plt.title(f\"Original RGB Image For ID: {ex_id}\", fontweight=\"bold\")\n plt.axis(False)\n plt.show()\n\n if plot_segmentation:\n print(f\"\\n\\n... SEGMENTATION MASK PLOT ({demo_ex['organ']})...\\n\")\n plt.figure(figsize=_figsize)\n plt.imshow((np.stack([rle_decode(demo_ex.rle, shape=(demo_ex.img_width, demo_ex.img_height), color=1),]*3, axis=-1)*O2C_MAP[demo_ex.organ]).astype(np.float32))\n plt.title(f\"Segmentation Mask ({demo_ex.organ})\", fontweight=\"bold\")\n plt.axis(False)\n plt.show()\n\n if plot_overlay:\n print(f\"\\n\\n... IMAGE WITH RGB SEGMENTATION MASK OVERLAY ({demo_ex['organ']}) ...\\n\")\n seg_overlay = get_overlay(demo_ex.img_path, demo_ex.organ, demo_ex.rle, img_shape=(demo_ex.img_width, demo_ex.img_height))\n\n plt.figure(figsize=_figsize)\n plt.imshow(seg_overlay)\n plt.title(f\"Segmentation Overlay id=`{ex_id}` (organ=`{demo_ex['organ']}`)\", fontweight=\"bold\")\n handles = [Rectangle((0,0),1,1, color=(*[__c/255. for __c in _c], 0.5)) for _c in _COLOURS]\n labels = ORGANS\n plt.legend(handles,labels)\n plt.axis(False)\n plt.show()\n\n print(\"\\n\\n... SINGLE ID EXPLORATION FINISHED ...\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.890505Z","iopub.execute_input":"2022-07-06T19:42:36.891211Z","iopub.status.idle":"2022-07-06T19:42:36.906181Z","shell.execute_reply.started":"2022-07-06T19:42:36.891181Z","shell.execute_reply":"2022-07-06T19:42:36.905095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for _o in ORGANS:\n print(f\"\\n\\n... RANDOM {_o.upper()} TRAIN EXAMPLE ...\\n\\n\")\n examine_id(train_df[train_df.organ==_o])\n\nprint(\"\\n\\n\\n\\n... TEST IMAGE ...\\n\\n\")\nexamine_id(test_df, plot_overlay=False, plot_original=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:42:36.910281Z","iopub.execute_input":"2022-07-06T19:42:36.910645Z","iopub.status.idle":"2022-07-06T19:43:04.064767Z","shell.execute_reply.started":"2022-07-06T19:42:36.910613Z","shell.execute_reply":"2022-07-06T19:43:04.063182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.img_width.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:04.067441Z","iopub.execute_input":"2022-07-06T19:43:04.068171Z","iopub.status.idle":"2022-07-06T19:43:04.085743Z","shell.execute_reply.started":"2022-07-06T19:43:04.068120Z","shell.execute_reply":"2022-07-06T19:43:04.084699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.1 INVESTIGATE THE FREQUENCY OF VARIOUS ORGAN SEGMENTATION EXAMPLES

\n\n---\n\n
OBSERVATIONS\n\nWe can observe the following distribution:\n* **kidney**\n * **99** images\n* **prostate**\n * **93** images\n* **large intestine**\n * **58** images\n* **spleen**\n * **53** images\n* **lung**\n * **48** images\n \nAll of the organ types are represented fairly well within our data, with **kidney** and **prostate** being about twice as common as the other three organ types: **large intestine**, **spleen**, and **lung**","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(train_df, \"organ\", color_discrete_map=O2C_HEX_MAP, color=\"organ\", title=\"Number of Segmentation Masks Per Organ Type\",)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:04.087501Z","iopub.execute_input":"2022-07-06T19:43:04.088211Z","iopub.status.idle":"2022-07-06T19:43:05.506418Z","shell.execute_reply.started":"2022-07-06T19:43:04.088166Z","shell.execute_reply":"2022-07-06T19:43:05.505411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.2 INVESTIGATE THE IMAGE SIZES

\n\n---\n\nIt's observable that not all images have the same size... however, given that, there is not that much variation between image sizes:\n\n
OBSERVATIONS\n* Globally, we can see that most of the images are dominated by a single size $(3000 \\times 3000)$\n* All the other sizes (19) only have 1 or 2 occurences each.\n* All image sizes are square\n\nNOTE: THE HOSTS TELL US THAT ALL HPA IMAGES SHOULD BE 3000x3000... HOWEVER, WE CAN SEE THAT ISN'T THE CASE!\n\n
OTHER INFORMATION\n\n* Expect roughly 550 images in the hidden test set. \n* All HPA images are 3000 x 3000 pixels with a tissue area within the image around 2500 x 2500 pixels. \n* The Hubmap images range in size from 4500x4500 down to 160x160 pixels.\n\n
","metadata":{}},{"cell_type":"code","source":"fig = px.scatter(train_df.drop_duplicates(subset=[\"img_width\", \"img_height\"]), x=\"img_width\", y=\"img_height\", \n size=train_df.groupby([\"img_width\", \"img_height\"])[\"id\"].transform(\"count\").iloc[train_df.drop_duplicates(subset=[\"img_width\", \"img_height\"]).index], \n color=\"(\"+train_df.drop_duplicates(subset=[\"img_width\", \"img_height\"])[\"img_width\"].astype(str)+\",\"+train_df.drop_duplicates(subset=[\"img_width\", \"img_height\"])[\"img_height\"].astype(str)+\")\", \n title=\"Bubble Chart Showing The Various Image Sizes\",\n labels={\"color\":\"Size Legend\", \n \"size\":\"Number Of Observations\",\n \"img_height\":\"Image Height (pixels)\",\n \"img_width\":\"Image Width (pixels)\"},\n size_max=150)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:05.507702Z","iopub.execute_input":"2022-07-06T19:43:05.508007Z","iopub.status.idle":"2022-07-06T19:43:05.660945Z","shell.execute_reply.started":"2022-07-06T19:43:05.507979Z","shell.execute_reply":"2022-07-06T19:43:05.659857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.3 INVESTIGATE AGE

\n\n---\n\n
OBSERVATIONS\n* The age distribution is skewed to be 50+\n* The distribution of examples with **large intestine** segmented is much higher among the oldest demographic\n* The general distribution of organ segmentation types across age groups appears relatively stratified (with the previously noted exception)\n\n
","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(train_df, train_df[\"age\"]*age_divisor, color_discrete_map=O2C_HEX_MAP, color=\"organ\", \n title=\"Number of Segmentation Masks Per Age Bin by Organ Type \",\n labels={\"organ\":\"Organ Legend\", \n \"count\":\"Number Of Observations\",\n \"x\":\"Age\"}, text_auto=True,\n )\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:05.662634Z","iopub.execute_input":"2022-07-06T19:43:05.662962Z","iopub.status.idle":"2022-07-06T19:43:05.736315Z","shell.execute_reply.started":"2022-07-06T19:43:05.662932Z","shell.execute_reply":"2022-07-06T19:43:05.735098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.4 INVESTIGATE SEX

\n\n---\n\n
OBSERVATIONS\n* Women do not have a prostate and as such there are no prostate examples where gender is Female.\n* Kidney's seem skewed towards the Male gender\n* All other organ's appear evenly distributed\n\n
","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(train_df, train_df[\"sex\"], color_discrete_map=O2C_HEX_MAP, color=\"organ\", \n title=\"Examples by Gender (0=Male, 1=Female) and Organ Type\",\n labels={\"organ\":\"Organ Legend\", \n \"count\":\"Number Of Observations\",\n \"sex\":\"Sex\"},text_auto=True, barmode=\"group\",\n )\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:05.737775Z","iopub.execute_input":"2022-07-06T19:43:05.738101Z","iopub.status.idle":"2022-07-06T19:43:05.810031Z","shell.execute_reply.started":"2022-07-06T19:43:05.738069Z","shell.execute_reply":"2022-07-06T19:43:05.808822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.5 INVESTIGATE FTUS ACROSS DIFFERENT ORGANS

\n\n---\n\nHere is a zoom sequence from the human body to the single-cell level for the kidney. \n* **Notice that FTU is one level higher than cells.**\n\n
\n\n
\n\nFTUs in the organs presented in this competition are: \n* **glomeruli** in the **kidney** – (a)\n* **crypt** in the **large intestine** – (b)\n* **alveolus** in the **lung** – (c)\n* **glandular acinus** in the **prostate** – (d)\n* **white pulp** in the **spleen** – (e)\n\n
\n\n
\n\n
OBSERVATIONS\n* TBD","metadata":{}},{"cell_type":"code","source":"def crop_ftu_polygons(img, cnts, pad=5):\n def get_crop_from_cnt(_img, _cnt, _pad):\n x1,y1,w,h = cv2.boundingRect(np.array(_cnt))\n x2,y2 = x1+w+_pad, y1+h+_pad\n x1, y1 = max(0, x1-_pad), max(0, y1-_pad)\n return _img[y1:y2, x1:x2]\n ftu_crops = [get_crop_from_cnt(img, cnt, pad) for cnt in cnts]\n return ftu_crops \n \ndef plot_ftus(ftu_crops, n_cols=4, height_per_row=7, max_crops=50):\n ftu_crops = ftu_crops[:max_crops]\n n_rows = int(np.ceil(len(ftu_crops)/n_cols))\n \n plt.figure(figsize=(20, n_rows*height_per_row))\n for i, ftu_crop in enumerate(ftu_crops):\n plt.subplot(n_rows, n_cols, i+1)\n plt.imshow(ftu_crop)\n plt.title(f\"FTU #{i+1} –– SHAPE={ftu_crop.shape[:-1]}\", fontweight=\"bold\")\n plt.axis(False)\n \n plt.tight_layout()\n plt.show()\n\nMAX_N_FTU = 50\nN_EX = 10\n\nftu_crop_map = {}\nfor _organ in ORGANS:\n sub_df = train_df[train_df.organ==_organ].sample(N_EX)\n imgs = [cv2.imread(img_path)[..., ::-1] for img_path in sub_df.img_path.values]\n cnts = [load_json_to_dict(json_path) for json_path in sub_df.lbl_path.values]\n ftu_crops = flatten_l_o_l([crop_ftu_polygons(img, _cnts, pad=0) for img, _cnts in zip(imgs, cnts)])[:MAX_N_FTU]\n ftu_crop_map[_organ] = ftu_crops.copy()\n print(f\"\\n\\n\\n... DISPLAYING {len(ftu_crops)} {_organ} FTU CROPS ...\\n\")\n plot_ftus(ftu_crops)\n\n# def get_ftus(df, organ, k=25, pad=25, one_per_img=False, data_source=\"HPA\", do_resize=False, resize_to=(128,128)):\n# pass","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:05.811874Z","iopub.execute_input":"2022-07-06T19:43:05.812206Z","iopub.status.idle":"2022-07-06T19:43:59.534261Z","shell.execute_reply.started":"2022-07-06T19:43:05.812177Z","shell.execute_reply":"2022-07-06T19:43:59.532559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.6 AVERAGE FTU SIZE (BOUNDING-RECT) BY ORGAN

\n\n---\n\nSince it would take too much processing to comb through all the available images and respective FTUs we will look at the previously obtained sampling of crops (bounding-rects) and the respective dimensions therewithin.\n\nAs we know all the HPA images have a pixel resolution (size) of 0.4”m we can therefore determine the real dimensions of the bounding rectangles around each FTU.\n\nNote dimensions are represented as (HEIGHTxWIDTH) unless otherwise communicated\n\n
OBSERVATIONS\n* **Kidney**\n * **Actual** FTU size is on-average SQUARE (most images aren't square but it averages as such) at **~157x155”m**\n * **HPA** FTU size in pixels will be **~392x389px**\n * **HuBMAP** FTU size in pixels will be **~314x310px**\n* **Large Intestine**\n * **Actual** FTU size is on-average RECTANGULAR (1Hx2W) (images are mostly horizontal but occasionally vertical or square) at **~60x131”m**\n * **HPA** FTU size in pixels will be **~150x328px**\n * **HuBMAP** FTU size in pixels will be **~262x572px**\n* **Lung**\n * FTUs are SMALL!\n * **Actual** FTU size is on-average SQUARE (images aren't square but average to mostly square) at **~90x80”m**\n * **HPA** FTU size in pixels will be **~225x200px**\n * **HuBMAP** FTU size in pixels will be **~119x106px**\n* **Prostate**\n * **Actual** FTU size is on-average SQUARE (images aren't square but average to mostly square) at **~68x77”m**\n * **HPA** FTU size in pixels will be **~172x193px**\n * **HuBMAP** FTU size in pixels will be **~11x13px**\n* **Spleen**\n * **Actual** FTU size is on-average RECTANGULAR (2Hx1W) at **~407x197”m**\n * **HPA** FTU size in pixels will be **~1018x493px**\n * **HuBMAP** FTU size in pixels will be **~823x399px**\n\n
","metadata":{}},{"cell_type":"code","source":"ftu_shape_map = {_organ:[c_map.shape[:-1] for c_map in crop_maps] for _organ, crop_maps in ftu_crop_map.items()}\nftu_area_map = {_organ:[c_map.shape[0]*c_map.shape[1] for c_map in crop_maps] for _organ, crop_maps in ftu_crop_map.items()}\n\norgan_ftu_df = pd.DataFrame({\"organ\":ORGANS})\norgan_ftu_df[\"average_area\"] = organ_ftu_df.organ.apply(lambda x: np.array(ftu_area_map[x]).mean())\norgan_ftu_df[\"average_area_”m\"] = organ_ftu_df[\"average_area\"]*0.4**2\norgan_ftu_df[\"average_height\"] = organ_ftu_df.organ.apply(lambda x: np.array(ftu_shape_map[x][0]).mean())\norgan_ftu_df[\"average_width\"] = organ_ftu_df.organ.apply(lambda x: np.array(ftu_shape_map[x][1]).mean())\norgan_ftu_df[\"average_shape_”m\"] = (\"(\"+(organ_ftu_df[\"average_height\"]*0.4).round(2).astype(str)+\",\"+(organ_ftu_df[\"average_width\"]*0.4).round(2).astype(str)+\")\").apply(ast.literal_eval)\norgan_ftu_df","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:59.536846Z","iopub.execute_input":"2022-07-06T19:43:59.537431Z","iopub.status.idle":"2022-07-06T19:43:59.569121Z","shell.execute_reply.started":"2022-07-06T19:43:59.537390Z","shell.execute_reply":"2022-07-06T19:43:59.567940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.7 MASK AREA BY ORGAN

\n\n---\n\n
OBSERVATIONS\n* Kidney's are the smallest by total area but the FTUs are individually larger than the Lung FTUs\n* Lung's are the next smallest by total area and have the second smallest average FTUs (more FTUs per image than Kidney)\n* Spleen is the next smallest by total area... however they are the largest FTUs on average\n* Prostate is the next largest and has the largest individual FTUs \n* Large Intestine constitutes the largest percentage of the respective images... however, they are actually the smallest FTUs on average individually.","metadata":{}},{"cell_type":"code","source":"examine_id(train_df, ex_id=train_df[train_df.organ==\"lung\"].id.sample(10).values[0])","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:43:59.570378Z","iopub.execute_input":"2022-07-06T19:43:59.570754Z","iopub.status.idle":"2022-07-06T19:44:03.856251Z","shell.execute_reply.started":"2022-07-06T19:43:59.570713Z","shell.execute_reply":"2022-07-06T19:44:03.854936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[\"mask_area_pct\"] = train_df.rle.progress_apply(lambda x: rle_decode(x, (3000,3000)).sum()/(3000**2))\n\nfig = px.histogram(train_df, \"mask_area_pct\", color=\"organ\", title=\"Mask Area Percentage By Organ Type\", color_discrete_map=O2C_HEX_MAP, barmode=\"group\", nbins=10)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:44:03.857671Z","iopub.execute_input":"2022-07-06T19:44:03.858032Z","iopub.status.idle":"2022-07-06T19:44:08.726153Z","shell.execute_reply.started":"2022-07-06T19:44:03.857995Z","shell.execute_reply":"2022-07-06T19:44:08.725089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

4.8 WHY DO WE HAVE JSON FILES AND RLE?

\n\n---\n\n
OBSERVATIONS\n* The RLE is the entire binary mask as a simple compressed string\n* The JSON contains the description of the masks as a collection (list) of polygons representing the respective masks.\n * A polygon is represented as a collection (list) of vertices.\n * A vertex is a point represented by a pair of values –– the x,y position\n * If you took these polygons and plotted them alongside the RLE decoded mask, they SHOULD be the same. I think it's simply another representation for us to use.\n\nNOTE: The RLE is more accurate than the Polygon Mask and should be used.\n\n
\n\n","metadata":{}},{"cell_type":"code","source":"# Use first row as example --> ex_id=10044\nex_row = train_df[train_df.id==10044]\n\nprint(\"\\n... First – Plot the image and the RLE overlapped ontop ...\\n\")\nexamine_id(ex_row)\n\nprint(\"\\n\\n\\n... Second – Load the .json file and display mask polygons ...\\n\")\nex_poly = load_json_to_dict(ex_row.lbl_path.values[0])\nraw_img = np.array(cv2.imread(ex_row.img_path[0])[..., ::-1])\nimg = np.zeros_like(raw_img)\ncolor_space = np.linspace(235, 20, len(ex_poly), dtype=int)\nfor i, ex_p in enumerate(ex_poly):\n img = cv2.fillPoly(img, [np.expand_dims(np.array(ex_p, np.int32), axis=0)], color=(int(color_space[i]),245,245))\nplt.figure(figsize=(20,20))\nplt.title(\"Mask Polygons\", fontweight=\"bold\")\nplt.imshow(img)\nplt.show()\n\nprint(\"\\n\\n\\n Lastly – Compare mask polygons to rle decoded mask\")\nrle_mask = rle_decode(ex_row.rle.values[0], (ex_row.img_width.values[0], ex_row.img_height.values[0]))\njson_mask = np.where(img>0, 1, 0)[..., 0].astype(np.float32)\ndiff_mask = np.abs(rle_mask-json_mask).astype(np.uint8)\n\nplt.figure(figsize=(20,20))\nplt.title(f\"Number of Non-Equal Pixels = {(rle_mask!=json_mask).sum()}\", fontweight=\"bold\")\nplt.imshow(diff_mask, cmap=\"gray\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:44:08.727756Z","iopub.execute_input":"2022-07-06T19:44:08.728295Z","iopub.status.idle":"2022-07-06T19:44:15.665163Z","shell.execute_reply.started":"2022-07-06T19:44:08.728250Z","shell.execute_reply":"2022-07-06T19:44:15.663988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**VISUALIZE DAB STAIN AND HEMATOXYLIN COUNTERSTAIN**\n* DAB is brown and present at locations of enzymatic activity\n* HEMATOXYLIN is light blue when found within cells\n* HEMATOXYLIN is dark blue when found within nuclei","metadata":{"_kg_hide-output":true,"_kg_hide-input":true}},{"cell_type":"code","source":"# original_prostate_img = cv2.imread(train_df.img_path[0])[..., ::-1]\n# resized_prostate_img = cv2.resize(original_prostate_img, (int(3000*(0.4/6.263)),int(3000*(0.4/6.263))))\n\n# plt.figure(figsize=(20,40))\n\n# plt.subplot(2,1,1)\n# plt.imshow(original_prostate_img)\n# plt.title(\"Original Prostate Image (3000x3000) – 0.4\", fontweight=\"bold\")\n\n# plt.subplot(2,1,2)\n# plt.imshow(resized_prostate_img)\n# plt.title(\"Resized Prostate Image (191x191) – 6.263\", fontweight=\"bold\")\n\n# plt.savefig(\"comparison.png\", dpi=300)\n# plt.show()","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-07-06T19:44:15.666590Z","iopub.execute_input":"2022-07-06T19:44:15.666915Z","iopub.status.idle":"2022-07-06T19:44:15.672089Z","shell.execute_reply.started":"2022-07-06T19:44:15.666866Z","shell.execute_reply":"2022-07-06T19:44:15.670966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

\n 5  MODELLING    ⤒\n

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n

5.1 ALL ONES WITH SOME INTELLIGENCE

\n\n---\n\nWe simply submit a mask made up of ones in a circular shape in the centre of the respective image.\n\n
","metadata":{}},{"cell_type":"code","source":"rles = []\nSMART_CIRCLE_FRAC = 0.85\n\nfor i, (img_w, img_h) in enumerate(zip(test_df[\"img_width\"], test_df[\"img_height\"])):\n tmp_img = np.zeros((img_w, img_h))\n tmp_img = cv2.circle(tmp_img, (int(np.round(img_w/2)), int(np.round(img_h/2))), int(np.round((img_w/2)*SMART_CIRCLE_FRAC)), 1, -1)\n rles.append(rle_encode(tmp_img))\n if i==0:\n plt.figure(figsize=(10,10))\n plt.imshow(rle_decode(rles[0], (img_w, img_h)))\n plt.title(\"Example of Smart Circle Mask\", fontweight=\"bold\")\n plt.show()\n\nss_df[\"rle\"] = rles\nss_df = ss_df[[\"id\", \"rle\"]]\nss_df.to_csv(\"submission.csv\", index=False)\ndisplay(ss_df)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T19:44:15.673985Z","iopub.execute_input":"2022-07-06T19:44:15.674514Z","iopub.status.idle":"2022-07-06T19:44:16.430642Z","shell.execute_reply.started":"2022-07-06T19:44:15.674471Z","shell.execute_reply":"2022-07-06T19:44:16.429475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n

\n 9  APPENDIX    ⤒\n

\n\n---\n\nThis section is mostly if not entirely copied from various sources listed below:\n* Yashvardhan Jain's Background Info Notebook","metadata":{}},{"cell_type":"markdown","source":"

9.1 HUMAN BIOMOLECULAR ATLAS PROGRAM (HUBMAP)

\n\n---\n\nHuBMAP aims to create an open, global atlas of the healthy human body at the cellular level, see the [**HuBMAP Consortium**](https://hubmapconsortium.org/about/) and [**Visible Human Massive Open Online Course (VHMOOC)**](https://expand.iu.edu/browse/sice/cns/courses/hubmap-visible-human-mooc). \n\nOne component of this overarching goal is to identify medically relevant functional tissue units (FTUs) within whole slide microscopy images of human tissues. Once these FTUs are detected, information on size, shape, variability in number and location within the tissue samples can be used to help build a spatially accurate and semantically explicit model of the human body.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

9.2 HUMAN PROTEIN ATLAS (HPA)

\n\n---\n\nThe [**Human Protein Atlas**](https://www.proteinatlas.org/) is a Swedish-based program initiated in 2003 with the aim to map all the human proteins in cells, tissues, and organs using an integration of various omics technologies, including antibody-based imaging, mass spectrometry-based proteomics, transcriptomics, and systems biology. All the data in the knowledge resource is open access to allow scientists both in academia and industry to freely access the data for exploration of the human proteome. The images for this competition are from the [**Tissue Atlas**](https://www.proteinatlas.org/humanproteome/tissue), which maps the protein expression profile across human tissues.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

9.3 FUNCTIONAL TISSUE UNITS (FTUs)

\n\n---\n\nBernard de Bono and his team coined the term “functional tissue unit” to support tissue modelling (**[de Bono, 2013](https://www.ncbi.nlm.nih.gov/pubmed/24103658)**) as:\n> “... a three-dimensional block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block”\n\nFor the purposes of Human Reference Atlas (**[Börner, 2021](https://www.nature.com/articles/s41556-021-00788-6)**, **[Godwin, 2021](https://www.biorxiv.org/content/10.1101/2021.11.09.467810v1)**, **[Börner, 2021](https://www.biorxiv.org/content/10.1101/2021.12.30.474265v1)**) construction and usage, **we define an FTU as the smallest level of tissue organization (i.e., a cell population neighborhood) that performs the organ’s major physiological function**. \n* One example of an FTU is the glomerulus found in the outer layer of kidney tissue known as the cortex, which in humans has an area of about 800 mm2 and average depth of about 9 mm (**[Mounier-Vehier, 2002](https://doi.org/10.1046/j.1523-1755.2002.00167.x)**). \n* Glomeruli consist of capillaries that facilitate filtration of waste products out of blood. \n* Normal glomeruli typically range from 100-350 ÎŒm in diameter with a roughly spherical shape (**[Kannan, 2019](https://www.kireports.org/article/S2468-0249(19))**). \n\n
\n\nRefer to Fig. 1 for a zoom sequence from the human body to single-cell level for the kidney.\n\n
\n\n
\n\nSimilarly, FTUs in other organs presented in this competition are: crypt in the large intestine, alveolus in the lung, glandular acinus in the prostate, and white pulp in the spleen, see Fig. 2.\n\n
\n\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n

9.4 TERMINOLOGY

\n\n---\n\n* **Tissue Preservation**: \n * Different types of tissue preservation techniques are used to maintain the integrity of tissue for downstream biomolecular assays analysis.\n* **Fresh Frozen Tissue Preservation**: \n * Fresh frozen tissue is frozen in liquid nitrogen (-190°C) within 30-60 minutes after surgical excision; this type of preservation has been the method of choice for transcriptomics and immunohistochemistry ([Robbe, 2018](https://doi.org/10.1038/gim.2017.241), [Stoeckli, 2007](https://doi.org/10.1016/j.ijms.2006.10.007)).\n* **Formalin Fixed, Paraffin Embedded (FFPE) Tissue Preservation**: \n * FFPE tissue is the preferred method for clinical pathology samples for histology assessment ([Bass, 2014](https://doi.org/10.5858/arpa.2013-0691-RA)).\n* **Periodic acid-Schiff (PAS) Stain Microscopy**: \n * [PAS](https://en.wikipedia.org/wiki/Periodic_acid%E2%80%93Schiff_stain) is a histology stain that detects complex sugars in tissue sections.\n* **Hematoxylin and Eosin (H&E) Stain Microscopy**: \n * [H&E](https://en.wikipedia.org/wiki/H%26E_stain) stain is one of the principal tissue stains used in histology.\n* **IHC staining**: \n * [Immunohistochemical](https://en.wikipedia.org/wiki/Immunohistochemistry) staining is accomplished with antibodies that recognize the target antigen.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

9.5 DATASETS

\n\n---\n\nThe Ground Truth segmentations were manually created and further reviewed by Subject Matter Experts (SMEs) for quality assurance.\n\n* **HPA Data** comprises images of 1 mm diameter tissue microarray cores stained with antibodies visualized with 3,3'-diaminobenzidine (DAB) and counterstained with hematoxylin.\n* **HuBMAP Data** comprises Periodic acid-Schiff (PAS)/hematoxylin and eosin (H&E) stain images of human tissue.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

9.6 RELATED WORK

\n\n---\n\n* Leah L. Godwin, Yingnan Ju, Naveksha Sood, Yashvardhan Jain, Ellen M. Quardokus, Andreas Bueckle, Teri Longacre, Aaron Horning, Yiing Lin, Edward D. Esplin, John W. Hickey, Michael P. Snyder, N. Heath Patterson, Jeffrey M. Spraggins, Katy Börner. “Robust and generalizable segmentation of human functional tissue units”; bioRxiv 2021.11.09.467810; doi: https://doi.org/10.1101/2021.11.09.467810\n* Sheehan, Susan M., and Ron Korstanje. 2018. “Automatic Glomerular Identification and Quantification of Histological Phenotypes Using Image Analysis and Machine Learning.” American Journal of Physiology - Renal Physiology 315 (6): F1644–51. https://doi.org/10.1152/ajprenal.00629.2017.\n* Gallego, Jaime, Anibal Pedraza, Samuel Lopez, Georg Steiner, Lucia Gonzalez, Arvydas Laurinavicius, and Gloria Bueno. 2018. “Glomerulus Classification and Detection Based on Convolutional Neural Networks.” Journal of Imaging 4 (1): 20. https://doi.org/10.3390/jimaging4010020.\n* Govind, Darshana, Brandon Ginley, Brendon Lutnick, John E. Tomaszewski, and Pinaki Sarder. 2018. “Glomerular Detection and Segmentation from Multimodal Microscopy Images Using a Butterworth Band-Pass Filter.” In Medical Imaging 2018: Digital Pathology, 10581:1058114. International Society for Optics and Photonics. https://doi.org/10.1117/12.2295446.\n* Bukowy, John D., Alex Dayton, Dustin Cloutier, Anna D. Manis, Alexander Staruschenko, Julian H. Lombard, Leah C. Solberg Woods, Daniel A. Beard, and Allen W. Cowley. 2018. “Region-Based Convolutional Neural Nets for Localization of Glomeruli in Trichrome-Stained Whole Kidney Sections.” Journal of the American Society of Nephrology 29 (8): 2081–88. https://doi.org/10.1681/ASN.2017111210.\n* Kannan, Shruti, Laura A. Morgan, Benjamin Liang, McKenzie G. Cheung, Christopher Q. Lin, Dan Mun, Ralph G. Nader, et al. 2019. “Segmentation of Glomeruli Within Trichrome Images Using Deep Learning.” Kidney International Reports 4 (7): 955–62. https://doi.org/10.1016/j.ekir.2019.04.008.\n* Nassim Bouteldja, Barbara M. Klinkhammer, Roman D. BĂŒlow, Patrick Droste, Simon W. Otten, Saskia Freifrau von Stillfried, Julia Moellmann, Susan M. Sheehan, Ron Korstanje, Sylvia Menzel, Peter Bankhead, Matthias Mietsch, Charis Drummer, Michael Lehrke, Rafael Kramann, JĂŒrgen Floege, Peter Boor, Dorit Merhof. “Deep Learning–Based Segmentation and Quantification in Experimental Kidney Histopathology”; JASN Jan 2021 32 (1) 52-68; https://doi.org/10.1681/ASN.2020050597\n* Brandon Ginley, Brendon Lutnick, Kuang-Yu Jen, Agnes B. Fogo, Sanjay Jain, Avi Rosenberg, Vighnesh Walavalkar, Gregory Wilding, John E. Tomaszewski, Rabi Yacoub, Giovanni Maria Rossi, Pinaki Sarder.“Computational Segmentation and Classification of Diabetic Glomerulosclerosis”; JASN Oct 2019, 30 (10) 1953-1967; https://doi.org/10.1681/ASN.2018121259","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}