\n \n \n This image was produced using masks and tiling methods from Jirka Borovec and custom toolkit for generating artifacts from Systems Imaging Bioinformatics Lab at the University of Michigan \n \n
\n
HuBMAP + HPA - Hacking the Human Body
\n\n# 1. Introduction\n\n### 1.1. About this notebook\n\nIn this notebook I will **merely showcase** work of a team that worked on [Stress Testing Pathology Models with Generated Artifacts](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8721870/pdf/JPI-12-54.pdf) paper and [custom toolkit](https://github.com/Systems-Imaging-Bioinformatics-Lab/histopath_failure_modes) they have developed for generating artifacts on tissue imaging. **I HAVE NOT DONE ANY WORK IN THIS NOTEBOOK**. I found this paper, read it and really liked it. Also I've used the [following notebook](https://www.kaggle.com/code/jirkaborovec/ftus-segm-decompose-large-images-to-tiles/notebook) by [Jirka Borovec](https://www.kaggle.com/jirkaborovec) for spliting image into tiles. As well as dataset he made public on [image annotation masks](https://www.kaggle.com/datasets/jirkaborovec/hacking-the-human-body-annotation-masks).\n\n**Again I will say it. This notebook does not showcase my work. It is work of [Stress Testing Pathology Models with Generated Artifacts](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8721870/pdf/JPI-12-54.pdf) paper and the people who worked hard on it.**\n\n**I have only found it and wanted to share and showcase their work.**\n\n### 1.2. Materials used directly\n#### Notebooks:\n- [FTUs⚕️Segm: decompose 🖽 large images to tiles](https://www.kaggle.com/code/jirkaborovec/ftus-segm-decompose-large-images-to-tiles/notebook) by [Jirka Borovec](https://www.kaggle.com/jirkaborovec)\n\n#### Papers:\n- [Stress Testing Pathology Models with Generated Artifacts](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8721870/pdf/JPI-12-54.pdf) by **Nicholas Chandler Wang, Jeremy Kaplan, Joonsang Lee, Jeffrey Hodgin, Aaron Udager, Arvind Rao**.\n\n Custom toolkit they developed can be found here: [Github repo link](https://github.com/Systems-Imaging-Bioinformatics-Lab/histopath_failure_modes) /Link in the Conclusions section under Availability headline/","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"# Here we import/install some libraries we will be using throughout this notebook\nimport numpy as np\nfrom scipy import interpolate\nfrom scipy.ndimage import morphology\nfrom scipy.stats import multivariate_normal\n\nfrom PIL import Image, ImageDraw\nfrom skimage import color\nimport cv2\n\nimport matplotlib.pyplot as plt\n\n!pip install -q deconvolution\nfrom deconvolution import Deconvolution\n\nimport os\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-01T03:06:24.469048Z","iopub.execute_input":"2022-07-01T03:06:24.470166Z","iopub.status.idle":"2022-07-01T03:06:36.386311Z","shell.execute_reply.started":"2022-07-01T03:06:24.470104Z","shell.execute_reply":"2022-07-01T03:06:36.384757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Image tiling method and mask annotation dataset by [Jirka Borovec](https://www.kaggle.com/jirkaborovec) from [FTUs⚕️Segm: decompose 🖽 large images to tiles](https://www.kaggle.com/code/jirkaborovec/ftus-segm-decompose-large-images-to-tiles/notebook) notebook","metadata":{}},{"cell_type":"code","source":"def tile_image(p_img, folder, size: int = 1024) -> list:\n w = h = size\n im = np.array(Image.open(p_img))\n # https://stackoverflow.com/a/47581978/4521646\n tiles = [im[i:(i + h), j:(j + w), ...] for i in range(0, im.shape[0], h) for j in range(0, im.shape[1], w)]\n idxs = [(i, (i + h), j, (j + w)) for i in range(0, im.shape[0], h) for j in range(0, im.shape[1], w)]\n name, _ = os.path.splitext(os.path.basename(p_img))\n files = []\n for k, tile in enumerate(tiles):\n if tile.shape[:2] != (h, w):\n tile_ = tile\n tile = np.zeros_like(tiles[0])\n tile[:tile_.shape[0], :tile_.shape[1], ...] = tile_\n p_img = os.path.join(folder, f\"{name}_{k:03}.png\")\n Image.fromarray(tile).save(p_img)\n files.append(p_img)\n return files, idxs\n\nDATASET_IMAGES = \"/kaggle/input/hubmap-organ-segmentation/train_images/\"\nDATASET_MASKS = \"/kaggle/input/hacking-the-human-body-annotation-masks/train_masks\"\n\n!mkdir -p /kaggle/working/temp/images\n!mkdir -p /kaggle/working/temp/masks\n\ntiles_img, _ = tile_image(os.path.join(DATASET_IMAGES, \"10044.tiff\"), \"/kaggle/working/temp/images\", size = 512)\ntiles_seg, idxs = tile_image(os.path.join(DATASET_MASKS, \"10044.png\"), \"/kaggle/working/temp/masks\", size = 512)\n\n!ls -lh /kaggle/working/temp/images\n!ls -lh /kaggle/working/temp/masks","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-01T03:06:36.388927Z","iopub.execute_input":"2022-07-01T03:06:36.389270Z","iopub.status.idle":"2022-07-01T03:06:43.791043Z","shell.execute_reply.started":"2022-07-01T03:06:36.389236Z","shell.execute_reply":"2022-07-01T03:06:43.789723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nb_tiles_sqrt = int(np.sqrt(len(tiles_img)))\n\nfig, axes = plt.subplots(nrows = nb_tiles_sqrt, ncols = nb_tiles_sqrt, figsize = (9, 9))\nfor i, (p_img, p_seg) in enumerate(zip(tiles_img, tiles_seg)):\n img = plt.imread(p_img)\n mask = np.array(Image.open(p_seg))\n axes[i // nb_tiles_sqrt, i % nb_tiles_sqrt].imshow(color.label2rgb(mask, img, bg_label = 0, bg_color = (1.,1.,1.), alpha = 0.25))\n axes[i // nb_tiles_sqrt, i % nb_tiles_sqrt].set_axis_off()\n\nfig.suptitle(\"Original Image\", fontsize = 18)\nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:06:43.793849Z","iopub.execute_input":"2022-07-01T03:06:43.794907Z","iopub.status.idle":"2022-07-01T03:06:49.870332Z","shell.execute_reply.started":"2022-07-01T03:06:43.794837Z","shell.execute_reply":"2022-07-01T03:06:49.868612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. [Histopath Failure Modes](https://github.com/Systems-Imaging-Bioinformatics-Lab/histopath_failure_modes.git) from [Systems Imaging Bioinformatics Lab at the University of Michigan](https://sibl.lab.medicine.umich.edu)","metadata":{}},{"cell_type":"code","source":"# Now we clone Histopath Failure Modes from Systems Imaging Bioinformatics Lab at the University of Michigan\n!git clone https://github.com/Systems-Imaging-Bioinformatics-Lab/histopath_failure_modes.git\n%cd histopath_failure_modes","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:06:49.874689Z","iopub.execute_input":"2022-07-01T03:06:49.875153Z","iopub.status.idle":"2022-07-01T03:07:04.113607Z","shell.execute_reply.started":"2022-07-01T03:06:49.875108Z","shell.execute_reply":"2022-07-01T03:07:04.112521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Now in their work they have studied 7 artifacts. \n\n- Bubbles\n- Tissue fold\n- Illumination\n- Marker line\n- Sectioning\n- Stain alteration\n- Tissue tears\n\n
\n \n \n This image is displayed in Methods section of \"Stress Testing Pathology Models with Generated Artifacts\" work from Systems Imaging Bioinformatics Lab at the University of Michigan \n \n
","metadata":{}},{"cell_type":"code","source":"# Importing custom artifact generation toolkit from \n# Systems Imaging Bioinformatics Lab at the University of Michigan\nfrom image_manipulation import img_manip","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:04.114935Z","iopub.execute_input":"2022-07-01T03:07:04.115278Z","iopub.status.idle":"2022-07-01T03:07:04.122325Z","shell.execute_reply.started":"2022-07-01T03:07:04.115242Z","shell.execute_reply":"2022-07-01T03:07:04.121081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now here let's define some functions for ease of use throughout this notebook.\ndef plot_tissue_with_generated_artifact(img, mask, augmented_img, augmentation):\n img = np.array(img)\n mask = np.array(mask)\n augmented_img = np.array(augmented_img)\n fig, axs = plt.subplots(1, 2, figsize = (16, 8), sharex = True, sharey = True)\n\n axs[0].imshow(color.label2rgb(mask, img, bg_label = 0, bg_color = (1.,1.,1.), alpha = 0.25))\n axs[0].set_title(\"Original image\", fontsize = 14)\n axs[0].axis('off')\n\n axs[1].imshow(color.label2rgb(mask, augmented_img, bg_label = 0, bg_color = (1.,1.,1.), alpha = 0.25))\n axs[1].set_title(f\"{augmentation}\", fontsize = 14)\n axs[1].axis('off')\n plt.show();\n \n# Following function is for inspecting what is happening on image when 'Illumination' is applied as \n# we can't directly observe the changes made.\ndef display_color_histogram(img, augmented_img, augmentation):\n augmented_img = np.array(augmented_img)\n colors = (\"red\", \"green\", \"blue\")\n channel_ids = (0, 1, 2)\n\n fig, axs = plt.subplots(1, 2, figsize = (18, 6), sharex = True, sharey = True)\n\n for channel_id, c in zip(channel_ids, colors):\n img = np.asarray(img)\n histogram, bin_edges = np.histogram(img[:, :, channel_id], bins = 12, range = (0, 256))\n axs[0].plot(bin_edges[0:-1], histogram, color = c, label = c)\n\n axs[0].spines['top'].set_visible(False)\n axs[0].spines['right'].set_visible(False)\n axs[0].set_title(\"Original image color histogram\", fontsize = 16)\n\n for channel_id, c in zip(channel_ids, colors):\n img = np.asarray(img)\n histogram, bin_edges = np.histogram(augmented_img[:, :, channel_id], bins = 12, range = (0, 256))\n axs[1].plot(bin_edges[0:-1], histogram, color = c, label = c)\n\n axs[1].spines['top'].set_visible(False)\n axs[1].spines['right'].set_visible(False)\n axs[1].set_title(f\"{augmentation}\", fontsize = 16)\n\n fig.text(0.5, 0.04, 'Color value', ha='center', fontsize = 12)\n fig.text(0.04, 0.5, 'Pixel count', va='center', rotation='vertical', fontsize = 12);","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:10:52.574793Z","iopub.execute_input":"2022-07-01T03:10:52.575285Z","iopub.status.idle":"2022-07-01T03:10:52.596280Z","shell.execute_reply.started":"2022-07-01T03:10:52.575248Z","shell.execute_reply":"2022-07-01T03:10:52.595007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's only use one tile we got from using Jirka Borovec's notebook and masks he made accessible\nfig, ax = plt.subplots(figsize = (14, 8))\n\nimg = plt.imread(tiles_img[8])\nmask = np.array(Image.open(tiles_seg[8]))\n\nax.imshow(color.label2rgb(mask, img, bg_label = 0, bg_color = (1.,1.,1.), alpha = 0.25))\nax.set_axis_off()","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:04.149373Z","iopub.execute_input":"2022-07-01T03:07:04.149900Z","iopub.status.idle":"2022-07-01T03:07:04.516994Z","shell.execute_reply.started":"2022-07-01T03:07:04.149850Z","shell.execute_reply":"2022-07-01T03:07:04.515667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.1. Bubbles\n\n**In THEIR WORK they have studied that** `Air bubbles can be introduced by air getting underneath the coverslip, while nuclear bubbles can be caused by heat or other conditions that cause protein coagulation.`","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n# Optional parameters for add_bubbles() method includes:\n # random_seed: int The random seed for numpy for consistent generation\n # nBubbles: int (+) The number of bubbles to generate in the image\n # maxWidth: float (+) The maximum width of the randomized bubbles (roughly), in pixels\n # alpha: float (0-1) The alpha transparency of the bubble layer (1 = opaque, 0 = transparent)\n # edgeWidth: float (+) The width of the darker edge of the bubble, in pixels\n # edgeColorMult: The RGB multiplier of the edge of the bubble \n # 3 float vector -Relative to the mean RGB color of the image\n # rgbVal: 3 float vector The RGB color of the bubbles\n\n# Here the default value fol nBubbles is 25\nimg_with_bubbles = img_manip.add_bubbles(img)\n\nplot_tissue_with_generated_artifact(img, mask, img_with_bubbles, \"Image with Bubbles\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:04.519027Z","iopub.execute_input":"2022-07-01T03:07:04.519983Z","iopub.status.idle":"2022-07-01T03:07:05.791255Z","shell.execute_reply.started":"2022-07-01T03:07:04.519934Z","shell.execute_reply":"2022-07-01T03:07:05.789726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's play around with different nBubbles value\nnBubbles = np.arange(10, 40, 5)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(3, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_tear = np.array(img_manip.add_bubbles(img, nBubbles = nBubbles[i]))\n ax.imshow(image_with_tear)\n ax.set_title(f\"Bubbles: nBubbles = {nBubbles[i]}\", fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding bubbles to image tile\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:05.792782Z","iopub.execute_input":"2022-07-01T03:07:05.793161Z","iopub.status.idle":"2022-07-01T03:07:10.450618Z","shell.execute_reply.started":"2022-07-01T03:07:05.793127Z","shell.execute_reply":"2022-07-01T03:07:10.449260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's play around with different maxWidth value\n# This parameter sets maximum width of randomly generated bubbles. Default value is 50\nmaxWidth = np.arange(50, 70, 5)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(2, 2, figsize = (16, 8))\nfor i, ax in enumerate(axs.flat):\n image_with_tear = np.array(img_manip.add_bubbles(img, maxWidth = maxWidth[i]))\n ax.imshow(image_with_tear)\n ax.set_title(f\"Bubbles: maxWidth = {maxWidth[i]}\", fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding bubbles to image tile. Playing around with maxWidth parameter\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:10.455500Z","iopub.execute_input":"2022-07-01T03:07:10.456385Z","iopub.status.idle":"2022-07-01T03:07:13.935241Z","shell.execute_reply.started":"2022-07-01T03:07:10.456327Z","shell.execute_reply":"2022-07-01T03:07:13.933721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.2. Tissue fold\n\n**In their work they have studied that** `Tissue folds occur when the thin tissue slice is not uniformly spread out and folds back over itself, often due to heterogeneous material properties within the sample.`[[16]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6097380/)","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n# Optional parameters include:\n # sampArr: numpy arr A numpy array the size of the input image\n # - used for recursion - if the input image is not where the tissue should be sampled from\n # sampSpl: n x 2 numpy arr You can optionally specify the sampled spline (non-random)\n # - used for recursion \n # inPts: n x 2 numpy arr Used to prespecify the handle points of the spline\n # - Note: this is not random\n # random_seed: int The random seed for numpy for consistent generation\n # scaleXY: 2 float vector Used to scale the sampling bounding box, if the sample region should be resized\n # Defaults to no change between original and sampling\n # large scale = larger sample region\n # width: float (+) The width of the tissue fold region, in pixels\n # sampShiftXY: 2 int vec You can optionally specify the direction to shift the spline region\n # Defaults to a random direction at most half the size of the image\n # randEdge: bool Whether to add some randomness to the edge of the tissue fold region\n # Defaults to off\n # nLayers: int (+) Number of tissue layers to add to the image\n # Runs the function recursively, defaults to 2 layers\n # nPts: int (+) The number of random handle points in the spline\n # endEdge: bool Whether or not the start of the spline should be on the edge of the image\n # int(0,1,2,3) If endEdge is a nonnegative int, it specifies which edge the spline stops on\n # 0 = Left, 1 = Top, 2 = Right, 3 = Bottom\n # int(-4,-3,-2,-1) If endEdge is a negative int, it specifies which edge the spline stops on \n # relative to the start\n # -4 = Same, -3 = End is 1 step clockwise (e.g. Bottom -> Left)\n # -2 = Opposite side, -1 = End is 1 step counterclockwise (e.g. Bottom -> Right)\n # Defaults to -2\nimage_with_tissue_fold = img_manip.add_fold(img, random_seed = 0)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_tissue_fold, \"Image with Tissue Folds\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:07:13.937706Z","iopub.execute_input":"2022-07-01T03:07:13.938307Z","iopub.status.idle":"2022-07-01T03:07:14.609930Z","shell.execute_reply.started":"2022-07-01T03:07:13.938254Z","shell.execute_reply":"2022-07-01T03:07:14.608702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.3. Illumination","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n# Optional parameters include:\n # random_seed: int The random seed for numpy for consistent generation\n # maxCov: float (+) The maximum covariance (governs the size of the distributions)\n # nNorms: int (+) The number of Gaussian distributions used to build the uneven illumination\n # scaleMin: float (<1) The minimum for the random factor used to adjust the illumination\n # scaleMax: float (>1) The maximum for the random factor used to adjust the illumination\n # minCovScale: float The minimum on the range of sizes across the set of distributions\n # Recommend >0 & ≤1\n # minDiagCovScale: float Affects the diagonal of the covariance matrix, and the minimum relative size \n # Recommend >0 & ≤1 of the two components compared to the scaled max covariance\n # maxCrCovScale: float The maximum relative cross covariance (1 = straight line, 0 = uncorrelated)\n # Recommend >0 & ≤1 Affects the shape of the distributions, a higher number means more eccentricity\n\n# Here default values for maxCov = 15, nNorms = 3, scaleMin = .8 and scaleMax = 1.1\nimage_with_illumination = img_manip.add_illumination(img)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_illumination, \"Image with evenly generated illumination\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:10:16.638920Z","iopub.execute_input":"2022-07-01T03:10:16.639479Z","iopub.status.idle":"2022-07-01T03:10:17.322214Z","shell.execute_reply.started":"2022-07-01T03:10:16.639441Z","shell.execute_reply":"2022-07-01T03:10:17.320701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's observe color histogram of both images above\ndisplay_color_histogram(img, image_with_illumination, \"Image with Illumination\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:10:57.275196Z","iopub.execute_input":"2022-07-01T03:10:57.276431Z","iopub.status.idle":"2022-07-01T03:10:57.645864Z","shell.execute_reply.started":"2022-07-01T03:10:57.276372Z","shell.execute_reply":"2022-07-01T03:10:57.644288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now let's play around with different values for scaleMin\n# Reminder that the default value is .8\nscaleMin = np.arange(.2, 1, .1)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(4, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_illumination = np.array(img_manip.add_illumination(img, scaleMin = scaleMin[i]))\n ax.imshow(image_with_illumination)\n ax.set_title(\"Illumination: scaleMin = %.1f\" % scaleMin[i], fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding illumination to image tile, playing around with scaleMin parameter\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:12:43.571859Z","iopub.execute_input":"2022-07-01T03:12:43.572259Z","iopub.status.idle":"2022-07-01T03:12:45.702095Z","shell.execute_reply.started":"2022-07-01T03:12:43.572223Z","shell.execute_reply":"2022-07-01T03:12:45.700902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Here we try applying different values for scaleMax\n# Reminder that the default value is 1.1\nscaleMax = np.arange(1.1, 2.3, .2)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(3, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_illumination = np.array(img_manip.add_illumination(img, scaleMax = scaleMax[i]))\n ax.imshow(image_with_illumination)\n ax.set_title(\"Illumination: scaleMax = %.1f\" % scaleMax[i], fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding illumination to image tile, playing around with scaleMax parameter\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:14:37.112057Z","iopub.execute_input":"2022-07-01T03:14:37.112502Z","iopub.status.idle":"2022-07-01T03:14:38.471858Z","shell.execute_reply.started":"2022-07-01T03:14:37.112467Z","shell.execute_reply":"2022-07-01T03:14:38.470545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.4. Marker line\n\n**From their work. In the methods section of their paper:**\n\n`Pathologists often use marking pens to delineate regions of interest on a histology slide, for measurement, or other uses. While a pathologist would recognize that these structures were not a feature of the tissue sample, an algorithm may not be able to accurately distinguish the two. As such, a marker artifact was generated to test algorithms for their sensitivity to this kind of change.`","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n# Optional parameters include:\n # random_seed: int The random seed for numpy for consistent generation\n # nPts: int The number of random handle points in the spline\n # sampSpl: n x 2 numpy arr You can optionally specify the sampled spline (non-random)\n # - Note: should be sampled densely enough (i.e. at least every pixel)\n # inPts: n x 2 numpy arr Used to prespecify the handle points of the spline\n # - Note: this is not random\n # width: float (+) The width of the marker line, in pixels\n # alpha: float (0-1) The alpha transparency of the marker layer (1 = opaque, 0 = transparent)\n # rgbVal: 3 uint8 vector The RGB color of the marker can be optionally specified\n # >=0 <=255\n # rgbRange: 3 x 2 uint8 arr The RGB range of the randomized color [[minR,maxR],[minG,maxG],[minB,maxB]]\n # >=0 <=255 - Leans more blue heavy by default\nimage_with_marker = img_manip.add_marker(img)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_marker, \"Image with marker lines\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:16:43.849256Z","iopub.execute_input":"2022-07-01T03:16:43.849759Z","iopub.status.idle":"2022-07-01T03:16:44.431144Z","shell.execute_reply.started":"2022-07-01T03:16:43.849723Z","shell.execute_reply":"2022-07-01T03:16:44.429803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.5. Sectioning\n\n***I'm sure I do not have to mention again at this point. But of course the following explanation is again from THEIR WORK and STUDY:***\n\n`Sectioning artifacts are the result of unevenly cut regions of the tissue section during microtomy leading to varying thicknesses, and thus varying staining, across the slide`","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n # width: float The width of the sectioning region, in pixels\n # random_seed: int The random seed for numpy for consistent generation\n # scaleMin: float The minimum level of saturation allowed at random\n # -Note: this scales based off of the distance from the spline\n # scaleMax: float The maximum level of saturtation allowed at random\n # -Note: this scales based off of the distance from the spline\n # randEdge: bool Whether to add some randomness to the edge of the sectioning region\n # Defaults to on\n # sampSpl: n x 2 numpy arr You can optionally specify the sampled spline (non-random)\n # - used for recursion \n # inPts: n x 2 numpy arr Used to prespecify the handle points of the spline\n # - Note: this is not random\n # nPts: int The number of random handle points in the spline\n # Defaults to 2\n # endEdge: bool Whether or not the start of the spline should be on the edge of the image\n # int(0,1,2,3) If endEdge is a nonnegative int, it specifies which edge the spline stops on\n # 0 = Left, 1 = Top, 2 = Right, 3 = Bottom\n # int(-4,-3,-2,-1) If endEdge is a negative int, it specifies which edge the spline stops on \n # relative to the start\n # -4 = Same, -3 = End is 1 step clockwise (e.g. Bottom -> Left)\n # -2 = Opposite side, -1 = End is 1 step counterclockwise (e.g. Bottom -> Right)\n # Defaults to -2\nimage_with_sectioning = img_manip.add_sectioning(img)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_sectioning, \"Image with sectioning\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:22:15.914263Z","iopub.execute_input":"2022-07-01T03:22:15.914793Z","iopub.status.idle":"2022-07-01T03:22:16.539365Z","shell.execute_reply.started":"2022-07-01T03:22:15.914755Z","shell.execute_reply":"2022-07-01T03:22:16.537811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_color_histogram(img, image_with_sectioning, \"Image with uneven sectioning applied\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:25:30.060040Z","iopub.execute_input":"2022-07-01T03:25:30.061004Z","iopub.status.idle":"2022-07-01T03:25:30.450920Z","shell.execute_reply.started":"2022-07-01T03:25:30.060948Z","shell.execute_reply":"2022-07-01T03:25:30.449512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Values to play around with\n# The default values for following parameters are width = 240, scaleMin = .5 and scaleMax = .8\nwidth = np.arange(40, 260, 30)\nscaleMin = np.arange(.1, .9, .1)\nscaleMax = np.arange(.8, 0, -.1)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(4, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_sectioning = np.array(img_manip.add_sectioning(img, width = width[i], scaleMin = scaleMin[i], \n scaleMax = scaleMax[i]))\n ax.imshow(image_with_sectioning)\n ax.set_title(\"Sectioning: w = %.1f, min = %.1f, max = %.1f\" % (width[i], scaleMin[i], scaleMax[i]), \n fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding sectioning image tile, playing around with width, scaleMin and scaleMax\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:24:01.157077Z","iopub.execute_input":"2022-07-01T03:24:01.157488Z","iopub.status.idle":"2022-07-01T03:24:02.455343Z","shell.execute_reply.started":"2022-07-01T03:24:01.157454Z","shell.execute_reply":"2022-07-01T03:24:02.453869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.6. Stain alteration","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n # adjFactor: 3 float vec The adjustment factor for each of the three basis vectors \n # (<1 = less stain, 1 = same, >1 = more stain)\n # Element 1: Eosin\n # Element 2: Hematoxylin\n # Element 3: Null (the remaining structure)\n # If set the change won't be random\n # scaleMax: 3 float vector The maximum amount of change (increase or decrease) to the stain levels\n # (>=1)\n # scaleMin: 3 float vector The minimum amount of change (increase or decrease) to the stain levels\n # (>=1)\n # random_seed: int The random seed for numpy for consistent generation\nimage_with_stain = img_manip.add_stain(img)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_stain, \"Image with stain\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:26:54.574233Z","iopub.execute_input":"2022-07-01T03:26:54.574723Z","iopub.status.idle":"2022-07-01T03:27:06.295124Z","shell.execute_reply.started":"2022-07-01T03:26:54.574687Z","shell.execute_reply":"2022-07-01T03:27:06.293724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_color_histogram(img, image_with_stain, \"Image with stain alteration applied\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:28:26.213196Z","iopub.execute_input":"2022-07-01T03:28:26.214017Z","iopub.status.idle":"2022-07-01T03:28:26.555075Z","shell.execute_reply.started":"2022-07-01T03:28:26.213971Z","shell.execute_reply":"2022-07-01T03:28:26.553339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.7. Tissue tears\n\n`Due to technical issues during the cutting process, for example, when hard particles are present in tissue samples or due to vibrations in the knife blade, tears can occur within the cut tissue section.`","metadata":{}},{"cell_type":"code","source":"img = Image.open(tiles_img[8])\nmask = Image.open(tiles_seg[8])\n # sampSpl: n x 2 numpy arr You can optionally specify the sampled spline (non-random)\n # random_seed: int The random seed for numpy for consistent generation\n # nPts: int (+) The number of random handle points in the spline\n # minSpacing: float (+) The minimum for the random spacing between tears, in pixels\n # maxSpacing: float (+) The maximum for the random spacing between tears, in pixels\n # tearStartFactor: The min for where the tear randomly starts along the spline, in percentage & \n # 2 float vec The max for where the tear randomly starts along the spline, in percentage\n # tearEndFactor: The min for where the tear randomly end along the spline, in percentage & \n # 2 float vec The max for where the tear randomly ends along the spline, in percentage\n # dirMin: float (+) The minimum for randomized inline and perpendicular direction max distance in pixels\n # dirMax: float (+) The maximum for randomized inline and perpendicular direction max distance in pixels\n # inLineMax: float (+) You can optionally set the maximum size of the tear in the spline direction\n # perpMax: float (+) You can optionally set the maximum size of the tear in the perpendicular direction\n # ptRadius: float (+) The size of each point's effect on the tear in pixels\n # tearAlpha: float (0-1) The alpha transparency of the tear layer (1 = opaque, 0 = transparent)\n #\n # inLinePercs: You can optionally set your own tear layer structure\n # 2 x n numpy float arr The values are the percentage of distance in the in line direction that the tears take up\n # n = number of layers [.5,.3,.2] means most of the structure of the tear is set early\n # rec. [[-,-,-],[+,+,+]] and later layers fill it out\n # rec. each row should Making the matrix asymmetric betw. the + and -, could give a force component to the tear\n # add up to 1 or -1 Should match the number of layers in the perpendicular side\n # \n # perpPercs: You can optionally set your own tear layer structure\n # 2 x n numpy float arr The values are the % of distance in the perpendicular direction that the tears take up\n # n = number of layers [.5,.3,.2] means most of the structure of the tear is set early\n # rec. [[-,-,-],[+,+,+]] and later layers fill it out\n # rec. each row should Making the matrix asymmetric betw. the + and -, could give a sided-ness to the tear\n # add up to 1 or -1 Should match the number of layers in the inline side\n # \n # l1MinCt: int (+) The first layer is set by number instead of density in the later layers\n # - Minimum # of pts in the first layer\n # l1MaxCt: int (+) The first layer is set by number instead of density in the later layers\n # - Maximum # of pts in the first layer\n # minDensity: The second layer and beyond are set by density instead of #\n # n-1 int (+) vector - Minimum density of points in the 2nd, 3rd, etc. layers\n # n = number of layers\n # maxDensity: The second layer and beyond are set by density instead of #\n # n-1 int (+) vector - Minimum density of points in the 2nd, 3rd, etc. layers\n # edgeAlpha: float (0-1) The alpha transparency of the edge layer (1 = opaque, 0 = transparent)\n # edgeColorMult: The RGB multiplier of the edge of the tear \n # 3 float vector -Relative to the mean RGB color of the image\n # rgbVal: 3 float vector The RGB color of the tear (i.e. background)\n # randEdge: bool Whether to add some randomness to the edge of the tear\n # Defaults to on\nimage_with_tear = img_manip.add_tear(img)\n\nplot_tissue_with_generated_artifact(img, mask, image_with_tear, \"Image tear stain\")","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:30:31.281297Z","iopub.execute_input":"2022-07-01T03:30:31.281781Z","iopub.status.idle":"2022-07-01T03:30:31.840569Z","shell.execute_reply.started":"2022-07-01T03:30:31.281745Z","shell.execute_reply":"2022-07-01T03:30:31.839589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The default values for minSpacing = 20, maxSpacing = 40\nminSpacing = np.arange(10, 70, 10)\nmaxSpacing = np.arange(30, 90, 10)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(3, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_tear = np.array(img_manip.add_tear(img, minSpacing = minSpacing[i], maxSpacing = maxSpacing[i]))\n ax.imshow(image_with_tear)\n ax.set_title(\"Tear: minSpacing = %.1f, maxSpacing = %.1f\" % (minSpacing[i], maxSpacing[i]), \n fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding tear image tile, playing around with min and max Spacing\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:31:37.971992Z","iopub.execute_input":"2022-07-01T03:31:37.973243Z","iopub.status.idle":"2022-07-01T03:31:39.325837Z","shell.execute_reply.started":"2022-07-01T03:31:37.973198Z","shell.execute_reply":"2022-07-01T03:31:39.324655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The default values for dirMin = 10, dirMax = 30\ndirMin = np.arange(10, 70, 10)\ndirMax = np.arange(30, 90, 10)\n\n# Image tile\nimg = Image.open(tiles_img[8])\n\nfig, axs = plt.subplots(3, 2, figsize = (20, 10))\nfor i, ax in enumerate(axs.flat):\n image_with_tear = np.array(img_manip.add_tear(img, dirMin = dirMin[i], dirMax = dirMax[i]))\n ax.imshow(image_with_tear)\n ax.set_title(\"Tear: dirMin = %.1f, dirMax = %.1f\" % (dirMin[i], dirMax[i]), \n fontsize = 14)\n ax.axis('off')\n \nfig.suptitle(\"Adding tear to image tile, playing around with dirMin and dirMax\", fontsize = 16)\nfig.tight_layout();","metadata":{"execution":{"iopub.status.busy":"2022-07-01T03:33:26.886464Z","iopub.execute_input":"2022-07-01T03:33:26.887010Z","iopub.status.idle":"2022-07-01T03:33:27.949298Z","shell.execute_reply.started":"2022-07-01T03:33:26.886971Z","shell.execute_reply":"2022-07-01T03:33:27.947896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Conclusion\n\nThen again, I am very impressed by their work. And I greatly enjoyed reading it. So I wanted to share it. If you are interested in their work then please by all means go to their [toolkit](https://github.com/Systems-Imaging-Bioinformatics-Lab/histopath_failure_modes.git) and [research paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8721870/pdf/JPI-12-54.pdf) and read it. Also do check out [notebook](https://www.kaggle.com/code/jirkaborovec/ftus-segm-decompose-large-images-to-tiles/notebook) used for splitting an image into tiles by [Jirka Borovec](https://www.kaggle.com/jirkaborovec) and [image mask annotation dataset](https://www.kaggle.com/datasets/jirkaborovec/hacking-the-human-body-annotation-masks) he made accessible for public.\n\n**Additional research papers and articles I have found interesting:**\n- [A review of artifacts in histopathology](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6097380/)\n- [Artifacts in Histological and Cytological Preparations](https://www.leicabiosystems.com/knowledge-pathway/artifacts-in-histological-and-cytological-preparations/)\n- [Biopsy and Tissue Processing Artifacts in Oral Mucosal Tissues](https://www.ijhns.com/doi/pdf/10.5005/jp-journals-10001-1102)\n- [Functional tissue units and their primary tissue motifs in multi-scale physiology](https://www.researchgate.net/publication/257533499_Functional_tissue_units_and_their_primary_tissue_motifs_in_multi-scale_physiology)\n- [Description of soft tissue artifacts and related consequences on hindlimb kinematics during canine gait](https://peerj.com/articles/9379/)","metadata":{}}]}
\n 
\n