{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Please **upvote** the notebook if you like the work done here. This will motivate me to make more of such notebook :)"},{"metadata":{},"cell_type":"markdown","source":"# This is a good consolidated source for anyone to get started with the domain understanding of the problem statement"},{"metadata":{},"cell_type":"markdown","source":"# Commonly used domain terms"},{"metadata":{},"cell_type":"markdown","source":"# Capillary\n\nA capillary is a small blood vessel from 5 to 10 micrometers (μm) in diameter, and having a wall of one endothelial cell thick. They are the smallest blood vessels in the body: they convey blood between the arterioles and venules. These microvessels are the site of the exchange of many substances with the interstitial fluid surrounding them.\n\n\n# Diffusion Distance\n\nIntuitively, Diffusion Distance between a pair of points (a, b) can be thought of as following: We create a unit impulse function centered at 'a'(imagine a fixed amount of energy concentrated at a point), and we let it diffuse for a period of time t. We create another impulse function centered at 'b'', and we also let it diffuse for a period time t. In the end, we look at the difference (measured by L2-norm) between the two distributions. And that is our Diffusion Distance.\n\nMathematically, this is described by:\n"},{"metadata":{},"cell_type":"markdown","source":"# Periodic acid-Schiff (PAS) Stain Microscopy\n\nPAS is a histology stain that detects complex sugars in tissue sections. Periodic acid is used to break specific bonds within these sugars. The resulting aldehydes react with the Schiff reagent to produce the purple-magenta color exhibited by these images. Glomeruli can be observed as the circular areas of dark stain.\n\n\n# Glomeruli\n\nGlomeruli consist of capillaries that facilitate the filtration of waste products out of blood. Normal glomeruli typically range from 100-350 μm in diameter with a roughly spherical shape.\n\nGlomeruli contain 4 cell types: \na. Parietal epithelial cells that form Bowman’s capsule\nb. Podocytes cover the outer layer of the filtration barrier\nc. Fenestrated endothelial cells that are coated with a glycolipid\nd. Glycoprotein matrix called glycocalyx that are in direct contact with blood and mesangial cells that occupy the space between the capillary blood vessel loops and are stained by the colorimetric histological stain called Periodic acid-Schiff (PAS) stain"},{"metadata":{},"cell_type":"markdown","source":"# Functional Tissue Unit (FTU)\n\na 3D block of cells centered around a capillary, such that each cell in this block is within diffusion distance from any other cell in the same block” (de Bono, 2013) One example of an FTU is the glomerulus found in the outer layer of kidney tissue known as the cortex, which in humans has an area of about 800 mm2 and an average depth of about 9 mm.\n"},{"metadata":{},"cell_type":"markdown","source":"# Dataset\n\nThe dataset is comprised of very large (>500MB - 5GB) TIFF files. The training set has 8, and the public test set has 5. The private test set is larger than the public test set.\n\nThe training set includes annotations in both RLE-encoded and unencoded (JSON) forms. The annotations denote segmentations of glomeruli."},{"metadata":{},"cell_type":"markdown","source":"# Glomeruli Segmentation Masks\n\nThe glomeruli segmentation masks are a mix of manually and deep learning (DL) generated annotations in a slightly modified geoJSON format. The JavaScript Object Notation (JSON) file lists all glomeruli identified for each of the 11 + 9 tissue sections. The position and shape of a glomeruli is represented by a set of coordinates. The “detection_score”, present only in the DL generated annotations, is a measure the DL model used during detection.\n\nEach item in the JSON list is an annotation with the following pertinent fields:\n\n- “geometry”:\n - “geometry/type”: All are “Polygon”\n - geometry/coordinates”: A list of each polygon vertex in x,y order\n- “properties”:\n - “properties/classification”:\n -“properties/classification/name”: Annotation class (in this case all are “Glomerulus”)\n - “properties/measurements”: list of key,value pairs for some quantitative property of the annotation. For annotations generated by the DL model, this includes “detection_score”."},{"metadata":{},"cell_type":"markdown","source":"# TIFF File format\n\n\nTag Image File Format, abbreviated TIFF or TIF, is a computer file format for storing raster graphics images, popular among graphic artists, the publishing industry,and photographers. TIFF is widely supported by scanning, faxing, word processing, optical character recognition, image manipulation, desktop publishing, and page-layout applications. The format was created by Aldus Corporation for use in desktop publishing. It published the latest version 6.0 in 1992, subsequently updated with an Adobe Systems copyright after the latter acquired Aldus in 1994. Several Aldus or Adobe technical notes have been published with minor extensions to the format, and several specifications have been based on TIFF 6.0, including TIFF/EP (ISO 12234-2), TIFF/IT (ISO 12639), TIFF-F (RFC 2306) and TIFF-FX (RFC 3949).\n\nTIFF is a flexible, adaptable file format for handling images and data within a single file, by including the header tags (size, definition, image-data arrangement, applied image compression) defining the image's geometry. A TIFF file, for example, can be a container holding JPEG (lossy) and PackBits (lossless) compressed images. A TIFF file also can include a vector-based clipping path (outlines, croppings, image frames). The ability to store image data in a lossless format makes a TIFF file a useful image archive, because, unlike standard JPEG files, a TIFF file using lossless compression (or none) may be edited and re-saved without losing image quality. This is not the case when using the TIFF as a container holding compressed JPEG. Other TIFF options are layers "},{"metadata":{},"cell_type":"markdown","source":"# Metric - Dice Coefficient"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Image Credits: https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient\n\nThe evaluation metric of this competition is Dice Coefficient. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:\n\n\n $\\Large \\frac{2*|X ∩ Y|}{|X|+|Y|}$ \n\n\nwhere X is the predicted set of pixels and Y is the ground truth.\n\nDice coefficient, is a statistical tool which measures the similarity between two sets of data. This index has become arguably the most broadly used tool in the validation of image segmentation algorithms created with AI, but it is a much more general concept which can be applied sets of data for a variety of applications including NLP.\n"},{"metadata":{},"cell_type":"markdown","source":"# Jaccard Score"},{"metadata":{},"cell_type":"markdown","source":"The Jaccard similarity index (sometimes called the Jaccard similarity coefficient) compares members for two sets to see which members are shared and which are distinct. It’s a measure of similarity for the two sets of data, with a range from 0% to 100%. The higher the percentage, the more similar the two populations. Although it’s easy to interpret, it is extremely sensitive to small samples sizes and may give erroneous results, especially with very small samples or data sets with missing observations.\n\nThe formula to find the Index is:\n\nJaccard Index = (the number in both sets) / (the number in either set) * 100\n\nIn Steps, that’s:\n\n1. Count the number of members which are shared between both sets.\n2. Count the total number of members in both sets (shared and un-shared).\n3. Divide the number of shared members (1) by the total number of members (2).\n4. Multiply the number you found in (3) by 100."},{"metadata":{},"cell_type":"markdown","source":"# Relation between Dice Coefficient and Jaccard Score.\n\n\n $\\Large D = \\frac{2*|X ∩ Y|}{|X|+|Y|}$ \n
\n $\\Large J = \\frac{|X ∩ Y|}{|X|+|Y|-|X ∩ Y|}$ \n
\n $\\Large D = \\frac{2J}{J+1}$ \n
\n $\\Large J = \\frac{D}{2-D}$ "},{"metadata":{},"cell_type":"markdown","source":"# Main Task \n\nSo basically our task here in this competition is to train a segmentation model that takes PAS kidney image input and identify the segments of glomeruli FTU in the PAS stained microscopy data"},{"metadata":{},"cell_type":"markdown","source":"Please comment on anything that I may have left out or wrongly written."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}