{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Contents\n\n1. [Introduction](#1)\n1. [About Dataset and data](#2)\n1. [Let see how kidney images looks like](#3)\n1. [Modeling](#4)"},{"metadata":{},"cell_type":"markdown","source":"
\n# Introduction
\n\n\n### Just as the Human Genome Project mapped the entirety of human DNA, the Human BioMolecular Atlas Program (HuBMAP) is a major endeavor. Sponsored by the National Institutes of Health (NIH), HuBMAP is working to catalyze the development of a framework for mapping the human body at a level of glomeruli functional tissue units for the first time in history. \n\n## Hoping to become one of the worldβs largest collaborative biological projects, HuBMAP aims to be an open map of the human body at the cellular level.\n"},{"metadata":{},"cell_type":"markdown","source":"# π π π Upvote if you find this Kernal useful "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"
\n# About Dataset and Data
"},{"metadata":{},"cell_type":"markdown","source":"\n# About Dataset\n\n\n### Size of Data\n\nThe data is huge (24.5 GB) The HuBMAP data used in this hackathon includes 11 fresh frozen and 9 Formalin Fixed Paraffin Embedded (FFPE) PAS kidney images. Glomeruli FTU annotations exist for all 20 tissue samples.\n\nThe dataset is comprised of very large (>500MB - 5GB) TIFF files. The training set has 8, and the public test set has 5. The private test set is larger than the public test set.\n\n### Train test Split \n\nThe training set includes annotations in both RLE-encoded and unencoded (JSON) forms. The annotations denote segmentations of glomeruli.\n\nBoth the training and public test sets also include anatomical structure segmentations. They are intended to help you identify the various parts of the tissue.\n\n\nThe training set includes annotations in both RLE-encoded and unencoded (JSON) forms. The annotations denote segmentations of glomeruli.\n"},{"metadata":{},"cell_type":"markdown","source":"# About Data\n\n\nFile structure\nThe JSON files are structured as follows, with each feature having:\n\nA type (Feature) and object type id (PathAnnotationObject). Note that these fields are the same between all files and do not offer signal.\nA geometry containing a Polygon with coordinates for the feature's enclosing volume\nAdditional properties, including the name and color of the feature in the image.\n\nThe IsLocked field is the same across file types (locked for glomerulus, unlocked for anatomical structure) and is not signal-bearing.\nNote that the objects themselves do NOT have unique IDs. The expected prediction for a given image is an RLE-encoded mask containing ALL objects in the image. The mask, as mentioned in the Evaluation page, should be binary when encoded - with 0 indicating the lack of a masked pixel, and 1 indicating a masked pixel.\n\n\nBoth the training and public test sets also include anatomical structure segmentations. They are intended to help you identify the various parts of the tissue.\n\nWe are provided with following files:\n\nFor each of the 11 training images we have been provided with a JSON file. Each JSON file has:\n\nA type (Feature) and object type id (PathAnnotationObject). Note that these fields are the same between all files and do not offer signal.\nA geometry containing a Polygon with coordinates for the feature's enclosing volume\n\nAdditional properties, including the name and color of the feature in the image.\nThe IsLocked field is the same across file types (locked for glomerulus, unlocked for anatomical structure) and is not signal-bearing.\ntrain.csv contains the unique IDs for each image, as well as an RLE-encoded representation of the mask for the objects in the image.\nSee the evaluation tab for details of the RLE encoding scheme\n\nHuBMAP-20-dataset_information.csv contains additional information (including anonymized patient data) about each image.\n\n\ntrain.csv :- \n It contains the unique IDs for each image, as well as an RLE-encoded representation of the mask for the objects in the image. See the evaluation tab for details of the RLE encoding scheme.\n\nHuBMAP-20-dataset_information.csv :- \n It contains additional information (including anonymized patient data) about each image."},{"metadata":{},"cell_type":"markdown","source":"# What we are prediciting?\n\nDevelop segmentation algorithms that identify glomeruli in the PAS stained microscopy data. Detect functional tissue units (FTUs) across different tissue preparation pipelines"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport tifffile as tiff \nimport seaborn as sns\n\n# segmentation\nfrom keras_segmentation.models.unet import vgg_unet\nfrom IPython.display import clear_output\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Thanks to divamgupta for Image segmentation in Keras module ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! pip install git+https://github.com/divamgupta/image-segmentation-keras","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/hubmap-kidney-segmentation/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/hubmap-kidney-segmentation/test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/hubmap-kidney-segmentation/train.csv\")\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Let see how kidney images looks like
\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"image = tiff.imread('../input/hubmap-kidney-segmentation/train/' + train.iloc[1,0] + \".tiff\")\nprint(\"This image's id:\", train.iloc[1,0])\nimage.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 15))\nplt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Decoding the mask in the image"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to - https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\n\ndef mask2rle(img):\n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels= img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n \ndef rle2mask(mask_rle, shape=(1600,256)):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (width,height) of array to return \n Returns numpy array, 1 - mask, 0 - background\n\n '''\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = rle2mask(train.iloc[1, 1], (image.shape[1], image.shape[0]))\nmask.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(image)\nplt.imshow(mask, cmap='coolwarm', alpha=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# Modeling
"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = '/kaggle/input/'\nos.listdir(data_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path_train = os.path.join(data_path, 'hubmap-256x256/train')\npath_masks = os.path.join(data_path, 'hubmap-256x256/masks')\n\npath_test = os.path.join(data_path, 'hubmap-256x256-test-data')\n\nprint(f'No. of training images: {len(os.listdir(path_train))}')\nprint(f'No. of masks: {len(os.listdir(path_masks))}')\nprint()\nprint(f'No. of test images: {len(os.listdir(path_test))}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for filename in os.listdir(path_train):\n if filename in os.listdir(path_masks):\n pass\n else:\n print('Filenames not same.')\nelse:\n print('All corresponding filenames are same.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_filename = os.listdir(path_train)[120]\n\nsample_image = plt.imread( os.path.join(path_train, sample_filename))\n\nsample_mask = plt.imread(os.path.join(path_masks, sample_filename))\n\n_, ax = plt.subplots(1, 2)\nax[0].imshow(sample_image)\nax[1].imshow(sample_mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = vgg_unet(n_classes = 2, input_height = 256, input_width = 256)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.train(train_images = path_train,train_annotations=path_masks,checkpoints_path='/kaggle/working/',epochs=6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = model.predict_segmentation(inp = os.path.join(path_train, sample_filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Architecture "},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.utils.vis_utils import plot_model\n\nplot_model(model, show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ..... Preparing Submission dataset"},{"metadata":{},"cell_type":"markdown","source":"# π π π Upvote if you find this Kernal useful "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}