{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <center style=\"font-family: consolas; font-size: 32px; font-weight: bold;\"> ❤️HuBMAP - Hacking the Human Vasculature - Exploratory Data Analysis</center>\n<p><center style=\"color:#949494; font-family: consolas; font-size: 20px;\">Segment instances of microvascular structures from healthy human kidney tissue slides.</center></p>\n\n***","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# <center style=\"font-family: consolas; font-size: 32px; font-weight: bold;\">(ಠಿ⁠_⁠ಠ) Overview</center>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>Goal</code>: The goal of the competition is to develop a model that can segment instances of microvascular structures, such as capillaries, arterioles, and venules, in 2D PAS-stained histology images from healthy human kidney tissue slides.</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>Importance</code>: Automating the segmentation of microvasculature structures will help improve researchers' understanding of how blood vessels are arranged in human tissues. This knowledge is crucial for studying the interaction, organization, and specialization of cells in the body.</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>Context</code>: The competition is aligned with the efforts of the Human BioMolecular Atlas Program (HuBMAP), which aims to map healthy cells in the human body. The Vasculature Common Coordinate Framework (VCCF) is a navigation system that uses blood vasculature to map cellular locations. However, there are gaps in knowledge about microvasculature, and automating segmentation can help fill those gaps and contribute to the development of the VCCF and a Human Reference Atlas (HRA).</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>Research Focus</code>: HuBMAP researchers are studying the connections between cells throughout the body using advanced molecular and cellular biology technologies. By segmenting microvasculature arrangements, machine learning insights can augment their understanding of how these small vessels are distributed, leading to a better understanding of the relationships between cells and their impact on human health.</p>\n\n","metadata":{}},{"cell_type":"markdown","source":"#### <a id=\"top\"></a>\n# <div style=\"box-shadow: rgb(60, 121, 245) 0px 0px 0px 3px inset, rgb(255, 255, 255) 10px -10px 0px -3px, rgb(31, 193, 27) 10px -10px, rgb(255, 255, 255) 20px -20px 0px -3px, rgb(255, 217, 19) 20px -20px, rgb(255, 255, 255) 30px -30px 0px -3px, rgb(255, 156, 85) 30px -30px, rgb(255, 255, 255) 40px -40px 0px -3px, rgb(255, 85, 85) 40px -40px; padding:20px; margin-right: 40px; font-size:30px; font-family: consolas; text-align:center; display:fill; border-radius:15px; color:rgb(60, 121, 245);\"><b>Table of contents</b></div>\n\n<div style=\"background-color: rgba(60, 121, 245, 0.03); padding:30px; font-size:15px; font-family: consolas;\">\n<ul>\n    <li><a href=\"#0\" target=\"_self\" rel=\" noreferrer nofollow\">0. Import all dependencies</a></li>\n    <li><a href=\"#1\" target=\"_self\" rel=\" noreferrer nofollow\">1. Data overview</a>\n        <ul>\n            <li><a href=\"#1.1\" target=\"_self\" rel=\" noreferrer nofollow\">1.1 Files and Field Descriptions</a></li>\n        </ul>\n        <ul>\n            <li><a href=\"#1.2\" target=\"_self\" rel=\" noreferrer nofollow\">1.2 Whole Slide Images metadata</a></li>\n        </ul>\n        <ul>\n            <li><a href=\"#1.3\" target=\"_self\" rel=\" noreferrer nofollow\">1.3 Each image metadata</a></li>\n        </ul>\n        <ul>\n            <li><a href=\"#1.4\" target=\"_self\" rel=\" noreferrer nofollow\">1.4 Tiles visualisation</a></li>\n        </ul>\n    </li>\n</ul>\n\n</div>\n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"0\"></a>\n# <div style=\"box-shadow: rgba(0, 0, 0, 0.16) 0px 1px 4px inset, rgb(51, 51, 51) 0px 0px 0px 3px inset; padding:20px; font-size:32px; font-family: consolas; text-align:center; display:fill; border-radius:15px;  color:rgb(34, 34, 34);\"> <b> 0. Import all dependencies </b></div>","metadata":{}},{"cell_type":"code","source":"import os\nimport json\nfrom PIL import Image\nfrom collections import Counter\n\nimport numpy as np\nimport pandas as pd\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport tifffile as tiff\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:38.446105Z","iopub.execute_input":"2023-05-22T23:34:38.446527Z","iopub.status.idle":"2023-05-22T23:34:39.828388Z","shell.execute_reply.started":"2023-05-22T23:34:38.446495Z","shell.execute_reply":"2023-05-22T23:34:39.826918Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    img_path_template: str = \"/kaggle/input/hubmap-hacking-the-human-vasculature/train/{}.tif\"","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:39.830533Z","iopub.execute_input":"2023-05-22T23:34:39.831849Z","iopub.status.idle":"2023-05-22T23:34:39.836444Z","shell.execute_reply.started":"2023-05-22T23:34:39.831809Z","shell.execute_reply":"2023-05-22T23:34:39.835358Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_cartesian_coords(coords, img_height):\n    coords_array = np.array(coords).squeeze()\n    xs = coords_array[:, 0]\n    ys = -coords_array[:, 1] + img_height\n    \n    return xs, ys","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:39.838109Z","iopub.execute_input":"2023-05-22T23:34:39.839565Z","iopub.status.idle":"2023-05-22T23:34:39.857130Z","shell.execute_reply.started":"2023-05-22T23:34:39.839527Z","shell.execute_reply":"2023-05-22T23:34:39.855744Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_annotated_image(image_dict, scale_factor: int = 1.0) -> None:\n    array = tiff.imread(CFG.img_path_template.format(image_dict[\"id\"]))\n    img_example = Image.fromarray(array)\n    annotations = image_dict[\"annotations\"]\n    \n    # create figure\n    fig = go.Figure()\n\n    # constants\n    img_width = img_example.size[0]\n    img_height = img_example.size[1]\n    \n\n    # add invisible scatter trace\n    fig.add_trace(\n        go.Scatter(\n            x=[0, img_width],\n            y=[0, img_height],\n            mode=\"markers\",\n            marker_opacity=0\n        )\n    )\n\n    # configure axes\n    fig.update_xaxes(\n        visible=False,\n        range=[0, img_width]\n    )\n\n    fig.update_yaxes(\n        visible=False,\n        range=[0, img_height],\n        # the scaleanchor attribute ensures that the aspect ratio stays constant\n        scaleanchor=\"x\"\n    )\n\n    # add image\n    fig.add_layout_image(dict(\n        x=0,\n        sizex=img_width,\n        y=img_height,\n        sizey=img_height,\n        xref=\"x\", yref=\"y\",\n        opacity=1.0,\n        layer=\"below\",\n        sizing=\"stretch\",\n        source=img_example\n    ))\n    \n    # add polygons\n    for annotation in annotations:\n        name = annotation[\"type\"]\n        xs, ys = get_cartesian_coords(annotation[\"coordinates\"], img_height)\n        fig.add_trace(go.Scatter(\n            x=xs, y=ys, fill=\"toself\",\n            name=name,\n            hovertemplate=\"%{name}\",\n            mode='lines'\n        ))\n\n    # configure other layout\n    fig.update_layout(\n        width=img_width * scale_factor,\n        height=img_height * scale_factor,\n        margin={\"l\": 0, \"r\": 0, \"t\": 0, \"b\": 0},\n        showlegend=False\n    )\n\n    # disable the autosize on double click because it adds unwanted margins around the image\n    # and finally show figure\n    fig.show(config={'doubleClick': 'reset'})","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:39.860296Z","iopub.execute_input":"2023-05-22T23:34:39.860724Z","iopub.status.idle":"2023-05-22T23:34:39.876581Z","shell.execute_reply.started":"2023-05-22T23:34:39.860689Z","shell.execute_reply":"2023-05-22T23:34:39.875008Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# <div style=\"box-shadow: rgba(0, 0, 0, 0.16) 0px 1px 4px inset, rgb(51, 51, 51) 0px 0px 0px 3px inset; padding:20px; font-size:32px; font-family: consolas; text-align:center; display:fill; border-radius:15px;  color:rgb(34, 34, 34);\"> <b> 1. Data overview</b></div>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">🔴 Our goal in this competition is to locate microvasculature structures (blood vessels) within human kidney histology slides.</p> \n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ The competition data comprises tiles extracted from <b>5</b> Whole Slide Images (WSI) split into <b>2</b> datasets. Tiles from Dataset 1 have annotations that have been expert reviewed. Dataset 2 comprises the remaining tiles from these same WSIs and contain sparse annotations that have not been expert reviewed.</p> \n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ Tiles from <b>Dataset 1</b> have annotations that <b>have been expert reviewed</b>. <b>Dataset 2</b> comprises the remaining tiles from these same WSIs and contain sparse annotations that <b>have not been expert reviewed</b>.</p> \n\n* <p style=\"font-family: consolas; font-size: 16px;\">All of the test set tiles are from Dataset 1.</p> \n* <p style=\"font-family: consolas; font-size: 16px;\"><b>1</b> of the WSIs make up <code>the training set</code>, <b>2</b> WSIs make up <code>the public test set</code>, and <b>1</b> WSI makes up <code>the private test set</code>.</p> \n* <p style=\"font-family: consolas; font-size: 16px;\">The training data includes Dataset 2 tiles from the <u>public</u> test WSI, but <b>not</b> from the <u>private</u> test WSI.</p> \n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ Dataset 3 includes tiles extracted from an additional <b>9</b> WSIs. <b>These tiles have not been annotated</b>. You may wish to apply semi- or self-supervised learning techniques on this data to support your predictions.</p> \n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1.1\"></a>\n## <div style=\"box-shadow: rgba(0, 0, 0, 0.18) 0px 2px 4px inset; padding:20px; font-size:24px; font-family: consolas; text-align:center; display:fill; border-radius:15px; color:rgb(67, 66, 66)\"> <b> 1.1 Files and Field Descriptions</b></div>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>{train|test}/</code> Folders containing TIFF images of the tiles. Each tile is 512x512 in size.</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>polygons.jsonl</code> Polygonal segmentation masks in JSONL format, available for Dataset 1 and Dataset 2. Each line gives JSON annotations for a single image.</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>wsi_meta.csv</code> Metadata for the Whole Slide Images the tiles were extracted from.</p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>tile_meta.csv</code> Metadata for each image.</p>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1.2\"></a>\n## <div style=\"box-shadow: rgba(0, 0, 0, 0.18) 0px 2px 4px inset; padding:20px; font-size:24px; font-family: consolas; text-align:center; display:fill; border-radius:15px; color:rgb(67, 66, 66)\"> <b> 1.2 Whole Slide Images metadata</b></div>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\"><code>Description</code></p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>wsi_meta.csv</code> Metadata for the Whole Slide Images the tiles were extracted from.</p>\n\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>source_wsi</code> Identifies the WSI.</p>\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>age</code>, <code>sex</code>, <code>race</code>, <code>height</code>, <code>weight</code>, and <code>bmi</code> demographic information about the tissue donor.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Let's load the wsi_meta.csv file and display it.</p>","metadata":{}},{"cell_type":"code","source":"wsi_meta_df = pd.read_csv(\"/kaggle/input/hubmap-hacking-the-human-vasculature/wsi_meta.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:39.878970Z","iopub.execute_input":"2023-05-22T23:34:39.880090Z","iopub.status.idle":"2023-05-22T23:34:39.913783Z","shell.execute_reply.started":"2023-05-22T23:34:39.880005Z","shell.execute_reply":"2023-05-22T23:34:39.912548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wsi_meta_df","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:39.915441Z","iopub.execute_input":"2023-05-22T23:34:39.916525Z","iopub.status.idle":"2023-05-22T23:34:39.953450Z","shell.execute_reply.started":"2023-05-22T23:34:39.916486Z","shell.execute_reply":"2023-05-22T23:34:39.952603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ As can be seen above, only 4 wsi are present in the available data out of the 5 mentioned.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Let's output the pie distribution for the sex and race values.</p>","metadata":{}},{"cell_type":"code","source":"sex_counts = wsi_meta_df.sex.value_counts()\n\nfig = px.pie(values=sex_counts.values, names=sex_counts.index)\nfig.update_traces(textposition='inside', textfont_size=14)\nfig.update_layout(\n    title={\n        'text': \"Pie distribution of sex values\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    },\n    legend_title_text='Aspect:'\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:39.954609Z","iopub.execute_input":"2023-05-22T23:34:39.955296Z","iopub.status.idle":"2023-05-22T23:34:42.181305Z","shell.execute_reply.started":"2023-05-22T23:34:39.955262Z","shell.execute_reply":"2023-05-22T23:34:42.180036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_counts = wsi_meta_df.race.value_counts()\n\nfig = px.pie(values=race_counts.values, names=race_counts.index)\nfig.update_traces(textposition='inside', textfont_size=14)\nfig.update_layout(\n    title={\n        'text': \"Pie distribution of race values\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    },\n    legend_title_text='Aspect:'\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.184305Z","iopub.execute_input":"2023-05-22T23:34:42.185523Z","iopub.status.idle":"2023-05-22T23:34:42.266524Z","shell.execute_reply.started":"2023-05-22T23:34:42.185473Z","shell.execute_reply":"2023-05-22T23:34:42.265134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Now let's look at the distribution of other WSI meta values.</p>","metadata":{}},{"cell_type":"code","source":"fig = px.violin(wsi_meta_df, x='race', y='age', box=True, color='sex', points=\"all\")\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.268584Z","iopub.execute_input":"2023-05-22T23:34:42.269107Z","iopub.status.idle":"2023-05-22T23:34:42.412163Z","shell.execute_reply.started":"2023-05-22T23:34:42.269040Z","shell.execute_reply":"2023-05-22T23:34:42.411011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.scatter(wsi_meta_df, x='height', y='weight', color='sex')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.416750Z","iopub.execute_input":"2023-05-22T23:34:42.417170Z","iopub.status.idle":"2023-05-22T23:34:42.524984Z","shell.execute_reply.started":"2023-05-22T23:34:42.417137Z","shell.execute_reply":"2023-05-22T23:34:42.523974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"1.3\"></a>\n## <div style=\"box-shadow: rgba(0, 0, 0, 0.18) 0px 2px 4px inset; padding:20px; font-size:24px; font-family: consolas; text-align:center; display:fill; border-radius:15px; color:rgb(67, 66, 66)\"> <b> 1.3 Each image metadata</b></div>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\"><code>Description</code></p>\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>tile_meta.csv</code> Metadata for each image.</p>\n\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>source_wsi</code> Identifies the WSI this tile was extracted from.</p>\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>{i|j}</code> The location of the upper-left corner within the WSI where the tile was extracted.</p>\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>dataset</code> The dataset this tile belongs to, as described above.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Let's load the tile_meta.csv file and see what it looks like.</p>","metadata":{}},{"cell_type":"code","source":"tile_meta_df = pd.read_csv(\"/kaggle/input/hubmap-hacking-the-human-vasculature/tile_meta.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:42.526187Z","iopub.execute_input":"2023-05-22T23:34:42.526503Z","iopub.status.idle":"2023-05-22T23:34:42.551163Z","shell.execute_reply.started":"2023-05-22T23:34:42.526476Z","shell.execute_reply":"2023-05-22T23:34:42.549939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:42.552618Z","iopub.execute_input":"2023-05-22T23:34:42.552996Z","iopub.status.idle":"2023-05-22T23:34:42.565666Z","shell.execute_reply.started":"2023-05-22T23:34:42.552955Z","shell.execute_reply":"2023-05-22T23:34:42.564604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df.describe()","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:42.567519Z","iopub.execute_input":"2023-05-22T23:34:42.568292Z","iopub.status.idle":"2023-05-22T23:34:42.605302Z","shell.execute_reply.started":"2023-05-22T23:34:42.568254Z","shell.execute_reply":"2023-05-22T23:34:42.604137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">❔ Let's see how many WSI there are.</p>","metadata":{}},{"cell_type":"code","source":"print(\"Unique source WSIs --\", list(np.unique(tile_meta_df.source_wsi)))\nprint(\"Number of unique source WSIs --\", len(np.unique(tile_meta_df.source_wsi)))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.607164Z","iopub.execute_input":"2023-05-22T23:34:42.607624Z","iopub.status.idle":"2023-05-22T23:34:42.616328Z","shell.execute_reply.started":"2023-05-22T23:34:42.607579Z","shell.execute_reply":"2023-05-22T23:34:42.615104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ As you can see from the output above, the 5th wsi is missing.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">❔ Now let's see how many datasets exist.</p>","metadata":{}},{"cell_type":"code","source":"print(\"Unique datasets --\", list(np.unique(tile_meta_df.dataset)))\nprint(\"Number of unique datasets --\", len(np.unique(tile_meta_df.dataset)))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.618281Z","iopub.execute_input":"2023-05-22T23:34:42.618763Z","iopub.status.idle":"2023-05-22T23:34:42.629881Z","shell.execute_reply.started":"2023-05-22T23:34:42.618721Z","shell.execute_reply":"2023-05-22T23:34:42.628596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"swsi_count = Counter(tile_meta_df.source_wsi)\n\nfig = px.bar(\n    x=list(swsi_count.values()), y=list(swsi_count.keys()),\n    color_discrete_sequence=['darkslateblue'],\n    orientation='h', height=700\n)\nfig.update_layout(\n    title={\n        'text': \"Source WSI Composition\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    },\n    xaxis_title=\"Frequency\", yaxis_title=\"Source WSI\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:42.631563Z","iopub.execute_input":"2023-05-22T23:34:42.632009Z","iopub.status.idle":"2023-05-22T23:34:42.739395Z","shell.execute_reply.started":"2023-05-22T23:34:42.631969Z","shell.execute_reply":"2023-05-22T23:34:42.738020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Most WSIs consists of 600 tiles, which is very, very much :).</p>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1.4\"></a>\n## <div style=\"box-shadow: rgba(0, 0, 0, 0.18) 0px 2px 4px inset; padding:20px; font-size:24px; font-family: consolas; text-align:center; display:fill; border-radius:15px; color:rgb(67, 66, 66)\"> <b> 1.4 Tiles visualisation</b></div>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Each Whole Slide Image (WSI) is divided into tiles (as seen above, basically 600 pieces). For some of these tiles, polygonal segmentation masks are given in the polygons.jsonl file. </p>\n\n<p style=\"font-family: consolas; font-size: 16px;\">⚪ <code>polygons.jsonl</code> description -- Polygonal segmentation masks in JSONL format, available for Dataset 1 and Dataset 2. Each line gives JSON annotations for a single image with:</p>\n\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>id</code> Identifies the corresponding image in <b>train/</b></p>\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>annotations</code> A list of mask annotations with:</p>\n* <p style=\"font-family: consolas; font-size: 16px;\"><code>type</code> Identifies the type of structure annotated:</p>\n\n    * <p style=\"font-family: consolas; font-size: 16px;\"><code>blood_vessel</code> <b>The target structure</b>. Our goal in this competition is to predict these kinds of masks on the test set.</p>\n    * <p style=\"font-family: consolas; font-size: 16px;\"><code>glomerulus</code> A capillary ball structure in the kidney. These parts of the images were excluded from blood vessel annotation. You should ensure none of your test set <b>predictions occur within glomerulus structures</b> as they will be counted as false positives. Annotations are provided for test set tiles.</p>\n    * <p style=\"font-family: consolas; font-size: 16px;\"><code>unsure</code> A structure the expert annotators cannot confidently distinguish as a blood vessel.</p>\n    \n* <p style=\"font-family: consolas; font-size: 16px;\"><code>coordinates</code> A list of polygon coordinates defining the segmentation mask.</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Let's load these annotations and see what they look like in an example.</p>","metadata":{}},{"cell_type":"code","source":"with open('/kaggle/input/hubmap-hacking-the-human-vasculature/polygons.jsonl', 'r') as json_file:\n    json_list = list(json_file)\n    \ntiles_dicts = []\nfor json_str in json_list:\n    tiles_dicts.append(json.loads(json_str))","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:42.741291Z","iopub.execute_input":"2023-05-22T23:34:42.741649Z","iopub.status.idle":"2023-05-22T23:34:49.391672Z","shell.execute_reply.started":"2023-05-22T23:34:42.741620Z","shell.execute_reply":"2023-05-22T23:34:49.390481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Count of annotated tiles --\", len(tiles_dicts))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-22T23:34:49.393199Z","iopub.execute_input":"2023-05-22T23:34:49.394206Z","iopub.status.idle":"2023-05-22T23:34:49.399939Z","shell.execute_reply.started":"2023-05-22T23:34:49.394169Z","shell.execute_reply":"2023-05-22T23:34:49.398714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ As you can see from the output above, the number of annotated tiles is over 4 times smaller for the total number of tiles (7033 tiles / 1633 annotated tiles).</p>","metadata":{}},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Now, let's go directly to the visualization. Take for example a tile with id <code>9b9349a10d8d</code>. In the polygons.jsonl list, this tile has an index of <i>1000</i>.</p>","metadata":{}},{"cell_type":"code","source":"tiles_dicts[1000][\"id\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.401933Z","iopub.execute_input":"2023-05-22T23:34:49.402699Z","iopub.status.idle":"2023-05-22T23:34:49.439998Z","shell.execute_reply.started":"2023-05-22T23:34:49.402650Z","shell.execute_reply":"2023-05-22T23:34:49.438852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df[tile_meta_df.id == \"9b9349a10d8d\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.441609Z","iopub.execute_input":"2023-05-22T23:34:49.442392Z","iopub.status.idle":"2023-05-22T23:34:49.464163Z","shell.execute_reply.started":"2023-05-22T23:34:49.442358Z","shell.execute_reply":"2023-05-22T23:34:49.463070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_annotated_image(tiles_dicts[1000])","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.465703Z","iopub.execute_input":"2023-05-22T23:34:49.466266Z","iopub.status.idle":"2023-05-22T23:34:49.699095Z","shell.execute_reply.started":"2023-05-22T23:34:49.466235Z","shell.execute_reply":"2023-05-22T23:34:49.697648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Take for example another tile with id <code>0870e4f9d580</code>. In the polygons.jsonl list, this tile has an index of <i>50</i>.</p>","metadata":{}},{"cell_type":"code","source":"tiles_dicts[50][\"id\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.700543Z","iopub.execute_input":"2023-05-22T23:34:49.700993Z","iopub.status.idle":"2023-05-22T23:34:49.709409Z","shell.execute_reply.started":"2023-05-22T23:34:49.700956Z","shell.execute_reply":"2023-05-22T23:34:49.708136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df[tile_meta_df.id == \"0870e4f9d580\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.711511Z","iopub.execute_input":"2023-05-22T23:34:49.711975Z","iopub.status.idle":"2023-05-22T23:34:49.736543Z","shell.execute_reply.started":"2023-05-22T23:34:49.711938Z","shell.execute_reply":"2023-05-22T23:34:49.735008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_annotated_image(tiles_dicts[50])","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.738622Z","iopub.execute_input":"2023-05-22T23:34:49.739167Z","iopub.status.idle":"2023-05-22T23:34:49.929975Z","shell.execute_reply.started":"2023-05-22T23:34:49.739117Z","shell.execute_reply":"2023-05-22T23:34:49.928748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Take for example another tile with id <code>0033bbc76b6b</code>. In the polygons.jsonl list, this tile has an index of <i>2</i>.</p>","metadata":{}},{"cell_type":"code","source":"tiles_dicts[2][\"id\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.931543Z","iopub.execute_input":"2023-05-22T23:34:49.931910Z","iopub.status.idle":"2023-05-22T23:34:49.939935Z","shell.execute_reply.started":"2023-05-22T23:34:49.931878Z","shell.execute_reply":"2023-05-22T23:34:49.938150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df[tile_meta_df.id == \"0033bbc76b6b\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.941980Z","iopub.execute_input":"2023-05-22T23:34:49.942440Z","iopub.status.idle":"2023-05-22T23:34:49.964690Z","shell.execute_reply.started":"2023-05-22T23:34:49.942402Z","shell.execute_reply":"2023-05-22T23:34:49.962764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_annotated_image(tiles_dicts[2])","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:49.966546Z","iopub.execute_input":"2023-05-22T23:34:49.966975Z","iopub.status.idle":"2023-05-22T23:34:50.133306Z","shell.execute_reply.started":"2023-05-22T23:34:49.966944Z","shell.execute_reply":"2023-05-22T23:34:50.130977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: consolas; font-size: 16px;\">⚪ Take for example another tile with id <code>4ca084aec87b</code>. In the polygons.jsonl list, this tile has an index of <i>500</i>.</p>","metadata":{}},{"cell_type":"code","source":"tiles_dicts[500][\"id\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:50.139457Z","iopub.execute_input":"2023-05-22T23:34:50.141032Z","iopub.status.idle":"2023-05-22T23:34:50.152351Z","shell.execute_reply.started":"2023-05-22T23:34:50.140982Z","shell.execute_reply":"2023-05-22T23:34:50.150673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tile_meta_df[tile_meta_df.id == \"4ca084aec87b\"]","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:50.154729Z","iopub.execute_input":"2023-05-22T23:34:50.155240Z","iopub.status.idle":"2023-05-22T23:34:50.176136Z","shell.execute_reply.started":"2023-05-22T23:34:50.155195Z","shell.execute_reply":"2023-05-22T23:34:50.174394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_annotated_image(tiles_dicts[500])","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:50.177377Z","iopub.execute_input":"2023-05-22T23:34:50.177746Z","iopub.status.idle":"2023-05-22T23:34:50.356160Z","shell.execute_reply.started":"2023-05-22T23:34:50.177707Z","shell.execute_reply":"2023-05-22T23:34:50.355113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# (⁠ ⁠ꈍ⁠ᴗ⁠ꈍ⁠) WORK STILL IN PROGRESS","metadata":{"execution":{"iopub.status.busy":"2023-05-22T23:34:50.357609Z","iopub.execute_input":"2023-05-22T23:34:50.358241Z","iopub.status.idle":"2023-05-22T23:34:50.362220Z","shell.execute_reply.started":"2023-05-22T23:34:50.358204Z","shell.execute_reply":"2023-05-22T23:34:50.361238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <div style=\"box-shadow: rgba(240, 46, 170, 0.4) -5px 5px inset, rgba(240, 46, 170, 0.3) -10px 10px inset, rgba(240, 46, 170, 0.2) -15px 15px inset, rgba(240, 46, 170, 0.1) -20px 20px inset, rgba(240, 46, 170, 0.05) -25px 25px inset; padding:20px; font-size:30px; font-family: consolas; display:fill; border-radius:15px; color: rgba(240, 46, 170, 0.7)\"> <b> ༼⁠ ⁠つ⁠ ⁠◕⁠‿⁠◕⁠ ⁠༽⁠つ Thank You!</b></div>\n\n<p style=\"font-family:verdana; color:rgb(34, 34, 34); font-family: consolas; font-size: 16px;\"> 💌 Thank you for taking the time to read through my notebook. I hope you found it interesting and informative. If you have any feedback or suggestions for improvement, please don't hesitate to let me know in the comments. <br><br> 🚀 If you liked this notebook, please consider upvoting it so that others can discover it too. Your support means a lot to me, and it helps to motivate me to create more content in the future. <br><br> ❤️ Once again, thank you for your support, and I hope to see you again soon!</p>","metadata":{}}]}