{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HuBMAP+HPA-Data Exploration\n\nThis notebook is a data exploration for [HuBMAP + HPA - Hacking the Human Body competition](http://https://www.kaggle.com/competitions/hubmap-organ-segmentation). This competition's goal is to identify and segment functional tissue units (FTUs) across five human organs.  \n  \nIn this notebook, I will read the data, visualize the images, and make some analysis. If you find it helpful, please give it a upvote. Thank you.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport tifffile\nimport matplotlib.pyplot as plt\nimport random\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:38:48.902976Z","iopub.execute_input":"2022-07-19T06:38:48.903458Z","iopub.status.idle":"2022-07-19T06:38:49.120192Z","shell.execute_reply.started":"2022-07-19T06:38:48.903361Z","shell.execute_reply":"2022-07-19T06:38:49.119259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Check the data directory","metadata":{}},{"cell_type":"code","source":"BASE_DIR = \"/kaggle/input/hubmap-organ-segmentation\"\nprint(os.listdir(BASE_DIR))","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:38:49.121634Z","iopub.execute_input":"2022-07-19T06:38:49.122713Z","iopub.status.idle":"2022-07-19T06:38:49.130134Z","shell.execute_reply.started":"2022-07-19T06:38:49.122677Z","shell.execute_reply":"2022-07-19T06:38:49.128786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Read CSV file and quick check images","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(os.path.join(BASE_DIR, \"train.csv\"))\ndisplay(train_df)\nprint(\"\\n\\n-----Dataframe info-----\")\ntrain_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:39:48.646011Z","iopub.execute_input":"2022-07-19T06:39:48.646426Z","iopub.status.idle":"2022-07-19T06:39:49.058986Z","shell.execute_reply.started":"2022-07-19T06:39:48.646389Z","shell.execute_reply":"2022-07-19T06:39:49.057690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_IMAGES_DIR = os.path.join(BASE_DIR, \"train_images\")\nTRAIN_IMAGE_NAMES = os.listdir(TRAIN_IMAGES_DIR)\nprint(f\"\\n\\nTraining CSV file has {len(train_df)} rows instances.\")\nprint(f\"\\n\\nTraining image directory has {len(TRAIN_IMAGE_NAMES)} images.\")\n\nID_SET_CSV = set(train_df[\"id\"])\nID_SET_IMAGES = set([int(x.split('.')[0]) for x in TRAIN_IMAGE_NAMES])\nprint(f\"\\n\\nAre these two ID sets are equal? {ID_SET_CSV == ID_SET_IMAGES}\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:51:43.294715Z","iopub.execute_input":"2022-07-19T06:51:43.296267Z","iopub.status.idle":"2022-07-19T06:51:43.306293Z","shell.execute_reply.started":"2022-07-19T06:51:43.296220Z","shell.execute_reply":"2022-07-19T06:51:43.305310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df[\"organ\"].value_counts())\nprint(f\"\\n\\nTraining images are from {len(train_df.organ.value_counts())} organ types.\\n\\n\")\n\nprint(train_df[\"data_source\"].value_counts())\nprint(f\"\\n\\nTraining images are from {len(train_df.data_source.value_counts())} data sources.\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:52:01.781280Z","iopub.execute_input":"2022-07-19T06:52:01.781671Z","iopub.status.idle":"2022-07-19T06:52:01.791904Z","shell.execute_reply.started":"2022-07-19T06:52:01.781640Z","shell.execute_reply":"2022-07-19T06:52:01.791023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Visualize images and masks  \n  \n  Utility functions","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/paulorzp/rle-functions-run-lenght-encode-decode\ndef rle2mask(mask_rle, shape):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (width,height) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [\n        np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])\n    ]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo : hi] = 1\n    return img.reshape(shape).T","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:55:55.384029Z","iopub.execute_input":"2022-07-19T06:55:55.384446Z","iopub.status.idle":"2022-07-19T06:55:55.393561Z","shell.execute_reply.started":"2022-07-19T06:55:55.384414Z","shell.execute_reply":"2022-07-19T06:55:55.392359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_images_and_masks(multiple=True, with_mask=True, img_id=None, organ=None):\n    plt.figure(figsize=(16, 16))\n    if multiple:\n        for ind, image_file_name in tqdm(enumerate(\n            train_df.sample(9)[\"id\"].values if not organ else train_df[train_df[\"organ\"]==organ].sample(9)[\"id\"].values\n        )):\n            plt.subplot(3, 3, ind + 1)\n            image = tifffile.imread(os.path.join(TRAIN_IMAGES_DIR, str(image_file_name)+\".tiff\"))\n            plt.imshow(image)\n            plt.title(str(image_file_name)+\" - \"+train_df[train_df[\"id\"]==image_file_name][\"organ\"].values[0])\n            if with_mask:\n                mask = rle2mask(train_df[train_df[\"id\"] == image_file_name][\"rle\"].values[0],\n                                (image.shape[1], image.shape[0]))\n                plt.imshow(mask, cmap=\"hot\", alpha=0.5)\n            plt.axis(\"off\")\n    else:\n        if not img_id:\n            img_id = random.choice(TRAIN_IMAGE_NAMES).split(\".\")[0]\n        image = tifffile.imread(os.path.join(TRAIN_IMAGES_DIR, img_id+\".tiff\"))\n        plt.imshow(image)\n        plt.title(img_id+\" - \"+train_df[train_df[\"id\"]==int(img_id)][\"organ\"].values[0])\n        if with_mask:\n            mask = rle2mask(train_df[train_df[\"id\"] == int(img_id)][\"rle\"].values[0],\n                            (image.shape[1], image.shape[0]))\n            plt.imshow(mask, cmap=\"hot\", alpha=0.5)\n        plt.axis(\"off\")\n    \n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:29:30.118764Z","iopub.execute_input":"2022-07-19T07:29:30.119164Z","iopub.status.idle":"2022-07-19T07:29:30.135823Z","shell.execute_reply.started":"2022-07-19T07:29:30.119132Z","shell.execute_reply":"2022-07-19T07:29:30.134414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize multiple images with masks","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:29:33.823619Z","iopub.execute_input":"2022-07-19T07:29:33.824839Z","iopub.status.idle":"2022-07-19T07:29:54.770890Z","shell.execute_reply.started":"2022-07-19T07:29:33.824770Z","shell.execute_reply":"2022-07-19T07:29:54.769670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize single image with mask","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=False, with_mask=True, img_id=\"23009\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:56:46.228515Z","iopub.execute_input":"2022-07-19T06:56:46.228868Z","iopub.status.idle":"2022-07-19T06:56:49.450229Z","shell.execute_reply.started":"2022-07-19T06:56:46.228837Z","shell.execute_reply":"2022-07-19T06:56:49.448900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Analyze image attributes\n  \n  Image sizes and pixel values  \n  The next cell may take one or two minutes, if you don't want to check all sizes, please skip it.","metadata":{}},{"cell_type":"code","source":"img_shapes = {}\nPIXEL_MIN_VALUE = 255\nPIXEL_MAX_VALUE = 0\nfor image_file_name in tqdm(TRAIN_IMAGE_NAMES):\n    image = tifffile.imread(os.path.join(TRAIN_IMAGES_DIR, image_file_name))\n    img_shapes[image.shape] = img_shapes.get(image.shape, 0) + 1\n    PIXEL_MIN_VALUE = np.min(image) if np.min(image) < PIXEL_MIN_VALUE else PIXEL_MIN_VALUE\n    PIXEL_MAX_VALUE = np.max(image) if np.max(image) > PIXEL_MAX_VALUE else PIXEL_MAX_VALUE\nprint(\"\\n\\nImages shapes: \", img_shapes)\nprint(f\"\\n\\nPixel values are in range: {PIXEL_MIN_VALUE} - {PIXEL_MAX_VALUE}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:57:02.079511Z","iopub.execute_input":"2022-07-19T06:57:02.079872Z","iopub.status.idle":"2022-07-19T06:58:45.363845Z","shell.execute_reply.started":"2022-07-19T06:57:02.079843Z","shell.execute_reply":"2022-07-19T06:58:45.362415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Compare with heights and widths in CSV file ","metadata":{}},{"cell_type":"code","source":"train_df[[\"img_height\", \"img_width\"]].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T06:59:00.136526Z","iopub.execute_input":"2022-07-19T06:59:00.136906Z","iopub.status.idle":"2022-07-19T06:59:00.161071Z","shell.execute_reply.started":"2022-07-19T06:59:00.136876Z","shell.execute_reply":"2022-07-19T06:59:00.159698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Count the target pixels or mask pixels","metadata":{}},{"cell_type":"code","source":"def count_pixels(row):\n    row[\"total_pixels\"] = row[\"img_height\"] * row[\"img_width\"]\n    row[\"target_pixels\"] = sum([int(x) for x in row[\"rle\"].split()[1::2]])\n    row[\"target_pixels_ratio\"] = row[\"target_pixels\"] / row[\"total_pixels\"]\n    return row\n\ntrain_df = train_df.apply(count_pixels, axis=1)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:04:02.719277Z","iopub.execute_input":"2022-07-19T07:04:02.719743Z","iopub.status.idle":"2022-07-19T07:04:03.856417Z","shell.execute_reply.started":"2022-07-19T07:04:02.719701Z","shell.execute_reply":"2022-07-19T07:04:03.855137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Different organs have different distributions of target pixels ratio","metadata":{}},{"cell_type":"code","source":"train_df.groupby(\"organ\")[\"target_pixels_ratio\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:04:13.524309Z","iopub.execute_input":"2022-07-19T07:04:13.525060Z","iopub.status.idle":"2022-07-19T07:04:13.571568Z","shell.execute_reply.started":"2022-07-19T07:04:13.525024Z","shell.execute_reply":"2022-07-19T07:04:13.570113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 6))\nfor ind, organ in enumerate([\"prostate\", \"spleen\", \"largeintestine\", \"kidney\", \"lung\"]):\n    train_df[train_df[\"organ\"]==organ][\"target_pixels_ratio\"].hist(alpha=0.5+0.1*ind, label=organ)\n\nplt.xlabel('Target pixels ratio')\nplt.ylabel('Counts')\nplt.title('Target pixels ratios of five organs')\nplt.grid(True)\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:04:20.568623Z","iopub.execute_input":"2022-07-19T07:04:20.569060Z","iopub.status.idle":"2022-07-19T07:04:20.997103Z","shell.execute_reply.started":"2022-07-19T07:04:20.569027Z","shell.execute_reply":"2022-07-19T07:04:20.995833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see from chart above:\n* Kidney and lung images have relatively small amount of target pixels. \n* Large intestine and prostate images have relatively large amount of target pixels. \n  \nOur models have to segment FTUs across images of different organs. We can again observe them in different organ groups.","metadata":{}},{"cell_type":"markdown","source":"* Kidney","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True, organ=\"kidney\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:04:33.419455Z","iopub.execute_input":"2022-07-19T07:04:33.419880Z","iopub.status.idle":"2022-07-19T07:04:51.576488Z","shell.execute_reply.started":"2022-07-19T07:04:33.419837Z","shell.execute_reply":"2022-07-19T07:04:51.574662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Lung","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True, organ=\"lung\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:04:59.754881Z","iopub.execute_input":"2022-07-19T07:04:59.755300Z","iopub.status.idle":"2022-07-19T07:05:17.950357Z","shell.execute_reply.started":"2022-07-19T07:04:59.755266Z","shell.execute_reply":"2022-07-19T07:05:17.949573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Spleen","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True, organ=\"spleen\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:05:24.206946Z","iopub.execute_input":"2022-07-19T07:05:24.207421Z","iopub.status.idle":"2022-07-19T07:05:41.732495Z","shell.execute_reply.started":"2022-07-19T07:05:24.207383Z","shell.execute_reply":"2022-07-19T07:05:41.731442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Large intestine","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True, organ=\"largeintestine\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:06:04.420956Z","iopub.execute_input":"2022-07-19T07:06:04.421644Z","iopub.status.idle":"2022-07-19T07:06:23.647169Z","shell.execute_reply.started":"2022-07-19T07:06:04.421598Z","shell.execute_reply":"2022-07-19T07:06:23.645885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Prostate","metadata":{}},{"cell_type":"code","source":"plot_images_and_masks(multiple=True, with_mask=True, organ=\"prostate\")","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:06:28.301582Z","iopub.execute_input":"2022-07-19T07:06:28.301964Z","iopub.status.idle":"2022-07-19T07:06:47.560546Z","shell.execute_reply.started":"2022-07-19T07:06:28.301934Z","shell.execute_reply":"2022-07-19T07:06:47.559769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From images above, we find that FTUs of different organs present different features. We may take that into account when shuffling the data or building models.","metadata":{}},{"cell_type":"markdown","source":"#### Check test CSV file and image","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv(os.path.join(BASE_DIR, \"test.csv\"))\ndisplay(test_df)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:08:49.656623Z","iopub.execute_input":"2022-07-19T07:08:49.658060Z","iopub.status.idle":"2022-07-19T07:08:49.682255Z","shell.execute_reply.started":"2022-07-19T07:08:49.658007Z","shell.execute_reply":"2022-07-19T07:08:49.681151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the organizers, the training dataset is from HPA, the public test set is a combination of HPA data and HuBMAP data, and the private test set contains only HuBMAP data.   \nThere is a HuBMAP image in test image directory. Let's compare it with a HPA one.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 16))\nplt.subplot(1, 2, 1)\nimg_HPA = tifffile.imread(os.path.join(TRAIN_IMAGES_DIR, \"26664.tiff\"))\nplt.imshow(img_HPA)\nplt.title(\"26664\"+\" - \"+train_df[train_df[\"id\"]==26664][\"organ\"].values[0]+\" - HPA\")\nplt.axis(\"off\")\n\nplt.subplot(1, 2, 2)\nimg_HuBMAP = tifffile.imread(os.path.join(BASE_DIR, \"test_images\", \"10078.tiff\"))\nplt.imshow(img_HuBMAP)\nplt.title(\"10078\"+\" - \"+train_df[train_df[\"id\"]==26664][\"organ\"].values[0]+\" - HuBMAP\")\nplt.axis(\"off\")\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:28:24.846375Z","iopub.execute_input":"2022-07-19T07:28:24.847402Z","iopub.status.idle":"2022-07-19T07:28:26.894982Z","shell.execute_reply.started":"2022-07-19T07:28:24.847355Z","shell.execute_reply":"2022-07-19T07:28:26.893438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that HuBMAP one is kind of zoomed in with different color. So the data mismatch between training set and test set can be really challenging.  \n  \nA previous competition [HuBMAP - Hacking the Kidney](https://www.kaggle.com/competitions/hubmap-kidney-segmentation) can be helpful. And the dataset is from HuBMAP.","metadata":{}},{"cell_type":"markdown","source":"#### Thank you for reading!  Comments are welcome!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}