{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":34547,"databundleVersionId":3897958,"sourceType":"competition"}],"dockerImageVersionId":30203,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

HuBMAP: Complete Understanding and EDA | W&B

\n \n
","metadata":{}},{"cell_type":"markdown","source":"

Contents

","metadata":{}},{"cell_type":"markdown","source":"> | S.No | Heading |\n> | :------------- | :-------------------: | \n> | 01 | [**Competition Overview**](#competition-overview) | \n> | 02 | [**Libraries**](#libraries) | \n> | 03 | [**Global Config**](#global-config) |\n> | 04 | [**Weights and Biases**](#weights-and-biases) |\n> | 05 | [**Load Datasets**](#load-datasets) |\n> | 06 | [**References**](#references) |","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Libraries

","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\nimport tifffile as tiff \nfrom tqdm.auto import tqdm\n\nplt.style.use(\"Solarize_Light2\")\n\n# Wandb Login\nimport wandb\nwandb.login()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-05-15T12:22:11.203362Z","iopub.execute_input":"2024-05-15T12:22:11.204439Z","iopub.status.idle":"2024-05-15T12:22:24.517315Z","shell.execute_reply.started":"2024-05-15T12:22:11.20434Z","shell.execute_reply":"2024-05-15T12:22:24.516224Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

Global Config

","metadata":{}},{"cell_type":"code","source":"class config:\n BASE_PATH = \"../input/hubmap-organ-segmentation/\"\n TRAIN_PATH = os.path.join(BASE_PATH, \"train\")\n\n# wandb config\nWANDB_CONFIG = {\n 'competition': 'HuBMAP', \n '_wandb_kernel': 'neuracort'\n }\n\n# Initialize W&B\nrun = wandb.init(\n project='hubmap-organ-segmentation', \n config= WANDB_CONFIG\n)","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:22:28.694294Z","iopub.execute_input":"2024-05-15T12:22:28.69523Z","iopub.status.idle":"2024-05-15T12:22:31.675342Z","shell.execute_reply.started":"2024-05-15T12:22:28.69519Z","shell.execute_reply":"2024-05-15T12:22:31.674076Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

Weights and Biases

","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"**Weights & Biases** is the machine learning platform for developers to build better models faster.\n\nYou can use W&B's lightweight, interoperable tools to\n\n- quickly track experiments,\n- version and iterate on datasets,\n- evaluate model performance,\n- reproduce models,\n- visualize results and spot regressions,\n- and share findings with colleagues.\n \nSet up W&B in 5 minutes, then quickly iterate on your machine learning pipeline with the confidence that your datasets and models are tracked and versioned in a reliable system of record.\n\nIn this notebook I will use Weights and Biases's amazing features to perform wonderful visualizations and logging seamlessly.","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Load Datasets

","metadata":{}},{"cell_type":"markdown","source":"## **Train Dataset** ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\n os.path.join(config.BASE_PATH, \"train.csv\")\n)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:22:37.026413Z","iopub.execute_input":"2024-05-15T12:22:37.027109Z","iopub.status.idle":"2024-05-15T12:22:37.435178Z","shell.execute_reply.started":"2024-05-15T12:22:37.027073Z","shell.execute_reply":"2024-05-15T12:22:37.434139Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%capture\nwandb.log({\"df_train\": df})","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:22:38.888908Z","iopub.execute_input":"2024-05-15T12:22:38.889672Z","iopub.status.idle":"2024-05-15T12:22:39.707073Z","shell.execute_reply.started":"2024-05-15T12:22:38.889637Z","shell.execute_reply":"2024-05-15T12:22:39.706238Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### [Interactive W&B Table for Dataframe $\\rightarrow$](https://wandb.ai/ishandutta/hubmap-organ-segmentation/runs/3j0orhhe?workspace=user-ishandutta)\n \nThe above gif shows the following features you can seamlessly use on wandb - \n- Interactively looking at the dataframe\n- Groupby based on columns and visualize the results\n- Select and deselect columns\n- Sorting columns in Ascending and Descending\n \n[![Animation.gif](https://iili.io/hDpaUJ.gif)]","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Image and Mask Single Sample

","metadata":{}},{"cell_type":"markdown","source":"## **Actual Image** ","metadata":{}},{"cell_type":"code","source":"img_id_1 = 10044\nimg_1 = tiff.imread(config.BASE_PATH + \"train_images/\" + str(img_id_1) + \".tiff\")\nprint(img_1.shape)","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:49.790467Z","iopub.execute_input":"2024-05-15T12:47:49.791421Z","iopub.status.idle":"2024-05-15T12:47:50.125058Z","shell.execute_reply.started":"2024-05-15T12:47:49.791382Z","shell.execute_reply":"2024-05-15T12:47:50.124098Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(15, 15))\nplt.imshow(img_1)\nplt.axis(\"off\")\nwandb.log({\"Image Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:50.126902Z","iopub.execute_input":"2024-05-15T12:47:50.12722Z","iopub.status.idle":"2024-05-15T12:47:52.562412Z","shell.execute_reply.started":"2024-05-15T12:47:50.127189Z","shell.execute_reply":"2024-05-15T12:47:52.561274Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Mask** ","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/paulorzp/rle-functions-run-length-encode-decode\ndef mask2rle(img):\n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels= img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n \ndef rle2mask(mask_rle, shape=(1600,256)):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (width,height) of array to return \n Returns numpy array, 1 - mask, 0 - background\n\n '''\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:52.564166Z","iopub.execute_input":"2024-05-15T12:47:52.564466Z","iopub.status.idle":"2024-05-15T12:47:52.575996Z","shell.execute_reply.started":"2024-05-15T12:47:52.564438Z","shell.execute_reply":"2024-05-15T12:47:52.574972Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"mask_1 = rle2mask(df[df[\"id\"]==img_id_1][\"rle\"].iloc[-1], (img_1.shape[1], img_1.shape[0]))\nmask_1.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:52.577111Z","iopub.execute_input":"2024-05-15T12:47:52.577366Z","iopub.status.idle":"2024-05-15T12:47:52.604662Z","shell.execute_reply.started":"2024-05-15T12:47:52.577342Z","shell.execute_reply":"2024-05-15T12:47:52.603856Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(mask_1, cmap='coolwarm', alpha=0.5)\nplt.axis(\"off\")\nwandb.log({\"Mask Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:52.606384Z","iopub.execute_input":"2024-05-15T12:47:52.606677Z","iopub.status.idle":"2024-05-15T12:47:54.174751Z","shell.execute_reply.started":"2024-05-15T12:47:52.60665Z","shell.execute_reply":"2024-05-15T12:47:54.173811Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Combining Image with Mask** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(img_1)\nplt.imshow(mask_1, cmap='coolwarm', alpha=0.5)\nplt.axis(\"off\")\nwandb.log({\"Image with Mask Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:47:54.17606Z","iopub.execute_input":"2024-05-15T12:47:54.176387Z","iopub.status.idle":"2024-05-15T12:47:57.344727Z","shell.execute_reply.started":"2024-05-15T12:47:54.176349Z","shell.execute_reply":"2024-05-15T12:47:57.343755Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Interactive Visualization of Images with Masks

","metadata":{}},{"cell_type":"code","source":"image_ids = df.id\nimage_files = glob.glob(config.BASE_PATH + \"/train_images/*\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:48:41.293774Z","iopub.execute_input":"2024-05-15T12:48:41.294744Z","iopub.status.idle":"2024-05-15T12:48:41.347384Z","shell.execute_reply.started":"2024-05-15T12:48:41.294705Z","shell.execute_reply":"2024-05-15T12:48:41.346406Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def save_table(table_name):\n table = wandb.Table(columns=['Id', 'Image', 'Mask', 'Image with Mask'], allow_mixed_types = True)\n\n for id, img in tqdm(zip(image_ids, image_files), total = len(image_ids)):\n\n img = tiff.imread(img)\n mask = rle2mask(df[df[\"id\"]==id][\"rle\"].iloc[-1], (img.shape[1], img.shape[0]))\n \n plt.figure(figsize=(10,10))\n plt.axis(\"off\")\n plt.imshow(img)\n plt.imshow(mask, cmap='coolwarm', alpha=0.5)\n plt.savefig(\"./image.jpg\")\n plt.close()\n\n table.add_data(\n id, \n wandb.Image(img), \n wandb.Image(mask),\n wandb.Image(cv2.cvtColor(cv2.imread(\"./image.jpg\"), cv2.COLOR_BGR2RGB))\n )\n\n wandb.log({table_name : table})\n \nsave_table(\"Images and Masks Record\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T12:48:41.912884Z","iopub.execute_input":"2024-05-15T12:48:41.913687Z","iopub.status.idle":"2024-05-15T13:24:53.480523Z","shell.execute_reply.started":"2024-05-15T12:48:41.913651Z","shell.execute_reply":"2024-05-15T13:24:53.479592Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### [Interactive W&B Table with all Images and their Masks $\\rightarrow$](https://wandb.ai/ishandutta/hubmap-organ-segmentation/runs/3t9d7ioa?workspace=user-ishandutta)\n \nUsers can easily load all the images in the form of wandb tables and view as well as compare the results.\n \n[![Animation.gif](https://iili.io/hDmNwB.gif)](https://iili.io/hDmNwB.gif)","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Metadata Analysis

","metadata":{}},{"cell_type":"markdown","source":"## **Organ** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.countplot(data=df, x=\"organ\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Organ Counts\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T13:24:53.482782Z","iopub.execute_input":"2024-05-15T13:24:53.484291Z","iopub.status.idle":"2024-05-15T13:24:53.665464Z","shell.execute_reply.started":"2024-05-15T13:24:53.484245Z","shell.execute_reply":"2024-05-15T13:24:53.664204Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Data Source** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.countplot(data=df, x=\"data_source\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Data Source\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T13:24:53.667317Z","iopub.execute_input":"2024-05-15T13:24:53.667717Z","iopub.status.idle":"2024-05-15T13:24:53.806306Z","shell.execute_reply.started":"2024-05-15T13:24:53.667679Z","shell.execute_reply":"2024-05-15T13:24:53.800609Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Age** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.histplot(data=df, x=\"age\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Age\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T13:27:30.939958Z","iopub.execute_input":"2024-05-15T13:27:30.940372Z","iopub.status.idle":"2024-05-15T13:27:31.219801Z","shell.execute_reply.started":"2024-05-15T13:27:30.94034Z","shell.execute_reply":"2024-05-15T13:27:31.218707Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## **Sex** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.histplot(data=df, x=\"sex\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Sex\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2024-05-15T13:27:31.615145Z","iopub.execute_input":"2024-05-15T13:27:31.61589Z","iopub.status.idle":"2024-05-15T13:27:31.816309Z","shell.execute_reply.started":"2024-05-15T13:27:31.615854Z","shell.execute_reply":"2024-05-15T13:27:31.815339Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n
\n

References

","metadata":{}},{"cell_type":"markdown","source":">- [HuBMAP - Exploratory Data Analysis](https://www.kaggle.com/code/ihelon/hubmap-exploratory-data-analysis)\n>- [HuBMAP Let's Visualize and Understand Dataset](https://www.kaggle.com/code/nayuts/hubmap-let-s-visualize-and-understand-dataset)","metadata":{}},{"cell_type":"markdown","source":"

More Plots coming soon!

\n \n
","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}}]}