{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

HuBMAP: Complete Understanding and EDA | W&B

\n \n
","metadata":{}},{"cell_type":"markdown","source":"

Contents

","metadata":{}},{"cell_type":"markdown","source":"> | S.No | Heading |\n> | :------------- | :-------------------: | \n> | 01 | [**Competition Overview**](#competition-overview) | \n> | 02 | [**Libraries**](#libraries) | \n> | 03 | [**Global Config**](#global-config) |\n> | 04 | [**Weights and Biases**](#weights-and-biases) |\n> | 05 | [**Load Datasets**](#load-datasets) |\n> | 06 | [**References**](#references) |","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Libraries

","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\nimport tifffile as tiff \nfrom tqdm.auto import tqdm\n\nplt.style.use(\"Solarize_Light2\")\n\n# Wandb Login\nimport wandb\nwandb.login()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-23T19:48:10.230355Z","iopub.execute_input":"2022-06-23T19:48:10.230980Z","iopub.status.idle":"2022-06-23T19:48:33.805364Z","shell.execute_reply.started":"2022-06-23T19:48:10.230887Z","shell.execute_reply":"2022-06-23T19:48:33.803979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Global Config

","metadata":{}},{"cell_type":"code","source":"class config:\n BASE_PATH = \"../input/hubmap-organ-segmentation/\"\n TRAIN_PATH = os.path.join(BASE_PATH, \"train\")\n\n# wandb config\nWANDB_CONFIG = {\n 'competition': 'HuBMAP', \n '_wandb_kernel': 'neuracort'\n }\n\n# Initialize W&B\nrun = wandb.init(\n project='hubmap-organ-segmentation', \n config= WANDB_CONFIG\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T20:42:54.411361Z","iopub.execute_input":"2022-06-23T20:42:54.411812Z","iopub.status.idle":"2022-06-23T20:43:00.606358Z","shell.execute_reply.started":"2022-06-23T20:42:54.411774Z","shell.execute_reply":"2022-06-23T20:43:00.605349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Weights and Biases

","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"**Weights & Biases** is the machine learning platform for developers to build better models faster.\n\nYou can use W&B's lightweight, interoperable tools to\n\n- quickly track experiments,\n- version and iterate on datasets,\n- evaluate model performance,\n- reproduce models,\n- visualize results and spot regressions,\n- and share findings with colleagues.\n \nSet up W&B in 5 minutes, then quickly iterate on your machine learning pipeline with the confidence that your datasets and models are tracked and versioned in a reliable system of record.\n\nIn this notebook I will use Weights and Biases's amazing features to perform wonderful visualizations and logging seamlessly.","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Load Datasets

","metadata":{}},{"cell_type":"markdown","source":"## **Train Dataset** ","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\n os.path.join(config.BASE_PATH, \"train.csv\")\n)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:48:48.003111Z","iopub.execute_input":"2022-06-23T19:48:48.003540Z","iopub.status.idle":"2022-06-23T19:48:48.361035Z","shell.execute_reply.started":"2022-06-23T19:48:48.003508Z","shell.execute_reply":"2022-06-23T19:48:48.359981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture\nwandb.log({\"df_train\": df})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Interactive W&B Table for Dataframe $\\rightarrow$](https://wandb.ai/ishandutta/hubmap-organ-segmentation/runs/3j0orhhe?workspace=user-ishandutta)\n \nThe above gif shows the following features you can seamlessly use on wandb - \n- Interactively looking at the dataframe\n- Groupby based on columns and visualize the results\n- Select and deselect columns\n- Sorting columns in Ascending and Descending\n \n[![Animation.gif](https://iili.io/hDpaUJ.gif)]","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Image and Mask Single Sample

","metadata":{}},{"cell_type":"markdown","source":"## **Actual Image** ","metadata":{}},{"cell_type":"code","source":"img_id_1 = 10044\nimg_1 = tiff.imread(config.BASE_PATH + \"train_images/\" + str(img_id_1) + \".tiff\")\nprint(img_1.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:29:44.079905Z","iopub.execute_input":"2022-06-23T19:29:44.080311Z","iopub.status.idle":"2022-06-23T19:29:44.744084Z","shell.execute_reply.started":"2022-06-23T19:29:44.080278Z","shell.execute_reply":"2022-06-23T19:29:44.742878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 15))\nplt.imshow(img_1)\nplt.axis(\"off\")\nwandb.log({\"Image Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:37:16.890039Z","iopub.execute_input":"2022-06-23T13:37:16.890421Z","iopub.status.idle":"2022-06-23T13:37:18.369898Z","shell.execute_reply.started":"2022-06-23T13:37:16.890389Z","shell.execute_reply":"2022-06-23T13:37:18.368994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Mask** ","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/paulorzp/rle-functions-run-length-encode-decode\ndef mask2rle(img):\n '''\n img: numpy array, 1 - mask, 0 - background\n Returns run length as string formated\n '''\n pixels= img.T.flatten()\n pixels = np.concatenate([[0], pixels, [0]])\n runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n runs[1::2] -= runs[::2]\n return ' '.join(str(x) for x in runs)\n \ndef rle2mask(mask_rle, shape=(1600,256)):\n '''\n mask_rle: run-length as string formated (start length)\n shape: (width,height) of array to return \n Returns numpy array, 1 - mask, 0 - background\n\n '''\n s = mask_rle.split()\n starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n starts -= 1\n ends = starts + lengths\n img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n for lo, hi in zip(starts, ends):\n img[lo:hi] = 1\n return img.reshape(shape).T","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:49:24.401314Z","iopub.execute_input":"2022-06-23T19:49:24.401705Z","iopub.status.idle":"2022-06-23T19:49:24.412247Z","shell.execute_reply.started":"2022-06-23T19:49:24.401672Z","shell.execute_reply":"2022-06-23T19:49:24.411346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask_1 = rle2mask(df[df[\"id\"]==img_id_1][\"rle\"].iloc[-1], (img_1.shape[1], img_1.shape[0]))\nmask_1.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:29:49.333029Z","iopub.execute_input":"2022-06-23T19:29:49.333424Z","iopub.status.idle":"2022-06-23T19:29:49.353781Z","shell.execute_reply.started":"2022-06-23T19:29:49.333373Z","shell.execute_reply":"2022-06-23T19:29:49.352832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(mask_1, cmap='coolwarm', alpha=0.5)\nplt.axis(\"off\")\nwandb.log({\"Mask Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:37:07.521568Z","iopub.execute_input":"2022-06-23T13:37:07.521944Z","iopub.status.idle":"2022-06-23T13:37:08.660278Z","shell.execute_reply.started":"2022-06-23T13:37:07.521907Z","shell.execute_reply":"2022-06-23T13:37:08.659162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Combining Image with Mask** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.imshow(img_1)\nplt.imshow(mask_1, cmap='coolwarm', alpha=0.5)\nplt.axis(\"off\")\nwandb.log({\"Image with Mask Sample 1\": plt})","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:36:47.516801Z","iopub.execute_input":"2022-06-23T13:36:47.517246Z","iopub.status.idle":"2022-06-23T13:36:49.860580Z","shell.execute_reply.started":"2022-06-23T13:36:47.517212Z","shell.execute_reply":"2022-06-23T13:36:49.859437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Interactive Visualization of Images with Masks

","metadata":{}},{"cell_type":"code","source":"image_ids = df.id\nimage_files = glob.glob(config.BASE_PATH + \"/train_images/*\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:49:05.399049Z","iopub.execute_input":"2022-06-23T19:49:05.399460Z","iopub.status.idle":"2022-06-23T19:49:05.450150Z","shell.execute_reply.started":"2022-06-23T19:49:05.399428Z","shell.execute_reply":"2022-06-23T19:49:05.449213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def save_table(table_name):\n table = wandb.Table(columns=['Id', 'Image', 'Mask', 'Image with Mask'], allow_mixed_types = True)\n\n for id, img in tqdm(zip(image_ids, image_files), total = len(image_ids)):\n\n img = tiff.imread(img)\n mask = rle2mask(df[df[\"id\"]==id][\"rle\"].iloc[-1], (img.shape[1], img.shape[0]))\n \n plt.figure(figsize=(10,10))\n plt.axis(\"off\")\n plt.imshow(img)\n plt.imshow(mask, cmap='coolwarm', alpha=0.5)\n plt.savefig(\"./image.jpg\")\n plt.close()\n\n table.add_data(\n id, \n wandb.Image(img), \n wandb.Image(mask),\n wandb.Image(cv2.cvtColor(cv2.imread(\"./image.jpg\"), cv2.COLOR_BGR2RGB))\n )\n\n wandb.log({table_name : table})\n \nsave_table(\"Images and Masks Record\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T19:49:29.544607Z","iopub.execute_input":"2022-06-23T19:49:29.545456Z","iopub.status.idle":"2022-06-23T20:31:53.541358Z","shell.execute_reply.started":"2022-06-23T19:49:29.545416Z","shell.execute_reply":"2022-06-23T20:31:53.539102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Interactive W&B Table with all Images and their Masks $\\rightarrow$](https://wandb.ai/ishandutta/hubmap-organ-segmentation/runs/3t9d7ioa?workspace=user-ishandutta)\n \nUsers can easily load all the images in the form of wandb tables and view as well as compare the results.\n \n[![Animation.gif](https://iili.io/hDmNwB.gif)](https://iili.io/hDmNwB.gif)","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Metadata Analysis

","metadata":{}},{"cell_type":"markdown","source":"## **Organ** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.countplot(data=df, x=\"organ\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Organ Counts\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:41:00.740268Z","iopub.execute_input":"2022-06-23T13:41:00.740634Z","iopub.status.idle":"2022-06-23T13:41:00.946917Z","shell.execute_reply.started":"2022-06-23T13:41:00.740605Z","shell.execute_reply":"2022-06-23T13:41:00.945403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Data Source** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.countplot(data=df, x=\"data_source\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Data Source\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:41:01.168090Z","iopub.execute_input":"2022-06-23T13:41:01.168483Z","iopub.status.idle":"2022-06-23T13:41:01.342411Z","shell.execute_reply.started":"2022-06-23T13:41:01.168451Z","shell.execute_reply":"2022-06-23T13:41:01.340963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Age** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.histplot(data=df, x=\"age\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Age\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:41:04.803595Z","iopub.execute_input":"2022-06-23T13:41:04.803995Z","iopub.status.idle":"2022-06-23T13:41:05.058106Z","shell.execute_reply.started":"2022-06-23T13:41:04.803960Z","shell.execute_reply":"2022-06-23T13:41:05.057300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Sex** ","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\ng = sns.histplot(data=df, x=\"sex\", palette=sns.color_palette(\"Set2\", 8))\ng.set_title(\"Sex\", color = \"black\")","metadata":{"execution":{"iopub.status.busy":"2022-06-23T13:41:05.799952Z","iopub.execute_input":"2022-06-23T13:41:05.800725Z","iopub.status.idle":"2022-06-23T13:41:05.982064Z","shell.execute_reply.started":"2022-06-23T13:41:05.800684Z","shell.execute_reply":"2022-06-23T13:41:05.981161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

References

","metadata":{}},{"cell_type":"markdown","source":">- [HuBMAP - Exploratory Data Analysis](https://www.kaggle.com/code/ihelon/hubmap-exploratory-data-analysis)\n>- [HuBMAP Let's Visualize and Understand Dataset](https://www.kaggle.com/code/nayuts/hubmap-let-s-visualize-and-understand-dataset)","metadata":{}},{"cell_type":"markdown","source":"

More Plots coming soon!

\n \n
","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation. This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}}]}