FCN | Blood Vessels in Kidney Semantic Segmentation
\n\n \n\n
\n \n \n \n
\n\n \n\n## 🎯 Introduction\nThis is a notebook for a past Kaggle competition [**HuBMAP - Hacking the Human Vasculature**.](https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature) The goal of this competition is to detect Blood Vessels from images of kidney tissue taken by a microscope, and the detection mask shall have [IoU (Intersection over Union)](https://learnopencv.com/intersection-over-union-iou-in-object-detection-and-segmentation/) greater than 0.6. The Kaggle score is calculated by Average Precision Over Confidence which is the same as [Open Images 2019 - Instance Segmentation](https://www.kaggle.com/c/open-images-2019-instance-segmentation/overview/evaluation). \n\nThis HuBMAP competition seems to be difficult for a few reasons. First, it looks almost impossible to correctly identify blood vessels by non-experts(refer to EDA). It would be hard for deep learning too. Second, only 1633 label data are given for images which is insufficient for such a complex segmentation. Another reason is an imbalanced label: only 3.3% of image pixels are positive. Therefore, I defined the following targets for this project. \n\n**Target**\n\n* To develop an effective training loop which includes **data augmentation** and **custom loss function**\n\n \n\n\n","metadata":{"execution":{"iopub.status.busy":"2024-02-22T19:53:17.818274Z","iopub.execute_input":"2024-02-22T19:53:17.81926Z","iopub.status.idle":"2024-02-22T19:53:17.83287Z","shell.execute_reply.started":"2024-02-22T19:53:17.819215Z","shell.execute_reply":"2024-02-22T19:53:17.831138Z"}}},{"cell_type":"markdown","source":"
\n\n\n# 3 | EDA\n\n\n ","metadata":{}},{"cell_type":"markdown","source":"\n# 3.1 | Basic Information","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# Assuming y_train is your labels\n# Ensure y_train is a 1-dimensional array\n# If y_train is not in the correct shape, reshape or extract labels accordingly\n\n# Example of flattening y_train if it has extra dimensions\n# Adjust this line if your labels are embedded in a different structure\nlabel_mat_flatten = np.array(label_mat).flatten()\n\n# Convert to pandas Series for easier manipulation\nlabel_mat_flatten = pd.Series(label_mat_flatten)\n\n# Count the occurrences of each label\nlabel_counts = label_mat_flatten.value_counts()\n\n# Print the label counts\nprint(\"Label counts:\\n\", label_counts)\n\n# Plot the label distribution\nplt.figure(figsize=(10, 5))\nlabel_counts.plot(kind='bar')\nplt.xlabel('Labels')\nplt.ylabel('Count')\nplt.title('Label Distribution in Data Set')\nplt.show()\n\n# Check if the dataset is balanced\nthreshold = 0.1 # Define your own threshold for imbalance\nmax_count = label_counts.max()\nmin_count = label_counts.min()\nbalance_ratio = min_count / max_count\n\nif balance_ratio < (1 - threshold):\n print(\"The dataset is unbalanced.\")\nelse:\n print(\"The dataset is balanced.\")\n","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:27.794517Z","iopub.execute_input":"2024-05-29T21:53:27.794810Z","iopub.status.idle":"2024-05-29T21:53:30.397810Z","shell.execute_reply.started":"2024-05-29T21:53:27.794785Z","shell.execute_reply":"2024-05-29T21:53:30.396922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To dispaly data distribution\nselect = [\"dataset\", \"source_wsi\", \"dataset_wsi\"]\n\nsum_data_df = kidney_df[select].groupby(select[0:-1]).count()\nsum_data_df.plot(kind = \"bar\", title = \"count of (dataset, source wsi)\", color=[\"#4682B4\", \"#4682B4\"])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.399029Z","iopub.execute_input":"2024-05-29T21:53:30.399376Z","iopub.status.idle":"2024-05-29T21:53:30.678639Z","shell.execute_reply.started":"2024-05-29T21:53:30.399340Z","shell.execute_reply":"2024-05-29T21:53:30.677702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To dispaly data distribution\nselect = [\"id\", \"dataset\"]\n\nsum_data_df = kidney_df[select].groupby(select[-1]).count()\nsum_data_df.plot(kind = \"bar\", title = \"count of dataset\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.679740Z","iopub.execute_input":"2024-05-29T21:53:30.680020Z","iopub.status.idle":"2024-05-29T21:53:30.858316Z","shell.execute_reply.started":"2024-05-29T21:53:30.679995Z","shell.execute_reply":"2024-05-29T21:53:30.857399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_photo(img, title = \"\"):\n \n N = img.shape[0]\n \n NC = 5\n NR = math.ceil(N/NC)\n fig, ax = plt.subplots(NR, NC, figsize = (12, NR*2.3))\n \n for k in range(N):\n i = int(k/NC)\n j = k % NC\n \n if N <= 5:\n ax[j].imshow(img[k],cmap='gray')\n ax[j].tick_params(left = False, right = False , labelleft = False ,\n labelbottom = False, bottom = False)\n else:\n ax[i,j].imshow(img[k],cmap='gray')\n ax[i,j].tick_params(left = False, right = False , labelleft = False ,\n labelbottom = False, bottom = False)\n \n if title != \"\":\n if N <= 5:\n ax[2].set_title(title)\n else:\n ax[0,2].set_title(title)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.859327Z","iopub.execute_input":"2024-05-29T21:53:30.859589Z","iopub.status.idle":"2024-05-29T21:53:30.867809Z","shell.execute_reply.started":"2024-05-29T21:53:30.859565Z","shell.execute_reply":"2024-05-29T21:53:30.866975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function to overlay masks and plot\ndef plot_with_masks(images, masks, title=\"\"):\n overlaid_images = [overlay_mask(image, mask) for image, mask in zip(images, masks)]\n plot_photo(np.array(overlaid_images), title)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.868982Z","iopub.execute_input":"2024-05-29T21:53:30.869246Z","iopub.status.idle":"2024-05-29T21:53:30.881569Z","shell.execute_reply.started":"2024-05-29T21:53:30.869223Z","shell.execute_reply":"2024-05-29T21:53:30.880698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.2 | Images and Labels\n\nNext, images and labels are visualized by each dataset and source (person). The size and quantity of blood vessels are different between sources. ","metadata":{}},{"cell_type":"code","source":"#This code select some samples from each dataset and source\nnp.random.seed(1)\n\nsamples_ds = []\nfor ds in [1,2]:\n filter1 = kidney_df[\"dataset\"]==ds\n samples = []\n for wsi in [1,2,3,4]:\n \n if ds ==1:\n if wsi in [1,2]:\n filter2 = filter1 & (kidney_df[\"source_wsi\"] == wsi)\n s_idx = np.random.choice(np.where(filter2)[0], 50)\n \n else:\n filter2 = filter1 & (kidney_df[\"source_wsi\"] == wsi)\n s_idx = np.random.choice(np.where(filter2)[0], 50)\n \n samples.append(s_idx)\n \n samples_ds.append(samples)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.882611Z","iopub.execute_input":"2024-05-29T21:53:30.882936Z","iopub.status.idle":"2024-05-29T21:53:30.895895Z","shell.execute_reply.started":"2024-05-29T21:53:30.882904Z","shell.execute_reply":"2024-05-29T21:53:30.895160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Dataset1**\n\nIn dataset 1, source 1 contains a larger size but smaller number of blood vessels, whereas source 2 has many smaller size blood vessels. ","metadata":{}},{"cell_type":"code","source":"# Plotting the images with overlaid masks\nplot_photo(X_train[samples_ds[0][0][0:5]], \"dataset 1, source 1\")\nplot_with_masks(X_train[samples_ds[0][0][0:5]], label_mat[samples_ds[0][0][0:5]], \"dataset 1, source 1 with masks\")\nplot_photo(label_mat[samples_ds[0][0][0:5]], \"dataset 1, source 1, Label\")\nplot_photo(X_train[samples_ds[0][1][0:5]], \"dataset 1, source 2\")\nplot_with_masks(X_train[samples_ds[0][1][0:5]], label_mat[samples_ds[0][1][0:5]], \"dataset 1, source 2 with masks\")\nplot_photo(label_mat[samples_ds[0][1][0:5]], \"dataset 1, source 2, Label\")","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:30.896947Z","iopub.execute_input":"2024-05-29T21:53:30.897239Z","iopub.status.idle":"2024-05-29T21:53:36.606502Z","shell.execute_reply.started":"2024-05-29T21:53:30.897215Z","shell.execute_reply":"2024-05-29T21:53:36.605458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Dataset 2**\n\nThe images and labels of dataset 2 look similar to dataset 1, but their color is slightly different. ","metadata":{"execution":{"iopub.status.busy":"2024-02-22T20:03:42.761321Z","iopub.execute_input":"2024-02-22T20:03:42.761739Z","iopub.status.idle":"2024-02-22T20:03:42.769009Z","shell.execute_reply.started":"2024-02-22T20:03:42.7617Z","shell.execute_reply":"2024-02-22T20:03:42.767481Z"}}},{"cell_type":"code","source":"plot_photo(X_train[samples_ds[1][0][0:5]], \"dataset 2, source 1\")\nplot_with_masks(X_train[samples_ds[1][0][0:5]], label_mat[samples_ds[1][0][0:5]], \"dataset 2, source 1 with masks\")\nplot_photo(label_mat[samples_ds[1][0][0:5]], \"dataset 2, source 1, Label\")\n\nplot_photo(X_train[samples_ds[1][1][0:5]], \"dataset 2, source 2\")\nplot_with_masks(X_train[samples_ds[1][1][0:5]], label_mat[samples_ds[1][1][0:5]], \"dataset 2, source 2 with masks\")\nplot_photo(label_mat[samples_ds[1][1][0:5]], \"dataset 2, source 2, Label\")\n\nplot_photo(X_train[samples_ds[1][2][0:5]], \"dataset 2, source 3\")\nplot_with_masks(X_train[samples_ds[1][2][0:5]], label_mat[samples_ds[1][2][0:5]], \"dataset 2, source 3 with masks\")\nplot_photo(label_mat[samples_ds[1][2][0:5]], \"dataset 2, source 3, Label\")\n\nplot_photo(X_train[samples_ds[1][3][0:5]], \"dataset 2, source 4\")\nplot_with_masks(X_train[samples_ds[1][3][0:5]], label_mat[samples_ds[1][3][0:5]], \"dataset 2, source 4 with masks\")\nplot_photo(label_mat[samples_ds[1][3][0:5]], \"dataset 2, source 4, Label\")","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:36.607787Z","iopub.execute_input":"2024-05-29T21:53:36.608841Z","iopub.status.idle":"2024-05-29T21:53:48.788970Z","shell.execute_reply.started":"2024-05-29T21:53:36.608803Z","shell.execute_reply":"2024-05-29T21:53:48.788061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def label_stat(Y_mat):\n \n #n_labels = number of labels\n # labels_mat= 2D matrix in which label of connected areas are assigned. \n \n n_labels, labels_mat = cv2.connectedComponents(Y_mat)\n size_list = []\n for i in range(1, n_labels):\n #Calculate number of pixels for each label\n filter1 = labels_mat == i\n size = filter1.sum()\n size_list.append(size)\n\n return n_labels - 1, np.mean(size_list)\n\n\ndef density_plot(dataset, n_source):\n \n fig, ax = plt.subplots(1,n_source, figsize = (3.5*n_source,2.7))\n \n for i in range(n_source):\n filter1 = (kidney_df[\"dataset\"] == dataset) & (kidney_df[\"source_wsi\"] == i + 1)\n sns.kdeplot(kidney_df[\"positive_ratio\"].loc[filter1], ax = ax[i])\n ax[i].set_xlim(0, 0.5)\n ax[i].set_title(\"(dataset, source) = (\" +str(dataset) + \", \" + str(i+1) + \")\" )\n if i > 0:\n ax[i].set_ylabel(\"\")\n \n plt.show()\n \n\ndef scatter_plot(dataset, n_source):\n fig, ax = plt.subplots(1,n_source, figsize = (3.5*n_source,2.7))\n\n for i in range(n_source):\n filter1 = (kidney_df[\"dataset\"] == dataset) & (kidney_df[\"source_wsi\"] == i + 1)\n sns.scatterplot(data = kidney_df.loc[filter1], x = \"n_BloodVessel\", y = \"meanSize_BloodVessel\", ax = ax[i], alpha = 0.4)\n ax[i].set_xlim(0, 35)\n ax[i].set_ylim(0, 15)\n if i > 0:\n ax[i].set_ylabel(\"\")\n ax[i].set_title(\"(dataset, source) = (\" +str(dataset) + \", \" + str(i+1) + \")\" )\n \n ax[0].set_ylabel(\"mean size (1000 pixel)\")\n plt.show()\n \n","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:48.790360Z","iopub.execute_input":"2024-05-29T21:53:48.791042Z","iopub.status.idle":"2024-05-29T21:53:48.803301Z","shell.execute_reply.started":"2024-05-29T21:53:48.791007Z","shell.execute_reply":"2024-05-29T21:53:48.802299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_label_list = []\nmean_size_list = []\nratio_list = []\nfor i in range(NP):\n n_label, mean_size = label_stat(label_mat[i])\n n_label_list.append(n_label)\n mean_size_list.append(mean_size)\n ratio_list.append(np.mean(label_mat[i]))","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:48.804619Z","iopub.execute_input":"2024-05-29T21:53:48.804961Z","iopub.status.idle":"2024-05-29T21:53:53.529025Z","shell.execute_reply.started":"2024-05-29T21:53:48.804930Z","shell.execute_reply":"2024-05-29T21:53:53.528197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kidney_df[\"n_BloodVessel\"] = n_label_list\nkidney_df[\"meanSize_BloodVessel\"] = np.array(mean_size_list)/1000\nkidney_df[\"positive_ratio\"] = ratio_list","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:53.530274Z","iopub.execute_input":"2024-05-29T21:53:53.530745Z","iopub.status.idle":"2024-05-29T21:53:53.539027Z","shell.execute_reply.started":"2024-05-29T21:53:53.530710Z","shell.execute_reply":"2024-05-29T21:53:53.538088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Ratio of positive pixel\n\nOverall, only 5.0% of pixels are positive (blood vessels). When it is calculated by each dataset and source, dataset 1 has more positive pixels than dataset 2.","metadata":{}},{"cell_type":"code","source":"pct = np.round(kidney_df[\"positive_ratio\"].mean()*100, 1)\nprint(pct, \"% of pixels are positive in all data\")","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:53.540123Z","iopub.execute_input":"2024-05-29T21:53:53.540384Z","iopub.status.idle":"2024-05-29T21:53:53.554429Z","shell.execute_reply.started":"2024-05-29T21:53:53.540360Z","shell.execute_reply":"2024-05-29T21:53:53.553469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kidney_df[[\"dataset\", \"source_wsi\", \"positive_ratio\"]].groupby([\"dataset\", \"source_wsi\"]).mean()","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:53.555955Z","iopub.execute_input":"2024-05-29T21:53:53.556763Z","iopub.status.idle":"2024-05-29T21:53:53.579073Z","shell.execute_reply.started":"2024-05-29T21:53:53.556730Z","shell.execute_reply":"2024-05-29T21:53:53.578033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The ratio of positive pixels is calculated by each image, and the distribution is plotted in the following density plots.","metadata":{"execution":{"iopub.status.busy":"2024-02-22T20:05:41.212307Z","iopub.execute_input":"2024-02-22T20:05:41.212998Z","iopub.status.idle":"2024-02-22T20:05:41.220194Z","shell.execute_reply.started":"2024-02-22T20:05:41.212961Z","shell.execute_reply":"2024-02-22T20:05:41.21873Z"}}},{"cell_type":"markdown","source":"**Dataset 1**","metadata":{}},{"cell_type":"code","source":"density_plot(1, 2)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:53.580289Z","iopub.execute_input":"2024-05-29T21:53:53.580602Z","iopub.status.idle":"2024-05-29T21:53:54.137196Z","shell.execute_reply.started":"2024-05-29T21:53:53.580576Z","shell.execute_reply":"2024-05-29T21:53:54.136243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Dataset 2**","metadata":{}},{"cell_type":"code","source":"density_plot(2, 4)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:54.138501Z","iopub.execute_input":"2024-05-29T21:53:54.138810Z","iopub.status.idle":"2024-05-29T21:53:55.001523Z","shell.execute_reply.started":"2024-05-29T21:53:54.138784Z","shell.execute_reply":"2024-05-29T21:53:55.000563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Size and quantity of Blood Vessels\n\nDataset 1 source 1 has a different distribution from others. It has less number of blood vessels, but their size is larger. Please note that **one dot corresponds to one image** in scatterplots.","metadata":{"execution":{"iopub.status.busy":"2024-02-22T20:06:53.752005Z","iopub.execute_input":"2024-02-22T20:06:53.75246Z","iopub.status.idle":"2024-02-22T20:06:53.760259Z","shell.execute_reply.started":"2024-02-22T20:06:53.752427Z","shell.execute_reply":"2024-02-22T20:06:53.758942Z"}}},{"cell_type":"markdown","source":"**dataset1**","metadata":{}},{"cell_type":"code","source":"scatter_plot(1, 2)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:55.002889Z","iopub.execute_input":"2024-05-29T21:53:55.003266Z","iopub.status.idle":"2024-05-29T21:53:55.347521Z","shell.execute_reply.started":"2024-05-29T21:53:55.003230Z","shell.execute_reply":"2024-05-29T21:53:55.346619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**dataset2**","metadata":{}},{"cell_type":"code","source":"scatter_plot(2, 4)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:55.348642Z","iopub.execute_input":"2024-05-29T21:53:55.348914Z","iopub.status.idle":"2024-05-29T21:53:56.003742Z","shell.execute_reply.started":"2024-05-29T21:53:55.348890Z","shell.execute_reply":"2024-05-29T21:53:56.002851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n# 5 | Training and Hyperparameter Tuning\n\n\n ","metadata":{}},{"cell_type":"markdown","source":"# 5.1 | Split Dataset\n\n80% of images are randomly selected for training. ","metadata":{"execution":{"iopub.status.busy":"2024-02-22T20:10:12.041187Z","iopub.execute_input":"2024-02-22T20:10:12.041615Z","iopub.status.idle":"2024-02-22T20:10:12.050959Z","shell.execute_reply.started":"2024-02-22T20:10:12.041581Z","shell.execute_reply":"2024-02-22T20:10:12.048901Z"}}},{"cell_type":"code","source":"batch_size = 4","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:57.447720Z","iopub.execute_input":"2024-05-29T21:53:57.447981Z","iopub.status.idle":"2024-05-29T21:53:57.452025Z","shell.execute_reply.started":"2024-05-29T21:53:57.447958Z","shell.execute_reply":"2024-05-29T21:53:57.451192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(1)\nn_data = NP\nn_train = int(n_data*0.8)\n\nall_idx = np.arange(n_data)\nnp.random.shuffle(all_idx)\n\ntrain_idx = all_idx[:n_train]\nval_idx = all_idx[n_train:]\n\nprint(\"N of data for training = \", train_idx.shape[0]) \nprint(\"N of data for validation = \", val_idx.shape[0])","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:57.452845Z","iopub.execute_input":"2024-05-29T21:53:57.453084Z","iopub.status.idle":"2024-05-29T21:53:57.464785Z","shell.execute_reply.started":"2024-05-29T21:53:57.453062Z","shell.execute_reply":"2024-05-29T21:53:57.463712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds = tf.data.Dataset.from_tensor_slices((X_train[train_idx], label_mat[train_idx])).shuffle(2000).batch(batch_size)\nX_val = X_train[val_idx]\nY_val = label_mat[val_idx]\nn_val = X_val.shape[0]","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:53:57.465940Z","iopub.execute_input":"2024-05-29T21:53:57.466222Z","iopub.status.idle":"2024-05-29T21:54:00.969264Z","shell.execute_reply.started":"2024-05-29T21:53:57.466199Z","shell.execute_reply":"2024-05-29T21:54:00.968448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_ds),len(X_val),len(Y_val)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:00.970364Z","iopub.execute_input":"2024-05-29T21:54:00.970663Z","iopub.status.idle":"2024-05-29T21:54:00.977295Z","shell.execute_reply.started":"2024-05-29T21:54:00.970638Z","shell.execute_reply":"2024-05-29T21:54:00.976450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5.3 | Vessel Segmentation Loss Function. ","metadata":{}},{"cell_type":"code","source":"import keras.backend as K\nimport tensorflow as tf\n\ndef vessel_segmentation_loss(y_true, y_pred):\n \"\"\"\n Computes the loss for semantic segmentation of blood vessels in the kidney.\n \"\"\"\n # Compute pixel-wise cross entropy loss\n ce_loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\n ce_loss = tf.reduce_mean(ce_loss)\n \n # Compute dice loss to encourage better segmentation of vessels\n smooth = 1e-5\n y_pred_sigmoid = tf.sigmoid(y_pred)\n y_true_binary = tf.cast(tf.equal(y_true, 1), tf.float32)\n intersection = tf.reduce_sum(y_pred_sigmoid * y_true_binary)\n union = tf.reduce_sum(y_pred_sigmoid) + tf.reduce_sum(y_true_binary)\n dice_loss = 1 - (2 * intersection + smooth) / (union + smooth)\n \n # Combine cross entropy and dice losses\n total_loss = ce_loss + dice_loss\n \n return total_loss","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:00.979191Z","iopub.execute_input":"2024-05-29T21:54:00.979509Z","iopub.status.idle":"2024-05-29T21:54:00.986740Z","shell.execute_reply.started":"2024-05-29T21:54:00.979486Z","shell.execute_reply":"2024-05-29T21:54:00.985898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5.4 | Image Augmentation\n\nThe function `augment` is applied to both the image and the label, because when the image is flipped, the label shall follow it. The first part of augmentation is **horizontal and vertical flip** at probability 0.5 respectively. Next is image rotation. **The rotation angle is selected from 0, 90, 180, and 270**. Finally, **the image and the label are zoomed** at 0.3 probability. It selects the upper left pixel randomly. Then the length is selected. It is clipped and resized to 512x512. ## 6.3 Image Augmentation\n\nThe funciton `augment` is applied to both image and label, because when image is flipped, label shall follow it. First part of augmentation is **horizontal and vertical flip** at probability 0.5 respectively. Next is image rotation. **The rotation angle is selected from 0, 90, 180 and 270**. Finally, **the image and the label is zoomed** at 0.3 probability. It select uppler left pixel randomly. Then the length is selected. It is clipped and resized to 512x512. ","metadata":{}},{"cell_type":"code","source":"\n#Image Augment Function\ndef augment(X, Y):\n \n #1. random flip--------------\n #1.1 horizontal\n if tf.random.uniform(shape=[1]) > 0.5:\n X = X[:,:,::-1]\n Y = Y[:,:,::-1]\n \n #1.2 vertical\n if tf.random.uniform(shape=[1]) > 0.5:\n X = X[:,::-1]\n Y = Y[:,::-1]\n \n #2. rotation------------------\n #2.1 set angle \n angle = np.random.randint(4) \n X = tf.image.rot90(X, k=angle)\n Y = tf.image.rot90(Y, k=angle)\n \n #3. zoom -----------------------------\n L = 512\n if tf.random.uniform(shape=[1]) < 0.3:\n \n #left upper\n x1 = np.random.choice(np.arange(0, int(L*0.2)), 1)[0]\n y1 = np.random.choice(np.arange(0, int(L*0.2)), 1)[0]\n \n #Length\n L2 = min(L-x1, L-y1)\n L3 = np.random.choice(np.arange(int(L2*0.8), L2), 1)[0]\n \n X = X[:,y1:(y1+L3), x1:(x1+L3),:] \n Y = Y[:,y1:(y1+L3), x1:(x1+L3),:] \n \n X = tf.cast(tf.image.resize(X,[L, L]), dtype = tf.uint8)\n Y = tf.cast(tf.image.resize(Y,[L, L]), dtype = tf.uint8)\n \n return X, Y","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:00.987625Z","iopub.execute_input":"2024-05-29T21:54:00.987897Z","iopub.status.idle":"2024-05-29T21:54:00.998864Z","shell.execute_reply.started":"2024-05-29T21:54:00.987874Z","shell.execute_reply":"2024-05-29T21:54:00.997966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def apply_augmentation(image, label):\n # Your augmentation operations here\n augmented_image, augmented_label = augment(image, label)\n return augmented_image, augmented_label","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:00.999876Z","iopub.execute_input":"2024-05-29T21:54:01.000245Z","iopub.status.idle":"2024-05-29T21:54:01.014678Z","shell.execute_reply.started":"2024-05-29T21:54:01.000213Z","shell.execute_reply":"2024-05-29T21:54:01.013693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds_augmented = train_ds.map(apply_augmentation)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:01.015853Z","iopub.execute_input":"2024-05-29T21:54:01.016397Z","iopub.status.idle":"2024-05-29T21:54:01.483750Z","shell.execute_reply.started":"2024-05-29T21:54:01.016365Z","shell.execute_reply":"2024-05-29T21:54:01.482763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5.5 | Hyperparameter Tuning\n**General**\n\n* adam optimizer with learning rate = 0.0002. (much better than default lr 0.001) with batch size = 4\n* weight of negative loss (false positive) in `custom_loss` = 3\n* best epoch: 20 ~ 30 \n\n**FCN**\n* Number of conv block at each level = 3\n* Convolution kernel size = 3\n","metadata":{}},{"cell_type":"code","source":"optimizer = tf.keras.optimizers.Adam(learning_rate=0.0002)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:01.485076Z","iopub.execute_input":"2024-05-29T21:54:01.485369Z","iopub.status.idle":"2024-05-29T21:54:01.494557Z","shell.execute_reply.started":"2024-05-29T21:54:01.485344Z","shell.execute_reply":"2024-05-29T21:54:01.493697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\n\n# Define early stopping callback\nearly_stopping = EarlyStopping(monitor='val_loss', patience=5, verbose=1, restore_best_weights=True)\n\n# Define model checkpoint callback to save the best model\nmodel_checkpoint = ModelCheckpoint('best_model.keras', monitor='val_loss', save_best_only=True)\n\n# Define learning rate reduction callback\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, min_lr=1e-9)\n\n# Compile the model with the new optimizer\nmodel_FCN.compile(optimizer=optimizer, loss=vessel_segmentation_loss, metrics=['accuracy'])\n","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:01.495873Z","iopub.execute_input":"2024-05-29T21:54:01.496168Z","iopub.status.idle":"2024-05-29T21:54:01.507897Z","shell.execute_reply.started":"2024-05-29T21:54:01.496142Z","shell.execute_reply":"2024-05-29T21:54:01.506993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train the model using the callbacks\nhistory = model_FCN.fit(train_ds_augmented, epochs=100, validation_data=(X_val, Y_val), callbacks=[model_checkpoint,reduce_lr])","metadata":{"execution":{"iopub.status.busy":"2024-05-29T21:54:01.508911Z","iopub.execute_input":"2024-05-29T21:54:01.509206Z","iopub.status.idle":"2024-05-29T22:07:39.574439Z","shell.execute_reply.started":"2024-05-29T21:54:01.509176Z","shell.execute_reply":"2024-05-29T22:07:39.573626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef plot_history(history):\n fig, ax = plt.subplots(figsize=(5, 4))\n ax.plot(history.history[\"loss\"], label=\"train loss\")\n ax.plot(history.history[\"val_loss\"], label=\"val loss\")\n ax.set_title(\"Loss\")\n ax.legend()\n ax.grid()\n\n# Assuming you have already trained your model and obtained the `history` object\nplot_history(history)\n","metadata":{"execution":{"iopub.status.busy":"2024-05-29T22:07:39.579293Z","iopub.execute_input":"2024-05-29T22:07:39.580216Z","iopub.status.idle":"2024-05-29T22:07:39.928243Z","shell.execute_reply.started":"2024-05-29T22:07:39.580175Z","shell.execute_reply":"2024-05-29T22:07:39.927300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n# 7 | Predicted labels\n\n\n \n FCN is trained well, it must be able to predict labels with good accuracy. The function `plot_result` visualizes an image, its label, and prediction results ","metadata":{}},{"cell_type":"code","source":"def plot_result(X, Y_true, Y_pred, cutoff = 0.5):\n \n N = X.shape[0]\n \n fig, ax = plt.subplots(N,4, figsize = (10, 3*N))\n \n for k in range(N):\n \n cutoff_img1 = (Y_pred[k,:,:,0] > cutoff).astype(int)\n\n true_img = np.zeros((512, 512, 3), dtype = np.uint8)\n true_img[:,:,1] = Y_true[k,:,:,0]*200\n \n cutoff1 = np.zeros((512, 512, 3), dtype = np.uint8)\n \n cutoff1[:,:,0] = cutoff_img1*230\n \n cutoff1[:,:,1] = cutoff_img1*50\n cutoff1[:,:,2] = cutoff_img1*50\n \n diff_photo1 = cutoff1.copy()\n diff_photo1[:,:,1] += (Y_true[k,:,:,0]*200).astype(np.uint8)\n \n ax[k, 0].imshow(X[k])\n ax[k, 1].imshow(true_img, cmap = \"gray\")\n ax[k, 2].imshow(cutoff1, cmap = \"gray\")\n ax[k, 3].imshow(diff_photo1)\n \n for j in range(4):\n ax[k,j].set_xticks([])\n ax[k,j].set_yticks([])\n \n if k == 0:\n ax[k, 0].set_title(\"val img\")\n ax[k, 1].set_title(\"true label\")\n ax[k, 2].set_title(\"model (cutoff at {})\".format(cutoff))\n ax[k, 3].set_title(\"Compare (Y:tp)\")","metadata":{"execution":{"iopub.status.busy":"2024-05-29T22:08:08.003799Z","iopub.execute_input":"2024-05-29T22:08:08.004159Z","iopub.status.idle":"2024-05-29T22:08:08.016209Z","shell.execute_reply.started":"2024-05-29T22:08:08.004126Z","shell.execute_reply":"2024-05-29T22:08:08.015230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Results**\n\nTen of the prediction results are shown below. Both models predict somewhat well, although it is not a simple task. The true label is **green** and the predicted label is **red**. The predicted label is created by cutting off the predicted probability at 0.8. When both are plotted in the same image (refer to **Compare (Y: tp)**), the true positive becomes **yellow**. Precisions of prediction are varied, some are very well predicted but some are not. The next question would be \"Does the result depend on the dataset?\" ","metadata":{}},{"cell_type":"code","source":"n_val = len(X_val)\nval_sample = np.random.choice(n_val, 10)\nplot_result(X_val[val_sample], Y_val[val_sample], Y_hat[val_sample], cutoff=best_threshold)","metadata":{"execution":{"iopub.status.busy":"2024-05-29T22:08:08.017331Z","iopub.execute_input":"2024-05-29T22:08:08.017638Z","iopub.status.idle":"2024-05-29T22:08:11.643865Z","shell.execute_reply.started":"2024-05-29T22:08:08.017612Z","shell.execute_reply":"2024-05-29T22:08:11.642896Z"},"trusted":true},"execution_count":null,"outputs":[]}]}
![\"Human\"](\"https://i.pinimg.com/originals/a4/7b/9a/a47b9a5a523cb4d49f79e2d73fee636d.gif\")
\n