{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Building a simple UNet-like model\n\nWe will build a simple **UNet-like model** using our generated data of images and masks in png format. You can refer to **[this notebook](https://www.kaggle.com/code/dingyan/hubmap-images-and-masks-original-size-3000x3000)** for generating images and masks.\n\nIf you feel this notebook is helpful, please **upvote**! Thank you.","metadata":{}},{"cell_type":"markdown","source":"**1. Get the images and masks path from our generated data. Random shuffle images and masks.**","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport os\nimport numpy as np\nfrom tqdm import tqdm\nimport random\nimport tifffile as tiff\nimport pandas as pd\n\nimg_size = 512\nimage_dir = '../input/hubmap-data/HuBMAP_train/images'\nmask_dir = '../input/hubmap-data/HuBMAP_train/masks'\n\nimage_path_list = os.listdir(image_dir)\nmask_path_list = os.listdir(mask_dir)\n\n# Shuffle the file paths (they were originally sorted by breed). \n# We use the same seed (1337) in both statements to ensure that \n# the input paths and target paths stay in the same order.\nrandom.Random(1337).shuffle(image_path_list)\nrandom.Random(1337).shuffle(mask_path_list)\n\nimg_num = len(image_path_list)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:43:11.420498Z","iopub.execute_input":"2022-08-07T07:43:11.421395Z","iopub.status.idle":"2022-08-07T07:43:18.098438Z","shell.execute_reply.started":"2022-08-07T07:43:11.421267Z","shell.execute_reply":"2022-08-07T07:43:18.097371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**2. Load images**\n\nWe use *keras.utils.load_img* to load images and masks and we can specify *target_size* (here we resize images to 512) to resize images which is convenient. We need to convert images to numpy array with *keras.utils.img_to_array*. Be careful, the values are range from 0 to 255. So we will need to rescale it to 0 - 1.","metadata":{}},{"cell_type":"code","source":"def get_image(path, img_size):\n    image = keras.utils.img_to_array(keras.utils.load_img(path, target_size=(img_size, img_size)))\n    return image/255.","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:43:24.246200Z","iopub.execute_input":"2022-08-07T07:43:24.247113Z","iopub.status.idle":"2022-08-07T07:43:24.252745Z","shell.execute_reply.started":"2022-08-07T07:43:24.247077Z","shell.execute_reply":"2022-08-07T07:43:24.251367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**3. Load masks**\n\nMasks we generated are already in range 0 - 1 so we don't need to rescale it. Just be careful that data type is in **float32** when we load images and convert to array with *keras.utils.img_to_array* . So we need to cast float32 to uint8 for model training. In this competition, we only need to classify the mask so it is a **binary classification** just like foreground and background.","metadata":{}},{"cell_type":"code","source":"def get_mask(path, img_size):\n    mask = keras.utils.img_to_array(keras.utils.load_img(path, target_size=(img_size, img_size), color_mode='grayscale'))\n    mask = tf.cast(mask, tf.uint8)\n    return mask","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:43:26.501047Z","iopub.execute_input":"2022-08-07T07:43:26.501494Z","iopub.status.idle":"2022-08-07T07:43:26.507090Z","shell.execute_reply.started":"2022-08-07T07:43:26.501440Z","shell.execute_reply":"2022-08-07T07:43:26.505975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**4. Since the dataset is not very big, we can just load everything into memory:**","metadata":{}},{"cell_type":"code","source":"input_images = np.zeros(shape=(img_num, img_size, img_size, 3), dtype='float32')\ninput_masks = np.zeros(shape=(img_num, img_size, img_size, 1), dtype='uint8')\n\nfor i in tqdm(range(len(image_path_list))):\n    image_path = os.path.join(image_dir, image_path_list[i])\n    mask_path = os.path.join(mask_dir, mask_path_list[i])\n    input_images[i, :, :, :] = get_image(image_path, img_size)\n    input_masks[i, :, :, :] = get_mask(mask_path, img_size)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:43:30.519532Z","iopub.execute_input":"2022-08-07T07:43:30.519958Z","iopub.status.idle":"2022-08-07T07:46:56.365496Z","shell.execute_reply.started":"2022-08-07T07:43:30.519919Z","shell.execute_reply":"2022-08-07T07:46:56.364471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**5. Let's take a look at the masked images**","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(15, 15))\nfor i in range(3*3):\n    plt.subplot(3, 3, i+1)\n    plt.imshow(input_images[i])\n    plt.imshow(input_masks[i], alpha=0.3)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:47:20.599421Z","iopub.execute_input":"2022-08-07T07:47:20.599944Z","iopub.status.idle":"2022-08-07T07:47:22.755647Z","shell.execute_reply.started":"2022-08-07T07:47:20.599902Z","shell.execute_reply":"2022-08-07T07:47:22.754395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The images and masks are ready for feeding into the model.","metadata":{}},{"cell_type":"markdown","source":"**6. We will only reserve 30 samples for validation as we don't have a lot of training data**","metadata":{}},{"cell_type":"code","source":"val_num = 30\ntrain_images = input_images[:-val_num]\ntrain_masks = input_masks[:-val_num]\nval_images = input_images[-val_num:]\nval_masks = input_masks[-val_num:]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:47:28.622837Z","iopub.execute_input":"2022-08-07T07:47:28.623286Z","iopub.status.idle":"2022-08-07T07:47:28.632975Z","shell.execute_reply.started":"2022-08-07T07:47:28.623245Z","shell.execute_reply":"2022-08-07T07:47:28.631909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**7. Build the model**\n\nThe first half of the model closely resembles a stack of Conv2D layers, with gradually increasing filter sizes, ending up with activations of size\n(32, 32, 512). It is like a kind of compression.\n\nThe second half of the model is a stack of Conv2DTranspose layers to get our final output to have the same shape as the target masks which is (512, 512, 1)\n\nThe last layers is used to do binary classification for mask so activation = 'sigmoid'. Let's first try loss = 'binary_crossentropy'.","metadata":{}},{"cell_type":"code","source":"# smaller network\ntf.keras.backend.clear_session()\ndef get_model(image_size, num_classes):\n    inputs = keras.Input(shape=(image_size, image_size, 3))\n    x = layers.Conv2D(32, 3, strides=2, activation='relu', padding='same')(inputs)\n    x = layers.Conv2D(32, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2D(64, 3, strides=2, activation='relu', padding='same')(x)\n    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2D(128, 3, strides=2, activation='relu', padding='same')(x)\n    x = layers.Conv2D(128, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2D(256, 3, strides=2, activation='relu', padding='same')(x)\n    x = layers.Conv2D(256, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2DTranspose(256, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2DTranspose(256, 3, activation='relu', strides=2, padding='same')(x)\n    x = layers.Conv2DTranspose(128, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2DTranspose(128, 3, activation='relu', strides=2, padding='same')(x)\n    x = layers.Conv2DTranspose(64, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2DTranspose(64, 3, activation='relu', strides=2, padding='same')(x)\n    x = layers.Conv2DTranspose(32, 3, activation='relu', padding='same')(x)\n    x = layers.Conv2DTranspose(32, 3, activation='relu', strides=2, padding='same')(x)\n    outputs = layers.Conv2D(num_classes, 3, activation='sigmoid', padding='same')(x)\n    model = keras.Model(inputs, outputs)\n    return model\nmodel = get_model(img_size, 1)\nmodel.compile(loss='binary_crossentropy',\n              optimizer='rmsprop',\n              metrics=['accuracy'])\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T04:15:15.462693Z","iopub.execute_input":"2022-08-07T04:15:15.463566Z","iopub.status.idle":"2022-08-07T04:15:15.655261Z","shell.execute_reply.started":"2022-08-07T04:15:15.463528Z","shell.execute_reply":"2022-08-07T04:15:15.654173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=30), keras.callbacks.ModelCheckpoint('mymodel', save_best_only=True)]\nhistory = model.fit(train_images, train_masks, epochs=100, validation_data=(val_images, val_masks), batch_size=4, callbacks=callbacks)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T04:15:20.410552Z","iopub.execute_input":"2022-08-07T04:15:20.410927Z","iopub.status.idle":"2022-08-07T04:24:27.969288Z","shell.execute_reply.started":"2022-08-07T04:15:20.410895Z","shell.execute_reply":"2022-08-07T04:24:27.968342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epochs = range(len(history.history['loss']))\nplt.plot(epochs[10:], history.history['loss'][10:], label='train loss')\nplt.plot(epochs[10:], history.history['val_loss'][10:], label='val loss')\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T04:25:30.581208Z","iopub.execute_input":"2022-08-07T04:25:30.582294Z","iopub.status.idle":"2022-08-07T04:25:46.381293Z","shell.execute_reply.started":"2022-08-07T04:25:30.582256Z","shell.execute_reply":"2022-08-07T04:25:46.380404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**8. Visualize the predictions on validation images.**","metadata":{}},{"cell_type":"code","source":"threshold = 0.5 # here I set 0.5 as threshold to classify the mask to be 0 or 1 \n\nmodel = keras.models.load_model('./mymodel')\n\ndef visualize_predictions(images, masks, ids):\n    plt.figure(figsize=(cols*5, rows*5))\n    for i in range(0, rows*cols, 2):\n        plt.subplot(rows, cols, i+1)\n        plt.axis('off')\n        plt.title('Groundtruth')\n        plt.imshow(images[ids[i]])\n        plt.imshow(masks[ids[i]], alpha=0.3)\n        \n        plt.subplot(rows, cols, i+2)\n        plt.title('Prediction')\n        plt.axis('off')\n        preds = model.predict(np.expand_dims(images[ids[i]], 0))[0]\n        pred_mask = np.where(preds > threshold, 1, 0)\n        plt.imshow(images[ids[i]])\n        plt.imshow(pred_mask, alpha=0.3)\n\nrows = 5\ncols = 2\nsampled_ids = np.random.choice(range(len(val_images)), rows*cols, replace=False)\nvisualize_predictions(val_images, val_masks, sampled_ids)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:48:05.592660Z","iopub.execute_input":"2022-08-07T07:48:05.593032Z","iopub.status.idle":"2022-08-07T07:48:16.544850Z","shell.execute_reply.started":"2022-08-07T07:48:05.593003Z","shell.execute_reply":"2022-08-07T07:48:16.543669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks pretty good in validation data given such a simple model. Moreover we haven't add any data augmentation yet. \n\n**Next steps we can try:**\n1. Add data augmentation.\n2. Try different model structures or loss functions.\n3. Seek some public external data.\n4. ...\n\nIf you have good ideas, please leave comments below. ","metadata":{}},{"cell_type":"markdown","source":"**9. Submission**","metadata":{}},{"cell_type":"code","source":"# ref.: https://www.kaggle.com/stainsby/fast-tested-rle\ndef rle_encode(img):\n    \"\"\" TBD\n    \n    Args:\n        img (np.array): \n            - 1 indicating mask\n            - 0 indicating background\n    \n    Returns: \n        run length as string formated\n    \"\"\"\n    pixels = img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\ndef preprocess_tiff_image(path):\n    image_array = tiff.imread(path)\n    original_shape = image_array.shape\n    image_processed = keras.utils.img_to_array(keras.utils.array_to_img(image_array).resize((512, 512)))/255\n    return image_processed, original_shape\n","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:54:58.335795Z","iopub.execute_input":"2022-08-07T07:54:58.336799Z","iopub.status.idle":"2022-08-07T07:54:58.345284Z","shell.execute_reply.started":"2022-08-07T07:54:58.336763Z","shell.execute_reply":"2022-08-07T07:54:58.344147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**First, let's visualize the sample test image**","metadata":{}},{"cell_type":"code","source":"sample_test_path = '../input/hubmap-organ-segmentation/test_images/10078.tiff'\nimage, _ = preprocess_tiff_image(sample_test_path)\npred = model.predict(np.expand_dims(image, axis=0))\npred_mask = np.where(pred > threshold, 1, 0)[0]\nplt.figure(figsize=(6, 6))\nplt.axis('off')\nplt.imshow(image)\nplt.imshow(pred_mask, alpha=0.3)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:59:15.858553Z","iopub.execute_input":"2022-08-07T07:59:15.858950Z","iopub.status.idle":"2022-08-07T07:59:16.367875Z","shell.execute_reply.started":"2022-08-07T07:59:15.858919Z","shell.execute_reply":"2022-08-07T07:59:16.367014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**At last, let's generate the submission file. Don't forget to resize the mask to original image size.**","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv('../input/hubmap-organ-segmentation/test.csv')\ntest_ids = test_df['id']\ntest_dir = '../input/hubmap-organ-segmentation/test_images'\n\nids = []\nrles = []\nfor id in test_ids:\n    path = os.path.join(test_dir, f\"{id}.tiff\")\n    image, original_shape = preprocess_tiff_image(path)\n    pred = model.predict(np.expand_dims(image, axis=0))\n    pred_mask = np.where(pred > 0.5, 1, 0)[0]\n    resized_pred_mask = keras.utils.array_to_img(pred_mask, scale=False).resize((original_shape[0], original_shape[1]), resample=0)\n    resized_pred_mask_array = keras.utils.img_to_array(resized_pred_mask, dtype='uint8')\n    rle = rle_encode(resized_pred_mask_array)\n    ids.append(id)\n    rles.append(rle)\n    \nsubmission_df = pd.DataFrame({'id':ids,'rle':rles})\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:06:38.035831Z","iopub.execute_input":"2022-08-07T08:06:38.039049Z","iopub.status.idle":"2022-08-07T08:06:38.451361Z","shell.execute_reply.started":"2022-08-07T08:06:38.039002Z","shell.execute_reply":"2022-08-07T08:06:38.450277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:07:05.286647Z","iopub.execute_input":"2022-08-07T08:07:05.287288Z","iopub.status.idle":"2022-08-07T08:07:05.297504Z","shell.execute_reply.started":"2022-08-07T08:07:05.287251Z","shell.execute_reply":"2022-08-07T08:07:05.296374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is a baseline for quickly building an image segmentation model. If you feel helpful, please **upvote**! ","metadata":{}}]}