{"cells":[{"metadata":{"_uuid":"f97f10bf-0cc6-49db-8cf5-5dd21ac1321b","_cell_guid":"b2c003db-3917-42b2-bdae-56ad2070d619","trusted":true,"id":"II9FPOosLk4C"},"cell_type":"markdown","source":"In this project, wa are asked to use image segmentation to identify the boundaries of a car in an image.\n\nThe training set contains 5088 of car images. Each car has 16 images taken at different angles, so we have 318 unique cars. We are also provided with the mask for each image in the training set. The test set contains 100 064 car images, again each car has 16 different images, so there are 6254 unique cars in the test set.\n\nYou can found the dataset [here](https://www.kaggle.com/c/carvana-image-masking-challenge/data?select=train.zip)"},{"metadata":{"_uuid":"89e48a34-f895-4e84-b65c-0a94648d0aa4","_cell_guid":"7eb185f2-a075-4ad6-b9d8-ba0f725e388b","trusted":true,"id":"T8JdO5w3MUL7"},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_uuid":"52fdc803-0579-4907-8af2-e01fbe998fbb","_cell_guid":"4d1924a6-92d3-402b-8110-1c3f7b50b444","trusted":true,"id":"-h1VwNw1MW9P"},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imread\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import Model, Sequential","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this notebook, I only used the training set (the test set is too big), this is why I commented the lines below."},{"metadata":{"_uuid":"f0271c05-ec5c-43d4-88fd-e4c9b0fba34a","_cell_guid":"29be8de1-ea06-4a6a-aa61-4b98dd5caf6b","trusted":true},"cell_type":"code","source":"from pathlib import Path\nimport zipfile\n\ntrain_zip_path = '../input/carvana-image-masking-challenge/train.zip'\nmasks_zip_path = '../input/carvana-image-masking-challenge/train_masks.zip'\ntest_zip_path = '../input/carvana-image-masking-challenge/test.zip'\n\nif not Path('/kaggle/working/train').exists():\n    with zipfile.ZipFile(train_zip_path,'r') as z:\n        z.extractall('/kaggle/working')\nif not Path('/kaggle/working/train_masks').exists():\n    with zipfile.ZipFile(masks_zip_path,'r') as z:\n        z.extractall('/kaggle/working')\nif not Path('/kaggle/working/test').exists():\n    pass\n    # with zipfile.ZipFile(test_zip_path,'r') as z:\n    #    z.extractall('/kaggle/working')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b426b27e-b060-4644-ac34-aa843307d1b5","_cell_guid":"d331532f-43ce-4534-a538-c54dccd9f636","trusted":true,"id":"RbQRxCZSMSyU"},"cell_type":"markdown","source":"# Analysis"},{"metadata":{"_uuid":"99d7a48f-42af-4bc9-a2dd-070d004dcb39","_cell_guid":"afd9193b-f7e5-4d91-8acd-5077a198c018","trusted":true,"id":"-ojB8nrwQa4q"},"cell_type":"markdown","source":"How many images do we have in the training and test set"},{"metadata":{"_uuid":"b92123cb-7a6f-4f09-8506-942b0279eccf","_cell_guid":"7160f213-3c2a-4f17-b007-fed5e78071a6","trusted":true,"id":"JgQTUGhTM2iy","outputId":"7d8db59d-2216-47c3-9c9e-ca5d9d993895"},"cell_type":"code","source":"print(\"train set:  \", len(os.listdir(\"/kaggle/working/train\")))\nprint(\"train masks:\", len(os.listdir(\"/kaggle/working/train_masks\")))\n#print(\"test set:   \", len(os.listdir(\"/kaggle/working/test\")))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d1b02d2-b182-42cb-889c-9f9f936dbef2","_cell_guid":"d2e31f36-a187-4e4c-a89e-18520906188c","trusted":true,"id":"HZaDw_iTO21Z"},"cell_type":"code","source":"from glob import glob\n\nroot_dir = \"/kaggle/working\"\ntrain_path = os.path.join(root_dir, \"train\")\ntrain_masks_path = os.path.join(root_dir, \"train_masks\")\n#test_path = os.path.join(root_dir, \"test\")\n\ntrain_filepaths = glob(os.path.join(train_path, \"*.jpg\"))\ntrain_masks_filepaths = glob(os.path.join(train_masks_path, \"*.gif\"))\n#test_filepaths = glob(os.path.join(test_path, \"*.jpg\"))\n\n# Get unique ids of images\ndef get_root_name(filepaths):\n    file_names = [os.path.basename(filepath) for filepath in filepaths]\n    root_name = [name.split(\"_\")[0] for name in file_names]\n    return root_name\n\nall_train_ids = set(get_root_name(train_filepaths))\nall_train_masks_ids = set(get_root_name(train_masks_filepaths))\n#all_test_ids = set(get_root_name(test_filepaths))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"80b2f323-2cab-4a0e-9454-3b68a2cb808f","_cell_guid":"62db4369-1805-4520-83bd-95ccdcc66168","trusted":true,"id":"NjAvHsHModxa"},"cell_type":"markdown","source":"How many different cars in training and test set"},{"metadata":{"_uuid":"34d4fecf-4804-46e5-8463-f8a514c9a215","_cell_guid":"73a58cd9-eb19-4c5c-ba0e-353093c73ce9","trusted":true,"id":"9JGRmjA5odY8","outputId":"ec9bd435-1329-43c9-ff08-73aa676202fe"},"cell_type":"code","source":"print(\"training set:       \", len(all_train_ids), \" different cars\")\nprint(\"training masks set: \", len(all_train_masks_ids), \"different cars\")\n#print(\"test set:           \", len(all_test_ids), \"different cars\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72bb204e-11fa-4343-bc85-fdf9fbdbd9f0","_cell_guid":"0e897b22-4dd8-4116-9e99-8eaadd0101c6","trusted":true,"id":"MrtnCzbCi5t6"},"cell_type":"markdown","source":"Display some cars with their masks"},{"metadata":{"_uuid":"8d0c0c23-0408-4ef4-8e2e-c5d57a59838e","_cell_guid":"1f5bff2d-ef92-439c-a202-91f3050f5264","trusted":true,"id":"Q9biz9KMluQ4","outputId":"3e0ac915-1ae9-44a9-af11-1ddf2b315ff5"},"cell_type":"code","source":"def display_images():\n    plt.figure(figsize=(15, 25))\n    title = ['Input Image', 'Mask']\n\n    for i in range(0, 10, 2):\n        plt.subplot(5, 2, i+1)\n        plt.title(title[0])\n        path_img = root_dir + \"/train/\" + list(all_train_ids)[i] + f\"_0{i+1}.jpg\"\n        plt.imshow(imread(path_img))\n        plt.axis(\"off\")\n\n        plt.subplot(5, 2, i+2)\n        plt.title(title[1])\n        path_mask_img = root_dir + \"/train_masks/\" + list(all_train_ids)[i] + f\"_0{i+1}_mask.gif\"\n        plt.imshow(imread(path_mask_img))\n        plt.axis(\"off\")\n    plt.show()\n\ndisplay_images()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7cd06ae-889a-4ab8-9998-97f03d109463","_cell_guid":"43c28d8c-3aca-4f31-80ac-b8d70661419d","trusted":true,"id":"cdJaYX_3Zsz9"},"cell_type":"markdown","source":"# Training and Validation Split\n\nWe split the data by car to avoid overfitting single images."},{"metadata":{"_uuid":"96ed9dca-e6e5-4330-8457-740944cbe1bf","_cell_guid":"f1284391-59d7-41bd-986c-1912dff1be63","trusted":true,"id":"A02l0tsEaSxt"},"cell_type":"code","source":"def get_image_id(path):\n    return os.path.splitext(os.path.basename(path))[0]\n\ndf = pd.DataFrame(dict(image_path=train_filepaths))\ndf['image_id'] = df['image_path'].map(lambda path: get_image_id(path))\ndf['mask_path'] = df['image_path'].map(\n    lambda x: x.replace('train', 'train_masks').replace('.jpg', '_mask.gif'))\ndf['car_id'] = df['image_id'].map(lambda img_id: img_id.split('_')[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43c99cfc-4214-4ddf-805e-0a2890955dc0","_cell_guid":"eb147c09-4a03-41e9-93a7-8568cb5d7ad8","trusted":true,"id":"aXxp20ZpCZpq","outputId":"2d5c9be1-7c8f-4c98-b764-fb6a5fb2b5b2"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndef split_data(ids, col=\"car_id\"):\n    train_ids, valid_ids = train_test_split(ids, random_state=42, test_size=.2)\n    valid_ids, test_ids = train_test_split(valid_ids, random_state=42, test_size=.5)\n    train_df = df[df[col].isin(train_ids)]\n    valid_df = df[df[col].isin(valid_ids)]\n    test_df = df[df[col].isin(test_ids)]\n    return train_df, valid_df, test_df\n\ntrain_df, valid_df, test_df = split_data(list(all_train_ids))\nprint(\"train_df: \", train_df.shape[0])\nprint(\"valid_df: \", valid_df.shape[0])\nprint(\"test_df:  \", test_df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e22003b-9785-4392-9611-838166ca91c2","_cell_guid":"01543605-8999-46b8-836d-adb97835bb05","trusted":true,"id":"KkBRWgblMFfb","outputId":"f68527c8-92ba-4101-cbd4-bc112a7075d0"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cb052c6-7c35-4dd5-a17f-7b36c8012c2d","_cell_guid":"be4495fc-4325-4797-93cb-8e794efa09f8","trusted":true,"id":"dURr9Zx1dJLz"},"cell_type":"markdown","source":"# Preprocessing the data\n\nHere, we build an input pipline using tf.data. We first load the data from the dataframe above, then we preprocess the image and the mask, and finally, we perform a simple data augmentation."},{"metadata":{"_uuid":"973bd280-7149-4487-bc55-e72f2b06cf7d","_cell_guid":"3aed48f7-8287-49f6-bbe7-1f3ed338fdd6","trusted":true,"id":"ZZXvcYzNQf9A"},"cell_type":"code","source":"from tensorflow.image import stateless_random_crop, stateless_random_brightness\n\nIMG_SIZE = [512, 512]\nrng = tf.random.Generator.from_seed(1)\n\ndef decode(path):\n    img = tf.io.read_file(path) \n    img = tf.image.decode_jpeg(img, channels=3)\n    img = tf.image.resize(img, IMG_SIZE)\n    img = img / 255.0\n    return img\n\n@tf.function\ndef preprocess(image_path, mask_path):\n    image = decode(image_path)\n    mask = decode(mask_path)\n    mask = mask[:, :, :1] # take one channel\n    return image, mask\n\n@tf.function\ndef data_augmentation(image, mask):\n    if rng.uniform(()) > 0.5: \n        image = tf.image.flip_left_right(image)\n        mask = tf.image.flip_left_right(mask)\n\n    seed = rng.make_seeds(2)[0]\n    image = stateless_random_brightness(image, max_delta=0.1, seed=seed)\n    return image, mask\n\ndef make_dataset(df, shuffle=False, augment=False, batch_size=16, buffer_size=1000):\n    ds = tf.data.Dataset.from_tensor_slices((df[\"image_path\"].values, df[\"mask_path\"].values))\n    ds = ds.map(preprocess, num_parallel_calls=5)\n    if shuffle:\n        ds = ds.shuffle(buffer_size)\n    if augment:\n        ds = ds.map(data_augmentation, num_parallel_calls=tf.data.AUTOTUNE)\n    ds = ds.batch(batch_size)\n    return ds.prefetch(1)\n\ntrain_data = make_dataset(train_df, shuffle=True, augment=True)\nvalid_data = make_dataset(valid_df)\ntest_data = make_dataset(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ecd6ff64-806c-40b4-9e2a-5369e99550dc","_cell_guid":"483b06f7-822a-4519-9ec2-c34dda1e223d","trusted":true,"id":"ISEgFqOwu2x2"},"cell_type":"markdown","source":"# Define the Model\n"},{"metadata":{"_uuid":"3f00a612-39f3-4f05-9333-c500dd6fa205","_cell_guid":"e716f166-d87d-4ab4-a6b2-c37236d35d25","trusted":true,"id":"KNikbhv-U0j6"},"cell_type":"code","source":"def upsample(filters, size, strides, dropout=None):\n    \"\"\"Upsample the input\"\"\"\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n\n    result = Sequential()\n    result.add(tf.keras.layers.Conv2DTranspose(filters, size, strides=strides,\n                                      padding=\"same\",\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n    result.add(tf.keras.layers.BatchNormalization())\n    if dropout:\n        result.add(tf.keras.layers.Dropout(dropout))\n    result.add(tf.keras.layers.ReLU())   \n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To have a better performance, I used the layers of the VGG19 as the encoder for the U-Net model. The decoder will be a series of the upsample block above."},{"metadata":{"_uuid":"47f16aec-7904-4458-bf32-eb9bcf75ac87","_cell_guid":"2669daec-5b08-491d-8772-6916afc1ab9a","trusted":true,"id":"dQ_SgBljx0BF"},"cell_type":"code","source":"from tensorflow.keras.applications import VGG19\n\nbase_model = VGG19(input_shape=IMG_SIZE + [3], include_top=False, weights=\"imagenet\")\n\nlayers_names = [\n    \"block2_conv1\",    # 256x256\n    \"block2_conv2\",    # 256x256\n    \"block3_conv1\",    # 128x128\n    \"block3_conv2\",    # 128x128\n    \"block4_conv1\",    # 64x64\n    \"block4_conv2\",    # 64x64\n    \"block5_conv1\",    # 32x32\n]\n\nlayers = [base_model.get_layer(name).output for name in layers_names]\ndown_stack = Model(inputs=base_model.input, outputs=layers)\ndown_stack.trainable = False\n\nup_stack = [\n    upsample(512, 3, 1),   # 32x32 -> 32x32\n    upsample(512, 3, 2),   # 32x32 -> 64x64\n    upsample(256, 3, 1),   # 64x64 -> 64x64 \n    upsample(256, 3, 2),   # 64x64 -> 128x128\n    upsample(128, 3, 1),   # 128x128 -> 128x128\n    upsample(128, 3, 2),   # 128x128 -> 256x256\n]     ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98007705-bc4a-4774-b400-2534a3664c90","_cell_guid":"6aafc653-3587-4e87-be69-3f0981a9ccc3","trusted":true,"id":"GyDqwirDwB_i"},"cell_type":"markdown","source":"# Build the U-Net model"},{"metadata":{"_uuid":"a91bbda1-210f-4cec-ba33-5c8bd708be8a","_cell_guid":"36c7e02f-c781-4111-9c6e-b1ac1a0071d6","trusted":true,"id":"rc9DEcC8s1sF"},"cell_type":"code","source":"keras.backend.clear_session()\n\ndef unet_generator(output_channels=1):\n    inputs = tf.keras.layers.Input(shape=IMG_SIZE + [3])\n    x = inputs\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    output = tf.keras.layers.Conv2DTranspose(\n        output_channels, 3, strides=2, activation='sigmoid',\n        padding=\"same\", kernel_initializer=initializer\n    )\n    \n    concat = tf.keras.layers.Concatenate()\n\n    # Downsampling \n    skips = down_stack(x)\n    x = skips[-1]\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connection\n    for up, skip in zip (up_stack, skips):\n        x = up(x)\n        if up.layers[0].strides == (2, 2):\n            concat = tf.keras.layers.Concatenate()\n            x = concat([x, skip])\n\n    x = output(x)\n    \n    return tf.keras.Model(inputs=inputs, outputs=x)\n\nmodel = unet_generator()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0400a558-4f74-4fa3-a8d7-3c455ee67991","_cell_guid":"fbcdbefc-b7eb-4322-a9ec-2d1ebc66fc02","trusted":true},"cell_type":"code","source":"keras.utils.plot_model(model, show_shapes=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is what the model predict before training"},{"metadata":{"trusted":true},"cell_type":"code","source":"for images, masks in train_data.take(1):\n    for img, mask in zip(images, masks):\n        sample_image = img\n        sample_mask = mask\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize(display_list):\n    plt.figure(figsize=(15, 15))\n    title = ['Input Image', 'True Mask', 'Predicted Mask']\n    for i in range(len(display_list)):\n        plt.subplot(1, len(display_list), i+1)\n        plt.title(title[i])\n        plt.imshow(tf.keras.preprocessing.image.array_to_img(display_list[i]))\n        plt.axis('off')\n    plt.show()\n\ndef show_predictions(sample_image, sample_mask):\n    pred_mask = model.predict(sample_image[tf.newaxis, ...])\n    pred_mask = pred_mask.reshape(IMG_SIZE[0],IMG_SIZE[1],1)\n    visualize([sample_image, sample_mask, pred_mask])\n    \nshow_predictions(sample_image, sample_mask)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba56524b-3352-41bd-b0f3-8bd91cdc4608","_cell_guid":"8684a619-5bf0-4e74-b763-d2657289356c","trusted":true,"id":"CFqXTEGmhE_O"},"cell_type":"markdown","source":"# Losses\n\nI saw in other kaggle kernels that users used the dice + binary cross-entropy loss but since only the binary cross-entropy gives me good results I didn't try the dice loss."},{"metadata":{"_uuid":"19f220c7-616d-47a2-a551-05c0d6c67dcc","_cell_guid":"34ea5743-1e93-479a-859b-a119331d12c4","trusted":true,"id":"UnYQFtHS4ANh","outputId":"f74e07ec-02d6-4360-fc0c-e6ec8fe898a0","collapsed":true},"cell_type":"code","source":"early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=4,\n                                                    restore_best_weights=True)\n\n# Maybe needs to train for more epochs, but kaggle time is limited\nepochs = 10\n\nclass DisplayCallback(tf.keras.callbacks.Callback):\n    def on_epoch_begin(self, epoch, logs=None):\n        if (epoch + 1) % 5 == 0:\n            show_predictions(sample_image, sample_mask)\n    \nmodel.compile(optimizer='adam',\n              loss=tf.keras.losses.BinaryCrossentropy(),\n              metrics=['accuracy'])\nmodel_history = model.fit(train_data, epochs=epochs,\n                          validation_data=valid_data,\n                          callbacks=[DisplayCallback(), early_stopping_cb])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning Curves"},{"metadata":{"trusted":true},"cell_type":"code","source":"loss = model_history.history['loss']\nval_loss = model_history.history['val_loss']\n\nacc = model_history.history['accuracy']\nval_acc = model_history.history['val_accuracy']\n\nplt.figure(figsize=(20, 5))\nplt.subplot(1, 2, 1)\nplt.plot(acc, label='Training Accuracy')\nplt.plot(val_acc, label='Validation Accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend()\n\nplt.subplot(1, 2, 2)\nplt.plot(loss, label='Training Loss')\nplt.plot(val_loss, label='Validation Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Binary Cross Entropy')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation on the Test Set"},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"for images, masks in test_data.take(1):\n    for img, mask in zip(images, masks):\n        show_predictions(img, mask)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ressources\n\n* [Image segmentation](https://www.tensorflow.org/tutorials/images/segmentation) tutorial from TensorFlow.\n* [VGG16+U-Net on Carvana](https://www.kaggle.com/kmader/vgg16-u-net-on-carvana) kaggle kernel.\n* [.99 loss - Simple Sol. using u-Net [Keras/TF] ](https://www.kaggle.com/phylake1337/99-loss-simple-sol-using-u-net-keras-tf) kaggle kernel.\n* [Data visualization](https://www.kaggle.com/vfdev5/data-visualization) kaggle kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}