{"cells":[{"metadata":{},"cell_type":"markdown","source":"#### This a simple and concise kernel to handle masking problem, I will also put some links for other helpful kernels, discussions, and videos that will answer all your questions and help you to the end.\n​\n* This problem is a segmentation problem in the core, which means that we can't use regular architectures/models eg:(vgg, resnet, etc), all those models are created for classification and regression problems.\n* The output we seek in this problem is different, it's not just a vector of values, it's an image which has the same size of the input image. So we're gonna use a special model called uNet which is an autoencoder in the core, here's a super helpful video for segmentation and uNet [video link](https://www.youtube.com/watch?v=azM57JuQpQI&t=945s).\n* After watching that video you should now know what's the segmentation problem and how uNet is different, if it isn't the case for you, then I encourage you to see this wonderful video from **Jeremy Howard** which explain uNet architecture and the segmentation problem then go through a solution for the same problem we are trying to solve (Caravana Competition), it's more than enough for us, isn't? [Jermey's Video](http://https://www.youtube.com/watch?v=nG3tT31nPmQ).\n* Also here are some helpful discussions from LB masters in which they are sharing their experiments and answer some good questions [Heng's Discussion and Pytorch Solution](https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37208) -- [Petros's Discussion and Keras Solution](https://www.kaggle.com/c/carvana-image-masking-challenge/discussion/37523).\n* Code is simple, concise and fully-commented. Feel free to ask for help / more info / more explanation in the comments.\n* Finally if this kernel helps you somehow, kindly don't forget to leave a little upvote.\n* Hope you enjoy it."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":false},"cell_type":"code","source":"import os, keras\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport keras.backend as K\n\nfrom sklearn.model_selection import train_test_split\nfrom keras.models import *\nfrom keras.layers import *\nfrom keras.optimizers import *\nfrom tqdm import tqdm\nfrom tensorflow.keras.preprocessing.image import load_img, ImageDataGenerator, img_to_array, array_to_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Set some directories\ntrainHQ_zip_path = '/kaggle/input/carvana-image-masking-challenge/train_hq.zip'\nmasks_zip_path = '/kaggle/input/carvana-image-masking-challenge/train_masks.zip'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"scrolled":false},"cell_type":"code","source":"import zipfile\n#Extract train images.\nwith zipfile.ZipFile(trainHQ_zip_path,'r') as zip_ref:\n    zip_ref.extractall('/kaggle/working')\n#Extract train masks/labels.\nwith zipfile.ZipFile(masks_zip_path,'r') as zip_ref:\n    zip_ref.extractall('/kaggle/working')\ndata_size = len(os.listdir('/kaggle/working/train_hq'))\nprint('Number of train images: ', len(os.listdir('/kaggle/working/train_hq')))\nprint('Number of train masks: ', len(os.listdir('/kaggle/working/train_masks')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Display ids for images and masks.\ncar_ids = sorted(os.listdir('/kaggle/working/train_hq'))\nmask_ids = sorted(os.listdir('/kaggle/working/train_masks'))\n#Generate some random index.\nrnd_ind = list(np.random.choice(data_size,8))\nfor i in rnd_ind:\n    print(\"Car image id: '{}' -- Corressponding Mask id '{}'\".format(car_ids[i], mask_ids[i]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, each car and its mask have almost the same id."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Pick the 1553th car&mask ids from ids lists.\nn = 1553\ncar_id = car_ids[n]\nmask_id = mask_ids[n]\n#Load car&mask images using thier ids.\ncar = load_img('/kaggle/working/train_hq/' + car_id)\nmask = load_img('/kaggle/working/train_masks/' + mask_id)\nprint(\"Image Size: \", car.size)\nprint(\"Mask Size: \", mask.size)\n#Plot them.\nfig, ax = plt.subplots(1, 2, figsize=(20,20))\nfig.subplots_adjust(hspace=.1, wspace=.01)\nax[0].imshow(car)\nax[0].axis('off')\nax[0].title.set_text('Car Image')\nax[1].imshow(mask)\nax[1].axis('off')\nax[1].title.set_text('Car Mask')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Note how the images have a large size. Thus we can't load all the data to the RAM at once, we will use a data generator to load the data from the HDD to the RAM batch by batch as needed."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Randomly split car&mask ids list to training and validation lists.\n#X is car image ids list, y is mask image ids list.\nX_train_ids, X_val_ids, y_train_ids, y_val_ids= train_test_split(car_ids, mask_ids,\n                                                                 test_size=.2, train_size=.8,\n                                                                 random_state=42)\nX_train_size = len(X_train_ids)\nX_val_size = len(X_val_ids)\nprint('Training images size: ', X_train_size)\nprint('Validation images size: ', X_val_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Input size could be 128 or 256 or 512 or 1024.\ninput_size = [128, 128, 3]\ndef data_generator(images_path, masks_path, image_ids, mask_ids, batch_size, img_size=input_size):\n    '''\n    images_path/masks_path: Images/Masks folder directory.\n    images_ids/mask_ids: Ids for '.jpg' images/masks.\n    img_size: Generated imgs/masks size.\n    \n    returns: batch of randomly-selected car&mask images value-scaled (0 -> 1). \n    '''\n    data_size = len(image_ids)\n    while True:\n        #Choose random indice for later picking.\n        rnd_ind = np.random.choice(np.arange(data_size),batch_size)\n        imgs = []\n        masks = []\n        for i in rnd_ind:\n            #Pick a random id for car&mask images.\n            img_id, mask_id = image_ids[i], mask_ids[i]\n            #Load/resize images.\n            img = load_img(images_path + img_id, target_size=img_size) \n            mask = load_img(masks_path + mask_id, target_size=img_size[:-1], color_mode = 'grayscale')\n            #Add to the batch data.\n            imgs.append(img_to_array(img))\n            masks.append(img_to_array(mask).reshape(img_size[:-1] + [1]))\n        yield np.array(imgs, dtype=np.float16) / 255., np.array(masks, dtype=np.float16) / 255.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Try out the generator, generate data samples from the validation set.\ngen = data_generator('/kaggle/working/train_hq/', '/kaggle/working/train_masks/',\n                    X_val_ids, y_val_ids, batch_size=32)\n\nimgs, masks = next(gen)\nprint('Images batch shape: ', imgs.shape)\nprint('Masks batch shape: ', imgs.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Plot output samples of the generator.\nfig, ax = plt.subplots(2, 4, figsize=(15,7))\nfig.subplots_adjust(hspace=.1, wspace=.05)\ncar_samples, mask_samples = imgs[:4].astype(np.float32), masks[:4][:,:,:,0].astype(np.float32)\nfor i, (car, mask) in enumerate(zip(car_samples, mask_samples)):\n    ax[0, i].imshow(car)\n    ax[0, i].axis('off')\n    ax[0, i].title.set_text('Car Image')\n    \n    ax[1, i].imshow(mask, cmap='gray')\n    ax[1, i].axis('off')\n    ax[1, i].title.set_text('Car Mask')\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* You may be asking yourself, which size should I resize the images, the short answer is 'bigger size is better'.\n* But why?, test set should be masked by (1918x1280) masks, which mean that we should resize the mask with whatever size we're working with up to (1918x1280), simply resizing a (1024x1024) mask up to (1918x1280) gives more accurate pixel values than resizing (128x128) up to (1918x1280) due to padding and interpolation.\n* Though big size images come with long training time, so I used (128x128).\n* You can find more experiments in LB masters' discussion that I previously linked to."},{"metadata":{},"cell_type":"markdown","source":"* We need to implement a custom loss function and metric.\n* I learned from a discussion that using mixed dice and cross-entropy loss and dice loss gives a better result."},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def dice_coef(y_true, y_pred):\n    '''\n    Metric\n    '''\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    score = (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return score\n\n\ndef dice_loss(y_true, y_pred):\n    '''\n    Loss function\n    '''\n    loss = 1 - dice_coef(y_true, y_pred)\n    return loss\n\n\ndef bce_dice_loss(y_true, y_pred):\n    '''\n    Mixed crossentropy and dice loss.\n    '''\n    loss = keras.losses.binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_unet_128(input_shape=(128, 128, 3),\n                 num_classes=1):\n    inputs = Input(shape=input_shape)\n    # 128\n\n    down1 = Conv2D(64, (3, 3), padding='same')(inputs)\n    down1 = BatchNormalization()(down1)\n    down1 = Activation('relu')(down1)\n    down1 = Conv2D(64, (3, 3), padding='same')(down1)\n    down1 = BatchNormalization()(down1)\n    down1 = Activation('relu')(down1)\n    down1_pool = MaxPooling2D((2, 2), strides=(2, 2))(down1)\n    # 64\n\n    down2 = Conv2D(128, (3, 3), padding='same')(down1_pool)\n    down2 = BatchNormalization()(down2)\n    down2 = Activation('relu')(down2)\n    down2 = Conv2D(128, (3, 3), padding='same')(down2)\n    down2 = BatchNormalization()(down2)\n    down2 = Activation('relu')(down2)\n    down2_pool = MaxPooling2D((2, 2), strides=(2, 2))(down2)\n    # 32\n\n    down3 = Conv2D(256, (3, 3), padding='same')(down2_pool)\n    down3 = BatchNormalization()(down3)\n    down3 = Activation('relu')(down3)\n    down3 = Conv2D(256, (3, 3), padding='same')(down3)\n    down3 = BatchNormalization()(down3)\n    down3 = Activation('relu')(down3)\n    down3_pool = MaxPooling2D((2, 2), strides=(2, 2))(down3)\n    # 16\n\n    down4 = Conv2D(512, (3, 3), padding='same')(down3_pool)\n    down4 = BatchNormalization()(down4)\n    down4 = Activation('relu')(down4)\n    down4 = Conv2D(512, (3, 3), padding='same')(down4)\n    down4 = BatchNormalization()(down4)\n    down4 = Activation('relu')(down4)\n    down4_pool = MaxPooling2D((2, 2), strides=(2, 2))(down4)\n    # 8\n\n    center = Conv2D(1024, (3, 3), padding='same')(down4_pool)\n    center = BatchNormalization()(center)\n    center = Activation('relu')(center)\n    center = Conv2D(1024, (3, 3), padding='same')(center)\n    center = BatchNormalization()(center)\n    center = Activation('relu')(center)\n    # center\n\n    up4 = UpSampling2D((2, 2))(center)\n    up4 = concatenate([down4, up4], axis=3)\n    up4 = Conv2D(512, (3, 3), padding='same')(up4)\n    up4 = BatchNormalization()(up4)\n    up4 = Activation('relu')(up4)\n    up4 = Conv2D(512, (3, 3), padding='same')(up4)\n    up4 = BatchNormalization()(up4)\n    up4 = Activation('relu')(up4)\n    up4 = Conv2D(512, (3, 3), padding='same')(up4)\n    up4 = BatchNormalization()(up4)\n    up4 = Activation('relu')(up4)\n    # 16\n\n    up3 = UpSampling2D((2, 2))(up4)\n    up3 = concatenate([down3, up3], axis=3)\n    up3 = Conv2D(256, (3, 3), padding='same')(up3)\n    up3 = BatchNormalization()(up3)\n    up3 = Activation('relu')(up3)\n    up3 = Conv2D(256, (3, 3), padding='same')(up3)\n    up3 = BatchNormalization()(up3)\n    up3 = Activation('relu')(up3)\n    up3 = Conv2D(256, (3, 3), padding='same')(up3)\n    up3 = BatchNormalization()(up3)\n    up3 = Activation('relu')(up3)\n    # 32\n\n    up2 = UpSampling2D((2, 2))(up3)\n    up2 = concatenate([down2, up2], axis=3)\n    up2 = Conv2D(128, (3, 3), padding='same')(up2)\n    up2 = BatchNormalization()(up2)\n    up2 = Activation('relu')(up2)\n    up2 = Conv2D(128, (3, 3), padding='same')(up2)\n    up2 = BatchNormalization()(up2)\n    up2 = Activation('relu')(up2)\n    up2 = Conv2D(128, (3, 3), padding='same')(up2)\n    up2 = BatchNormalization()(up2)\n    up2 = Activation('relu')(up2)\n    # 64\n\n    up1 = UpSampling2D((2, 2))(up2)\n    up1 = concatenate([down1, up1], axis=3)\n    up1 = Conv2D(64, (3, 3), padding='same')(up1)\n    up1 = BatchNormalization()(up1)\n    up1 = Activation('relu')(up1)\n    up1 = Conv2D(64, (3, 3), padding='same')(up1)\n    up1 = BatchNormalization()(up1)\n    up1 = Activation('relu')(up1)\n    up1 = Conv2D(64, (3, 3), padding='same')(up1)\n    up1 = BatchNormalization()(up1)\n    up1 = Activation('relu')(up1)\n    # 128\n\n    classify = Conv2D(num_classes, (1, 1), activation='sigmoid')(up1)\n\n    model = Model(inputs=inputs, outputs=classify)\n\n    model.compile(optimizer=RMSprop(lr=0.0001), loss=bce_dice_loss, metrics=[dice_coef])\n\n    return model\n\nuNet = get_unet_128()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note: If you wanna try bigger/smaller  input size you should then deepen/shallow-en the architecture even more, take a look at [Patros repo](https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge/blob/master/model/u_net.py).\n"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Prepare callbacks\nLR_callback = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', patience=2, verbose=10, factor=.2, min_lr=.00001)\nEarlyStop_callback = keras.callbacks.EarlyStopping(monitor='val_loss',patience=10, restore_best_weights=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"#Perpare data generators.\nbatch_size = 32\ntrain_gen = data_generator('/kaggle/working/train_hq/', '/kaggle/working/train_masks/',\n                           X_train_ids, y_train_ids, batch_size=batch_size)\nval_gen = data_generator('/kaggle/working/train_hq/', '/kaggle/working/train_masks/',\n                           X_val_ids, y_val_ids, batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = uNet.fit_generator(train_gen, steps_per_epoch=int(X_train_size/batch_size),\n                             epochs=35, validation_data=val_gen,\n                             validation_steps=int(X_val_size/batch_size),\n                             callbacks=[LR_callback, EarlyStop_callback])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation\nfig, ax = plt.subplots(2,1, figsize=(15,7))\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\nlegend = ax[0].legend(loc='best', shadow=True)\n\nax[1].plot(history.history['dice_coef'], color='b', label=\"Training dice loss\")\nax[1].plot(history.history['val_dice_coef'], color='r',label=\"Validation dice loss\")\nlegend = ax[1].legend(loc='best', shadow=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"1. * Now let's try  to pridect masks for a batch of 32 images from the validation set."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Perdict some imgs.\npred_masks = uNet.predict(imgs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(32, 3, figsize=(20,150))\nfig.subplots_adjust(hspace=.1, wspace=.05)\nfor i in range(32):\n    ax[i, 0].imshow(imgs[i].astype(np.float32))\n    ax[i, 0].axis('off')\n    ax[i, 0].title.set_text('Car')\n    \n    ax[i, 1].imshow(masks[i,:,:,0].astype(np.float32), cmap='gray')\n    ax[i, 1].axis('off')\n    ax[i, 1].title.set_text('Real Mask')\n    \n    ax[i, 2].imshow(pred_masks[i,:,:,0], cmap='gray')\n    ax[i, 2].axis('off')\n    ax[i, 2].title.set_text('Predicted Mask')\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Output is not that bad, actully if it's not perfect, it should save much time for editors, as the model get most of the work done.\n* Here's the end, we can't test the model using testset unfortuntly the availabe size for on hdd disk is only just 5 GB and the testset is ~= 8 GB, Hope it was helpful.\n* Thank you for reading."},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.rmtree('/kaggle/working/train_hq')\nshutil.rmtree('/kaggle/working/train_masks')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}