{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](https://www.vaya-antarctica.com/wp-content/uploads/Orion-Header-1920x1020.jpg)","metadata":{"_uuid":"984d7378d2deeb66f729d631e15bf35ea2087ba7"}},{"cell_type":"markdown","source":"I have already posted one [kernel](https://www.kaggle.com/rackovic1994/feature-engineering) on Airbus Ship Detection Challenge with brunt on Image Processing. In this kernel I will go through **end-to-end U-Net model** for Object Detection trying to find and localize ship on satelite images.\n\nThings I will cover here are\n1.         Problem setting of this competition\n2.        Image generator for CNN\n3.        Metrics and loss function\n4.        Building CNN and making predictions\n\nSo, let's start with **Problem Setting**. The goal of competition is to detect ship at the satelite image. We have original images (around 100 k of them) and corresponding masks for each occuring ship (in .csv format, so we need to transform it into an image). So, make NN model, feed it with images and masks and get predictions. Piece of cake, right?\n\nLet's go step by step. (And yes, if you don't have GPU it's better not to try on this competition)","metadata":{"_uuid":"6ea1080a7026d0311fb140f82bc72a1bff63721f"}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom skimage import io\nfrom keras.models import Model, load_model\nfrom keras.layers import Input\nfrom keras.layers.core import Lambda\nfrom keras.layers.convolutional import Conv2D, Conv2DTranspose\nfrom keras.layers.pooling import MaxPooling2D\nfrom keras.layers.merge import concatenate\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint\nfrom keras import backend as K\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nmarks = pd.read_csv('../input/train_ship_segmentations_v2.csv') # Markers for ships\nimages = os.listdir('../input/train_v2') # Images for training\nos.chdir(\"../input/train_v2\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-27T17:46:33.798676Z","iopub.execute_input":"2022-05-27T17:46:33.799335Z","iopub.status.idle":"2022-05-27T17:46:34.751444Z","shell.execute_reply.started":"2022-05-27T17:46:33.799255Z","shell.execute_reply":"2022-05-27T17:46:34.750411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def mask_part(pic):\n    '''\n    Function that encodes mask for single ship from .csv entry into numpy matrix\n    '''\n    back = np.zeros(768**2)\n    starts = pic.split()[0::2]\n    lens = pic.split()[1::2]\n    for i in range(len(lens)):\n        back[(int(starts[i])-1):(int(starts[i])-1+int(lens[i]))] = 1\n    return np.reshape(back, (768, 768, 1))\n\ndef is_empty(key):\n    '''\n    Function that checks if there is a ship in image\n    '''\n    df = marks[marks['ImageId'] == key].iloc[:,1]\n    if len(df) == 1 and type(df.iloc[0]) != str and np.isnan(df.iloc[0]):\n        return True\n    else:\n        return False\n    \ndef masks_all(key):\n    '''\n    Merges together all the ship markers corresponding to a single image\n    '''\n    df = marks[marks['ImageId'] == key].iloc[:,1]\n    masks= np.zeros((768,768,1))\n    if is_empty(key):\n        return masks\n    else:\n        for i in range(len(df)):\n            masks += mask_part(df.iloc[i])\n        return np.transpose(masks, (1,0,2))","metadata":{"_uuid":"d9d89705dd864b815220a8aa28b1827227f5693d","execution":{"iopub.status.busy":"2022-05-27T17:46:42.392709Z","iopub.execute_input":"2022-05-27T17:46:42.393146Z","iopub.status.idle":"2022-05-27T17:46:42.401146Z","shell.execute_reply.started":"2022-05-27T17:46:42.393083Z","shell.execute_reply":"2022-05-27T17:46:42.400094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw(lista):\n    l = int(len(lista)/2)\n    plt.figure(figsize = (20,5))\n    for i in range(2*l):\n        plt.subplot(2,2*l,2*i+1)\n        plt.imshow(plt.imread(lista[i]))\n        plt.axis('off')\n        plt.subplot(2,2*l,2*i+2)\n        plt.imshow(masks_all(lista[i])[:,:,0])\n        plt.axis('off')\n        #plt.suptitle('Images and corresponding masks', fontsize = 22)\n    plt.show()\nprint('Randomly chosen images with corresponding masks')\ndraw(np.random.choice([i for i in images[:200] if not is_empty(i)], size = 8))\ndraw(np.random.choice([i for i in images[200:400] if not is_empty(i)], size = 8))\ndraw(np.random.choice([i for i in images[400:600] if not is_empty(i)], size = 8))","metadata":{"_uuid":"049b54bc5dd5d6f2390fe3bd6265817db00f0365","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-27T17:46:47.265028Z","iopub.execute_input":"2022-05-27T17:46:47.265361Z","iopub.status.idle":"2022-05-27T17:47:04.333785Z","shell.execute_reply.started":"2022-05-27T17:46:47.265311Z","shell.execute_reply":"2022-05-27T17:47:04.332738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you look closely on these plots, you'll notise that there are a lot of ships which are really hard to find; so don't be surprised if model struggles detect them. But, on the bright side, there are no overllaping of masks, which is a commong problem in some similar tasks; this is one burden less.","metadata":{"_uuid":"569d35b70efab079d13454ce4571b959f148a684"}},{"cell_type":"markdown","source":"## Image Generator\nPopular tactics in computer vision problems for fighting lack of data is **Data Augmentation**. It consists of performing different transformations on existing images in order to get 'new' images that are in it's core the same, but still differ enough that machine considers them different.\n\nIn order to feed our Net with continous stream of images, we need image generator. Generators in Python are simply speaking *functions that have varying number of outputs*.  Keras is equiped with built-in *Image Data Generator*, but for this competition I want to make mine own. It's not that hard when you go through it.","metadata":{"_uuid":"b705ac3b934e26ab368af4352fd0c0396d14a652","trusted":true}},{"cell_type":"code","source":"def transform(X, Y):\n    '''\n    Function for augmenting images. \n    It takes original image and corresponding mask and performs the\n    same flipping and rotation transforamtions on both in order to \n    perserve the overlapping of ships and their masks\n    '''\n# add noise:\n    x = np.copy(X)\n    y = np.copy(Y)\n    x[:,:,0] = x[:,:,0] + np.random.normal(loc=0.0, scale=0.01, size=(768,768))\n    x[:,:,1] = x[:,:,1] + np.random.normal(loc=0.0, scale=0.01, size=(768,768))\n    x[:,:,2] = x[:,:,2] + np.random.normal(loc=0.0, scale=0.01, size=(768,768))\n    # Adding Gaussian noise on each rgb channel; this way we will NEVER get two completely same images.\n    # Note that this transformation is not performed on Y \n    x[np.where(x<0)] = 0\n    x[np.where(x>1)] = 1\n# axes swap:\n    if np.random.rand()<0.5: # 0.5 chances for this transformation to occur (same for two below)\n        x = np.swapaxes(x, 0,1)\n        y = np.swapaxes(y, 0,1)\n# vertical flip:\n    if np.random.rand()<0.5:\n        x = np.flip(x, 0)\n        y = np.flip(y, 0)\n# horizontal flip:\n    if np.random.rand()<0.5:\n        x = np.flip(x, 1)\n        y = np.flip(y, 1)\n    return x, y  ","metadata":{"_uuid":"755d3109260322be52e149859312b5ce147d98cf","execution":{"iopub.status.busy":"2022-05-27T17:47:06.201185Z","iopub.execute_input":"2022-05-27T17:47:06.201539Z","iopub.status.idle":"2022-05-27T17:47:06.208972Z","shell.execute_reply.started":"2022-05-27T17:47:06.201476Z","shell.execute_reply":"2022-05-27T17:47:06.208277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_transformed(file):\n    '''\n    Plots specified Original image and five augmentations\n    (five images created transforming original with function 'transform()')\n    '''\n    X, Y = plt.imread(file), masks_all(file)\n    plt.figure(figsize = (19,8))\n    plt.subplot(253, title ='Original Image')\n    X, Y = plt.imread(file)/255, masks_all(file)\n    plt.imshow(X)\n    plt.axis('off')\n    plt.subplot(256, title ='Transformed Image')\n    plt.imshow(transform(X,Y)[0])\n    plt.axis('off')\n    plt.subplot(257, title ='Transformed Image')\n    plt.imshow(transform(X,Y)[0])\n    plt.axis('off')    \n    plt.subplot(258, title ='Transformed Image')\n    plt.imshow(transform(X,Y)[0])\n    plt.axis('off')\n    plt.subplot(259, title ='Transformed Image')\n    plt.imshow(transform(X,Y)[0])\n    plt.axis('off')    \n    plt.subplot(2,5,10, title ='Transformed Image')\n    plt.imshow(transform(X,Y)[0])\n    plt.axis('off')\n    plt.suptitle(file,x=0.3, y=0.7, verticalalignment ='top', fontsize = 22)\n    plt.show()","metadata":{"_uuid":"47f96bd7246b3ef37bd69ceb9c6b9d4f2a9ab1bb","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-27T17:47:10.097082Z","iopub.execute_input":"2022-05-27T17:47:10.09747Z","iopub.status.idle":"2022-05-27T17:47:10.1059Z","shell.execute_reply.started":"2022-05-27T17:47:10.097402Z","shell.execute_reply":"2022-05-27T17:47:10.104951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_transformed('0270d7317.jpg')","metadata":{"_kg_hide-input":true,"_uuid":"ed0d0b3a1e3e2ced8906dcbfa75bb5837be09803","execution":{"iopub.status.busy":"2022-05-27T17:47:14.378725Z","iopub.execute_input":"2022-05-27T17:47:14.379463Z","iopub.status.idle":"2022-05-27T17:47:16.064976Z","shell.execute_reply.started":"2022-05-27T17:47:14.3794Z","shell.execute_reply":"2022-05-27T17:47:16.063899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At the figure abowe you can see image *0270d7317.jpg* and five 'new' images created transforming it.","metadata":{"_uuid":"71d3491b4eec3d2541f8039d0f8ad8ad42b0fec9"}},{"cell_type":"code","source":"def make_batch(files, batch_size):\n    '''\n    Creates batches of images and masks in order to feed them to NN\n    '''\n    X = np.zeros((batch_size, 768, 768, 3))\n    Y = np.zeros((batch_size, 768, 768, 1)) # I add 1 here to get 4D batch\n    for i in range(batch_size):\n        ship = np.random.choice(files)\n        X[i] = (io.imread(ship))/255.0 # Original images are in 0-255 range, I want it in 0-1\n        Y[i]= masks_all(ship)\n    return X, Y","metadata":{"_uuid":"2987d753974ef67c7d6d43cd597528c0ddca7adc","execution":{"iopub.status.busy":"2022-05-27T17:47:20.007118Z","iopub.execute_input":"2022-05-27T17:47:20.007479Z","iopub.status.idle":"2022-05-27T17:47:20.014414Z","shell.execute_reply.started":"2022-05-27T17:47:20.007425Z","shell.execute_reply":"2022-05-27T17:47:20.013039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, it is time to create Generator function combining two functions abowe. Images in batch are chosen at random so we can have one image occuring multiple times even in a single batch, but on each image in batch 'transform()' function is invoked fighting this problem.","metadata":{"_uuid":"5922a33c18324bdcfd28f93320d2afa5fffec3a9"}},{"cell_type":"code","source":"def Generator(files, batch_size):\n    '''\n    Generates batches of images and corresponding masks\n    '''\n    while True:\n        X, Y = make_batch(files, batch_size)\n        for i in range(batch_size):\n            X[i], Y[i] = transform(X[i], Y[i])\n        yield X, Y","metadata":{"_uuid":"e62f6697f196d5bc0ee22af3e6df9d5882655ba6","execution":{"iopub.status.busy":"2022-05-27T17:47:26.316305Z","iopub.execute_input":"2022-05-27T17:47:26.317017Z","iopub.status.idle":"2022-05-27T17:47:26.322191Z","shell.execute_reply.started":"2022-05-27T17:47:26.316594Z","shell.execute_reply":"2022-05-27T17:47:26.321328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metrics and Loss Function","metadata":{"_uuid":"8217c19aa0cdf254af5a5d9c7ef02ea4bb59d578"}},{"cell_type":"markdown","source":"Keras package has a lot of built-in metrics, but default metrics are not always good choice. In this problem we need to make our own function for measuring quality of model. Metric I'll be using is called **Intersection over Union**. It is nicely presented at the figure below, where *B1* would present ground truth mask of a ship and *B2* is predicted mask. When this mattching is peerfect, metric value is 1 and the lower predicting precison is, the lower is this value (down to zero).\n![](https://lovesnowbest.site/2018/02/27/Intro-to-Object-Detection/iou.png)\nBelow are implementations of *IoU*, which calculates this ratio for masks, and *back_IoU*, that does the same thing only now consiering background insted of masks. For metrics in model you can use both of these or only the first one (I wouldn't advise only the second one) .\n\nBut, when training NN, the **loss function** is the one that really matters. Simple way to create loss function is to take negative value of IoU metric. In this way we obtain function that decreases when our prediction is improving and increases otherwise.","metadata":{"_uuid":"f0c9ff27796d7acec3da0510eeef5e4362b6b539"}},{"cell_type":"code","source":"# Intersection over Union for Objects\ndef IoU(y_true, y_pred, tresh=1e-10):\n    Intersection = K.sum(y_true * y_pred, axis=[1,2,3])\n    Union = K.sum(y_true, axis=[1,2,3]) + K.sum(y_pred, axis=[1,2,3]) - Intersection\n    return K.mean( (Intersection + tresh) / (Union + tresh), axis=0)\n# Intersection over Union for Background\ndef back_IoU(y_true, y_pred):\n    return IoU(1-y_true, 1-y_pred)\n# Loss function\ndef IoU_loss(in_gt, in_pred):\n    #return 2 - back_IoU(in_gt, in_pred) - IoU(in_gt, in_pred)\n    return 1 - IoU(in_gt, in_pred)","metadata":{"_uuid":"f0db98b6d7c3b5bddd1e0ba538fedc283f5ace77","execution":{"iopub.status.busy":"2022-05-27T17:47:30.291255Z","iopub.execute_input":"2022-05-27T17:47:30.291642Z","iopub.status.idle":"2022-05-27T17:47:30.298225Z","shell.execute_reply.started":"2022-05-27T17:47:30.29159Z","shell.execute_reply":"2022-05-27T17:47:30.29686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## U-Net Architecture\nU-Net is a Convolutional Neural Network developed primarily for segmentation and object detection on biomedical images. It is suitable for this problem too cause we need model that takes image as input and also outputs an image.","metadata":{"_uuid":"136de4aa8f89dcb46e13aca5fcfc2d4f64a5c8dc","trusted":true}},{"cell_type":"markdown","source":"![](https://i.imgur.com/ZXj1nzK.png)\nOur architecture is basicaly the same as the one presented at this shema, borrowed from [here](https://imgur.com/gallery/7FHkQ). Network consists of two 'parts', first beeing common convolutional model with convolutions and pooling layers, decreasing image width and height but increasing number of channels. What makes it different is second 'part' consisting of *upsamplig*, where we have practically the inverse model, increasing width and height but decreasing channels number. This is acheived using transposed convolution and concatenation.","metadata":{"_uuid":"bad164385456584d2914cc007dc908f4d9f04377"}},{"cell_type":"markdown","source":"### Building model\nModel here is fairly simple in order to execute it easily, but you can create more layers and play with parameters, only looking carefully on the input and output dimensions. As you can see from the model summary both have the same dimensions (except for the number of channels). It is easy to preserve dimensions, you only need to use *padding='same'* and concatenate layers that are supposed to have the same dimensions.","metadata":{"_uuid":"79dcfadac05d5ae6533e4717948a9a59a5322927"}},{"cell_type":"code","source":"inputs = Input((768, 768, 3))\n\nc1 = Conv2D(8, (3, 3), activation='relu', padding='same') (inputs)\nc1 = Conv2D(8, (3, 3), activation='relu', padding='same') (c1)\np1 = MaxPooling2D((2, 2)) (c1)\n\nc2 = Conv2D(16, (3, 3), activation='relu', padding='same') (p1)\nc2 = Conv2D(16, (3, 3), activation='relu', padding='same') (c2)\np2 = MaxPooling2D((2, 2)) (c2)\n\nc3 = Conv2D(32, (3, 3), activation='relu', padding='same') (p2)\nc3 = Conv2D(32, (3, 3), activation='relu', padding='same') (c3)\np3 = MaxPooling2D((2, 2)) (c3)\n\nc4 = Conv2D(64, (3, 3), activation='relu', padding='same') (p3)\nc4 = Conv2D(64, (3, 3), activation='relu', padding='same') (c4)\n\nu5 = Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same') (c4)\nu5 = concatenate([u5, c3])\nc5 = Conv2D(32, (3, 3), activation='relu', padding='same') (u5)\nc5 = Conv2D(32, (3, 3), activation='relu', padding='same') (c5)\n\nu6 = Conv2DTranspose(16, (2, 2), strides=(2, 2), padding='same') (c5)\nu6 = concatenate([u6, c2])\nc6 = Conv2D(16, (3, 3), activation='relu', padding='same') (u6)\nc6 = Conv2D(16, (3, 3), activation='relu', padding='same') (c6)\n\nu7 = Conv2DTranspose(8, (2, 2), strides=(2, 2), padding='same') (c6)\nu7 = concatenate([u7, c1], axis=3)\nc7 = Conv2D(8, (3, 3), activation='relu', padding='same') (u7)\nc7 = Conv2D(8, (3, 3), activation='relu', padding='same') (c7)\n\noutputs = Conv2D(1, (1, 1), activation='sigmoid') (c7)\n\nmodel = Model(inputs=[inputs], outputs=[outputs])\nmodel.compile(optimizer='adam', loss= IoU_loss, metrics=[IoU, back_IoU])\nmodel.summary()","metadata":{"_uuid":"cf82ffd5aef03961b69f9b618ee08dd2bc7f4a08","execution":{"iopub.status.busy":"2022-05-27T17:47:42.19911Z","iopub.execute_input":"2022-05-27T17:47:42.199489Z","iopub.status.idle":"2022-05-27T17:47:42.726551Z","shell.execute_reply.started":"2022-05-27T17:47:42.199402Z","shell.execute_reply":"2022-05-27T17:47:42.725733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Setting batch_size = 200 and steps_per_epoch = 500 gives 200*500 = 100.000 images to train on.\n That is around the same size of available training set, so you shouldn't go under this, but\n considering our generator function, probably a lot of images won't be taken in consideration, so\n I encourage you to increase both, batch size and steps, freely.  \n \n Here, I will put absurdly small parameters just in order to check if everithing is working fine.","metadata":{"_uuid":"7e7f7d1e0a7a416831016e2151dc5da5773bbd6d"}},{"cell_type":"code","source":"#results = model.fit_generator(Generator(images, batch_size = 200), steps_per_epoch = 500, epochs = 30)\nresults = model.fit_generator(Generator(images, batch_size = 20), steps_per_epoch = 50, epochs = 3)","metadata":{"_uuid":"1311872f0a6012f93a015963b8bc80a3dc94910f","execution":{"iopub.status.busy":"2022-05-27T17:47:47.46989Z","iopub.execute_input":"2022-05-27T17:47:47.470232Z","iopub.status.idle":"2022-05-27T19:40:58.069217Z","shell.execute_reply.started":"2022-05-27T17:47:47.47019Z","shell.execute_reply":"2022-05-27T19:40:58.067534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### That's all folks","metadata":{"_uuid":"b5eb679dbfcc6b00007ce925192471e1133f9e60"}},{"cell_type":"code","source":"","metadata":{"_uuid":"469e8b1573b725907672550fd58fad8176673817","trusted":true},"execution_count":null,"outputs":[]}]}