{"cells":[{"metadata":{"_uuid":"446db166cce01b2217bcde4a255a8bce4afa442e"},"cell_type":"markdown","source":"# Overview: \nHere, we will prepeare the data needed for Keras_MaskRCNN which implements Retinanet as the object detector, the original MASKRCNN uses fasterRCNN, when we used the original MASKRCNN, the model identified really good number of false postive. So we used Keras_MaskRCNN which utilizes the Retinanet as it's detector.  For the scope of this kernel, we will prepare the required data to train a beseline model. Due to the libraries requried for the model to train, it was not convinient to write a kernel through which we could have used kaggle resources to train the model. However after following the below step, you can simply start training the model in your desired machine.  For your convenience, the prepread data is already in the data section, also a trained model snaptshot has been added, so that you could visualize the result right in this Kernel.\n\n# Result: \nThis baseline model, if you manage to train until 20 epochs will give you fairly nice public score around 67 - 68 % . \n\n# Beyond: \nWe actually used it to ensemble the result from other models like MaskRcnn, which prediction was quite fine, however with lots of false positive. This model helped us to identify the images in the test set where the ship weren't present. \n\n# What We Learnt: \nThis was our first competition ever in the Kaggle, we had never touched the instance segmentation problem before, we started off with Matterport's version of MaskRCNN, after post-processing the result and ensembling 3 different models, we got around 68.7% score in public leaderboard.  After many weeks of struggle with MaskRCNN, we could hardly improve the score, then we realized that our result contained many false positives, thus the idea sprung while doing some study, and we tried to use RetinaNet instead of FasterRCNN for the detector, but we were already out of time, so we simply chose [Keras_MaskRCNN](https://github.com/fizyr/keras-maskrcnn/) which already had that.  After all we have lot to learn. Thank you Kaggle. \n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Import Necessary Packages"},{"metadata":{"trusted":true,"_uuid":"dc4ac43c960c185a15c3e4fb657ba0d6f63451df"},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom PIL import Image\nimport os\nprint(os.listdir(\"../input/airbus-ship-detection/\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Loading the dataset and preparation "},{"metadata":{"trusted":true,"_uuid":"d68cdafb6d391ca9d3f10ae96d2c161832432a4e"},"cell_type":"code","source":"def extract_bboxes(mask):\n    \"\"\"Compute bounding boxes from masks.\n    mask: [height, width, num_instances]. Mask pixels are either 1 or 0.\n\n    Returns: bbox array [num_instances, (y1, x1, y2, x2)].\n    \"\"\"\n    boxes = np.zeros([mask.shape[-1], 4], dtype=np.int32)\n    for i in range(mask.shape[-1]):\n        m = mask[:, :, i]\n        # Bounding box.\n        horizontal_indicies = np.where(np.any(m, axis=0))[0]\n        vertical_indicies = np.where(np.any(m, axis=1))[0]\n        if horizontal_indicies.shape[0]:\n            x1, x2 = horizontal_indicies[[0, -1]]\n            y1, y2 = vertical_indicies[[0, -1]]\n            # x2 and y2 should not be part of the box. Increment by 1.\n            x2 += 1\n            y2 += 1\n        else:\n            # No mask for this instance. Might happen due to\n            # resizing or cropping. Set bbox to zeros\n            x1, x2, y1, y2 = 0, 0, 0, 0\n        boxes[i] = np.array([y1, x1, y2, x2])\n    return boxes.astype(np.int32)\n\n\ndef load_mask(data):\n\n    \"\"\"Generate instance masks for an image.\n   Returns:\n    masks: A bool array of shape [height, width, instance count] with\n        one mask per instance.\n    class_ids: a 1D array of class IDs of the instance masks.\n    \"\"\"\n    # If not a ship dataset image, delegate to parent class.\n    \n    height =  width =  768\n    rle = [data[\"EncodedPixels\"]]\n    mask = np.zeros((height,width,len(rle)))\n    for p,m in enumerate(rle):\n        all_masks = np.zeros((height,width))\n        all_masks += rle_decode(m)\n        mask[:, :, p] = all_masks\n\n    return mask.astype(np.bool)\n\n\ndef rleToMask(rleString,height,width):\n    rows,cols = height,width\n    rleNumbers = [int(numstring) for numstring in rleString.split(' ')]\n    rlePairs = np.array(rleNumbers).reshape(-1,2)\n    img = np.zeros(rows*cols,dtype=np.uint8)\n    for index,length in rlePairs:\n        index -= 1\n        img[index:index+length] = 255\n    img = img.reshape(cols,rows)\n    img = img.T\n    return img\n\n\ndef rle_decode(mask_rle, shape=(768, 768)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T  # Needed to align to RLE direction","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6750eaf0d45f9abba6aac41a39dfacbc91347fba"},"cell_type":"markdown","source":"# Make the data required later for training purposes"},{"metadata":{"trusted":true,"_uuid":"69ed8e5f808c3a5407c8d6e5e655affe8e1197bc"},"cell_type":"code","source":"    from tqdm import tqdm_notebook\n    #Load Images\n    PATH_TO_SAVE = \"DIR/TO/SAVE/MASKS/\"\n    DATA_DIR = \"PATH/FOR/CSV/\"\n    datadir = os.listdir(\"../input\")\n    masks = pd.read_csv('../input/train_ship_segmentations_v2.csv')\n    print(masks.shape[0], 'masks found')\n    print(masks['ImageId'].value_counts().shape[0])\n    \n    images_with_ship = masks[~masks.EncodedPixels.isnull()].unique().values()\n    print('There are ' +str(len(images_with_ship)) + ' image files  with masks found')\n    data = pd.DataFrame(columns=[\"ImageId\", \"x1\", \"y1\", \"x2\", \"y2\", \"classname\", \"mask\"])\n    masks.fillna('', inplace=True)\n    grouped = masks.groupby([\"ImageId\"])\n    \n    for index, row in tqdm_notebook(masks.iterrows()):\n        if(row[\"EncodedPixels\"]==''):\n            rowtoappend = {\"ImageId\":row[\"ImageId\"], \"x1\":'', \"y1\":'', \"x2\":'', \"y2\":'', \"classname\":'', 'mask':''}\n            data.loc[count] = rowtoappend\n        else:\n            number_of_masks = masks[masks[\"ImageId\"]==row[\"ImageId\"]]\n            rle = [row[\"EncodedPixels\"]] \n            imgdata = rleToMask(row[\"EncodedPixels\"], 768, 768)\n            im = Image.fromarray(imgdata)\n            i = 0\n            '''While running in local mode this line would save the instance mask associated with a particulr image inside the desired directory\n            It has been commented here because the masks dataset has been loaded as a seperate dataset in this kernel. \n            '''\n            while os.path.exists(PATH_TO_SAVE + row[\"ImageId\"] + \"%s.png\" % i):\n                i += 1\n            im.save(PATH_TO_SAVE + row[\"ImageId\"] + \"%s.png\" % i)\n            m = load_mask(row)\n            box = extract_bboxes(m)\n            box = box[0]\n            \n            rowtoappend = {\"ImageId\":row[\"ImageId\"], \"x1\":box[1], \"y1\":box[0], \"x2\":box[3], \n                           \"y2\":box[2], \"classname\":\"ship\", 'mask':PATH_TO_SAVE + row[\"ImageId\"]+\"%s.png\" % i}\n            data.loc[count] = rowtoappend\n            count+=1\n    data.to_csv(DATA_DIR + \"annotation.csv\", index=None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"96278e32a8aa14d0bed48c881a7dff58d672cc8f"},"cell_type":"code","source":"# Reading the data\ndata = pd.read_csv('../input/maskdata/annotation_working.csv')\ndata.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac56688023628be2b68b785f7b54690cd0ffaef6"},"cell_type":"markdown","source":"# The Results:\nNow we have annotation_working.csv file inside the maskdata folder, we have class label csv under the same folder, and the masks folder consists all the maks images that we will need for training the model. "},{"metadata":{"_uuid":"29ca8aa708739494df6e32f259f7f7c0bac716bc"},"cell_type":"markdown","source":"# Actual Training\nLoading all the necessary libraries and putting the needed codes would be cumbersome for this kernel. For that simplicity, we shifted the codes into github, through the listed approach in the repo, you can train the model. \nThe major step, since you already have the masks images, annotation csv and class_ids would be just to run: \n`./keras_maskrcnn/bin/train.py --weights=PATH/TO/COCO/WEIGHT/ --epochs=50 --steps=1000 --config=config/config.ini csv data/annotation.csv data/class_ids.csv` \nor more precisely, follow the steps in https://github.com/vaghawan/airbus-ship-detection-using-keras-retinanet-maskrcnn#training\n\nAfter you got your model trained, the `detect.py` performs the main actions.  Here we have done the modifcation of the model outputs and transformed them into the output that we would get from Matterport's version of MaskRCNN, because we were already using the inference of Matterport's version of MaskRCNN. "},{"metadata":{"_uuid":"0100f86833e40400ddce9f3afe69e1b2101d4173"},"cell_type":"markdown","source":"# Post-Processing Of KerasMaskRCNN Outputs:\n- The output shape of masks is always (100, ImageHeight, ImageWidth, 1), and we transformed the masks into the shape of (ImageHeight, ImageWidth, NUM_OF_DETECTED_INSTANCE) \n- We applied Non-Max Supression to the predicted bounding boxes to remove lots of overlaps. \n- After that, we removed the overlap in the masks by assigning the overlap pixels to the highest scored mask. \n- We thresholded the predicted output. \n\nBelow is the code that were used for post-processing. "},{"metadata":{"trusted":true,"_uuid":"f83cd529e6f26924c733f7fda2c22c70c783ae02"},"cell_type":"code","source":"import keras\nimport sys\nsys.path.append(\"../input/kerasmaskrcnn/keras-maskrcnn-master/keras-maskrcnn-master/\")\n# import keras_retinanet\nfrom keras_maskrcnn import models\nfrom keras_maskrcnn.utils.visualization import draw_mask\nfrom keras_retinanet.utils.visualization import draw_box, draw_caption, draw_annotations\nfrom keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image\nfrom keras_retinanet.utils.colors import label_color\n\n# import miscellaneous modules\nimport matplotlib.pyplot as plt\nimport cv2\nimport os\nimport numpy as np\nimport time\n\n# set tf backend to allow memory to grow, instead of claiming everything\nimport tensorflow as tf\nimport skimage\nfrom skimage.morphology import binary_opening, disk, label,binary_closing,binary_dilation\n\nfrom skimage.measure import find_contours\nfrom skimage.measure import label as label_lib\nfrom scipy.ndimage.morphology import binary_fill_holes\nfrom skimage.morphology import dilation, erosion\nimport scipy\nimport skimage.color\nimport skimage.io\nimport pandas as pd\nimport csv, datetime\n\ndef get_session():\n    config = tf.ConfigProto()\n    config.gpu_options.allow_growth = True\n    return tf.Session(config=config)\n\n# use this environment flag to change which GPU to use\n#os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"1\"\n\n# set the modified tf session as backend in keras\nkeras.backend.tensorflow_backend.set_session(get_session())\n\n\n\ndef draw_test(image, boxes, masks, scores, labels=None, color=None, binarize_threshold=0.5):\n    \"\"\" Draws a mask in a given box.\n\n    Args\n        image              : Three dimensional image to draw on.\n        box                : Vector of at least 4 values (x1, y1, x2, y2) representing a box in the image.\n        mask               : A 2D float mask which will be reshaped to the size of the box, binarized and drawn over the image.\n        color              : Color to draw the mask with. If the box has 5 values, the last value is assumed to be the label and used to construct a default color.\n        binarize_threshold : Threshold used for binarizing the mask.\n        scores             : A 1D Numpy array of scores\n    \"\"\"\n    resulting_masks = []\n    kept_scores = []\n    kept_labels = []\n    kept_boxes = []\n    \n    if labels is None:\n        labels = [None for _ in range(boxes.shape[0])]\n    \n\n    for box, mask, label, score in zip(boxes, masks, labels, scores):\n        \n        # resize to fit the box\n        if label != -1.:\n            kept_boxes.append(box)\n            box = box.astype(int)\n            mask = cv2.resize(mask, (box[2] - box[0], box[3] - box[1]))\n\n            # binarize the mask\n            mask = (mask > binarize_threshold).astype(np.uint8)\n            # print(mask, mask.shape)\n            # draw the mask in the image\n            mask_image = np.zeros((image.shape[0], image.shape[1]), np.uint8)\n            mask_image[box[1]:box[3], box[0]:box[2]] = mask\n            mask = mask_image\n            resulting_masks.append(mask)\n            kept_scores.append(score)\n            kept_labels.append(label)\n            #resulting_masks = np.append(resulting_masks, mask, axis=0)\n    if len(resulting_masks) <1 :\n        resulting_masks = np.zeros((image.shape[0], image.shape[1], 0), np.uint8)\n        kept_scores = np.array([])\n        kept_labels = np.array([])\n        kept_boxes = np.array([])\n    else:   \n        resulting_masks = np.asarray(resulting_masks)\n        resulting_masks = resulting_masks.reshape(resulting_masks.shape[0],768, 768, 1 )\n        kept_scores = np.asarray(kept_scores)\n        kept_labels = np.asarray(kept_labels)\n        kept_boxes = np.asarray(kept_boxes)\n        \n    print(resulting_masks.shape)\n    if resulting_masks.shape[-1] == None:\n        print(\"here\")\n        resulting_masks = resulting_masks.reshape(0, 768, 768, 1)\n    \n    return resulting_masks, kept_scores, kept_labels, kept_boxes\n\n\n\n\ndef postprocess_masks(result, image, min_nuc_size=0):\n\n    \"\"\"Clean overlaps between bounding boxes, fill small holes, smooth boundaries\"\"\"\n    print(np.where(result[\"masks\"][0]==1))\n    height, width = image.shape[:2]\n\n    # If there is no mask prediction do the following\n    print(\"inside post-process\", result['masks'].shape)\n    if result['masks'].shape[0] == 0:\n        print(\"we were supposed to be here\")\n        result['masks'] = np.zeros([height, width, 1])\n        result['masks'][0, 0, 0] = 1\n        result['scores'] = np.ones(1)\n        result['class_ids'] = np.zeros(1)\n\n    keep_ind = np.where(np.sum(result['masks'], axis=(0, 1)) > min_nuc_size)[0]\n    \n    if len(keep_ind) < result['masks'].shape[-1]:\n        # print('Deleting',len(result['masks'])-len(keep_ind), ' empty result['masks']')\n        result['masks'] = result['masks'][..., keep_ind]\n        result['scores'] = result['scores'][keep_ind]\n        result['rois'] = result['rois'][keep_ind]\n        result['class_ids'] = result['class_ids'][keep_ind]\n\n    sort_ind = np.argsort(result['scores'])[::-1]\n    \n    result['masks'] = result['masks'][..., sort_ind]\n    overlap = np.zeros([height, width])\n\n    # Removes overlaps from masks with lower score\n    for mm in range(result['masks'].shape[-1]):\n        # Fill holes inside the mask\n        mask = binary_fill_holes(result['masks'][..., mm]).astype(np.uint8)\n        # Smoothen edges using dilation and erosion\n        mask = erosion(dilation(mask))\n        # Delete overlaps\n        overlap += mask\n        \n        mask[overlap > 1] = 0\n        \n        out_label = label_lib(mask)\n        \n        # Remove all the pieces if there are more than one pieces\n        if out_label.max() > 1:\n            mask[()] = 0\n            print('removed something here')\n        result['masks'][..., mm] = mask\n    \n    keep_ind = np.where(np.sum(result['masks'], axis=(0, 1)) > min_nuc_size)[0]\n    \n    if len(keep_ind) < result['masks'].shape[-1]:\n        result['masks'] = result['masks'][..., keep_ind]\n        result['scores'] = result['scores'][keep_ind]\n        result['rois'] = result['rois'][keep_ind]\n        result['class_ids'] = result['class_ids'][keep_ind]\n\n    return result\n\n\n\ndef rle_encode(img, min_max_threshold=1e-3, max_mean_threshold=None):\n    '''\n    img: numpy array, 1 - mask, 0 - background\n    Returns run length as string formated\n    '''\n    if np.max(img) < min_max_threshold:\n        return '' ## no need to encode if it's all zeros\n    if max_mean_threshold and np.mean(img) > max_mean_threshold:\n        return '' ## ignore overfilled mask\n    pixels = img.T.flatten()\n    pixels = np.concatenate([[0], pixels, [0]])\n    runs = np.where(pixels[1:] != pixels[:-1])[0] + 1\n    runs[1::2] -= runs[::2]\n    return ' '.join(str(x) for x in runs)\n\n\n\ndef compute_iou(box, boxes, box_area, boxes_area):\n    \"\"\"Calculates IoU of the given box with the array of the given boxes.\n    box: 1D vector [y1, x1, y2, x2]\n    boxes: [boxes_count, (y1, x1, y2, x2)]\n    box_area: float. the area of 'box'\n    boxes_area: array of length boxes_count.\n\n    Note: the areas are passed in rather than calculated here for\n    efficiency. Calculate once in the caller to avoid duplicate work.\n    \"\"\"\n    # Calculate intersection areas\n    # y1 = np.maximum(box[0], boxes[:, 0])\n    # y2 = np.minimum(box[2], boxes[:, 2])\n    # x1 = np.maximum(box[1], boxes[:, 1])\n    # x2 = np.minimum(box[3], boxes[:, 3])\n\n    #New for us: \n    y1 = np.maximum(box[1], boxes[:, 1])\n    y2 = np.minimum(box[3], boxes[:, 3])\n    x1 = np.maximum(box[0], boxes[:, 0])\n    x2 = np.minimum(box[2], boxes[:, 2])\n    intersection = np.maximum(x2 - x1, 0) * np.maximum(y2 - y1, 0)\n    union = box_area + boxes_area[:] - intersection[:]\n    iou = intersection / union\n    return iou\n\n\ndef non_max_suppression(boxes, scores, threshold):\n    \"\"\"Performs non-maximum suppression and returns indices of kept boxes.\n    boxes: [N, (y1, x1, y2, x2)]. Notice that (y2, x2) lays outside the box.\n    scores: 1-D array of box scores.\n    threshold: Float. IoU threshold to use for filtering.\n    \"\"\"\n    assert boxes.shape[0] > 0\n    if boxes.dtype.kind != \"f\":\n        boxes = boxes.astype(np.float32)\n\n    # Compute box areas\n    # y1 = boxes[:, 0]\n    # x1 = boxes[:, 1]\n    # y2 = boxes[:, 2]\n    # x2 = boxes[:, 3]\n    \n    #New for us:\n    y1 = boxes[:, 1]\n    x1 = boxes[:, 0]\n    y2 = boxes[:, 3]\n    x2 = boxes[:, 2]\n    area = (y2 - y1) * (x2 - x1)\n\n    # Get indicies of boxes sorted by scores (highest first)\n    ixs = scores.argsort()[::-1]\n\n    pick = []\n    while len(ixs) > 0:\n        # Pick top box and add its index to the list\n        i = ixs[0]\n        pick.append(i)\n        # Compute IoU of the picked box with the rest\n        iou = compute_iou(boxes[i], boxes[ixs[1:]], area[i], area[ixs[1:]])\n        # Identify boxes with IoU over the threshold. This\n        # returns indices into ixs[1:], so add 1 to get\n        # indices into ixs.\n        remove_ixs = np.where(iou > threshold)[0] + 1\n        # Remove indices of the picked and overlapped boxes.\n        ixs = np.delete(ixs, remove_ixs)\n        ixs = np.delete(ixs, 0)\n    return np.array(pick, dtype=np.int32)\n\ndef color_splash(image, mask):\n    \"\"\"Apply color splash effect.\n    image: RGB image [height, width, 3]\n    mask: instance segmentation mask [height, width, instance count]\n\n    Returns result image.\n    \"\"\"\n    # Make a grayscale copy of the image. The grayscale copy still\n    # has 3 RGB channels, though.\n    red = image*[1,1,0]\n    gray = skimage.color.gray2rgb(skimage.color.rgb2gray(image)) * 255\n    # Copy color pixels from the original color image where mask is set\n\n    if mask.shape[-1] > 0:\n        # We're treating all instances as one, so collapse the mask into one layer\n        mask = (np.sum(mask, -1, keepdims=True) >= 1)\n        mask = mask.astype(int)\n        #print(rle_encode(mask))\n        splash = np.where(mask, red, image).astype(np.uint8)\n    else:\n        splash = gray.astype(np.uint8)\n    return splash \n\n\ndef rle_encoding(x):\n    dots = np.where(x.T.flatten() == 1)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if (b > prev+1):\n            run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n\n    return ' '.join(str(x) for x in run_lengths)\n\n\ndef prob_to_rles(masks, height, width):\n\n    if masks.sum() < 1:\n        masks = np.zeros([height, width, 1])\n        # print('no masks')\n        masks[0, 0, 0] = 1\n\n    if np.any(masks.sum(axis=-1) > 1):\n        print('Overlap', masks.shape)\n\n    for mm in range(masks.shape[-1]):\n        yield rle_encoding(masks[..., mm].astype(np.int32)), np.sum(masks[..., mm].astype(np.int32)==1)\n        \ndef test_on_single_image(model, imagepath, labels_names:dict, SCORE_THRES= 0.2, IOU_THRES = 0.5):\n\t\n    image = read_image_bgr(imagepath)\n\n    # copy to draw on\n    draw = image.copy()\n    draw = cv2.cvtColor(draw, cv2.COLOR_BGR2RGB)\n\n    # preprocess image for network\n    image = preprocess_image(image)\n    image, scale = resize_image(image)\n\n    # process image\n    start = time.time()\n    outputs = model.predict_on_batch(np.expand_dims(image, axis=0))\n    print(\"processing time: \", time.time() - start)\n    boxes  = outputs[-4][0]\n    scores = outputs[-3][0]\n    labels = outputs[-2][0]\n    masks  = outputs[-1][0]\n\n    # correct for image scale\n    boxes /= scale\n\n    # visualize detections\n    #print(boxes.shape, scores.shape, masks.shape)\n    masks, scores, labels, boxes = draw_test(draw, boxes, masks, scores, labels, color=label_color(0))\n    if boxes.size !=0:\n        keep_ind = non_max_suppression(boxes, scores, IOU_THRES)\n        masks = masks[keep_ind, :, :]\n        scores = scores[keep_ind]\n        labels = labels[keep_ind]\n        rois = boxes[keep_ind]\n        result = {\"masks\":masks, \"scores\":scores, \"class_ids\":labels, \"rois\":rois}\n        idxtokeep = np.where(result['scores']>SCORE_THRES)[0]\n        result['masks'] = masks[idxtokeep,:, :]\n        result['scores'] = scores[idxtokeep]\n        result['class_ids'] = labels[idxtokeep]\n        result['rois'] = rois[idxtokeep]\n\n        image_arr = skimage.io.imread(imagepath)\n\n        masks_resulted = []\n        if result['masks'].size !=0:\n            firstmask = result['masks'][0]\n\n            result['masks'] = result['masks'][1:]\n            for box, score, label, mask in zip(result['rois'], result['scores'], result['class_ids'], result['masks']):\n                color = label_color(label)\n                b = box.astype(int)\n                #draw_box(draw, b, color=color)\n\n                firstmask = np.append(firstmask, mask, axis=2)\n                mask = mask[:,:,label]\n                #draw_mask_overlap(draw, b, mask)\n\n                caption = \"{} {:.3f}\".format(labels_to_names[label], score)\n\n            \n            \n\n            print(\"concatinated mask\", firstmask.shape)\n            result['masks'] = firstmask\n            print(result['scores'])\n            splash = color_splash(image_arr, result['masks'])\n            plt.figure(figsize=(15,15))\n            skimage.io.imshow(splash)\n            plt.show()\n            plt.close()\n\n        else:\n            print(\"the result were removed due to the thresholding.\")\n\n    else:\n        print(\"no instance found.\")\n\n\n\ndef generate_result(model, imagedir, labels_names:dict, csv_path:str, output_image_path:str=None, SCORE_THRES= 0.2, IOU_THRES = 0.5):\n    \n    already_tested = list(pd.read_csv(csv_path)[\"ImageId\"].unique())\n    allimagesindir = list(os.listdir(imagedir))\n    yettotest = list(set(allimagesindir) - set(already_tested)) if (len(allimagesindir)>len(already_tested)) else list(set(already_tested) - set(allimagesindir))\n    print(yettotest) \n    print(\"number of images yet to  test is:\", len(yettotest), len(already_tested))\n\n    count = 0\n    for image_path in yettotest:\n        img_path = imagedir+image_path\n        # load image\n        image = read_image_bgr(img_path)\n        # copy to draw on\n        draw = image.copy()\n        draw = cv2.cvtColor(draw, cv2.COLOR_BGR2RGB)\n        # preprocess image for network\n        image = preprocess_image(image)\n        image, scale = resize_image(image)\n        # process image\n        start = time.time()\n        outputs = model.predict_on_batch(np.expand_dims(image, axis=0))\n\n        print(\"processing time: \", time.time() - start)\n        \n\n        boxes  = outputs[-4][0]\n        scores = outputs[-3][0]\n        labels = outputs[-2][0]\n        masks  = outputs[-1][0]\n\n        boxes /= scale\n\n        #This is for the preparation of submission csv file.\n        imagelist=[]\n        encodelist=[]\n        overlappedrm = []\n        scorelist = []\n        lengthlist = []\n        \n        masks, scores, labels, boxes = draw_test(draw, boxes, masks, scores, labels, color=label_color(0))\n        if boxes.size !=0:\n            keep_ind = non_max_suppression(boxes, scores, IOU_THRES)\n            masks = masks[keep_ind, :, :]\n            scores = scores[keep_ind]\n            labels = labels[keep_ind]\n            rois = boxes[keep_ind]\n            result = {\"masks\":masks, \"scores\":scores, \"class_ids\":labels, \"rois\":rois}\n            idxtokeep = np.where(result['scores']>SCORE_THRES)[0]\n            result['masks'] = masks[idxtokeep,:, :]\n            result['scores'] = scores[idxtokeep]\n            result['class_ids'] = labels[idxtokeep]\n            result['rois'] = rois[idxtokeep]\n\n            image_arr = skimage.io.imread(img_path)\n\n            masks_resulted = []\n\n            if result['masks'].size != 0:\n                firstmask = result['masks'][0]\n\n                result['masks'] = result['masks'][1:]\n                for box, score, label, mask in zip(result['rois'], result['scores'], result['class_ids'], result['masks']):\n                    color = label_color(label)\n                    b = box.astype(int)\n                    firstmask = np.append(firstmask, mask, axis=2)\n                    mask = mask[:,:,label]\n\n                \n                result['masks'] = firstmask\n                print(\"kept scores.. \" , result[\"scores\"])\n                if output_image_path:\n                    splash = color_splash(image_arr, result['masks'])\n                    file_name = image_path+\"splash_{:%Y%m%dT%H%M%S}.png\".format(datetime.datetime.now())\n                    skimage.io.imsave(output_image_path+file_name, splash)\n                    \n                \n\n            else:\n                print(\"the result were removed due to the score thresholding.\")\n\n            height, width = image_arr.shape[:2]\n\n            if result[\"scores\"].size == 0:\n                imagelist.append(image_path)\n                encodelist.append('')\n                print(\"the mask is blank\")\n                scorelist.append(0.0)\n                lengthlist.append(0)\n\n            else:\n                masks = result[\"masks\"].astype(int)\n                \n                encode = list(prob_to_rles(masks, height, width))\n                \n                \n                scores = list(result['scores'])\n                \n                if encode !=None:\n                    for en, score in zip(encode, scores):\n                        imagelist.append(image_path)\n                        encodelist.append(en[0])\n                        scorelist.append(score)\n                        lengthlist.append(en[1])\n                        #overlappedrm.append(rmoverlapped)\n                else:\n                    imagelist.append(image_path)\n                    encodelist.append('')\n                    scorelist.append(0.0)\n                    lengthlist.append(0)\n\n        else:\n            imagelist.append(image_path)\n            encodelist.append('')\n            print(\"the mask is blank\")\n            scorelist.append(0.0)\n            lengthlist.append(0)\n\n        with open(csv_path, 'a') as outcsv:\n\n            fieldnames = ['ImageId', 'EncodedPixels', 'Score', 'Length']\n            writer = csv.DictWriter(outcsv, fieldnames=fieldnames)\n            writer.writerows([{\"ImageId\":img, \"EncodedPixels\":enc, \"Score\":scr, \"Length\":lengt} for img, enc, scr, lengt in zip(imagelist, encodelist, scorelist, lengthlist)])\n\n\n\nif __name__=='__main__':\n    #adjust this to point to your downloaded/trained model\n    model_path = '../input/snapshots/resnet50_csv_84.h5'\n    #load retinanet model\n    model = models.load_model(model_path, backbone_name='resnet50')\n    #print(model.summary())\n    SCORE_THRES = 0.2\n    IOU_THRES = 0.5\n    #load label to names mapping for visualization purposes\n    labels_to_names = {0: 'ship'}\n\n    img_path = '../input/airbus-ship-detection/test_v2/0a89c4e4b.jpg'\n    #For test in single image\n    test_on_single_image(model, img_path, labels_to_names)\n    #To generate the overall Results: \n    #generate_result(model, \"/var/www/mask-rcnn/data/all/test_v2/\", labels_to_names, \"/var/www/mask-rcnn/data/all/ensemble170-32.csv\", output_image_path=\"/var/www/airbus-competition-using-keras-maskrcnn/examples/splash/\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"922952d303a9b8f5687a0471e26cfc241e6aefdf"},"cell_type":"markdown","source":"# References:\nLots of references of codes and implemntation were taken from the various sources, which are not least but listed below:\n\n[https://github.com/fizyr/keras-maskrcnn/ ] - Original Keras Mask RCNN. \n\n[https://github.com/matterport/Mask_RCNN] - Many of the utility functions were actually the modified version available in the utility.py of matterport's Mask RCNN.\n\n[https://github.com/mirzaevinom/data_science_bowl_2018] -  Some of the utility functions were actually modified/used from mirzaevinom implementation.\n\nKaggle Airbus Ship Detection competition, Kernels. \n\n**All of these implementation are in our repo:** https://github.com/vaghawan/airbus-ship-detection-using-keras-retinanet-maskrcnn"},{"metadata":{"trusted":true,"_uuid":"bbc2bcb63e79d1eaa119b3dbb028bb2c8d88dfee"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}