{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nfrom skimage.io import imread\nfrom matplotlib import pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display(display_list):\n    plt.figure(figsize=(10, 10))\n\n    for i in range(len(display_list)):\n        plt.subplot(1, len(display_list), i+1)\n        plt.imshow(display_list[i])\n        plt.axis('off')\n    plt.show()\ndef masks_as_image(in_mask_list):\n    '''\n    Take the individual ship masks and create a single mask array for all ships\n    '''\n    all_masks = np.zeros(IMG_SIZE, dtype = np.uint8)\n    for mask in in_mask_list:\n        if isinstance(mask, str):\n            all_masks |= rle_decode(mask)\n    return all_masks\n\ndef rle_decode(mask_rle, shape=(768, 768)):\n    '''\n    mask_rle: run-length as string formated (start length)\n    shape: (height,width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0]*shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape).T  # Needed to align to RLE direction\n\n# https://github.com/ternaus/TernausNet/blob/master/Example.ipynb\ndef mask_overlay(image, mask):\n    \"\"\"\n    Helper function to visualize mask\n    \"\"\"\n    mask = mask.astype(np.uint8)\n    weighted_sum = cv2.addWeighted(mask, 0.75, image, 0.5, 0.)\n    img = image.copy()\n    ind = mask[:, :, 1] > 0    \n    img[ind] = weighted_sum[ind]    \n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH   = '../input/airbus-ship-detection/'\nTRAIN_PATH  = DATA_PATH+'train_v2/'\nTEST_PATH   = DATA_PATH+'test_v2/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The directories contain roughly 192,556  and 15606 images each."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"train = os.listdir(TRAIN_PATH)\ntest = os.listdir(TEST_PATH)\nprint(len(train), len(test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The first thing we can do is to open and display some of the images from whitin Jupyter notebook itself."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"images = []\nfor ImageId in train[43:46]:\n    image = imread(TRAIN_PATH+ImageId)\n    images += [image]\ndisplay(images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## We need to detection ship in this images. lets plotting ship count."},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(DATA_PATH+'train_ship_segmentations_v2.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.reset_index()\ndf['ship_count'] = df.groupby('ImageId')['ImageId'].transform('count')\ndf.loc[df['EncodedPixels'].isnull().values,'ship_count'] = 0  #see infocusp's comment\n\nsns.set_style(\"white\")\nsns.set_context(\"notebook\", font_scale=1.5, rc={\"lines.linewidth\": 2.5})\nsns.distplot(df['ship_count'],kde=False)\nplt.title('Ship Count Distribution in Train Set')\n\nprint(df['ship_count'].describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Look at class balance**"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.bar(\n    ['Ships', 'No Ships'], \n    [len(df[~df.EncodedPixels.isna()].ImageId.unique()),\n    len(df[df.EncodedPixels.isna()].ImageId.unique())]);\nplt.ylabel('Number of Images');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Counting pixels: verifying the class imbalance between ship and no-ship pixels"},{"metadata":{},"cell_type":"markdown","source":"The Challenge of detecting the ships in the images can be thought as a classification problem for pixels, where, for each image, we need to classify 768  ×  768 pixels in one of two classes: ship and no-ship. \n\nLets present the imbalance of the classes considering a pixel-level granularity, this is, we will check how many pixels in the dataset corresponds to ships and how many to other stuff (no-ships)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function transforms EncodedPixels into a list of pixels\n# Check our previous notebook for a detailed explanation:\n# https://www.kaggle.com/julian3833/2-understanding-and-plotting-rle-bounding-boxes\ndef rle_to_pixels(rle_code):\n    rle_code = [int(i) for i in rle_code.split()]\n    pixels = [(pixel_position % 768, pixel_position // 768) \n                 for start, length in list(zip(rle_code[0:-1:2], rle_code[1:-2:2])) \n                 for pixel_position in range(start, start + length)]\n    return pixels\n\ndef show_pixels_distribution(df):\n    \"\"\"\n    Prints the amount of ship and no-ship pixels in the df\n    \"\"\"\n    # Total images in the df\n    n_images = df['ImageId'].nunique() \n    \n    # Total pixels in the df\n    total_pixels = n_images * 768 * 768 \n\n    # Keep only rows with RLE boxes, transform them into list of pixels, sum the lengths of those lists\n    ship_pixels = df['EncodedPixels'].dropna().apply(rle_to_pixels).str.len().sum() \n\n    ratio = ship_pixels / total_pixels\n    print(f\"Ship: {round(ratio, 3)} ({ship_pixels})\")\n    print(f\"No ship: {round(1 - ratio, 3)} ({total_pixels - ship_pixels})\")\ndf = pd.read_csv(DATA_PATH+\"train_ship_segmentations_v2.csv\").append(pd.read_csv(DATA_PATH+\"sample_submission_v2.csv\"))\nshow_pixels_distribution(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see above, only 1‰ of the pixels are ships, while 99.9% of the pixels are no-ships.\n\nAnd, as you can see below, dropping all the images with no ships in them the class imbalance is reduced, but it's still very high: 5‰, this is, 0.5% of the pixels are ships while 99.5% are no-ships.\n\nAs we will analyse in detail on the following notebook of the series, this extreme class imbalance condition of the dataset will trigger actions in the construction of the public models (in particular, the stack of a ship/no-ship image classifier for the general problem with a ship/no-ship image segmentation for only the 22% of the images with ships)."},{"metadata":{"trusted":true},"cell_type":"code","source":"show_pixels_distribution(df.dropna())","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}