{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# 3. Basic exploratory analysis\n### Airbus Ship Detection Challenge - A quick overview for computer vision noobs\n\n&nbsp;\n\n\nHi, and welcome! This is the third kernel of the series `Airbus Ship Detection Challenge - A quick overview for computer vision noobs.` In this short kernel we will review the data very briefly. We will, first, count ship/no-ship images and plot the ships-per-image distribution and, second, present the strong imbalance in the total amount of ship/no-ship pixels.\n\n\n\nThe full series consist of the following notebooks:\n1. [Loading and visualizing the images](https://www.kaggle.com/julian3833/1-loading-and-visualizing-the-images)\n2. [Understanding and plotting rle bounding boxes](https://www.kaggle.com/julian3833/2-understanding-and-plotting-rle-bounding-boxes) \n3. *[Basic exploratory analysis](https://www.kaggle.com/julian3833/3-basic-exploratory-analysis)*\n4. [Exploring public models](https://www.kaggle.com/julian3833/4-exploring-models-shared-by-the-community)\n5. [1.0 submission: submitting the test file](https://www.kaggle.com/julian3833/5-1-0-submission-submitting-the-test-file)\n\nThis is an ongoing project, so expect more notebooks to be added to the series soon. Actually, we are currently working on the following ones:\n* Understanding and exploiting the data leak\n* A quick overview of image segmentation domain\n* Jumping into Pytorch\n* Understanding U-net\n* Proposing a simple improvement to U-net model"},{"metadata":{"_uuid":"7794e6c2e76eb6e738dd25ea99a0dc2b45ddab4e"},"cell_type":"markdown","source":"## 1. 10,000 foot view "},{"metadata":{"_uuid":"a6ad58daba72890fad68659c2a5b06f7abb25511"},"cell_type":"markdown","source":"The dataset consist of 3 csvs and 2 image sets: \n* The `train/` and the `test/` set of images, with 104,070 and 88,500 images each\n   - Refer to the [first kernel](https://www.kaggle.com/julian3833/1-loading-and-visualizing-images) of the series to display these images\n* The `sample_submission.csv` is a submit example, with the format of a solution. It has exactly 88,486 images to process (14 images from `test/` should be excluded for the submission, as stated on the Challenge's [Data](https://www.kaggle.com/c/airbus-ship-detection/data) tab)\n* The `train_ship_segmentations.csv` contains the run-length encoded bounding boxes for the ships in the `train/` directory, while the `test_ship_segmentations.csv` contains the bouding boxes for the `test/` directory. \n   - These `dfs` have two columns: `ImageId` and `EncodedPixels`\n   - An image with more than one ship will have n rows in these csvs, one for each ship\n  \n  \n <span style='color:blue'>Why is there a `test_ship_segmentations.csv` at all? Isn't that csv like... the solution? Well, it actually is. There was a data leakage in the dataset and the organizers decided to make the segmentations for the `test/` images public. We are currently working on a notebook explaining the situation and we [shared another one](https://www.kaggle.com/julian3833/5-1-0-submission-submitting-the-test-file) creating a 1.00 submission from this test set</span>"},{"metadata":{"trusted":true,"_uuid":"068bc40cd17220a931eb55328cdc1e8b999a58a7"},"cell_type":"code","source":"ls ../input/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc14b9d44653f86bf06907a74a2cdd700140624c"},"cell_type":"code","source":"import os\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nprint(f\"Images in train/: {len(os.listdir('../input/train/'))}\")\nprint(f\"Images in test/ :  {len(os.listdir('../input/test/'))}\")\n\nprint()\n\nn_submit_images = pd.read_csv(\"../input/sample_submission.csv\").shape[0]\nprint(f\"Images for submission: {n_submit_images}\")\n\npd.read_csv(f\"../input/train_ship_segmentations.csv\").head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d03bd02ec219873f76301de03ca237be437fa2b"},"cell_type":"markdown","source":"## 2. A first glance at the csvs: ship vs. no-ship and total ships distribution"},{"metadata":{"_uuid":"cd4aa939b12d7b1e30b586838ccb2a09656639a0"},"cell_type":"markdown","source":"We will define two simple functions `load_df()` and `show_df()`. The first one loads a csv to pandas and creates the fields `HasShip` and `TotalShips` from `EncodedPixels`.  The second one displays the amount of images with and without ships and the distribution of total ships per image.\n\nIf you don't understand the `EncodedPixels`, you can refer to the [previous kernel](https://www.kaggle.com/julian3833/2-understanding-and-plotting-rle-bounding-boxes) of this series, where we explain the `run-length encoding` in detail. \n\nAs you can see, only the 22% of the images have at least one ship present, and more than 60% of those have only one ship."},{"metadata":{"trusted":true,"_uuid":"73fe96f76089c60a9f311ca13dc9835c6bb7344d"},"cell_type":"code","source":"def load_df(file=\"train\"):\n    \"\"\"\n    Loads a csv, creates the fields `HasShip` and `TotalShips` dropping `EncodedPixels` and setting `ImageId` as index.\n    \"\"\"\n    df = pd.read_csv(f\"../input/{file}_ship_segmentations.csv\")\n    df['HasShip'] = df['EncodedPixels'].notnull()\n    df = df.groupby(\"ImageId\").agg({'HasShip': ['first', 'sum']}) # counts amount of ships per image, sets ImageId to index\n    df.columns = ['HasShip', 'TotalShips']\n    return df\n\ndef show_df(df):\n    \"\"\"\n    Prints and displays the ship/no-ship ratio and the ship count distribution of df\n    \"\"\"\n    total = len(df)\n    ship = df['HasShip'].sum()\n    no_ship = total - ship\n    total_ships = int(df['TotalShips'].sum())\n        \n    print(f\"Images: {total} \\nShips:  {total_ships}\")\n    print(f\"Images with ships:    {round(ship/total,2)} ({ship})\")\n    print(f\"Images with no ships: {round(no_ship/total,2)} ({no_ship})\")\n    \n    _, axes = plt.subplots(nrows=1, ncols=2, figsize=(30, 8), gridspec_kw = {'width_ratios':[1, 3]})\n    \n    # Plot ship/no-ship with a bar plot\n    ship_ratio = df['HasShip'].value_counts() / total\n    ship_ratio = ship_ratio.rename(index={True: 'Ship', False: 'No Ship'})\n    ship_ratio.plot.bar(ax=axes[0], color=['red', 'lime'], rot=0, title=\"Ship/No-ship distribution\");\n    \n    # Plot TotalShips distribution with a bar plot\n    total_ships_distribution = df.loc[df['HasShip'], 'TotalShips'].value_counts().sort_index() / ship\n    total_ships_distribution.plot(kind='bar', ax=axes[1], rot=0, title=\"Total ships distribution\");","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7650a6075909c7d7aecc8d1b410a677f9d42d92d"},"cell_type":"code","source":"df_train = load_df(\"train\")\ndf_test = load_df(\"test\")\nshow_df(df_train.append(df_test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"810f6e45698823308b28d08031b0cc42d9d57f2d"},"cell_type":"markdown","source":"The class imbalance of images get worse for the `test set` and, as we will see in the next title, it gets even worse when we don't consider the `images` but the `pixels`."},{"metadata":{"trusted":true,"_uuid":"b0ab30d56a97cad4be48f7d46f53b0647e640f86"},"cell_type":"code","source":"show_df(df_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d8110778c9f38b9cf1b0343d9e81e0512f51c9f"},"cell_type":"markdown","source":"## 3. Counting pixels: verifying the class imbalance between ship and no-ship pixels"},{"metadata":{"_uuid":"c2870f382f41c32a73583d0fb650bb24501a8fad"},"cell_type":"markdown","source":"The Challenge of detecting the ships in the images can be thought as a `classification problem` for pixels, where, for each image, we need to classify 768 $\\times$ 768 pixels in one of two classes: `ship` and `no-ship`. This is actually the common approach to the `image segmentation` problem as we will discuss in further notebooks.\n\nIn this notebook we will just present the imbalance of the classes considering a `pixel-level` granularity, this is, we will check *how many pixels in the dataset corresponds to ships and how many to other stuff (no-ships)*\n\nFew notes before diving into the code:\n* The `total_pixels` is $ 768 \\times 768 \\times \\text{n_imgs} $\n* The total amount of `ship_pixels` is encoded in the `EncodedPixels`: it's actually the sum of the all the pair positions of those strings. \n   - Since we have defined a `rle_to_pixels` function [before](https://www.kaggle.com/julian3833/2-understanding-and-plotting-rle-bounding-boxes), we will just use it and count the amount of pixels after that transformation\n* The total amount of `no_ship_pixels` is `total_pixels - ship_pixels`\n"},{"metadata":{"trusted":true,"_uuid":"61b026c916aeb574ea657493f9ba243761240f7f"},"cell_type":"code","source":"# This function transforms EncodedPixels into a list of pixels\n# Check our previous notebook for a detailed explanation:\n# https://www.kaggle.com/julian3833/2-understanding-and-plotting-rle-bounding-boxes\ndef rle_to_pixels(rle_code):\n    rle_code = [int(i) for i in rle_code.split()]\n    pixels = [(pixel_position % 768, pixel_position // 768) \n                 for start, length in list(zip(rle_code[0:-1:2], rle_code[1:-2:2])) \n                 for pixel_position in range(start, start + length)]\n    return pixels\n\ndef show_pixels_distribution(df):\n    \"\"\"\n    Prints the amount of ship and no-ship pixels in the df\n    \"\"\"\n    # Total images in the df\n    n_images = df['ImageId'].nunique() \n    \n    # Total pixels in the df\n    total_pixels = n_images * 768 * 768 \n\n    # Keep only rows with RLE boxes, transform them into list of pixels, sum the lengths of those lists\n    ship_pixels = df['EncodedPixels'].dropna().apply(rle_to_pixels).str.len().sum() \n\n    ratio = ship_pixels / total_pixels\n    print(f\"Ship: {round(ratio, 3)} ({ship_pixels})\")\n    print(f\"No ship: {round(1 - ratio, 3)} ({total_pixels - ship_pixels})\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7bd2c8c5776462158f78034192ce609f95cc9d2"},"cell_type":"code","source":"df = pd.read_csv(\"../input/train_ship_segmentations.csv\").append(pd.read_csv(\"../input/test_ship_segmentations.csv\"))\nshow_pixels_distribution(df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36a8564b7949355917e9170f0c9df82c1004261e"},"cell_type":"markdown","source":"As you can see above,  only 1‰ of the pixels are `ships`, while 99.9% of the pixels are `no-ships`. \n\nAnd, as you can see below, dropping all the images with no ships in them the class imbalance is reduced, but it's still very high: 5‰, this is, 0.5% of the pixels are `ships` while 99.5% are `no-ships`.\n\nAs we will analyse in detail on the [following notebook](https://www.kaggle.com/julian3833/4-exploring-public-models) of the series, this extreme class imbalance condition of the dataset will trigger actions in the construction of the public models (in particular, the stack of a `ship/no-ship image classifier` for the general problem with a `ship/no-ship image segmentation` for only the 22% of the images with ships)."},{"metadata":{"trusted":true,"_uuid":"df49a6f8c345a7a4308bfe629ded43edbb417384"},"cell_type":"code","source":"show_pixels_distribution(df.dropna())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"471db951794aab051e04a8c40471b0d188c35e43"},"cell_type":"markdown","source":"### References\n* [Airbus EDA](https://www.kaggle.com/ezietsman/airbus-eda) - a more advanced and very nice exploratory data analysis kernel\n* [Fine tuning resnet34 on ship detection](https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection) - the kernel from which we red about the pixel class imbalance as a strong problem for the first time. We will refer to this *awesome kernel* various times on this notebook series.\n* [Class imbalance problem](http://www.chioka.in/class-imbalance-problem/) - a blog post to recap about class imbalance\n\n\n### What's next?\nYou can check the [next kernel](https://www.kaggle.com/julian3833/4-exploring-public-models) of the series, where we explore the hottests available public models and present the main ideas and approaches behind them.\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}