{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#  🚢 Ship Detection: Image Visualizations and EDA\n\n<img src= \"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcR-Yxy_PxDspM7V39A-jp2dBv29geaV4OiDxg&usqp=CAU\" alt =\"Ships\" style='width: 400px;'>\n\nThe goal of this challange is to detect and mask ships from satellite images. There are both trainig and test data available with masks on the ships encoded with a run-lenght encoding style, as we will se later. The main difficult is due to the fact that the dataset is very big and unbalanced in a sense of presence of ships in the images. Furthermore the images are from satellities and the ships rapresent only a small section over the total if we consider the pixels. We will follow a summary as follow:","metadata":{}},{"cell_type":"markdown","source":"<a id=\"introduction\"></a>\n## Segmentation in Image Processing\n\nIn image processing and computer vision image segmentation is a very important and useful task. It consists in partitioning the image into multiple segments, or sets of pixels that belong to a patrticular class. In particular the task is to assing a particular label to every pixel in the image based on some shared characteristics, the results produces a mask, a set of segments that cover the entire or part of the image. The applications of this kind of machine learning task are huge for example in the medical imaging of face recogniction.\n\nThere are two main segmentation approches:\n* **Semantic Segmentation**: all pixels belonging to a particular class are considered together as one mask.\n* **Instance Segmentation**: the pixels are divided into different objects, even if belong to the same class.\n\nIn our case the problem is a Instance Segmentation one, in particular we have to match every ship in the images and find the pixels belonging to that particular class. \n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Table of contents","metadata":{}},{"cell_type":"markdown","source":"Intro\n[Segmentation in Image Processing](#introduction)\n\n1.[Loading libraries and functions](#section-one)\n - [Libraries](#subsection-one-1)\n - [Functions](#subsection-two-1)\n \n2.[Exploratory Data Analysis (EDA)](#section-two)\n - [Statistics about data](#subsection-one-2)\n - [Images Visualizations](#subsection-two-2)\n - [Data Sampling](#subsection-three-2) \n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n## Loading libraries and functions\n\nIn this section we will load the libraries and the function that will be used in the notebook.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-one-1\"></a>\n## Libraries\nImporting all the libraries used in the notebook ","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing\nimport matplotlib.pyplot as plt\n\n\nimport os #to load the data\nimport random\nimport PIL #manage images \n\nfrom skimage.io import imread, imshow, concatenate_images\nfrom skimage import io, transform\nfrom skimage.measure import label, regionprops","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-29T08:11:58.336538Z","iopub.execute_input":"2021-06-29T08:11:58.337143Z","iopub.status.idle":"2021-06-29T08:11:59.767329Z","shell.execute_reply.started":"2021-06-29T08:11:58.337031Z","shell.execute_reply":"2021-06-29T08:11:59.766379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-one-2\"></a>\n## Functions","metadata":{}},{"cell_type":"markdown","source":"In this section we will report the function we used for different purpose, this will help us in the computation and in the readness of the notebook.\n\n","metadata":{}},{"cell_type":"code","source":"#apply a particluar mask over the image \ndef apply_mask(image, mask):\n    for x, y in mask:\n        image[x, y, [0, 1]] = 255\n    return image","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:11:59.768837Z","iopub.execute_input":"2021-06-29T08:11:59.769145Z","iopub.status.idle":"2021-06-29T08:11:59.775521Z","shell.execute_reply.started":"2021-06-29T08:11:59.769118Z","shell.execute_reply":"2021-06-29T08:11:59.774697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def rle_to_pixels(rle_code):\n    '''\n    Transforms a RLE code string into a list of pixels of a (768, 768) canvas\n    '''\n    # Divide the rle in a list of pairs of ints rapresenring the (start,lenght)\n    rle_code = [int(i) for i in rle_code.split()] \n    \n    pixels = [\n        #Find the 2d coordinate for the canva using the mod function (%) and the integer division function(//)\n        (pixel_position % 768, pixel_position // 768) \n        # I select the start pixel and the lenght of the line\n                 for start, length in list(zip(rle_code[0:-1:2], rle_code[1::2])) \n        # I screen all the pixel positions rapresenting (start,end)\n                 for pixel_position in range(start, start + length)] \n    return pixels","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:11:59.777639Z","iopub.execute_input":"2021-06-29T08:11:59.777891Z","iopub.status.idle":"2021-06-29T08:11:59.78912Z","shell.execute_reply.started":"2021-06-29T08:11:59.777866Z","shell.execute_reply":"2021-06-29T08:11:59.788206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-two\"></a>\n# Exploratory Data Analysis","metadata":{}},{"cell_type":"markdown","source":"### Loading the Datasets","metadata":{}},{"cell_type":"markdown","source":"Are available two different datasets, one will be used for the training section. It is composed by 192556 different images in a .jpg format and a size of 768x768. The other dataset will be used for the test section and it is composed by 15606 images with the same characteristics. ","metadata":{}},{"cell_type":"code","source":"train = os.listdir(\"../input/airbus-ship-detection/train_v2\")\ntest = os.listdir(\"../input/airbus-ship-detection/test_v2\")\nsub= pd.read_csv(\"../input/airbus-ship-detection/sample_submission_v2.csv\", index_col=0).dropna()\n\n# Set paths\ndata_root = '../input/airbus-ship-detection/'\npath_train = os.path.join(data_root,'train_v2')\npath_test = os.path.join(data_root,'test_v2')\n\nprint(f\"Train files: {len(train)}. ---> {train[:3]}\")\nprint(f\"Test files :  {len(test)}. ---> {test[:3]}\")\n\n#PIL.Image.open(\"../input/airbus-ship-detection/train_v2/000c34352.jpg\")\nprint(\"The dimension of the images is: \"+str(PIL.Image.open(\"../input/airbus-ship-detection/train_v2/000c34352.jpg\").size))","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:11:59.790654Z","iopub.execute_input":"2021-06-29T08:11:59.791029Z","iopub.status.idle":"2021-06-29T08:12:03.372266Z","shell.execute_reply.started":"2021-06-29T08:11:59.790992Z","shell.execute_reply":"2021-06-29T08:12:03.371226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot some images from the train set \nw = 9\nh = 3\n\n# we resize the format of the images with a 200x200 shape\n# this function uses the open, resize and array functions to manage data visualizations\nload_img = lambda filename: np.array(PIL.Image.open(f\"../input/airbus-ship-detection/train_v2/{filename}\").resize((200, 200)))\n\n_, axes_list = plt.subplots(h, w, figsize=(2*w, 2*h)) # define a grid of (w, h)\n\nfor axes in axes_list:\n    for ax in axes:\n        ax.axis('off')\n        img = np.random.choice(train) # take a random train filename\n        ax.imshow(load_img(img)) # load and show\n        ax.set_title(img)\n        ","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:03.374738Z","iopub.execute_input":"2021-06-29T08:12:03.375171Z","iopub.status.idle":"2021-06-29T08:12:07.336752Z","shell.execute_reply.started":"2021-06-29T08:12:03.375143Z","shell.execute_reply":"2021-06-29T08:12:07.335472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Run-length encoding \n\nThe encoded string looks like this: start, length, start, length, ... , where each pair of (start, length) draws a line of length pixeles starting from position start. The start position, in turn, is not a (x, y) coordinate but an index of the 1-d array resulting of flattening the 2-d image into a rows-after-row 1-d sequence of pixels. Knowing the shape of the images we can just unfold this 1-d representating into a 2-dimensions mask using // and %.\n\nWe load this .cvs file and write some useful functions to better use the econding.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/airbus-ship-detection/train_ship_segmentations_v2.csv\", index_col=0).dropna()\ndisplay(df.head())\n\nprint('Example of a encoding run-lenght format: \\n')\ndf['EncodedPixels']['000155de5.jpg']","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:07.338302Z","iopub.execute_input":"2021-06-29T08:12:07.338679Z","iopub.status.idle":"2021-06-29T08:12:08.735624Z","shell.execute_reply.started":"2021-06-29T08:12:07.338642Z","shell.execute_reply":"2021-06-29T08:12:08.734602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-one-\"></a>\n\n## Statistics about data","metadata":{}},{"cell_type":"code","source":"# Read CSV as dataframe\ndf_count = pd.read_csv(os.path.join('../input/airbus-ship-detection/train_ship_segmentations_v2.csv'))\nprint('Total number of images (original): %d' % df_count['ImageId'].value_counts().shape[0])\n\n# Create a dataframe with unique images id as indexes and number of ships and image sizes as new columns\ndf_count = df_count[~df_count['ImageId'].isin(['6384c3e78.jpg'])] # remove corrupted file \nunique_img_ids = df_count.groupby('ImageId').size().reset_index(name='counts')\nprint('Total number of images (after removing corrupted images): %d' % df_count['ImageId'].value_counts().shape[0])","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:08.737211Z","iopub.execute_input":"2021-06-29T08:12:08.737616Z","iopub.status.idle":"2021-06-29T08:12:09.997128Z","shell.execute_reply.started":"2021-06-29T08:12:08.737574Z","shell.execute_reply":"2021-06-29T08:12:09.99581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count number of ships per image\ndf_wships = df_count.dropna()\ndf_wships = df_wships.groupby('ImageId').size().reset_index(name='counts')\ndf_woships = df_count[df_count['EncodedPixels'].isna()]\n\nprint('Number of images with ships :     %d \\nNumber of images without ships : %d\\n  \\nProportion: %0.1f\\n ' \\\n      % (df_wships.shape[0], df_woships.shape[0], df_wships.shape[0] / df_woships.shape[0]))\n\n\nprint('Ration with ships:     ' +str(round((df_wships.shape[0]/len(train)),2)))\nprint('Ration without ships:  ' +str(round((df_woships.shape[0]/len(train)),2)))\n\n#make plots\n\nplt.figure(figsize=(15, 6))\n\nplt.subplot(1,2,1)\nplt.bar(['With ships','Without ships'], [len(df_wships),len(df_woships)], color = ['lightblue','pink'])\nplt.ylabel('Number of images')\nplt.title('Unbalanced Trainig Data')\nplt.grid()\n\nplt.subplot(1,2,2)\nplt.bar(['With ships','Without ships'], [len(df_wships)/len(train),len(df_woships)/len(train)], \n        color = ['lightblue','pink'])\nplt.ylabel('Number of images')\nplt.title('Unbalanced Trainig Data (Normalized)')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.000625Z","iopub.execute_input":"2021-06-29T08:12:10.00092Z","iopub.status.idle":"2021-06-29T08:12:10.423959Z","shell.execute_reply.started":"2021-06-29T08:12:10.000892Z","shell.execute_reply":"2021-06-29T08:12:10.423207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot histogram\nhist = df_wships.hist(bins=np.arange(df_wships['counts'].max())+0.5)\nplt.xticks(range(15))\nplt.title(\"Histogram of ships count\")\nplt.xlabel(\"Number of ships\")\nplt.ylabel(\"Number of images\")\nplt.show(hist)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.42578Z","iopub.execute_input":"2021-06-29T08:12:10.426304Z","iopub.status.idle":"2021-06-29T08:12:10.667959Z","shell.execute_reply.started":"2021-06-29T08:12:10.426264Z","shell.execute_reply":"2021-06-29T08:12:10.666868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tabulate import tabulate\n\ncounts = df_wships['counts'].value_counts(sort=False)\nresults = [(i, counts[i]) for i in range(1,16)]\nprint(tabulate(results, headers=['number of ships','number of images'], tablefmt='presto'))","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.669566Z","iopub.execute_input":"2021-06-29T08:12:10.66997Z","iopub.status.idle":"2021-06-29T08:12:10.684664Z","shell.execute_reply.started":"2021-06-29T08:12:10.669927Z","shell.execute_reply":"2021-06-29T08:12:10.683447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Challenge of detecting the ships in the images can be thought as a classification problem for pixels, where, for each image, we need to classify 768 × 768 pixels in one of two classes: ship and no-ship. At pixel level the problem of imbalance data is more accentuated than the case of considering the whole image. To find the proportion between ship and no-ship at pixel level we need to consider the total number of pixel and the pixel where the ships are. The total number of pixel is 768×768×n_imgs while the number of 'ship pixels' is the sum of the all the pair positions of the  strings encoded in 'EncodedPixels'. Finally the total amount of 'no ship pixels' is just: total pixels - 'ship pixels'. We tried to do that but, while counting the numbers of ship pixels, we exceeded the ammount of memory provide by Kaggle. Therefore we decided to report the results of other notebooks and discuss the results.\n","metadata":{}},{"cell_type":"code","source":"def show_pixels_distribution(df):\n    \"\"\"\n    Prints the amount of ship and no-ship pixels in the df\n    \"\"\"\n    # Total images in the df\n    n_images = df['ImageId'].nunique() \n    \n    # Total pixels in the df\n    total_pixels = n_images * 768 * 768 \n\n    # Keep only rows with RLE boxes, transform them into list of pixels, sum the lengths of those lists\n    ship_pixels = df['EncodedPixels'].dropna().apply(rle_to_pixels).str.len().sum() \n\n    ratio = ship_pixels / total_pixels\n    print(f\"Ship: {round(ratio, 3)} ({ship_pixels})\")\n    print(f\"No ship: {round(1 - ratio, 3)} ({total_pixels - ship_pixels})\")\n    ","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.685756Z","iopub.execute_input":"2021-06-29T08:12:10.686051Z","iopub.status.idle":"2021-06-29T08:12:10.692636Z","shell.execute_reply.started":"2021-06-29T08:12:10.686021Z","shell.execute_reply":"2021-06-29T08:12:10.691563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df = pd.read_csv(\"../input/airbus-ship-detection/train_ship_segmentations_v2.csv\")\n# show_pixels_distribution(df)\n","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.693808Z","iopub.execute_input":"2021-06-29T08:12:10.694134Z","iopub.status.idle":"2021-06-29T08:12:10.706959Z","shell.execute_reply.started":"2021-06-29T08:12:10.694094Z","shell.execute_reply":"2021-06-29T08:12:10.706173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Results** (considering the whole dataset):\n\nShip:    0.001 (127777104) \n\nNo ship: 0.999 (113446373040)","metadata":{}},{"cell_type":"markdown","source":"As we can see from the results,  only 1‰ of the pixels are *ships*, while 99.9% of the pixels are *no-ships*. This imply that the dataset is very unbalanced at pixel level. Below we have reported the results considering only the images with the ships, to understand if the problem persists.","metadata":{}},{"cell_type":"code","source":"# show_pixels_distribution(df.dropna())","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.708389Z","iopub.execute_input":"2021-06-29T08:12:10.708941Z","iopub.status.idle":"2021-06-29T08:12:10.717691Z","shell.execute_reply.started":"2021-06-29T08:12:10.708898Z","shell.execute_reply":"2021-06-29T08:12:10.716783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Results** (considering only the images with ships):\n\n\nShip: 0.005 (127777104)\n\n\nNo ship: 0.995 (24972773040)","metadata":{}},{"cell_type":"markdown","source":"The class imbalance is reduced, but it is still very high: 5‰. So in the images with ships only 0.5% of the pixels are ships while 99.5% are no-ships. This condition of extreme class imbalance of the dataset is a problem and so we had to take this into account when deciding which strategy was best for solving the ship detection task.","metadata":{}},{"cell_type":"markdown","source":"## Images Visualizations\n<a id=\"subsection-two-2\"></a>\n\nIn this section we will investigate better the dataset, in particular the images and the masks that will be used in the training part. \nFirstly we plot only the maks, while in after we merge the images with the mask obtaining the real satellite image with the presenceof the ship detected. ","metadata":{}},{"cell_type":"code","source":"# Plot some masks\nw = 8\nh = 2\n\n_, axes_list = plt.subplots(h, w, figsize=(2*w, 2*h))\nplt.subplots_adjust(wspace=0.4)\nax.set(xlim=(0, 768), ylim=(0, 768))\nfor axes in axes_list:\n    for ax in axes:\n        ax.axis('auto')\n        canvas = np.zeros((768, 768))\n        pixels = rle_to_pixels(np.random.choice(df['EncodedPixels']))\n        canvas[tuple(zip(*pixels))] = 1\n        ax.imshow(canvas);","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:10.719176Z","iopub.execute_input":"2021-06-29T08:12:10.719581Z","iopub.status.idle":"2021-06-29T08:12:13.258613Z","shell.execute_reply.started":"2021-06-29T08:12:10.71954Z","shell.execute_reply":"2021-06-29T08:12:13.257546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#In some images there are more that one ship so the df has more than one row, we merge this data into one rle code\ndf = df.groupby(\"ImageId\")[['EncodedPixels']].agg(lambda rle_codes: ' '.join(rle_codes)).reset_index()\nload_img = lambda filename: np.array(PIL.Image.open(f\"../input/airbus-ship-detection/train_v2/{filename}\"))","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:13.259851Z","iopub.execute_input":"2021-06-29T08:12:13.260159Z","iopub.status.idle":"2021-06-29T08:12:14.041996Z","shell.execute_reply.started":"2021-06-29T08:12:13.260129Z","shell.execute_reply":"2021-06-29T08:12:14.040955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot some images with maks\nw = 8\nh = 3\n\n_, axes_list = plt.subplots(h, w, figsize=(2*w, 2*h))\n\nfor axes in axes_list:\n    for ax in axes:\n        ax.axis('off')\n        row_index = np.random.randint(len(df)) # take a random row from the df\n        ax.imshow(apply_mask(load_img(df.loc[row_index, 'ImageId']), rle_to_pixels(df.loc[row_index, 'EncodedPixels'])))\n        ax.set_title(df.loc[row_index, 'ImageId'])","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:14.043132Z","iopub.execute_input":"2021-06-29T08:12:14.043407Z","iopub.status.idle":"2021-06-29T08:12:17.851514Z","shell.execute_reply.started":"2021-06-29T08:12:14.04338Z","shell.execute_reply":"2021-06-29T08:12:17.850337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Sampling\n<a id=\"subsection-three-2\"></a>\n\nIt is possible to notive the unblancing of the dataset regarding the presence or not pf ships in the images. In particular in the trainig set we have arounf 22% of the images with ships while the others 78% without. As we will se later we want to select only the images with ships to manage the detection task, so it is a good idea to classify before the images according to the presence of ships and then proceeed with the detection. \n\nTo have a better performance in the classification it could be better to manipulate a little bit the images in such a way to have a better differentiation among the images and a better balancing between the high and low presence classes. In this section we will explore this kind of arguments.","metadata":{}},{"cell_type":"markdown","source":"**Create different datasets**\n\nOne way to proceed can be to make the dataset used for the trainig in the classification task more balanced, in this case we can use all the images with ships and balance the dataset with part of the non-ship images. We will create two different dataset as follow:\n\n* 50% Ship + 50% No Ship ---> Total of 85.112 Images\n* 100% Ship + 0% No Ship ---> Total of 42.556  Images\n\nWe will train the classifier and test the performace for different trainig sets. ","metadata":{}},{"cell_type":"markdown","source":"Recall that each image corresponds to a number of masks equal to the number of ships contained in the image.","metadata":{}},{"cell_type":"code","source":"masks = pd.read_csv(os.path.join('../input/airbus-ship-detection/train_ship_segmentations_v2.csv'))\nmasks = masks[~masks['ImageId'].isin(['6384c3e78.jpg'])] # remove corrupted file \n\nprint(masks.shape[0], 'masks found')\nprint(masks['ImageId'].value_counts().shape[0], 'unique images found')","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:17.853168Z","iopub.execute_input":"2021-06-29T08:12:17.853604Z","iopub.status.idle":"2021-06-29T08:12:18.70202Z","shell.execute_reply.started":"2021-06-29T08:12:17.853564Z","shell.execute_reply":"2021-06-29T08:12:18.701155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To obtain the first dataset (50% Ship + 50% No Ship) we need to undersample the empty images. The number of images with ships are 42.556 so we need to remove 107443 (=149999 - 42.556) images from the no ship class.","metadata":{}},{"cell_type":"code","source":"# To an empty image correspond just one mask\nn_rem = 107443 # = 149999 - 42.556\nmasks = masks.drop(masks[masks.EncodedPixels.isnull()].sample(n_rem,random_state=42).index)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:18.703119Z","iopub.execute_input":"2021-06-29T08:12:18.703512Z","iopub.status.idle":"2021-06-29T08:12:18.819169Z","shell.execute_reply.started":"2021-06-29T08:12:18.703484Z","shell.execute_reply":"2021-06-29T08:12:18.818297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Split into training and validation groups**\n\nWe stratify by the number of boats appearing so we keep the proportion of occurrences before the split.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nunique_img_ids = masks.groupby('ImageId').size().reset_index(name='counts')\ntrain_ids, valid_ids = train_test_split(unique_img_ids, \n                 test_size = 0.05, \n                 stratify = unique_img_ids['counts'],\n                 random_state=42\n                )\ntrain_df = pd.merge(masks, train_ids)\nvalid_df = pd.merge(masks, valid_ids)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:18.820251Z","iopub.execute_input":"2021-06-29T08:12:18.8207Z","iopub.status.idle":"2021-06-29T08:12:19.351447Z","shell.execute_reply.started":"2021-06-29T08:12:18.820671Z","shell.execute_reply":"2021-06-29T08:12:19.350413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['counts'] = train_df.apply(lambda c_row: c_row['counts'] if \n                                    isinstance(c_row['EncodedPixels'], str) else\n                                    0, 1)\nvalid_df['counts'] = valid_df.apply(lambda c_row: c_row['counts'] if \n                                    isinstance(c_row['EncodedPixels'], str) else\n                                    0, 1)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:19.352694Z","iopub.execute_input":"2021-06-29T08:12:19.353025Z","iopub.status.idle":"2021-06-29T08:12:21.377527Z","shell.execute_reply.started":"2021-06-29T08:12:19.352969Z","shell.execute_reply":"2021-06-29T08:12:21.376729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.ImageId.nunique(), 'unique images in train set')\nprint(valid_df.ImageId.nunique(),' unique images in test set')\n","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:21.378567Z","iopub.execute_input":"2021-06-29T08:12:21.378988Z","iopub.status.idle":"2021-06-29T08:12:21.425276Z","shell.execute_reply.started":"2021-06-29T08:12:21.378938Z","shell.execute_reply":"2021-06-29T08:12:21.424216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot histogram\n\nhist = train_df.hist(bins=np.arange(train_df['counts'].max())-0.5)\nplt.xticks(range(15))\nplt.title(\"Histogram of ships count in training set\")\nplt.xlabel(\"Number of ships\")\nplt.ylabel(\"Number of images\")\nplt.show(hist)\n\n\nhist = valid_df.hist(bins=np.arange(valid_df['counts'].max())-0.5)\nplt.xticks(range(15))\nplt.title(\"Histogram of ships count in validation set\")\nplt.xlabel(\"Number of ships\")\nplt.ylabel(\"Number of images\")\nplt.show(hist)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:21.426662Z","iopub.execute_input":"2021-06-29T08:12:21.426995Z","iopub.status.idle":"2021-06-29T08:12:21.914578Z","shell.execute_reply.started":"2021-06-29T08:12:21.426944Z","shell.execute_reply":"2021-06-29T08:12:21.913851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#save the dataset into .csv files\ntrain_df.to_csv('train_df.csv',index=False)\nvalid_df.to_csv('valid_df.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:21.917269Z","iopub.execute_input":"2021-06-29T08:12:21.917711Z","iopub.status.idle":"2021-06-29T08:12:23.905213Z","shell.execute_reply.started":"2021-06-29T08:12:21.917666Z","shell.execute_reply":"2021-06-29T08:12:23.904259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To obtain the second dataset (100% Ship + 0% No Ship) we need to remove all the images without ships from the original dataset.","metadata":{}},{"cell_type":"code","source":"masks = pd.read_csv(os.path.join('../input/airbus-ship-detection/train_ship_segmentations_v2.csv'))\nmasks = masks[~masks['ImageId'].isin(['6384c3e78.jpg'])] # remove corrupted file \nmasks_ship = masks.drop(masks[masks.EncodedPixels.isnull()].index) # remove all the images without ships\n","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:23.90802Z","iopub.execute_input":"2021-06-29T08:12:23.90836Z","iopub.status.idle":"2021-06-29T08:12:24.637912Z","shell.execute_reply.started":"2021-06-29T08:12:23.908326Z","shell.execute_reply":"2021-06-29T08:12:24.63687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Split into training and validation groups**\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nunique_img_ids = masks_ship.groupby('ImageId').size().reset_index(name='counts')\ntrain_ids_ship, valid_ids_ship = train_test_split(unique_img_ids, \n                 test_size = 0.05, \n                 stratify = unique_img_ids['counts'],\n                 random_state=42\n                )\ntrain_df_ship = pd.merge(masks_ship, train_ids_ship)\nvalid_df_ship = pd.merge(masks_ship, valid_ids_ship)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:24.639439Z","iopub.execute_input":"2021-06-29T08:12:24.640047Z","iopub.status.idle":"2021-06-29T08:12:24.813992Z","shell.execute_reply.started":"2021-06-29T08:12:24.640003Z","shell.execute_reply":"2021-06-29T08:12:24.81297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_ship['counts'] = train_df_ship.apply(lambda c_row: c_row['counts'] if \n                                            isinstance(c_row['EncodedPixels'], str) else\n                                            0, 1)\nvalid_df_ship['counts'] = valid_df_ship.apply(lambda c_row: c_row['counts'] if\n                                            isinstance(c_row['EncodedPixels'], str) else\n                                            0, 1)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:24.815216Z","iopub.execute_input":"2021-06-29T08:12:24.815523Z","iopub.status.idle":"2021-06-29T08:12:26.285447Z","shell.execute_reply.started":"2021-06-29T08:12:24.815493Z","shell.execute_reply":"2021-06-29T08:12:26.28429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df_ship.ImageId.nunique(), 'unique images in train set')\nprint(valid_df_ship.ImageId.nunique(),' unique images in test set')\n","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:26.286878Z","iopub.execute_input":"2021-06-29T08:12:26.287318Z","iopub.status.idle":"2021-06-29T08:12:26.323443Z","shell.execute_reply.started":"2021-06-29T08:12:26.287276Z","shell.execute_reply":"2021-06-29T08:12:26.322454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot histogram\n\nhist = train_df_ship.hist(bins=np.arange(train_df_ship['counts'].max())+0.5)\nplt.xticks(range(15))\nplt.title(\"Histogram of ships count in training set\")\nplt.xlabel(\"Number of ships\")\nplt.ylabel(\"Number of images\")\nplt.show(hist)\n\n\nhist = valid_df_ship.hist(bins=np.arange(valid_df_ship['counts'].max())+0.5)\nplt.xticks(range(15))\nplt.title(\"Histogram of ships count in validation set\")\nplt.xlabel(\"Number of ships\")\nplt.ylabel(\"Number of images\")\nplt.show(hist)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:26.324931Z","iopub.execute_input":"2021-06-29T08:12:26.325262Z","iopub.status.idle":"2021-06-29T08:12:26.811307Z","shell.execute_reply.started":"2021-06-29T08:12:26.325229Z","shell.execute_reply":"2021-06-29T08:12:26.810165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#save the dataset into .csv files\ntrain_df_ship.to_csv('train_df_ship.csv',index=False)\nvalid_df_ship.to_csv('valid_df_ship.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-29T08:12:26.812708Z","iopub.execute_input":"2021-06-29T08:12:26.813115Z","iopub.status.idle":"2021-06-29T08:12:28.75033Z","shell.execute_reply.started":"2021-06-29T08:12:26.813082Z","shell.execute_reply":"2021-06-29T08:12:28.749238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will continue the challange in another notebook to have a better reading and organization. In this first notebook we faced the problem and analyze the data avalibale. In particulat we detected an high unbalanced classification problem (pixel level). We shown the masks and the images and the distributions of images with and without ships. In the next notebooks we will explore the classification of images into two classes, one with ship and another without ships. ","metadata":{}}]}