{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":9988,"databundleVersionId":868324,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Initialization**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ncsv_file = \"/kaggle/input/airbus-ship-detection/train_ship_segmentations_v2.csv\"\nroot_dir = \"/kaggle/input/airbus-ship-detection/train_v2/\"\ndata = pd.read_csv(csv_file)\n\nprint(data.head())","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:38.565627Z","iopub.execute_input":"2024-07-30T07:48:38.566110Z","iopub.status.idle":"2024-07-30T07:48:39.310100Z","shell.execute_reply.started":"2024-07-30T07:48:38.566072Z","shell.execute_reply":"2024-07-30T07:48:39.308719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data.info())","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:39.312647Z","iopub.execute_input":"2024-07-30T07:48:39.313093Z","iopub.status.idle":"2024-07-30T07:48:39.379697Z","shell.execute_reply.started":"2024-07-30T07:48:39.313050Z","shell.execute_reply":"2024-07-30T07:48:39.378166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Unique images: {data['ImageId'].nunique()}\")","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:39.381346Z","iopub.execute_input":"2024-07-30T07:48:39.381762Z","iopub.status.idle":"2024-07-30T07:48:39.478628Z","shell.execute_reply.started":"2024-07-30T07:48:39.381728Z","shell.execute_reply":"2024-07-30T07:48:39.477253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the distribution of missing data\nprint(data.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:39.481911Z","iopub.execute_input":"2024-07-30T07:48:39.483229Z","iopub.status.idle":"2024-07-30T07:48:39.578396Z","shell.execute_reply.started":"2024-07-30T07:48:39.483166Z","shell.execute_reply":"2024-07-30T07:48:39.576483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add a column to mark the presence of ships\ndata['has_ship'] = data['EncodedPixels'].notnull()\n\n# Group by ImageId and check the presence of ships\ngrouped_data = data.groupby('ImageId')['has_ship'].any().reset_index()\n\n# Count of images with and without ships\nship_count = grouped_data['has_ship'].value_counts()\nprint(ship_count)\n\nplt.figure(figsize=(12, 8))\nship_count.plot(kind='bar', color=['skyblue', 'salmon'])\nplt.title('Distribution of Images with and without Ships')\nplt.xlabel('Presence of Ships')\nplt.ylabel('Number of Images')\nplt.xticks(ticks=[0, 1], labels=['Without Ships', 'With Ships'], rotation=0)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:39.580448Z","iopub.execute_input":"2024-07-30T07:48:39.582125Z","iopub.status.idle":"2024-07-30T07:48:40.237552Z","shell.execute_reply.started":"2024-07-30T07:48:39.582064Z","shell.execute_reply":"2024-07-30T07:48:40.236169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Group by ImageId and count the number of ships on each image\nship_counts_per_image = data.groupby('ImageId')['EncodedPixels'].count().reset_index()\nship_counts_per_image.columns = ['ImageId', 'ship_count']\n\n# Distribution of the number of ships on images\nship_distribution = ship_counts_per_image['ship_count'].value_counts()\nprint(ship_distribution)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:40.239451Z","iopub.execute_input":"2024-07-30T07:48:40.239931Z","iopub.status.idle":"2024-07-30T07:48:40.526362Z","shell.execute_reply.started":"2024-07-30T07:48:40.239860Z","shell.execute_reply":"2024-07-30T07:48:40.524982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Distribution of the Number of Ships on Images**\n","metadata":{}},{"cell_type":"code","source":"# Filter images with at least one ship\nship_counts_per_image = ship_counts_per_image[ship_counts_per_image['ship_count'] > 0]\n\n# Distribution of the number of ships on images\nship_distribution = ship_counts_per_image['ship_count'].value_counts().sort_index()\n\nplt.figure(figsize=(12, 8))\nax = ship_distribution.plot(kind='bar', color='skyblue')\nplt.title('Distribution of the Number of Ships on Images')\nplt.xlabel('Number of Ships')\nplt.ylabel('Number of Images')\n\nfor i in ax.containers:\n    ax.bar_label(i)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:40.528001Z","iopub.execute_input":"2024-07-30T07:48:40.529110Z","iopub.status.idle":"2024-07-30T07:48:41.062268Z","shell.execute_reply.started":"2024-07-30T07:48:40.529069Z","shell.execute_reply":"2024-07-30T07:48:41.060780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Example of an Input Image and сorresponding Mask**\n","metadata":{}},{"cell_type":"code","source":"from skimage.io import imread\nimport os\n\n# Filter images with ships\ndata_with_ships = data[data['EncodedPixels'].notnull()]\n\n# Decoding masks in Run-Length Encoding format\ndef rle_decode(mask_rle, shape=(768, 768)):\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0::2], s[1::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape, order='F')\n\n# Visualize the image and mask\ndef visualize_image_and_mask(image_id, root_dir, title_suffix=''):\n    image_path = os.path.join(root_dir, image_id)\n    image = imread(image_path)\n    \n    masks = data_with_ships[data_with_ships['ImageId'] == image_id]['EncodedPixels'].tolist()\n    mask = np.zeros((768, 768), dtype=np.uint8)\n    \n    for mask_rle in masks:\n        mask += rle_decode(mask_rle)\n    \n    fig, ax = plt.subplots(1, 2, figsize=(12, 6))\n    \n    ax[0].imshow(image)\n    ax[0].set_title(f'Input Image {title_suffix}')\n    \n    ax[1].imshow(mask, cmap='gray')\n    ax[1].set_title(f'Mask {title_suffix}')\n    \n    plt.show()\n\n# Visualization for one example\nexample_image_id = data_with_ships['ImageId'].values[0]\nvisualize_image_and_mask(example_image_id, root_dir)\n","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:41.064245Z","iopub.execute_input":"2024-07-30T07:48:41.064752Z","iopub.status.idle":"2024-07-30T07:48:41.919290Z","shell.execute_reply.started":"2024-07-30T07:48:41.064707Z","shell.execute_reply":"2024-07-30T07:48:41.918078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Percentage of the Image Area Occupied by Ships**\n","metadata":{}},{"cell_type":"code","source":"# Filter images with ships\ndata_with_ships = data[data['EncodedPixels'].notnull()]\n\n# Calculate the percentage of the image occupied by ships\ndef calculate_ship_percentage(rle_list, image_shape=(768, 768)):\n    total_pixels = image_shape[0] * image_shape[1]\n    ship_pixels = sum(np.sum(rle_decode(rle)) for rle in rle_list)\n    return (ship_pixels / total_pixels) * 100\n\n# Group masks by ImageId\ngrouped_data = data_with_ships.groupby('ImageId')['EncodedPixels'].apply(list).reset_index()\n\n# Add a column with the percentage of the image occupied by ships\ngrouped_data['ship_percentage'] = grouped_data['EncodedPixels'].apply(calculate_ship_percentage)\n\nplt.figure(figsize=(12, 8))\nplt.hist(grouped_data['ship_percentage'], bins=30, color='skyblue')\nplt.title('Distribution of the Percentage of the Image Occupied by Ships')\nplt.xlabel('Percentage of the Image')\nplt.ylabel('Number of Images')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:48:41.921115Z","iopub.execute_input":"2024-07-30T07:48:41.921481Z","iopub.status.idle":"2024-07-30T07:49:28.527126Z","shell.execute_reply.started":"2024-07-30T07:48:41.921451Z","shell.execute_reply":"2024-07-30T07:49:28.525652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Heatmap of Ship Positions**\n","metadata":{}},{"cell_type":"code","source":"heatmap = np.zeros((768, 768))\n\nfor mask_rle in data_with_ships['EncodedPixels']:\n    heatmap += rle_decode(mask_rle)\n\nplt.figure(figsize=(10, 10))\nplt.imshow(heatmap, cmap='hot', interpolation='nearest')\nplt.title('Heatmap of Ship Positions')\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:49:28.531207Z","iopub.execute_input":"2024-07-30T07:49:28.531615Z","iopub.status.idle":"2024-07-30T07:51:18.733597Z","shell.execute_reply.started":"2024-07-30T07:49:28.531581Z","shell.execute_reply":"2024-07-30T07:51:18.732341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Average, Minimum, and Maximum Ship Area**\n","metadata":{}},{"cell_type":"code","source":"# Adding 'ship_area' column to calculate the ship area\ndef calculate_ship_area(mask_rle):\n    mask = rle_decode(mask_rle)\n    return np.sum(mask)\n\ndata_with_ships['ship_area'] = data_with_ships['EncodedPixels'].apply(calculate_ship_area)\n\n# Calculating the average, minimum, and maximum ship areas\naverage_ship_area = data_with_ships['ship_area'].mean()\nmin_ship_area = data_with_ships['ship_area'].min()\nmax_ship_area = data_with_ships['ship_area'].max()\n\nprint(f'Average ship area: {average_ship_area:.2f} pixels')\nprint(f'Minimum ship area: {min_ship_area} pixels')\nprint(f'Maximum ship area: {max_ship_area} pixels')","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:51:18.734986Z","iopub.execute_input":"2024-07-30T07:51:18.735336Z","iopub.status.idle":"2024-07-30T07:52:02.564556Z","shell.execute_reply.started":"2024-07-30T07:51:18.735306Z","shell.execute_reply":"2024-07-30T07:52:02.563197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualization for the largest ship\nmax_ship = data_with_ships[data_with_ships['ship_area'] == max_ship_area].iloc[0]\nvisualize_image_and_mask(max_ship['ImageId'], root_dir, title_suffix='(Largest Ship)')\n\n# Visualization for the smallest ship\nmin_ship = data_with_ships[data_with_ships['ship_area'] == min_ship_area].iloc[0]\nvisualize_image_and_mask(min_ship['ImageId'], root_dir, title_suffix='(Smallest Ship)')","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:52:02.565956Z","iopub.execute_input":"2024-07-30T07:52:02.566314Z","iopub.status.idle":"2024-07-30T07:52:04.066268Z","shell.execute_reply.started":"2024-07-30T07:52:02.566285Z","shell.execute_reply":"2024-07-30T07:52:04.064962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Visualizations for Clarity**\n","metadata":{}},{"cell_type":"code","source":"from scipy.spatial import distance\n\n# Calculate pixel coordinates of the mask\ndef get_pixel_coordinates(mask_rle):\n    mask = rle_decode(mask_rle)\n    return np.argwhere(mask == 1)\n\n# Check if ships are close to each other\ndef are_ships_close(coords_list, min_distance=50):\n    \"\"\"\n    Determine if any ships in the list of coordinates are close to each other.\n    \"\"\"\n    for i, coords1 in enumerate(coords_list):\n        for j, coords2 in enumerate(coords_list):\n            if i >= j:\n                continue\n            if distance.cdist(coords1, coords2).min() < min_distance:\n                return True\n    return False\n\n# Find images with more than two ships\ndef find_images_with_more_than_two_ships(data_with_ships):\n    images_with_more_than_two_ships = []\n    for image_id in data_with_ships['ImageId'].unique():\n        masks = data_with_ships[data_with_ships['ImageId'] == image_id]['EncodedPixels'].tolist()\n        if len(masks) > 2:\n            coords_list = []\n            for mask_rle in masks:\n                coords = get_pixel_coordinates(mask_rle)\n                coords_list.append(coords)\n            \n            if are_ships_close(coords_list):\n                images_with_more_than_two_ships.append(image_id)\n    \n    return images_with_more_than_two_ships\n\n# Find images with more than two ships\nimages_with_more_than_two_ships = find_images_with_more_than_two_ships(data_with_ships)\n\n# Visualize one such image (e.g., the fifth in the list)\nif images_with_more_than_two_ships:\n    example_image_id = images_with_more_than_two_ships[4]\n    visualize_image_and_mask(example_image_id, root_dir)\nelse:\n    print(\"No images with more than two ships found.\")","metadata":{"execution":{"iopub.status.busy":"2024-07-30T07:52:04.068028Z","iopub.execute_input":"2024-07-30T07:52:04.068619Z","iopub.status.idle":"2024-07-30T08:11:42.389599Z","shell.execute_reply.started":"2024-07-30T07:52:04.068574Z","shell.execute_reply":"2024-07-30T08:11:42.388438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if images_with_more_than_two_ships:\n    example_image_id = images_with_more_than_two_ships[24]\n    visualize_image_and_mask(example_image_id, root_dir)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T08:11:42.391260Z","iopub.execute_input":"2024-07-30T08:11:42.391708Z","iopub.status.idle":"2024-07-30T08:11:43.142400Z","shell.execute_reply.started":"2024-07-30T08:11:42.391665Z","shell.execute_reply":"2024-07-30T08:11:43.141090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if images_with_more_than_two_ships:\n    example_image_id = images_with_more_than_two_ships[55]\n    visualize_image_and_mask(example_image_id, root_dir)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T08:11:43.143923Z","iopub.execute_input":"2024-07-30T08:11:43.144284Z","iopub.status.idle":"2024-07-30T08:11:43.991418Z","shell.execute_reply.started":"2024-07-30T08:11:43.144253Z","shell.execute_reply":"2024-07-30T08:11:43.990100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if images_with_more_than_two_ships:\n    example_image_id = images_with_more_than_two_ships[58]\n    visualize_image_and_mask(example_image_id, root_dir)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T08:11:43.993031Z","iopub.execute_input":"2024-07-30T08:11:43.993596Z","iopub.status.idle":"2024-07-30T08:11:44.762334Z","shell.execute_reply.started":"2024-07-30T08:11:43.993481Z","shell.execute_reply":"2024-07-30T08:11:44.760497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if images_with_more_than_two_ships:\n    example_image_id = images_with_more_than_two_ships[27]\n    visualize_image_and_mask(example_image_id, root_dir)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T08:11:44.764277Z","iopub.execute_input":"2024-07-30T08:11:44.764706Z","iopub.status.idle":"2024-07-30T08:11:45.590364Z","shell.execute_reply.started":"2024-07-30T08:11:44.764670Z","shell.execute_reply":"2024-07-30T08:11:45.588966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Conclusion**\n","metadata":{}},{"cell_type":"markdown","source":"There are 192,556 unique images, with 150,000 images without ships and 42,556 images with ships. Due to the significant imbalance, **it will be necessary to balance the data**. There is also a notable imbalance in the number of ships per image:   \n\n1 ship: 27,104 images - 63.69%  \n2 ships: 7,674 images - 18.03%  \n3 ships: 2,954 images - 6.94%  \n4 ships: 1,622 images - 3.81%  \n5 ships: 925 images - 2.17%  \n6 ships: 657 images - 1.54%  \n7 ships: 406 images - 0.95%  \n8 ships: 318 images - 0.75%  \n9 ships: 243 images - 0.57%  \n10 ships: 168 images - 0.39%  \n11 ships: 144 images - 0.34%  \n12 ships: 124 images - 0.29%  \n13 ships: 75 images - 0.18%  \n14 ships: 76 images - 0.18%  \n15 ships: 66 images - 0.16%  \n\nBy constructing a heatmap of ship positions, it is noticeable that ships rarely occupy positions at the edges of the image and are much more often located in the center. **It might be worthwhile to apply data augmentation to address this.**\n\nAdditionally, by calculating the area occupied by ships in the images, it is observed that **ships occupy on average between 0.1% to 2% of the image, and in rare cases, between 2% to 4%.**\n\nThe smallest ship is depicted on just 2 pixels, while the largest one occupies 25,904 pixels. **On average, a ship occupies 1,567 pixels.**\n\nMoreover, when examining the image and mask of the largest ship, it can be seen that **the ship's shape and the mask do not perfectly match**. It was also found that the masks do not have rounded shapes, as ships usually do, which could negatively affect the results. **It is worth considering the use of Pseudo-Labeling.**\n\nAs illustrated in the images, **ships are also often positioned close to each other**, making it difficult for a simple model to distinguish between them, and it may be necessary to apply techniques to address this feature.\n\nAs seen in the last image, ships can be found in bays very close to similarly shaped buildings.","metadata":{}}]}