{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MASK R-CNN\ncreated by Kaushal Shah edited by Niclas Bürger","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# Motivation\n\n* Erkennung und Klassifzierung einzelner Objkete innerhalb eines Bildes oder Videos\n* Ziel dieses Notebooks ist es einzelne Kleidungstücke zu erkennen und zu klassifzieren\n    * Der erste Teil des Notebooks beschäftig sich mit der Visualiserung und bereinigung der Daten\n    * Anschließend wird das Modell mithilfe von Traingsdaten trainiert\n    * Zu guter letzt wird das trainierte Modell getestet ","metadata":{}},{"cell_type":"markdown","source":"# Theoretischer Überblick\n\n* Mask R-CNN ist ein neuronales Netzt, welches auf die Lösung des Instanzsegmentierungsproblems in maschinellen lernen abziehlt.\n* Die Instanz Segmentierung befasst sich hierbei mit der korrekten Erkennung aller Objekte in einem Bild oder Video, \n  bei gleichzeitig präziser Segmentierung jeder Instanz (also klare Trennung von einzelnen Objekten, die als ähnliche Instanz klassifiziert werden).\n* Die Instanz Segmentierung ist also eine Kombination aus:\n    * Objekterkennung\n    * Objektlokalisierung \n    * Objektklassifizierung \n    \n    \n\n* Aufbau bestehend aus: \n    * CNN (Convolutional Neural Networks): Kann ein einzelnes Objekt in einem Bild erkennen und klassifzieren \n    * R-CNN (Region Based CNN): Verwendet Bounding Boxes und ROI (Region of Interest) um Regionen auszuwerten. Architektur bildet Grundlage für Mask R-CNN\n    * Faster R-CNN: Weriterentwicklung von R-CNN. Extrahiert mithife von RPN (Region Proposal Network) Features aus einem Bild und führt eine Klassifzierung und eine Bounding Box Regression durch.  \n\n\n* Vorgehen für die Klassifzierung in zwei Schritten:\n    1. Generierung von Interessanten Regionen in denen sich jeweils ein Objekt befinden könnte\n    2. Zuweisung einer Klasse, Verfeinerung der Bounding Box und Erzeugung einer Maske auf Pixel Ebene\n\nAuf dem Bild is die Maske, die Bounding Box und die Klassizierung zu erkennen.\n\n<img src= \"https://blog.paperspace.com/content/images/2020/11/maskrcnn.png\" alt =\"MASK R-CNN Example Image\" style='width: 800px;'>\n\n","metadata":{}},{"cell_type":"markdown","source":"# Datensatz\n\nDer in diesem Notebook verwendete Datensatz besteht aus:\n* 45600 Trainingsbildern\n* 3200 Testing-Bildern \n\nDie Bilder stellen jeweils eine Person dar, die ein oder mehrere Kleidungsstücke trägt.\n\nHinzu kommt eine 'train.csv' Datei, welche für jedes Bild den folgenden Datensatz beinhaltet: \n* ImageId: Der Bildername\n* EncodedPixels: Gibt in Form von 'run-length encodeten' pixeln die Maske des Objekts an\n* Height: Höhe des Bildes\n* Width: Breite des Bildes\n* ClassId: Verknüfung in form eines Integer Wertes zur Objektklasse \n* AttributsIds: Ids die eine Verknüfung zu Attributen herstellen die das Objekt näher beschreiben\n  (Die ClassIds und AttributIDs sind in der 'label-description.json' näher aufgelistet. Hierin befinden sich 46 Klassifiaktionen und 294 Attribute)\n  \n  \n  ","metadata":{}},{"cell_type":"markdown","source":"# Training\n\nBevor mit der Training des Modells begonnen werden kann muss der Datensatz zu aller erst bereinigt werden. \n\nSo wird unteranderm die Spalte AttributesId nicht benötigt da diese NaN (Not a Number) Werter beinhaltet, gut 38% aller Daten, und somit den Datensatz verfälscht.\n\nAnschschließend wird der Trainingsdatensatz nochmal in zwei unter Datensätze aufgeteilt. \nEinmal in das eigentliche Train-Dataset und in ein Valid-Dataset. Dieses soll die Trainierten Daten validieren.\n\n","metadata":{}},{"cell_type":"markdown","source":"# Testing\n","metadata":{}},{"cell_type":"markdown","source":"# Fazit\n\n* Das Notebook läuft ohne Fehler\n* Das trainierte Modell ist noch nicht genau genug um alle Kleidungsstücke valide zu Klassifizieren\n* Eine Längere Trainingszeit mit mehr Epochen könnte helfen das Modell zu verfeinern\n* Ebenfalls könnten die Attribute verwendet werden um eine genauere Klassifizierung zu ermöglichen","metadata":{}},{"cell_type":"code","source":"# Import required libraries\nimport os\nimport gc\nimport sys\nimport json\nimport random\nfrom pathlib import Path\n\nimport cv2 # CV2 for image manipulation\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom tqdm import tqdm\n\nfrom imgaug import augmenters as iaa\n\nimport seaborn as sns\nimport matplotlib.image as mpimg\nfrom matplotlib import pyplot as plt\n\nfrom sklearn.model_selection import StratifiedKFold, KFold","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install tensorflow==1.14 #updated tensorflow version from 1.5 to 1.14\n!pip install keras==2.1.5\n\nimport tensorflow\n\nprint(f'Tensorflow Version: {tensorflow.__version__}')\nimport keras\nprint(f'Keras Version: {keras.__version__}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Outputs all files in the specific kaggle folder \n\"\"\"\"\"\"\n\n!ls /kaggle/input/imaterialist-fashion-2020-fgvc7/","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# reads in sample data in form of csv and json data\n# label_description.json: describes the objects to be classified\n# smaple_submission.csv\n# train.csv: holds the training data. Maps the ImageIDs to the contained AttributIDs. Also holds the location off the Attribute in form off encoded pixels \n\"\"\"\"\"\"\n\nwith open('/kaggle/input/imaterialist-fashion-2020-fgvc7/label_descriptions.json', 'r') as file:\n    label_desc = json.load(file)\nsample_sub_df = pd.read_csv('/kaggle/input/imaterialist-fashion-2020-fgvc7/sample_submission.csv')\ntrain_df = pd.read_csv('/kaggle/input/imaterialist-fashion-2020-fgvc7/train.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Displays the first five rows in the 'train' dataframe \n\"\"\"\"\"\"\n\ntrain_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Displays the first five rows in the 'sample' dataframe\n\"\"\"\"\"\"\n\nsample_sub_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Tuple representing the dimensions (rows x columns) of the DataFrame \n\"\"\"\"\"\"\n\nprint(f'Shape of training dataset: {train_df.shape}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Prints out the number of images used in the training and test dataset\n# Note: Training set is bigger to better train the model\n\"\"\"\"\"\"\n\nprint(f'# of images in training set: {train_df[\"ImageId\"].nunique()}')\nprint(f'# of images in test set: {sample_sub_df[\"ImageId\"].nunique()}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image size analysis in training dataset","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the min, max and average height and width of all images contained in the 'train' dateframe\n\"\"\"\"\"\"\n\npd.DataFrame([train_df['Height'].describe(), train_df['Width'].describe()]).T.loc[['max', 'min', 'mean']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Height and Width destribution of training images","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# \n\"\"\"\"\"\"\n\nimage_shape_df = train_df.groupby(\"ImageId\")[[\"Height\", \"Width\"]].first()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows height and width distribution accross the dataset as an plot\n# X-axis: Height/Width of Image\n# Y-axis: Number of Images\n\"\"\"\"\"\"\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 5))\nax1.hist(image_shape_df['Height'], bins=100)\nax1.set_title(\"Height distribution\")\nax2.hist(image_shape_df['Width'], bins=100)\nax2.set_title(\"Width distribution\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with minimum height","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the image with the minimum height in the 'train' dateframe\n\"\"\"\"\"\"\n\nplt.figure(figsize = (70,7))\nmin_height = list(set(train_df[train_df['Height'] == train_df['Height'].min()]['ImageId']))[0]\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{min_height}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with maximum height","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the image with the maximum height in the 'train' dateframe\n\"\"\"\"\"\"\n\nplt.figure(figsize = (70,7))\nmax_height = list(set(train_df[train_df['Height'] == train_df['Height'].max()]['ImageId']))[0]\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{max_height}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with minimum width","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the image with the minumum width in the 'train' dateframe\n\"\"\"\"\"\"\n\nplt.figure(figsize = (70,7))\nmin_width = list(set(train_df[train_df['Width'] == train_df['Width'].min()]['ImageId']))[0]\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{min_width}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with maximum width","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the image with the maximum width in the 'train' dateframe\n\"\"\"\"\"\" \n\nplt.figure(figsize = (70,7))\nmax_width = list(set(train_df[train_df['Width'] == train_df['Width'].max()]['ImageId']))[0]\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{max_width}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Sets varibales with the imageID for the image with maximum area und minumum area\n\"\"\"\"\"\" \n\narea_df = pd.DataFrame()\narea_df['ImageId'] = train_df['ImageId']\narea_df['area'] = train_df['Height'] * train_df['Width']\nmin_area = list(set(area_df[area_df['area'] == area_df['area'].min()]['ImageId']))[0]\nmax_area = list(set(area_df[area_df['area'] == area_df['area'].max()]['ImageId']))[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with minimum area","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows image with the minumim area\n\"\"\"\"\"\" \n\nplt.figure(figsize = (70,7))\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{min_area}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image with maximum area","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows image with the maximum area\n\"\"\"\"\"\"\n\nplt.figure(figsize = (70,7))\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{max_area}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Details about Classes and Attributes","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the number of classes to be categorized and their attributes\n\"\"\"\"\"\"\n\nnum_classes = len(label_desc['categories'])\nnum_attributes = len(label_desc['attributes'])\nprint(f'Total # of classes: {num_classes}')\nprint(f'Total # of attributes: {num_attributes}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the categories\n\"\"\"\"\"\"\n\ncategories_df = pd.DataFrame(label_desc['categories'])\nattributes_df = pd.DataFrame(label_desc['attributes'])\ncategories_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the attributes\n\"\"\"\"\"\"\n\npd.set_option('display.max_rows', 300)\nattributes_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting a few training images without any masks","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Function to plot the first twelve unique imgages in the 'train' dataframe without applying any masks\n\"\"\"\"\"\"\n\ndef plot_images(size=12, figsize=(12, 12)):\n    # First get some images to be plotted\n    image_ids = train_df['ImageId'].unique()[:12]\n    images=[]\n    \n    for image in image_ids:\n        images.append(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{image}.jpg'))\n    \n    # Plot images in groups of 4 images\n    n_groups = 4\n    \n    count = 0\n    for index in range(size // 4):\n        fig, ax = plt.subplots(nrows=2, ncols=2, figsize=figsize)\n        for row in ax:\n            for col in row:\n                col.imshow(images[count])\n                col.axis('off')\n                count += 1\n        plt.show()\n    gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Run function to plot the first twelve unique images\n\"\"\"\"\"\"\n\nplot_images()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting a few images with given segments","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Function to apply encoded pixels to the twelve first unique images\n# First the respective image metadata is added to the image\n# Then iterate over every image and apply the specific encoded pixels to create the mask\n\"\"\"\"\"\"\n\ndef create_mask(size):\n    image_ids = train_df['ImageId'].unique()[:size]\n    images_meta=[]\n\n    for image_id in image_ids:\n        img = mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{image_id}.jpg')\n        images_meta.append({\n            'image': img,\n            'shape': img.shape,\n            'encoded_pixels': train_df[train_df['ImageId'] == image_id]['EncodedPixels'],\n            'class_ids':  train_df[train_df['ImageId'] == image_id]['ClassId']\n        })\n\n    masks = []\n    for image in images_meta:\n        shape = image.get('shape')\n        encoded_pixels = list(image.get('encoded_pixels'))\n        class_ids = list(image.get('class_ids'))\n        \n        # Initialize numpy array with shape same as image size\n        height, width = shape[:2]\n        mask = np.zeros((height, width)).reshape(-1)\n        \n        # Iterate over encoded pixels and create mask\n        for segment, (pixel_str, class_id) in enumerate(zip(encoded_pixels, class_ids)):\n            splitted_pixels = list(map(int, pixel_str.split()))\n            pixel_starts = splitted_pixels[::2]\n            run_lengths = splitted_pixels[1::2]\n            assert max(pixel_starts) < mask.shape[0]\n            for pixel_start, run_length in zip(pixel_starts, run_lengths):\n                pixel_start = int(pixel_start) - 1\n                run_length = int(run_length)\n                mask[pixel_start:pixel_start+run_length] = 255 - class_id * 4\n        masks.append(mask.reshape((height, width), order='F'))  # https://stackoverflow.com/questions/45973722/how-does-numpy-reshape-with-order-f-work\n    return masks, images_meta","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Function to plot the first twelve unique imgages in the 'train' dataframe with the applyed mask\n\"\"\"\"\"\"\n\ndef plot_segmented_images(size=12, figsize=(14, 14)):\n    # First create masks from given segments\n    masks, images_meta = create_mask(size)\n    \n    # Plot images in groups of 4 images\n    n_groups = 4\n    \n    count = 0\n    for index in range(size // 4):\n        fig, ax = plt.subplots(nrows=2, ncols=2, figsize=figsize)\n        for row in ax:\n            for col in row:\n                col.imshow(images_meta[count]['image'])\n                col.imshow(masks[count], alpha=0.75)\n                col.axis('off')\n                count += 1\n        plt.show()\n    gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Run the function to show the images with mask\n\"\"\"\"\"\"\n\nplot_segmented_images()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Analysing Categories and Attributes","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Create Dataframes with the categories und and the attributes\n\"\"\"\"\"\"\n\ncategories_df = pd.DataFrame(label_desc.get('categories'))\nattributes_df = pd.DataFrame(label_desc.get('attributes'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the ammount of categories and attributes\n\"\"\"\"\"\"\n\nprint(f'# of categories: {len(categories_df)}')\nprint(f'# of attributes: {len(attributes_df)}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So there are 46 categories (classes) and 294 attributes. Let's see some of the categories and attributes","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the first items of the categegories\n\"\"\"\"\"\"\n\ncategories_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the first items of the categegories\n\"\"\"\"\"\"\n\nattributes_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Builds two maps. \n# A map that holds the category and the corresponding id\n# A map that holds the attribute and the corresponding id. \n\"\"\"\"\"\"\n\ncategory_map, attribute_map = {}, {}\nfor cat in label_desc.get('categories'):\n    category_map[cat.get('id')] = cat.get('name')\nfor attr in label_desc.get('attributes'):\n    attribute_map[attr.get('id')] = attr.get('name')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the category map \n\"\"\"\"\"\"\n\ncategory_map","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Shows the attribute map \n\"\"\"\"\"\"\n\ncategory_map","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Print out the 'test' dataframe to show how the 'ClassId' looks like before mapping\n\"\"\"\"\"\"\n\ntrain_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Map the name of the category to the corresponding integer number (0 --> shirt/blouse) \n\"\"\"\"\"\"\n\ntrain_df['ClassId'] = train_df['ClassId'].map(category_map)\ntrain_df['ClassId'] = train_df['ClassId'].astype('category')\ntrain_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's see the class wise distribution of segments in training dataset","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the distibutation of the classes\n\"\"\"\"\"\"\n\nsns.set(style='darkgrid')\nfig, ax = plt.subplots(figsize = (10,10))\nsns.countplot(y='ClassId',data=train_df , ax=ax, order = train_df['ClassId'].value_counts().index)\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Now let's visualize an image with all its classes and attributes","metadata":{}},{"cell_type":"code","source":"IMAGE_ID = '000b3ec2c6eaffb491a5abb72c2e3e26'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Create new dataframe with an specific image and the corosponding classes and attributes\n# Show the dataframe\n# The image has 8 segments. \n# Note: Not every segment has an attribute\n\"\"\"\"\"\"\n\n# Get the an image id given in the training set for visualization\nvis_df = train_df[train_df['ImageId'] == IMAGE_ID]\nvis_df['ClassId'] = vis_df['ClassId'].cat.codes\nvis_df = vis_df.reset_index(drop=True)\nvis_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From above table, this image has 8 segmentes and a few attributes. Let's visualize all of them!","metadata":{}},{"cell_type":"markdown","source":"## Let's first the plot the plain image","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the specific plain image with no shown segments\n# The segments to visualize are:\n# right and left shoe\n# pants\n# top, t-shirt, sweatshirt\n# pocket\n# right and left sleeve\n# neckline\n\"\"\"\"\"\"\n\nplt.figure(figsize = (110,11))\nimage = mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{IMAGE_ID}.jpg')\nplt.grid(False)\nplt.imshow(image)\nplt.plot()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[train_df['ImageId'] == IMAGE_ID]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Now let's plot each segment in a separate image","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Create array with masks for every segment in the seleceted image\n\"\"\"\"\"\"\n\nsegments = list(vis_df['EncodedPixels'])\nclass_ids = list(vis_df['ClassId'])\nmasks = []\nfor segment, class_id in zip(segments, class_ids):\n    \n    height = vis_df['Height'][0]\n    width = vis_df['Width'][0]\n    # Initialize empty mask\n    mask = np.zeros((height, width)).reshape(-1)\n    \n    # Iterate over encoded pixels and create mask\n    splitted_pixels = list(map(int, segment.split()))\n    pixel_starts = splitted_pixels[::2]\n    run_lengths = splitted_pixels[1::2]\n    assert max(pixel_starts) < mask.shape[0]\n    for pixel_start, run_length in zip(pixel_starts, run_lengths):\n        pixel_start = int(pixel_start) - 1\n        run_length = int(run_length)\n        mask[pixel_start:pixel_start+run_length] = 255 - class_id * 4\n\n    mask = mask.reshape((height, width), order='F')\n    masks.append(mask)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Function to visualize a specific segment in the selected image\n\"\"\"\"\"\"\n\ndef plot_individual_segment(*masks, image, figsize=(110, 11)):\n    plt.figure(figsize = figsize)\n    plt.imshow(image)\n    for mask in masks:\n        plt.imshow(mask, alpha=0.6)\n    plt.axis('off')\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 1st Segment: ClassId: \"Shoe\" and no attributes ","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the first segment: 'shoe'\n# Note: This segment has no attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[0], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 2nd Segment: ClassId: \"shoe\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the second segment: 'shoe'\n# Note: This segment has no attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[1], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 3rd Segment with ClassId: \"pants\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the third segment: 'pants'\n# Note: This segment has attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[2], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 4th Segment with ClassId: \"top, t-shirt, sweatshirt\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the fourth segment: 'top, t-shirt, sweatshirt'\n# Note: This segment has attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[3], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 5th Segment with ClassId: \"pocket\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the fifth segment: 'pocket'\n# Note: This segment has attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[4], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 6th Segment with ClassId: \"sleeve\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the sixth segment: 'sleeve'\n# Note: This segment has attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[5], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 7th Segment with ClassId: \"sleeve\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the seventh segment: 'sleeve'\n# Note: This segment has attributes\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[6], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting 8th segment with Class \"neckline\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Plot the image with the seventh segment: 'sleeve'\n# Note: This segment has attributes\n# Note: fixed integer number in array\n\"\"\"\"\"\"\n\nplot_individual_segment(masks[7], image=image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of the segments have no attributes. Let's check how many such segment exists in training dataset.","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Not ervery segement in the 'train' dataset has an attribute.\n# Show number of segements that have no attributes in percent. \n\"\"\"\"\"\"\n\nprint(f'Segments that do not have attributes: {train_df[\"AttributesIds\"].isna().sum()/len(train_df) * 100} %')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check of missing values in training dataset for columns other than \"AttributeIds\"","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Count all values (ImageId, EncodedPixels, Height, Width and ClassId) in the dataframe with NaN\n# Note: Only column 'AttributeId' has NaN values\n\"\"\"\"\"\"\n\ntrain_df[['ImageId', 'EncodedPixels', 'Height', 'Width', 'ClassId']].isna().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preparation and modeling","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Show 'train' dataframe\n\"\"\"\"\"\"\n\ntrain_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Transform ClassIds text into integers (pants --> 24) \n\"\"\"\"\"\"\n\ntrain_df['ClassId'] = train_df['ClassId'].cat.codes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Show 'train' dataframe\n\"\"\"\"\"\"\n\ntrain_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Drop attributeIds for simplicity for now. TODO: Need to take this in consideration once the basic model is ready with ClassId","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Remove AttributesIds from dataframe. This column contains NaN values \n\"\"\"\"\"\"\n\ntrain_df = train_df.drop('AttributesIds', axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Transforms column values from column 'EncodedPixels' and 'ClassId' into an array \n# Note: Bug fixed\n\"\"\"\"\"\"\n\nimage_df = train_df.groupby('ImageId')[['EncodedPixels', 'ClassId']].agg(lambda x: list(x))\nsize_df = train_df.groupby('ImageId')[['Height', 'Width']].mean()\nimage_df = image_df.join(size_df, on='ImageId')\n\nprint(\"Total images: \", len(image_df))\nimage_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reference: https://github.com/matterport/Mask_RCNN/blob/master/samples/shapes/train_shapes.ipynb","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Import Mask R-CNN code from GitGub\n\"\"\"\"\"\"\n\nimport os\nfrom pathlib import Path\n!git clone https://www.github.com/matterport/Mask_RCNN.git\nos.chdir('Mask_RCNN')\n\n!rm -rf .git # to prevent an error when the kernel is committed\n!rm -rf images assets # to prevent displaying images at the bottom of a kernel","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Load COCO Dataset\n\"\"\"\"\"\"\n\n!wget --quiet https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5\n!ls -lh mask_rcnn_coco.h5\n\nCOCO_WEIGHTS_PATH = 'mask_rcnn_coco.h5'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !cat /kaggle/working/Mask_RCNN/mrcnn/model.py","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_DIR = Path('/kaggle/input/imaterialist-fashion-2020-fgvc7')\nROOT_DIR = Path('/kaggle/working')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sys.path = sys.path[:-1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Import Mask R-CNN libary\n\"\"\"\"\"\"\n\n# sys.path.append(ROOT_DIR/'Mask_RCNN')\nfrom mrcnn.config import Config\nfrom mrcnn import utils\nimport mrcnn.model as modellib\nfrom mrcnn import visualize\nfrom mrcnn.model import log","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Config for Training\n# Example: Computer Ressources and number of Epochs\n\"\"\"\"\"\"\n\nclass FashionConfig(Config):\n    \"\"\"Configuration for training on the toy shapes dataset.\n    Derives from the base Config class and overrides values specific\n    to the toy shapes dataset.\n    \"\"\"\n    # Give the configuration a recognizable name\n    NAME = \"class\"\n\n    # Train on 1 GPU and 8 images per GPU. We can put multiple images on each\n    # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 8\n\n    # Number of classes (including background)\n    NUM_CLASSES = 1 + len(categories_df)  # background + 46 classes\n\n    # Use small images for faster training. Set the limits of the small side\n    # the large side, and that determines the image shape.\n    IMAGE_MIN_DIM = 256\n    IMAGE_MAX_DIM = 256\n\n    # Use smaller anchors because our image and objects are small\n    RPN_ANCHOR_SCALES = (8, 16, 32, 64, 128)  # anchor side in pixels\n\n    # Reduce training ROIs per image because the images are small and have\n    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.\n    TRAIN_ROIS_PER_IMAGE = 32\n\n    # Use a small epoch since the data is simple\n    STEPS_PER_EPOCH = 100\n\n    # use small validation steps since the epoch is small\n    VALIDATION_STEPS = 5\n    \nconfig = FashionConfig()\nconfig.display()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Create config class for the fashion dataset\n\"\"\"\"\"\"\n\nclass FashionDataset(utils.Dataset):\n    def __init__(self, df):\n        super().__init__(self)\n        \n        self.IMAGE_SIZE = 256\n        \n        # Add classes\n        for cat in label_desc['categories']:\n            self.add_class('fashion', cat.get('id'), cat.get('name'))\n        \n        # Add images\n        for i, row in df.iterrows():\n            self.add_image(\"fashion\", \n                           image_id=row.name, \n                           path=str(DATA_DIR/'train'/row.name) + '.jpg', \n                           labels=row['ClassId'],\n                           annotations=row['EncodedPixels'], \n                           height=row['Height'], width=row['Width'])\n            \n    def _resize_image(self, image_path):\n        img = cv2.imread(image_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (self.IMAGE_SIZE, self.IMAGE_SIZE), interpolation=cv2.INTER_AREA)  \n        return img\n        \n    def load_image(self, image_id):\n        return self._resize_image(self.image_info[image_id]['path'])\n    \n    def image_reference(self, image_id):\n        info = self.image_info[image_id]\n        return info['path'], [x for x in info['labels']]\n    \n    def load_mask(self, image_id):\n        info = self.image_info[image_id]\n                \n        mask = np.zeros((self.IMAGE_SIZE, self.IMAGE_SIZE, len(info['annotations'])), dtype=np.uint8)\n        labels = []\n        \n        for m, (annotation, label) in enumerate(zip(info['annotations'], info['labels'])):\n            sub_mask = np.full(info['height']*info['width'], 0, dtype=np.uint8)\n            annotation = [int(x) for x in annotation.split(' ')]\n            \n            for i, start_pixel in enumerate(annotation[::2]):\n                sub_mask[start_pixel: start_pixel+annotation[2*i+1]] = 1\n\n            sub_mask = sub_mask.reshape((info['height'], info['width']), order='F')\n            sub_mask = cv2.resize(sub_mask, (self.IMAGE_SIZE, self.IMAGE_SIZE), interpolation=cv2.INTER_NEAREST)\n            \n            mask[:, :, m] = sub_mask\n            labels.append(int(label)+1)\n            \n        return mask, np.array(labels)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Show Masks for testing\n\"\"\"\"\"\"\n\ndataset = FashionDataset(image_df)\ndataset.prepare()\n\nfor i in range(6):\n    image_id = random.choice(dataset.image_ids)\n\n    image = dataset.load_image(image_id)\n    mask, class_ids = dataset.load_mask(image_id)\n    visualize.display_top_masks(image, mask, class_ids, dataset.class_names, limit=4)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Training on a very small subset of the data for fast results for now.\n# image_df = image_df.iloc[0:2]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Split the trainig dataset in a train and a valid dataset. \n\"\"\"\"\"\"\n\n# This code partially supports k-fold training, \n# you can specify the fold to train and the total number of folds here\nFOLD = 0\nN_FOLDS = 2\n\nkf = KFold(n_splits=N_FOLDS, random_state=42, shuffle=True)\nsplits = kf.split(image_df) # ideally, this should be multilabel stratification\n\ndef get_fold():    \n    for i, (train_index, valid_index) in enumerate(splits):\n        if i == FOLD:\n            return image_df.iloc[train_index], image_df.iloc[valid_index]\n        \ntrain_df, valid_df = get_fold()\n\ntrain_dataset = FashionDataset(train_df)\ntrain_dataset.prepare()\n\nvalid_dataset = FashionDataset(valid_df)\nvalid_dataset.prepare()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Show shape of the datasets\n# Note: Both datasets have the same size\n\"\"\"\"\"\"\n\nprint(train_df.shape)\nprint(valid_df.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Config for Training\n\"\"\"\"\"\"\n\n# Note that any hyperparameters here, such as LR, may still not be optimal\nLR = 1e-4\nEPOCHS = [1, 6, 8]\n\nimport warnings \nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's load the COCO dataset weights to our Model.","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Create modell for training\n\"\"\"\"\"\"\n\nmodel = modellib.MaskRCNN(mode='training', config=config, model_dir=ROOT_DIR)\n\n# Load weights trained on MS COCO, but skip layers that\n# are different due to the different number of classes\n# See README for instructions to download the COCO weights\nmodel.load_weights(COCO_WEIGHTS_PATH, by_name=True,\n                   exclude=[\"mrcnn_class_logits\", \"mrcnn_bbox_fc\", \n                            \"mrcnn_bbox\", \"mrcnn_mask\"])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"augmentation = iaa.Sequential([\n    iaa.Fliplr(0.5) # only horizontal flip here\n])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Run training with 1 Epochs\n# Trainign time: ca 1,5h\n\"\"\"\"\"\"\n\n\nmodel.train(train_dataset, valid_dataset,\n            learning_rate=LR*2, # train heads with higher lr to speedup learning\n            epochs=EPOCHS[0],\n            layers='heads',\n            augmentation=None)\n\nhistory = model.keras_model.history.history","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# model.train(train_dataset, valid_dataset,\n#             learning_rate=LR,\n#             epochs=EPOCHS[1],\n#             layers='all',\n#             augmentation=augmentation)\n\n# new_history = model.keras_model.history.history\n# for k in new_history: history[k] = history[k] + new_history[k]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# model.train(train_dataset, valid_dataset,\n#             learning_rate=LR/5,\n#             epochs=EPOCHS[2],\n#             layers='all',\n#             augmentation=augmentation)\n\n# new_history = model.keras_model.history.history\n# for k in new_history: history[k] = history[k] + new_history[k]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epochs = range(EPOCHS[0])\n\nplt.figure(figsize=(18, 6))\n\nplt.subplot(131)\nplt.plot(epochs, history['loss'], label=\"train loss\")\nplt.plot(epochs, history['val_loss'], label=\"valid loss\")\nplt.legend()\nplt.subplot(132)\nplt.plot(epochs, history['mrcnn_class_loss'], label=\"train class loss\")\nplt.plot(epochs, history['val_mrcnn_class_loss'], label=\"valid class loss\")\nplt.legend()\nplt.subplot(133)\nplt.plot(epochs, history['mrcnn_mask_loss'], label=\"train mask loss\")\nplt.plot(epochs, history['val_mrcnn_mask_loss'], label=\"valid mask loss\")\nplt.legend()\n\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_epoch = np.argmin(history[\"val_loss\"]) + 1\nprint(\"Best epoch: \", best_epoch)\nprint(\"Valid loss: \", history[\"val_loss\"][best_epoch-1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import glob","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glob_list = glob.glob(f'/kaggle/working/class*/mask_rcnn_class_{best_epoch:04d}.h5')\nmodel_path = glob_list[0] if glob_list else ''","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = sample_sub_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Load the trained weights into the modell\n\"\"\"\"\"\"\n\nclass InferenceConfig(FashionConfig):\n    GPU_COUNT = 1\n    IMAGES_PER_GPU = 1\n\ninference_config = InferenceConfig()\n\nmodel = modellib.MaskRCNN(mode='inference', \n                          config=inference_config,\n                          model_dir=ROOT_DIR)\n\nassert model_path != '', \"Provide path to trained weights\"\nprint(\"Loading weights from \", model_path)\nmodel.load_weights(model_path, by_name=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert data to run-length encoding\ndef to_rle(bits):\n    rle = []\n    pos = 0\n    for bit, group in itertools.groupby(bits):\n        group_list = list(group)\n        if bit:\n            rle.extend([pos, sum(group_list)])\n        pos += len(group_list)\n    return rle","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since the submission system does not permit overlapped masks, we have to fix them\ndef refine_masks(masks, rois):\n    areas = np.sum(masks.reshape(-1, masks.shape[-1]), axis=0)\n    mask_index = np.argsort(areas)\n    union_mask = np.zeros(masks.shape[:-1], dtype=bool)\n    for m in mask_index:\n        masks[:, :, m] = np.logical_and(masks[:, :, m], np.logical_not(union_mask))\n        union_mask = np.logical_or(masks[:, :, m], union_mask)\n    for m in range(masks.shape[-1]):\n        mask_pos = np.where(masks[:, :, m]==True)\n        if np.any(mask_pos):\n            y1, x1 = np.min(mask_pos, axis=1)\n            y2, x2 = np.max(mask_pos, axis=1)\n            rois[m, :] = [y1, x1, y2, x2]\n    return masks, rois","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE = 256","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def resize_image(image_path):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_AREA)  \n    return img","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# import itertools to fix missing dependency bug in next cell\n\"\"\"\"\"\"\n\nimport itertools","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Iterate over the testing dataset and validate if the mask are recognized.\n\"\"\"\"\"\"\n\n\nsub_list = []\nmissing_count = 0\nfor i, row in tqdm(sample_df.iterrows(), total=len(sample_df)):\n    image = resize_image(str(DATA_DIR/'test'/row['ImageId']) + '.jpg')\n    result = model.detect([image])[0]\n    if result['masks'].size > 0:\n        masks, _ = refine_masks(result['masks'], result['rois'])\n        for m in range(masks.shape[-1]):\n            mask = masks[:, :, m].ravel(order='F')\n            rle = to_rle(mask)\n            label = result['class_ids'][m] - 1\n            sub_list.append([row['ImageId'], ' '.join(list(map(str, rle))), label, np.NaN])\n    else:\n        # The system does not allow missing ids, this is an easy way to fill them \n        sub_list.append([row['ImageId'], '1 1', 23, np.NaN])\n        missing_count += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub_df.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = pd.DataFrame(sub_list, columns=sample_df.columns.values)\nprint(\"Total image results: \", submission_df['ImageId'].nunique())\nprint(\"Missing Images: \", missing_count)\nsubmission_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\"\"\"\n# Show the test images with Mask, classification and bounding box\n\"\"\"\"\"\"\n\nfor i in range(9):\n    image_id = sample_df.sample()['ImageId'].values[0]  + '.jpg'\n    image_path = str(DATA_DIR/'test'/image_id)\n    \n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    \n    result = model.detect([resize_image(image_path)])\n    r = result[0]\n    \n    if r['masks'].size > 0:\n        masks = np.zeros((img.shape[0], img.shape[1], r['masks'].shape[-1]), dtype=np.uint8)\n        for m in range(r['masks'].shape[-1]):\n            masks[:, :, m] = cv2.resize(r['masks'][:, :, m].astype('uint8'), \n                                        (img.shape[1], img.shape[0]), interpolation=cv2.INTER_NEAREST)\n        \n        y_scale = img.shape[0]/IMAGE_SIZE\n        x_scale = img.shape[1]/IMAGE_SIZE\n        rois = (r['rois'] * [y_scale, x_scale, y_scale, x_scale]).astype(int)\n        \n        masks, rois = refine_masks(masks, rois)\n    else:\n        masks, rois = r['masks'], r['rois']\n        \n    visualize.display_instances(img, rois, masks, r['class_ids'], \n                                valid_dataset.class_names, r['scores'],\n                                title=image_id, figsize=(12, 12))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# In Progress... Stay Tuned!","metadata":{}}]}