{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Trabajo Práctico Visión por computadora","metadata":{}},{"cell_type":"markdown","source":"### Dataset description\nIn this dataset, we are provided a large number of images and corresponding fashion/apparel segmentations. Images are named with a unique ImageId. You must segment and classify the images in the test set. This dataset contains images of people wearing a variety of clothing types in a variety of poses. \n\n#### Files\n* train/ - The training images\n* test/ - The test images (you are segmenting and classifying these images)\n* train.csv - Training annotations, contains images with both segmented apparel categories and fine-grained attributes; and images with segmented apparel categories only.\n* label_descriptions.json - A file giving the apparel categories and fine-grained attributes descriptions.\n* sample_submission.csv - A sample submission file in the correct format.\n\n#### Columns\n* ImageId - the unique Id of an image\n* EncodedPixels - masks in run-length encoded format (please refer to evaluation page for details).\n* ClassId - the class id for this mask. It represents the apparel category.\n* AttributesIds - the attributes ids for this mask. We concatenate all the attributes (if any) together.","metadata":{}},{"cell_type":"markdown","source":"#### Import libraries","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport json\nimport os\nimport random\nimport cv2\n\nprint(f\"pandas version: {pd.__version__}\")\nprint(f\"numpy version: {np.__version__}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:12:24.338610Z","iopub.execute_input":"2021-12-23T22:12:24.339333Z","iopub.status.idle":"2021-12-23T22:12:24.604349Z","shell.execute_reply.started":"2021-12-23T22:12:24.339227Z","shell.execute_reply":"2021-12-23T22:12:24.602491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explorary data analysis","metadata":{}},{"cell_type":"markdown","source":"### Upload datasets","metadata":{}},{"cell_type":"code","source":"# Training dataset\ntrain_df = pd.read_csv('/kaggle/input/imaterialist-fashion-2020-fgvc7/train.csv')\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:13:50.488462Z","iopub.execute_input":"2021-12-23T22:13:50.489055Z","iopub.status.idle":"2021-12-23T22:14:19.400293Z","shell.execute_reply.started":"2021-12-23T22:13:50.489018Z","shell.execute_reply":"2021-12-23T22:14:19.399210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dataset details","metadata":{}},{"cell_type":"markdown","source":"#### Datasets size","metadata":{}},{"cell_type":"code","source":"# Get datasets shapes\nprint(f'Training dataset shape: {train_df.shape}')\nprint(f'Unique images training: {train_df[\"ImageId\"].nunique()}')","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:19.402731Z","iopub.execute_input":"2021-12-23T22:14:19.403367Z","iopub.status.idle":"2021-12-23T22:14:19.546624Z","shell.execute_reply.started":"2021-12-23T22:14:19.403324Z","shell.execute_reply":"2021-12-23T22:14:19.545300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Size distribution","metadata":{}},{"cell_type":"code","source":"# Get image size distribution\nshape_df = train_df.groupby(\"ImageId\")[[\"Height\", \"Width\"]].first()\nfor dim in [\"Height\", \"Width\"]:\n    plt.figure()\n    plt.hist(shape_df[dim], bins=50)\n    plt.grid()\n    plt.title(f\"{dim} distribution\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:19.548551Z","iopub.execute_input":"2021-12-23T22:14:19.549172Z","iopub.status.idle":"2021-12-23T22:14:20.604617Z","shell.execute_reply.started":"2021-12-23T22:14:19.549128Z","shell.execute_reply":"2021-12-23T22:14:20.603603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Classes per image distribution","metadata":{}},{"cell_type":"code","source":"# Get classes per image distribution\nplt.hist(train_df.groupby(['ImageId'], as_index=False).count()[['ClassId']], bins=30)\nplt.grid()\nplt.title(\"Clases per image distribution distribution\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-23T22:14:20.607818Z","iopub.execute_input":"2021-12-23T22:14:20.608855Z","iopub.status.idle":"2021-12-23T22:14:21.153602Z","shell.execute_reply.started":"2021-12-23T22:14:20.608806Z","shell.execute_reply":"2021-12-23T22:14:21.151116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plotting random images","metadata":{}},{"cell_type":"code","source":"# Plot randomly selected image\nplt.figure(figsize=(70,7))\nrandom_image = train_df.sample()[\"ImageId\"].item()\nplt.imshow(mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{random_image}.jpg'))\nplt.grid(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.155396Z","iopub.execute_input":"2021-12-23T22:14:21.156830Z","iopub.status.idle":"2021-12-23T22:14:21.607611Z","shell.execute_reply.started":"2021-12-23T22:14:21.156773Z","shell.execute_reply":"2021-12-23T22:14:21.606602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Label description analysis","metadata":{}},{"cell_type":"code","source":"# Get label file\nwith open('/kaggle/input/imaterialist-fashion-2020-fgvc7/label_descriptions.json', 'r') as file:\n    label_d = json.load(file)\n\nprint(\"Label description columns {}\".format(list(label_d.keys())))","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.608917Z","iopub.execute_input":"2021-12-23T22:14:21.609246Z","iopub.status.idle":"2021-12-23T22:14:21.625189Z","shell.execute_reply.started":"2021-12-23T22:14:21.609173Z","shell.execute_reply":"2021-12-23T22:14:21.624063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Separate label description into categories and attributes\ncategories_df = pd.DataFrame(label_d['categories'])\nattributes_df = pd.DataFrame(label_d['attributes'])","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.626705Z","iopub.execute_input":"2021-12-23T22:14:21.627222Z","iopub.status.idle":"2021-12-23T22:14:21.636202Z","shell.execute_reply.started":"2021-12-23T22:14:21.627176Z","shell.execute_reply":"2021-12-23T22:14:21.635151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Categories\ncategories_df","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.638159Z","iopub.execute_input":"2021-12-23T22:14:21.638829Z","iopub.status.idle":"2021-12-23T22:14:21.658822Z","shell.execute_reply.started":"2021-12-23T22:14:21.638781Z","shell.execute_reply":"2021-12-23T22:14:21.657696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categ_names = categories_df[\"name\"].unique()\nprint(categ_names)\nprint(f\"Number of attributes {len(categ_names)}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.660753Z","iopub.execute_input":"2021-12-23T22:14:21.661111Z","iopub.status.idle":"2021-12-23T22:14:21.673105Z","shell.execute_reply.started":"2021-12-23T22:14:21.661042Z","shell.execute_reply":"2021-12-23T22:14:21.671724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Attributes\npd.set_option('display.max_rows', 500)\nattributes_df","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.677657Z","iopub.execute_input":"2021-12-23T22:14:21.678087Z","iopub.status.idle":"2021-12-23T22:14:21.727299Z","shell.execute_reply.started":"2021-12-23T22:14:21.678053Z","shell.execute_reply":"2021-12-23T22:14:21.726325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"attr_names = attributes_df[\"name\"].unique()\nprint(attr_names)\nprint(f\"Number of attributes {len(attr_names)}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.728718Z","iopub.execute_input":"2021-12-23T22:14:21.729557Z","iopub.status.idle":"2021-12-23T22:14:21.739366Z","shell.execute_reply.started":"2021-12-23T22:14:21.729506Z","shell.execute_reply":"2021-12-23T22:14:21.738158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create dictionaries to map the IDs with the category and attributes strings\ncat_map = {category[\"id\"]: category[\"name\"] for category in label_d['categories']}\ncat_map_inv = {category[\"name\"]: category[\"id\"] for category in label_d['categories']}\n\nattr_map = {category[\"id\"]: category[\"name\"] for category in label_d['attributes']}\nattr_map_inv = {category[\"name\"]: category[\"id\"] for category in label_d['attributes']}","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.740788Z","iopub.execute_input":"2021-12-23T22:14:21.741882Z","iopub.status.idle":"2021-12-23T22:14:21.750278Z","shell.execute_reply.started":"2021-12-23T22:14:21.741821Z","shell.execute_reply":"2021-12-23T22:14:21.749355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plot segmented images","metadata":{}},{"cell_type":"code","source":"def plot_raw_segmented_image(df, figsize=(15,15)):\n    # Read random image\n    random_id = df.sample()[\"ImageId\"].item()\n    image = mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{random_id}.jpg')\n    shape = image.shape,\n    encoded_pixels = df[train_df['ImageId'] == random_id]['EncodedPixels']\n    class_ids = df[train_df['ImageId'] == random_id]['ClassId']\n    \n    # Create mask\n    height, width = shape[0][:2]\n    mask = np.zeros((height, width)).reshape(-1)\n    for pixels, class_id in zip(encoded_pixels, class_ids):\n        pixels_split = list(map(int, pixels.split()))\n        pixel_starts = pixels_split[::2]\n        run_lengths = pixels_split[1::2]\n        for pixel_start, run_length in zip(pixel_starts, run_lengths):\n            mask[pixel_start:pixel_start + run_length] = 255 - class_id * 4\n    mask = mask.reshape(height, width, order='F')    \n    \n    # Plot images\n    fig, axs = plt.subplots(1, 2,figsize=(15,15))\n    axs[0].imshow(image)    \n    axs[1].imshow(image)    \n    axs[1].imshow(mask, alpha=0.8)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.752913Z","iopub.execute_input":"2021-12-23T22:14:21.753751Z","iopub.status.idle":"2021-12-23T22:14:21.767886Z","shell.execute_reply.started":"2021-12-23T22:14:21.753705Z","shell.execute_reply":"2021-12-23T22:14:21.766632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot raw and segmented images\nsize = 3\nfor _ in range(size):\n    plot_raw_segmented_image(train_df, \"ImageId\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:21.769445Z","iopub.execute_input":"2021-12-23T22:14:21.770690Z","iopub.status.idle":"2021-12-23T22:14:25.748225Z","shell.execute_reply.started":"2021-12-23T22:14:21.770612Z","shell.execute_reply":"2021-12-23T22:14:25.746102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_classes_image(df, figsize=(15,15)):\n    # Select random image\n    random_id = df.sample()[\"ImageId\"].item()\n    image = mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{random_id}.jpg')\n    shape = image.shape,\n    encoded_pixels = df[train_df['ImageId'] == random_id]['EncodedPixels']\n    class_ids = df[train_df['ImageId'] == random_id]['ClassId']\n    \n    # Create mask and plot every specific class in the image\n    height, width = shape[0][:2]\n    for pixels, class_id in zip(encoded_pixels, class_ids):\n        mask = np.zeros((height, width)).reshape(-1)\n        pixels_split = list(map(int, pixels.split()))\n        pixel_starts = pixels_split[::2]\n        run_lengths = pixels_split[1::2]\n        for pixel_start, run_length in zip(pixel_starts, run_lengths):\n            mask[pixel_start:pixel_start + run_length] = 255 - class_id * 4\n        mask = mask.reshape(height, width, order='F')\n        \n        # Plot masked image\n        plt.figure(figsize=(15, 15))\n        plt.title(cat_map[class_id])\n        plt.imshow(image)    \n        plt.imshow(mask, alpha=0.8)\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:25.750322Z","iopub.execute_input":"2021-12-23T22:14:25.750929Z","iopub.status.idle":"2021-12-23T22:14:25.766569Z","shell.execute_reply.started":"2021-12-23T22:14:25.750886Z","shell.execute_reply":"2021-12-23T22:14:25.765516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_classes_image(train_df)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:25.768513Z","iopub.execute_input":"2021-12-23T22:14:25.769097Z","iopub.status.idle":"2021-12-23T22:14:47.309932Z","shell.execute_reply.started":"2021-12-23T22:14:25.769037Z","shell.execute_reply":"2021-12-23T22:14:47.309111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Replace ClassId value","metadata":{}},{"cell_type":"code","source":"# Replace ClassId for class string \ntrain_df['ClassId'] = train_df['ClassId'].map(cat_map)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:47.311392Z","iopub.execute_input":"2021-12-23T22:14:47.311851Z","iopub.status.idle":"2021-12-23T22:14:47.343634Z","shell.execute_reply.started":"2021-12-23T22:14:47.311812Z","shell.execute_reply":"2021-12-23T22:14:47.342409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot class value count\ntrain_df['ClassId'].value_counts()[:20].plot(kind='barh')\nplt.grid()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:47.345896Z","iopub.execute_input":"2021-12-23T22:14:47.346263Z","iopub.status.idle":"2021-12-23T22:14:47.767391Z","shell.execute_reply.started":"2021-12-23T22:14:47.346219Z","shell.execute_reply":"2021-12-23T22:14:47.766408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform ClassId back to int to perform the training\ntrain_df['ClassId'] = train_df['ClassId'].map(cat_map_inv)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:47.770426Z","iopub.execute_input":"2021-12-23T22:14:47.771265Z","iopub.status.idle":"2021-12-23T22:14:47.816307Z","shell.execute_reply.started":"2021-12-23T22:14:47.771218Z","shell.execute_reply":"2021-12-23T22:14:47.815297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Detectron","metadata":{}},{"cell_type":"markdown","source":"#### Install dependencies","metadata":{}},{"cell_type":"code","source":"!pip install -q cython pyyaml","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:47.818016Z","iopub.execute_input":"2021-12-23T22:14:47.818613Z","iopub.status.idle":"2021-12-23T22:14:59.502825Z","shell.execute_reply.started":"2021-12-23T22:14:47.818536Z","shell.execute_reply":"2021-12-23T22:14:59.501677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install pycocotools==2.0.2","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:14:59.506476Z","iopub.execute_input":"2021-12-23T22:14:59.507225Z","iopub.status.idle":"2021-12-23T22:15:20.947538Z","shell.execute_reply.started":"2021-12-23T22:14:59.507146Z","shell.execute_reply":"2021-12-23T22:15:20.946435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip install 'git+https://github.com/facebookresearch/detectron2.git'","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:15:20.950637Z","iopub.execute_input":"2021-12-23T22:15:20.950999Z","iopub.status.idle":"2021-12-23T22:19:13.960486Z","shell.execute_reply.started":"2021-12-23T22:15:20.950948Z","shell.execute_reply":"2021-12-23T22:19:13.959379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Import libraries","metadata":{}},{"cell_type":"code","source":"from detectron2 import model_zoo\nfrom detectron2.engine import DefaultPredictor\nfrom detectron2.config import get_cfg\nfrom detectron2.utils.visualizer import Visualizer\nfrom detectron2.data import MetadataCatalog, DatasetCatalog\nfrom detectron2.structures import BoxMode","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:13.967737Z","iopub.execute_input":"2021-12-23T22:19:13.968104Z","iopub.status.idle":"2021-12-23T22:19:15.285044Z","shell.execute_reply.started":"2021-12-23T22:19:13.968070Z","shell.execute_reply":"2021-12-23T22:19:15.281718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Use detectron in current dataset","metadata":{}},{"cell_type":"code","source":"# Use detectron to predict images in the current dataset,\n# use the default weights and labels from the network\nconfig_file = \"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\"\ncfg = get_cfg()\ncfg.merge_from_file(model_zoo.get_config_file(config_file))\ncfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5\ncfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(config_file)\n\npredictor = DefaultPredictor(cfg)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:15.290235Z","iopub.execute_input":"2021-12-23T22:19:15.291149Z","iopub.status.idle":"2021-12-23T22:19:38.132366Z","shell.execute_reply.started":"2021-12-23T22:19:15.291066Z","shell.execute_reply":"2021-12-23T22:19:38.131309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot images classified by the pretrained detectron\nrows, cols = 2, 2\nplt.figure(figsize=(20, 20))\n\nfor i in range(int(rows * cols)):\n    plt.subplot(rows, cols, i + 1)\n    \n    # Get random image\n    random_id = train_df.sample()[\"ImageId\"].item()\n    im = mpimg.imread(f'/kaggle/input/imaterialist-fashion-2020-fgvc7/train/{random_id}.jpg')\n    height, width = im.shape[:2]\n    \n    # Get detectron prediction from the selected image\n    outputs = predictor(im)\n    \n    # Create visualizer\n    visualizer = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=0.4)\n    \n    # Change font size for better reading\n    visualizer._default_font_size = np.sqrt(height * width) // 20\n    visualizer = visualizer.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    \n    # Plot images\n    plt.axis('off')\n    plt.imshow(visualizer.get_image()[:, :, ::-1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:38.134254Z","iopub.execute_input":"2021-12-23T22:19:38.134703Z","iopub.status.idle":"2021-12-23T22:19:50.650435Z","shell.execute_reply.started":"2021-12-23T22:19:38.134629Z","shell.execute_reply":"2021-12-23T22:19:50.649527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Modify dataset to use detectron","metadata":{}},{"cell_type":"code","source":"\ndef rle_decode_string(string, h, w):\n    \"\"\"\n    Transforms rle string into a pixel mask\n    \n    :param string: rle string to transform into mask\n    :type string: str\n    :param string: image height\n    :type string: int\n    :param string: image width\n    :type string: int\n    :return: image mask\n    :rtype: numpy array\n\n    \"\"\"\n    mask = np.full(h * w, 0, dtype=np.uint8)\n    annotation = [int(x) for x in string.split(' ')]\n    for i, start_pixel in enumerate(annotation[::2]):\n        mask[start_pixel: start_pixel + annotation[2 * i + 1]] = 1\n    mask = mask.reshape((h, w), order='F')\n    return mask\n\ndef rle2bbox(rle, shape):\n    '''\n    Get a bbox from a mask which is required for Detectron 2 dataset\n    :param rle: run-length encoded image mask, as string\n    :type rle: str\n    :param shape: (height, width) of image on which RLE was produced\n    :type rle: tuple\n    :return: (x0, y0, x1, y1) tuple describing the bounding box of the rle mask\n    :rtype: tuple\n    '''\n    \n    a = np.fromiter(rle.split(), dtype=np.uint)\n    a = a.reshape((-1, 2))  # an array of (start, length) pairs\n    a[:,0] -= 1  # `start` is 1-indexed\n    \n    y0 = a[:,0] % shape[0]\n    y1 = y0 + a[:,1]\n    if np.any(y1 > shape[0]):\n        # got `y` overrun, meaning that there are a pixels in mask on 0 and shape[0] position\n        y0 = 0\n        y1 = shape[0]\n    else:\n        y0 = np.min(y0)\n        y1 = np.max(y1)\n    \n    x0 = a[:,0] // shape[0]\n    x1 = (a[:,0] + a[:,1]) // shape[0]\n    x0 = np.min(x0)\n    x1 = np.max(x1)\n    \n    if x1 > shape[1]:\n        # just went out of the image dimensions\n        raise ValueError(\"invalid RLE or image dimensions: x1=%d > shape[1]=%d\" % (\n            x1, shape[1]\n        ))\n\n    return x0, y0, x1, y1","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:50.651875Z","iopub.execute_input":"2021-12-23T22:19:50.652920Z","iopub.status.idle":"2021-12-23T22:19:50.672320Z","shell.execute_reply.started":"2021-12-23T22:19:50.652878Z","shell.execute_reply":"2021-12-23T22:19:50.670634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform ImageId into image path\nimage_dir = '/kaggle/input/imaterialist-fashion-2020-fgvc7/train/'\ntrain_df['ImageId'] = image_dir + train_df['ImageId'] + '.jpg'\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:50.673901Z","iopub.execute_input":"2021-12-23T22:19:50.674723Z","iopub.status.idle":"2021-12-23T22:19:50.942627Z","shell.execute_reply.started":"2021-12-23T22:19:50.674669Z","shell.execute_reply":"2021-12-23T22:19:50.941666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create boxes list\nbboxes = [rle2bbox(c.EncodedPixels, (c.Height, c.Width)) for n, c in train_df.iterrows()]\nbboxes_array = np.array(bboxes)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:19:50.945325Z","iopub.execute_input":"2021-12-23T22:19:50.946068Z","iopub.status.idle":"2021-12-23T22:22:07.989219Z","shell.execute_reply.started":"2021-12-23T22:19:50.946018Z","shell.execute_reply":"2021-12-23T22:22:07.988125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill NaNs\ntrain_df = train_df.fillna(999)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:07.991262Z","iopub.execute_input":"2021-12-23T22:22:07.991553Z","iopub.status.idle":"2021-12-23T22:22:08.146535Z","shell.execute_reply.started":"2021-12-23T22:22:07.991509Z","shell.execute_reply":"2021-12-23T22:22:08.145448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add bounding boxes coordinates to train using detectron\ntrain_df['x0'], train_df['y0'], train_df['x1'], train_df['y1'] = bboxes_array[:,0], bboxes_array[:,1], bboxes_array[:,2], bboxes_array[:,3]\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:08.148485Z","iopub.execute_input":"2021-12-23T22:22:08.148860Z","iopub.status.idle":"2021-12-23T22:22:08.175786Z","shell.execute_reply.started":"2021-12-23T22:22:08.148804Z","shell.execute_reply":"2021-12-23T22:22:08.174783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def transform_to_array(value):\n    if isinstance(value, (np.ndarray, np.generic)):\n        return value\n    elif isinstance(value, str):\n        array = [int(val) for val in value.split(\",\")]\n    elif isinstance(value, int):\n        array = [999] \n    array = np.array(array)\n    return np.pad(array, (0, 14 - len(array)))","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:08.177644Z","iopub.execute_input":"2021-12-23T22:22:08.178336Z","iopub.status.idle":"2021-12-23T22:22:08.186770Z","shell.execute_reply.started":"2021-12-23T22:22:08.178289Z","shell.execute_reply":"2021-12-23T22:22:08.185534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Transform attribute string into tensor\ntrain_df[\"AttributesIds\"] = train_df[\"AttributesIds\"].map(transform_to_array)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:08.188654Z","iopub.execute_input":"2021-12-23T22:22:08.189082Z","iopub.status.idle":"2021-12-23T22:22:24.541340Z","shell.execute_reply.started":"2021-12-23T22:22:08.189035Z","shell.execute_reply":"2021-12-23T22:22:24.540334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Store modified train_df\ntrain_df.to_pickle(\"train_df.pickle\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:24.542852Z","iopub.execute_input":"2021-12-23T22:22:24.543714Z","iopub.status.idle":"2021-12-23T22:22:29.530858Z","shell.execute_reply.started":"2021-12-23T22:22:24.543657Z","shell.execute_reply":"2021-12-23T22:22:29.529814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train_df))","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:22:29.532549Z","iopub.execute_input":"2021-12-23T22:22:29.532894Z","iopub.status.idle":"2021-12-23T22:22:29.540205Z","shell.execute_reply.started":"2021-12-23T22:22:29.532852Z","shell.execute_reply":"2021-12-23T22:22:29.539035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pycocotools\ndef get_materialist_dicts(df):\n    \"\"\"\n    Transforms dataframe into dictionary used to train using detectron\n    \"\"\"\n    dataset_dicts = []\n    for idx, filename in enumerate(df[\"ImageId\"].unique()):\n        record = {}\n        # Get useful image information\n        height, width = df[df[\"ImageId\"] == filename][[\"Height\", \"Width\"]].values[0]\n        record[\"file_name\"] = filename\n        record[\"image_id\"] = idx\n        record[\"height\"] = int(height)\n        record[\"width\"] = int(width)\n        \n        if idx % 1000 == 0:\n            print(idx)\n        \n        objs = []\n        for i, row in df[(df['ImageId'] == filename)].iterrows():\n            \n            # Get segmentation polygons\n            mask = rle_decode_string(row['EncodedPixels'], row['Height'], row['Width'])\n            # segmentation = pycocotools.mask.encode(np.asarray(mask, order=\"F\"))\n            contours, hierarchy = cv2.findContours((mask).astype(np.uint8), cv2.RETR_TREE,\n                                                    cv2.CHAIN_APPROX_SIMPLE)\n            segmentation = []\n\n            for contour in contours:\n                contour = contour.flatten().tolist()\n                if len(contour) > 4:\n                    segmentation.append(contour)\n\n            obj = {\n                \"bbox\": [row['x0'], row['y0'], row['x1'], row['y1']],\n                \"bbox_mode\": BoxMode.XYXY_ABS,\n                \"segmentation\": segmentation,\n                \"category_id\": row['ClassId'],\n                \"attributes\": row['AttributesIds'],\n                \"iscrowd\": 0,\n            }\n            objs.append(obj)\n        \n        record['annotations'] = objs\n        dataset_dicts.append(record)\n    return dataset_dicts\n\n# Use reduced dictionary to reduce the time to transform into dictionaries\n#df_copy = train_df[:8000].copy()\n\ndf_copy = train_df.copy()\n\ndf_copy = train_df[:23000].copy()\ndf_copy_val = train_df[23000:24000].copy()\n\n# Full dictionary\n# df_copy = train_df.copy()\n\nmaterialist_dict = get_materialist_dicts(df_copy)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-12-23T22:22:29.542385Z","iopub.execute_input":"2021-12-23T22:22:29.543249Z","iopub.status.idle":"2021-12-23T22:31:31.572229Z","shell.execute_reply.started":"2021-12-23T22:22:29.543129Z","shell.execute_reply":"2021-12-23T22:31:31.571202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"materialist_dict[0].keys()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:31:31.573971Z","iopub.execute_input":"2021-12-23T22:31:31.574313Z","iopub.status.idle":"2021-12-23T22:31:31.582866Z","shell.execute_reply.started":"2021-12-23T22:31:31.574250Z","shell.execute_reply":"2021-12-23T22:31:31.581622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(df_copy))\nprint(len(materialist_dict))\nprint(len(train_df))\nprint(len(train_df[\"ImageId\"].unique()))\nprint(len(df_copy[\"ImageId\"].unique()))","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:46:16.410741Z","iopub.execute_input":"2021-12-24T00:46:16.411050Z","iopub.status.idle":"2021-12-24T00:46:16.546083Z","shell.execute_reply.started":"2021-12-24T00:46:16.411016Z","shell.execute_reply":"2021-12-24T00:46:16.544841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Register the custom dataset to detectron2,\nfor d in [\"train\", \"val\"]:\n    if d == \"train\":\n        used_df = df_copy\n    else:\n        used_df = df_copy_val\n    DatasetCatalog.register(\"mat_\" + d, lambda df=used_df: get_materialist_dicts(df))\n    # DatasetCatalog.register(\"mat_\" + d, lambda df=df_copy: get_materialist_dicts(df))\n    MetadataCatalog.get(\"mat_\" + d).set(thing_classes=list(categories_df.name))\nmaterialist_metadata = MetadataCatalog.get(\"mat_train\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:31:31.584541Z","iopub.execute_input":"2021-12-23T22:31:31.585141Z","iopub.status.idle":"2021-12-23T22:31:31.599509Z","shell.execute_reply.started":"2021-12-23T22:31:31.585079Z","shell.execute_reply":"2021-12-23T22:31:31.598439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To verify the data loading is correct we visualize the annotations of randomly selected samples in the training set\nfor d in random.sample(materialist_dict, 5):\n    img = cv2.imread(d[\"file_name\"])\n    img = mpimg.imread(d[\"file_name\"])\n    height, width = img.shape[:2]\n    plt.figure(figsize=(20, 20))\n    visualizer = Visualizer(img[:, :, ::-1], metadata=materialist_metadata, scale=0.5)\n    visualizer._default_font_size = np.sqrt(height * width) // 20\n    out = visualizer.draw_dataset_dict(d)\n    plt.imshow(out.get_image()[:, :, ::-1])\n    plt.axis('off')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:31:31.601455Z","iopub.execute_input":"2021-12-23T22:31:31.601934Z","iopub.status.idle":"2021-12-23T22:31:41.301132Z","shell.execute_reply.started":"2021-12-23T22:31:31.601885Z","shell.execute_reply":"2021-12-23T22:31:41.300126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## FPN","metadata":{}},{"cell_type":"code","source":"from detectron2.engine import DefaultTrainer\nos.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\n\n# Fine-tune a COCO-pretrained R50-FPN Mask R-CNN model on the dataset\ncfg_FPN = get_cfg()\ncfg_FPN.merge_from_file(model_zoo.get_config_file(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\"))\ncfg_FPN.DATASETS.TRAIN = (\"mat_train\",)\ncfg_FPN.DATASETS.TEST = (\"mat_val\",)\ncfg_FPN.DATALOADER.NUM_WORKERS = 1\ncfg_FPN.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml\")\ncfg_FPN.SOLVER.IMS_PER_BATCH = 2\ncfg_FPN.SOLVER.BASE_LR = 0.00025 \ncfg_FPN.SOLVER.MAX_ITER = 1000  \ncfg_FPN.SOLVER.STEPS = []       \ncfg_FPN.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  \ncfg_FPN.MODEL.ROI_HEADS.NUM_CLASSES = 46 \n\n# Train\ncfg_FPN.OUTPUT_DIR = \"./output_FPN\"\nos.makedirs(cfg_FPN.OUTPUT_DIR, exist_ok=True)\ntrainer_FPN = DefaultTrainer(cfg_FPN) \ntrainer_FPN.resume_or_load(resume=False)\ntrainer_FPN.train()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:31:41.303140Z","iopub.execute_input":"2021-12-23T22:31:41.303914Z","iopub.status.idle":"2021-12-23T22:54:15.257205Z","shell.execute_reply.started":"2021-12-23T22:31:41.303851Z","shell.execute_reply":"2021-12-23T22:54:15.255976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create predictor from the weigths obtained during the training\ncfg_FPN.MODEL.WEIGHTS = os.path.join(cfg_FPN.OUTPUT_DIR, \"model_final.pth\")\ncfg_FPN.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5   # set the testing threshold for this model\ncfg_FPN.DATASETS.TEST = ('mat_val',)\npredictor_FPN = DefaultPredictor(cfg_FPN)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:54:15.260951Z","iopub.execute_input":"2021-12-23T22:54:15.261430Z","iopub.status.idle":"2021-12-23T22:54:16.421437Z","shell.execute_reply.started":"2021-12-23T22:54:15.261356Z","shell.execute_reply":"2021-12-23T22:54:16.420015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from detectron2.utils.visualizer import ColorMode\nplt.figure(figsize=(20,20))\nfor d in random.sample(materialist_dict, 3):    \n    im = cv2.imread(d[\"file_name\"])\n    outputs = predictor_FPN(im)\n    visualizer = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = visualizer.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:54:16.433626Z","iopub.execute_input":"2021-12-23T22:54:16.434042Z","iopub.status.idle":"2021-12-23T22:54:35.434800Z","shell.execute_reply.started":"2021-12-23T22:54:16.433989Z","shell.execute_reply":"2021-12-23T22:54:35.433780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from detectron2.utils.visualizer import ColorMode\n\n# Show different images at random\nrows, cols = 3, 3\nplt.figure(figsize=(20,20))\n\nfor i, d in enumerate(random.sample(materialist_dict, 9)):\n    # Process image\n    plt.subplot(rows, cols, i+1)\n\n    im = cv2.imread(d[\"file_name\"])\n    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n    \n    # Run through predictor\n    outputs = predictor_FPN(im)\n    \n    # Visualize\n    v = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T22:54:35.436173Z","iopub.execute_input":"2021-12-23T22:54:35.436589Z","iopub.status.idle":"2021-12-23T22:54:46.029861Z","shell.execute_reply.started":"2021-12-23T22:54:35.436548Z","shell.execute_reply":"2021-12-23T22:54:46.028825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from detectron2.evaluation import COCOEvaluator, inference_on_dataset\nfrom detectron2.data import build_detection_test_loader\n\n# Evaluate model\nevaluator_FPN = COCOEvaluator(\"mat_val\", output_dir=\"./output\")\nval_loader_FPN = build_detection_test_loader(cfg_FPN, \"mat_val\")","metadata":{"execution":{"iopub.status.busy":"2021-12-23T23:11:10.764168Z","iopub.execute_input":"2021-12-23T23:11:10.764486Z","iopub.status.idle":"2021-12-23T23:11:50.268198Z","shell.execute_reply.started":"2021-12-23T23:11:10.764452Z","shell.execute_reply":"2021-12-23T23:11:50.266984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get results\nresult_FPN = inference_on_dataset(predictor_FPN.model, val_loader_FPN, evaluator_FPN)","metadata":{"execution":{"iopub.status.busy":"2021-12-23T23:13:08.788847Z","iopub.execute_input":"2021-12-23T23:13:08.789726Z","iopub.status.idle":"2021-12-23T23:14:32.356892Z","shell.execute_reply.started":"2021-12-23T23:13:08.789690Z","shell.execute_reply":"2021-12-23T23:14:32.355689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_FPN","metadata":{"execution":{"iopub.status.busy":"2021-12-23T23:14:38.607251Z","iopub.execute_input":"2021-12-23T23:14:38.607627Z","iopub.status.idle":"2021-12-23T23:14:38.620072Z","shell.execute_reply.started":"2021-12-23T23:14:38.607589Z","shell.execute_reply":"2021-12-23T23:14:38.618907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DC5","metadata":{}},{"cell_type":"code","source":"from detectron2.engine import DefaultTrainer\nos.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\n\n# Fine-tune a COCO-pretrained R50-DC5 Mask R-CNN model on the dataset\ncfg_DC5 = get_cfg()\ncfg_DC5.merge_from_file(model_zoo.get_config_file(\"COCO-InstanceSegmentation/mask_rcnn_R_50_DC5_3x.yaml\"))\ncfg_DC5.DATASETS.TRAIN = (\"mat_train\",)\ncfg_DC5.DATASETS.TEST = ()\ncfg_DC5.DATALOADER.NUM_WORKERS = 1\ncfg_DC5.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_50_DC5_3x.yaml\")\ncfg_DC5.SOLVER.IMS_PER_BATCH = 2\ncfg_DC5.SOLVER.BASE_LR = 0.00025  \ncfg_DC5.SOLVER.MAX_ITER = 1000 \ncfg_DC5.SOLVER.STEPS = []    \ncfg_DC5.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  \ncfg_DC5.MODEL.ROI_HEADS.NUM_CLASSES = 46  \n\n# Train\ncfg_DC5.OUTPUT_DIR = \"./output_DC5\"\nos.makedirs(cfg_DC5.OUTPUT_DIR, exist_ok=True)\ntrainer_DC5 = DefaultTrainer(cfg_DC5) \ntrainer_DC5.resume_or_load(resume=False)\ntrainer_DC5.train()","metadata":{"execution":{"iopub.status.busy":"2021-12-23T23:39:24.778815Z","iopub.execute_input":"2021-12-23T23:39:24.779807Z","iopub.status.idle":"2021-12-24T00:04:24.492906Z","shell.execute_reply.started":"2021-12-23T23:39:24.779769Z","shell.execute_reply":"2021-12-24T00:04:24.491492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cfg_DC5.MODEL.WEIGHTS = os.path.join(cfg_DC5.OUTPUT_DIR, \"model_final.pth\")\ncfg_DC5.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5   # set the testing threshold for this model\ncfg_DC5.DATASETS.TEST = ('mat_val')\npredictor_DC5 = DefaultPredictor(cfg_DC5)","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:04:24.497146Z","iopub.execute_input":"2021-12-24T00:04:24.497526Z","iopub.status.idle":"2021-12-24T00:04:28.719217Z","shell.execute_reply.started":"2021-12-24T00:04:24.497467Z","shell.execute_reply":"2021-12-24T00:04:28.718165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from detectron2.utils.visualizer import ColorMode\nplt.figure(figsize=(20,20))\nfor d in random.sample(materialist_dict, 3):    \n    im = cv2.imread(d[\"file_name\"])\n    outputs = predictor_DC5(im)\n    visualizer = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = visualizer.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:06:28.006012Z","iopub.execute_input":"2021-12-24T00:06:28.006340Z","iopub.status.idle":"2021-12-24T00:06:30.768713Z","shell.execute_reply.started":"2021-12-24T00:06:28.006310Z","shell.execute_reply":"2021-12-24T00:06:30.766521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show different images at random\nrows, cols = 3, 3\nplt.figure(figsize=(20,20))\nfor i, d in enumerate(random.sample(materialist_dict, 9)):\n    # Process image\n    plt.subplot(rows, cols, i+1)\n    im = cv2.imread(d[\"file_name\"])\n    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n    \n    # Run through predictor\n    outputs = predictor_DC5(im)\n    \n    # Visualize\n    v = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:06:41.051797Z","iopub.execute_input":"2021-12-24T00:06:41.052131Z","iopub.status.idle":"2021-12-24T00:07:11.436482Z","shell.execute_reply.started":"2021-12-24T00:06:41.052097Z","shell.execute_reply":"2021-12-24T00:07:11.435495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate model\nevaluator_DC5 = COCOEvaluator(\"mat_val\", output_dir=cfg_DC5.OUTPUT_DIR)\nval_loader_DC5 = build_detection_test_loader(cfg_DC5, \"mat_val\")\nresult_DC5 = inference_on_dataset(predictor_DC5.model, val_loader_DC5, evaluator_DC5)","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:04:28.720706Z","iopub.execute_input":"2021-12-24T00:04:28.721025Z","iopub.status.idle":"2021-12-24T00:06:07.210863Z","shell.execute_reply.started":"2021-12-24T00:04:28.720978Z","shell.execute_reply":"2021-12-24T00:06:07.209693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_DC5","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:06:07.214034Z","iopub.execute_input":"2021-12-24T00:06:07.214349Z","iopub.status.idle":"2021-12-24T00:06:07.225012Z","shell.execute_reply.started":"2021-12-24T00:06:07.214304Z","shell.execute_reply":"2021-12-24T00:06:07.223961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## C4","metadata":{}},{"cell_type":"code","source":"from detectron2.engine import DefaultTrainer\n\n# Fine-tune a COCO-pretrained R50-C4 Mask R-CNN model on the dataset\ncfg_C4 = get_cfg()\ncfg_C4.merge_from_file(model_zoo.get_config_file(\"COCO-InstanceSegmentation/mask_rcnn_R_50_C4_3x.yaml\"))\ncfg_C4.DATASETS.TRAIN = (\"mat_train\",)\ncfg_C4.DATASETS.TEST = ()\ncfg_C4.DATALOADER.NUM_WORKERS = 2\ncfg_C4.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(\"COCO-InstanceSegmentation/mask_rcnn_R_50_C4_3x.yaml\")\ncfg_C4.SOLVER.IMS_PER_BATCH = 2\ncfg_C4.SOLVER.BASE_LR = 0.00025\ncfg_C4.SOLVER.MAX_ITER = 1000\ncfg_C4.SOLVER.STEPS = []\ncfg_C4.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128\ncfg_C4.MODEL.ROI_HEADS.NUM_CLASSES = 46\n\n# Train\ncfg_C4.OUTPUT_DIR = \"./output_C4\"\nos.makedirs(cfg_C4.OUTPUT_DIR, exist_ok=True)\ntrainer_C4 = DefaultTrainer(cfg_C4) \ntrainer_C4.resume_or_load(resume=False)\ntrainer_C4.train()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:07:28.632183Z","iopub.execute_input":"2021-12-24T00:07:28.632501Z","iopub.status.idle":"2021-12-24T00:34:39.782822Z","shell.execute_reply.started":"2021-12-24T00:07:28.632468Z","shell.execute_reply":"2021-12-24T00:34:39.780068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create predictor from the weigths obtained during the training\ncfg_C4.MODEL.WEIGHTS = os.path.join(cfg_C4.OUTPUT_DIR, \"model_final.pth\")\ncfg_C4.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5\ncfg_C4.DATASETS.TEST = ('mat_val')\npredictor_C4 = DefaultPredictor(cfg_C4)","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:34:39.787365Z","iopub.execute_input":"2021-12-24T00:34:39.788234Z","iopub.status.idle":"2021-12-24T00:34:40.913287Z","shell.execute_reply.started":"2021-12-24T00:34:39.788174Z","shell.execute_reply":"2021-12-24T00:34:40.912249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from detectron2.utils.visualizer import ColorMode\nplt.figure(figsize=(20,20))\nfor d in random.sample(materialist_dict, 3):    \n    im = cv2.imread(d[\"file_name\"])\n    outputs = predictor_C4(im)\n    visualizer = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = visualizer.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:34:40.915345Z","iopub.execute_input":"2021-12-24T00:34:40.915752Z","iopub.status.idle":"2021-12-24T00:34:46.154462Z","shell.execute_reply.started":"2021-12-24T00:34:40.915648Z","shell.execute_reply":"2021-12-24T00:34:46.153488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show different images at random\nrows, cols = 3, 3\nplt.figure(figsize=(20,20))\nfor i, d in enumerate(random.sample(materialist_dict, 9)):\n    # Process image\n    plt.subplot(rows, cols, i+1)\n    im = cv2.imread(d[\"file_name\"])\n    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)\n    \n    # Run through predictor\n    outputs = predictor_C4(im)\n    \n    # Visualize\n    v = Visualizer(im[:, :, ::-1],\n                   metadata=materialist_metadata, \n                   scale=0.8, \n                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels\n    )\n    v = v.draw_instance_predictions(outputs[\"instances\"].to(\"cpu\"))\n    plt.imshow(v.get_image()[:, :, ::-1])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:34:46.156971Z","iopub.execute_input":"2021-12-24T00:34:46.157462Z","iopub.status.idle":"2021-12-24T00:35:15.771767Z","shell.execute_reply.started":"2021-12-24T00:34:46.157414Z","shell.execute_reply":"2021-12-24T00:35:15.770362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Evaluate model\nevaluator_C4 = COCOEvaluator(\"mat_val\", output_dir=\"./output\")\nval_loader_C4 = build_detection_test_loader(cfg_C4, \"mat_val\")\nresult_C4 = inference_on_dataset(predictor_C4.model, val_loader_C4, evaluator_C4)","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:35:15.773889Z","iopub.execute_input":"2021-12-24T00:35:15.774469Z","iopub.status.idle":"2021-12-24T00:37:37.100826Z","shell.execute_reply.started":"2021-12-24T00:35:15.774422Z","shell.execute_reply":"2021-12-24T00:37:37.099688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_C4","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:37:37.103165Z","iopub.execute_input":"2021-12-24T00:37:37.103514Z","iopub.status.idle":"2021-12-24T00:37:37.114765Z","shell.execute_reply.started":"2021-12-24T00:37:37.103459Z","shell.execute_reply":"2021-12-24T00:37:37.113576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation","metadata":{}},{"cell_type":"code","source":"results_dict = {}\n\n# Create table to compare results\nresults_dict['bbox_DC5'] = result_DC5['bbox']\nresults_dict['bbox_C4'] = result_C4['bbox']\nresults_dict['bbox_FPN'] = result_FPN['bbox']\nresults_dict['segm_DC5'] = result_DC5['segm']\nresults_dict['segm_C4'] = result_C4['segm']\nresults_dict['segm_FPN'] = result_FPN['segm']\n\ndf_results = pd.DataFrame.from_dict(results_dict)\ndf_results.loc[['AP', 'AP50', 'AP75', 'APs', 'APm', 'APl']]","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:42:44.056369Z","iopub.execute_input":"2021-12-24T00:42:44.056739Z","iopub.status.idle":"2021-12-24T00:42:44.096905Z","shell.execute_reply.started":"2021-12-24T00:42:44.056705Z","shell.execute_reply":"2021-12-24T00:42:44.095972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As shown in the table, most of the metrics obtained during the training are similar. All the networks used to train have the same number of layers. If we wanted to improve those metrics, we could use a bigger training set (since we are using only a small subset to accelerate the training process), use a higher number of epochs or use a network architecture with a higher number of layers, such as res101.   ","metadata":{}},{"cell_type":"markdown","source":"Due to hardawre limitations the metrics are not as good as expected. ","metadata":{}},{"cell_type":"markdown","source":"As shown in the different results for each model, the AP metric is higher for the classes that are more frequent in the dataset (for instance shoes, sleeves, etc.), as expected.","metadata":{}},{"cell_type":"code","source":"import ast\n\nmetrics = {}\n\n# Create losses plots\nfor folder in [\"FPN\", \"C4\", \"DC5\"]:\n    with open(f'output_{folder}/metrics.json') as file:\n        lines = file.readlines()\n    \n    metrics_model = []\n    for line in lines:\n        metrics_model.append(ast.literal_eval(line))\n    metrics[folder] = metrics_model","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:43:05.801308Z","iopub.execute_input":"2021-12-24T00:43:05.801692Z","iopub.status.idle":"2021-12-24T00:43:05.836407Z","shell.execute_reply.started":"2021-12-24T00:43:05.801634Z","shell.execute_reply":"2021-12-24T00:43:05.835316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nfig, axs = plt.subplots(3, 4, figsize=(20,10))\n\nfor idx, model in enumerate([\"FPN\", \"C4\", \"DC5\"]):\n    met = metrics[model]\n    for loss_id, loss in enumerate([\"loss_box_reg\", \"loss_mask\", \"loss_cls\", \"total_loss\"]):\n        ax = axs[idx, loss_id]\n        total_loss = [loss_dict[loss] for loss_dict in met]\n        iterations = [loss_dict['iteration'] for loss_dict in met]\n\n        ax.set_title(f\"{model} - {loss}\")\n        ax.set_xlabel(\"iterations\")\n        ax.plot(iterations, total_loss)\n        ax.grid()\n        \nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2021-12-24T00:43:08.446819Z","iopub.execute_input":"2021-12-24T00:43:08.447885Z","iopub.status.idle":"2021-12-24T00:43:11.025191Z","shell.execute_reply.started":"2021-12-24T00:43:08.447826Z","shell.execute_reply":"2021-12-24T00:43:11.024199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The increase in the box loss is normal, and it is because the box regression loss is only applied to the positive boxes.\nGiven that the amount of positive boxes is very small in the beginning of training (due to the objectness classifier not being very good), the box regression loss is smaller. But as training goes on, we have better objectness, and thus more boxes to be regressed, and the regression loss increases a bit at the beginning.","metadata":{}}]}