![\"Instance](\"https://i.ibb.co/3y1QksP/Screenshot-2024-01-09-at-10-20-23-AM.png\")
\nImage classification can be defined as the prediction of a class that the whole image belongs to, fabric classification, for instance. On the other hand, object detection and localisation techniques aim for a more refined classification approach. In these cases, the goal is to discern and locate individual instances within the image, assigning each instance to its respective class (Hafiz and Bhat, 2020). Taking a step further in classification refinement, the semantic segmentation technique adopts an incremental approach by attributing each pixel in an image to one of the predefined output classes.
\n\nAn exploratory analysis of DeepFashion2 indicates that the image dimensions fluctuate vastly. In the dataset, the largest images have dimensions exceeding 3000 pixels in height and 2000 pixels in width. Conversely, the smallest images have dimensions of around 200 pixels for both height and width.
\nTo handle the variation in image sizes, one alternative is to specify a target height and width, along with a resizing method in the Mask R-CNN configuration which will then perform resizing as part of training. However, this approach can result in longer training times and higher computational costs. As an alternative, we resize the entire dataset to a dimension of 256 × 256 along with their corresponding annotations, as part of data pre-processing.
\nThe Kaggle notebook where I pre-processed and resized DeepFashion2 can be found here - Resizing DeepFashion2 (256 x 256) and Basic EDA
\nDeepFashion2 is a large dataset, containing over 491k images with a total of 801k items (Ge et al., 2019). Due to constraints on time and computational resources, working with the full dataset can be impractical. One solution is to create a subset of the training and validation sets while keeping the original class balance. However, care must be taken to ensure that when an image is sampled, the information for all garment instances within the image is retained.
\nWe analyse the distributions of different garment category combinations in the dataset, such as the number of images containing both short sleeve tops and skirts. The dataset was sampled based on these category combinations rather than individual categories. This approach ensures that if an image is included in the sample, the information for all garment instances within the image is also retained.
\nTo address the class imbalance observed in the original dataset, we strategically refined our sampling method to ensure an approximately balanced representation of each garment category. With this approach, we generated a training dataset of 16k distinct images and a validation set of 8k images. An inspection of the categories within these sets reveals a more even distribution of garment classes.
\n\nWe use an open-source variant (Wijkhuizen, 2021) of Matterplot’s Mask R-CNN implementation (Abdulla, 2017) as well as the original implementation for the experiment. This variant allows the use of any model from the EfficientNetV2 family as the backbone (Wijkhuizen, 2021), in contrast to the original implementation, which uses a ResNet backbone (Abdulla, 2017). Several default settings of Mask R-CNN were modified to better fit our dataset and the available computational resources.\n
\n![\"Model](\"https://i.ibb.co/tcJpxM4/Screenshot-2024-01-09-at-10-17-27-AM.png\")
\nA Dataset class is configured which transforms the tabular data created in the earlier phase into variables accessible by the network functions. A custom configuration sets up the hyperparameters and other options for training. For instance, some hyperparameters control the number of instances identified in an image during training and another set of hyperparameters define the size and shape of the instance masks detected by the model.\n
\n\nThe ResNet50 backbone model is initialised with COCO weights. During training and validation, we begin by training only the top layers of the model, using a learning rate of 0.0002. Subsequently, all layers are trained with a reduced learning rate of 0.0001, but with a momentum of 0.9. This step is repeated with an even smaller learning rate of 0.00002. To introduce variety and enhance the generalising capabilities of the model, in-place data augmentation is employed. The input images are flipped horizontally with a 50% probability. This augmentation technique leads to about half of the images in the batch being flipped horizontally, while the others are left as is.\n
\n\nFor garment instance segmentation, the model is trained utilising a single NVIDIA TESLA P100 Graphics Processing Unit (GPU). Due to resource limitations, the training is capped at 12 epochs, selecting the epoch with the smallest validation loss for the network weights to optimise the model performance.\n
\n\nFor the testing phase, the model weights that yield the lowest validation loss are selected. The performance of the models is then assessed using a testing dataset comprised of approximately 8k samples.\n
\n\nTypically, the evaluation of object detection and instance segmentation tasks is carried out using the Average Precision (AP) metric at various Intersection over Union (IoU) thresholds (He et al., 2018). IoU is a measure that quantifies the overlap between predicted and ground truth bounding boxes or masks by calculating the ratio of their intersection to their union. An IoU value of 1 signifies perfectly aligned masks. \n
\n \n\nTo evaluate and compare the performance of the models, the Mean Average Precision (mAP) is computed across all testing samples at four distinct IoU thresholds, 0.5, 0.65, 0.75 and 0.85, which correspond to 50%, 65%, 75% and 85% overlap, respectively.\n
\n\nTo evaluate the performance of the models across various garment categories, the validation dataset was filtered to include only images annotated with a single garment.\n
\n \n\nThe ResNet50 model exhibits good precision for certain garment categories, such as trousers, shorts, and short sleeve tops with average confidences of 0.9, 0.79 and 0.72, respectively. However, there is a possibility of the model performing poorly with other garment categories such as skirt where the highest confidence score of 0.84 is when the predicted value is long sleeve top.\n
\n\nWe filter the validation data to include images with only one annotated item and record the AP at 50% and 75% thresholds. This process is repeated across all garment categories to see how the model performs for each individual category.\n
\n \n\nAlthough confidence scores cannot be directly interpreted as the prediction efficiency of the model, this suggests that the model tends to predict with higher confidence for categories that are more frequently represented in the dataset.\n
\n \n\nWe filter the validation data to include images with atleast one annotated item and record the AP at 50% and 75% thresholds. This process is repeated across all garment categories to see how the model performs for each individual category.\n
\n \n\n A similar version of this notebook but with the EfficientNetV2-B0 backbone can be found here - Fashion Instance Segmentation using MaskRCNN(EfficientNetV2-B0)\n
\n![](\"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExa2lmeHhzaGU0b3ZlNnBsNzc0ZTdtY25obWtidm14MnFiZWdyNmxtZiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/6tHy8UAbv3zgs/giphy.gif\")
\n