![](\"https://i.imgur.com/xeXdDYC.png\")
We will present an introduction to the semantic segmentation task. But before we understand semantic segmentation we first need to understand object detection as semantic segmentation is object detection \"with an extra step\".
\nThere are many different tasks in Computer Vision, some of which you may already be familiary with:
\n
For example, some Computer Vision tasks are:
\nAs said before, semantic segmentation is like an \"extension\" of object detection. So let's get first a clear understanding of the object detection task.
\nWhat's the input and the output?
\n(height, width, channels).What is the notation for bounding boxes?
\nThese are some common types of notations:
\n(x1, y1, x2, y2) where (x1, y1) correspond to the upper left corner coordinates of the box and (x2, y2) correspond to the lower right corner coordinates of the box.(x, y, width, height) where (x, y) are the coordinates of the center of the box and the height and width of the box.(x_min, y_min) of its lower left corner within the image together with its width and height in pixels. What would be the approach? We can do it with a simple NN architecture which uses common backbones (like VGG, AlexNet, ResNet, etc) and attach to them:
\nSince we have two loss functions and we need only one single loss to compute gradient descent we usually just sum up these two loss functions as a weighted sum. We do a weighted sum so that the weights of each loss can be learned and thus one problem does not overcomes the other.
\nImages can have more than one object! What do we do? If we must detect multiple objects in an image we don't know beforehand how many objects will be present and the same object can appear multiple times in the same image too! For example, in a single image:
\nOne approach to deal with this problem is the sliding window, which consists in cropping or dividing the input image in multiple crops. For each crop we apply the CNN and classify each crop as object or background. The obvious problems that rise with this approach are:
\nIf we have a 800x600 image we have ~58M boxes! No way we can evaluate all of them!
\nOne solution to this problem are Region Proposal mechanisms.
\n![](\"https://i.imgur.com/9yLr02y.png\")
Problems:
\nAs we can see, the R-CNN needs to do ~2000 forward passes for each image since we had ~2000 RoI. This is very expensive computationally and thus slow. The solution that people has come up with is running the CNN before warping. This allows us to reshare a lot of computation across image regions.
\nThis new solution is usually called Fast R-CNN. It basically runs the whole input image through a CNN and outputs a feature map or also called image features. This CNN we use is usually called the \"backbone\" network (like AlexNet, VGG, ResNet, etc). Then, the regions of interest are applied to the image features. Next, we warp/crop/resize the RoIs and input them into a light-weight CNN that outputs categories and box transforms per crop.
\nThis is much much faster since most of the computation happens in the backbone network (this saves work for overlapping region proposals) and the per-region network is relatively light-weight.
\nFast R-CNN can be up to 10 times faster than Slow R-CNN at training time and up to 20 times faster at test time.
\n![](\"https://i.imgur.com/mQeWzlJ.png\")
Even though that Fast R-CNN is much faster than Slow R-CNN most of the compute time is dominated by region proposals! Region proposals are computed by an heuristic (Selective search algorithm) which runs on CPU. We could learn region proposals with a CNN instead!
\nFaster R-CNN inserts a Region Proposal Network (RPN) to predict proposals from features. Otherwise, it is the same as Fast R-CNN. RPNs use \"anchor boxes\" of fixed size at each point in the feature map. For each anchor box we predict whether the corresponding anchor contains an object or not and a box transform to regress from anchor box to object box.
\nFaster R-CNN has 4 train losses!
\nOn a GPU Faster R-CNN is up to 10 times faster than Fast R-CNN.
\n![](\"https://i.imgur.com/bYlP0RJ.jpeg\")
Semantic segmentation's objective is to label each pixel in the image with a category label. It does not care about differentiating instances, it only cares about pixels. What this means, for example, is that if we have to adjacent categories next to each other (e.g: two cows next to each other) they will not be classified as two different instances but rather one single cloud of points that belong to the same category (one cloud/blob of points which belong to the class \"cow\" but does not tell us which pixels belong to which cow).
\nThe architecture used in Semantic Segmentation is very much the same as Faster RCNN but with one more branch at the end called the mask branch.
\nWhat is commonly used in semantic segmentations mask branchs are Fully Convolutional Networks. This is a convolutional network architecture that does not have any fully connected layers nor pooling layers but rather only have convolutional layers. In this way an input image with a fixed size is passed through all these convolutional layers until on the final layer we perfom a softmax() on each pixel that gives us a set of class scores for every pixel followed by an argmax() that classifies each pixel to a category. We can train this using a cross entropy loss on each pixel.
In practice, what is used is convolutional networks with downsampling and upsampling, like the famous U-Net. These architectures have high resolution layers followed by mid-resolution layers followed by low-resolution layers on the downsampling stage and the other way round on the upsampling stage.
\n![](\"https://i.imgur.com/zYpAleW.png\")
Mask R-CNN is just like Faster R-CNN but with an extra branch that predicts a mask.
\n![](\"https://i.imgur.com/8DyNisQ.png\")
So the way this works in summary is:
\nSome popular Python libraries are PyTorch Segmentation Models and OpenMMLab Semantic Segmentation.
\nExample of a Segmentation Network in segmentation_models_pytorch:
segmentation_models_pytorch smp\n\nmodel = smp.Unet(\n encoder_name=, \n encoder_weights=, \n in_channels=, \n classes=, \n)\nSemantic Segmentation is a Computer Vision task where we want to classify each pixel in an image to detect an object and its shape/mask. It's similar to Object Detection but with one extra step!
\nIn this competition our input is an RGB image and its corresponding mask: an image with the same dimension as the original but with 1s and 0s (1 to indicate where the object is present and 0 where it's not). The output is the predicted mask.
\nSome popular architectures are UNets and you can find models implemented in PyTorch in segmentation_models_pytorch.
Some common loss functions used in Segmentation tasks are Dice Loss and Jaccard Loss. Some common metrics used are Dice Coefficient and IoU Coefficient.
\nHope you liked it! 🍀
\nSlides were taken from Justin Johnson's amazing Computer Vision course. You can find the course here.
", "messages": [ { "id": "2189652", "postDate": "03/20/2023 16:53:32", "content": "We will present an introduction to the semantic segmentation task. But before we understand semantic segmentation we first need to understand object detection as semantic segmentation is object detection \"with an extra step\".
\nThere are many different tasks in Computer Vision, some of which you may already be familiary with:
\n
For example, some Computer Vision tasks are:
\nAs said before, semantic segmentation is like an \"extension\" of object detection. So let's get first a clear understanding of the object detection task.
\nWhat's the input and the output?
\n(height, width, channels).What is the notation for bounding boxes?
\nThese are some common types of notations:
\n(x1, y1, x2, y2) where (x1, y1) correspond to the upper left corner coordinates of the box and (x2, y2) correspond to the lower right corner coordinates of the box.(x, y, width, height) where (x, y) are the coordinates of the center of the box and the height and width of the box.(x_min, y_min) of its lower left corner within the image together with its width and height in pixels. What would be the approach? We can do it with a simple NN architecture which uses common backbones (like VGG, AlexNet, ResNet, etc) and attach to them:
\nSince we have two loss functions and we need only one single loss to compute gradient descent we usually just sum up these two loss functions as a weighted sum. We do a weighted sum so that the weights of each loss can be learned and thus one problem does not overcomes the other.
\nImages can have more than one object! What do we do? If we must detect multiple objects in an image we don't know beforehand how many objects will be present and the same object can appear multiple times in the same image too! For example, in a single image:
\nOne approach to deal with this problem is the sliding window, which consists in cropping or dividing the input image in multiple crops. For each crop we apply the CNN and classify each crop as object or background. The obvious problems that rise with this approach are:
\nIf we have a 800x600 image we have ~58M boxes! No way we can evaluate all of them!
\nOne solution to this problem are Region Proposal mechanisms.
\n
Problems:
\nAs we can see, the R-CNN needs to do ~2000 forward passes for each image since we had ~2000 RoI. This is very expensive computationally and thus slow. The solution that people has come up with is running the CNN before warping. This allows us to reshare a lot of computation across image regions.
\nThis new solution is usually called Fast R-CNN. It basically runs the whole input image through a CNN and outputs a feature map or also called image features. This CNN we use is usually called the \"backbone\" network (like AlexNet, VGG, ResNet, etc). Then, the regions of interest are applied to the image features. Next, we warp/crop/resize the RoIs and input them into a light-weight CNN that outputs categories and box transforms per crop.
\nThis is much much faster since most of the computation happens in the backbone network (this saves work for overlapping region proposals) and the per-region network is relatively light-weight.
\nFast R-CNN can be up to 10 times faster than Slow R-CNN at training time and up to 20 times faster at test time.
\n
Even though that Fast R-CNN is much faster than Slow R-CNN most of the compute time is dominated by region proposals! Region proposals are computed by an heuristic (Selective search algorithm) which runs on CPU. We could learn region proposals with a CNN instead!
\nFaster R-CNN inserts a Region Proposal Network (RPN) to predict proposals from features. Otherwise, it is the same as Fast R-CNN. RPNs use \"anchor boxes\" of fixed size at each point in the feature map. For each anchor box we predict whether the corresponding anchor contains an object or not and a box transform to regress from anchor box to object box.
\nFaster R-CNN has 4 train losses!
\nOn a GPU Faster R-CNN is up to 10 times faster than Fast R-CNN.
\n
Semantic segmentation's objective is to label each pixel in the image with a category label. It does not care about differentiating instances, it only cares about pixels. What this means, for example, is that if we have to adjacent categories next to each other (e.g: two cows next to each other) they will not be classified as two different instances but rather one single cloud of points that belong to the same category (one cloud/blob of points which belong to the class \"cow\" but does not tell us which pixels belong to which cow).
\nThe architecture used in Semantic Segmentation is very much the same as Faster RCNN but with one more branch at the end called the mask branch.
\nWhat is commonly used in semantic segmentations mask branchs are Fully Convolutional Networks. This is a convolutional network architecture that does not have any fully connected layers nor pooling layers but rather only have convolutional layers. In this way an input image with a fixed size is passed through all these convolutional layers until on the final layer we perfom a softmax() on each pixel that gives us a set of class scores for every pixel followed by an argmax() that classifies each pixel to a category. We can train this using a cross entropy loss on each pixel.
In practice, what is used is convolutional networks with downsampling and upsampling, like the famous U-Net. These architectures have high resolution layers followed by mid-resolution layers followed by low-resolution layers on the downsampling stage and the other way round on the upsampling stage.
\n
Mask R-CNN is just like Faster R-CNN but with an extra branch that predicts a mask.
\n
So the way this works in summary is:
\nSome popular Python libraries are PyTorch Segmentation Models and OpenMMLab Semantic Segmentation.
\nExample of a Segmentation Network in segmentation_models_pytorch:
segmentation_models_pytorch smp\n\nmodel = smp.Unet(\n encoder_name=, \n encoder_weights=, \n in_channels=, \n classes=, \n)\nSemantic Segmentation is a Computer Vision task where we want to classify each pixel in an image to detect an object and its shape/mask. It's similar to Object Detection but with one extra step!
\nIn this competition our input is an RGB image and its corresponding mask: an image with the same dimension as the original but with 1s and 0s (1 to indicate where the object is present and 0 where it's not). The output is the predicted mask.
\nSome popular architectures are UNets and you can find models implemented in PyTorch in segmentation_models_pytorch.
Some common loss functions used in Segmentation tasks are Dice Loss and Jaccard Loss. Some common metrics used are Dice Coefficient and IoU Coefficient.
\nHope you liked it! 🍀
\nSlides were taken from Justin Johnson's amazing Computer Vision course. You can find the course here.
", "rawMarkdown": "### Table of Contents\n\n1. Introduction\n2. We need to understand Object Detection first!\n3. Slow R-CNN\n4. Fast R-CNN\n5. Faster R-CNN\n6. Finally... Semantic Segmentation!\n7. Mask R-CNN\n8. Where can I find segmentation models?\n9. Conclusion\n\n\n### Introduction\n\nWe will present an introduction to the semantic segmentation task. But before we understand semantic segmentation we first need to understand object detection as semantic segmentation is object detection \"with an extra step\".\n\nThere are many different tasks in Computer Vision, some of which you may already be familiary with:\n\n\n\nFor example, some Computer Vision tasks are:\n\n- **Image classification** consists of predicting whether a class is present or not in an image. \n- **Multi-class classification:** multi-class classification is the extension of the **binary classification** problem. In this type of problem each image can belong to only one category, but the amount of categories is more than two. This means that class labels or class membership are mutually exclusive. For example, predicting if the animal present in an image is a cat, a dog or a mouse.\n- **Multi-label classification:** in multi-label classification one single image can have multiple labels present. This means that class labels or class membership are not mutually exclusive. In multi-label classification, zero or more labels are required as output for each input sample. For example, predicting which animals are present in an image.\n- **Object detection:** object detection involves drawing a bounding box around one or more objects in an image. In this way, object detection does not only detect if a class is present in an image or not but it also tells where it is located providing a box surrounding the object.\n- **Semantic segmentation**: labels all parts of an image with a label. Each pixel in an image is assigned a label. The goal of semantic image segmentation is to label each pixel of an image with a corresponding class of what is being represented. Because we're predicting for every pixel in the image, this task is commonly referred to as dense prediction.\n- **Instance segmentation:** instance segmentation deals with detecting instances of objects and demarcating their boundaries. It is similar to semantic segmentation in the sense that it clearly outputs the object boundaries (irregular boundaries), but it is different in the sense that it does not segment all the image but rather the objects of interest. It is similar to object detection in the sense that it does not segment all the image, but it is different in the sense that it provides a more fine grained boundary than just a bounding box.\n\n### We need to understand Object Detection first!\n\nAs said before, semantic segmentation is like an \"extension\" of object detection. So let's get first a clear understanding of the object detection task.\n\n**What's the input and the output?**\n\n- **Input:** Single RGB image. The shape is then `(height, width, channels)`.\n- **Output:** a set of detected objects. For each object predict:\n 1. Category label, from a fixed known set of categories, like \"cat\", \"dog\", \"mouse\", etc.\n 2. Bounding box (four numbers: x, y, width, height).\n \n**What is the notation for bounding boxes?**\n\nThese are some common types of notations:\n1. **Upper left corner to lower right corner:** we provide four numbers `(x1, y1, x2, y2)` where `(x1, y1)` correspond to the upper left corner coordinates of the box and `(x2, y2)` correspond to the lower right corner coordinates of the box.\n2. **Center box:** we provide four numbers `(x, y, width, height)` where `(x, y)` are the coordinates of the center of the box and the height and width of the box.\n3. **COCO format:** A bounding box is described by the pixel coordinate `(x_min, y_min)` of its lower left corner within the image together with its width and height in pixels. \n\n### Detecting a single object\n\nWhat would be the approach? We can do it with a simple NN architecture which uses common backbones (like VGG, AlexNet, ResNet, etc) and attach to them:\n- One branch (**\"what\" branch**) of layers that handle the classification problem which consists of some fully connected layers plus a softmax function (to classify which of the possible categories are present) with a Softmax Loss. \n- One other branch (**\"where\" branch**) of layers that handle the localization problem which consists of some fully connected layers which end up in 4 nodes (x, y, width, height) with a regression loss, like L2 Loss.\n\nSince we have two loss functions and we need only one single loss to compute gradient descent we usually just sum up these two loss functions as a weighted sum. We do a weighted sum so that the weights of each loss can be learned and thus one problem does not overcomes the other.\n\n### Detecting multiple objects: a more challenging challenge\n\nImages can have more than one object! What do we do? If we must detect multiple objects in an image we don't know beforehand how many objects will be present and the same object can appear multiple times in the same image too! For example, in a single image:\n- One object can appear one or more times (10 dogs in an image).\n- Multiple objects can appear and more than one time (2 dogs and 3 cats).\n- The objects can appear in different regions.\n\nOne approach to deal with this problem is the **sliding window**, which consists in cropping or dividing the input image in multiple crops. For each crop we apply the CNN and classify each crop as object or background. The obvious problems that rise with this approach are:\n- **Which should be the shape of the bounding box?**\n- **How many possible boxes are there in an image of size HxW?** There are many! For each image we need to consider all the possible box sizes and ratios which yield the next formula: \n$$ Total\\:possible\\:boxes = \\frac{H(H+1)}{2}\\frac{W(W+1)}{2} $$\n\nIf we have a 800x600 image we have ~58M boxes! No way we can evaluate all of them!\n\nOne solution to this problem are **Region Proposal** mechanisms.\n\n- Find a small set of boxes that are likely to cover all objects.\n- Often based on heuristics: e.g. look for \"blob-like\" image regions that have high probabilites of containing objects.\n- Relatively fast to run: e.g. Selective Search gives ~2000 region proposals in a few seconds on CPU.\n- They will end up being replaced by CNN (much faster approach and with learnable parameters).\n- A proposal method gives us **regions of interest** (RoI)\n\n### R-CNN: Region-based CNN (usually nowadays called \"Slow\" R-CNN)\n\n\n\n1. We start with an input image and a Region Proposal mechanism which gives us ~2k regions of interest (RoI).\n2. For each proposed region of interest we warp/resize each region into a fixed size region since the RoI can be of variable heights and widths. The warp size depends on the backbone architecure we use for classifying each warp. Some CNNs require input images to be 224x224 for example.\n3. For each warped region we run it through a CNN and classify each region. In this way, one image will yield many regions which will be passed through the CNN.\n4. Once we get a score for each region we try to select a subset of region proposals to output. To explain this better, let's say we have a cat and a dog in an image, but the model outputs ~2000 predictions for all the RoIs, we need a way to summarize all the predicitons so that we just output one single box for the dog and one for the cat. Many choices here: threshold on background or per-category, take top K proposals per image, etc.\n5. Compare with ground-truth boxes.\n\n\n\nProblems:\n- What happen if the proposed regions do not match well to our objects? To overcome this problem, in addition to classifying the class of the region we also perform a bounding box regression which tries to predict or \"transform\" the correct RoI's x, y, height and width. This way we iteratively try to get better RoIs which help us better detect the object in our image.\n \n### Fast R-CNN\n\nAs we can see, the R-CNN needs to do ~2000 forward passes for each image since we had ~2000 RoI. This is very expensive computationally and thus slow. The solution that people has come up with is running the CNN **before** warping. This allows us to reshare a lot of computation across image regions. \n\nThis new solution is usually called **Fast R-CNN**. It basically runs the whole input image through a CNN and outputs a feature map or also called image features. This CNN we use is usually called the \"backbone\" network (like AlexNet, VGG, ResNet, etc). Then, the regions of interest are applied to the image features. Next, we warp/crop/resize the RoIs and input them into a light-weight CNN that outputs categories and box transforms per crop.\n\nThis is much much faster since most of the computation happens in the backbone network (this saves work for overlapping region proposals) and the per-region network is relatively light-weight.\n\nFast R-CNN can be up to 10 times faster than Slow R-CNN at training time and up to 20 times faster at test time.\n\n\n\n### Faster R-CNN: learning region proposals.\n\nEven though that Fast R-CNN is much faster than Slow R-CNN most of the compute time is dominated by region proposals! Region proposals are computed by an heuristic (Selective search algorithm) which runs on CPU. We could learn region proposals with a CNN instead!\n\n\nFaster R-CNN inserts a **Region Proposal Network (RPN)** to predict proposals from features. Otherwise, it is the same as Fast R-CNN. RPNs use \"anchor boxes\" of fixed size at each point in the feature map. For each anchor box we predict whether the corresponding anchor contains an object or not and a box transform to regress from anchor box to object box.\n\nFaster R-CNN has 4 train losses!\n\n1. RPN classification: anchor box is object or not object.\n2. RPN regression: predict transform from anchor box to proposal box.\n3. Object classification: classify proposals as background or object class.\n4. Object regression: predict transform from proposal box to object box.\n\nOn a GPU Faster R-CNN is up to 10 times faster than Fast R-CNN.\n\n\n\n\n### Finally... Semantic Segmentation!\n\nSemantic segmentation's objective is to label each pixel in the image with a category label. It does not care about differentiating instances, it only cares about pixels. What this means, for example, is that if we have to adjacent categories next to each other (e.g: two cows next to each other) they will not be classified as two different instances but rather one single cloud of points that belong to the same category (one cloud/blob of points which belong to the class \"cow\" but does not tell us which pixels belong to which cow).\n\n\n### Semantic segmentation: fully convolutional networks\n\nThe architecture used in Semantic Segmentation is very much the same as Faster RCNN but with one more branch at the end called the **mask branch**.\n\nWhat is commonly used in semantic segmentations mask branchs are Fully Convolutional Networks. This is a convolutional network architecture that does not have any fully connected layers nor pooling layers but rather only have convolutional layers. In this way an input image with a fixed size is passed through all these convolutional layers until on the final layer we perfom a `softmax()` on each pixel that gives us a set of class scores for every pixel followed by an `argmax()` that classifies each pixel to a category. We can train this using a cross entropy loss on each pixel.\n\nIn practice, what is used is convolutional networks with downsampling and upsampling, like the famous U-Net. These architectures have high resolution layers followed by mid-resolution layers followed by low-resolution layers on the downsampling stage and the other way round on the upsampling stage.\n\n\n\n\n### Mask R-CNN\n\nMask R-CNN is just like Faster R-CNN but with an extra branch that predicts a mask.\n\n\n\nSo the way this works in summary is:\n1. We input our RGB image into the CNN backbone (this can be any CNN, like EfficientNet) which outputs image-level features.\n2. We pass the image-level features to the Region Proposal Network (RPN) to get the region proposals.\n3. For each region proposal we pass it through a small semantic segmentation network (like Unet) to get the prediction masks.\n4. There are many branches involved!\n - Two for the RPN: one to detect the ROIs bounding boxes and another to detect if an object is present in the ROI.\n - One for classification to detect which object is present.\n - One to detect the bounding box.\n - One to detect the mask.\n \n### Where can I find segmentation models?\n\n\nSome popular Python libraries are [PyTorch Segmentation Models](https://github.com/qubvel/segmentation_models.pytorch) and [OpenMMLab Semantic Segmentation](https://github.com/open-mmlab/mmsegmentation). \n\nExample of a Segmentation Network in `segmentation_models_pytorch`:\n```python\nimport segmentation_models_pytorch as smp\n\nmodel = smp.Unet(\n encoder_name=\"resnet34\", # choose encoder, e.g. mobilenet_v2 or efficientnet-b7\n encoder_weights=\"imagenet\", # use `imagenet` pre-trained weights for encoder initialization\n in_channels=1, # model input channels (1 for gray-scale images, 3 for RGB, etc.)\n classes=3, # model output channels (number of classes in your dataset)\n)\n```\n\n### Conclusion\n\nSemantic Segmentation is a Computer Vision task where we want to classify each pixel in an image to detect an object and its shape/mask. It's similar to Object Detection but with one extra step!\n\nIn this competition our input is an RGB image and its corresponding mask: an image with the same dimension as the original but with 1s and 0s (1 to indicate where the object is present and 0 where it's not). The output is the predicted mask.\n\nSome popular architectures are UNets and you can find models implemented in PyTorch in `segmentation_models_pytorch`.\n\nSome common loss functions used in Segmentation tasks are [Dice Loss](https://smp.readthedocs.io/en/latest/losses.html#diceloss) and [Jaccard Loss](https://smp.readthedocs.io/en/latest/losses.html#jaccardloss). Some common metrics used are [Dice Coefficient](https://pbs.twimg.com/media/EWci84bWsAIBWvV?format=png&name=small) and [IoU Coefficient](https://miro.medium.com/v2/resize:fit:1400/1*FSauyBPV0fiVa_BUVutVTg.png).\n\nHope you liked it! 🍀\n\n\n### References\n\nSlides were taken from Justin Johnson's amazing Computer Vision course. You can find the course [here](https://www.youtube.com/playlist?list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r).",
"votes": null
},
{
"id": "2193696",
"postDate": "03/23/2023 12:29:47",
"content": "Thanks @alejopaullier for sharing a very useful source of knowledge. Nice elaboration full of insights👍
", "rawMarkdown": "Thanks @alejopaullier for sharing a very useful source of knowledge. Nice elaboration full of insights👍", "votes": null }, { "id": "2264021", "postDate": "05/18/2023 05:10:32", "content": "Excellent explanation @alejopaullier
", "rawMarkdown": "Excellent explanation @alejopaullier", "votes": null } ], "comments": [ { "id": 2193696, "author_name": "tariqbashir", "author_url": "", "post_date": "03/23/2023 12:29:47", "content": "Thanks @alejopaullier for sharing a very useful source of knowledge. Nice elaboration full of insights👍
", "votes": null, "replies": [] }, { "id": 2264021, "author_name": "tamannaakterswarna", "author_url": "", "post_date": "05/18/2023 05:10:32", "content": "Excellent explanation @alejopaullier
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2189652": "### Table of Contents\n\n1. Introduction\n2. We need to understand Object Detection first!\n3. Slow R-CNN\n4. Fast R-CNN\n5. Faster R-CNN\n6. Finally... Semantic Segmentation!\n7. Mask R-CNN\n8. Where can I find segmentation models?\n9. Conclusion\n\n\n### Introduction\n\nWe will present an introduction to the semantic segmentation task. But before we understand semantic segmentation we first need to understand object detection as semantic segmentation is object detection \"with an extra step\".\n\nThere are many different tasks in Computer Vision, some of which you may already be familiary with:\n\n\n\nFor example, some Computer Vision tasks are:\n\n- **Image classification** consists of predicting whether a class is present or not in an image. \n- **Multi-class classification:** multi-class classification is the extension of the **binary classification** problem. In this type of problem each image can belong to only one category, but the amount of categories is more than two. This means that class labels or class membership are mutually exclusive. For example, predicting if the animal present in an image is a cat, a dog or a mouse.\n- **Multi-label classification:** in multi-label classification one single image can have multiple labels present. This means that class labels or class membership are not mutually exclusive. In multi-label classification, zero or more labels are required as output for each input sample. For example, predicting which animals are present in an image.\n- **Object detection:** object detection involves drawing a bounding box around one or more objects in an image. In this way, object detection does not only detect if a class is present in an image or not but it also tells where it is located providing a box surrounding the object.\n- **Semantic segmentation**: labels all parts of an image with a label. Each pixel in an image is assigned a label. The goal of semantic image segmentation is to label each pixel of an image with a corresponding class of what is being represented. Because we're predicting for every pixel in the image, this task is commonly referred to as dense prediction.\n- **Instance segmentation:** instance segmentation deals with detecting instances of objects and demarcating their boundaries. It is similar to semantic segmentation in the sense that it clearly outputs the object boundaries (irregular boundaries), but it is different in the sense that it does not segment all the image but rather the objects of interest. It is similar to object detection in the sense that it does not segment all the image, but it is different in the sense that it provides a more fine grained boundary than just a bounding box.\n\n### We need to understand Object Detection first!\n\nAs said before, semantic segmentation is like an \"extension\" of object detection. So let's get first a clear understanding of the object detection task.\n\n**What's the input and the output?**\n\n- **Input:** Single RGB image. The shape is then `(height, width, channels)`.\n- **Output:** a set of detected objects. For each object predict:\n 1. Category label, from a fixed known set of categories, like \"cat\", \"dog\", \"mouse\", etc.\n 2. Bounding box (four numbers: x, y, width, height).\n \n**What is the notation for bounding boxes?**\n\nThese are some common types of notations:\n1. **Upper left corner to lower right corner:** we provide four numbers `(x1, y1, x2, y2)` where `(x1, y1)` correspond to the upper left corner coordinates of the box and `(x2, y2)` correspond to the lower right corner coordinates of the box.\n2. **Center box:** we provide four numbers `(x, y, width, height)` where `(x, y)` are the coordinates of the center of the box and the height and width of the box.\n3. **COCO format:** A bounding box is described by the pixel coordinate `(x_min, y_min)` of its lower left corner within the image together with its width and height in pixels. \n\n### Detecting a single object\n\nWhat would be the approach? We can do it with a simple NN architecture which uses common backbones (like VGG, AlexNet, ResNet, etc) and attach to them:\n- One branch (**\"what\" branch**) of layers that handle the classification problem which consists of some fully connected layers plus a softmax function (to classify which of the possible categories are present) with a Softmax Loss. \n- One other branch (**\"where\" branch**) of layers that handle the localization problem which consists of some fully connected layers which end up in 4 nodes (x, y, width, height) with a regression loss, like L2 Loss.\n\nSince we have two loss functions and we need only one single loss to compute gradient descent we usually just sum up these two loss functions as a weighted sum. We do a weighted sum so that the weights of each loss can be learned and thus one problem does not overcomes the other.\n\n### Detecting multiple objects: a more challenging challenge\n\nImages can have more than one object! What do we do? If we must detect multiple objects in an image we don't know beforehand how many objects will be present and the same object can appear multiple times in the same image too! For example, in a single image:\n- One object can appear one or more times (10 dogs in an image).\n- Multiple objects can appear and more than one time (2 dogs and 3 cats).\n- The objects can appear in different regions.\n\nOne approach to deal with this problem is the **sliding window**, which consists in cropping or dividing the input image in multiple crops. For each crop we apply the CNN and classify each crop as object or background. The obvious problems that rise with this approach are:\n- **Which should be the shape of the bounding box?**\n- **How many possible boxes are there in an image of size HxW?** There are many! For each image we need to consider all the possible box sizes and ratios which yield the next formula: \n$$ Total\\:possible\\:boxes = \\frac{H(H+1)}{2}\\frac{W(W+1)}{2} $$\n\nIf we have a 800x600 image we have ~58M boxes! No way we can evaluate all of them!\n\nOne solution to this problem are **Region Proposal** mechanisms.\n\n- Find a small set of boxes that are likely to cover all objects.\n- Often based on heuristics: e.g. look for \"blob-like\" image regions that have high probabilites of containing objects.\n- Relatively fast to run: e.g. Selective Search gives ~2000 region proposals in a few seconds on CPU.\n- They will end up being replaced by CNN (much faster approach and with learnable parameters).\n- A proposal method gives us **regions of interest** (RoI)\n\n### R-CNN: Region-based CNN (usually nowadays called \"Slow\" R-CNN)\n\n\n\n1. We start with an input image and a Region Proposal mechanism which gives us ~2k regions of interest (RoI).\n2. For each proposed region of interest we warp/resize each region into a fixed size region since the RoI can be of variable heights and widths. The warp size depends on the backbone architecure we use for classifying each warp. Some CNNs require input images to be 224x224 for example.\n3. For each warped region we run it through a CNN and classify each region. In this way, one image will yield many regions which will be passed through the CNN.\n4. Once we get a score for each region we try to select a subset of region proposals to output. To explain this better, let's say we have a cat and a dog in an image, but the model outputs ~2000 predictions for all the RoIs, we need a way to summarize all the predicitons so that we just output one single box for the dog and one for the cat. Many choices here: threshold on background or per-category, take top K proposals per image, etc.\n5. Compare with ground-truth boxes.\n\n\n\nProblems:\n- What happen if the proposed regions do not match well to our objects? To overcome this problem, in addition to classifying the class of the region we also perform a bounding box regression which tries to predict or \"transform\" the correct RoI's x, y, height and width. This way we iteratively try to get better RoIs which help us better detect the object in our image.\n \n### Fast R-CNN\n\nAs we can see, the R-CNN needs to do ~2000 forward passes for each image since we had ~2000 RoI. This is very expensive computationally and thus slow. The solution that people has come up with is running the CNN **before** warping. This allows us to reshare a lot of computation across image regions. \n\nThis new solution is usually called **Fast R-CNN**. It basically runs the whole input image through a CNN and outputs a feature map or also called image features. This CNN we use is usually called the \"backbone\" network (like AlexNet, VGG, ResNet, etc). Then, the regions of interest are applied to the image features. Next, we warp/crop/resize the RoIs and input them into a light-weight CNN that outputs categories and box transforms per crop.\n\nThis is much much faster since most of the computation happens in the backbone network (this saves work for overlapping region proposals) and the per-region network is relatively light-weight.\n\nFast R-CNN can be up to 10 times faster than Slow R-CNN at training time and up to 20 times faster at test time.\n\n\n\n### Faster R-CNN: learning region proposals.\n\nEven though that Fast R-CNN is much faster than Slow R-CNN most of the compute time is dominated by region proposals! Region proposals are computed by an heuristic (Selective search algorithm) which runs on CPU. We could learn region proposals with a CNN instead!\n\n\nFaster R-CNN inserts a **Region Proposal Network (RPN)** to predict proposals from features. Otherwise, it is the same as Fast R-CNN. RPNs use \"anchor boxes\" of fixed size at each point in the feature map. For each anchor box we predict whether the corresponding anchor contains an object or not and a box transform to regress from anchor box to object box.\n\nFaster R-CNN has 4 train losses!\n\n1. RPN classification: anchor box is object or not object.\n2. RPN regression: predict transform from anchor box to proposal box.\n3. Object classification: classify proposals as background or object class.\n4. Object regression: predict transform from proposal box to object box.\n\nOn a GPU Faster R-CNN is up to 10 times faster than Fast R-CNN.\n\n\n\n\n### Finally... Semantic Segmentation!\n\nSemantic segmentation's objective is to label each pixel in the image with a category label. It does not care about differentiating instances, it only cares about pixels. What this means, for example, is that if we have to adjacent categories next to each other (e.g: two cows next to each other) they will not be classified as two different instances but rather one single cloud of points that belong to the same category (one cloud/blob of points which belong to the class \"cow\" but does not tell us which pixels belong to which cow).\n\n\n### Semantic segmentation: fully convolutional networks\n\nThe architecture used in Semantic Segmentation is very much the same as Faster RCNN but with one more branch at the end called the **mask branch**.\n\nWhat is commonly used in semantic segmentations mask branchs are Fully Convolutional Networks. This is a convolutional network architecture that does not have any fully connected layers nor pooling layers but rather only have convolutional layers. In this way an input image with a fixed size is passed through all these convolutional layers until on the final layer we perfom a `softmax()` on each pixel that gives us a set of class scores for every pixel followed by an `argmax()` that classifies each pixel to a category. We can train this using a cross entropy loss on each pixel.\n\nIn practice, what is used is convolutional networks with downsampling and upsampling, like the famous U-Net. These architectures have high resolution layers followed by mid-resolution layers followed by low-resolution layers on the downsampling stage and the other way round on the upsampling stage.\n\n\n\n\n### Mask R-CNN\n\nMask R-CNN is just like Faster R-CNN but with an extra branch that predicts a mask.\n\n\n\nSo the way this works in summary is:\n1. We input our RGB image into the CNN backbone (this can be any CNN, like EfficientNet) which outputs image-level features.\n2. We pass the image-level features to the Region Proposal Network (RPN) to get the region proposals.\n3. For each region proposal we pass it through a small semantic segmentation network (like Unet) to get the prediction masks.\n4. There are many branches involved!\n - Two for the RPN: one to detect the ROIs bounding boxes and another to detect if an object is present in the ROI.\n - One for classification to detect which object is present.\n - One to detect the bounding box.\n - One to detect the mask.\n \n### Where can I find segmentation models?\n\n\nSome popular Python libraries are [PyTorch Segmentation Models](https://github.com/qubvel/segmentation_models.pytorch) and [OpenMMLab Semantic Segmentation](https://github.com/open-mmlab/mmsegmentation). \n\nExample of a Segmentation Network in `segmentation_models_pytorch`:\n```python\nimport segmentation_models_pytorch as smp\n\nmodel = smp.Unet(\n encoder_name=\"resnet34\", # choose encoder, e.g. mobilenet_v2 or efficientnet-b7\n encoder_weights=\"imagenet\", # use `imagenet` pre-trained weights for encoder initialization\n in_channels=1, # model input channels (1 for gray-scale images, 3 for RGB, etc.)\n classes=3, # model output channels (number of classes in your dataset)\n)\n```\n\n### Conclusion\n\nSemantic Segmentation is a Computer Vision task where we want to classify each pixel in an image to detect an object and its shape/mask. It's similar to Object Detection but with one extra step!\n\nIn this competition our input is an RGB image and its corresponding mask: an image with the same dimension as the original but with 1s and 0s (1 to indicate where the object is present and 0 where it's not). The output is the predicted mask.\n\nSome popular architectures are UNets and you can find models implemented in PyTorch in `segmentation_models_pytorch`.\n\nSome common loss functions used in Segmentation tasks are [Dice Loss](https://smp.readthedocs.io/en/latest/losses.html#diceloss) and [Jaccard Loss](https://smp.readthedocs.io/en/latest/losses.html#jaccardloss). Some common metrics used are [Dice Coefficient](https://pbs.twimg.com/media/EWci84bWsAIBWvV?format=png&name=small) and [IoU Coefficient](https://miro.medium.com/v2/resize:fit:1400/1*FSauyBPV0fiVa_BUVutVTg.png).\n\nHope you liked it! 🍀\n\n\n### References\n\nSlides were taken from Justin Johnson's amazing Computer Vision course. You can find the course [here](https://www.youtube.com/playlist?list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r).",
"2193696": "Thanks @alejopaullier for sharing a very useful source of knowledge. Nice elaboration full of insights👍",
"2264021": "Excellent explanation @alejopaullier"
},
"source": "meta"
}