One of the techniques that many people used early on (me included), was removing connected components smaller than a certain size. This typically helps if your output masks are somewhat noisy and might have some loose speckles or small disconnected regions that are inaccurate.
\n\nThe way most people began using it in this competition though was a little more than noise reduction. In this competition, many people were using thresholds high enough that it acted as a false positive rejector that would prevent a small mask prediction being a false positive. Only allowing the large predictions through meant that we had to be confident over a large region of an image that the cloud type was present. This was useful early on when our models weren't great but was a huge hindrance in the later stages of the competition because it made it so that it was physically impossible to get some of the images correct and in others damaged our masks severely.
\n\nLet me explain with some histograms.
\n\nHere is the distribution of non-empty cloud sizes\n.png?generation=1574120298593370&alt=media)
This graph is showing the cumulative distribution of the sum of the non-empty masks. We can determine from this graph that if we set our min_size high like some were doing with values up to 10k-20k you might be gimping your model and making it incapable of generating a mask for as much as ~16% of masks. This initially works when your model is weak because the filter will block any it is unclear about but once you have multiple models in ensemble and they are confident about a small region being a cloud it would still be impossible for it to get it correct. \n.png?generation=1574120625891547&alt=media)\nWe can see from the flower cloud type we would have completely lost this mask. Even with an ensemble of 10 models all saying that region has a flower type cloud it is impossible to get anything other than a 0/1 for that type.
That causes total failure of the model, but there is the even worse case when you have an a piece of a mask that falls under this size limit. Many of the masks are fragmented, either because of multiple annotators disagreeing or because of the stripe going across the images. \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1035002%2Fbbd61585a8213d737c83c2e3190825cc%2Fdownload.png?generation=1574120787827227&alt=media\")
Looking at the smallest non-zero mask connected component we can see that we are missing even more small segments via filtering like this. With a 20k threshold ~57% of images have a small chunk removed. You segmentation model may correctly locate both the major and minor pieces of the mask, but the small fragments on the opposite side of the strip will always be missed.
\n\n.png?generation=1574121319540529&alt=media)
Looking at the flower type cloud on this one we can see that it would be able to find the larger right portion but the left portion would probably be filtered out if you set a large enough min_size.
\n\nThis was the main contributor behind my jump in the leaderboard from ~300. I had tried many things and built ensembles but it was very difficult to break through that ceiling. In the end, I don't think the things Heng pointed out were really of much benefit to him. I think the primary boost he got was from using pixel max thresholding instead of min_size thresholding. Encoder, Decoder, Augmentation, different resolutions played no part in my final solution. It was primarily just a pile of lightweight resnet18 unet models trained with bce. Trying larger resolution, special losses, and tons of augmentation combinations didn't really contribute anything imo.
", "messages": [ { "id": "676049", "postDate": "11/18/2019 23:58:27", "content": "One of the techniques that many people used early on (me included), was removing connected components smaller than a certain size. This typically helps if your output masks are somewhat noisy and might have some loose speckles or small disconnected regions that are inaccurate.
\n\nThe way most people began using it in this competition though was a little more than noise reduction. In this competition, many people were using thresholds high enough that it acted as a false positive rejector that would prevent a small mask prediction being a false positive. Only allowing the large predictions through meant that we had to be confident over a large region of an image that the cloud type was present. This was useful early on when our models weren't great but was a huge hindrance in the later stages of the competition because it made it so that it was physically impossible to get some of the images correct and in others damaged our masks severely.
\n\nLet me explain with some histograms.
\n\nHere is the distribution of non-empty cloud sizes\n.png?generation=1574120298593370&alt=media)
This graph is showing the cumulative distribution of the sum of the non-empty masks. We can determine from this graph that if we set our min_size high like some were doing with values up to 10k-20k you might be gimping your model and making it incapable of generating a mask for as much as ~16% of masks. This initially works when your model is weak because the filter will block any it is unclear about but once you have multiple models in ensemble and they are confident about a small region being a cloud it would still be impossible for it to get it correct. \n.png?generation=1574120625891547&alt=media)\nWe can see from the flower cloud type we would have completely lost this mask. Even with an ensemble of 10 models all saying that region has a flower type cloud it is impossible to get anything other than a 0/1 for that type.
That causes total failure of the model, but there is the even worse case when you have an a piece of a mask that falls under this size limit. Many of the masks are fragmented, either because of multiple annotators disagreeing or because of the stripe going across the images. \n
Looking at the smallest non-zero mask connected component we can see that we are missing even more small segments via filtering like this. With a 20k threshold ~57% of images have a small chunk removed. You segmentation model may correctly locate both the major and minor pieces of the mask, but the small fragments on the opposite side of the strip will always be missed.
\n\n.png?generation=1574121319540529&alt=media)
Looking at the flower type cloud on this one we can see that it would be able to find the larger right portion but the left portion would probably be filtered out if you set a large enough min_size.
\n\nThis was the main contributor behind my jump in the leaderboard from ~300. I had tried many things and built ensembles but it was very difficult to break through that ceiling. In the end, I don't think the things Heng pointed out were really of much benefit to him. I think the primary boost he got was from using pixel max thresholding instead of min_size thresholding. Encoder, Decoder, Augmentation, different resolutions played no part in my final solution. It was primarily just a pile of lightweight resnet18 unet models trained with bce. Trying larger resolution, special losses, and tons of augmentation combinations didn't really contribute anything imo.
", "rawMarkdown": "One of the techniques that many people used early on (me included), was removing connected components smaller than a certain size. This typically helps if your output masks are somewhat noisy and might have some loose speckles or small disconnected regions that are inaccurate. \n\nThe way most people began using it in this competition though was a little more than noise reduction. In this competition, many people were using thresholds high enough that it acted as a false positive rejector that would prevent a small mask prediction being a false positive. Only allowing the large predictions through meant that we had to be confident over a large region of an image that the cloud type was present. This was useful early on when our models weren't great but was a huge hindrance in the later stages of the competition because it made it so that it was physically impossible to get some of the images correct and in others damaged our masks severely. \n\nLet me explain with some histograms. \n\nHere is the distribution of non-empty cloud sizes\n.png?generation=1574120298593370&alt=media)\n\nThis graph is showing the cumulative distribution of the sum of the non-empty masks. We can determine from this graph that if we set our min_size high like some were doing with values up to 10k-20k you might be gimping your model and making it incapable of generating a mask for as much as ~16% of masks. This initially works when your model is weak because the filter will block any it is unclear about but once you have multiple models in ensemble and they are confident about a small region being a cloud it would still be impossible for it to get it correct. \n.png?generation=1574120625891547&alt=media)\nWe can see from the flower cloud type we would have completely lost this mask. Even with an ensemble of 10 models all saying that region has a flower type cloud it is impossible to get anything other than a 0/1 for that type. \n\nThat causes total failure of the model, but there is the even worse case when you have an a piece of a mask that falls under this size limit. Many of the masks are fragmented, either because of multiple annotators disagreeing or because of the stripe going across the images. \n\n\nLooking at the smallest non-zero mask connected component we can see that we are missing even more small segments via filtering like this. With a 20k threshold ~57% of images have a small chunk removed. You segmentation model may correctly locate both the major and minor pieces of the mask, but the small fragments on the opposite side of the strip will always be missed. \n\n.png?generation=1574121319540529&alt=media)\n\nLooking at the flower type cloud on this one we can see that it would be able to find the larger right portion but the left portion would probably be filtered out if you set a large enough min_size. \n\nThis was the main contributor behind my jump in the leaderboard from ~300. I had tried many things and built ensembles but it was very difficult to break through that ceiling. In the end, I don't think the things Heng pointed out were really of much benefit to him. I think the primary boost he got was from using pixel max thresholding instead of min_size thresholding. Encoder, Decoder, Augmentation, different resolutions played no part in my final solution. It was primarily just a pile of lightweight resnet18 unet models trained with bce. Trying larger resolution, special losses, and tons of augmentation combinations didn't really contribute anything imo.", "votes": null }, { "id": "676108", "postDate": "11/19/2019 00:57:15", "content": "That's interesting. Thanks for pointing this out. I'm gonna run some tests on my models without min-size reduction and see if they improve.
\n\nIf you read the original paper here (first paragraph page 5),
\n\n\n\n\nParticipants had the possibility to draw any number of boxes, including none, with the caveat that the box would cover at least 10% of the image.
\n
you will see that rectangles needed to be at least 18375 pixels in a 350x525 mask. It seems that this was before the authors removed the black strips. So I guess the problem is if a small box overlaps the black strips then it results in a small mask.
", "rawMarkdown": "That's interesting. Thanks for pointing this out. I'm gonna run some tests on my models without min-size reduction and see if they improve.\n\nIf you read the original paper [here][1] (first paragraph page 5), \n\n> Participants had the possibility to draw any number of boxes, including none, with the caveat that the box would cover at least 10% of the image.\n\nyou will see that rectangles needed to be at least 18375 pixels in a 350x525 mask. It seems that this was before the authors removed the black strips. So I guess the problem is if a small box overlaps the black strips then it results in a small mask.\n\n[1]: https://arxiv.org/pdf/1906.01906.pdf", "votes": null }, { "id": "676150", "postDate": "11/19/2019 01:49:20", "content": "I did notice that in my holdout set as well.. Need to do more experiments on this :)
", "rawMarkdown": "I did notice that in my holdout set as well.. Need to do more experiments on this :)", "votes": null } ], "comments": [ { "id": 676108, "author_name": "cdeotte", "author_url": "", "post_date": "11/19/2019 00:57:15", "content": "That's interesting. Thanks for pointing this out. I'm gonna run some tests on my models without min-size reduction and see if they improve.
\n\nIf you read the original paper here (first paragraph page 5),
\n\n\n\n\nParticipants had the possibility to draw any number of boxes, including none, with the caveat that the box would cover at least 10% of the image.
\n
you will see that rectangles needed to be at least 18375 pixels in a 350x525 mask. It seems that this was before the authors removed the black strips. So I guess the problem is if a small box overlaps the black strips then it results in a small mask.
", "votes": null, "replies": [] }, { "id": 676150, "author_name": "phoenix9032", "author_url": "", "post_date": "11/19/2019 01:49:20", "content": "I did notice that in my holdout set as well.. Need to do more experiments on this :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "676049": "One of the techniques that many people used early on (me included), was removing connected components smaller than a certain size. This typically helps if your output masks are somewhat noisy and might have some loose speckles or small disconnected regions that are inaccurate. \n\nThe way most people began using it in this competition though was a little more than noise reduction. In this competition, many people were using thresholds high enough that it acted as a false positive rejector that would prevent a small mask prediction being a false positive. Only allowing the large predictions through meant that we had to be confident over a large region of an image that the cloud type was present. This was useful early on when our models weren't great but was a huge hindrance in the later stages of the competition because it made it so that it was physically impossible to get some of the images correct and in others damaged our masks severely. \n\nLet me explain with some histograms. \n\nHere is the distribution of non-empty cloud sizes\n.png?generation=1574120298593370&alt=media)\n\nThis graph is showing the cumulative distribution of the sum of the non-empty masks. We can determine from this graph that if we set our min_size high like some were doing with values up to 10k-20k you might be gimping your model and making it incapable of generating a mask for as much as ~16% of masks. This initially works when your model is weak because the filter will block any it is unclear about but once you have multiple models in ensemble and they are confident about a small region being a cloud it would still be impossible for it to get it correct. \n.png?generation=1574120625891547&alt=media)\nWe can see from the flower cloud type we would have completely lost this mask. Even with an ensemble of 10 models all saying that region has a flower type cloud it is impossible to get anything other than a 0/1 for that type. \n\nThat causes total failure of the model, but there is the even worse case when you have an a piece of a mask that falls under this size limit. Many of the masks are fragmented, either because of multiple annotators disagreeing or because of the stripe going across the images. \n\n\nLooking at the smallest non-zero mask connected component we can see that we are missing even more small segments via filtering like this. With a 20k threshold ~57% of images have a small chunk removed. You segmentation model may correctly locate both the major and minor pieces of the mask, but the small fragments on the opposite side of the strip will always be missed. \n\n.png?generation=1574121319540529&alt=media)\n\nLooking at the flower type cloud on this one we can see that it would be able to find the larger right portion but the left portion would probably be filtered out if you set a large enough min_size. \n\nThis was the main contributor behind my jump in the leaderboard from ~300. I had tried many things and built ensembles but it was very difficult to break through that ceiling. In the end, I don't think the things Heng pointed out were really of much benefit to him. I think the primary boost he got was from using pixel max thresholding instead of min_size thresholding. Encoder, Decoder, Augmentation, different resolutions played no part in my final solution. It was primarily just a pile of lightweight resnet18 unet models trained with bce. Trying larger resolution, special losses, and tons of augmentation combinations didn't really contribute anything imo.", "676108": "That's interesting. Thanks for pointing this out. I'm gonna run some tests on my models without min-size reduction and see if they improve.\n\nIf you read the original paper [here][1] (first paragraph page 5), \n\n> Participants had the possibility to draw any number of boxes, including none, with the caveat that the box would cover at least 10% of the image.\n\nyou will see that rectangles needed to be at least 18375 pixels in a 350x525 mask. It seems that this was before the authors removed the black strips. So I guess the problem is if a small box overlaps the black strips then it results in a small mask.\n\n[1]: https://arxiv.org/pdf/1906.01906.pdf", "676150": "I did notice that in my holdout set as well.. Need to do more experiments on this :)" }, "source": "meta" }