\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F291298%2Fb8afbe951cc83c92dfd405af7f112317%2FScreen%20Shot%202021-01-27%20at%2010.34.19%20AM.png?generation=1611772475581218&alt=media\")
As the images are \"weak labeled\", how can we do local evaluation without the label of the mask?
", "messages": [ { "id": 1172248, "postDate": "2021-01-27T10:16:34.510Z", "content": "As the images are \"weak labeled\", how can we do local evaluation without the label of the mask?
", "rawMarkdown": "As the images are \"weak labeled\", how can we do local evaluation without the label of the mask?", "votes": 9 }, { "id": 1172998, "postDate": "2021-01-27T16:39:22.063Z", "content": "Semantic segmentation with only weak classification labels β that's what makes this competition sexy π
", "rawMarkdown": "Semantic segmentation with only weak classification labels — that's what makes this competition sexy π", "votes": 7, "replies": [ { "id": 1173425, "postDate": "2021-01-27T21:37:30.287Z", "content": "Isnβt the competition for instance segmentation not semantic segmentation?
", "rawMarkdown": "Isnβt the competition for instance segmentation not semantic segmentation?", "votes": 2 }, { "id": 1173621, "postDate": "2021-01-28T02:30:57.307Z", "content": "Soooo sexy, never met a competition like this π€
", "rawMarkdown": "Soooo sexy, never met a competition like this π€" } ] }, { "id": 1172968, "postDate": "2021-01-27T16:22:07.190Z", "content": "I had the same question @snaker. At first glance it looks like a multi-label classification challenge but when I saw this with the evaluation tab:
\n\n\nThe OpenImages version of the metric is described in detail here. See also this tutorial on running the evaluation in Python, with the only difference being the use of F1 rather than average precision.
\nSegmentation is calculated using IoU with a threshold of 0.6
\n
so it leads me to believe its label + masks that are used for the evaluation but there are not ground truth / bounding boxes masks provided.
\nI could be mistaken though and looking to much into it.
\nUPDATE:
\nif I actually click into the link that describes the metric in detail it leads here - https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval
and there is a description that include ground truth masks:
\n\n\nFor each image:
\nFirst, all detection masks are matched to all existing ground-truth masks. Two masks are considered as a potential match if mask-to-mask IoU > 0.5. Matched detections are considered True Positives.
\n
\n The remaining unmatched detections are now matched with ground-truth boxes that do not have a corresponding mask and that are not marked as group-of. Two such boxes are considered a potential match if box-to-box IoU > 0.5. Matched detections are ignored. These are probably true detections, but no ground-truth mask is available to evaluate in detail.
\n The remaining unmatched detections are finally matched to ground-truth group-of boxes. Two such boxes are considered a potential match if box-to-box intersection over area > 0.5 (like in the object detection case, multiple detections can match the same group-of box). Matched detections are ignored. The remaining non-matched detections do not match any mask nor box, and are thus considered False Positives.After this three-stage matching all detections have been tagged as True Positive, False Positive or to be ignored, and precision/recall values can be computed to generate per-class AP values.
\n
and they state the only difference is at the end here instead of using AP we use F1. I think another difference would also be that mask-to-mask IoU > 0.6.
", "rawMarkdown": "I had the same question @snaker. At first glance it looks like a multi-label classification challenge but when I saw this with the evaluation tab:\n\n> The OpenImages version of the metric is described in detail here. See also this tutorial on running the evaluation in Python, with the only difference being the use of F1 rather than average precision.\n\n> Segmentation is calculated using IoU with a threshold of 0.6\n\nso it leads me to believe its label + masks that are used for the evaluation but there are not ground truth / bounding boxes masks provided.\n\nI could be mistaken though and looking to much into it.\n\n**UPDATE**:\nif I actually click into the link that describes the metric in detail it leads here - https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval\n\nand there is a description that include ground truth masks:\n\n> For each image:\n\n> First, all detection masks are matched to all existing ground-truth masks. Two masks are considered as a potential match if mask-to-mask IoU > 0.5. Matched detections are considered True Positives.\nThe remaining unmatched detections are now matched with ground-truth boxes that do not have a corresponding mask and that are not marked as group-of. Two such boxes are considered a potential match if box-to-box IoU > 0.5. Matched detections are ignored. These are probably true detections, but no ground-truth mask is available to evaluate in detail.\nThe remaining unmatched detections are finally matched to ground-truth group-of boxes. Two such boxes are considered a potential match if box-to-box intersection over area > 0.5 (like in the object detection case, multiple detections can match the same group-of box). Matched detections are ignored. The remaining non-matched detections do not match any mask nor box, and are thus considered False Positives.\n\n> After this three-stage matching all detections have been tagged as True Positive, False Positive or to be ignored, and precision/recall values can be computed to generate per-class AP values.\n\nand they state the only difference is at the end here instead of using AP we use F1. I think another difference would also be that mask-to-mask IoU > 0.6.", "votes": 1, "replies": [ { "id": 1173074, "postDate": "2021-01-27T17:08:28.027Z", "content": "This is a weakly-labeled multi-labeled classification challenge. And you need to do 2 tasks here: identify the cell (segmentation) and classify it. The metrics works like this: For each image, all predicted masks are matched with ground-truth masks (this is not provided, but you can easily get this by any type of segmentation model, an example here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. . Then mAP is calculated. If the model predict a class 0 wrongly for cell 1 in image A, this is counted as 1 False Positive for class 0. If the model predicts a class on a region of the image that is not a cell, this is also counted as 1 False Positive for that class.
", "rawMarkdown": "This is a weakly-labeled multi-labeled classification challenge. And you need to do 2 tasks here: identify the cell (segmentation) and classify it. The metrics works like this: For each image, all predicted masks are matched with ground-truth masks (this is not provided, but you can easily get this by any type of segmentation model, an example here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. ~~Then F1 is calculated for each class on these cells. Then macro-F1 is calculated for all classes. So ideally you want to maximize the macro-F1 of all cells in the images with defined patterns (high confidence score for a class)~~. Then mAP is calculated. If the model predict a class 0 wrongly for cell 1 in image A, this is counted as 1 False Positive for class 0. If the model predicts a class on a region of the image that is not a cell, this is also counted as 1 False Positive for that class. ", "votes": 6 }, { "id": 1173229, "postDate": "2021-01-27T18:34:45.110Z", "content": "Thank you for the explanation @lnhtrang. The link is leading to a 404 error though:
\n
Updated! typo ')' in the link :)
", "rawMarkdown": "Updated! typo ')' in the link :)", "votes": 1 }, { "id": 1173676, "postDate": "2021-01-28T03:51:08.453Z", "content": "Thanks for the great explaination. Just one question
\n\n\n\"you need to do 2 tasks here: identify the cell (segmentation) and classify it\"
\n
Actually we are classifying every pixel of the image right?
", "rawMarkdown": "Thanks for the great explaination. Just one question\n\n> \"you need to do 2 tasks here: identify the cell (segmentation) and classify it\"\n\nActually we are classifying every pixel of the image right?" }, { "id": 1174316, "postDate": "2021-01-28T12:01:34.783Z", "content": "You can certainly view the problem as such!
", "rawMarkdown": "You can certainly view the problem as such!" }, { "id": 1174380, "postDate": "2021-01-28T12:29:15.797Z", "content": "It depends on your approach to solve the problem. What we are asking you to submit is labels per cell (not per pixel).
", "rawMarkdown": "It depends on your approach to solve the problem. What we are asking you to submit is labels per cell (not per pixel)." }, { "id": 1179923, "postDate": "2021-02-01T01:27:54.683Z", "content": "Does this mean that every single cell should just belongs to ony one cell type? (so the pixel type of one cell should all be the same type)
", "rawMarkdown": "Does this mean that every single cell should just belongs to ony one cell type? (so the pixel type of one cell should all be the same type)" }, { "id": 1180264, "postDate": "2021-02-01T07:35:57.830Z", "content": "In the challenge dataset there are 17 different cell types. All cells in one image are always the same cell type. The task is perform 'protein organelle classification label' per cell, which may differ between the cells in one image.
", "rawMarkdown": "In the challenge dataset there are 17 different cell types. All cells in one image are always the same cell type. The task is perform 'protein organelle classification label' per cell, which may differ between the cells in one image." }, { "id": 1180659, "postDate": "2021-02-01T11:59:10.303Z", "content": "Hey.
\nWe will be using instance segmentation to perform segmentation (pixel wise prediction) to identify masks for all cells in an image.
\nYou will then have to predict, for a given instance/cell, what the labels are (which organelles is the protein of interest found in).
", "rawMarkdown": "Hey.\n\nWe will be using instance segmentation to perform segmentation (pixel wise prediction) to identify masks for all cells in an image.\n\nYou will then have to predict, for a given instance/cell, what the labels are (which organelles is the protein of interest found in)." }, { "id": 1220281, "postDate": "2021-02-27T20:00:15.677Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1172998, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2021-01-27T16:39:22.063000", "content": "Semantic segmentation with only weak classification labels β that's what makes this competition sexy π
", "votes": 7, "replies": [ { "id": 1173425, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-01-27T21:37:30.287000", "content": "Isnβt the competition for instance segmentation not semantic segmentation?
", "votes": 2, "replies": [] }, { "id": 1173621, "author_name": "Chenglu", "author_url": "", "post_date": "2021-01-28T02:30:57.307000", "content": "Soooo sexy, never met a competition like this π€
", "votes": 0, "replies": [] } ] }, { "id": 1172968, "author_name": "RDizzl3", "author_url": "", "post_date": "2021-01-27T16:22:07.190000", "content": "I had the same question @snaker. At first glance it looks like a multi-label classification challenge but when I saw this with the evaluation tab:
\n\n\nThe OpenImages version of the metric is described in detail here. See also this tutorial on running the evaluation in Python, with the only difference being the use of F1 rather than average precision.
\nSegmentation is calculated using IoU with a threshold of 0.6
\n
so it leads me to believe its label + masks that are used for the evaluation but there are not ground truth / bounding boxes masks provided.
\nI could be mistaken though and looking to much into it.
\nUPDATE:
\nif I actually click into the link that describes the metric in detail it leads here - https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval
and there is a description that include ground truth masks:
\n\n\nFor each image:
\nFirst, all detection masks are matched to all existing ground-truth masks. Two masks are considered as a potential match if mask-to-mask IoU > 0.5. Matched detections are considered True Positives.
\n
\n The remaining unmatched detections are now matched with ground-truth boxes that do not have a corresponding mask and that are not marked as group-of. Two such boxes are considered a potential match if box-to-box IoU > 0.5. Matched detections are ignored. These are probably true detections, but no ground-truth mask is available to evaluate in detail.
\n The remaining unmatched detections are finally matched to ground-truth group-of boxes. Two such boxes are considered a potential match if box-to-box intersection over area > 0.5 (like in the object detection case, multiple detections can match the same group-of box). Matched detections are ignored. The remaining non-matched detections do not match any mask nor box, and are thus considered False Positives.After this three-stage matching all detections have been tagged as True Positive, False Positive or to be ignored, and precision/recall values can be computed to generate per-class AP values.
\n
and they state the only difference is at the end here instead of using AP we use F1. I think another difference would also be that mask-to-mask IoU > 0.6.
", "votes": 1, "replies": [ { "id": 1173074, "author_name": "Trang Le", "author_url": "", "post_date": "2021-01-27T17:08:28.027000", "content": "This is a weakly-labeled multi-labeled classification challenge. And you need to do 2 tasks here: identify the cell (segmentation) and classify it. The metrics works like this: For each image, all predicted masks are matched with ground-truth masks (this is not provided, but you can easily get this by any type of segmentation model, an example here https://www.kaggle.com/lnhtrang/hpa-public-data-download-and-hpacellseg. . Then mAP is calculated. If the model predict a class 0 wrongly for cell 1 in image A, this is counted as 1 False Positive for class 0. If the model predicts a class on a region of the image that is not a cell, this is also counted as 1 False Positive for that class.
", "votes": 6, "replies": [] }, { "id": 1173229, "author_name": "RDizzl3", "author_url": "", "post_date": "2021-01-27T18:34:45.110000", "content": "Thank you for the explanation @lnhtrang. The link is leading to a 404 error though:
\n
Updated! typo ')' in the link :)
", "votes": 1, "replies": [] }, { "id": 1173676, "author_name": "Chenglu", "author_url": "", "post_date": "2021-01-28T03:51:08.453000", "content": "Thanks for the great explaination. Just one question
\n\n\n\"you need to do 2 tasks here: identify the cell (segmentation) and classify it\"
\n
Actually we are classifying every pixel of the image right?
", "votes": 0, "replies": [] }, { "id": 1174316, "author_name": "Trang Le", "author_url": "", "post_date": "2021-01-28T12:01:34.783000", "content": "You can certainly view the problem as such!
", "votes": 0, "replies": [] }, { "id": 1174380, "author_name": "Emma Lundberg", "author_url": "", "post_date": "2021-01-28T12:29:15.797000", "content": "It depends on your approach to solve the problem. What we are asking you to submit is labels per cell (not per pixel).
", "votes": 0, "replies": [] }, { "id": 1179923, "author_name": "Chenglu", "author_url": "", "post_date": "2021-02-01T01:27:54.683000", "content": "Does this mean that every single cell should just belongs to ony one cell type? (so the pixel type of one cell should all be the same type)
", "votes": 0, "replies": [] }, { "id": 1180264, "author_name": "Trang Le", "author_url": "", "post_date": "2021-02-01T07:35:57.830000", "content": "In the challenge dataset there are 17 different cell types. All cells in one image are always the same cell type. The task is perform 'protein organelle classification label' per cell, which may differ between the cells in one image.
", "votes": 0, "replies": [] }, { "id": 1180659, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-02-01T11:59:10.303000", "content": "Hey.
\nWe will be using instance segmentation to perform segmentation (pixel wise prediction) to identify masks for all cells in an image.
\nYou will then have to predict, for a given instance/cell, what the labels are (which organelles is the protein of interest found in).
", "votes": 0, "replies": [] }, { "id": 1220281, "author_name": "", "author_url": "", "post_date": "2021-02-27T20:00:15.677000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1172248": "As the images are \"weak labeled\", how can we do local evaluation without the label of the mask?", "1172998": "Semantic segmentation with only weak classification labels — that's what makes this competition sexy π", "1172968": "I had the same question @snaker. At first glance it looks like a multi-label classification challenge but when I saw this with the evaluation tab:\n\n> The OpenImages version of the metric is described in detail here. See also this tutorial on running the evaluation in Python, with the only difference being the use of F1 rather than average precision.\n\n> Segmentation is calculated using IoU with a threshold of 0.6\n\nso it leads me to believe its label + masks that are used for the evaluation but there are not ground truth / bounding boxes masks provided.\n\nI could be mistaken though and looking to much into it.\n\n**UPDATE**:\nif I actually click into the link that describes the metric in detail it leads here - https://storage.googleapis.com/openimages/web/evaluation.html#instance_segmentation_eval\n\nand there is a description that include ground truth masks:\n\n> For each image:\n\n> First, all detection masks are matched to all existing ground-truth masks. Two masks are considered as a potential match if mask-to-mask IoU > 0.5. Matched detections are considered True Positives.\nThe remaining unmatched detections are now matched with ground-truth boxes that do not have a corresponding mask and that are not marked as group-of. Two such boxes are considered a potential match if box-to-box IoU > 0.5. Matched detections are ignored. These are probably true detections, but no ground-truth mask is available to evaluate in detail.\nThe remaining unmatched detections are finally matched to ground-truth group-of boxes. Two such boxes are considered a potential match if box-to-box intersection over area > 0.5 (like in the object detection case, multiple detections can match the same group-of box). Matched detections are ignored. The remaining non-matched detections do not match any mask nor box, and are thus considered False Positives.\n\n> After this three-stage matching all detections have been tagged as True Positive, False Positive or to be ignored, and precision/recall values can be computed to generate per-class AP values.\n\nand they state the only difference is at the end here instead of using AP we use F1. I think another difference would also be that mask-to-mask IoU > 0.6." } }