I now understand that the task is not only labeling pixels (1=neuron, 0=background), but also separating the 1 pixels into non-overlapping components corresponding to individual neurons. My question is: How does the Evaluation mechanism, which talks about calculating a precision value at different IoU thresholds. Can someone please explain to me how this computes the correctness of object segmentation? Thanks in advance…
", "messages": [ { "id": "1557843", "postDate": "10/25/2021 21:25:03", "content": "I now understand that the task is not only labeling pixels (1=neuron, 0=background), but also separating the 1 pixels into non-overlapping components corresponding to individual neurons. My question is: How does the Evaluation mechanism, which talks about calculating a precision value at different IoU thresholds. Can someone please explain to me how this computes the correctness of object segmentation? Thanks in advance…
", "rawMarkdown": "I now understand that the task is not only labeling pixels (1=neuron, 0=background), but also separating the 1 pixels into non-overlapping components corresponding to individual neurons. My question is: How does the Evaluation mechanism, which talks about calculating a precision value at different IoU thresholds. Can someone please explain to me how this computes the correctness of object segmentation? Thanks in advance...", "votes": null }, { "id": "1558386", "postDate": "10/26/2021 07:41:51", "content": "To simplify it let’s say we only have one threshold.
\nYou find all pairs of (prediction, target) that overlap above that threshold and then calculate:
Now to maximize the precision value you want to bring down the values of FP and FN. In other words get as close as you can to one to one matching between predictions and targets.
", "rawMarkdown": "To simplify it let’s say we only have one threshold.\nYou find all pairs of (prediction, target) that overlap above that threshold and then calculate:\n - TP: number of matched predictions \n - FP: number of unmatched predictions\n - FN: number of unmatched targets\n\nNow to maximize the precision value you want to bring down the values of FP and FN. In other words get as close as you can to one to one matching between predictions and targets.", "votes": null }, { "id": "1558731", "postDate": "10/26/2021 13:14:14", "content": "Sorry to be thick, but I still don't understand. When you say \"all pairs\", do you mean \"all pairs of blobs\" or \"all pairs of pixels\"? By the way, it would be great if someone could publish a reference evaluator.
", "rawMarkdown": "Sorry to be thick, but I still don't understand. When you say \"all pairs\", do you mean \"all pairs of blobs\" or \"all pairs of pixels\"? By the way, it would be great if someone could publish a reference evaluator.", "votes": null }, { "id": "1558831", "postDate": "10/26/2021 14:13:54", "content": "I mean objects we are detecting - neural cells.
\nEach row in the input file (and submission) corresponds to a single object. Lets say we predicted 100 rows, and there are 200 target rows - we have 20000 potential pairs. Out of these maybe 60 have IoU above some threshold. Then our score for that threshold would be: 60 / (60 + 40 + 140)*
There is a notebook which implements the metric here: https://www.kaggle.com/theoviel/competition-metric-map-iou Though it doesn't handle overlapping targets. I'm going to share my implementation, which doesn't have that problem once I finish integrating it into my detectron training.
\n*Ignoring a rare case of 1:2 mapping
", "rawMarkdown": "I mean objects we are detecting - neural cells.\nEach row in the input file (and submission) corresponds to a single object. Lets say we predicted 100 rows, and there are 200 target rows - we have 20000 potential pairs. Out of these maybe 60 have IoU above some threshold. Then our score for that threshold would be: 60 / (60 + 40 + 140)*\n\nThere is a notebook which implements the metric here: https://www.kaggle.com/theoviel/competition-metric-map-iou Though it doesn't handle overlapping targets. I'm going to share my implementation, which doesn't have that problem once I finish integrating it into my detectron training.\n\n**Ignoring a rare case of [1:2 mapping](https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/280583)*", "votes": null } ], "comments": [ { "id": 1558386, "author_name": "slawekbiel", "author_url": "", "post_date": "10/26/2021 07:41:51", "content": "To simplify it let’s say we only have one threshold.
\nYou find all pairs of (prediction, target) that overlap above that threshold and then calculate:
Now to maximize the precision value you want to bring down the values of FP and FN. In other words get as close as you can to one to one matching between predictions and targets.
", "votes": null, "replies": [ { "id": 1558731, "author_name": "markalavin", "author_url": "", "post_date": "10/26/2021 13:14:14", "content": "Sorry to be thick, but I still don't understand. When you say \"all pairs\", do you mean \"all pairs of blobs\" or \"all pairs of pixels\"? By the way, it would be great if someone could publish a reference evaluator.
", "votes": null, "replies": [] }, { "id": 1558831, "author_name": "slawekbiel", "author_url": "", "post_date": "10/26/2021 14:13:54", "content": "I mean objects we are detecting - neural cells.
\nEach row in the input file (and submission) corresponds to a single object. Lets say we predicted 100 rows, and there are 200 target rows - we have 20000 potential pairs. Out of these maybe 60 have IoU above some threshold. Then our score for that threshold would be: 60 / (60 + 40 + 140)*
There is a notebook which implements the metric here: https://www.kaggle.com/theoviel/competition-metric-map-iou Though it doesn't handle overlapping targets. I'm going to share my implementation, which doesn't have that problem once I finish integrating it into my detectron training.
\n*Ignoring a rare case of 1:2 mapping
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1557843": "I now understand that the task is not only labeling pixels (1=neuron, 0=background), but also separating the 1 pixels into non-overlapping components corresponding to individual neurons. My question is: How does the Evaluation mechanism, which talks about calculating a precision value at different IoU thresholds. Can someone please explain to me how this computes the correctness of object segmentation? Thanks in advance...", "1558386": "To simplify it let’s say we only have one threshold.\nYou find all pairs of (prediction, target) that overlap above that threshold and then calculate:\n - TP: number of matched predictions \n - FP: number of unmatched predictions\n - FN: number of unmatched targets\n\nNow to maximize the precision value you want to bring down the values of FP and FN. In other words get as close as you can to one to one matching between predictions and targets.", "1558731": "Sorry to be thick, but I still don't understand. When you say \"all pairs\", do you mean \"all pairs of blobs\" or \"all pairs of pixels\"? By the way, it would be great if someone could publish a reference evaluator.", "1558831": "I mean objects we are detecting - neural cells.\nEach row in the input file (and submission) corresponds to a single object. Lets say we predicted 100 rows, and there are 200 target rows - we have 20000 potential pairs. Out of these maybe 60 have IoU above some threshold. Then our score for that threshold would be: 60 / (60 + 40 + 140)*\n\nThere is a notebook which implements the metric here: https://www.kaggle.com/theoviel/competition-metric-map-iou Though it doesn't handle overlapping targets. I'm going to share my implementation, which doesn't have that problem once I finish integrating it into my detectron training.\n\n**Ignoring a rare case of [1:2 mapping](https://www.kaggle.com/c/sartorius-cell-instance-segmentation/discussion/280583)*" }, "source": "meta" }