Many people have noted the issue of a Submission- / Scoring-error when submitting a dummy prediction, such as '1 0', which corresponds to an empty mask, see for example this post by Harshit or this post by Djata.
\nI have taken a look at the implementation of the surface dice metric linked in the competition overview and it seems that the formula they use is $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}}{y+y_{pred}}$$ (applied componentwise, then summed over all entries).
\nHowever, if both $y$ and $y_{pred}$ are $0$, then we may face an issue with this metric, since the expression evaluates to $0/0$. Typically, you'd add a \"smoothing parameter\" $\\epsilon$ and compute the metric via $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}+\\epsilon}{y+y_{pred}+\\epsilon}$$ to avoid division by $0$.
\nI'm not 100% certain that this is the issue, I just wanted to throw it into the discussion to hear what you guys think.
\nHere's the relevant snippet of the code linked above:
\n\n", "messages": [ { "id": "2523108", "postDate": "11/13/2023 09:04:08", "content": "distances_gt_to_pred = surface_distances[\"distances_gt_to_pred\"]
\n
\n distances_pred_to_gt = surface_distances[\"distances_pred_to_gt\"]
\n surfel_areas_gt = surface_distances[\"surfel_areas_gt\"]
\n surfel_areas_pred = surface_distances[\"surfel_areas_pred\"]
\n overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm])
\n overlap_pred = np.sum(
\n surfel_areas_pred[distances_pred_to_gt <= tolerance_mm])
\n surface_dice = (overlap_gt + overlap_pred) / (np.sum(surfel_areas_gt) +
\n np.sum(surfel_areas_pred))
\n return surface_dice
Many people have noted the issue of a Submission- / Scoring-error when submitting a dummy prediction, such as '1 0', which corresponds to an empty mask, see for example this post by Harshit or this post by Djata.
\nI have taken a look at the implementation of the surface dice metric linked in the competition overview and it seems that the formula they use is $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}}{y+y_{pred}}$$ (applied componentwise, then summed over all entries).
\nHowever, if both $y$ and $y_{pred}$ are $0$, then we may face an issue with this metric, since the expression evaluates to $0/0$. Typically, you'd add a \"smoothing parameter\" $\\epsilon$ and compute the metric via $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}+\\epsilon}{y+y_{pred}+\\epsilon}$$ to avoid division by $0$.
\nI'm not 100% certain that this is the issue, I just wanted to throw it into the discussion to hear what you guys think.
\nHere's the relevant snippet of the code linked above:
\n\n", "rawMarkdown": "Many people have noted the issue of a Submission- / Scoring-error when submitting a dummy prediction, such as '1 0', which corresponds to an empty mask, see for example [this post by Harshit](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455001) or [this post by Djata](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455008). \n\nI have taken a look at the [implementation of the surface dice metric](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) linked in the competition overview and it seems that the formula they use is $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}}{y+y_{pred}}$$ (applied componentwise, then summed over all entries). \n\nHowever, if both $y$ and $y_{pred}$ are $0$, then we may face an issue with this metric, since the expression evaluates to $0/0$. Typically, you'd add a \"smoothing parameter\" $\\epsilon$ and compute the metric via $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}+\\epsilon}{y+y_{pred}+\\epsilon}$$ to avoid division by $0$. \n\nI'm not 100% certain that this is the issue, I just wanted to throw it into the discussion to hear what you guys think. \n\nHere's the relevant snippet of the code linked above: \n\n> distances_gt_to_pred = surface_distances[\"distances_gt_to_pred\"]\n distances_pred_to_gt = surface_distances[\"distances_pred_to_gt\"]\n surfel_areas_gt = surface_distances[\"surfel_areas_gt\"]\n surfel_areas_pred = surface_distances[\"surfel_areas_pred\"]\n overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm])\n overlap_pred = np.sum(\n surfel_areas_pred[distances_pred_to_gt <= tolerance_mm])\n surface_dice = (overlap_gt + overlap_pred) / (np.sum(surfel_areas_gt) +\n np.sum(surfel_areas_pred))\n return surface_dice", "votes": null }, { "id": "2529363", "postDate": "11/18/2023 06:55:50", "content": "distances_gt_to_pred = surface_distances[\"distances_gt_to_pred\"]
\n
\n distances_pred_to_gt = surface_distances[\"distances_pred_to_gt\"]
\n surfel_areas_gt = surface_distances[\"surfel_areas_gt\"]
\n surfel_areas_pred = surface_distances[\"surfel_areas_pred\"]
\n overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm])
\n overlap_pred = np.sum(
\n surfel_areas_pred[distances_pred_to_gt <= tolerance_mm])
\n surface_dice = (overlap_gt + overlap_pred) / (np.sum(surfel_areas_gt) +
\n np.sum(surfel_areas_pred))
\n return surface_dice
The implementation of the Surface-Dice metric in this competition could potentially lead to submission challenges, particularly due to its high sensitivity to precise segmentation of intricate vascular structures. Given the complexity of 3D Hierarchical Phase-Contrast Tomography (HiP-CT) scans of human kidneys, any minor inaccuracies in segmenting tiny blood vessels can significantly impact the metric, thereby affecting the evaluation of submissions. Additionally, the need for high precision in segmentation underlines the importance of developing robust models that can handle the variability inherent in the provided datasets.
", "rawMarkdown": "The implementation of the Surface-Dice metric in this competition could potentially lead to submission challenges, particularly due to its high sensitivity to precise segmentation of intricate vascular structures. Given the complexity of 3D Hierarchical Phase-Contrast Tomography (HiP-CT) scans of human kidneys, any minor inaccuracies in segmenting tiny blood vessels can significantly impact the metric, thereby affecting the evaluation of submissions. Additionally, the need for high precision in segmentation underlines the importance of developing robust models that can handle the variability inherent in the provided datasets.", "votes": null } ], "comments": [ { "id": 2529363, "author_name": "dhinkris", "author_url": "", "post_date": "11/18/2023 06:55:50", "content": "The implementation of the Surface-Dice metric in this competition could potentially lead to submission challenges, particularly due to its high sensitivity to precise segmentation of intricate vascular structures. Given the complexity of 3D Hierarchical Phase-Contrast Tomography (HiP-CT) scans of human kidneys, any minor inaccuracies in segmenting tiny blood vessels can significantly impact the metric, thereby affecting the evaluation of submissions. Additionally, the need for high precision in segmentation underlines the importance of developing robust models that can handle the variability inherent in the provided datasets.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2523108": "Many people have noted the issue of a Submission- / Scoring-error when submitting a dummy prediction, such as '1 0', which corresponds to an empty mask, see for example [this post by Harshit](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455001) or [this post by Djata](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455008). \n\nI have taken a look at the [implementation of the surface dice metric](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) linked in the competition overview and it seems that the formula they use is $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}}{y+y_{pred}}$$ (applied componentwise, then summed over all entries). \n\nHowever, if both $y$ and $y_{pred}$ are $0$, then we may face an issue with this metric, since the expression evaluates to $0/0$. Typically, you'd add a \"smoothing parameter\" $\\epsilon$ and compute the metric via $$\\text{DICE}(y,y_{pred})=\\frac{y\\cdot y_{pred}+\\epsilon}{y+y_{pred}+\\epsilon}$$ to avoid division by $0$. \n\nI'm not 100% certain that this is the issue, I just wanted to throw it into the discussion to hear what you guys think. \n\nHere's the relevant snippet of the code linked above: \n\n> distances_gt_to_pred = surface_distances[\"distances_gt_to_pred\"]\n distances_pred_to_gt = surface_distances[\"distances_pred_to_gt\"]\n surfel_areas_gt = surface_distances[\"surfel_areas_gt\"]\n surfel_areas_pred = surface_distances[\"surfel_areas_pred\"]\n overlap_gt = np.sum(surfel_areas_gt[distances_gt_to_pred <= tolerance_mm])\n overlap_pred = np.sum(\n surfel_areas_pred[distances_pred_to_gt <= tolerance_mm])\n surface_dice = (overlap_gt + overlap_pred) / (np.sum(surfel_areas_gt) +\n np.sum(surfel_areas_pred))\n return surface_dice", "2529363": "The implementation of the Surface-Dice metric in this competition could potentially lead to submission challenges, particularly due to its high sensitivity to precise segmentation of intricate vascular structures. Given the complexity of 3D Hierarchical Phase-Contrast Tomography (HiP-CT) scans of human kidneys, any minor inaccuracies in segmenting tiny blood vessels can significantly impact the metric, thereby affecting the evaluation of submissions. Additionally, the need for high precision in segmentation underlines the importance of developing robust models that can handle the variability inherent in the provided datasets." }, "source": "meta" }