![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14675197%2Fd95b9fac815eb3027ed0e39255ca81f0%2Fscreen2.png?generation=1686219935732069&alt=media\")
Let's discuss CV vs LB scores here.
\nTo start with CV:0.494 LB:0.303 model: resnet50_mask_rcnn. I use the splits from this notebook
\nTBH so far I haven't yet found a good relation between CV and LB & neither a good method to compute CV. There might be something wrong with my CV calculation as well. I use COCO metric with IoU threshold set to 0.6 for CV calculation (this might not be the best method so feel free to correct me).
\nWhat are your CV vs LB score :) How are you measuring your CV :)
", "messages": [ { "id": "2290379", "postDate": "06/06/2023 18:20:46", "content": "Let's discuss CV vs LB scores here.
\nTo start with CV:0.494 LB:0.303 model: resnet50_mask_rcnn. I use the splits from this notebook
\nTBH so far I haven't yet found a good relation between CV and LB & neither a good method to compute CV. There might be something wrong with my CV calculation as well. I use COCO metric with IoU threshold set to 0.6 for CV calculation (this might not be the best method so feel free to correct me).
\nWhat are your CV vs LB score :) How are you measuring your CV :)
", "rawMarkdown": "Let's discuss CV vs LB scores here.\n\nTo start with CV:0.494 LB:0.303 model: resnet50_mask_rcnn. I use the splits from this [notebook](https://www.kaggle.com/code/benihime91/hubmap-2023-create-coco-annotations)\n\nTBH so far I haven't yet found a good relation between CV and LB & neither a good method to compute CV. There might be something wrong with my CV calculation as well. I use COCO metric with IoU threshold set to 0.6 for CV calculation (this might not be the best method so feel free to correct me).\n\nWhat are your CV vs LB score :) How are you measuring your CV :)", "votes": null }, { "id": "2290591", "postDate": "06/07/2023 01:04:32", "content": "LB 0.412:conf0.001 …val_coco-iou@0.5:0.95=0.39
\ndata=only from dataset1,only use \"blood_vessel\" (dismiss \"unsure\" label)
\nsplit :random 8:2
I still don't know what to do with the \"unsure\" label
\nThis split is not strictly correlated with cv/lb 😢
Dataet 1 and 2 have different proportions of blood_vessel and unsure, so I think the quality of annotation is quite different.
", "rawMarkdown": "LB 0.412:conf0.001 ...val_coco-iou@0.5:0.95=0.39 \n\ndata=only from dataset1,only use \"blood_vessel\" (dismiss \"unsure\" label)\nsplit :random 8:2\n\nI still don't know what to do with the \"unsure\" label\nThis split is not strictly correlated with cv/lb 😢\n\nDataet 1 and 2 have different proportions of blood_vessel and unsure, so I think the quality of annotation is quite different.", "votes": null }, { "id": "2290964", "postDate": "06/07/2023 08:20:03", "content": "I see you are using the default coco metric as validation , have you seen better correlaton with it? So far i have been using coco_metric@IoU=0.6 but correlation with LB is bad.
", "rawMarkdown": "I see you are using the default coco metric as validation , have you seen better correlaton with it? So far i have been using coco_metric@IoU=0.6 but correlation with LB is bad.", "votes": null }, { "id": "2291315", "postDate": "06/07/2023 13:20:13", "content": "I'm surprised that you can reach 0.4 with only 400 images from dataset1
", "rawMarkdown": "I'm surprised that you can reach 0.4 with only 400 images from dataset1", "votes": null }, { "id": "2292416", "postDate": "06/08/2023 10:27:06", "content": "It is not clear about glomerulus:
\n\"You should ensure none of your test set predictions occur within glomerulus structures as they will be counted as false positives.\"
Any predicted mask that intersects with the glomerulus at a non-zero IoU causes FP? This question to competition host, i think.
\nThe attached image shows the GT masks.
No good correlation. In fact, LB0.44 is overscored compared to val_coco-iou@0.5:0.95=0.406
", "rawMarkdown": "No good correlation. In fact, LB0.44 is overscored compared to val_coco-iou@0.5:0.95=0.406", "votes": null }, { "id": "2293643", "postDate": "06/09/2023 11:24:24", "content": "LB is iou@0.6, it's easier than iou@0.5:0.95
", "rawMarkdown": "LB is iou@0.6, it's easier than iou@0.5:0.95", "votes": null }, { "id": "2294020", "postDate": "06/09/2023 16:58:54", "content": "how do I evaluate mAP@0.6IOU how can I replicate submission evaluation
", "rawMarkdown": "how do I evaluate mAP@0.6IOU how can I replicate submission evaluation", "votes": null }, { "id": "2294034", "postDate": "06/09/2023 17:07:14", "content": "Disregarding glomeruli, this article was helpful for me: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
", "rawMarkdown": "Disregarding glomeruli, this article was helpful for me: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation", "votes": null }, { "id": "2297425", "postDate": "06/12/2023 15:09:05", "content": "Wouldn't it be around 338 images as 0.8 * 442 = 337.6?, so even less!
", "rawMarkdown": "Wouldn't it be around 338 images as 0.8 * 442 = 337.6?, so even less!", "votes": null }, { "id": "2312039", "postDate": "06/21/2023 16:18:53", "content": "\"I still don't know what to do with the \"unsure\" label\"
\nif you want, you can just a model to predict just the unsure.
\nthen submit the sure prediction. see how much unsure label contribute to score in hidden test
with simple coding, you can check if the vessel is partially or completely overlap with the glomerulus structure.
\ntrain a two class model for pixel label:
\nmake submission for one class and two class separately. see which has better score.
\nBut i think the difference are negligible
", "rawMarkdown": "with simple coding, you can check if the vessel is partially or completely overlap with the glomerulus structure.\n\n\ntrain a two class model for pixel label:\n1. vessel only\n2. vessel that are partially overlap\n\nmake submission for one class and two class separately. see which has better score.\n\nBut i think the difference are negligible", "votes": null }, { "id": "2320180", "postDate": "06/27/2023 14:55:12", "content": "so wired, i trained the mask-rcnn in one fold and local socre is 0.4+ but online only have 0.14
", "rawMarkdown": "so wired, i trained the mask-rcnn in one fold and local socre is 0.4+ but online only have 0.14", "votes": null } ], "comments": [ { "id": 2290591, "author_name": "abebe9849", "author_url": "", "post_date": "06/07/2023 01:04:32", "content": "LB 0.412:conf0.001 …val_coco-iou@0.5:0.95=0.39
\ndata=only from dataset1,only use \"blood_vessel\" (dismiss \"unsure\" label)
\nsplit :random 8:2
I still don't know what to do with the \"unsure\" label
\nThis split is not strictly correlated with cv/lb 😢
Dataet 1 and 2 have different proportions of blood_vessel and unsure, so I think the quality of annotation is quite different.
", "votes": null, "replies": [ { "id": 2290964, "author_name": "benihime91", "author_url": "", "post_date": "06/07/2023 08:20:03", "content": "I see you are using the default coco metric as validation , have you seen better correlaton with it? So far i have been using coco_metric@IoU=0.6 but correlation with LB is bad.
", "votes": null, "replies": [ { "id": 2293088, "author_name": "abebe9849", "author_url": "", "post_date": "06/09/2023 00:16:32", "content": "No good correlation. In fact, LB0.44 is overscored compared to val_coco-iou@0.5:0.95=0.406
", "votes": null, "replies": [ { "id": 2293643, "author_name": "ptran1203", "author_url": "", "post_date": "06/09/2023 11:24:24", "content": "LB is iou@0.6, it's easier than iou@0.5:0.95
", "votes": null, "replies": [] } ] } ] }, { "id": 2291315, "author_name": "ptran1203", "author_url": "", "post_date": "06/07/2023 13:20:13", "content": "I'm surprised that you can reach 0.4 with only 400 images from dataset1
", "votes": null, "replies": [ { "id": 2297425, "author_name": "camillagretschel", "author_url": "", "post_date": "06/12/2023 15:09:05", "content": "Wouldn't it be around 338 images as 0.8 * 442 = 337.6?, so even less!
", "votes": null, "replies": [] } ] }, { "id": 2294020, "author_name": "bibhabasumohapatra", "author_url": "", "post_date": "06/09/2023 16:58:54", "content": "how do I evaluate mAP@0.6IOU how can I replicate submission evaluation
", "votes": null, "replies": [ { "id": 2294034, "author_name": "tsobolev", "author_url": "", "post_date": "06/09/2023 17:07:14", "content": "Disregarding glomeruli, this article was helpful for me: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
", "votes": null, "replies": [] } ] }, { "id": 2312039, "author_name": "hengck23", "author_url": "", "post_date": "06/21/2023 16:18:53", "content": "\"I still don't know what to do with the \"unsure\" label\"
\nif you want, you can just a model to predict just the unsure.
\nthen submit the sure prediction. see how much unsure label contribute to score in hidden test
It is not clear about glomerulus:
\n\"You should ensure none of your test set predictions occur within glomerulus structures as they will be counted as false positives.\"
Any predicted mask that intersects with the glomerulus at a non-zero IoU causes FP? This question to competition host, i think.
\nThe attached image shows the GT masks.
with simple coding, you can check if the vessel is partially or completely overlap with the glomerulus structure.
\ntrain a two class model for pixel label:
\nmake submission for one class and two class separately. see which has better score.
\nBut i think the difference are negligible
", "votes": null, "replies": [] } ] }, { "id": 2320180, "author_name": "leehann", "author_url": "", "post_date": "06/27/2023 14:55:12", "content": "so wired, i trained the mask-rcnn in one fold and local socre is 0.4+ but online only have 0.14
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2290379": "Let's discuss CV vs LB scores here.\n\nTo start with CV:0.494 LB:0.303 model: resnet50_mask_rcnn. I use the splits from this [notebook](https://www.kaggle.com/code/benihime91/hubmap-2023-create-coco-annotations)\n\nTBH so far I haven't yet found a good relation between CV and LB & neither a good method to compute CV. There might be something wrong with my CV calculation as well. I use COCO metric with IoU threshold set to 0.6 for CV calculation (this might not be the best method so feel free to correct me).\n\nWhat are your CV vs LB score :) How are you measuring your CV :)", "2290591": "LB 0.412:conf0.001 ...val_coco-iou@0.5:0.95=0.39 \n\ndata=only from dataset1,only use \"blood_vessel\" (dismiss \"unsure\" label)\nsplit :random 8:2\n\nI still don't know what to do with the \"unsure\" label\nThis split is not strictly correlated with cv/lb 😢\n\nDataet 1 and 2 have different proportions of blood_vessel and unsure, so I think the quality of annotation is quite different.", "2290964": "I see you are using the default coco metric as validation , have you seen better correlaton with it? So far i have been using coco_metric@IoU=0.6 but correlation with LB is bad.", "2291315": "I'm surprised that you can reach 0.4 with only 400 images from dataset1", "2292416": "It is not clear about glomerulus:\n\"You should ensure none of your test set predictions occur within glomerulus structures as they will be counted as false positives.\"\n\nAny predicted mask that intersects with the glomerulus at a non-zero IoU causes FP? This question to competition host, i think.\nThe attached image shows the GT masks. \n\n", "2293088": "No good correlation. In fact, LB0.44 is overscored compared to val_coco-iou@0.5:0.95=0.406", "2293643": "LB is iou@0.6, it's easier than iou@0.5:0.95", "2294020": "how do I evaluate mAP@0.6IOU how can I replicate submission evaluation", "2294034": "Disregarding glomeruli, this article was helpful for me: https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation", "2297425": "Wouldn't it be around 338 images as 0.8 * 442 = 337.6?, so even less!", "2312039": "\"I still don't know what to do with the \"unsure\" label\"\n\n1. just ignore them in computation of loss for back propagation (i.e. these unsure pixels do not back propagate)\n2. if you think say 30% of unsure label are actually correct, you can set ground truth =0.3 for unsure\n\nif you want, you can just a model to predict just the unsure.\nthen submit the sure prediction. see how much unsure label contribute to score in hidden test", "2312048": "with simple coding, you can check if the vessel is partially or completely overlap with the glomerulus structure.\n\n\ntrain a two class model for pixel label:\n1. vessel only\n2. vessel that are partially overlap\n\nmake submission for one class and two class separately. see which has better score.\n\nBut i think the difference are negligible", "2320180": "so wired, i trained the mask-rcnn in one fold and local socre is 0.4+ but online only have 0.14" }, "source": "meta" }