I have been getting very decent Dice score for my training and validation sets, where I am using Kidney_1 and kidney_2 as train and kidney_3 as valid set. I am using a SegNet model. I am using a loss function which is a combination of binary cross entroy and dice loss, and also evaluating two metrics, IOU score and Dice Coefficient.
\nThe error for training is 0.3 and for valid set is 0.35
\nDice score for train is 0.85 and for valid is 0.73
\nIOU score for train is 0.91 and for valid is 0.85
But when I score my submission it always gives me \"0\" score.
\nWhat could be the reason that I am getting a 0 score?
\nAny help or suggestion would be much appreciated!
", "messages": [ { "id": "2541568", "postDate": "11/28/2023 15:41:10", "content": "I have been getting very decent Dice score for my training and validation sets, where I am using Kidney_1 and kidney_2 as train and kidney_3 as valid set. I am using a SegNet model. I am using a loss function which is a combination of binary cross entroy and dice loss, and also evaluating two metrics, IOU score and Dice Coefficient.
\nThe error for training is 0.3 and for valid set is 0.35
\nDice score for train is 0.85 and for valid is 0.73
\nIOU score for train is 0.91 and for valid is 0.85
But when I score my submission it always gives me \"0\" score.
\nWhat could be the reason that I am getting a 0 score?
\nAny help or suggestion would be much appreciated!
", "rawMarkdown": "I have been getting very decent Dice score for my training and validation sets, where I am using Kidney_1 and kidney_2 as train and kidney_3 as valid set. I am using a SegNet model. I am using a loss function which is a combination of binary cross entroy and dice loss, and also evaluating two metrics, IOU score and Dice Coefficient.\n\nThe error for training is 0.3 and for valid set is 0.35 \nDice score for train is 0.85 and for valid is 0.73\nIOU score for train is 0.91 and for valid is 0.85\n\nBut when I score my submission it always gives me \"0\" score.\n\nWhat could be the reason that I am getting a 0 score? \n\nAny help or suggestion would be much appreciated!", "votes": null }, { "id": "2541684", "postDate": "11/28/2023 17:20:46", "content": "Did you check how you model performs in 3D space?
\nFrom what you've described, I can assume that you're measuring the metric by slices (let's call them horizontal direction), but there's also a vertical direction
Yes, I am measuring the metric by slices.
\nI am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space?
I have seen some of the notebooks approaching it as 2D problem and getting a non-zero result on LB, shouldn't I at least get a very small non-zero score?
", "rawMarkdown": "Yes, I am measuring the metric by slices. \nI am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \n\nI have seen some of the notebooks approaching it as 2D problem and getting a non-zero result on LB, shouldn't I at least get a very small non-zero score?", "votes": null }, { "id": "2541835", "postDate": "11/28/2023 20:31:27", "content": "Not always a good metric in the horizontal direction gives a good result in the vertical direction. Each kidney is a connected structure, and continuity in depth is important (especially considering the metric of this competition, it bans solutions with big gaps in the vertical and horizontal direction).
\nYou can train a 2D model (I also train a 2D model, but I treat the task as a 3D one) and get a pretty good result, but you need to figure out how you will combine slides into a 3D solution with a continuous mask.
Thank you @viktoriaskorik !
\nI will look into some morphological connected analysis as post-processing.
One more question, I don't know if it is a stupid question, but here it is:
\nI am resizing my images and masks for training the model, let's say into 512x512 now when I predict segmentation mask from my trained model my mask would be 512x512, so I would have to convert it back into the same resolution as the inferred image was, so that when I convert my mask to rle, it gets evaluated with the rle of the original mask not the resized one? right?
\nSince I am submitting rle of a mask of size 512x512 that could be the reason I am getting the Dice Score 0? or should it give me an error instead of 0?
", "rawMarkdown": "Thank you @viktoriaskorik !\nI will look into some morphological connected analysis as post-processing.\n\nOne more question, I don't know if it is a stupid question, but here it is:\n\nI am resizing my images and masks for training the model, let's say into 512x512 now when I predict segmentation mask from my trained model my mask would be 512x512, so I would have to convert it back into the same resolution as the inferred image was, so that when I convert my mask to rle, it gets evaluated with the rle of the original mask not the resized one? right? \n\nSince I am submitting rle of a mask of size 512x512 that could be the reason I am getting the Dice Score 0? or should it give me an error instead of 0?", "votes": null }, { "id": "2544506", "postDate": "11/30/2023 21:39:30", "content": "\"I am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \"
\nif 2D perform well says, 100% accuracy then 3D won't perform better.
\nnow if 2D performs say 95%, 3D may perform at least 95% or better.
\n(assuming that you don't overfit in 3d etc ….)
why? we assume that given more information, you can make better decision
", "rawMarkdown": "\"I am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \"\n\nif 2D perform well says, 100% accuracy then 3D won't perform better.\n\nnow if 2D performs say 95%, 3D may perform at least 95% or better.\n(assuming that you don't overfit in 3d etc ....)\n\nwhy? we assume that given more information, you can make better decision", "votes": null }, { "id": "2544817", "postDate": "12/01/2023 05:10:32", "content": "You need to resize to the original size first, then encode rle. If you encode 512x512 mask into rle, there will be no error, because rle decoder will decode relative to the original size of the image, but it will decode incorrectly.
", "rawMarkdown": "You need to resize to the original size first, then encode rle. If you encode 512x512 mask into rle, there will be no error, because rle decoder will decode relative to the original size of the image, but it will decode incorrectly.", "votes": null }, { "id": "2545539", "postDate": "12/01/2023 14:54:05", "content": "I meant to say that if on 2D I am getting a really good score, on 3D I should get a better score than 0.000. I am not expecting a great score in 3D based on the 2D model.
\nBoth of you @hengck23 @viktoriaskorik have cleared out most of my confusions. I really appreciate the help and time.
", "rawMarkdown": "I meant to say that if on 2D I am getting a really good score, on 3D I should get a better score than 0.000. I am not expecting a great score in 3D based on the 2D model.\n\nBoth of you @hengck23 @viktoriaskorik have cleared out most of my confusions. I really appreciate the help and time.", "votes": null }, { "id": "2545547", "postDate": "12/01/2023 14:57:47", "content": "Yes, that was my mistake. I was wondering that I should be getting something above 0 atleast. I was not resizing my masks during inference into original size.
\nI really appreciate the feedback!
", "rawMarkdown": "Yes, that was my mistake. I was wondering that I should be getting something above 0 atleast. I was not resizing my masks during inference into original size. \n\nI really appreciate the feedback!", "votes": null }, { "id": "2588497", "postDate": "01/05/2024 14:35:07", "content": "I'm experiencing something like this right now as well. validation IoU coefficient is 0.9475 on about 45% of the train images and the corresponding dice coefficient is 0.9457 but getting score of 0.0000 despite increasing the threshold to about 85% confidence. what could be the cause.
\nthis is the notebook link for anyone that wishes to help https://www.kaggle.com/code/b45370/inferencenb
", "rawMarkdown": "I'm experiencing something like this right now as well. validation IoU coefficient is 0.9475 on about 45% of the train images and the corresponding dice coefficient is 0.9457 but getting score of 0.0000 despite increasing the threshold to about 85% confidence. what could be the cause.\n\nthis is the notebook link for anyone that wishes to help https://www.kaggle.com/code/b45370/inferencenb", "votes": null } ], "comments": [ { "id": 2541684, "author_name": "viktoriaskorik", "author_url": "", "post_date": "11/28/2023 17:20:46", "content": "Did you check how you model performs in 3D space?
\nFrom what you've described, I can assume that you're measuring the metric by slices (let's call them horizontal direction), but there's also a vertical direction
Yes, I am measuring the metric by slices.
\nI am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space?
I have seen some of the notebooks approaching it as 2D problem and getting a non-zero result on LB, shouldn't I at least get a very small non-zero score?
", "votes": null, "replies": [ { "id": 2541835, "author_name": "viktoriaskorik", "author_url": "", "post_date": "11/28/2023 20:31:27", "content": "Not always a good metric in the horizontal direction gives a good result in the vertical direction. Each kidney is a connected structure, and continuity in depth is important (especially considering the metric of this competition, it bans solutions with big gaps in the vertical and horizontal direction).
\nYou can train a 2D model (I also train a 2D model, but I treat the task as a 3D one) and get a pretty good result, but you need to figure out how you will combine slides into a 3D solution with a continuous mask.
Thank you @viktoriaskorik !
\nI will look into some morphological connected analysis as post-processing.
One more question, I don't know if it is a stupid question, but here it is:
\nI am resizing my images and masks for training the model, let's say into 512x512 now when I predict segmentation mask from my trained model my mask would be 512x512, so I would have to convert it back into the same resolution as the inferred image was, so that when I convert my mask to rle, it gets evaluated with the rle of the original mask not the resized one? right?
\nSince I am submitting rle of a mask of size 512x512 that could be the reason I am getting the Dice Score 0? or should it give me an error instead of 0?
", "votes": null, "replies": [ { "id": 2544506, "author_name": "hengck23", "author_url": "", "post_date": "11/30/2023 21:39:30", "content": "\"I am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \"
\nif 2D perform well says, 100% accuracy then 3D won't perform better.
\nnow if 2D performs say 95%, 3D may perform at least 95% or better.
\n(assuming that you don't overfit in 3d etc ….)
why? we assume that given more information, you can make better decision
", "votes": null, "replies": [ { "id": 2545539, "author_name": "salmankhaliq22", "author_url": "", "post_date": "12/01/2023 14:54:05", "content": "I meant to say that if on 2D I am getting a really good score, on 3D I should get a better score than 0.000. I am not expecting a great score in 3D based on the 2D model.
\nBoth of you @hengck23 @viktoriaskorik have cleared out most of my confusions. I really appreciate the help and time.
", "votes": null, "replies": [] } ] }, { "id": 2544817, "author_name": "viktoriaskorik", "author_url": "", "post_date": "12/01/2023 05:10:32", "content": "You need to resize to the original size first, then encode rle. If you encode 512x512 mask into rle, there will be no error, because rle decoder will decode relative to the original size of the image, but it will decode incorrectly.
", "votes": null, "replies": [ { "id": 2545547, "author_name": "salmankhaliq22", "author_url": "", "post_date": "12/01/2023 14:57:47", "content": "Yes, that was my mistake. I was wondering that I should be getting something above 0 atleast. I was not resizing my masks during inference into original size.
\nI really appreciate the feedback!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2588497, "author_name": "b45370", "author_url": "", "post_date": "01/05/2024 14:35:07", "content": "I'm experiencing something like this right now as well. validation IoU coefficient is 0.9475 on about 45% of the train images and the corresponding dice coefficient is 0.9457 but getting score of 0.0000 despite increasing the threshold to about 85% confidence. what could be the cause.
\nthis is the notebook link for anyone that wishes to help https://www.kaggle.com/code/b45370/inferencenb
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2541568": "I have been getting very decent Dice score for my training and validation sets, where I am using Kidney_1 and kidney_2 as train and kidney_3 as valid set. I am using a SegNet model. I am using a loss function which is a combination of binary cross entroy and dice loss, and also evaluating two metrics, IOU score and Dice Coefficient.\n\nThe error for training is 0.3 and for valid set is 0.35 \nDice score for train is 0.85 and for valid is 0.73\nIOU score for train is 0.91 and for valid is 0.85\n\nBut when I score my submission it always gives me \"0\" score.\n\nWhat could be the reason that I am getting a 0 score? \n\nAny help or suggestion would be much appreciated!", "2541684": "Did you check how you model performs in 3D space?\nFrom what you've described, I can assume that you're measuring the metric by slices (let's call them horizontal direction), but there's also a vertical direction", "2541788": "Yes, I am measuring the metric by slices. \nI am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \n\nI have seen some of the notebooks approaching it as 2D problem and getting a non-zero result on LB, shouldn't I at least get a very small non-zero score?", "2541835": "Not always a good metric in the horizontal direction gives a good result in the vertical direction. Each kidney is a connected structure, and continuity in depth is important (especially considering the metric of this competition, it bans solutions with big gaps in the vertical and horizontal direction).\nYou can train a 2D model (I also train a 2D model, but I treat the task as a 3D one) and get a pretty good result, but you need to figure out how you will combine slides into a 3D solution with a continuous mask.", "2544487": "Thank you @viktoriaskorik !\nI will look into some morphological connected analysis as post-processing.\n\nOne more question, I don't know if it is a stupid question, but here it is:\n\nI am resizing my images and masks for training the model, let's say into 512x512 now when I predict segmentation mask from my trained model my mask would be 512x512, so I would have to convert it back into the same resolution as the inferred image was, so that when I convert my mask to rle, it gets evaluated with the rle of the original mask not the resized one? right? \n\nSince I am submitting rle of a mask of size 512x512 that could be the reason I am getting the Dice Score 0? or should it give me an error instead of 0?", "2544506": "\"I am considering it a 2D problem, so if it is performing well for 2D slices wouldn't it do well for 3D space? \"\n\nif 2D perform well says, 100% accuracy then 3D won't perform better.\n\nnow if 2D performs say 95%, 3D may perform at least 95% or better.\n(assuming that you don't overfit in 3d etc ....)\n\nwhy? we assume that given more information, you can make better decision", "2544817": "You need to resize to the original size first, then encode rle. If you encode 512x512 mask into rle, there will be no error, because rle decoder will decode relative to the original size of the image, but it will decode incorrectly.", "2545539": "I meant to say that if on 2D I am getting a really good score, on 3D I should get a better score than 0.000. I am not expecting a great score in 3D based on the 2D model.\n\nBoth of you @hengck23 @viktoriaskorik have cleared out most of my confusions. I really appreciate the help and time.", "2545547": "Yes, that was my mistake. I was wondering that I should be getting something above 0 atleast. I was not resizing my masks during inference into original size. \n\nI really appreciate the feedback!", "2588497": "I'm experiencing something like this right now as well. validation IoU coefficient is 0.9475 on about 45% of the train images and the corresponding dice coefficient is 0.9457 but getting score of 0.0000 despite increasing the threshold to about 85% confidence. what could be the cause.\n\nthis is the notebook link for anyone that wishes to help https://www.kaggle.com/code/b45370/inferencenb" }, "source": "meta" }