When actually taking CT scans, it is expected that information such as resolution and beamline is already known. Therefore, when performing segmentation using a model, these data should be available and inference can be performed.
\nWhy haven't these details been provided? Of course, it's possible to work on this issue as a challenge, but there's a concern that it might lead to insignificant changes in rankings and negatively impact the competition.
\nIs there a practical intention behind this?
", "messages": [ { "id": "2536358", "postDate": "11/24/2023 06:28:17", "content": "When actually taking CT scans, it is expected that information such as resolution and beamline is already known. Therefore, when performing segmentation using a model, these data should be available and inference can be performed.
\nWhy haven't these details been provided? Of course, it's possible to work on this issue as a challenge, but there's a concern that it might lead to insignificant changes in rankings and negatively impact the competition.
\nIs there a practical intention behind this?
", "rawMarkdown": "When actually taking CT scans, it is expected that information such as resolution and beamline is already known. Therefore, when performing segmentation using a model, these data should be available and inference can be performed.\n\nWhy haven't these details been provided? Of course, it's possible to work on this issue as a challenge, but there's a concern that it might lead to insignificant changes in rankings and negatively impact the competition. \n\nIs there a practical intention behind this?", "votes": null }, { "id": "2536556", "postDate": "11/24/2023 09:24:42", "content": "Hi, thanks for the question. So I'm not sure if you read the preprint that we just posted. I suggest you take a look at it, and carefully read the sections about HiP-CT as a technique and the discussion and conclusion, this should clarify our intentions.
\nA short and incomplete answer would be: We are able to scan human kidneys with increasing speed thanks to the development of the HiP-CT technique. This means we have scanned and continue to scan many human kidneys, all of which have anatomical variations as well as some changes in CNR, SNR and achievable resolution, due to the setup improvements we have been making to push the HiP-CT technique to its maximum potential. What is really needed to realise the capability of HiP-CT, and impact on human health, is to develop a good and rhobust model that is as generalizable as possible to HiP-CT imaging of human kidney.
Thank you for such a quick and detailed response!
\nI am a radiologist and see kidney images daily, so I understand that there are many individual differences and each person has optimal imaging conditions. However, I am unclear about the situation where metadata cannot be used.
\nI understand the intention to efficiently segment images under unknown imaging conditions. However, what does it mean to acquire medical images and not be able to reference the metadata?
\nOf course, I understand that it cannot be disclosed beforehand, but I do not understand why it cannot be used even under confidential conditions during inference.
\nIn this regard, for example, one could create a model to estimate imaging conditions to switch segmentation models depend on the conditions, but it’s difficult to imagine scenarios where the approach would be useful outside of the competition.
\nWhat I am trying to say is that, similar to other competitions, being able to reference metadata during inference would perhaps be closer to a real-world environment.
\nP.S.
\nI was unaware of HiP-CT until I joined this competition, and I am astounded by its high quality. I might believe it if I were told they are actual cross-sectional images of a kidney. It’s an amazing technology!
Hello @clairewalsh,
\nThanks for organizing the competition.
\nCould you please share your views on the current metric limitations. It has been shown here and here that the current implementation won't accept \"complex\" solutions simply for a metric computation reason. Two solutions running on the same hardware and time limitations won't be considered equally in this competition: the more \"complex\" one will be discarded disregarding the fact the it would generate a better solution according to the metric chosen by organizers.
Don't you think it would be a shame that the best solution for this problem won't be awarded because of technical issues that won't have any impact in real applications ? I'm trying to get your thoughts here because organizers are the ones that can make Kaggle invest time in changing this and make this competition fairer. The sooner this gets fixed, the more kagglers will join in and the better the final solution will be: I think it is in your best interest to make sure that this is fixed soon.
\nIt's also important for me to know how much time I'll invest in this competition. :)
", "rawMarkdown": "Hello @clairewalsh,\n\nThanks for organizing the competition.\nCould you please share your views on the current metric limitations. It has been shown [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456761) and [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455220) that the current implementation won't accept \"complex\" solutions simply for a metric computation reason. Two solutions running on the same hardware and time limitations won't be considered equally in this competition: the more \"complex\" one will be discarded disregarding the fact the it would generate a better solution according to the metric chosen by organizers.\n\nDon't you think it would be a shame that the best solution for this problem won't be awarded because of technical issues that won't have any impact in real applications ? I'm trying to get your thoughts here because organizers are the ones that can make Kaggle invest time in changing this and make this competition fairer. The sooner this gets fixed, the more kagglers will join in and the better the final solution will be: I think it is in your best interest to make sure that this is fixed soon.\n\nIt's also important for me to know how much time I'll invest in this competition. :)", "votes": null }, { "id": "2536937", "postDate": "11/24/2023 15:43:55", "content": "Thanks for this, I appreciate the feedback. We are discussing as a team.
", "rawMarkdown": "Thanks for this, I appreciate the feedback. We are discussing as a team.", "votes": null }, { "id": "2537037", "postDate": "11/24/2023 17:19:00", "content": "As someone who was only able to sabmit my baseline for the 11th time, it seems to me that the problem is that OOM is not getting more complex solutions, but solutions that give non-continues segmentation on all three axes.
\nA complex solution does not mean a high-quality solution. I may be wrong, but so far it seems to me that the problem is not in the complexity of the solution, but in the non-continuity of the mask.
OOM is only related to the size of the 3D surface delineated by the predictions, called border_pred in the metric’s code. “Simple” or “complex” predictions is indeed a non sense, just a way of talking. “Non-continuity” as you say will indeed create a larger border in 3D as it multiplies “holes”. They are many ways of creating OOM, multiple small cubes will create that as well.
\nIt does not really matter what solutions creates OOM, any binary predictions should be scored no matter what -> only the metric should be the judge of what is a good or bad solution.
", "rawMarkdown": "OOM is only related to the size of the 3D surface delineated by the predictions, called border_pred in the metric’s code. “Simple” or “complex” predictions is indeed a non sense, just a way of talking. “Non-continuity” as you say will indeed create a larger border in 3D as it multiplies “holes”. They are many ways of creating OOM, multiple small cubes will create that as well.\n\nIt does not really matter what solutions creates OOM, any binary predictions should be scored no matter what -> only the metric should be the judge of what is a good or bad solution.", "votes": null }, { "id": "2537226", "postDate": "11/24/2023 23:13:13", "content": "Now I'm not sure we're talking about the same thing. Are you talking about Submission Scoring Error?
", "rawMarkdown": "Now I'm not sure we're talking about the same thing. Are you talking about Submission Scoring Error?", "votes": null }, { "id": "2538420", "postDate": "11/26/2023 04:57:23", "content": "in my case complex model with a threshold of 0.3 gives Submission scoring error
", "rawMarkdown": "in my case complex model with a threshold of 0.3 gives Submission scoring error", "votes": null } ], "comments": [ { "id": 2536556, "author_name": "clairewalsh", "author_url": "", "post_date": "11/24/2023 09:24:42", "content": "Hi, thanks for the question. So I'm not sure if you read the preprint that we just posted. I suggest you take a look at it, and carefully read the sections about HiP-CT as a technique and the discussion and conclusion, this should clarify our intentions.
\nA short and incomplete answer would be: We are able to scan human kidneys with increasing speed thanks to the development of the HiP-CT technique. This means we have scanned and continue to scan many human kidneys, all of which have anatomical variations as well as some changes in CNR, SNR and achievable resolution, due to the setup improvements we have been making to push the HiP-CT technique to its maximum potential. What is really needed to realise the capability of HiP-CT, and impact on human health, is to develop a good and rhobust model that is as generalizable as possible to HiP-CT imaging of human kidney.
Thank you for such a quick and detailed response!
\nI am a radiologist and see kidney images daily, so I understand that there are many individual differences and each person has optimal imaging conditions. However, I am unclear about the situation where metadata cannot be used.
\nI understand the intention to efficiently segment images under unknown imaging conditions. However, what does it mean to acquire medical images and not be able to reference the metadata?
\nOf course, I understand that it cannot be disclosed beforehand, but I do not understand why it cannot be used even under confidential conditions during inference.
\nIn this regard, for example, one could create a model to estimate imaging conditions to switch segmentation models depend on the conditions, but it’s difficult to imagine scenarios where the approach would be useful outside of the competition.
\nWhat I am trying to say is that, similar to other competitions, being able to reference metadata during inference would perhaps be closer to a real-world environment.
\nP.S.
\nI was unaware of HiP-CT until I joined this competition, and I am astounded by its high quality. I might believe it if I were told they are actual cross-sectional images of a kidney. It’s an amazing technology!
Hello @clairewalsh,
\nThanks for organizing the competition.
\nCould you please share your views on the current metric limitations. It has been shown here and here that the current implementation won't accept \"complex\" solutions simply for a metric computation reason. Two solutions running on the same hardware and time limitations won't be considered equally in this competition: the more \"complex\" one will be discarded disregarding the fact the it would generate a better solution according to the metric chosen by organizers.
Don't you think it would be a shame that the best solution for this problem won't be awarded because of technical issues that won't have any impact in real applications ? I'm trying to get your thoughts here because organizers are the ones that can make Kaggle invest time in changing this and make this competition fairer. The sooner this gets fixed, the more kagglers will join in and the better the final solution will be: I think it is in your best interest to make sure that this is fixed soon.
\nIt's also important for me to know how much time I'll invest in this competition. :)
", "votes": null, "replies": [ { "id": 2536937, "author_name": "clairewalsh", "author_url": "", "post_date": "11/24/2023 15:43:55", "content": "Thanks for this, I appreciate the feedback. We are discussing as a team.
", "votes": null, "replies": [ { "id": 2537037, "author_name": "viktoriaskorik", "author_url": "", "post_date": "11/24/2023 17:19:00", "content": "As someone who was only able to sabmit my baseline for the 11th time, it seems to me that the problem is that OOM is not getting more complex solutions, but solutions that give non-continues segmentation on all three axes.
\nA complex solution does not mean a high-quality solution. I may be wrong, but so far it seems to me that the problem is not in the complexity of the solution, but in the non-continuity of the mask.
OOM is only related to the size of the 3D surface delineated by the predictions, called border_pred in the metric’s code. “Simple” or “complex” predictions is indeed a non sense, just a way of talking. “Non-continuity” as you say will indeed create a larger border in 3D as it multiplies “holes”. They are many ways of creating OOM, multiple small cubes will create that as well.
\nIt does not really matter what solutions creates OOM, any binary predictions should be scored no matter what -> only the metric should be the judge of what is a good or bad solution.
", "votes": null, "replies": [ { "id": 2537226, "author_name": "viktoriaskorik", "author_url": "", "post_date": "11/24/2023 23:13:13", "content": "Now I'm not sure we're talking about the same thing. Are you talking about Submission Scoring Error?
", "votes": null, "replies": [] }, { "id": 2538420, "author_name": "arunodhayan", "author_url": "", "post_date": "11/26/2023 04:57:23", "content": "in my case complex model with a threshold of 0.3 gives Submission scoring error
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2536358": "When actually taking CT scans, it is expected that information such as resolution and beamline is already known. Therefore, when performing segmentation using a model, these data should be available and inference can be performed.\n\nWhy haven't these details been provided? Of course, it's possible to work on this issue as a challenge, but there's a concern that it might lead to insignificant changes in rankings and negatively impact the competition. \n\nIs there a practical intention behind this?", "2536556": "Hi, thanks for the question. So I'm not sure if you read the preprint that we just posted. I suggest you take a look at it, and carefully read the sections about HiP-CT as a technique and the discussion and conclusion, this should clarify our intentions. \nA short and incomplete answer would be: We are able to scan human kidneys with increasing speed thanks to the development of the HiP-CT technique. This means we have scanned and continue to scan many human kidneys, all of which have anatomical variations as well as some changes in CNR, SNR and achievable resolution, due to the setup improvements we have been making to push the HiP-CT technique to its maximum potential. What is really needed to realise the capability of HiP-CT, and impact on human health, is to develop a good and rhobust model that is as generalizable as possible to HiP-CT imaging of human kidney.", "2536597": "Thank you for such a quick and detailed response!\n\nI am a radiologist and see kidney images daily, so I understand that there are many individual differences and each person has optimal imaging conditions. However, I am unclear about the situation where metadata cannot be used.\n\nI understand the intention to efficiently segment images under unknown imaging conditions. However, what does it mean to acquire medical images and not be able to reference the metadata?\n\nOf course, I understand that it cannot be disclosed beforehand, but I do not understand why it cannot be used even under confidential conditions during inference.\n\nIn this regard, for example, one could create a model to estimate imaging conditions to switch segmentation models depend on the conditions, but it’s difficult to imagine scenarios where the approach would be useful outside of the competition.\n\nWhat I am trying to say is that, similar to other competitions, being able to reference metadata during inference would perhaps be closer to a real-world environment.\n\nP.S.\nI was unaware of HiP-CT until I joined this competition, and I am astounded by its high quality. I might believe it if I were told they are actual cross-sectional images of a kidney. It’s an amazing technology!", "2536607": "Hello @clairewalsh,\n\nThanks for organizing the competition.\nCould you please share your views on the current metric limitations. It has been shown [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/456761) and [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/455220) that the current implementation won't accept \"complex\" solutions simply for a metric computation reason. Two solutions running on the same hardware and time limitations won't be considered equally in this competition: the more \"complex\" one will be discarded disregarding the fact the it would generate a better solution according to the metric chosen by organizers.\n\nDon't you think it would be a shame that the best solution for this problem won't be awarded because of technical issues that won't have any impact in real applications ? I'm trying to get your thoughts here because organizers are the ones that can make Kaggle invest time in changing this and make this competition fairer. The sooner this gets fixed, the more kagglers will join in and the better the final solution will be: I think it is in your best interest to make sure that this is fixed soon.\n\nIt's also important for me to know how much time I'll invest in this competition. :)", "2536937": "Thanks for this, I appreciate the feedback. We are discussing as a team.", "2537037": "As someone who was only able to sabmit my baseline for the 11th time, it seems to me that the problem is that OOM is not getting more complex solutions, but solutions that give non-continues segmentation on all three axes.\nA complex solution does not mean a high-quality solution. I may be wrong, but so far it seems to me that the problem is not in the complexity of the solution, but in the non-continuity of the mask.", "2537198": "OOM is only related to the size of the 3D surface delineated by the predictions, called border_pred in the metric’s code. “Simple” or “complex” predictions is indeed a non sense, just a way of talking. “Non-continuity” as you say will indeed create a larger border in 3D as it multiplies “holes”. They are many ways of creating OOM, multiple small cubes will create that as well.\n\nIt does not really matter what solutions creates OOM, any binary predictions should be scored no matter what -> only the metric should be the judge of what is a good or bad solution.", "2537226": "Now I'm not sure we're talking about the same thing. Are you talking about Submission Scoring Error?", "2538420": "in my case complex model with a threshold of 0.3 gives Submission scoring error" }, "source": "meta" }