Final update: the problem is now solved. It turns out that my clipping parameter derived from training data does not match the test data intensity. It may clip nearly the whole image to a constant value, and hence there is nothing to infer. Thanks for all the suggestions.
\n**
\nI have tried to use my submission notebook to generate submission.csv from my validation set (Kidney 3 Dense). I scored it using the official surface dice notebook and I got a score that looks fine. So I belive my submission notebook works ok. However, I still got a score of 0.0 when I made the submission. I have some questions that I think may cause the problem. I would be appreciated if anyone could kindly give me some help on them (I do not have much experience with Kaggle, so sorry if some of them look stupid).
\nDataloading: I think it is possible that I am not loading all the test images in my submission. In the official database in the notebook, there are only two kidneys with three slices each under the /kaggle/input/blood-vessel-segmentation/test folder. Can I safely assume that this folder will be placed with full data in the submission scoring? Is there a way to check it (or similarly, is there a way to check if I transverse /kaggle/input/blood-vessel-segmentation/test, I do transverse the full data during the scoring process, with correct folder structure so I am also getting correct dataset and slice name )?
Submission.csv: it could be the case that my generated submission.csv has problems. Is there a way to inspect the submission.csv generated during the submission scoring to see if anything is wrong?
Related to the above one. If I get a score rather than other error message, does it mean I generated submission.csv successfully? Is there a way to see if my notebook does generate this file during the scoring process?
Metrics calculation: the segmentation masked produced by my model may lead to problems in surface dice calculation. Could the current implementation of surface dice simply return 0 when it finds problems in the calculation of surface dice (i.e. it interrupts the calculation and simply returns 0 as the result when it catches some exception)? Is there a way to inspect if such case happens?
It is also possible some errors happen in my submission notebook during the scoring process, which interrupts the inference process but somehow the line that generates submission.csv still runs. This may give an incomplete submission.csv. I think I need to see error messages during the scoring process to find such errors. Is there a way to do it?
Thanks for your help!
\nUpdate 2023/12/19: I find the model predicts nearly no foreground on Kidney 2, although it works well on Kidney 3. I am now considering it more likely to be poor model training / data normalization statictics calculation due to a lack of training data.
", "messages": [ { "id": "2566091", "postDate": "12/18/2023 12:50:30", "content": "Final update: the problem is now solved. It turns out that my clipping parameter derived from training data does not match the test data intensity. It may clip nearly the whole image to a constant value, and hence there is nothing to infer. Thanks for all the suggestions.
\n**
\nI have tried to use my submission notebook to generate submission.csv from my validation set (Kidney 3 Dense). I scored it using the official surface dice notebook and I got a score that looks fine. So I belive my submission notebook works ok. However, I still got a score of 0.0 when I made the submission. I have some questions that I think may cause the problem. I would be appreciated if anyone could kindly give me some help on them (I do not have much experience with Kaggle, so sorry if some of them look stupid).
\nDataloading: I think it is possible that I am not loading all the test images in my submission. In the official database in the notebook, there are only two kidneys with three slices each under the /kaggle/input/blood-vessel-segmentation/test folder. Can I safely assume that this folder will be placed with full data in the submission scoring? Is there a way to check it (or similarly, is there a way to check if I transverse /kaggle/input/blood-vessel-segmentation/test, I do transverse the full data during the scoring process, with correct folder structure so I am also getting correct dataset and slice name )?
Submission.csv: it could be the case that my generated submission.csv has problems. Is there a way to inspect the submission.csv generated during the submission scoring to see if anything is wrong?
Related to the above one. If I get a score rather than other error message, does it mean I generated submission.csv successfully? Is there a way to see if my notebook does generate this file during the scoring process?
Metrics calculation: the segmentation masked produced by my model may lead to problems in surface dice calculation. Could the current implementation of surface dice simply return 0 when it finds problems in the calculation of surface dice (i.e. it interrupts the calculation and simply returns 0 as the result when it catches some exception)? Is there a way to inspect if such case happens?
It is also possible some errors happen in my submission notebook during the scoring process, which interrupts the inference process but somehow the line that generates submission.csv still runs. This may give an incomplete submission.csv. I think I need to see error messages during the scoring process to find such errors. Is there a way to do it?
Thanks for your help!
\nUpdate 2023/12/19: I find the model predicts nearly no foreground on Kidney 2, although it works well on Kidney 3. I am now considering it more likely to be poor model training / data normalization statictics calculation due to a lack of training data.
", "rawMarkdown": "Final update: the problem is now solved. It turns out that my clipping parameter derived from training data does not match the test data intensity. It may clip nearly the whole image to a constant value, and hence there is nothing to infer. Thanks for all the suggestions. \n\n**\n\nI have tried to use my submission notebook to generate submission.csv from my validation set (Kidney 3 Dense). I scored it using the official [surface dice notebook](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) and I got a score that looks fine. So I belive my submission notebook works ok. However, I still got a score of 0.0 when I made the submission. I have some questions that I think may cause the problem. I would be appreciated if anyone could kindly give me some help on them (I do not have much experience with Kaggle, so sorry if some of them look stupid).\n\n1. Dataloading: I think it is possible that I am not loading all the test images in my submission. In the official database in the notebook, there are only two kidneys with three slices each under the `/kaggle/input/blood-vessel-segmentation/test` folder. Can I safely assume that this folder will be placed with full data in the submission scoring? Is there a way to check it (or similarly, is there a way to check if I transverse `/kaggle/input/blood-vessel-segmentation/test`, I do transverse the full data during the scoring process, with correct folder structure so I am also getting correct dataset and slice name )?\n\n2. Submission.csv: it could be the case that my generated submission.csv has problems. Is there a way to inspect the submission.csv generated during the submission scoring to see if anything is wrong?\n\n3. Related to the above one. If I get a score rather than other error message, does it mean I generated submission.csv successfully? Is there a way to see if my notebook does generate this file during the scoring process?\n\n4. Metrics calculation: the segmentation masked produced by my model may lead to problems in surface dice calculation. Could the current implementation of surface dice simply return 0 when it finds problems in the calculation of surface dice (i.e. it interrupts the calculation and simply returns 0 as the result when it catches some exception)? Is there a way to inspect if such case happens?\n\n5. It is also possible some errors happen in my submission notebook during the scoring process, which interrupts the inference process but somehow the line that generates submission.csv still runs. This may give an incomplete submission.csv. I think I need to see error messages during the scoring process to find such errors. Is there a way to do it?\n\nThanks for your help!\n\n***\n\nUpdate 2023/12/19: I find the model predicts nearly no foreground on Kidney 2, although it works well on Kidney 3. I am now considering it more likely to be poor model training / data normalization statictics calculation due to a lack of training data.", "votes": null }, { "id": "2566342", "postDate": "12/18/2023 17:12:37", "content": "One possible issue I equally encountered before is data normalization during preprocessing. For some reason, I made a wrong assumption on what to use to divide the data in my submission script compared to the training/validation. This turns all my input slice images to nearly zeros leading to no reasonable result…. hence a zero score.
\nI will suggest you try to visualize the test data with your code. It will likely show all zeros….
", "rawMarkdown": "One possible issue I equally encountered before is data normalization during preprocessing. For some reason, I made a wrong assumption on what to use to divide the data in my submission script compared to the training/validation. This turns all my input slice images to nearly zeros leading to no reasonable result.... hence a zero score.\n\nI will suggest you try to visualize the test data with your code. It will likely show all zeros....", "votes": null }, { "id": "2566548", "postDate": "12/19/2023 00:31:55", "content": "Thank you for the suggestion. Since my inference notebook gives reasonable score on Kidney 3, I think the inference pipeline should be ok. But I find the model predicts nearly no foreground on Kidney 2. I am now suspecting this is a problem due to lack of training data rather than the online submission pipeline.
", "rawMarkdown": "Thank you for the suggestion. Since my inference notebook gives reasonable score on Kidney 3, I think the inference pipeline should be ok. But I find the model predicts nearly no foreground on Kidney 2. I am now suspecting this is a problem due to lack of training data rather than the online submission pipeline.", "votes": null }, { "id": "2566712", "postDate": "12/19/2023 05:29:14", "content": "Not sure what model etc. you are using, but some public nference notebooks for pytorch do load state dictionary with strict=False:
\nmodel.load_state_dict(state_dict, strict=False)
If there are a lot of missing keys then this could be the problem with zero or low submission scores. When you are scoring on validation set, if this is in the training notebook it might be OK on that model. But the model in the inference notebook could be different.
", "rawMarkdown": "Not sure what model etc. you are using, but some public nference notebooks for pytorch do load state dictionary with strict=False: \n`model.load_state_dict(state_dict, strict=False)`\n\nIf there are a lot of missing keys then this could be the problem with zero or low submission scores. When you are scoring on validation set, if this is in the training notebook it might be OK on that model. But the model in the inference notebook could be different.", "votes": null }, { "id": "2566896", "postDate": "12/19/2023 08:44:23", "content": "A closer inspection reveals that it is indeed the problem of the inference pipeline. My data clipping parameter is obtained from training data. It seems the test data intensity is quite different from the training data, so the clipping parameter leads to loss of information. Thanks again for your suggestion!
", "rawMarkdown": "A closer inspection reveals that it is indeed the problem of the inference pipeline. My data clipping parameter is obtained from training data. It seems the test data intensity is quite different from the training data, so the clipping parameter leads to loss of information. Thanks again for your suggestion!", "votes": null }, { "id": "2581267", "postDate": "12/31/2023 15:31:10", "content": "hi , i am getting the public score 0.0 .if possible pls help me .
\nnotebook link - https://www.kaggle.com/code/rohitdileep/unet-tensorflow
In my experience, it is really helpful to check the input/output of your network in inference. A possible way to find the problem is to run inference on the kidney data you have. In my case I found I get outputs of all-zeros when I ran inference on Kidney 2. Then a further investigation revealed that my clipping parameters obtained from Kidney 1 (my training split) was not suitbale for other Kidneys. Also this is only my case and other people may find different causes of the problem in their experiment. It might help to read how other people solve similar problems as well. You could check if the cause of their problem applies to your case.
", "rawMarkdown": "In my experience, it is really helpful to check the input/output of your network in inference. A possible way to find the problem is to run inference on the kidney data you have. In my case I found I get outputs of all-zeros when I ran inference on Kidney 2. Then a further investigation revealed that my clipping parameters obtained from Kidney 1 (my training split) was not suitbale for other Kidneys. Also this is only my case and other people may find different causes of the problem in their experiment. It might help to read how other people solve similar problems as well. You could check if the cause of their problem applies to your case.", "votes": null } ], "comments": [ { "id": 2566342, "author_name": "alabibojesomo", "author_url": "", "post_date": "12/18/2023 17:12:37", "content": "One possible issue I equally encountered before is data normalization during preprocessing. For some reason, I made a wrong assumption on what to use to divide the data in my submission script compared to the training/validation. This turns all my input slice images to nearly zeros leading to no reasonable result…. hence a zero score.
\nI will suggest you try to visualize the test data with your code. It will likely show all zeros….
", "votes": null, "replies": [ { "id": 2566548, "author_name": "czuryh", "author_url": "", "post_date": "12/19/2023 00:31:55", "content": "Thank you for the suggestion. Since my inference notebook gives reasonable score on Kidney 3, I think the inference pipeline should be ok. But I find the model predicts nearly no foreground on Kidney 2. I am now suspecting this is a problem due to lack of training data rather than the online submission pipeline.
", "votes": null, "replies": [ { "id": 2566896, "author_name": "czuryh", "author_url": "", "post_date": "12/19/2023 08:44:23", "content": "A closer inspection reveals that it is indeed the problem of the inference pipeline. My data clipping parameter is obtained from training data. It seems the test data intensity is quite different from the training data, so the clipping parameter leads to loss of information. Thanks again for your suggestion!
", "votes": null, "replies": [] } ] } ] }, { "id": 2566712, "author_name": "something4kag", "author_url": "", "post_date": "12/19/2023 05:29:14", "content": "Not sure what model etc. you are using, but some public nference notebooks for pytorch do load state dictionary with strict=False:
\nmodel.load_state_dict(state_dict, strict=False)
If there are a lot of missing keys then this could be the problem with zero or low submission scores. When you are scoring on validation set, if this is in the training notebook it might be OK on that model. But the model in the inference notebook could be different.
", "votes": null, "replies": [] }, { "id": 2581267, "author_name": "rohitdileep", "author_url": "", "post_date": "12/31/2023 15:31:10", "content": "hi , i am getting the public score 0.0 .if possible pls help me .
\nnotebook link - https://www.kaggle.com/code/rohitdileep/unet-tensorflow
In my experience, it is really helpful to check the input/output of your network in inference. A possible way to find the problem is to run inference on the kidney data you have. In my case I found I get outputs of all-zeros when I ran inference on Kidney 2. Then a further investigation revealed that my clipping parameters obtained from Kidney 1 (my training split) was not suitbale for other Kidneys. Also this is only my case and other people may find different causes of the problem in their experiment. It might help to read how other people solve similar problems as well. You could check if the cause of their problem applies to your case.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2566091": "Final update: the problem is now solved. It turns out that my clipping parameter derived from training data does not match the test data intensity. It may clip nearly the whole image to a constant value, and hence there is nothing to infer. Thanks for all the suggestions. \n\n**\n\nI have tried to use my submission notebook to generate submission.csv from my validation set (Kidney 3 Dense). I scored it using the official [surface dice notebook](https://www.kaggle.com/code/metric/surface-dice-metric/notebook) and I got a score that looks fine. So I belive my submission notebook works ok. However, I still got a score of 0.0 when I made the submission. I have some questions that I think may cause the problem. I would be appreciated if anyone could kindly give me some help on them (I do not have much experience with Kaggle, so sorry if some of them look stupid).\n\n1. Dataloading: I think it is possible that I am not loading all the test images in my submission. In the official database in the notebook, there are only two kidneys with three slices each under the `/kaggle/input/blood-vessel-segmentation/test` folder. Can I safely assume that this folder will be placed with full data in the submission scoring? Is there a way to check it (or similarly, is there a way to check if I transverse `/kaggle/input/blood-vessel-segmentation/test`, I do transverse the full data during the scoring process, with correct folder structure so I am also getting correct dataset and slice name )?\n\n2. Submission.csv: it could be the case that my generated submission.csv has problems. Is there a way to inspect the submission.csv generated during the submission scoring to see if anything is wrong?\n\n3. Related to the above one. If I get a score rather than other error message, does it mean I generated submission.csv successfully? Is there a way to see if my notebook does generate this file during the scoring process?\n\n4. Metrics calculation: the segmentation masked produced by my model may lead to problems in surface dice calculation. Could the current implementation of surface dice simply return 0 when it finds problems in the calculation of surface dice (i.e. it interrupts the calculation and simply returns 0 as the result when it catches some exception)? Is there a way to inspect if such case happens?\n\n5. It is also possible some errors happen in my submission notebook during the scoring process, which interrupts the inference process but somehow the line that generates submission.csv still runs. This may give an incomplete submission.csv. I think I need to see error messages during the scoring process to find such errors. Is there a way to do it?\n\nThanks for your help!\n\n***\n\nUpdate 2023/12/19: I find the model predicts nearly no foreground on Kidney 2, although it works well on Kidney 3. I am now considering it more likely to be poor model training / data normalization statictics calculation due to a lack of training data.", "2566342": "One possible issue I equally encountered before is data normalization during preprocessing. For some reason, I made a wrong assumption on what to use to divide the data in my submission script compared to the training/validation. This turns all my input slice images to nearly zeros leading to no reasonable result.... hence a zero score.\n\nI will suggest you try to visualize the test data with your code. It will likely show all zeros....", "2566548": "Thank you for the suggestion. Since my inference notebook gives reasonable score on Kidney 3, I think the inference pipeline should be ok. But I find the model predicts nearly no foreground on Kidney 2. I am now suspecting this is a problem due to lack of training data rather than the online submission pipeline.", "2566712": "Not sure what model etc. you are using, but some public nference notebooks for pytorch do load state dictionary with strict=False: \n`model.load_state_dict(state_dict, strict=False)`\n\nIf there are a lot of missing keys then this could be the problem with zero or low submission scores. When you are scoring on validation set, if this is in the training notebook it might be OK on that model. But the model in the inference notebook could be different.", "2566896": "A closer inspection reveals that it is indeed the problem of the inference pipeline. My data clipping parameter is obtained from training data. It seems the test data intensity is quite different from the training data, so the clipping parameter leads to loss of information. Thanks again for your suggestion!", "2581267": "hi , i am getting the public score 0.0 .if possible pls help me .\nnotebook link - https://www.kaggle.com/code/rohitdileep/unet-tensorflow", "2581682": "In my experience, it is really helpful to check the input/output of your network in inference. A possible way to find the problem is to run inference on the kidney data you have. In my case I found I get outputs of all-zeros when I ran inference on Kidney 2. Then a further investigation revealed that my clipping parameters obtained from Kidney 1 (my training split) was not suitbale for other Kidneys. Also this is only my case and other people may find different causes of the problem in their experiment. It might help to read how other people solve similar problems as well. You could check if the cause of their problem applies to your case." }, "source": "meta" }