I seems that the top baseline: https://www.kaggle.com/nguynvnthtr/the-training-image-is-1024-x-1024 only uses data from kidney 1. I did try training on all kidneys but they didn't help (lb was wrose).
\nI didn't find any related discussion, I wander why data of kidney 2/3 doesn't help?
I seems that the top baseline: https://www.kaggle.com/nguynvnthtr/the-training-image-is-1024-x-1024 only uses data from kidney 1. I did try training on all kidneys but they didn't help (lb was wrose).
\nI didn't find any related discussion, I wander why data of kidney 2/3 doesn't help?
kidney 2 is sparsely labelled, if you use it as it's given you are passing incomplete labels to your loss. So your model will learn to ignore some vessels and will think is ok since it wont be penalized for his mistakes.
\nAnd there is also the fact that those notebooks works so well for the specific situation of his hypertuned hyperparameters for the public LB score.
one trick is to train with other self-supervsied target
", "rawMarkdown": "one trick is to train with other self-supervsied target", "votes": 2, "replies": [ { "id": 2599453, "postDate": "2024-01-12T20:40:57.180Z", "content": "I was thinking at simply, when I had a confident enough model, pseudolabel it toguether with the sparsed labels.
", "rawMarkdown": "I was thinking at simply, when I had a confident enough model, pseudolabel it toguether with the sparsed labels." }, { "id": 2602300, "postDate": "2024-01-15T03:55:57.167Z", "content": "It depends on computation power to quickly try some ideas, which is not ok for most users.
", "rawMarkdown": "It depends on computation power to quickly try some ideas, which is not ok for most users." }, { "id": 2602565, "postDate": "2024-01-15T07:39:31.933Z", "content": "I just tried using dense data of kidney 3 but lb was worse. May be kidney 1 data is somehow similar to public test data.
", "rawMarkdown": "I just tried using dense data of kidney 3 but lb was worse. May be kidney 1 data is somehow similar to public test data.", "votes": 1 }, { "id": 2602742, "postDate": "2024-01-15T10:10:11.263Z", "content": "not true
\nmodel1 : train = kidney1 dense
\nmodel2 : train = kidney1 dense + kidney3 dense
model 2 is definitely better than model1. the issue is how to validate model2 (since you do not have another dense data set)
\nfurther, due to lack of train/valid data, the correaltion of local cv and public is weak
", "rawMarkdown": "not true\n\nmodel1 : train = kidney1 dense\nmodel2 : train = kidney1 dense + kidney3 dense \n\n---\n\nmodel 2 is definitely better than model1. the issue is how to validate model2 (since you do not have another dense data set)\n\n---\n\nfurther, due to lack of train/valid data, the correaltion of local cv and public is weak", "votes": 1 }, { "id": 2603795, "postDate": "2024-01-16T04:08:11.527Z", "content": "Are your model1 and model2 with same config/hyperparams? I just checked my code but no bug was found. I'm sure that with 501 kidney 3 dense samples my lb went worse(0.86 without kidney 3 and 0.853 with kidney 3).
", "rawMarkdown": "Are your model1 and model2 with same config/hyperparams? I just checked my code but no bug was found. I'm sure that with 501 kidney 3 dense samples my lb went worse(0.86 without kidney 3 and 0.853 with kidney 3)." }, { "id": 2603801, "postDate": "2024-01-16T04:28:03.930Z", "content": "@hengck23 what is your validation set for model 2? kidney_2? or your model 2 is retrained of model 1 without validation set?
", "rawMarkdown": "@hengck23 what is your validation set for model 2? kidney_2? or your model 2 is retrained of model 1 without validation set?" }, { "id": 2603825, "postDate": "2024-01-16T05:01:19.153Z", "content": "use kidney2 for validation.
\nbut need to pay attnetion to the metrics for monitoring.
\ne.g. becuase of sparse annotation, \"false positive\" may just missed annotation. (i.e. fp is not error)
Interesting! With kidney 3, the training dice score is higher and fp is lower, which is understandable, but lb is lower.
", "rawMarkdown": "Interesting! With kidney 3, the training dice score is higher and fp is lower, which is understandable, but lb is lower." }, { "id": 2607762, "postDate": "2024-01-18T12:16:38.417Z", "content": "@hengck23 how did you select kidney 2 for validation with slices >= 900?
", "rawMarkdown": "@hengck23 how did you select kidney 2 for validation with slices >= 900?" } ] }, { "id": 2602552, "postDate": "2024-01-15T07:29:48.167Z", "content": "Thank you so much for your wonderful discussion! What about kidney 3？I used kidney 3 for testing. Is that right?
", "rawMarkdown": "Thank you so much for your wonderful discussion! What about kidney 3？I used kidney 3 for testing. Is that right?" } ] }, { "id": 2603828, "postDate": "2024-01-16T05:07:53.337Z", "content": "anyone has results for train=kidney2 only, valid =xxx, lb= ???
\ni wonder if we just ignore the small vessel, what is the lb score
I did a small test with similar model, epochs, etc.
\ntrain only kidney 2 validate kidney 3 dense
\nvalidation 0.90
\nLB 0.77
train only kidney 1 validate kidney 3 dense
\nvalidation 0.87
\nLB 0.855
It seems that the public LB at least may be more similar to kidney 1 and kidney 3. Thought perhaps since kidney 2 has less annotation it may only learn the larger vessels well? Also think its image sizes are different to the others so how it is tiled, resized, etc. may have a different effect but may not be the same for 3D as 2D.
", "rawMarkdown": "I did a small test with similar model, epochs, etc. \n\ntrain only kidney 2 validate kidney 3 dense \nvalidation 0.90\nLB 0.77\n\ntrain only kidney 1 validate kidney 3 dense\nvalidation 0.87\nLB 0.855\n\nIt seems that the public LB at least may be more similar to kidney 1 and kidney 3. Thought perhaps since kidney 2 has less annotation it may only learn the larger vessels well? Also think its image sizes are different to the others so how it is tiled, resized, etc. may have a different effect but may not be the same for 3D as 2D. ", "votes": 4, "replies": [ { "id": 2607947, "postDate": "2024-01-18T14:11:49.703Z", "content": "I shared a similar experiment with quite different CV - LB results here
", "rawMarkdown": "I shared a similar experiment with quite different CV - LB results [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/468525#2607904)", "votes": 2 } ] } ] } ], "comments": [ { "id": 2599040, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-01-12T16:14:33.567000", "content": "kidney 2 is sparsely labelled, if you use it as it's given you are passing incomplete labels to your loss. So your model will learn to ignore some vessels and will think is ok since it wont be penalized for his mistakes.
\nAnd there is also the fact that those notebooks works so well for the specific situation of his hypertuned hyperparameters for the public LB score.
one trick is to train with other self-supervsied target
", "votes": 2, "replies": [ { "id": 2599453, "author_name": "Ángel Jacinto Sánchez Ruiz", "author_url": "", "post_date": "2024-01-12T20:40:57.180000", "content": "I was thinking at simply, when I had a confident enough model, pseudolabel it toguether with the sparsed labels.
", "votes": 0, "replies": [] }, { "id": 2602300, "author_name": "dragon zhang", "author_url": "", "post_date": "2024-01-15T03:55:57.167000", "content": "It depends on computation power to quickly try some ideas, which is not ok for most users.
", "votes": 0, "replies": [] }, { "id": 2602565, "author_name": "Dylan Liu", "author_url": "", "post_date": "2024-01-15T07:39:31.933000", "content": "I just tried using dense data of kidney 3 but lb was worse. May be kidney 1 data is somehow similar to public test data.
", "votes": 1, "replies": [] }, { "id": 2602742, "author_name": "hengck23", "author_url": "", "post_date": "2024-01-15T10:10:11.263000", "content": "not true
\nmodel1 : train = kidney1 dense
\nmodel2 : train = kidney1 dense + kidney3 dense
model 2 is definitely better than model1. the issue is how to validate model2 (since you do not have another dense data set)
\nfurther, due to lack of train/valid data, the correaltion of local cv and public is weak
", "votes": 1, "replies": [] }, { "id": 2603795, "author_name": "Dylan Liu", "author_url": "", "post_date": "2024-01-16T04:08:11.527000", "content": "Are your model1 and model2 with same config/hyperparams? I just checked my code but no bug was found. I'm sure that with 501 kidney 3 dense samples my lb went worse(0.86 without kidney 3 and 0.853 with kidney 3).
", "votes": 0, "replies": [] }, { "id": 2603801, "author_name": "FGPC", "author_url": "", "post_date": "2024-01-16T04:28:03.930000", "content": "@hengck23 what is your validation set for model 2? kidney_2? or your model 2 is retrained of model 1 without validation set?
", "votes": 0, "replies": [] }, { "id": 2603825, "author_name": "hengck23", "author_url": "", "post_date": "2024-01-16T05:01:19.153000", "content": "use kidney2 for validation.
\nbut need to pay attnetion to the metrics for monitoring.
\ne.g. becuase of sparse annotation, \"false positive\" may just missed annotation. (i.e. fp is not error)
Interesting! With kidney 3, the training dice score is higher and fp is lower, which is understandable, but lb is lower.
", "votes": 0, "replies": [] }, { "id": 2607762, "author_name": "tabgen1", "author_url": "", "post_date": "2024-01-18T12:16:38.417000", "content": "@hengck23 how did you select kidney 2 for validation with slices >= 900?
", "votes": 0, "replies": [] } ] }, { "id": 2602552, "author_name": "Mingfeng Li_CPU", "author_url": "", "post_date": "2024-01-15T07:29:48.167000", "content": "Thank you so much for your wonderful discussion! What about kidney 3？I used kidney 3 for testing. Is that right?
", "votes": 0, "replies": [] } ] }, { "id": 2603828, "author_name": "hengck23", "author_url": "", "post_date": "2024-01-16T05:07:53.337000", "content": "anyone has results for train=kidney2 only, valid =xxx, lb= ???
\ni wonder if we just ignore the small vessel, what is the lb score
I did a small test with similar model, epochs, etc.
\ntrain only kidney 2 validate kidney 3 dense
\nvalidation 0.90
\nLB 0.77
train only kidney 1 validate kidney 3 dense
\nvalidation 0.87
\nLB 0.855
It seems that the public LB at least may be more similar to kidney 1 and kidney 3. Thought perhaps since kidney 2 has less annotation it may only learn the larger vessels well? Also think its image sizes are different to the others so how it is tiled, resized, etc. may have a different effect but may not be the same for 3D as 2D.
", "votes": 4, "replies": [ { "id": 2607947, "author_name": "Optimo", "author_url": "", "post_date": "2024-01-18T14:11:49.703000", "content": "I shared a similar experiment with quite different CV - LB results here
", "votes": 2, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2598765": "I seems that the top baseline: https://www.kaggle.com/nguynvnthtr/the-training-image-is-1024-x-1024 only uses data from kidney 1. I did try training on all kidneys but they didn't help (lb was wrose).\nI didn't find any related discussion, I wander why data of kidney 2/3 doesn't help?", "2599040": "kidney 2 is sparsely labelled, if you use it as it's given you are passing incomplete labels to your loss. So your model will learn to ignore some vessels and will think is ok since it wont be penalized for his mistakes.\nAnd there is also the fact that those notebooks works so well for the specific situation of his hypertuned hyperparameters for the public LB score.", "2603828": "anyone has results for train=kidney2 only, valid =xxx, lb= ???\ni wonder if we just ignore the small vessel, what is the lb score" } }