![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F856355%2F6ffe902ddc7efeaddfbc65dd31c2c3f7%2F5631705451322_.pic.jpg?generation=1705451397035794&alt=media\")
I trained on K1 and val on K3, or train K1,K3 val on K2, both not found any correlation between CV and LB.
\nHas anyone use K2 sparse, K3 sparse or other mixture of them for train, what's the result?
I trained on K1 and val on K3, or train K1,K3 val on K2, both not found any correlation between CV and LB.
\nHas anyone use K2 sparse, K3 sparse or other mixture of them for train, what's the result?
the possible problem is the public/private dataset distribution? if they are same type of data.
\neither models trained on all data would be little bit lower on Public while stable on Private,
\nor models with 2 stages(classification and prediction) would be sustainable to shake up.
However, without local computation power, I am lazy to try out my simple ideas.
", "rawMarkdown": "the possible problem is the public/private dataset distribution? if they are same type of data. \neither models trained on all data would be little bit lower on Public while stable on Private,\nor models with 2 stages(classification and prediction) would be sustainable to shake up.\n\nHowever, without local computation power, I am lazy to try out my simple ideas.", "votes": null }, { "id": "2606272", "postDate": "01/17/2024 14:29:25", "content": "Can you please clarify what you mean by k2 and k2 sparse? So that I can share comparable results, thanks!
", "rawMarkdown": "Can you please clarify what you mean by k2 and k2 sparse? So that I can share comparable results, thanks!", "votes": null }, { "id": "2606278", "postDate": "01/17/2024 14:36:11", "content": "May be k2 is semipseudolabeled
", "rawMarkdown": "May be k2 is semipseudolabeled", "votes": null }, { "id": "2607125", "postDate": "01/18/2024 04:07:37", "content": "I mean use kidney2 to train or kidney3 sparse to train, what’s the difference between using kidney1 dense in LB
", "rawMarkdown": "I mean use kidney2 to train or kidney3 sparse to train, what’s the difference between using kidney1 dense in LB", "votes": null }, { "id": "2607654", "postDate": "01/18/2024 11:00:59", "content": "is your score for the entire k2 or k2 on slices > 900 as your teammate reports here ?
", "rawMarkdown": "is your score for the entire k2 or k2 on slices > 900 as your teammate reports [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/468826) ?", "votes": null }, { "id": "2607884", "postDate": "01/18/2024 13:49:22", "content": "Out of curiosity, have you tried training using different seeds and checking the LB vs CV correlation (for base models and averaged over seeds)?
", "rawMarkdown": "Out of curiosity, have you tried training using different seeds and checking the LB vs CV correlation (for base models and averaged over seeds)?", "votes": null }, { "id": "2607904", "postDate": "01/18/2024 13:58:17", "content": "Here are the results of one experiment I just did (k2 here is the original kidney with all slices), LB threshold is 0.5:
\n| Train | \nValid | \nk1_dense | \nk2 | \nk3_dense | \nLB (Th 0.5) | \n
|---|---|---|---|---|---|
| k1_dense, k3_dense | \nk2 | \n\n | Thresh 0.10: 0.7446 Thresh 0.20: 0.7780 Thresh 0.30: 0.7909 Thresh 0.35: 0.7941 Thresh 0.45: 0.7968 Thresh 0.50: 0.7970 Thresh 0.55: 0.7963 Thresh 0.60: 0.7940 Thresh 0.70: 0.7839 | \n\n | 0.799 | \n
| k2 | \nk1_dense, k3_dense | \nThresh 0.10: 0.6312 Thresh 0.20: 0.7462 Thresh 0.30: 0.8034 Thresh 0.35: 0.8215 Thresh 0.45: 0.8428 Thresh 0.50: 0.8477 Thresh 0.55: 0.8487 Thresh 0.60: 0.8459 Thresh 0.70: 0.8264 | \n\n | Thresh 0.10: 0.8047 Thresh 0.20: 0.8712 Thresh 0.30: 0.8991 Thresh 0.35: 0.9064 Thresh 0.45: 0.9122 Thresh 0.50: 0.9114 Thresh 0.55: 0.9084 Thresh 0.60: 0.9030 Thresh 0.70: 0.8835 | \n0.724 | \n
Still same experiment, threshold adapted to CV (0.2)
\n| Train | \nValid | \nk1_dense | \nk2 | \nk3_dense | \nLB (Th 0.2) | \n
|---|---|---|---|---|---|
| k1_dense | \nk2, k3_dense | \n\n | Thresh 0.10: 0.7884 Thresh 0.20: 0.8002 Thresh 0.30: 0.8008 Thresh 0.35: 0.7991 Thresh 0.45: 0.7930 Thresh 0.50: 0.7890 Thresh 0.55: 0.7842 Thresh 0.60: 0.7785 Thresh 0.70: 0.7629 | \nThresh 0.10: 0.9005 Thresh 0.20: 0.9117 Thresh 0.30: 0.9089 Thresh 0.35: 0.9052 Thresh 0.45: 0.8945 Thresh 0.50: 0.8873 Thresh 0.55: 0.8785 Thresh 0.60: 0.8682 Thresh 0.70: 0.8406 | \n0.838 | \n
Thanks for sharing! in my experiments, k1_dense, k3_dense train together is slightly higher in LB than only use k1 dense.
\nOnly use kidney2 train, LB is pretty low also around 0.72, similar with yours
I tried different seeds and also not found better correlation, have you found?
", "rawMarkdown": "I tried different seeds and also not found better correlation, have you found?", "votes": null }, { "id": "2608640", "postDate": "01/19/2024 01:26:38", "content": "Train a simple NN classifier online, class1: kidney1, public; class2: kidney2; class3: kidney3; and then we can prob out the private? but this is maybe classified by intensity, actually we need the metric domain.
", "rawMarkdown": "Train a simple NN classifier online, class1: kidney1, public; class2: kidney2; class3: kidney3; and then we can prob out the private? but this is maybe classified by intensity, actually we need the metric domain.", "votes": null }, { "id": "2608646", "postDate": "01/19/2024 01:37:34", "content": "Useful links:
\nMay I inquire whether the metrics 'hit' and 'fp' are calculated based on a single axis or three axes?
", "rawMarkdown": "May I inquire whether the metrics 'hit' and 'fp' are calculated based on a single axis or three axes?", "votes": null }, { "id": "2609230", "postDate": "01/19/2024 10:25:23", "content": "Not really. I need to do further experiments but I am lacking time. 😬
", "rawMarkdown": "Not really. I need to do further experiments but I am lacking time. 😬", "votes": null }, { "id": "2609359", "postDate": "01/19/2024 12:19:37", "content": "my scores on kidney 2 seem much lower than yours… Do you report scores for the lower part of kindey 2 or the entire kidney 2? (on lower part of kidney 2 I have scores around 0.84-0.85)
", "rawMarkdown": "my scores on kidney 2 seem much lower than yours... Do you report scores for the lower part of kindey 2 or the entire kidney 2? (on lower part of kidney 2 I have scores around 0.84-0.85)", "votes": null }, { "id": "2610212", "postDate": "01/20/2024 02:02:24", "content": "three axes
", "rawMarkdown": "three axes", "votes": null }, { "id": "2610213", "postDate": "01/20/2024 02:04:15", "content": "the result is on lower part of kidney 2, after 900 slice. my score is not stable range from 0.82--0.89
", "rawMarkdown": "the result is on lower part of kidney 2, after 900 slice. my score is not stable range from 0.82--0.89", "votes": null }, { "id": "2610228", "postDate": "01/20/2024 02:53:45", "content": "if you perform TTA, you can easily go beyond 0.9+ for validation on kidney1,2(lower) or 3 !!
", "rawMarkdown": "if you perform TTA, you can easily go beyond 0.9+ for validation on kidney1,2(lower) or 3 !!", "votes": null }, { "id": "2612187", "postDate": "01/21/2024 08:49:51", "content": "@optimo - have you considered to use kidney 2 like for fine tuning on one of your kidney 1 or 1 and 3 models?
\nwas considering doing that on the lower part of kidney 2 but not sure what to use for validation unless split part of it off, or use the rest but not sure how good it will be, if it is data leakage. etc. may have a go, see what happens…
is your score for the entire k2 or k2 on slices >= 900 ?
", "rawMarkdown": "is your score for the entire k2 or k2 on slices >= 900 ?", "votes": null }, { "id": "2616820", "postDate": "01/23/2024 21:14:21", "content": "i think we may made a fundamental mistake.
\nWhen the validation data is scare, you should use validation data + their augmentation for CV.
\nyou cannot validate with only \"one dataset\"
\nThis will have a more robust CV.
Note: this require you to understand what is the different image characteristics of your test set (e.g. size, resolution, intensity ….)
", "rawMarkdown": "i think we may made a fundamental mistake.\n\nWhen the validation data is scare, you should use validation data + their augmentation for CV.\nyou cannot validate with only \"one dataset\"\nThis will have a more robust CV.\n\nNote: this require you to understand what is the different image characteristics of your test set (e.g. size, resolution, intensity ....)", "votes": null }, { "id": "2629724", "postDate": "02/01/2024 01:47:15", "content": "Here's what I've seen, in case you find it interesting.
\nMetrics are shown as dice (one dimension, k2), surface dice (one dimension, k2), surface dice lb (3 dimensional TTA with 3,2,1 votes respectively, threshold for a vote is 0.5).
\nk1 train only: 0.82, 0.68, (0.77, 0.77, 0.65) 3x TTA w/threshold of 3,2,1 votes.
\nk1 & k3 train together: 0.91, 0.86, (0.64, 0.75, 0.79) 3x TTA w/threshold of 3,2,1 votes.
\nI feel like I'm missing something in this competition…trying different ways to preprocess, more augmentations, and a little bit with loss functions, but I really struggle to get lb scores better than these.
", "rawMarkdown": "Here's what I've seen, in case you find it interesting.\n\nMetrics are shown as dice (one dimension, k2), surface dice (one dimension, k2), surface dice lb (3 dimensional TTA with 3,2,1 votes respectively, threshold for a vote is 0.5). \n\nk1 train only: 0.82, 0.68, (0.77, 0.77, 0.65) 3x TTA w/threshold of 3,2,1 votes.\n\nk1 & k3 train together: 0.91, 0.86, (0.64, 0.75, 0.79) 3x TTA w/threshold of 3,2,1 votes.\n\nI feel like I'm missing something in this competition...trying different ways to preprocess, more augmentations, and a little bit with loss functions, but I really struggle to get lb scores better than these.", "votes": null } ], "comments": [ { "id": 2605446, "author_name": "dragonzhang", "author_url": "", "post_date": "01/17/2024 04:38:21", "content": "the possible problem is the public/private dataset distribution? if they are same type of data.
\neither models trained on all data would be little bit lower on Public while stable on Private,
\nor models with 2 stages(classification and prediction) would be sustainable to shake up.
However, without local computation power, I am lazy to try out my simple ideas.
", "votes": null, "replies": [] }, { "id": 2606272, "author_name": "optimo", "author_url": "", "post_date": "01/17/2024 14:29:25", "content": "Can you please clarify what you mean by k2 and k2 sparse? So that I can share comparable results, thanks!
", "votes": null, "replies": [ { "id": 2606278, "author_name": "sacuscreed", "author_url": "", "post_date": "01/17/2024 14:36:11", "content": "May be k2 is semipseudolabeled
", "votes": null, "replies": [] }, { "id": 2607125, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/18/2024 04:07:37", "content": "I mean use kidney2 to train or kidney3 sparse to train, what’s the difference between using kidney1 dense in LB
", "votes": null, "replies": [ { "id": 2607654, "author_name": "optimo", "author_url": "", "post_date": "01/18/2024 11:00:59", "content": "is your score for the entire k2 or k2 on slices > 900 as your teammate reports here ?
", "votes": null, "replies": [] }, { "id": 2616346, "author_name": "zz1zzi", "author_url": "", "post_date": "01/23/2024 15:32:30", "content": "is your score for the entire k2 or k2 on slices >= 900 ?
", "votes": null, "replies": [] } ] } ] }, { "id": 2607884, "author_name": "yassinealouini", "author_url": "", "post_date": "01/18/2024 13:49:22", "content": "Out of curiosity, have you tried training using different seeds and checking the LB vs CV correlation (for base models and averaged over seeds)?
", "votes": null, "replies": [ { "id": 2608618, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/19/2024 00:40:41", "content": "I tried different seeds and also not found better correlation, have you found?
", "votes": null, "replies": [ { "id": 2609230, "author_name": "yassinealouini", "author_url": "", "post_date": "01/19/2024 10:25:23", "content": "Not really. I need to do further experiments but I am lacking time. 😬
", "votes": null, "replies": [] } ] } ] }, { "id": 2607904, "author_name": "optimo", "author_url": "", "post_date": "01/18/2024 13:58:17", "content": "Here are the results of one experiment I just did (k2 here is the original kidney with all slices), LB threshold is 0.5:
\n| Train | \nValid | \nk1_dense | \nk2 | \nk3_dense | \nLB (Th 0.5) | \n
|---|---|---|---|---|---|
| k1_dense, k3_dense | \nk2 | \n\n | Thresh 0.10: 0.7446 Thresh 0.20: 0.7780 Thresh 0.30: 0.7909 Thresh 0.35: 0.7941 Thresh 0.45: 0.7968 Thresh 0.50: 0.7970 Thresh 0.55: 0.7963 Thresh 0.60: 0.7940 Thresh 0.70: 0.7839 | \n\n | 0.799 | \n
| k2 | \nk1_dense, k3_dense | \nThresh 0.10: 0.6312 Thresh 0.20: 0.7462 Thresh 0.30: 0.8034 Thresh 0.35: 0.8215 Thresh 0.45: 0.8428 Thresh 0.50: 0.8477 Thresh 0.55: 0.8487 Thresh 0.60: 0.8459 Thresh 0.70: 0.8264 | \n\n | Thresh 0.10: 0.8047 Thresh 0.20: 0.8712 Thresh 0.30: 0.8991 Thresh 0.35: 0.9064 Thresh 0.45: 0.9122 Thresh 0.50: 0.9114 Thresh 0.55: 0.9084 Thresh 0.60: 0.9030 Thresh 0.70: 0.8835 | \n0.724 | \n
Still same experiment, threshold adapted to CV (0.2)
\n| Train | \nValid | \nk1_dense | \nk2 | \nk3_dense | \nLB (Th 0.2) | \n
|---|---|---|---|---|---|
| k1_dense | \nk2, k3_dense | \n\n | Thresh 0.10: 0.7884 Thresh 0.20: 0.8002 Thresh 0.30: 0.8008 Thresh 0.35: 0.7991 Thresh 0.45: 0.7930 Thresh 0.50: 0.7890 Thresh 0.55: 0.7842 Thresh 0.60: 0.7785 Thresh 0.70: 0.7629 | \nThresh 0.10: 0.9005 Thresh 0.20: 0.9117 Thresh 0.30: 0.9089 Thresh 0.35: 0.9052 Thresh 0.45: 0.8945 Thresh 0.50: 0.8873 Thresh 0.55: 0.8785 Thresh 0.60: 0.8682 Thresh 0.70: 0.8406 | \n0.838 | \n
Thanks for sharing! in my experiments, k1_dense, k3_dense train together is slightly higher in LB than only use k1 dense.
\nOnly use kidney2 train, LB is pretty low also around 0.72, similar with yours
my scores on kidney 2 seem much lower than yours… Do you report scores for the lower part of kindey 2 or the entire kidney 2? (on lower part of kidney 2 I have scores around 0.84-0.85)
", "votes": null, "replies": [ { "id": 2610213, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/20/2024 02:04:15", "content": "the result is on lower part of kidney 2, after 900 slice. my score is not stable range from 0.82--0.89
", "votes": null, "replies": [ { "id": 2610228, "author_name": "hengck23", "author_url": "", "post_date": "01/20/2024 02:53:45", "content": "if you perform TTA, you can easily go beyond 0.9+ for validation on kidney1,2(lower) or 3 !!
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2612187, "author_name": "something4kag", "author_url": "", "post_date": "01/21/2024 08:49:51", "content": "@optimo - have you considered to use kidney 2 like for fine tuning on one of your kidney 1 or 1 and 3 models?
\nwas considering doing that on the lower part of kidney 2 but not sure what to use for validation unless split part of it off, or use the rest but not sure how good it will be, if it is data leakage. etc. may have a go, see what happens…
Train a simple NN classifier online, class1: kidney1, public; class2: kidney2; class3: kidney3; and then we can prob out the private? but this is maybe classified by intensity, actually we need the metric domain.
", "votes": null, "replies": [] }, { "id": 2608646, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/19/2024 01:37:34", "content": "Useful links:
\nMay I inquire whether the metrics 'hit' and 'fp' are calculated based on a single axis or three axes?
", "votes": null, "replies": [ { "id": 2610212, "author_name": "lihaoweicvch", "author_url": "", "post_date": "01/20/2024 02:02:24", "content": "three axes
", "votes": null, "replies": [] } ] }, { "id": 2616820, "author_name": "hengck23", "author_url": "", "post_date": "01/23/2024 21:14:21", "content": "i think we may made a fundamental mistake.
\nWhen the validation data is scare, you should use validation data + their augmentation for CV.
\nyou cannot validate with only \"one dataset\"
\nThis will have a more robust CV.
Note: this require you to understand what is the different image characteristics of your test set (e.g. size, resolution, intensity ….)
", "votes": null, "replies": [] }, { "id": 2629724, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/01/2024 01:47:15", "content": "Here's what I've seen, in case you find it interesting.
\nMetrics are shown as dice (one dimension, k2), surface dice (one dimension, k2), surface dice lb (3 dimensional TTA with 3,2,1 votes respectively, threshold for a vote is 0.5).
\nk1 train only: 0.82, 0.68, (0.77, 0.77, 0.65) 3x TTA w/threshold of 3,2,1 votes.
\nk1 & k3 train together: 0.91, 0.86, (0.64, 0.75, 0.79) 3x TTA w/threshold of 3,2,1 votes.
\nI feel like I'm missing something in this competition…trying different ways to preprocess, more augmentations, and a little bit with loss functions, but I really struggle to get lb scores better than these.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2605231": "I trained on K1 and val on K3, or train K1,K3 val on K2, both not found any correlation between CV and LB.\nHas anyone use K2 sparse, K3 sparse or other mixture of them for train, what's the result?\n\n\n", "2605446": "the possible problem is the public/private dataset distribution? if they are same type of data. \neither models trained on all data would be little bit lower on Public while stable on Private,\nor models with 2 stages(classification and prediction) would be sustainable to shake up.\n\nHowever, without local computation power, I am lazy to try out my simple ideas.", "2606272": "Can you please clarify what you mean by k2 and k2 sparse? So that I can share comparable results, thanks!", "2606278": "May be k2 is semipseudolabeled", "2607125": "I mean use kidney2 to train or kidney3 sparse to train, what’s the difference between using kidney1 dense in LB", "2607654": "is your score for the entire k2 or k2 on slices > 900 as your teammate reports [here](https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/468826) ?", "2607884": "Out of curiosity, have you tried training using different seeds and checking the LB vs CV correlation (for base models and averaged over seeds)?", "2607904": "Here are the results of one experiment I just did (k2 here is the original kidney with all slices), LB threshold is 0.5:\n\n| Train | Valid | k1_dense | k2 | k3_dense | LB (Th 0.5) |\n|-----------|----------|--------------|----------|-------------|------|\n| k1_dense, k3_dense | k2 | | Thresh 0.10: 0.7446