{ "id": 236752, "title": "Tackle d488c759a", "url": "/competitions/hubmap-kidney-segmentation/discussion/236752", "author_name": "Bessenyei Szilárd", "post_date": "2021-05-05T16:57:55.892000", "votes": 5, "comment_count": 18, "views": 0, "content": "

Hi everyone!

\n

I measured the dice for each image, and I determined, that my algorithm is the weakest on d488c759a.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ImageDice
aa05346ff0.910
2ec3f1bb90.945
57512b7f10.930
d488c759a0.820
3589adb900.950
\n

\"d488c759a\"

\n

I noticed, that higher CV doesn't mean that my network performs better on the test set, because the glomeruli cells look different on d488c759a than on the others. My network has CV dice 0.93+ and I got 0.915 LB. I see that this is
\nI tried HSV, contrast, gain augmentations and reduce the threshold, but none of them helped. Does any has any suggestion on what type of augmentation should I use to tackle this?

\n

Thanks for the help!

", "messages": [ { "id": 1294448, "postDate": "2021-05-05T16:57:55.893Z", "content": "

Hi everyone!

\n

I measured the dice for each image, and I determined, that my algorithm is the weakest on d488c759a.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ImageDice
aa05346ff0.910
2ec3f1bb90.945
57512b7f10.930
d488c759a0.820
3589adb900.950
\n

\"d488c759a\"

\n

I noticed, that higher CV doesn't mean that my network performs better on the test set, because the glomeruli cells look different on d488c759a than on the others. My network has CV dice 0.93+ and I got 0.915 LB. I see that this is
\nI tried HSV, contrast, gain augmentations and reduce the threshold, but none of them helped. Does any has any suggestion on what type of augmentation should I use to tackle this?

\n

Thanks for the help!

", "rawMarkdown": "Hi everyone!\n\nI measured the dice for each image, and I determined, that my algorithm is the weakest on d488c759a. \n\n| Image | Dice |\n| ------------- | --------- |\n| aa05346ff | 0.910 |\n| 2ec3f1bb9 | 0.945 |\n| 57512b7f1 | 0.930 |\n| **d488c759a** | **0.820** |\n| 3589adb90 | 0.950 |\n\n![d488c759a](https://i.imgur.com/VoiYGIS.jpg)\n\nI noticed, that higher CV doesn't mean that my network performs better on the test set, because the glomeruli cells look different on `d488c759a` than on the others. My network has CV dice 0.93+ and I got 0.915 LB. I see that this is \nI tried HSV, contrast, gain augmentations and reduce the threshold, but none of them helped. Does any has any suggestion on what type of augmentation should I use to tackle this?\n\nThanks for the help!", "votes": 5 }, { "id": 1298025, "postDate": "2021-05-08T14:00:37.607Z", "content": "

I just retrained some models yesterday that penalize FPs more (haven't done this in a month) and while my CV-dice score goes up slightly and my LB dice stays pretty much constant around 0.95+ for the 4 test files excluding d488, the LB dice goes down a lot for d488.

\n

This leads me to think that d488 contains what would otherwise would be FP glomes.

\n

The latest comment from the organizers is that the healthy/sclerosed classification is not as clear cut as we might want but there is a continuum, i.e. a glome could be something like 65% sclerosed, and it's up to the pathologists to decide in which bucket they place it.

\n

I`m not a pathologist and while I think that's completely plausible I don't think that's what's wrong with d488.

\n

imo the problem with d488 is that:

\n
    \n
  1. the organizers messed up the annotations again and decided not to fix them because it was already too late or wtv
    2. \n
  2. there`s some tricky property of glomes that most if not all people missed
    3. \n
\n

I lean more towards (1) tho bc d488 is the only tiff out of 20 that has this problem and people have found lots of glomes similar to those in d488 in other files but which have't been annotated as such.

", "rawMarkdown": "I just retrained some models yesterday that penalize FPs more (haven't done this in a month) and while my CV-dice score goes up slightly and my LB dice stays pretty much constant around 0.95+ for the 4 test files excluding d488, the LB dice goes down a lot for d488.\n\nThis leads me to think that d488 contains what would otherwise would be FP glomes.\n\nThe latest comment from the organizers is that the healthy/sclerosed classification is not as clear cut as we might want but there is a continuum, i.e. a glome could be something like 65% sclerosed, and it's up to the pathologists to decide in which bucket they place it.\n\nI`m not a pathologist and while I think that's completely plausible I don't think that's what's wrong with d488.\n\nimo the problem with d488 is that:\n1. the organizers messed up the annotations again and decided not to fix them because it was already too late or wtv\n2. there`s some tricky property of glomes that most if not all people missed\n\nI lean more towards (1) tho bc d488 is the only tiff out of 20 that has this problem and people have found lots of glomes similar to those in d488 in other files but which have't been annotated as such. ", "votes": 6, "replies": [ { "id": 1298521, "postDate": "2021-05-09T00:30:11.650Z", "content": "

I'm with you. Since they did not want to correct the mistake yet again, the organizers authorized hand labeling

", "rawMarkdown": "I'm with you. Since they did not want to correct the mistake yet again, the organizers authorized hand labeling " }, { "id": 1308600, "postDate": "2021-05-15T10:28:18.357Z", "content": "

I vote for #1 and allowing competition was not a good idea then.

", "rawMarkdown": "I vote for #1 and allowing competition was not a good idea then.", "votes": 1 } ] }, { "id": 1294462, "postDate": "2021-05-05T17:08:01.053Z", "content": "

D48 is infamous. A lot of us train using Zhao's D48 hand labels, see this topic:

\n

https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616

\n

But we don't know if this is a good idea. Many people think the glomeruli in d48 should not be labelled (i.e. it's appropriate for your model not to flag them). If you read the thread above, you'll see multiple takes on this.

\n

Furthermore, it makes public leader board performance difficult to judge.

", "rawMarkdown": "D48 is infamous. A lot of us train using Zhao's D48 hand labels, see this topic:\n\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616\n\nBut we don't know if this is a good idea. Many people think the glomeruli in d48 should *not* be labelled (i.e. it's appropriate for your model not to flag them). If you read the thread above, you'll see multiple takes on this.\n\nFurthermore, it makes public leader board performance difficult to judge.", "votes": 3, "replies": [ { "id": 1294577, "postDate": "2021-05-05T18:43:06.793Z", "content": "

Great, It is good to know, that others struggled with this as well, but they have a solution. This is very useful!

", "rawMarkdown": "Great, It is good to know, that others struggled with this as well, but they have a solution. This is very useful! " } ] }, { "id": 1294840, "postDate": "2021-05-06T01:37:42.103Z", "content": "

Will some data like d48 in private set?

", "rawMarkdown": "Will some data like d48 in private set?", "votes": 1, "replies": [ { "id": 1294886, "postDate": "2021-05-06T03:05:09.507Z", "content": "

Good question, as I know we couldn't predict that

", "rawMarkdown": "Good question, as I know we couldn't predict that" } ] }, { "id": 1295889, "postDate": "2021-05-06T19:24:02.490Z", "content": "

I tried multiple training/validation runs with the D48 hand labels and constantly a slight decrease in validation score compared to using my best ensemble and pseudo label the D48 myself and then perform the same training and validation.

\n

I also tried a few submissions with it but had a rather consistent drop in performance.

\n

So it likely depends on your current setup what the effect would be of using the D48 labels.
\nWith a performance drop in both local CV and Public LB I'am taking my chances on not using it.

\n

But I'am very curious to see the final and real outcome on Private LB.

", "rawMarkdown": "I tried multiple training/validation runs with the D48 hand labels and constantly a slight decrease in validation score compared to using my best ensemble and pseudo label the D48 myself and then perform the same training and validation.\n\nI also tried a few submissions with it but had a rather consistent drop in performance.\n\nSo it likely depends on your current setup what the effect would be of using the D48 labels.\nWith a performance drop in both local CV and Public LB I'am taking my chances on not using it.\n\nBut I'am very curious to see the final and real outcome on Private LB.", "votes": 2, "replies": [ { "id": 1296262, "postDate": "2021-05-07T05:51:38.883Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1295002, "postDate": "2021-05-06T06:08:51.903Z", "content": "

I wonder how do you calculate the dice since we don't have the public test label

", "rawMarkdown": "I wonder how do you calculate the dice since we don't have the public test label", "replies": [ { "id": 1295021, "postDate": "2021-05-06T06:30:46.700Z", "content": "

If you are training using pseudo labels on public test data, you can use these labels to calculate dice_loss with your predicted results. But is better calculate val dice_loss on only training data.

", "rawMarkdown": "If you are training using pseudo labels on public test data, you can use these labels to calculate dice_loss with your predicted results. But is better calculate val dice_loss on only training data." }, { "id": 1295037, "postDate": "2021-05-06T06:45:14.677Z", "content": "

He may have done 5 separate submissions, predicting only 1 of the 5 images at the time (and predicting all zeros for the other files), and then multiplied the resultant scores by 5.

", "rawMarkdown": "He may have done 5 separate submissions, predicting only 1 of the 5 images at the time (and predicting all zeros for the other files), and then multiplied the resultant scores by 5.", "votes": 2 }, { "id": 1295125, "postDate": "2021-05-06T08:12:10.360Z", "content": "

yes, @jamesphoward is right.

", "rawMarkdown": "yes, @jamesphoward is right." } ] }, { "id": 1294946, "postDate": "2021-05-06T04:38:24.620Z", "content": "

I don't think hand labeling d48 is a good idea because in private test data there may be or may not be such type of images. So it is better to train model with noise. In general private data is similar to training data.

", "rawMarkdown": "I don't think hand labeling d48 is a good idea because in private test data there may be or may not be such type of images. So it is better to train model with noise. In general private data is similar to training data.", "replies": [ { "id": 1295221, "postDate": "2021-05-06T09:41:59.843Z", "content": "
\n

In general private data is similar to training data

\n
\n

How can you infer about the same ?

", "rawMarkdown": "> In general private data is similar to training data\n\nHow can you infer about the same ?" }, { "id": 1296166, "postDate": "2021-05-07T03:36:39.417Z", "content": "

Based on past experience of competitions. If you go through the discussions a lot of people are saying that you should make your final submission based on your CV not on public LB.
\nAnother point is that whole data is prepared from the same source some part of which is given to us as training data and rest is kept for testing. Public lb is calculated only a small portion of test data because of which public lb may be biased towards some images.

", "rawMarkdown": "Based on past experience of competitions. If you go through the discussions a lot of people are saying that you should make your final submission based on your CV not on public LB. \nAnother point is that whole data is prepared from the same source some part of which is given to us as training data and rest is kept for testing. Public lb is calculated only a small portion of test data because of which public lb may be biased towards some images.", "votes": 1 }, { "id": 1296265, "postDate": "2021-05-07T05:58:44.463Z", "content": "

Using pseudo label with D48 HAND label also drop my LB,

", "rawMarkdown": "Using pseudo label with D48 HAND label also drop my LB," }, { "id": 1296269, "postDate": "2021-05-07T06:05:54.487Z", "content": "

@zxyu1995 Similar is the case with me, I added Zhao's hand labelled dataset in my training pipeline, the training was fine but the values for cv and lb were significantly lower/worse.

", "rawMarkdown": "@zxyu1995 Similar is the case with me, I added Zhao's hand labelled dataset in my training pipeline, the training was fine but the values for cv and lb were significantly lower/worse." } ] } ], "comments": [ { "id": 1298025, "author_name": "rosuluc", "author_url": "", "post_date": "2021-05-08T14:00:37.607000", "content": "

I just retrained some models yesterday that penalize FPs more (haven't done this in a month) and while my CV-dice score goes up slightly and my LB dice stays pretty much constant around 0.95+ for the 4 test files excluding d488, the LB dice goes down a lot for d488.

\n

This leads me to think that d488 contains what would otherwise would be FP glomes.

\n

The latest comment from the organizers is that the healthy/sclerosed classification is not as clear cut as we might want but there is a continuum, i.e. a glome could be something like 65% sclerosed, and it's up to the pathologists to decide in which bucket they place it.

\n

I`m not a pathologist and while I think that's completely plausible I don't think that's what's wrong with d488.

\n

imo the problem with d488 is that:

\n
    \n
  1. the organizers messed up the annotations again and decided not to fix them because it was already too late or wtv
    2. \n
  2. there`s some tricky property of glomes that most if not all people missed
    3. \n
\n

I lean more towards (1) tho bc d488 is the only tiff out of 20 that has this problem and people have found lots of glomes similar to those in d488 in other files but which have't been annotated as such.

", "votes": 6, "replies": [ { "id": 1298521, "author_name": "andras", "author_url": "", "post_date": "2021-05-09T00:30:11.650000", "content": "

I'm with you. Since they did not want to correct the mistake yet again, the organizers authorized hand labeling

", "votes": 0, "replies": [] }, { "id": 1308600, "author_name": "MPWARE", "author_url": "", "post_date": "2021-05-15T10:28:18.357000", "content": "

I vote for #1 and allowing competition was not a good idea then.

", "votes": 1, "replies": [] } ] }, { "id": 1294462, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-05T17:08:01.053000", "content": "

D48 is infamous. A lot of us train using Zhao's D48 hand labels, see this topic:

\n

https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616

\n

But we don't know if this is a good idea. Many people think the glomeruli in d48 should not be labelled (i.e. it's appropriate for your model not to flag them). If you read the thread above, you'll see multiple takes on this.

\n

Furthermore, it makes public leader board performance difficult to judge.

", "votes": 3, "replies": [ { "id": 1294577, "author_name": "Bessenyei Szilárd", "author_url": "", "post_date": "2021-05-05T18:43:06.793000", "content": "

Great, It is good to know, that others struggled with this as well, but they have a solution. This is very useful!

", "votes": 0, "replies": [] } ] }, { "id": 1294840, "author_name": "Zekun", "author_url": "", "post_date": "2021-05-06T01:37:42.103000", "content": "

Will some data like d48 in private set?

", "votes": 1, "replies": [ { "id": 1294886, "author_name": "Bessenyei Szilárd", "author_url": "", "post_date": "2021-05-06T03:05:09.507000", "content": "

Good question, as I know we couldn't predict that

", "votes": 0, "replies": [] } ] }, { "id": 1295889, "author_name": "Robin Smits", "author_url": "", "post_date": "2021-05-06T19:24:02.490000", "content": "

I tried multiple training/validation runs with the D48 hand labels and constantly a slight decrease in validation score compared to using my best ensemble and pseudo label the D48 myself and then perform the same training and validation.

\n

I also tried a few submissions with it but had a rather consistent drop in performance.

\n

So it likely depends on your current setup what the effect would be of using the D48 labels.
\nWith a performance drop in both local CV and Public LB I'am taking my chances on not using it.

\n

But I'am very curious to see the final and real outcome on Private LB.

", "votes": 2, "replies": [ { "id": 1296262, "author_name": "", "author_url": "", "post_date": "2021-05-07T05:51:38.883000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1295002, "author_name": "zxyu", "author_url": "", "post_date": "2021-05-06T06:08:51.903000", "content": "

I wonder how do you calculate the dice since we don't have the public test label

", "votes": 0, "replies": [ { "id": 1295021, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-05-06T06:30:46.700000", "content": "

If you are training using pseudo labels on public test data, you can use these labels to calculate dice_loss with your predicted results. But is better calculate val dice_loss on only training data.

", "votes": 0, "replies": [] }, { "id": 1295037, "author_name": "James Howard", "author_url": "", "post_date": "2021-05-06T06:45:14.677000", "content": "

He may have done 5 separate submissions, predicting only 1 of the 5 images at the time (and predicting all zeros for the other files), and then multiplied the resultant scores by 5.

", "votes": 2, "replies": [] }, { "id": 1295125, "author_name": "Bessenyei Szilárd", "author_url": "", "post_date": "2021-05-06T08:12:10.360000", "content": "

yes, @jamesphoward is right.

", "votes": 0, "replies": [] } ] }, { "id": 1294946, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-05-06T04:38:24.620000", "content": "

I don't think hand labeling d48 is a good idea because in private test data there may be or may not be such type of images. So it is better to train model with noise. In general private data is similar to training data.

", "votes": 0, "replies": [ { "id": 1295221, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-05-06T09:41:59.843000", "content": "
\n

In general private data is similar to training data

\n
\n

How can you infer about the same ?

", "votes": 0, "replies": [] }, { "id": 1296166, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-05-07T03:36:39.417000", "content": "

Based on past experience of competitions. If you go through the discussions a lot of people are saying that you should make your final submission based on your CV not on public LB.
\nAnother point is that whole data is prepared from the same source some part of which is given to us as training data and rest is kept for testing. Public lb is calculated only a small portion of test data because of which public lb may be biased towards some images.

", "votes": 1, "replies": [] }, { "id": 1296265, "author_name": "zxyu", "author_url": "", "post_date": "2021-05-07T05:58:44.463000", "content": "

Using pseudo label with D48 HAND label also drop my LB,

", "votes": 0, "replies": [] }, { "id": 1296269, "author_name": "ilovepotatoes", "author_url": "", "post_date": "2021-05-07T06:05:54.487000", "content": "

@zxyu1995 Similar is the case with me, I added Zhao's hand labelled dataset in my training pipeline, the training was fine but the values for cv and lb were significantly lower/worse.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1294448": "Hi everyone!\n\nI measured the dice for each image, and I determined, that my algorithm is the weakest on d488c759a. \n\n| Image | Dice |\n| ------------- | --------- |\n| aa05346ff | 0.910 |\n| 2ec3f1bb9 | 0.945 |\n| 57512b7f1 | 0.930 |\n| **d488c759a** | **0.820** |\n| 3589adb90 | 0.950 |\n\n![d488c759a](https://i.imgur.com/VoiYGIS.jpg)\n\nI noticed, that higher CV doesn't mean that my network performs better on the test set, because the glomeruli cells look different on `d488c759a` than on the others. My network has CV dice 0.93+ and I got 0.915 LB. I see that this is \nI tried HSV, contrast, gain augmentations and reduce the threshold, but none of them helped. Does any has any suggestion on what type of augmentation should I use to tackle this?\n\nThanks for the help!", "1298025": "I just retrained some models yesterday that penalize FPs more (haven't done this in a month) and while my CV-dice score goes up slightly and my LB dice stays pretty much constant around 0.95+ for the 4 test files excluding d488, the LB dice goes down a lot for d488.\n\nThis leads me to think that d488 contains what would otherwise would be FP glomes.\n\nThe latest comment from the organizers is that the healthy/sclerosed classification is not as clear cut as we might want but there is a continuum, i.e. a glome could be something like 65% sclerosed, and it's up to the pathologists to decide in which bucket they place it.\n\nI`m not a pathologist and while I think that's completely plausible I don't think that's what's wrong with d488.\n\nimo the problem with d488 is that:\n1. the organizers messed up the annotations again and decided not to fix them because it was already too late or wtv\n2. there`s some tricky property of glomes that most if not all people missed\n\nI lean more towards (1) tho bc d488 is the only tiff out of 20 that has this problem and people have found lots of glomes similar to those in d488 in other files but which have't been annotated as such. ", "1294462": "D48 is infamous. A lot of us train using Zhao's D48 hand labels, see this topic:\n\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616\n\nBut we don't know if this is a good idea. Many people think the glomeruli in d48 should *not* be labelled (i.e. it's appropriate for your model not to flag them). If you read the thread above, you'll see multiple takes on this.\n\nFurthermore, it makes public leader board performance difficult to judge.", "1294840": "Will some data like d48 in private set?", "1295889": "I tried multiple training/validation runs with the D48 hand labels and constantly a slight decrease in validation score compared to using my best ensemble and pseudo label the D48 myself and then perform the same training and validation.\n\nI also tried a few submissions with it but had a rather consistent drop in performance.\n\nSo it likely depends on your current setup what the effect would be of using the D48 labels.\nWith a performance drop in both local CV and Public LB I'am taking my chances on not using it.\n\nBut I'am very curious to see the final and real outcome on Private LB.", "1295002": "I wonder how do you calculate the dice since we don't have the public test label", "1294946": "I don't think hand labeling d48 is a good idea because in private test data there may be or may not be such type of images. So it is better to train model with noise. In general private data is similar to training data." } }