{ "id": 235941, "title": "Should there be a shake-up?", "url": "/competitions/hubmap-kidney-segmentation/discussion/235941", "author_name": "", "post_date": "2021-05-02T00:38:28.700142400Z", "votes": 11, "comment_count": 42, "views": 0, "content": "

I think there should be a shake-up in this competition.
\nThe reasons are as follows:

\n", "messages": [ { "id": "1290426", "postDate": "05/02/2021 00:38:28", "content": "

I think there should be a shake-up in this competition.
\nThe reasons are as follows:

\n", "rawMarkdown": "I think there should be a shake-up in this competition.\nThe reasons are as follows:\n\n- Manually pseudo labelled image in public test set (public LB score should be higher)\n- Deepflash 2 notebook can be overfitted on the private leaderboard (the public kernel doesn't use Kfold cross-validation)\n- There are two kernels with high scores on the public leaderboard (public csv file can be submitted)", "votes": null }, { "id": "1290461", "postDate": "05/02/2021 02:35:33", "content": "

Is the Deepflash 2 only gets lb 922?

", "rawMarkdown": "Is the Deepflash 2 only gets lb 922?", "votes": null }, { "id": "1290466", "postDate": "05/02/2021 02:43:42", "content": "

I got 92.8 on public LB and CV is around 94.5 using pseudo labeling. You can also read this discussion and comments https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234575
\nThere is someone got more than 93.5 on public LB.

", "rawMarkdown": "I got 92.8 on public LB and CV is around 94.5 using pseudo labeling. You can also read this discussion and comments https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234575\nThere is someone got more than 93.5 on public LB.", "votes": null }, { "id": "1290528", "postDate": "05/02/2021 05:05:25", "content": "

Would u like to join us ? We use keras models.

", "rawMarkdown": "Would u like to join us ? We use keras models.", "votes": null }, { "id": "1290534", "postDate": "05/02/2021 05:11:44", "content": "

Thank you for asking. I'm sorry I can't join your team.

", "rawMarkdown": "Thank you for asking. I'm sorry I can't join your team.", "votes": null }, { "id": "1290696", "postDate": "05/02/2021 09:41:54", "content": "

There will be a ridiculous shakeup at the top because I suspect most people towards the top have used the D48 labels (which may or may not have been a good idea). Furthermore, anyone who's done extensive test set labelling will inevitably be very high at the moment. Their scores will certainly diverge, either in a good way or a very bad way, from the others. It's going to be very exciting.

", "rawMarkdown": "There will be a ridiculous shakeup at the top because I suspect most people towards the top have used the D48 labels (which may or may not have been a good idea). Furthermore, anyone who's done extensive test set labelling will inevitably be very high at the moment. Their scores will certainly diverge, either in a good way or a very bad way, from the others. It's going to be very exciting.", "votes": null }, { "id": "1290698", "postDate": "05/02/2021 09:50:47", "content": "

I totally agree with you

", "rawMarkdown": "I totally agree with you", "votes": null }, { "id": "1290717", "postDate": "05/02/2021 10:12:02", "content": "

Aha,have you done these?

", "rawMarkdown": "Aha,have you done these?", "votes": null }, { "id": "1290721", "postDate": "05/02/2021 10:22:51", "content": "

I've used Zhao's D48 labels, but no further labelling.

", "rawMarkdown": "I've used Zhao's D48 labels, but no further labelling.", "votes": null }, { "id": "1290785", "postDate": "05/02/2021 12:07:10", "content": "

This is the problem with pseudo labeling. When I inspect my model's output (0.919 LB), I can't find any glomeruli that it hasn't identified! In total I can find 15 cases where it has either missed or hasn't created a perfect mask. And 15 misses does not explain the 0.081 dice score miss from a full score. Given that in cross validation I've achieved dice scores as high as 0.96, it has come to labeling issues. So if the top LB scores are obtained not because they overfit, but because the pseudo/manual labels they used are better, then they could score higher in the private test too. And that just sucks.

", "rawMarkdown": "This is the problem with pseudo labeling. When I inspect my model's output (0.919 LB), I can't find any glomeruli that it hasn't identified! In total I can find 15 cases where it has either missed or hasn't created a perfect mask. And 15 misses does not explain the 0.081 dice score miss from a full score. Given that in cross validation I've achieved dice scores as high as 0.96, it has come to labeling issues. So if the top LB scores are obtained not because they overfit, but because the pseudo/manual labels they used are better, then they could score higher in the private test too. And that just sucks.", "votes": null }, { "id": "1290999", "postDate": "05/02/2021 16:07:20", "content": "

Hey @jamesphoward Question about the D48…How do you use it then?

\n
    \n
  1. Do you use it to retrain a model?
    2. \n
  2. Or only use it to modify your final submission?
    3. \n
\n

I wonder how Kagglers put it to use where it would benefit both Public and Private LB. If it is only used to modify the submission file…then what's the use at all…beside looking good at the Public LB? It won't help for the Private LB in that case.

\n

In that case there could indeed be nice shakeup.

", "rawMarkdown": "Hey @jamesphoward Question about the D48...How do you use it then?\n\n1. Do you use it to retrain a model?\n2. Or only use it to modify your final submission?\n\nI wonder how Kagglers put it to use where it would benefit both Public and Private LB. If it is only used to modify the submission file...then what's the use at all...beside looking good at the Public LB? It won't help for the Private LB in that case.\n\nIn that case there could indeed be nice shakeup.", "votes": null }, { "id": "1291032", "postDate": "05/02/2021 16:35:13", "content": "

I did (1).

\n

I think (2) is incredibly dangerous (although (1) might be, too) if you do it in a simple way, at least, because unless you've trained on D48, your model will make very unconfident predictions on those types of glomeruli. If you then use the D48 labels to calibrate your predictions, then you'll end up with an incredibly low prediction threshold for all other slides.

", "rawMarkdown": "I did (1).\n\nI think (2) is incredibly dangerous (although (1) might be, too) if you do it in a simple way, at least, because unless you've trained on D48, your model will make very unconfident predictions on those types of glomeruli. If you then use the D48 labels to calibrate your predictions, then you'll end up with an incredibly low prediction threshold for all other slides.", "votes": null }, { "id": "1291062", "postDate": "05/02/2021 17:08:55", "content": "

Hi @jamesphoward Thanks for your answer.
\nInterresting choice to make there …. i you use it to train a model then there is not that much difference I suppose between automatically pseudo labelled data or manually labelled data.

\n

Given a proper validation strategy you should be able to get a fair indication I guess of whether it is 'safe' or not.

\n

That said….the final week and finish will be interresting for sure :-)

", "rawMarkdown": "Hi @jamesphoward Thanks for your answer. \nInterresting choice to make there .... i you use it to train a model then there is not that much difference I suppose between automatically pseudo labelled data or manually labelled data.\n\nGiven a proper validation strategy you should be able to get a fair indication I guess of whether it is 'safe' or not.\n\nThat said....the final week and finish will be interresting for sure :-)", "votes": null }, { "id": "1292140", "postDate": "05/03/2021 17:40:15", "content": "

@lhagiimn Deepflash use randomized tiles for training and another set of tiles for validation not mixing with train as such. SO just trying to understand reason for overfit.
\nkfold also does similar the scheme that is based not on Tiff ids

", "rawMarkdown": "lhagiimn Deepflash use randomized tiles for training and another set of tiles for validation not mixing with train as such. SO just trying to understand reason for overfit.\nkfold also does similar the scheme that is based not on Tiff ids", "votes": null }, { "id": "1292148", "postDate": "05/03/2021 17:43:28", "content": "

@jamesphoward u have not used PL of other test tiff ids ?

", "rawMarkdown": "jamesphoward u have not used PL of other test tiff ids ?", "votes": null }, { "id": "1292182", "postDate": "05/03/2021 18:08:04", "content": "

Nope I haven't

", "rawMarkdown": "Nope I haven't", "votes": null }, { "id": "1292204", "postDate": "05/03/2021 18:32:26", "content": "

Did u find a big diff in LB score between with and without D4 label training( meaning setting just low th during inference no training on using labels)

", "rawMarkdown": "Did u find a big diff in LB score between with and without D4 label training( meaning setting just low th during inference no training on using labels)", "votes": null }, { "id": "1292205", "postDate": "05/03/2021 18:34:54", "content": "

I think my best model I moved from 0.926 to 0.934 by adding in D4 labels.

", "rawMarkdown": "I think my best model I moved from 0.926 to 0.934 by adding in D4 labels.", "votes": null }, { "id": "1292264", "postDate": "05/03/2021 19:40:37", "content": "

Hi @jamesphoward that's an interresting improvement. So the remainder of your current score is based on TTA and ensemble likely.

\n

With 2 submissions that can be selected I think I should at least try it out…given that my current single fold best is over 0.931..so would be interresting to see what the D4 labels would do in that case. Combining in an ensemble some models trained with D4 and without D4 should keep it on the safe side. Interresting…

\n

Good luck with the remainder of the competition!

", "rawMarkdown": "Hi @jamesphoward that's an interresting improvement. So the remainder of your current score is based on TTA and ensemble likely.\n\nWith 2 submissions that can be selected I think I should at least try it out...given that my current single fold best is over 0.931..so would be interresting to see what the D4 labels would do in that case. Combining in an ensemble some models trained with D4 and without D4 should keep it on the safe side. Interresting...\n\nGood luck with the remainder of the competition!", "votes": null }, { "id": "1292309", "postDate": "05/03/2021 20:18:32", "content": "

I certainly think it's wise to have a 2nd submission pipeline without D4, given the controversy around the glomeruli it contains.

", "rawMarkdown": "I certainly think it's wise to have a 2nd submission pipeline _without_ D4, given the controversy around the glomeruli it contains.", "votes": null }, { "id": "1292583", "postDate": "05/04/2021 05:53:00", "content": "

There is no separation of which images are used for train or validation, both use all the files in deepflash public notebooks. Train does random tiles considering pdfs > random.random [0.0-1.0] to choose centers. Validation randomly samples the designated number of tiles. There is no guarantee of no overlap of tiles or sections of tiles. The use of random in various places tend to make results hard to reproduce even with seeds.

\n

Using stratified kfold e.g. based on mask coverage for tiles like from the iafoss datasets, or hold out of different ids for validation sampling, would ensure folds have different tiles (or images) for train or validation.

", "rawMarkdown": "There is no separation of which images are used for train or validation, both use all the files in deepflash public notebooks. Train does random tiles considering pdfs > random.random [0.0-1.0] to choose centers. Validation randomly samples the designated number of tiles. There is no guarantee of no overlap of tiles or sections of tiles. The use of random in various places tend to make results hard to reproduce even with seeds. \n\nUsing stratified kfold e.g. based on mask coverage for tiles like from the iafoss datasets, or hold out of different ids for validation sampling, would ensure folds have different tiles (or images) for train or validation.", "votes": null }, { "id": "1292602", "postDate": "05/04/2021 06:26:31", "content": "

But when I use kfold,the lb decrease.

", "rawMarkdown": "But when I use kfold,the lb decrease.", "votes": null }, { "id": "1292643", "postDate": "05/04/2021 07:01:02", "content": "

Hello,is the whole test set 5 images?

", "rawMarkdown": "Hello,is the whole test set 5 images?", "votes": null }, { "id": "1292654", "postDate": "05/04/2021 07:17:03", "content": "

No, it should be 15 images.

", "rawMarkdown": "No, it should be 15 images.", "votes": null }, { "id": "1292656", "postDate": "05/04/2021 07:23:40", "content": "

Yes, your right. I just said that it could be overfitted. So it couldn’t be overfitted as well.
\nMy opinion is that training and validation sets can be come from same image for current public notebook because validation set is random sample tiles. So trained model could easily be overfitted. The test set is different from training set, I think GroupKFold could be good evaluation strategy.

", "rawMarkdown": "Yes, your right. I just said that it could be overfitted. So it couldn’t be overfitted as well. \nMy opinion is that training and validation sets can be come from same image for current public notebook because validation set is random sample tiles. So trained model could easily be overfitted. The test set is different from training set, I think GroupKFold could be good evaluation strategy.", "votes": null }, { "id": "1292785", "postDate": "05/04/2021 09:24:33", "content": "

I tried kfold split based on tiff ids , gkf results were uncorrelated completely. Model possibly overfits to position of ftu given all tiles of same tiff in one set

", "rawMarkdown": "I tried kfold split based on tiff ids , gkf results were uncorrelated completely. Model possibly overfits to position of ftu given all tiles of same tiff in one set", "votes": null }, { "id": "1293769", "postDate": "05/05/2021 06:42:13", "content": "

@jamesphoward could u help me interpret 7.26 to 7.34. Assuming your base score .900 .so improvement is 908 ?

", "rawMarkdown": "jamesphoward could u help me interpret 7.26 to 7.34. Assuming your base score .900 .so improvement is 908 ?", "votes": null }, { "id": "1293792", "postDate": "05/05/2021 07:03:49", "content": "

Sorry that was a bizarre typo - I meant to say 0.926 to 0.934.

", "rawMarkdown": "Sorry that was a bizarre typo - I meant to say 0.926 to 0.934.", "votes": null }, { "id": "1293823", "postDate": "05/05/2021 07:28:38", "content": "
\n

Sorry that was a bizarre typo - I meant to say 0.924 to 0.936.

\n
\n

do you mean 0.926 to 0.934 ;)

", "rawMarkdown": "> Sorry that was a bizarre typo - I meant to say 0.924 to 0.936.\n\ndo you mean 0.926 to 0.934 ;)", "votes": null }, { "id": "1293828", "postDate": "05/05/2021 07:33:32", "content": "

😳 yes, sorry (again)

", "rawMarkdown": "😳 yes, sorry (again)", "votes": null }, { "id": "1295240", "postDate": "05/06/2021 10:08:06", "content": "

The final shakeup probably wont be too different from this
\n\"The

", "rawMarkdown": "The final shakeup probably wont be too different from this\n![The final shakeup probably wont be too different from this](https://i.ibb.co/Xz0q1Q7/Screenshot-2021-05-06-at-11-04-31.png)", "votes": null }, { "id": "1295303", "postDate": "05/06/2021 11:06:16", "content": "

I think so. I think gold zone would be safe.

", "rawMarkdown": "I think so. I think gold zone would be safe.", "votes": null }, { "id": "1296082", "postDate": "05/07/2021 00:48:13", "content": "

Nooooo! It's sort ascending 😏

", "rawMarkdown": "Nooooo! It's sort **ascending** 😏", "votes": null }, { "id": "1296628", "postDate": "05/07/2021 11:41:41", "content": "

@jamesphoward I trained my models with hand labelled D4 tfrecord (using Zhao's dataset) but instead of giving me a boost on cv and lb it dropped the score from 0.920 to 0.744 on public lb. Also, another thing I noticed was my models started facing exploding gradients problem on appending the D4 tfrecord to the train set (which was fixed using clipnorm) but still the result was unexpected. Is there something wrong with the generation of D4 tfrecords ? or is that the models fails to train and learn on completely different data (D4) compared to the initial training set ?

", "rawMarkdown": "jamesphoward I trained my models with hand labelled D4 tfrecord (using Zhao's dataset) but instead of giving me a boost on cv and lb it dropped the score from 0.920 to 0.744 on public lb. Also, another thing I noticed was my models started facing exploding gradients problem on appending the D4 tfrecord to the train set (which was fixed using clipnorm) but still the result was unexpected. Is there something wrong with the generation of D4 tfrecords ? or is that the models fails to train and learn on completely different data (D4) compared to the initial training set ?", "votes": null }, { "id": "1296641", "postDate": "05/07/2021 11:51:39", "content": "

Strange. I don't use tfrecords so I can't say, but I found my efficientnet linknet models trained almost identically when the D4 data was added in.

", "rawMarkdown": "Strange. I don't use tfrecords so I can't say, but I found my efficientnet linknet models trained almost identically when the D4 data was added in.", "votes": null }, { "id": "1297173", "postDate": "05/07/2021 19:09:05", "content": "

I would prefer no shake-up 😉
\nBut I must admit that probability for strong shake-up is not null.

", "rawMarkdown": "I would prefer no shake-up 😉\nBut I must admit that probability for strong shake-up is not null.", "votes": null }, { "id": "1297222", "postDate": "05/07/2021 20:02:26", "content": "

@mpware did you guys use the infamous D48 ? 😜

", "rawMarkdown": "mpware did you guys use the infamous **D48** ? 😜", "votes": null }, { "id": "1297699", "postDate": "05/08/2021 08:28:01", "content": "

Zhao's label is not the key. By probing LB i noticed that my CV prediction of d4 is more accurate than Zhao's hand-label

", "rawMarkdown": "Zhao's label is not the key. By probing LB i noticed that my CV prediction of d4 is more accurate than Zhao's hand-label", "votes": null }, { "id": "1297700", "postDate": "05/08/2021 08:29:54", "content": "

I would rather trust CV than LB. believe you me. I lost two golds medals 😨

", "rawMarkdown": "I would rather trust CV than LB. believe you me. I lost two golds medals 😨", "votes": null }, { "id": "1297701", "postDate": "05/08/2021 08:30:24", "content": "

my d4 is more accurate than Zhao's too but I dont think it is useful for my model. I am still afraid to use that.

", "rawMarkdown": "my d4 is more accurate than Zhao's too but I dont think it is useful for my model. I am still afraid to use that.", "votes": null }, { "id": "1297708", "postDate": "05/08/2021 08:40:50", "content": "

Nice to see that you came to the same conclusion

", "rawMarkdown": "Nice to see that you came to the same conclusion", "votes": null }, { "id": "1301244", "postDate": "05/11/2021 01:55:12", "content": "

It happened :(

", "rawMarkdown": "It happened :(", "votes": null }, { "id": "1301580", "postDate": "05/11/2021 06:37:26", "content": "

Well that aged like a fine wine.

", "rawMarkdown": "Well that aged like a fine wine.", "votes": null } ], "comments": [ { "id": 1290461, "author_name": "zekunn", "author_url": "", "post_date": "05/02/2021 02:35:33", "content": "

Is the Deepflash 2 only gets lb 922?

", "votes": null, "replies": [ { "id": 1290466, "author_name": "lhagiimn", "author_url": "", "post_date": "05/02/2021 02:43:42", "content": "

I got 92.8 on public LB and CV is around 94.5 using pseudo labeling. You can also read this discussion and comments https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234575
\nThere is someone got more than 93.5 on public LB.

", "votes": null, "replies": [] }, { "id": 1290528, "author_name": "zekunn", "author_url": "", "post_date": "05/02/2021 05:05:25", "content": "

Would u like to join us ? We use keras models.

", "votes": null, "replies": [] }, { "id": 1290534, "author_name": "lhagiimn", "author_url": "", "post_date": "05/02/2021 05:11:44", "content": "

Thank you for asking. I'm sorry I can't join your team.

", "votes": null, "replies": [] } ] }, { "id": 1290696, "author_name": "jamesphoward", "author_url": "", "post_date": "05/02/2021 09:41:54", "content": "

There will be a ridiculous shakeup at the top because I suspect most people towards the top have used the D48 labels (which may or may not have been a good idea). Furthermore, anyone who's done extensive test set labelling will inevitably be very high at the moment. Their scores will certainly diverge, either in a good way or a very bad way, from the others. It's going to be very exciting.

", "votes": null, "replies": [ { "id": 1290698, "author_name": "lhagiimn", "author_url": "", "post_date": "05/02/2021 09:50:47", "content": "

I totally agree with you

", "votes": null, "replies": [] }, { "id": 1290717, "author_name": "zekunn", "author_url": "", "post_date": "05/02/2021 10:12:02", "content": "

Aha,have you done these?

", "votes": null, "replies": [] }, { "id": 1290721, "author_name": "jamesphoward", "author_url": "", "post_date": "05/02/2021 10:22:51", "content": "

I've used Zhao's D48 labels, but no further labelling.

", "votes": null, "replies": [] }, { "id": 1290785, "author_name": "erikdali", "author_url": "", "post_date": "05/02/2021 12:07:10", "content": "

This is the problem with pseudo labeling. When I inspect my model's output (0.919 LB), I can't find any glomeruli that it hasn't identified! In total I can find 15 cases where it has either missed or hasn't created a perfect mask. And 15 misses does not explain the 0.081 dice score miss from a full score. Given that in cross validation I've achieved dice scores as high as 0.96, it has come to labeling issues. So if the top LB scores are obtained not because they overfit, but because the pseudo/manual labels they used are better, then they could score higher in the private test too. And that just sucks.

", "votes": null, "replies": [] }, { "id": 1290999, "author_name": "rsmits", "author_url": "", "post_date": "05/02/2021 16:07:20", "content": "

Hey @jamesphoward Question about the D48…How do you use it then?

\n
    \n
  1. Do you use it to retrain a model?
    2. \n
  2. Or only use it to modify your final submission?
    3. \n
\n

I wonder how Kagglers put it to use where it would benefit both Public and Private LB. If it is only used to modify the submission file…then what's the use at all…beside looking good at the Public LB? It won't help for the Private LB in that case.

\n

In that case there could indeed be nice shakeup.

", "votes": null, "replies": [] }, { "id": 1291032, "author_name": "jamesphoward", "author_url": "", "post_date": "05/02/2021 16:35:13", "content": "

I did (1).

\n

I think (2) is incredibly dangerous (although (1) might be, too) if you do it in a simple way, at least, because unless you've trained on D48, your model will make very unconfident predictions on those types of glomeruli. If you then use the D48 labels to calibrate your predictions, then you'll end up with an incredibly low prediction threshold for all other slides.

", "votes": null, "replies": [] }, { "id": 1291062, "author_name": "rsmits", "author_url": "", "post_date": "05/02/2021 17:08:55", "content": "

Hi @jamesphoward Thanks for your answer.
\nInterresting choice to make there …. i you use it to train a model then there is not that much difference I suppose between automatically pseudo labelled data or manually labelled data.

\n

Given a proper validation strategy you should be able to get a fair indication I guess of whether it is 'safe' or not.

\n

That said….the final week and finish will be interresting for sure :-)

", "votes": null, "replies": [] }, { "id": 1292148, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/03/2021 17:43:28", "content": "

@jamesphoward u have not used PL of other test tiff ids ?

", "votes": null, "replies": [] }, { "id": 1292182, "author_name": "jamesphoward", "author_url": "", "post_date": "05/03/2021 18:08:04", "content": "

Nope I haven't

", "votes": null, "replies": [] }, { "id": 1292204, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/03/2021 18:32:26", "content": "

Did u find a big diff in LB score between with and without D4 label training( meaning setting just low th during inference no training on using labels)

", "votes": null, "replies": [] }, { "id": 1292205, "author_name": "jamesphoward", "author_url": "", "post_date": "05/03/2021 18:34:54", "content": "

I think my best model I moved from 0.926 to 0.934 by adding in D4 labels.

", "votes": null, "replies": [] }, { "id": 1292264, "author_name": "rsmits", "author_url": "", "post_date": "05/03/2021 19:40:37", "content": "

Hi @jamesphoward that's an interresting improvement. So the remainder of your current score is based on TTA and ensemble likely.

\n

With 2 submissions that can be selected I think I should at least try it out…given that my current single fold best is over 0.931..so would be interresting to see what the D4 labels would do in that case. Combining in an ensemble some models trained with D4 and without D4 should keep it on the safe side. Interresting…

\n

Good luck with the remainder of the competition!

", "votes": null, "replies": [] }, { "id": 1292309, "author_name": "jamesphoward", "author_url": "", "post_date": "05/03/2021 20:18:32", "content": "

I certainly think it's wise to have a 2nd submission pipeline without D4, given the controversy around the glomeruli it contains.

", "votes": null, "replies": [] }, { "id": 1292643, "author_name": "zekunn", "author_url": "", "post_date": "05/04/2021 07:01:02", "content": "

Hello,is the whole test set 5 images?

", "votes": null, "replies": [] }, { "id": 1292654, "author_name": "lhagiimn", "author_url": "", "post_date": "05/04/2021 07:17:03", "content": "

No, it should be 15 images.

", "votes": null, "replies": [] }, { "id": 1293769, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/05/2021 06:42:13", "content": "

@jamesphoward could u help me interpret 7.26 to 7.34. Assuming your base score .900 .so improvement is 908 ?

", "votes": null, "replies": [] }, { "id": 1293792, "author_name": "jamesphoward", "author_url": "", "post_date": "05/05/2021 07:03:49", "content": "

Sorry that was a bizarre typo - I meant to say 0.926 to 0.934.

", "votes": null, "replies": [] }, { "id": 1293823, "author_name": "fabiendaniel", "author_url": "", "post_date": "05/05/2021 07:28:38", "content": "
\n

Sorry that was a bizarre typo - I meant to say 0.924 to 0.936.

\n
\n

do you mean 0.926 to 0.934 ;)

", "votes": null, "replies": [] }, { "id": 1293828, "author_name": "jamesphoward", "author_url": "", "post_date": "05/05/2021 07:33:32", "content": "

😳 yes, sorry (again)

", "votes": null, "replies": [] }, { "id": 1296628, "author_name": "thakurudit", "author_url": "", "post_date": "05/07/2021 11:41:41", "content": "

@jamesphoward I trained my models with hand labelled D4 tfrecord (using Zhao's dataset) but instead of giving me a boost on cv and lb it dropped the score from 0.920 to 0.744 on public lb. Also, another thing I noticed was my models started facing exploding gradients problem on appending the D4 tfrecord to the train set (which was fixed using clipnorm) but still the result was unexpected. Is there something wrong with the generation of D4 tfrecords ? or is that the models fails to train and learn on completely different data (D4) compared to the initial training set ?

", "votes": null, "replies": [] }, { "id": 1296641, "author_name": "jamesphoward", "author_url": "", "post_date": "05/07/2021 11:51:39", "content": "

Strange. I don't use tfrecords so I can't say, but I found my efficientnet linknet models trained almost identically when the D4 data was added in.

", "votes": null, "replies": [] }, { "id": 1297699, "author_name": "ulrich07", "author_url": "", "post_date": "05/08/2021 08:28:01", "content": "

Zhao's label is not the key. By probing LB i noticed that my CV prediction of d4 is more accurate than Zhao's hand-label

", "votes": null, "replies": [] }, { "id": 1297701, "author_name": "southsakura", "author_url": "", "post_date": "05/08/2021 08:30:24", "content": "

my d4 is more accurate than Zhao's too but I dont think it is useful for my model. I am still afraid to use that.

", "votes": null, "replies": [] }, { "id": 1297708, "author_name": "ulrich07", "author_url": "", "post_date": "05/08/2021 08:40:50", "content": "

Nice to see that you came to the same conclusion

", "votes": null, "replies": [] } ] }, { "id": 1292140, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/03/2021 17:40:15", "content": "

@lhagiimn Deepflash use randomized tiles for training and another set of tiles for validation not mixing with train as such. SO just trying to understand reason for overfit.
\nkfold also does similar the scheme that is based not on Tiff ids

", "votes": null, "replies": [ { "id": 1292583, "author_name": "something4kag", "author_url": "", "post_date": "05/04/2021 05:53:00", "content": "

There is no separation of which images are used for train or validation, both use all the files in deepflash public notebooks. Train does random tiles considering pdfs > random.random [0.0-1.0] to choose centers. Validation randomly samples the designated number of tiles. There is no guarantee of no overlap of tiles or sections of tiles. The use of random in various places tend to make results hard to reproduce even with seeds.

\n

Using stratified kfold e.g. based on mask coverage for tiles like from the iafoss datasets, or hold out of different ids for validation sampling, would ensure folds have different tiles (or images) for train or validation.

", "votes": null, "replies": [] }, { "id": 1292602, "author_name": "zekunn", "author_url": "", "post_date": "05/04/2021 06:26:31", "content": "

But when I use kfold,the lb decrease.

", "votes": null, "replies": [] }, { "id": 1292656, "author_name": "lhagiimn", "author_url": "", "post_date": "05/04/2021 07:23:40", "content": "

Yes, your right. I just said that it could be overfitted. So it couldn’t be overfitted as well.
\nMy opinion is that training and validation sets can be come from same image for current public notebook because validation set is random sample tiles. So trained model could easily be overfitted. The test set is different from training set, I think GroupKFold could be good evaluation strategy.

", "votes": null, "replies": [] }, { "id": 1292785, "author_name": "jaideepvalani", "author_url": "", "post_date": "05/04/2021 09:24:33", "content": "

I tried kfold split based on tiff ids , gkf results were uncorrelated completely. Model possibly overfits to position of ftu given all tiles of same tiff in one set

", "votes": null, "replies": [] }, { "id": 1297700, "author_name": "ulrich07", "author_url": "", "post_date": "05/08/2021 08:29:54", "content": "

I would rather trust CV than LB. believe you me. I lost two golds medals 😨

", "votes": null, "replies": [] } ] }, { "id": 1295240, "author_name": "jamesphoward", "author_url": "", "post_date": "05/06/2021 10:08:06", "content": "

The final shakeup probably wont be too different from this
\n\"The

", "votes": null, "replies": [ { "id": 1295303, "author_name": "lhagiimn", "author_url": "", "post_date": "05/06/2021 11:06:16", "content": "

I think so. I think gold zone would be safe.

", "votes": null, "replies": [] }, { "id": 1296082, "author_name": "jihangz", "author_url": "", "post_date": "05/07/2021 00:48:13", "content": "

Nooooo! It's sort ascending 😏

", "votes": null, "replies": [] }, { "id": 1297173, "author_name": "mpware", "author_url": "", "post_date": "05/07/2021 19:09:05", "content": "

I would prefer no shake-up 😉
\nBut I must admit that probability for strong shake-up is not null.

", "votes": null, "replies": [] }, { "id": 1297222, "author_name": "awsaf49", "author_url": "", "post_date": "05/07/2021 20:02:26", "content": "

@mpware did you guys use the infamous D48 ? 😜

", "votes": null, "replies": [] }, { "id": 1301580, "author_name": "jamesphoward", "author_url": "", "post_date": "05/11/2021 06:37:26", "content": "

Well that aged like a fine wine.

", "votes": null, "replies": [] } ] }, { "id": 1301244, "author_name": "lhagiimn", "author_url": "", "post_date": "05/11/2021 01:55:12", "content": "

It happened :(

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1290426": "I think there should be a shake-up in this competition.\nThe reasons are as follows:\n\n- Manually pseudo labelled image in public test set (public LB score should be higher)\n- Deepflash 2 notebook can be overfitted on the private leaderboard (the public kernel doesn't use Kfold cross-validation)\n- There are two kernels with high scores on the public leaderboard (public csv file can be submitted)", "1290461": "Is the Deepflash 2 only gets lb 922?", "1290466": "I got 92.8 on public LB and CV is around 94.5 using pseudo labeling. You can also read this discussion and comments https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234575\nThere is someone got more than 93.5 on public LB.", "1290528": "Would u like to join us ? We use keras models.", "1290534": "Thank you for asking. I'm sorry I can't join your team.", "1290696": "There will be a ridiculous shakeup at the top because I suspect most people towards the top have used the D48 labels (which may or may not have been a good idea). Furthermore, anyone who's done extensive test set labelling will inevitably be very high at the moment. Their scores will certainly diverge, either in a good way or a very bad way, from the others. It's going to be very exciting.", "1290698": "I totally agree with you", "1290717": "Aha,have you done these?", "1290721": "I've used Zhao's D48 labels, but no further labelling.", "1290785": "This is the problem with pseudo labeling. When I inspect my model's output (0.919 LB), I can't find any glomeruli that it hasn't identified! In total I can find 15 cases where it has either missed or hasn't created a perfect mask. And 15 misses does not explain the 0.081 dice score miss from a full score. Given that in cross validation I've achieved dice scores as high as 0.96, it has come to labeling issues. So if the top LB scores are obtained not because they overfit, but because the pseudo/manual labels they used are better, then they could score higher in the private test too. And that just sucks.", "1290999": "Hey @jamesphoward Question about the D48...How do you use it then?\n\n1. Do you use it to retrain a model?\n2. Or only use it to modify your final submission?\n\nI wonder how Kagglers put it to use where it would benefit both Public and Private LB. If it is only used to modify the submission file...then what's the use at all...beside looking good at the Public LB? It won't help for the Private LB in that case.\n\nIn that case there could indeed be nice shakeup.", "1291032": "I did (1).\n\nI think (2) is incredibly dangerous (although (1) might be, too) if you do it in a simple way, at least, because unless you've trained on D48, your model will make very unconfident predictions on those types of glomeruli. If you then use the D48 labels to calibrate your predictions, then you'll end up with an incredibly low prediction threshold for all other slides.", "1291062": "Hi @jamesphoward Thanks for your answer. \nInterresting choice to make there .... i you use it to train a model then there is not that much difference I suppose between automatically pseudo labelled data or manually labelled data.\n\nGiven a proper validation strategy you should be able to get a fair indication I guess of whether it is 'safe' or not.\n\nThat said....the final week and finish will be interresting for sure :-)", "1292140": "lhagiimn Deepflash use randomized tiles for training and another set of tiles for validation not mixing with train as such. SO just trying to understand reason for overfit.\nkfold also does similar the scheme that is based not on Tiff ids", "1292148": "jamesphoward u have not used PL of other test tiff ids ?", "1292182": "Nope I haven't", "1292204": "Did u find a big diff in LB score between with and without D4 label training( meaning setting just low th during inference no training on using labels)", "1292205": "I think my best model I moved from 0.926 to 0.934 by adding in D4 labels.", "1292264": "Hi @jamesphoward that's an interresting improvement. So the remainder of your current score is based on TTA and ensemble likely.\n\nWith 2 submissions that can be selected I think I should at least try it out...given that my current single fold best is over 0.931..so would be interresting to see what the D4 labels would do in that case. Combining in an ensemble some models trained with D4 and without D4 should keep it on the safe side. Interresting...\n\nGood luck with the remainder of the competition!", "1292309": "I certainly think it's wise to have a 2nd submission pipeline _without_ D4, given the controversy around the glomeruli it contains.", "1292583": "There is no separation of which images are used for train or validation, both use all the files in deepflash public notebooks. Train does random tiles considering pdfs > random.random [0.0-1.0] to choose centers. Validation randomly samples the designated number of tiles. There is no guarantee of no overlap of tiles or sections of tiles. The use of random in various places tend to make results hard to reproduce even with seeds. \n\nUsing stratified kfold e.g. based on mask coverage for tiles like from the iafoss datasets, or hold out of different ids for validation sampling, would ensure folds have different tiles (or images) for train or validation.", "1292602": "But when I use kfold,the lb decrease.", "1292643": "Hello,is the whole test set 5 images?", "1292654": "No, it should be 15 images.", "1292656": "Yes, your right. I just said that it could be overfitted. So it couldn’t be overfitted as well. \nMy opinion is that training and validation sets can be come from same image for current public notebook because validation set is random sample tiles. So trained model could easily be overfitted. The test set is different from training set, I think GroupKFold could be good evaluation strategy.", "1292785": "I tried kfold split based on tiff ids , gkf results were uncorrelated completely. Model possibly overfits to position of ftu given all tiles of same tiff in one set", "1293769": "jamesphoward could u help me interpret 7.26 to 7.34. Assuming your base score .900 .so improvement is 908 ?", "1293792": "Sorry that was a bizarre typo - I meant to say 0.926 to 0.934.", "1293823": "> Sorry that was a bizarre typo - I meant to say 0.924 to 0.936.\n\ndo you mean 0.926 to 0.934 ;)", "1293828": "😳 yes, sorry (again)", "1295240": "The final shakeup probably wont be too different from this\n![The final shakeup probably wont be too different from this](https://i.ibb.co/Xz0q1Q7/Screenshot-2021-05-06-at-11-04-31.png)", "1295303": "I think so. I think gold zone would be safe.", "1296082": "Nooooo! It's sort **ascending** 😏", "1296628": "jamesphoward I trained my models with hand labelled D4 tfrecord (using Zhao's dataset) but instead of giving me a boost on cv and lb it dropped the score from 0.920 to 0.744 on public lb. Also, another thing I noticed was my models started facing exploding gradients problem on appending the D4 tfrecord to the train set (which was fixed using clipnorm) but still the result was unexpected. Is there something wrong with the generation of D4 tfrecords ? or is that the models fails to train and learn on completely different data (D4) compared to the initial training set ?", "1296641": "Strange. I don't use tfrecords so I can't say, but I found my efficientnet linknet models trained almost identically when the D4 data was added in.", "1297173": "I would prefer no shake-up 😉\nBut I must admit that probability for strong shake-up is not null.", "1297222": "mpware did you guys use the infamous **D48** ? 😜", "1297699": "Zhao's label is not the key. By probing LB i noticed that my CV prediction of d4 is more accurate than Zhao's hand-label", "1297700": "I would rather trust CV than LB. believe you me. I lost two golds medals 😨", "1297701": "my d4 is more accurate than Zhao's too but I dont think it is useful for my model. I am still afraid to use that.", "1297708": "Nice to see that you came to the same conclusion", "1301244": "It happened :(", "1301580": "Well that aged like a fine wine." }, "source": "meta" }