![\"\"](\"https://nsa40.casimages.com/img/2021/03/27/210327061249285069.png\")
Hi,
\nDo we work on both non-sclerotic and sclerotic glomeruli on this competition?
\nCould organizer let us know (if allowed)? @leahscherschel ?
Looking at labels in training dataset, it looks it's non-sclerotic only. I'm not sure if there are some sclerotic at the bottom of aaa6a05cc.tiff train image. Some expert from business domain could confirm? If they are and not labelled they we have the answer (or the labels are noisy which is not expected since the update)
According to this document, both exist:
\nhttps://www.mdpi.com/2079-9292/9/3/503
Here is an example:
\n![\"\"](\"https://nsa40.casimages.com/img/2021/03/27/21032706224374201.png\")
All masks and training images here:
\nhttps://www.kaggle.com/mpware/masks-quick-eda-updated-data
\nI see only non-scletoric.
Thanks a lot.
", "messages": [ { "id": "1254439", "postDate": "03/27/2021 17:14:50", "content": "Hi,
\nDo we work on both non-sclerotic and sclerotic glomeruli on this competition?
\nCould organizer let us know (if allowed)? @leahscherschel ?
Looking at labels in training dataset, it looks it's non-sclerotic only. I'm not sure if there are some sclerotic at the bottom of aaa6a05cc.tiff train image. Some expert from business domain could confirm? If they are and not labelled they we have the answer (or the labels are noisy which is not expected since the update)
According to this document, both exist:
\nhttps://www.mdpi.com/2079-9292/9/3/503
Here is an example:
\n
All masks and training images here:
\nhttps://www.kaggle.com/mpware/masks-quick-eda-updated-data
\nI see only non-scletoric.
Thanks a lot.
", "rawMarkdown": "Hi,\n\nDo we work on both non-sclerotic and sclerotic glomeruli on this competition?\nCould organizer let us know (if allowed)? @leahscherschel ?\n\nLooking at labels in training dataset, it looks it's non-sclerotic only. I'm not sure if there are some sclerotic at the bottom of `aaa6a05cc.tiff` train image. Some expert from business domain could confirm? If they are and not labelled they we have the answer (or the labels are noisy which is not expected since the update)\n\n\n\nAccording to this document, both exist:\nhttps://www.mdpi.com/2079-9292/9/3/503\n\nHere is an example:\n\n\nAll masks and training images here:\nhttps://www.kaggle.com/mpware/masks-quick-eda-updated-data\nI see only non-scletoric.\n\nThanks a lot.", "votes": null }, { "id": "1254511", "postDate": "03/27/2021 18:29:04", "content": "Here is an assertive question from @mhermsen.
\nSclerotic glomerulis seem to be more difficult to detect. Maybe that's one of the reasons why PL scores went up after new data was introduced. If so, the old training set annotations/masks should be different than new ones.
\nMaybe someone who has kept the old masks can confirm whether they are different from the new ones.
", "rawMarkdown": "[Here](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/207884) is an assertive question from @mhermsen.\n\nSclerotic glomerulis seem to be more difficult to detect. Maybe that's one of the reasons why PL scores went up after new data was introduced. If so, the old training set annotations/masks should be different than new ones.\n\nMaybe someone who has kept the old masks can confirm whether they are different from the new ones.", "votes": null }, { "id": "1254536", "postDate": "03/27/2021 19:03:26", "content": "Thanks for the link, someone asked the same question 🙂.
\nWhat is weird is that, according to this post, some sclerotic glomeruli look to be in the public test ground truth. New data had high quality controls so I might miss something.
", "rawMarkdown": "Thanks for the link, someone asked the same question 🙂. \n\nWhat is weird is that, according to this [post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616), some sclerotic glomeruli look to be in the public test ground truth. New data had high quality controls so I might miss something.", "votes": null }, { "id": "1254755", "postDate": "03/28/2021 04:49:43", "content": "This image (and b9a3865fc.tiff) is also questioned in the Annotations Updated thread -
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116#1250326
Looks like hand labels are going to be needed here unless the public test image d488c759a was incorrectly labelled.
", "rawMarkdown": "This image (and b9a3865fc.tiff) is also questioned in the Annotations Updated thread -\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116#1250326\n\nLooks like hand labels are going to be needed here unless the public test image d488c759a was incorrectly labelled.", "votes": null }, { "id": "1254940", "postDate": "03/28/2021 09:08:07", "content": "I'm wondering if organizers could let us know how the annotators were organized?
\nSince this is the 2nd (I think) go at images for this competition I am not sure if there are differences between train, public test and new private test in these annotations. I thought there were issues even with train originally and while new private test was being sorted some of these were resolved especially with masks.
\nIf you have not seen this dataset already, it could be worth considering to include or at least some of these. Around 550+ have positive mask images and based on the info in the associated paper includes normal glomeruli and sclerosed glomeruli.
\nhttps://www.kaggle.com/baesiann/glomeruli-hubmap-external-1024x1024?select=masks_1024
Yes I've seen it. It is good but the author did not process all raw images to avoid most of sclerotic ones:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/208972#1198328
Also the initial dataset does not come with the masks (only bounding boxes images). Anyway, that's true that something could be done with this dataset.
", "rawMarkdown": "Yes I've seen it. It is good but the author did not process all raw images to avoid most of sclerotic ones:\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/208972#1198328\n\nAlso the initial dataset does not come with the masks (only bounding boxes images). Anyway, that's true that something could be done with this dataset.", "votes": null }, { "id": "1255304", "postDate": "03/28/2021 17:02:50", "content": "Would be good if the host can confirm this because I also getting the feeling that there are some sclerotic glomeruli labelled in the test set.
\nMaybe to make the private set evaluation even more objective, sclerotic glomeruli can be evaluated with a lower weight, which can be probably defined as an image with value 1 everywhere, but with lower values over sclerotic glomeruli. The best would be if medical experts can provide these weights per glomerulus, varying between 0 to 1 and representing a probability of the region to be sclerotic glomerulus (or weight values from 1 to 0, or some low value, to focus less on sclerotic ones). This way, models will still be evaluated with a more focus on non-sclerotic glomeruli. Essentially, this would mean that the evaluation metric will be slightly improved and they can leave it to us how things are labelled. This way, the amount of work to be done on the host side will be minimized as well.
", "rawMarkdown": "Would be good if the host can confirm this because I also getting the feeling that there are some sclerotic glomeruli labelled in the test set. \n\nMaybe to make the private set evaluation even more objective, sclerotic glomeruli can be evaluated with a lower weight, which can be probably defined as an image with value 1 everywhere, but with lower values over sclerotic glomeruli. The best would be if medical experts can provide these weights per glomerulus, varying between 0 to 1 and representing a probability of the region to be sclerotic glomerulus (or weight values from 1 to 0, or some low value, to focus less on sclerotic ones). This way, models will still be evaluated with a more focus on non-sclerotic glomeruli. Essentially, this would mean that the evaluation metric will be slightly improved and they can leave it to us how things are labelled. This way, the amount of work to be done on the host side will be minimized as well.", "votes": null }, { "id": "1255697", "postDate": "03/29/2021 04:44:36", "content": "Apologies, did not see the thread, just found the paper and dataset. The paper was interesting in that it looks at non-glomerular and glomerular areas first, then glomerular into normal and sclerosed glomeruli. Have not found anything so far in this competition that says they are looking for normal only and wonder if that would even make sense for its objectives. May try to track back to the original data to see how many sclerotic ones made it to masks in the dataset by baeslann. At least if a few it could be good to have as additional data, maybe eliminate having to try to hand label train.
", "rawMarkdown": "Apologies, did not see the thread, just found the paper and dataset. The paper was interesting in that it looks at non-glomerular and glomerular areas first, then glomerular into normal and sclerosed glomeruli. Have not found anything so far in this competition that says they are looking for normal only and wonder if that would even make sense for its objectives. May try to track back to the original data to see how many sclerotic ones made it to masks in the dataset by baeslann. At least if a few it could be good to have as additional data, maybe eliminate having to try to hand label train.", "votes": null }, { "id": "1255913", "postDate": "03/29/2021 10:58:58", "content": "Just to underline a little bit more this problem:
\nIf you look carefully at the training mask before and after the update, you will notice that the vas majority of the sclerotic glomeruli has been discarded in the updated version , for example into 1e22425f28 masks: at least 10 sclerotic glom have been discarded into the new version (I don't speak about the artifacts that have been also discarded !)
\nI didn't check for the other images but I'm rather confident that this is also the case because some of my old fold model (trained with data before the update) are strkingly good to almost capture every sclerosed glomerulli from d488c759a and on the contrary my newer model are not so good at this task.
\nThe problem is that the training set policy seems to have changed before and after the update( = it seems that there is no sclerotic glomerulli into the train set anymore) but it seems that the public test has not been designed to take this change into account leading to this crazy situation of competitors manually drawing mask to fit better the LB with the \"hope\" that the private test set follow the same policy.
\nAccording to me there are three possibilities:
\n1) As the training set now seems not to include sclerotic glom and public/private test do include: we need to work on an \"class imbalance\" problem and develop models that are able to deal with \"rare sclerotic\" cases (I mean \"rare\" according to the training set). No problem for me but this should be known because this is a different problem that the one stated into the competition: \"glomerulli segmentation\" becomes \"segment glomerulli using an partially incomplete dataset\"
\n2) or the public/private test are updated accordingly to the training set policy by discarding sclerotic glomerulli.
\n3) the training set is updated to include sclerotic glomerulli as before. Impossible to try to train a \"good model\" using only the old masks as the quality was really poor (shift, hallucinated gloms, …) and the bunch of new training data doesn't follow the same policy.
\nI don't know what is the best but the organizer should clarify their minds as the deadline is approaching and models are long to concetpualize,train and tune.
\nFa.
", "rawMarkdown": "Just to underline a little bit more this problem:\n\nIf you look carefully at the training mask before and after the update, you will notice that the vas majority of the sclerotic glomeruli has been discarded in the updated version , for example into 1e22425f28 masks: at least 10 sclerotic glom have been discarded into the new version (I don't speak about the artifacts that have been also discarded !)\n\n I didn't check for the other images but I'm rather confident that this is also the case because some of my old fold model (trained with data before the update) are strkingly good to almost capture every sclerosed glomerulli from d488c759a and on the contrary my newer model are not so good at this task.\n\nThe problem is that the training set policy seems to have changed before and after the update( = it seems that there is no sclerotic glomerulli into the train set anymore) but it seems that the public test has not been designed to take this change into account leading to this crazy situation of competitors manually drawing mask to fit better the LB with the \"hope\" that the private test set follow the same policy. \n\nAccording to me there are three possibilities:\n\n1) As the training set now seems not to include sclerotic glom and public/private test do include: we need to work on an \"class imbalance\" problem and develop models that are able to deal with \"rare sclerotic\" cases (I mean \"rare\" according to the training set). No problem for me but this should be known because this is a different problem that the one stated into the competition: \"glomerulli segmentation\" becomes \"segment glomerulli using an partially incomplete dataset\"\n\n2) or the public/private test are updated accordingly to the training set policy by discarding sclerotic glomerulli.\n\n3) the training set is updated to include sclerotic glomerulli as before. Impossible to try to train a \"good model\" using only the old masks as the quality was really poor (shift, hallucinated gloms, ...) and the bunch of new training data doesn't follow the same policy.\n\nI don't know what is the best but the organizer should clarify their minds as the deadline is approaching and models are long to concetpualize,train and tune.\n\nFa.", "votes": null }, { "id": "1256238", "postDate": "03/29/2021 17:20:08", "content": "Thanks for this excellent feedback! Maybe the organizers want us to train a model without any tagged sclerotic examples to detect sclerotic 🤕. Or it's just a mistake in public test data, or the test data has not been reviewed by the quality control or it's intended noise. I agree they should clarify. @addisonhoward Could you shed a light on this?
", "rawMarkdown": "Thanks for this excellent feedback! Maybe the organizers want us to train a model without any tagged sclerotic examples to detect sclerotic 🤕. Or it's just a mistake in public test data, or the test data has not been reviewed by the quality control or it's intended noise. I agree they should clarify. @addisonhoward Could you shed a light on this?", "votes": null }, { "id": "1256424", "postDate": "03/29/2021 21:03:25", "content": "Hello all ,
\nFollowing the data update, the glomeruli annotated in all datasets (public and private, train and test) are non-sclerotic.
", "rawMarkdown": "Hello all ,\n\nFollowing the data update, the glomeruli annotated in all datasets (public and private, train and test) are non-sclerotic.", "votes": null }, { "id": "1256446", "postDate": "03/29/2021 21:44:21", "content": "Thanks a lot for your answer. What was confusing is the color change (darker) that looks like a sclerotic one but now we know it's not.
", "rawMarkdown": "Thanks a lot for your answer. What was confusing is the color change (darker) that looks like a sclerotic one but now we know it's not.", "votes": null }, { "id": "1256455", "postDate": "03/29/2021 22:02:33", "content": "Thanks Leah for your quick response.
\nNevertheless Carno Zhao has pointed out some strong evidences that at least one of the public test image (d488c759a) should contain sclerotic glomeruli in the ground truth (or segmental sclerotic glomeruli … I don't know precisely the exact name but to be short not \"normal\" or \"hypertrophic\" ones); Hand labelling of \"rare glomeruli\" (\"rare\" according to the updated training set) lead to a huge LB score improvement.
\nCould you confirm that this particular image (d488c759a) doesn't have any \"not normal/hypertrophic\" glomerulli in the ground truth ?
\nThank you very much in advance for all the further clarification you will provide.
\nFa.
", "rawMarkdown": "Thanks Leah for your quick response.\n\nNevertheless Carno Zhao has pointed out some strong evidences that at least one of the public test image (d488c759a) should contain sclerotic glomeruli in the ground truth (or segmental sclerotic glomeruli ... I don't know precisely the exact name but to be short not \"normal\" or \"hypertrophic\" ones); Hand labelling of \"rare glomeruli\" (\"rare\" according to the updated training set) lead to a huge LB score improvement.\n\nCould you confirm that this particular image (d488c759a) doesn't have any \"not normal/hypertrophic\" glomerulli in the ground truth ?\n\nThank you very much in advance for all the further clarification you will provide.\n\nFa.", "votes": null }, { "id": "1256523", "postDate": "03/30/2021 01:23:44", "content": "Thank you for clarifying the intent of the annotation, @leahscherschel. Going further into the details, how should we interpret this finding by @carnozhao : https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616 ? It seems some sclerotic glomeruli are annotated in the public test set (maybe not in the private, but probably in the public).
", "rawMarkdown": "Thank you for clarifying the intent of the annotation, @leahscherschel. Going further into the details, how should we interpret this finding by @carnozhao : https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616 ? It seems some sclerotic glomeruli are annotated in the public test set (maybe not in the private, but probably in the public).", "votes": null }, { "id": "1256561", "postDate": "03/30/2021 03:00:01", "content": "thank you for statement @leahscherschel.
", "rawMarkdown": "thank you for statement @leahscherschel.", "votes": null }, { "id": "1256680", "postDate": "03/30/2021 06:45:18", "content": "it would definetly help we get to know for sure the type of FTU presents in tiff,can we conclude tht there are although rare Unhealthy Glomeruli present.
", "rawMarkdown": "it would definetly help we get to know for sure the type of FTU presents in tiff,can we conclude tht there are although rare Unhealthy Glomeruli present.", "votes": null }, { "id": "1257021", "postDate": "03/30/2021 13:14:53", "content": "Thanks, but sclerotic or not, it seems like there is some unusual type of glomeruli in public test set, which is not present in new annotated data. By adding that type to train ( via aaa6a05cc, d488c759a or some other images) one can improve score on public private set (which is ok imho).
\nProblem can arise if for example only d488c759a image has that type annotated by mistake (making LB all wrong), and all other images from private set are correlated with 'standart' glomeruli from train set
There are also seem to be different types of glomeruli sclerosis, so, I hope I understand correctly that by \"non-sclerotic\" @leahscherschel means that we should be ruling out glomeruli where \"global sclerosis\" is present, like those in Figure 1: https://www.kidneypathology.com/English_version/Histologic_patterns.html? And not the ones which are partially sclerotic. Most of us are not domain experts, so bear with me if I'm writing some nonsense here 🙂
", "rawMarkdown": "There are also seem to be different types of glomeruli sclerosis, so, I hope I understand correctly that by \"non-sclerotic\" @leahscherschel means that we should be ruling out glomeruli where \"global sclerosis\" is present, like those in Figure 1: https://www.kidneypathology.com/English_version/Histologic_patterns.html? And not the ones which are partially sclerotic. Most of us are not domain experts, so bear with me if I'm writing some nonsense here 🙂", "votes": null }, { "id": "1257326", "postDate": "03/30/2021 18:15:23", "content": "Your comment makes sense, @gdonchyts. Maybe that's what's happening.
", "rawMarkdown": "Your comment makes sense, @gdonchyts. Maybe that's what's happening.", "votes": null }, { "id": "1257450", "postDate": "03/30/2021 21:14:00", "content": "It seems that what we called \"sclerotic glomeruli\" should be called \"fibrous crescent\" according to this lab link : https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/
\nWe can see well the difference on the 2 pictures comparing fibrous crescent and sclerotic glomeruli.
So I reformulate my previous question using the good wording now :) :
\n@leahscherschel : Does d488c759a contain annotated fibrous crescent or crescent glomerulis in the ground truth ?
\nThanks you by advance for the kind help you could provide.
\nFa.
", "rawMarkdown": "It seems that what we called \"sclerotic glomeruli\" should be called \"fibrous crescent\" according to this lab link : https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/\nWe can see well the difference on the 2 pictures comparing fibrous crescent and sclerotic glomeruli.\n\nSo I reformulate my previous question using the good wording now :) :\n\n@leahscherschel : Does d488c759a contain annotated fibrous crescent or crescent glomerulis in the ground truth ?\n\nThanks you by advance for the kind help you could provide.\n\nFa.", "votes": null }, { "id": "1272538", "postDate": "04/13/2021 14:55:51", "content": "Up ! Why is this not clarified yet ? Can anyone confirm one of the following things :
\nFibrous crescent glomeruli are expected to be predicted by the model. Which is why they are annotated in d488c759a (public_test), as well in the private test set.
Fibrous crescent glomeruli are not expected to be predicted by the model. Which is why they are not annotated in aaa6a05cc (train), as well in the private test set. They were annotated in d488c759a (public_test) by mistake.
The competition is ending in a month and we still don't know what we're supposed to predict…
\nPoke @leahscherschel @addissonhoward @katyborner
", "rawMarkdown": "Up ! Why is this not clarified yet ? Can anyone confirm one of the following things :\n\n- [Fibrous crescent glomeruli](https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/) are expected to be predicted by the model. Which is why they are annotated in `d488c759a (public_test)`, as well in the private test set.\n\n- [Fibrous crescent glomeruli](https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/) are **not** expected to be predicted by the model. Which is why they are not annotated in `aaa6a05cc (train)`, as well in the private test set. They were annotated in `d488c759a (public_test)` by mistake.\n\nThe competition is ending in a month and we still don't know what we're supposed to predict...\n\nPoke @leahscherschel @addissonhoward @katyborner", "votes": null }, { "id": "1272577", "postDate": "04/13/2021 15:27:55", "content": "100% agree. No matter the real definition for such different glomeruli, this question should be answered.
", "rawMarkdown": "100% agree. No matter the real definition for such different glomeruli, this question should be answered.", "votes": null }, { "id": "1272585", "postDate": "04/13/2021 15:36:31", "content": "I feel the same
", "rawMarkdown": "I feel the same", "votes": null }, { "id": "1272702", "postDate": "04/13/2021 17:36:25", "content": "I think the same way.
\nThe d488c759a(public_test) and aaa6a05cc(train) have the same
\nFibrous crescent glomeruli, but only one side is annotated.
\nI don't know why they keep silent on this matter.
Will the competition be extended again?
\nI just want to know what's correct or not.
", "rawMarkdown": "Will the competition be extended again?\n\nI just want to know what's correct or not.", "votes": null }, { "id": "1272926", "postDate": "04/13/2021 22:09:12", "content": "Hum … Has someone been able to increase his LB by \"hand pseudo labeling\" the fibrous crescent glom present into 57512b7f1 ?
\nI ask this serious question because at one point I'm more and more suspecting the fact that ONLY d488c759a contains annotated Fibrous glom into the ground truth. Why ?
\nI detail my strategy:
\nOne of my \"best\" base model has got a LB of 0.924 without trying to detect this fibrous glom: OK
\n1) I've trained a dummy model able to detect ONLY fibrous glom (and no others !) using my \"own home made d488c759a fibrous glom hand labelling\".
\n2) I use this model in combination with one of my \"best\" base model to calculate the d488c759a mask + my best base only to calculate the other masks: my LB increased to 0.933 (LB +0.009)
\n3) As I noticed that my model was able to detect such glom in great proportion into 57512b7f1 I also included this image: so I used a combination of my \"best\" base + my fibrous glom model for only 57512b7f1 and my \"best\" base for the others : my LB drops to 0.915 (-0.009) … Am I crazy ? Has someone else noticed the same thing ? Someone did included 57512b7f1 fibrous glom into their models (may be top LB kagglers ??) ? When I used a combination of my best model + my fibrous glom for every public test image my LB remains almost the same: LB 0.923 meaning that all my successful d488 predictions has been cleared by my \"not successful\" 57512b7f1 predictions (as the other images seem to not contain such fibrous glom)
Either my model has greatly greatly overfited (which is a real possibility !) or the public test GT contains \"mistakenly annotated\" fibrous gloms for d488c759a, or the public test GT contains \"mistakenly missed\" fibrous glom for 57512b7f1…
\nIs someone noticed the same or ?
", "rawMarkdown": "Hum ... Has someone been able to increase his LB by \"hand pseudo labeling\" the fibrous crescent glom present into **57512b7f1** ?\n\nI ask this serious question because at one point I'm more and more suspecting the fact that ONLY d488c759a contains annotated Fibrous glom into the ground truth. Why ?\n\nI detail my strategy:\n\nOne of my \"best\" base model has got a LB of 0.924 without trying to detect this fibrous glom: OK\n\n1) I've trained a dummy model able to detect ONLY fibrous glom (and no others !) using my \"own home made d488c759a fibrous glom hand labelling\".\n2) I use this model in combination with one of my \"best\" base model to calculate the d488c759a mask + my best base only to calculate the other masks: my LB increased to 0.933 (LB +0.009)\n3) As I noticed that my model was able to detect such glom in great proportion into 57512b7f1 I also included this image: so I used a combination of my \"best\" base + my fibrous glom model for only 57512b7f1 and my \"best\" base for the others : my LB drops to 0.915 (-0.009) ... Am I crazy ? Has someone else noticed the same thing ? Someone did included 57512b7f1 fibrous glom into their models (may be top LB kagglers ??) ? When I used a combination of my best model + my fibrous glom for every public test image my LB remains almost the same: LB 0.923 meaning that all my successful d488 predictions has been cleared by my \"not successful\" 57512b7f1 predictions (as the other images seem to not contain such fibrous glom)\n\nEither my model has greatly greatly overfited (which is a real possibility !) or the public test GT contains \"mistakenly annotated\" fibrous gloms for d488c759a, or the public test GT contains \"mistakenly missed\" fibrous glom for 57512b7f1...\n\nIs someone noticed the same or ?", "votes": null }, { "id": "1272948", "postDate": "04/13/2021 22:57:56", "content": "🙄
\n![\"game\"](\"https://upload.wikimedia.org/wikipedia/en/9/96/A_commonly_used_style_for_many_microsoft_games%2C_originating_with_Microsoft_Minesweeper.png\")
Upvoted. I still dont know which strategies to be chosen. In such case, LB will be meaningless and untrustworthy. Meanwhile I have also found some (around 50) mistakes-like samples in the training data.
", "rawMarkdown": "Upvoted. I still dont know which strategies to be chosen. In such case, LB will be meaningless and untrustworthy. Meanwhile I have also found some (around 50) mistakes-like samples in the training data.", "votes": null }, { "id": "1273300", "postDate": "04/14/2021 08:23:27", "content": "I agree. There are something need to be clear to figure out why d488 image be the outliers. And are our models proposed to predict the fibrous glom?
", "rawMarkdown": "I agree. There are something need to be clear to figure out why d488 image be the outliers. And are our models proposed to predict the fibrous glom?", "votes": null }, { "id": "1273331", "postDate": "04/14/2021 08:52:15", "content": "@siltoon That's interesting. I think that means fibrous-crescent glomeruli were not annotated in 57512b7f1, your model seems fine.
Now our model is trained without such kind of dark glomeruli.
\nCannot decide to include them if there is no reliable response by the host, although it will increase public score.
@siltoon haveu tried this combination best base model+ fib for 575* and d488c759a
", "rawMarkdown": "siltoon haveu tried this combination best base model+ fib for 575* and d488c759a", "votes": null }, { "id": "1273674", "postDate": "04/14/2021 14:33:06", "content": "@Jaideep: Yes and the result is rather similar to my base LB meaning that the fibrous predicted into 575 seems to not be included into the GT.
\nBut as for now I've absolutely NO mathematical proof of this possible discrepancies between 575 and d488 fibrous … that's why I was asking whether someone here has got the same feeling …
", "rawMarkdown": "Jaideep: Yes and the result is rather similar to my base LB meaning that the fibrous predicted into 575 seems to not be included into the GT.\n\nBut as for now I've absolutely NO mathematical proof of this possible discrepancies between 575 and d488 fibrous ... that's why I was asking whether someone here has got the same feeling ...", "votes": null }, { "id": "1274303", "postDate": "04/15/2021 07:12:46", "content": "Adding some links,excerpts here that may be of interest wrt Glomerular crescent -
\nmechanisms-of-glomerular-crescent-formation
\n\"Glomerular crescent formation appears to represent a nonspecific response to severe injury to the glomerular capillary wall. The initiating event is the development of physical gaps (also called rents or holes) in the glomerular capillary wall, glomerular basement membrane, and Bowman's capsule. These gaps permit the entry into Bowman's space of coagulation factors, which lead to fibrin formation (due to conversion of fibrinogen to fibrin polymers and delayed fibrinolysis) and cellular elements (such as monocytes and lymphocytes), both of which promote crescent formation \"
The glomerular crescent: triggers, evolution, resolution, and implications for therapy
\n\"Crescents are classical histopathological lesions found in severe forms of rapidly progressive glomerulonephritis, also referred to as crescentic glomerulonephritis (CGN). …
\nCrescent formation requires an additional level of injury to the glomerular filtration barrier, that is,vascular injury with holes or major breaks of the glomerular basement membrane (GBM) that trigger the plasmatic coagulation cascade within Bowman‘s space \"
Now, i never use any manually anotated label, and now i also want to know:
\n\"Fibrous crescent glomeruli are expected to be predicted by the model. Which is why they are annotated in d488c759a (public_test), as well in the private test set.
Fibrous crescent glomeruli are not expected to be predicted by the model. Which is why they are not annotated in aaa6a05cc (train), as well in the private test set. They were annotated in d488c759a (public_test) by mistake.
\n\"
\n@leahscherschel @addissonhoward @katyborner
👍 great insight
", "rawMarkdown": "👍 great insight", "votes": null } ], "comments": [ { "id": 1254511, "author_name": "isakev", "author_url": "", "post_date": "03/27/2021 18:29:04", "content": "Here is an assertive question from @mhermsen.
\nSclerotic glomerulis seem to be more difficult to detect. Maybe that's one of the reasons why PL scores went up after new data was introduced. If so, the old training set annotations/masks should be different than new ones.
\nMaybe someone who has kept the old masks can confirm whether they are different from the new ones.
", "votes": null, "replies": [ { "id": 1254536, "author_name": "mpware", "author_url": "", "post_date": "03/27/2021 19:03:26", "content": "Thanks for the link, someone asked the same question 🙂.
\nWhat is weird is that, according to this post, some sclerotic glomeruli look to be in the public test ground truth. New data had high quality controls so I might miss something.
", "votes": null, "replies": [] }, { "id": 1256680, "author_name": "jaideepvalani", "author_url": "", "post_date": "03/30/2021 06:45:18", "content": "it would definetly help we get to know for sure the type of FTU presents in tiff,can we conclude tht there are although rare Unhealthy Glomeruli present.
", "votes": null, "replies": [] } ] }, { "id": 1254755, "author_name": "something4kag", "author_url": "", "post_date": "03/28/2021 04:49:43", "content": "This image (and b9a3865fc.tiff) is also questioned in the Annotations Updated thread -
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116#1250326
Looks like hand labels are going to be needed here unless the public test image d488c759a was incorrectly labelled.
", "votes": null, "replies": [ { "id": 1254940, "author_name": "mpware", "author_url": "", "post_date": "03/28/2021 09:08:07", "content": "I'm wondering if organizers could let us know how the annotators were organized?
\nSince this is the 2nd (I think) go at images for this competition I am not sure if there are differences between train, public test and new private test in these annotations. I thought there were issues even with train originally and while new private test was being sorted some of these were resolved especially with masks.
\nIf you have not seen this dataset already, it could be worth considering to include or at least some of these. Around 550+ have positive mask images and based on the info in the associated paper includes normal glomeruli and sclerosed glomeruli.
\nhttps://www.kaggle.com/baesiann/glomeruli-hubmap-external-1024x1024?select=masks_1024
Yes I've seen it. It is good but the author did not process all raw images to avoid most of sclerotic ones:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/208972#1198328
Also the initial dataset does not come with the masks (only bounding boxes images). Anyway, that's true that something could be done with this dataset.
", "votes": null, "replies": [] }, { "id": 1255697, "author_name": "something4kag", "author_url": "", "post_date": "03/29/2021 04:44:36", "content": "Apologies, did not see the thread, just found the paper and dataset. The paper was interesting in that it looks at non-glomerular and glomerular areas first, then glomerular into normal and sclerosed glomeruli. Have not found anything so far in this competition that says they are looking for normal only and wonder if that would even make sense for its objectives. May try to track back to the original data to see how many sclerotic ones made it to masks in the dataset by baeslann. At least if a few it could be good to have as additional data, maybe eliminate having to try to hand label train.
", "votes": null, "replies": [] } ] }, { "id": 1255304, "author_name": "gdonchyts", "author_url": "", "post_date": "03/28/2021 17:02:50", "content": "Would be good if the host can confirm this because I also getting the feeling that there are some sclerotic glomeruli labelled in the test set.
\nMaybe to make the private set evaluation even more objective, sclerotic glomeruli can be evaluated with a lower weight, which can be probably defined as an image with value 1 everywhere, but with lower values over sclerotic glomeruli. The best would be if medical experts can provide these weights per glomerulus, varying between 0 to 1 and representing a probability of the region to be sclerotic glomerulus (or weight values from 1 to 0, or some low value, to focus less on sclerotic ones). This way, models will still be evaluated with a more focus on non-sclerotic glomeruli. Essentially, this would mean that the evaluation metric will be slightly improved and they can leave it to us how things are labelled. This way, the amount of work to be done on the host side will be minimized as well.
", "votes": null, "replies": [] }, { "id": 1255913, "author_name": "siltoon", "author_url": "", "post_date": "03/29/2021 10:58:58", "content": "Just to underline a little bit more this problem:
\nIf you look carefully at the training mask before and after the update, you will notice that the vas majority of the sclerotic glomeruli has been discarded in the updated version , for example into 1e22425f28 masks: at least 10 sclerotic glom have been discarded into the new version (I don't speak about the artifacts that have been also discarded !)
\nI didn't check for the other images but I'm rather confident that this is also the case because some of my old fold model (trained with data before the update) are strkingly good to almost capture every sclerosed glomerulli from d488c759a and on the contrary my newer model are not so good at this task.
\nThe problem is that the training set policy seems to have changed before and after the update( = it seems that there is no sclerotic glomerulli into the train set anymore) but it seems that the public test has not been designed to take this change into account leading to this crazy situation of competitors manually drawing mask to fit better the LB with the \"hope\" that the private test set follow the same policy.
\nAccording to me there are three possibilities:
\n1) As the training set now seems not to include sclerotic glom and public/private test do include: we need to work on an \"class imbalance\" problem and develop models that are able to deal with \"rare sclerotic\" cases (I mean \"rare\" according to the training set). No problem for me but this should be known because this is a different problem that the one stated into the competition: \"glomerulli segmentation\" becomes \"segment glomerulli using an partially incomplete dataset\"
\n2) or the public/private test are updated accordingly to the training set policy by discarding sclerotic glomerulli.
\n3) the training set is updated to include sclerotic glomerulli as before. Impossible to try to train a \"good model\" using only the old masks as the quality was really poor (shift, hallucinated gloms, …) and the bunch of new training data doesn't follow the same policy.
\nI don't know what is the best but the organizer should clarify their minds as the deadline is approaching and models are long to concetpualize,train and tune.
\nFa.
", "votes": null, "replies": [ { "id": 1256238, "author_name": "mpware", "author_url": "", "post_date": "03/29/2021 17:20:08", "content": "Thanks for this excellent feedback! Maybe the organizers want us to train a model without any tagged sclerotic examples to detect sclerotic 🤕. Or it's just a mistake in public test data, or the test data has not been reviewed by the quality control or it's intended noise. I agree they should clarify. @addisonhoward Could you shed a light on this?
", "votes": null, "replies": [] } ] }, { "id": 1256424, "author_name": "leahscherschel", "author_url": "", "post_date": "03/29/2021 21:03:25", "content": "Hello all ,
\nFollowing the data update, the glomeruli annotated in all datasets (public and private, train and test) are non-sclerotic.
", "votes": null, "replies": [ { "id": 1256446, "author_name": "mpware", "author_url": "", "post_date": "03/29/2021 21:44:21", "content": "Thanks a lot for your answer. What was confusing is the color change (darker) that looks like a sclerotic one but now we know it's not.
", "votes": null, "replies": [] }, { "id": 1256523, "author_name": "felipekitamura", "author_url": "", "post_date": "03/30/2021 01:23:44", "content": "Thank you for clarifying the intent of the annotation, @leahscherschel. Going further into the details, how should we interpret this finding by @carnozhao : https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616 ? It seems some sclerotic glomeruli are annotated in the public test set (maybe not in the private, but probably in the public).
", "votes": null, "replies": [] }, { "id": 1256561, "author_name": "cswwp347724", "author_url": "", "post_date": "03/30/2021 03:00:01", "content": "thank you for statement @leahscherschel.
", "votes": null, "replies": [] }, { "id": 1257021, "author_name": "bakeryproducts", "author_url": "", "post_date": "03/30/2021 13:14:53", "content": "Thanks, but sclerotic or not, it seems like there is some unusual type of glomeruli in public test set, which is not present in new annotated data. By adding that type to train ( via aaa6a05cc, d488c759a or some other images) one can improve score on public private set (which is ok imho).
\nProblem can arise if for example only d488c759a image has that type annotated by mistake (making LB all wrong), and all other images from private set are correlated with 'standart' glomeruli from train set
There are also seem to be different types of glomeruli sclerosis, so, I hope I understand correctly that by \"non-sclerotic\" @leahscherschel means that we should be ruling out glomeruli where \"global sclerosis\" is present, like those in Figure 1: https://www.kidneypathology.com/English_version/Histologic_patterns.html? And not the ones which are partially sclerotic. Most of us are not domain experts, so bear with me if I'm writing some nonsense here 🙂
", "votes": null, "replies": [] }, { "id": 1257326, "author_name": "felipekitamura", "author_url": "", "post_date": "03/30/2021 18:15:23", "content": "Your comment makes sense, @gdonchyts. Maybe that's what's happening.
", "votes": null, "replies": [] }, { "id": 1257450, "author_name": "siltoon", "author_url": "", "post_date": "03/30/2021 21:14:00", "content": "It seems that what we called \"sclerotic glomeruli\" should be called \"fibrous crescent\" according to this lab link : https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/
\nWe can see well the difference on the 2 pictures comparing fibrous crescent and sclerotic glomeruli.
So I reformulate my previous question using the good wording now :) :
\n@leahscherschel : Does d488c759a contain annotated fibrous crescent or crescent glomerulis in the ground truth ?
\nThanks you by advance for the kind help you could provide.
\nFa.
", "votes": null, "replies": [] } ] }, { "id": 1256455, "author_name": "siltoon", "author_url": "", "post_date": "03/29/2021 22:02:33", "content": "Thanks Leah for your quick response.
\nNevertheless Carno Zhao has pointed out some strong evidences that at least one of the public test image (d488c759a) should contain sclerotic glomeruli in the ground truth (or segmental sclerotic glomeruli … I don't know precisely the exact name but to be short not \"normal\" or \"hypertrophic\" ones); Hand labelling of \"rare glomeruli\" (\"rare\" according to the updated training set) lead to a huge LB score improvement.
\nCould you confirm that this particular image (d488c759a) doesn't have any \"not normal/hypertrophic\" glomerulli in the ground truth ?
\nThank you very much in advance for all the further clarification you will provide.
\nFa.
", "votes": null, "replies": [] }, { "id": 1272538, "author_name": "theoviel", "author_url": "", "post_date": "04/13/2021 14:55:51", "content": "Up ! Why is this not clarified yet ? Can anyone confirm one of the following things :
\nFibrous crescent glomeruli are expected to be predicted by the model. Which is why they are annotated in d488c759a (public_test), as well in the private test set.
Fibrous crescent glomeruli are not expected to be predicted by the model. Which is why they are not annotated in aaa6a05cc (train), as well in the private test set. They were annotated in d488c759a (public_test) by mistake.
The competition is ending in a month and we still don't know what we're supposed to predict…
\nPoke @leahscherschel @addissonhoward @katyborner
", "votes": null, "replies": [ { "id": 1272577, "author_name": "mpware", "author_url": "", "post_date": "04/13/2021 15:27:55", "content": "100% agree. No matter the real definition for such different glomeruli, this question should be answered.
", "votes": null, "replies": [] }, { "id": 1272585, "author_name": "bakeryproducts", "author_url": "", "post_date": "04/13/2021 15:36:31", "content": "I feel the same
", "votes": null, "replies": [] }, { "id": 1272702, "author_name": "gwanghan", "author_url": "", "post_date": "04/13/2021 17:36:25", "content": "I think the same way.
\nThe d488c759a(public_test) and aaa6a05cc(train) have the same
\nFibrous crescent glomeruli, but only one side is annotated.
\nI don't know why they keep silent on this matter.
Will the competition be extended again?
\nI just want to know what's correct or not.
", "votes": null, "replies": [] }, { "id": 1272926, "author_name": "siltoon", "author_url": "", "post_date": "04/13/2021 22:09:12", "content": "Hum … Has someone been able to increase his LB by \"hand pseudo labeling\" the fibrous crescent glom present into 57512b7f1 ?
\nI ask this serious question because at one point I'm more and more suspecting the fact that ONLY d488c759a contains annotated Fibrous glom into the ground truth. Why ?
\nI detail my strategy:
\nOne of my \"best\" base model has got a LB of 0.924 without trying to detect this fibrous glom: OK
\n1) I've trained a dummy model able to detect ONLY fibrous glom (and no others !) using my \"own home made d488c759a fibrous glom hand labelling\".
\n2) I use this model in combination with one of my \"best\" base model to calculate the d488c759a mask + my best base only to calculate the other masks: my LB increased to 0.933 (LB +0.009)
\n3) As I noticed that my model was able to detect such glom in great proportion into 57512b7f1 I also included this image: so I used a combination of my \"best\" base + my fibrous glom model for only 57512b7f1 and my \"best\" base for the others : my LB drops to 0.915 (-0.009) … Am I crazy ? Has someone else noticed the same thing ? Someone did included 57512b7f1 fibrous glom into their models (may be top LB kagglers ??) ? When I used a combination of my best model + my fibrous glom for every public test image my LB remains almost the same: LB 0.923 meaning that all my successful d488 predictions has been cleared by my \"not successful\" 57512b7f1 predictions (as the other images seem to not contain such fibrous glom)
Either my model has greatly greatly overfited (which is a real possibility !) or the public test GT contains \"mistakenly annotated\" fibrous gloms for d488c759a, or the public test GT contains \"mistakenly missed\" fibrous glom for 57512b7f1…
\nIs someone noticed the same or ?
", "votes": null, "replies": [] }, { "id": 1272948, "author_name": "gdonchyts", "author_url": "", "post_date": "04/13/2021 22:57:56", "content": "🙄
\n
Upvoted. I still dont know which strategies to be chosen. In such case, LB will be meaningless and untrustworthy. Meanwhile I have also found some (around 50) mistakes-like samples in the training data.
", "votes": null, "replies": [] }, { "id": 1273300, "author_name": "shinewine", "author_url": "", "post_date": "04/14/2021 08:23:27", "content": "I agree. There are something need to be clear to figure out why d488 image be the outliers. And are our models proposed to predict the fibrous glom?
", "votes": null, "replies": [] }, { "id": 1273331, "author_name": "theoviel", "author_url": "", "post_date": "04/14/2021 08:52:15", "content": "@siltoon That's interesting. I think that means fibrous-crescent glomeruli were not annotated in 57512b7f1, your model seems fine.
@siltoon haveu tried this combination best base model+ fib for 575* and d488c759a
", "votes": null, "replies": [] }, { "id": 1273674, "author_name": "siltoon", "author_url": "", "post_date": "04/14/2021 14:33:06", "content": "@Jaideep: Yes and the result is rather similar to my base LB meaning that the fibrous predicted into 575 seems to not be included into the GT.
\nBut as for now I've absolutely NO mathematical proof of this possible discrepancies between 575 and d488 fibrous … that's why I was asking whether someone here has got the same feeling …
", "votes": null, "replies": [] }, { "id": 1275172, "author_name": "cswwp347724", "author_url": "", "post_date": "04/16/2021 03:12:59", "content": "Now, i never use any manually anotated label, and now i also want to know:
\n\"Fibrous crescent glomeruli are expected to be predicted by the model. Which is why they are annotated in d488c759a (public_test), as well in the private test set.
Fibrous crescent glomeruli are not expected to be predicted by the model. Which is why they are not annotated in aaa6a05cc (train), as well in the private test set. They were annotated in d488c759a (public_test) by mistake.
\n\"
\n@leahscherschel @addissonhoward @katyborner
Now our model is trained without such kind of dark glomeruli.
\nCannot decide to include them if there is no reliable response by the host, although it will increase public score.
Adding some links,excerpts here that may be of interest wrt Glomerular crescent -
\nmechanisms-of-glomerular-crescent-formation
\n\"Glomerular crescent formation appears to represent a nonspecific response to severe injury to the glomerular capillary wall. The initiating event is the development of physical gaps (also called rents or holes) in the glomerular capillary wall, glomerular basement membrane, and Bowman's capsule. These gaps permit the entry into Bowman's space of coagulation factors, which lead to fibrin formation (due to conversion of fibrinogen to fibrin polymers and delayed fibrinolysis) and cellular elements (such as monocytes and lymphocytes), both of which promote crescent formation \"
The glomerular crescent: triggers, evolution, resolution, and implications for therapy
\n\"Crescents are classical histopathological lesions found in severe forms of rapidly progressive glomerulonephritis, also referred to as crescentic glomerulonephritis (CGN). …
\nCrescent formation requires an additional level of injury to the glomerular filtration barrier, that is,vascular injury with holes or major breaks of the glomerular basement membrane (GBM) that trigger the plasmatic coagulation cascade within Bowman‘s space \"
👍 great insight
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1254439": "Hi,\n\nDo we work on both non-sclerotic and sclerotic glomeruli on this competition?\nCould organizer let us know (if allowed)? @leahscherschel ?\n\nLooking at labels in training dataset, it looks it's non-sclerotic only. I'm not sure if there are some sclerotic at the bottom of `aaa6a05cc.tiff` train image. Some expert from business domain could confirm? If they are and not labelled they we have the answer (or the labels are noisy which is not expected since the update)\n\n\n\nAccording to this document, both exist:\nhttps://www.mdpi.com/2079-9292/9/3/503\n\nHere is an example:\n\n\nAll masks and training images here:\nhttps://www.kaggle.com/mpware/masks-quick-eda-updated-data\nI see only non-scletoric.\n\nThanks a lot.", "1254511": "[Here](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/207884) is an assertive question from @mhermsen.\n\nSclerotic glomerulis seem to be more difficult to detect. Maybe that's one of the reasons why PL scores went up after new data was introduced. If so, the old training set annotations/masks should be different than new ones.\n\nMaybe someone who has kept the old masks can confirm whether they are different from the new ones.", "1254536": "Thanks for the link, someone asked the same question 🙂. \n\nWhat is weird is that, according to this [post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616), some sclerotic glomeruli look to be in the public test ground truth. New data had high quality controls so I might miss something.", "1254755": "This image (and b9a3865fc.tiff) is also questioned in the Annotations Updated thread -\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116#1250326\n\nLooks like hand labels are going to be needed here unless the public test image d488c759a was incorrectly labelled.", "1254940": "I'm wondering if organizers could let us know how the annotators were organized?\n- Did they have the same initial requirements?\n- Did they cross-check each other?\n- What is behind the high quality controls?", "1254959": "Since this is the 2nd (I think) go at images for this competition I am not sure if there are differences between train, public test and new private test in these annotations. I thought there were issues even with train originally and while new private test was being sorted some of these were resolved especially with masks. \n\nIf you have not seen this dataset already, it could be worth considering to include or at least some of these. Around 550+ have positive mask images and based on the info in the associated paper includes normal glomeruli and sclerosed glomeruli. \nhttps://www.kaggle.com/baesiann/glomeruli-hubmap-external-1024x1024?select=masks_1024", "1254971": "Yes I've seen it. It is good but the author did not process all raw images to avoid most of sclerotic ones:\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/208972#1198328\n\nAlso the initial dataset does not come with the masks (only bounding boxes images). Anyway, that's true that something could be done with this dataset.", "1255304": "Would be good if the host can confirm this because I also getting the feeling that there are some sclerotic glomeruli labelled in the test set. \n\nMaybe to make the private set evaluation even more objective, sclerotic glomeruli can be evaluated with a lower weight, which can be probably defined as an image with value 1 everywhere, but with lower values over sclerotic glomeruli. The best would be if medical experts can provide these weights per glomerulus, varying between 0 to 1 and representing a probability of the region to be sclerotic glomerulus (or weight values from 1 to 0, or some low value, to focus less on sclerotic ones). This way, models will still be evaluated with a more focus on non-sclerotic glomeruli. Essentially, this would mean that the evaluation metric will be slightly improved and they can leave it to us how things are labelled. This way, the amount of work to be done on the host side will be minimized as well.", "1255697": "Apologies, did not see the thread, just found the paper and dataset. The paper was interesting in that it looks at non-glomerular and glomerular areas first, then glomerular into normal and sclerosed glomeruli. Have not found anything so far in this competition that says they are looking for normal only and wonder if that would even make sense for its objectives. May try to track back to the original data to see how many sclerotic ones made it to masks in the dataset by baeslann. At least if a few it could be good to have as additional data, maybe eliminate having to try to hand label train.", "1255913": "Just to underline a little bit more this problem:\n\nIf you look carefully at the training mask before and after the update, you will notice that the vas majority of the sclerotic glomeruli has been discarded in the updated version , for example into 1e22425f28 masks: at least 10 sclerotic glom have been discarded into the new version (I don't speak about the artifacts that have been also discarded !)\n\n I didn't check for the other images but I'm rather confident that this is also the case because some of my old fold model (trained with data before the update) are strkingly good to almost capture every sclerosed glomerulli from d488c759a and on the contrary my newer model are not so good at this task.\n\nThe problem is that the training set policy seems to have changed before and after the update( = it seems that there is no sclerotic glomerulli into the train set anymore) but it seems that the public test has not been designed to take this change into account leading to this crazy situation of competitors manually drawing mask to fit better the LB with the \"hope\" that the private test set follow the same policy. \n\nAccording to me there are three possibilities:\n\n1) As the training set now seems not to include sclerotic glom and public/private test do include: we need to work on an \"class imbalance\" problem and develop models that are able to deal with \"rare sclerotic\" cases (I mean \"rare\" according to the training set). No problem for me but this should be known because this is a different problem that the one stated into the competition: \"glomerulli segmentation\" becomes \"segment glomerulli using an partially incomplete dataset\"\n\n2) or the public/private test are updated accordingly to the training set policy by discarding sclerotic glomerulli.\n\n3) the training set is updated to include sclerotic glomerulli as before. Impossible to try to train a \"good model\" using only the old masks as the quality was really poor (shift, hallucinated gloms, ...) and the bunch of new training data doesn't follow the same policy.\n\nI don't know what is the best but the organizer should clarify their minds as the deadline is approaching and models are long to concetpualize,train and tune.\n\nFa.", "1256238": "Thanks for this excellent feedback! Maybe the organizers want us to train a model without any tagged sclerotic examples to detect sclerotic 🤕. Or it's just a mistake in public test data, or the test data has not been reviewed by the quality control or it's intended noise. I agree they should clarify. @addisonhoward Could you shed a light on this?", "1256424": "Hello all ,\n\nFollowing the data update, the glomeruli annotated in all datasets (public and private, train and test) are non-sclerotic.", "1256446": "Thanks a lot for your answer. What was confusing is the color change (darker) that looks like a sclerotic one but now we know it's not.", "1256455": "Thanks Leah for your quick response.\n\nNevertheless Carno Zhao has pointed out some strong evidences that at least one of the public test image (d488c759a) should contain sclerotic glomeruli in the ground truth (or segmental sclerotic glomeruli ... I don't know precisely the exact name but to be short not \"normal\" or \"hypertrophic\" ones); Hand labelling of \"rare glomeruli\" (\"rare\" according to the updated training set) lead to a huge LB score improvement.\n\nCould you confirm that this particular image (d488c759a) doesn't have any \"not normal/hypertrophic\" glomerulli in the ground truth ?\n\nThank you very much in advance for all the further clarification you will provide.\n\nFa.", "1256523": "Thank you for clarifying the intent of the annotation, @leahscherschel. Going further into the details, how should we interpret this finding by @carnozhao : https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616 ? It seems some sclerotic glomeruli are annotated in the public test set (maybe not in the private, but probably in the public).", "1256561": "thank you for statement @leahscherschel.", "1256680": "it would definetly help we get to know for sure the type of FTU presents in tiff,can we conclude tht there are although rare Unhealthy Glomeruli present.", "1257021": "Thanks, but sclerotic or not, it seems like there is some unusual type of glomeruli in public test set, which is not present in new annotated data. By adding that type to train ( via aaa6a05cc, d488c759a or some other images) one can improve score on public private set (which is ok imho).\nProblem can arise if for example only d488c759a image has that type annotated by mistake (making LB all wrong), and all other images from private set are correlated with 'standart' glomeruli from train set", "1257217": "There are also seem to be different types of glomeruli sclerosis, so, I hope I understand correctly that by \"non-sclerotic\" @leahscherschel means that we should be ruling out glomeruli where \"global sclerosis\" is present, like those in Figure 1: https://www.kidneypathology.com/English_version/Histologic_patterns.html? And not the ones which are partially sclerotic. Most of us are not domain experts, so bear with me if I'm writing some nonsense here 🙂", "1257326": "Your comment makes sense, @gdonchyts. Maybe that's what's happening.", "1257450": "It seems that what we called \"sclerotic glomeruli\" should be called \"fibrous crescent\" according to this lab link : https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/\nWe can see well the difference on the 2 pictures comparing fibrous crescent and sclerotic glomeruli.\n\nSo I reformulate my previous question using the good wording now :) :\n\n@leahscherschel : Does d488c759a contain annotated fibrous crescent or crescent glomerulis in the ground truth ?\n\nThanks you by advance for the kind help you could provide.\n\nFa.", "1272538": "Up ! Why is this not clarified yet ? Can anyone confirm one of the following things :\n\n- [Fibrous crescent glomeruli](https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/) are expected to be predicted by the model. Which is why they are annotated in `d488c759a (public_test)`, as well in the private test set.\n\n- [Fibrous crescent glomeruli](https://www.arkanalabs.com/chronic-lesions-of-anca-fibrous-cresecent/) are **not** expected to be predicted by the model. Which is why they are not annotated in `aaa6a05cc (train)`, as well in the private test set. They were annotated in `d488c759a (public_test)` by mistake.\n\nThe competition is ending in a month and we still don't know what we're supposed to predict...\n\nPoke @leahscherschel @addissonhoward @katyborner", "1272577": "100% agree. No matter the real definition for such different glomeruli, this question should be answered.", "1272585": "I feel the same", "1272702": "I think the same way.\nThe **d488c759a(public_test)** and **aaa6a05cc(train)** have the same \n`Fibrous crescent glomeruli`, but only one side is annotated.\nI don't know why they keep silent on this matter.", "1272797": "Will the competition be extended again?\n\nI just want to know what's correct or not.", "1272926": "Hum ... Has someone been able to increase his LB by \"hand pseudo labeling\" the fibrous crescent glom present into **57512b7f1** ?\n\nI ask this serious question because at one point I'm more and more suspecting the fact that ONLY d488c759a contains annotated Fibrous glom into the ground truth. Why ?\n\nI detail my strategy:\n\nOne of my \"best\" base model has got a LB of 0.924 without trying to detect this fibrous glom: OK\n\n1) I've trained a dummy model able to detect ONLY fibrous glom (and no others !) using my \"own home made d488c759a fibrous glom hand labelling\".\n2) I use this model in combination with one of my \"best\" base model to calculate the d488c759a mask + my best base only to calculate the other masks: my LB increased to 0.933 (LB +0.009)\n3) As I noticed that my model was able to detect such glom in great proportion into 57512b7f1 I also included this image: so I used a combination of my \"best\" base + my fibrous glom model for only 57512b7f1 and my \"best\" base for the others : my LB drops to 0.915 (-0.009) ... Am I crazy ? Has someone else noticed the same thing ? Someone did included 57512b7f1 fibrous glom into their models (may be top LB kagglers ??) ? When I used a combination of my best model + my fibrous glom for every public test image my LB remains almost the same: LB 0.923 meaning that all my successful d488 predictions has been cleared by my \"not successful\" 57512b7f1 predictions (as the other images seem to not contain such fibrous glom)\n\nEither my model has greatly greatly overfited (which is a real possibility !) or the public test GT contains \"mistakenly annotated\" fibrous gloms for d488c759a, or the public test GT contains \"mistakenly missed\" fibrous glom for 57512b7f1...\n\nIs someone noticed the same or ?", "1272948": "🙄\n\n", "1273061": "Upvoted. I still dont know which strategies to be chosen. In such case, LB will be meaningless and untrustworthy. Meanwhile I have also found some (around 50) mistakes-like samples in the training data.", "1273300": "I agree. There are something need to be clear to figure out why d488 image be the outliers. And are our models proposed to predict the fibrous glom?", "1273331": "siltoon That's interesting. I think that means fibrous-crescent glomeruli were not annotated in `57512b7f1 `, your model seems fine.", "1273382": "Now our model is trained without such kind of dark glomeruli.\nCannot decide to include them if there is no reliable response by the host, although it will increase public score.", "1273495": "siltoon haveu tried this combination best base model+ fib for 575* and d488c759a", "1273674": "Jaideep: Yes and the result is rather similar to my base LB meaning that the fibrous predicted into 575 seems to not be included into the GT.\n\nBut as for now I've absolutely NO mathematical proof of this possible discrepancies between 575 and d488 fibrous ... that's why I was asking whether someone here has got the same feeling ...", "1274303": "Adding some links,excerpts here that may be of interest wrt Glomerular crescent -\n\n[mechanisms-of-glomerular-crescent-formation](https://www.uptodate.com/contents/mechanisms-of-glomerular-crescent-formation)\n\"Glomerular crescent formation appears to represent a nonspecific response to severe injury to the glomerular capillary wall. The initiating event is the development of physical gaps (also called rents or holes) in the glomerular capillary wall, glomerular basement membrane, and Bowman's capsule. These gaps permit the entry into Bowman's space of coagulation factors, which lead to fibrin formation (due to conversion of fibrinogen to fibrin polymers and delayed fibrinolysis) and cellular elements (such as monocytes and lymphocytes), both of which promote crescent formation \"\n\n[The glomerular crescent: triggers, evolution, resolution, and implications for therapy](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7170443/)\n\"Crescents are classical histopathological lesions found in severe forms of rapidly progressive glomerulonephritis, also referred to as crescentic glomerulonephritis (CGN). ...\nCrescent formation requires an additional level of injury to the glomerular filtration barrier, that is,vascular injury with holes or major breaks of the glomerular basement membrane (GBM) that trigger the plasmatic coagulation cascade within Bowman‘s space \"", "1275172": "Now, i never use any manually anotated label, and now i also want to know:\n\"Fibrous crescent glomeruli are expected to be predicted by the model. Which is why they are annotated in d488c759a (public_test), as well in the private test set.\n\nFibrous crescent glomeruli are not expected to be predicted by the model. Which is why they are not annotated in aaa6a05cc (train), as well in the private test set. They were annotated in d488c759a (public_test) by mistake.\n\"\n@leahscherschel @addissonhoward @katyborner", "1280708": "👍 great insight" }, "source": "meta" }