{ "id": 198116, "title": "Annotations updated", "url": "/competitions/hubmap-kidney-segmentation/discussion/198116", "author_name": "Phil Culliton", "post_date": "2020-11-19T21:05:28.005000", "votes": 34, "comment_count": 22, "views": 0, "content": "

Hi all - thank you for your participation and your diligent work. Thanks especially to @felipekitamura who pointed out annotation issues here and for everyone involved in that discussion.

\n

The annotations for one training and one public test file have been updated - that would be c68fe75ea in test and e79de561c in train. Annotations were re-checked for all files and updated if necessary. No updates were required in the private test set.

\n

We will be re-running the non-sample kernels that were submitted, but this may take some time, depending on the amount of processing time required (we usually re-run each kernel several times to ensure proper functioning). In the meantime, any new submissions you make will reflect the updated data.

\n

Thanks again!

", "messages": [ { "id": 1084234, "postDate": "2020-11-19T21:05:28.007Z", "content": "

Hi all - thank you for your participation and your diligent work. Thanks especially to @felipekitamura who pointed out annotation issues here and for everyone involved in that discussion.

\n

The annotations for one training and one public test file have been updated - that would be c68fe75ea in test and e79de561c in train. Annotations were re-checked for all files and updated if necessary. No updates were required in the private test set.

\n

We will be re-running the non-sample kernels that were submitted, but this may take some time, depending on the amount of processing time required (we usually re-run each kernel several times to ensure proper functioning). In the meantime, any new submissions you make will reflect the updated data.

\n

Thanks again!

", "rawMarkdown": "Hi all - thank you for your participation and your diligent work. Thanks especially to @felipekitamura who pointed out annotation issues [here](https://admin.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664) and for everyone involved in that discussion.\n\nThe annotations for one training and one public test file have been updated - that would be `c68fe75ea` in test and `e79de561c` in train. Annotations were re-checked for all files and updated if necessary. No updates were required in the private test set.\n\nWe will be re-running the non-sample kernels that were submitted, but this may take some time, depending on the amount of processing time required (we usually re-run each kernel several times to ensure proper functioning). In the meantime, any new submissions you make will reflect the updated data.\n\nThanks again!", "votes": 34 }, { "id": 1250326, "postDate": "2021-03-24T00:42:38.593Z", "content": "

What are we supposed to do with labels which still seem to be incorrectly labelled in the training images, would it be useful to report their coordinates somewhere?

\n

For example, I could see a mismatch between my model predictions and training labels in aaa6a05cc.tiff and after inspection, they look like some glomeruli missing in the input labels (or maybe it's my wishful thinking πŸ™‚). I've just inspected a few regions in a single image, there are probably more:

\n

~[9260, 3743] - approximate center of the region
\n\"error1\"

\n

~[5300, 10258], here inset images on the right are original labels (top) and model prediction (bottom)
\n\"error2\"

\n

Would be nice to prevent these missing labels at least in the private dataset. This can be probably done by double-checking all predicted glomeruli regions using several top models and inspect regions mismatching with current private set labels, fixing incorrect or missing ones after consulting with medical experts.

", "rawMarkdown": "What are we supposed to do with labels which still seem to be incorrectly labelled in the training images, would it be useful to report their coordinates somewhere? \n\nFor example, I could see a mismatch between my model predictions and training labels in aaa6a05cc.tiff and after inspection, they look like some glomeruli missing in the input labels (or maybe it's my wishful thinking πŸ™‚). I've just inspected a few regions in a single image, there are probably more: \n\n~[9260, 3743] - approximate center of the region\n![error1](https://i.ibb.co/tM1bVHD/image.png)\n\n~[5300, 10258], here inset images on the right are original labels (top) and model prediction (bottom)\n![error2](https://i.ibb.co/TkrHmbQ/image.png)\n\nWould be nice to prevent these missing labels at least in the private dataset. This can be probably done by double-checking all predicted glomeruli regions using several top models and inspect regions mismatching with current private set labels, fixing incorrect or missing ones after consulting with medical experts.", "votes": 12, "replies": [ { "id": 1250331, "postDate": "2021-03-24T00:58:58.567Z", "content": "

Two more unlabeled in b9a3865fc.tiff:

\n

\"image\"

", "rawMarkdown": "Two more unlabeled in b9a3865fc.tiff:\n\n![image](https://i.ibb.co/42ssqcF/image.png)", "votes": 6 }, { "id": 1250769, "postDate": "2021-03-24T09:04:16.873Z", "content": "

I've just noticed a reply from the host in another thread that hand-labelling is allowed, https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1250442 - thanks! This makes the competition more interesting, though as non-pathologists we probably lack the experience to label everything correctly.

\n

Let's hope that the private dataset is labelled much more rigorously than the training set so that the errors like the ones above won't be present during the final evaluation.

", "rawMarkdown": "I've just noticed a reply from the host in another thread that hand-labelling is allowed, https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1250442 - thanks! This makes the competition more interesting, though as non-pathologists we probably lack the experience to label everything correctly.\n\nLet's hope that the private dataset is labelled much more rigorously than the training set so that the errors like the ones above won't be present during the final evaluation.", "votes": 2 }, { "id": 1253298, "postDate": "2021-03-26T15:17:19.503Z", "content": "

https://ibb.co/pj382JH

\n

It also seems like afae8098 has multiple issues.

\n

https://ibb.co/m6rY0rz

\n

095bf7a1 too

", "rawMarkdown": "https://ibb.co/pj382JH\n\nIt also seems like afae8098 has multiple issues. \n\nhttps://ibb.co/m6rY0rz\n\n095bf7a1 too" }, { "id": 1253898, "postDate": "2021-03-27T06:40:41.593Z", "content": "

@gdonchyts Thanks for sharing. Also realize there still seems to be labeling errors in the train dataset.
\nI don't think test has more accurate labels than train…, but I hope so.
\nIf there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.
\nHand-labeling is allowed, but I have no methods to judge whether they are correct or not.
\nFinally I don't want to spend time on labeling to be honest.

", "rawMarkdown": "@gdonchyts Thanks for sharing. Also realize there still seems to be labeling errors in the train dataset.\nI don't think test has more accurate labels than train..., but I hope so.\nIf there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.\nHand-labeling is allowed, but I have no methods to judge whether they are correct or not.\nFinally I don't want to spend time on labeling to be honest.", "votes": 2 }, { "id": 1254137, "postDate": "2021-03-27T11:21:21.550Z", "content": "
\n

If there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.

\n
\n

Yes, if they are also incomplete, our private runs will be evaluated using a \"rolling a dice\" metric πŸ˜ƒ. And since labelling of the training set is allowed - even when we reach 99.9% by manually labelling the public test set - this will be useless if the private test set is not up to the same quality. I hope the competition hosts will run some of our notebooks randomly and inspect the results very carefully, adding missing and removing bad labels in the private dataset, to make sure these labels are 99% good.

", "rawMarkdown": "> If there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.\n\nYes, if they are also incomplete, our private runs will be evaluated using a \"rolling a dice\" metric πŸ˜ƒ. And since labelling of the training set is allowed - even when we reach 99.9% by manually labelling the public test set - this will be useless if the private test set is not up to the same quality. I hope the competition hosts will run some of our notebooks randomly and inspect the results very carefully, adding missing and removing bad labels in the private dataset, to make sure these labels are 99% good.", "votes": 1 }, { "id": 1255051, "postDate": "2021-03-28T11:30:07.143Z", "content": "

About b9a3865fc.tiff this is probably due to being in the artifact / ruptured part of the specimen. I dont think, that those glomeruli are of particular interest. The issue is, that the annotation in these areas is not too consistent.

", "rawMarkdown": "About b9a3865fc.tiff this is probably due to being in the artifact / ruptured part of the specimen. I dont think, that those glomeruli are of particular interest. The issue is, that the annotation in these areas is not too consistent." }, { "id": 1276352, "postDate": "2021-04-17T12:51:36.950Z", "content": "

@gdonchyts \"rolling a Dice\" you mean. ;)

\n

And indeed, if the error is the same in the private dataset, it isn't useful to correct the training unfortunately. It might be even better to randomly introduce more errors just in case. :p

", "rawMarkdown": "@gdonchyts \"rolling a Dice\" you mean. ;)\n\nAnd indeed, if the error is the same in the private dataset, it isn't useful to correct the training unfortunately. It might be even better to randomly introduce more errors just in case. :p " }, { "id": 1276356, "postDate": "2021-04-17T13:03:31.350Z", "content": "

Yes, if there are similar systematic errors present in labels in the private dataset.

", "rawMarkdown": "Yes, if there are similar systematic errors present in labels in the private dataset. ", "votes": 2 }, { "id": 1288904, "postDate": "2021-04-30T13:02:03.877Z", "content": "

Gain on one Tiff could result into Loss at other .

", "rawMarkdown": "Gain on one Tiff could result into Loss at other . " }, { "id": 1290181, "postDate": "2021-05-01T17:32:32.310Z", "content": "

@gdonchyts are all missing labels in train are SC or some genuine ones are missed too that is healthy ones

", "rawMarkdown": "@gdonchyts are all missing labels in train are SC or some genuine ones are missed too that is healthy ones " }, { "id": 1290222, "postDate": "2021-05-01T18:09:28.953Z", "content": "

Some of them look sufficiently healthy, when compared to true labels. The question is if this is also the case in private data.

", "rawMarkdown": "Some of them look sufficiently healthy, when compared to true labels. The question is if this is also the case in private data." } ] }, { "id": 1104503, "postDate": "2020-12-07T02:12:54.373Z", "content": "

@philculliton @juyingnan

\n

please refer to the evidence and findings at:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201816

\n

the annotations following train images are shifted
\n54f2eec69
\n095bf7a1f
\ne79de561c (still shifted after update)

\n

maybe the public and private test dataset suffer the same issues
\ne.g c68fe75ea and afa5e8098

\n

Thank you very much.

", "rawMarkdown": "@philculliton @juyingnan \n\nplease refer to the evidence and findings at:\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201816\n\nthe annotations following train images are shifted\n54f2eec69\n095bf7a1f\ne79de561c (still shifted after update)\n\nmaybe the public and private test dataset suffer the same issues\ne.g c68fe75ea and afa5e8098\n\nThank you very much.", "votes": 15 }, { "id": 1161827, "postDate": "2021-01-20T19:11:10.037Z", "content": "

Does anyone know when the new train/public test datasets will be released?

", "rawMarkdown": "Does anyone know when the new train/public test datasets will be released?", "votes": 1, "replies": [ { "id": 1161894, "postDate": "2021-01-20T19:59:10.293Z", "content": "

Soonish I hope. :D
\nJoke aside, I have no idea but hopefully by the end of the month maybe?

", "rawMarkdown": "Soonish I hope. :D\nJoke aside, I have no idea but hopefully by the end of the month maybe?", "votes": 1 }, { "id": 1238158, "postDate": "2021-03-14T17:47:03.183Z", "content": "

It has been now: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/224826

", "rawMarkdown": "It has been now: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/224826" } ] }, { "id": 1085186, "postDate": "2020-11-20T18:20:37.927Z", "content": "

Thank you!

\n

pointed out annotation issues here and for everyone involved in that discussion.
\nI think you have attached a wrong link to here.

", "rawMarkdown": "Thank you!\n\n`pointed out annotation issues here and for everyone involved in that discussion.`\nI think you have attached a wrong link to **here**.", "votes": 1 }, { "id": 1150708, "postDate": "2021-01-12T19:30:46.237Z", "content": "

Is seems the annotation issues link https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116 is protected.

\n

It seems the correct link is the following: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664.

", "rawMarkdown": "Is seems the annotation issues link https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116 is protected. \n\nIt seems the correct link is the following: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664. " }, { "id": 1086286, "postDate": "2020-11-21T14:48:15.520Z", "content": "

Nice, I just found many wrong annotations 20 minutes ago.

", "rawMarkdown": "Nice, I just found many wrong annotations 20 minutes ago. " }, { "id": 1085257, "postDate": "2020-11-20T19:34:18.210Z", "content": "

I am also unable to access that link.

", "rawMarkdown": "I am also unable to access that link.", "replies": [ { "id": 1085825, "postDate": "2020-11-21T08:11:25.900Z", "content": "

https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664

\n

This is the link, if you haven't got it yet.

", "rawMarkdown": "https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664\n\nThis is the link, if you haven't got it yet." } ] }, { "id": 1108812, "postDate": "2020-12-11T03:10:23.380Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1250326, "author_name": "Gena", "author_url": "", "post_date": "2021-03-24T00:42:38.593000", "content": "

What are we supposed to do with labels which still seem to be incorrectly labelled in the training images, would it be useful to report their coordinates somewhere?

\n

For example, I could see a mismatch between my model predictions and training labels in aaa6a05cc.tiff and after inspection, they look like some glomeruli missing in the input labels (or maybe it's my wishful thinking πŸ™‚). I've just inspected a few regions in a single image, there are probably more:

\n

~[9260, 3743] - approximate center of the region
\n\"error1\"

\n

~[5300, 10258], here inset images on the right are original labels (top) and model prediction (bottom)
\n\"error2\"

\n

Would be nice to prevent these missing labels at least in the private dataset. This can be probably done by double-checking all predicted glomeruli regions using several top models and inspect regions mismatching with current private set labels, fixing incorrect or missing ones after consulting with medical experts.

", "votes": 12, "replies": [ { "id": 1250331, "author_name": "Gena", "author_url": "", "post_date": "2021-03-24T00:58:58.567000", "content": "

Two more unlabeled in b9a3865fc.tiff:

\n

\"image\"

", "votes": 6, "replies": [] }, { "id": 1250769, "author_name": "Gena", "author_url": "", "post_date": "2021-03-24T09:04:16.873000", "content": "

I've just noticed a reply from the host in another thread that hand-labelling is allowed, https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1250442 - thanks! This makes the competition more interesting, though as non-pathologists we probably lack the experience to label everything correctly.

\n

Let's hope that the private dataset is labelled much more rigorously than the training set so that the errors like the ones above won't be present during the final evaluation.

", "votes": 2, "replies": [] }, { "id": 1253298, "author_name": "Zero", "author_url": "", "post_date": "2021-03-26T15:17:19.503000", "content": "

https://ibb.co/pj382JH

\n

It also seems like afae8098 has multiple issues.

\n

https://ibb.co/m6rY0rz

\n

095bf7a1 too

", "votes": 0, "replies": [] }, { "id": 1253898, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-27T06:40:41.593000", "content": "

@gdonchyts Thanks for sharing. Also realize there still seems to be labeling errors in the train dataset.
\nI don't think test has more accurate labels than train…, but I hope so.
\nIf there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.
\nHand-labeling is allowed, but I have no methods to judge whether they are correct or not.
\nFinally I don't want to spend time on labeling to be honest.

", "votes": 2, "replies": [] }, { "id": 1254137, "author_name": "Gena", "author_url": "", "post_date": "2021-03-27T11:21:21.550000", "content": "
\n

If there are incorrect or missing labels in the test especially private one, our model and ranking won't be evaluated appropriately.

\n
\n

Yes, if they are also incomplete, our private runs will be evaluated using a \"rolling a dice\" metric πŸ˜ƒ. And since labelling of the training set is allowed - even when we reach 99.9% by manually labelling the public test set - this will be useless if the private test set is not up to the same quality. I hope the competition hosts will run some of our notebooks randomly and inspect the results very carefully, adding missing and removing bad labels in the private dataset, to make sure these labels are 99% good.

", "votes": 1, "replies": [] }, { "id": 1255051, "author_name": "Philipp Sodmann", "author_url": "", "post_date": "2021-03-28T11:30:07.143000", "content": "

About b9a3865fc.tiff this is probably due to being in the artifact / ruptured part of the specimen. I dont think, that those glomeruli are of particular interest. The issue is, that the annotation in these areas is not too consistent.

", "votes": 0, "replies": [] }, { "id": 1276352, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-04-17T12:51:36.950000", "content": "

@gdonchyts \"rolling a Dice\" you mean. ;)

\n

And indeed, if the error is the same in the private dataset, it isn't useful to correct the training unfortunately. It might be even better to randomly introduce more errors just in case. :p

", "votes": 0, "replies": [] }, { "id": 1276356, "author_name": "Gena", "author_url": "", "post_date": "2021-04-17T13:03:31.350000", "content": "

Yes, if there are similar systematic errors present in labels in the private dataset.

", "votes": 2, "replies": [] }, { "id": 1288904, "author_name": "Jaideep", "author_url": "", "post_date": "2021-04-30T13:02:03.877000", "content": "

Gain on one Tiff could result into Loss at other .

", "votes": 0, "replies": [] }, { "id": 1290181, "author_name": "Jaideep", "author_url": "", "post_date": "2021-05-01T17:32:32.310000", "content": "

@gdonchyts are all missing labels in train are SC or some genuine ones are missed too that is healthy ones

", "votes": 0, "replies": [] }, { "id": 1290222, "author_name": "Gena", "author_url": "", "post_date": "2021-05-01T18:09:28.953000", "content": "

Some of them look sufficiently healthy, when compared to true labels. The question is if this is also the case in private data.

", "votes": 0, "replies": [] } ] }, { "id": 1104503, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-07T02:12:54.373000", "content": "

@philculliton @juyingnan

\n

please refer to the evidence and findings at:
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201816

\n

the annotations following train images are shifted
\n54f2eec69
\n095bf7a1f
\ne79de561c (still shifted after update)

\n

maybe the public and private test dataset suffer the same issues
\ne.g c68fe75ea and afa5e8098

\n

Thank you very much.

", "votes": 15, "replies": [] }, { "id": 1161827, "author_name": "Ramin Nateghi", "author_url": "", "post_date": "2021-01-20T19:11:10.037000", "content": "

Does anyone know when the new train/public test datasets will be released?

", "votes": 1, "replies": [ { "id": 1161894, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-01-20T19:59:10.293000", "content": "

Soonish I hope. :D
\nJoke aside, I have no idea but hopefully by the end of the month maybe?

", "votes": 1, "replies": [] }, { "id": 1238158, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-03-14T17:47:03.183000", "content": "

It has been now: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/224826

", "votes": 0, "replies": [] } ] }, { "id": 1085186, "author_name": "Kishan Joshi", "author_url": "", "post_date": "2020-11-20T18:20:37.927000", "content": "

Thank you!

\n

pointed out annotation issues here and for everyone involved in that discussion.
\nI think you have attached a wrong link to here.

", "votes": 1, "replies": [] }, { "id": 1150708, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-01-12T19:30:46.237000", "content": "

Is seems the annotation issues link https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116 is protected.

\n

It seems the correct link is the following: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664.

", "votes": 0, "replies": [] }, { "id": 1086286, "author_name": "aria", "author_url": "", "post_date": "2020-11-21T14:48:15.520000", "content": "

Nice, I just found many wrong annotations 20 minutes ago.

", "votes": 0, "replies": [] }, { "id": 1085257, "author_name": "Brendan Doohan", "author_url": "", "post_date": "2020-11-20T19:34:18.210000", "content": "

I am also unable to access that link.

", "votes": 0, "replies": [ { "id": 1085825, "author_name": "Kishan Joshi", "author_url": "", "post_date": "2020-11-21T08:11:25.900000", "content": "

https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664

\n

This is the link, if you haven't got it yet.

", "votes": 0, "replies": [] } ] }, { "id": 1108812, "author_name": "", "author_url": "", "post_date": "2020-12-11T03:10:23.380000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1084234": "Hi all - thank you for your participation and your diligent work. Thanks especially to @felipekitamura who pointed out annotation issues [here](https://admin.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664) and for everyone involved in that discussion.\n\nThe annotations for one training and one public test file have been updated - that would be `c68fe75ea` in test and `e79de561c` in train. Annotations were re-checked for all files and updated if necessary. No updates were required in the private test set.\n\nWe will be re-running the non-sample kernels that were submitted, but this may take some time, depending on the amount of processing time required (we usually re-run each kernel several times to ensure proper functioning). In the meantime, any new submissions you make will reflect the updated data.\n\nThanks again!", "1250326": "What are we supposed to do with labels which still seem to be incorrectly labelled in the training images, would it be useful to report their coordinates somewhere? \n\nFor example, I could see a mismatch between my model predictions and training labels in aaa6a05cc.tiff and after inspection, they look like some glomeruli missing in the input labels (or maybe it's my wishful thinking πŸ™‚). I've just inspected a few regions in a single image, there are probably more: \n\n~[9260, 3743] - approximate center of the region\n![error1](https://i.ibb.co/tM1bVHD/image.png)\n\n~[5300, 10258], here inset images on the right are original labels (top) and model prediction (bottom)\n![error2](https://i.ibb.co/TkrHmbQ/image.png)\n\nWould be nice to prevent these missing labels at least in the private dataset. This can be probably done by double-checking all predicted glomeruli regions using several top models and inspect regions mismatching with current private set labels, fixing incorrect or missing ones after consulting with medical experts.", "1104503": "@philculliton @juyingnan \n\nplease refer to the evidence and findings at:\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/201816\n\nthe annotations following train images are shifted\n54f2eec69\n095bf7a1f\ne79de561c (still shifted after update)\n\nmaybe the public and private test dataset suffer the same issues\ne.g c68fe75ea and afa5e8098\n\nThank you very much.", "1161827": "Does anyone know when the new train/public test datasets will be released?", "1085186": "Thank you!\n\n`pointed out annotation issues here and for everyone involved in that discussion.`\nI think you have attached a wrong link to **here**.", "1150708": "Is seems the annotation issues link https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/198116 is protected. \n\nIt seems the correct link is the following: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/197664. ", "1086286": "Nice, I just found many wrong annotations 20 minutes ago. ", "1085257": "I am also unable to access that link.", "1108812": "" } }