![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6197543%2F9b3efffbbe418bfb299d90f66b0d79de%2FScreenshot_4.jpg?generation=1686689058146565&alt=media\")
Hello everyone. After studying the data from datasets 1 and 2, I noticed that the data is annotated in completely different ways. In some instances, the annotations follow the lines, while in other images, they deviate from them. I would like to inquire with the competition organizers about how the test data was annotated or selected. Are these strictly annotated data? Or is the annotation of a similar level? Th u
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6197543%2F47596f0b79ed5d8a8120969ee1411426%2FScreenshot_3.jpg?generation=1686689082173218&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6197543%2Fb2d5de8f842635d33d280d7405104789%2FScreenshot_2.jpg?generation=1686689091526052&alt=media\")
Hello everyone. After studying the data from datasets 1 and 2, I noticed that the data is annotated in completely different ways. In some instances, the annotations follow the lines, while in other images, they deviate from them. I would like to inquire with the competition organizers about how the test data was annotated or selected. Are these strictly annotated data? Or is the annotation of a similar level? Th u
\n
\n
great finding, so probably it is one of the reasons why dilation boost both local CV and public lb score https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901
", "rawMarkdown": "great finding, so probably it is one of the reasons why dilation boost both local CV and public lb score https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901", "votes": 5, "replies": [ { "id": 2301489, "postDate": "2023-06-14T01:43:30.783Z", "content": "So it could boost your LB, high probability that it harms private lb
", "rawMarkdown": "So it could boost your LB, high probability that it harms private lb", "replies": [ { "id": 2301577, "postDate": "2023-06-14T03:29:07.080Z", "content": "But public test set is also from dataset1, which means smaller contours. Why it benefits from dilation so much?
", "rawMarkdown": "But public test set is also from dataset1, which means smaller contours. Why it benefits from dilation so much?", "votes": 3 }, { "id": 2301606, "postDate": "2023-06-14T04:06:37.197Z", "content": "dataset1 is split into public test and private test, I think it means they are sharing the same labeling rules.
", "rawMarkdown": "dataset1 is split into public test and private test, I think it means they are sharing the same labeling rules." }, { "id": 2301668, "postDate": "2023-06-14T04:59:27.790Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2301502, "postDate": "2023-06-14T01:58:41.307Z", "content": "We should be sure by experimentation, but that may not be relevant to the diffrence of dataset.
\nThe reason is that dataset1 seems to fit the actual contours better, while dataset2 seems to create masks those are rougher than the actual contours.
\nIf dataset2 was used for the test data, then the hypothesis might fit, but in reality dataset1 is used.
If your post-processing improves the score in all CVs (especially in dataset1) and in the test data, then optimization to loss would not mean optimization to metrics(mAP).
\nBy the way, your findings are really interesting and very surprising for me. Thanks for sharing!
", "rawMarkdown": "We should be sure by experimentation, but that may not be relevant to the diffrence of dataset.\n\nThe reason is that dataset1 seems to fit the actual contours better, while dataset2 seems to create masks those are rougher than the actual contours.\nIf dataset2 was used for the test data, then the hypothesis might fit, but in reality dataset1 is used.\n\nIf your post-processing improves the score in all CVs (especially in dataset1) and in the test data, then optimization to loss would not mean optimization to metrics(mAP).\n\nBy the way, your findings are really interesting and very surprising for me. Thanks for sharing!" } ] }, { "id": 2304251, "postDate": "2023-06-15T21:03:59.187Z", "content": "Hello,
\nThank you for your question regarding the annotations in the dataset. First, I would like to note that generation of manual annotations is a time consuming and expensive process. Some noise in the labels is to be expected with any such dataset. In this particular dataset, the borders of microvasculature structures are not always clearly distinguishable given the tissue thickness and resolution of the images. For example, when annotating peritubular capillaries, the endothelial cell membrane may be indistinguishable from the tubular epithelial membrane, leading to some overlap of annotation borders with tubular epithelium borders. One other thing to note is that often the borders of the annotations themselves can obscure important visual information used to distinguish the vessels from surrounding structures. It is helpful to have a side by side comparison of both annotated and un-annotated images to distinguish vessels.
\nBest,
\nThanks for reply!
\nI also respect the great efforts of the annotators, thank you all!
The only thing that concerns me is that the annotation trends are very different even in dataset1. Perhaps this is unavoidable since multiple annotators are working on it.
\nI just want to be clear about the distribution of test data.
\nAre separate annotators responsible for train, public and private testing respectively?
\nOr is each annotator randomly distributed to each dataset?
\nFor example, is annotator 1 responsible for both the train and the public and private test datasets?
If the unknown annotator is only included in the private test, then it would be much more difficult to create a robust model. The post-processing such as dilation and erosion can cause huge shake-up.
\nWe look forward to your reply. Thanks!
", "rawMarkdown": "Thanks for reply!\nI also respect the great efforts of the annotators, thank you all!\n\nThe only thing that concerns me is that the annotation trends are very different even in dataset1. Perhaps this is unavoidable since multiple annotators are working on it.\n\nI just want to be clear about the distribution of test data.\nAre separate annotators responsible for train, public and private testing respectively?\nOr is each annotator randomly distributed to each dataset?\nFor example, is annotator 1 responsible for both the train and the public and private test datasets?\n\nIf the unknown annotator is only included in the private test, then it would be much more difficult to create a robust model. The post-processing such as dilation and erosion can cause huge shake-up.\n\nWe look forward to your reply. Thanks!", "votes": 2, "replies": [ { "id": 2305579, "postDate": "2023-06-16T19:23:42.740Z", "content": "YYama,
\nThe initial annotations were completed by multiple annotators, however all data was validated by the same expert. Thus, differences in annotation trends are minimal and have gone through the same validation process for the training, private and public test sets. I hope this is helpful.
\nBest,
\nKate Gustilo
Thanks for the reply!
\nSo there is no bias in the annotation trend from one dataset to another.
\nVery good news!
\"The competition data comprises tiles extracted from five Whole Slide Images (WSI) split into two datasets. Tiles from Dataset 1 have annotations that have been expert reviewed. Dataset 2 comprises the remaining tiles from these same WSIs and contain sparse annotations that have not been expert reviewed.\"
\nhttps://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/data
This is how the dataset description states it. In other words, the annotations in dataset1 are of good accuracy and the annotations in dataset2 are of slightly poorer quality. This information may have an impact on the validation method.
\nEDIT:
\nAccording to the description, only data from dataset1 is used for the test data. This is very important information.
Yes, but i send img from data_1. For example this image from data_1 too.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6197543%2Fa8fc9e565fae02cb36b7107032eb9aa8%2F1.jpg?generation=1686761343902051&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6197543%2Fe02e1ca0b30bb65d400392732bcf4b78%2FScreenshot_2.jpg?generation=1686761353747613&alt=media\")
Ah, I see. So both were from DATASET1. Thank you for sharing!
\nThen I can only assume that they were annotated by separate annotators.
\nSome of them have gone beyond the extravascular membrane and some of them have even included tubular epithelial cells.
I would like the host to disclose information about the annotator too. Are the test sets made by more than one annotator, and is there any bias between public and private annotators?
", "rawMarkdown": "Ah, I see. So both were from DATASET1. Thank you for sharing!\n\nThen I can only assume that they were annotated by separate annotators.\nSome of them have gone beyond the extravascular membrane and some of them have even included tubular epithelial cells.\n\nI would like the host to disclose information about the annotator too. Are the test sets made by more than one annotator, and is there any bias between public and private annotators?", "votes": 2 } ] } ] } ], "comments": [ { "id": 2301416, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-06-13T22:39:24.207000", "content": "great finding, so probably it is one of the reasons why dilation boost both local CV and public lb score https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901
", "votes": 5, "replies": [ { "id": 2301489, "author_name": "Phat Tran", "author_url": "", "post_date": "2023-06-14T01:43:30.783000", "content": "So it could boost your LB, high probability that it harms private lb
", "votes": 0, "replies": [ { "id": 2301577, "author_name": "Leon", "author_url": "", "post_date": "2023-06-14T03:29:07.080000", "content": "But public test set is also from dataset1, which means smaller contours. Why it benefits from dilation so much?
", "votes": 3, "replies": [] }, { "id": 2301606, "author_name": "Yi Wu", "author_url": "", "post_date": "2023-06-14T04:06:37.197000", "content": "dataset1 is split into public test and private test, I think it means they are sharing the same labeling rules.
", "votes": 0, "replies": [] }, { "id": 2301668, "author_name": "", "author_url": "", "post_date": "2023-06-14T04:59:27.790000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2301502, "author_name": "YYama", "author_url": "", "post_date": "2023-06-14T01:58:41.307000", "content": "We should be sure by experimentation, but that may not be relevant to the diffrence of dataset.
\nThe reason is that dataset1 seems to fit the actual contours better, while dataset2 seems to create masks those are rougher than the actual contours.
\nIf dataset2 was used for the test data, then the hypothesis might fit, but in reality dataset1 is used.
If your post-processing improves the score in all CVs (especially in dataset1) and in the test data, then optimization to loss would not mean optimization to metrics(mAP).
\nBy the way, your findings are really interesting and very surprising for me. Thanks for sharing!
", "votes": 0, "replies": [] } ] }, { "id": 2304251, "author_name": "Katherine Gustilo", "author_url": "", "post_date": "2023-06-15T21:03:59.187000", "content": "Hello,
\nThank you for your question regarding the annotations in the dataset. First, I would like to note that generation of manual annotations is a time consuming and expensive process. Some noise in the labels is to be expected with any such dataset. In this particular dataset, the borders of microvasculature structures are not always clearly distinguishable given the tissue thickness and resolution of the images. For example, when annotating peritubular capillaries, the endothelial cell membrane may be indistinguishable from the tubular epithelial membrane, leading to some overlap of annotation borders with tubular epithelium borders. One other thing to note is that often the borders of the annotations themselves can obscure important visual information used to distinguish the vessels from surrounding structures. It is helpful to have a side by side comparison of both annotated and un-annotated images to distinguish vessels.
\nBest,
\nThanks for reply!
\nI also respect the great efforts of the annotators, thank you all!
The only thing that concerns me is that the annotation trends are very different even in dataset1. Perhaps this is unavoidable since multiple annotators are working on it.
\nI just want to be clear about the distribution of test data.
\nAre separate annotators responsible for train, public and private testing respectively?
\nOr is each annotator randomly distributed to each dataset?
\nFor example, is annotator 1 responsible for both the train and the public and private test datasets?
If the unknown annotator is only included in the private test, then it would be much more difficult to create a robust model. The post-processing such as dilation and erosion can cause huge shake-up.
\nWe look forward to your reply. Thanks!
", "votes": 2, "replies": [ { "id": 2305579, "author_name": "Katherine Gustilo", "author_url": "", "post_date": "2023-06-16T19:23:42.740000", "content": "YYama,
\nThe initial annotations were completed by multiple annotators, however all data was validated by the same expert. Thus, differences in annotation trends are minimal and have gone through the same validation process for the training, private and public test sets. I hope this is helpful.
\nBest,
\nKate Gustilo
Thanks for the reply!
\nSo there is no bias in the annotation trend from one dataset to another.
\nVery good news!
\"The competition data comprises tiles extracted from five Whole Slide Images (WSI) split into two datasets. Tiles from Dataset 1 have annotations that have been expert reviewed. Dataset 2 comprises the remaining tiles from these same WSIs and contain sparse annotations that have not been expert reviewed.\"
\nhttps://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/data
This is how the dataset description states it. In other words, the annotations in dataset1 are of good accuracy and the annotations in dataset2 are of slightly poorer quality. This information may have an impact on the validation method.
\nEDIT:
\nAccording to the description, only data from dataset1 is used for the test data. This is very important information.
Yes, but i send img from data_1. For example this image from data_1 too.
\n
Ah, I see. So both were from DATASET1. Thank you for sharing!
\nThen I can only assume that they were annotated by separate annotators.
\nSome of them have gone beyond the extravascular membrane and some of them have even included tubular epithelial cells.
I would like the host to disclose information about the annotator too. Are the test sets made by more than one annotator, and is there any bias between public and private annotators?
", "votes": 2, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2301372": "Hello everyone. After studying the data from datasets 1 and 2, I noticed that the data is annotated in completely different ways. In some instances, the annotations follow the lines, while in other images, they deviate from them. I would like to inquire with the competition organizers about how the test data was annotated or selected. Are these strictly annotated data? Or is the annotation of a similar level? Th u \n\n\n\n\n", "2301416": "great finding, so probably it is one of the reasons why dilation boost both local CV and public lb score https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/416901", "2304251": "Hello,\n\nThank you for your question regarding the annotations in the dataset. First, I would like to note that generation of manual annotations is a time consuming and expensive process. Some noise in the labels is to be expected with any such dataset. In this particular dataset, the borders of microvasculature structures are not always clearly distinguishable given the tissue thickness and resolution of the images. For example, when annotating peritubular capillaries, the endothelial cell membrane may be indistinguishable from the tubular epithelial membrane, leading to some overlap of annotation borders with tubular epithelium borders. One other thing to note is that often the borders of the annotations themselves can obscure important visual information used to distinguish the vessels from surrounding structures. It is helpful to have a side by side comparison of both annotated and un-annotated images to distinguish vessels. \n\nBest,\n - Kate Gustilo, Research Analyst and Anatomist", "2301431": "\"The competition data comprises tiles extracted from five Whole Slide Images (WSI) split into two datasets. Tiles from Dataset 1 have annotations that have been expert reviewed. Dataset 2 comprises the remaining tiles from these same WSIs and contain sparse annotations that have not been expert reviewed.\"\nhttps://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/data\n\nThis is how the dataset description states it. In other words, the annotations in dataset1 are of good accuracy and the annotations in dataset2 are of slightly poorer quality. This information may have an impact on the validation method.\n\nEDIT:\nAccording to the description, only data from dataset1 is used for the test data. This is very important information." } }