As discussed in this post, the rule has been changed to allow hand labeling public test data.
\nFrom A. COMPETITION-SPECIFIC RULES in the rule
\n\n\nSubmissions may use or incorporate information from hand labeling or human prediction of the validation dataset or public test data records.
\n
However, I am strongly opposed to this rule change.
\nHand labeling should be prohibited.
If the hand labeling is allowed, public leaderboard doesn't work anymore.
\nIn addition, if some teams get good LB score by hand labeling, then they can add it to train data. I think this means hand labeling skill is important to increase training data and to get good final score of private test.
So now, unfortunately, this competition got an annotation skill competition. Unless the rule change again, your next action is to hand label public data until you get 1.0 in the LB score.
", "messages": [ { "id": 1252075, "postDate": "2021-03-25T12:09:01.180Z", "content": "As discussed in this post, the rule has been changed to allow hand labeling public test data.
\nFrom A. COMPETITION-SPECIFIC RULES in the rule
\n\n\nSubmissions may use or incorporate information from hand labeling or human prediction of the validation dataset or public test data records.
\n
However, I am strongly opposed to this rule change.
\nHand labeling should be prohibited.
If the hand labeling is allowed, public leaderboard doesn't work anymore.
\nIn addition, if some teams get good LB score by hand labeling, then they can add it to train data. I think this means hand labeling skill is important to increase training data and to get good final score of private test.
So now, unfortunately, this competition got an annotation skill competition. Unless the rule change again, your next action is to hand label public data until you get 1.0 in the LB score.
", "rawMarkdown": "As discussed in [this post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616), the rule has been changed to allow hand labeling public test data.\n\n**From A. COMPETITION-SPECIFIC RULES in the [rule](https://www.kaggle.com/c/hubmap-kidney-segmentation/rules)**\n> Submissions may use or incorporate information from hand labeling or human prediction of the validation dataset or public test data records.\n\n**However, I am strongly opposed to this rule change.If the hand labeling of the public test data is allowed, we will give up this competition. Makes no sense at all.
", "rawMarkdown": "If the hand labeling of the public test data is allowed, we will give up this competition. Makes no sense at all.", "votes": 7 }, { "id": 1252142, "postDate": "2021-03-25T13:06:27.267Z", "content": "I agree with you so much. Public LB is not for evaluating annotation skill.
", "rawMarkdown": "I agree with you so much. Public LB is not for evaluating annotation skill.", "votes": 8 }, { "id": 1252293, "postDate": "2021-03-25T15:05:49.067Z", "content": "Although I don't like the rule change, I do not entirely agree with you :
\n\n\nSo now, unfortunately, this competition got an annotation skill competition.
\n
Labelling the public LB is the same as labelling external data on this regard. So it was already (sort of) an annotation competition.
\n\n\nPublic leaderboard doesn't work anymore.
\n
Public LB is often unrealiable, which is why you always need to have a reliable CV.
", "rawMarkdown": "Although I don't like the rule change, I do not entirely agree with you : \n\n> So now, unfortunately, this competition got an annotation skill competition.\n\nLabelling the public LB is the same as labelling external data on this regard. So it was already (sort of) an annotation competition.\n\n> Public leaderboard doesn't work anymore.\n\nPublic LB is often unrealiable, which is why you always need to have a reliable CV.", "votes": 1, "replies": [ { "id": 1252316, "postDate": "2021-03-25T15:25:56.750Z", "content": "@theoviel
\nIf you annotate external data, you don't know how well it was annotated.
\nHowever, in this competition, the quality of the annotations can be checked on the Public leaderboard. Moreover, the quality of the annotations based on LB can be of the quality required by this competition. I think this is also different from using external data.
\n\nPublic LB is often unrealiable,
\n
That's true. But, if you get additional 5 test images validated by LB added to train data, your CV will get more reliable.
", "rawMarkdown": "@theoviel \nIf you annotate external data, you don't know how well it was annotated.\nHowever, in this competition, the quality of the annotations can be checked on the Public leaderboard. Moreover, the quality of the annotations based on LB can be of the quality required by this competition. I think this is also different from using external data.\n\n> Public LB is often unrealiable,\n\nThat's true. But, if you get additional 5 test images validated by LB added to train data, your CV will get more reliable.", "votes": 3 }, { "id": 1252328, "postDate": "2021-03-25T15:33:37.810Z", "content": "That's a good point indeed, I didn't consider that you can get feedback on your annotations.
", "rawMarkdown": "That's a good point indeed, I didn't consider that you can get feedback on your annotations.\n" } ] }, { "id": 1252088, "postDate": "2021-03-25T12:18:43.183Z", "content": "I think prohibition of hand-labeling public test dataset is hard to conducted, for you cannot judge whether one’s model was trained by that. IMO the proper way to get around is giving us new public test dataset and DO NOT make it visible to us.
", "rawMarkdown": "I think prohibition of hand-labeling public test dataset is hard to conducted, for you cannot judge whether one’s model was trained by that. IMO the proper way to get around is giving us new public test dataset and DO NOT make it visible to us.", "votes": 1 }, { "id": 1255447, "postDate": "2021-03-28T20:17:23.173Z", "content": "I think the rules clearly state that we should share datasets if we use external data (section B.7.C). The addition of A.3 does not require that. Hand labelling is just a faster way to overfit the public test set and then focus on the development of a robust model. And given that there are still errors present in the public test set (~2-6%?) and the competition is a code competition - it sounds like a fair change to me, LB was of a limited value anyway. This will ensure that we focus more on the generalization of our models faster instead of spending time just on the overfitting to the public test set. Even the best hand-labelled submissions with the LB 99% will be useless on the private test set anyway. So it became kind of two-step competition right now: 1) probe the public test set and develop a good model with that knowledge, and 2) develop a model that will generalize well, and pray that it will perform well on the private test set and that the quality of labels of the private test set is better than in the public train/test set.
", "rawMarkdown": "I think the rules clearly state that we should share datasets if we use external data (section B.7.C). The addition of A.3 does not require that. Hand labelling is just a faster way to overfit the public test set and then focus on the development of a robust model. And given that there are still errors present in the public test set (~2-6%?) and the competition is a code competition - it sounds like a fair change to me, LB was of a limited value anyway. This will ensure that we focus more on the generalization of our models faster instead of spending time just on the overfitting to the public test set. Even the best hand-labelled submissions with the LB 99% will be useless on the private test set anyway. So it became kind of two-step competition right now: 1) probe the public test set and develop a good model with that knowledge, and 2) develop a model that will generalize well, and pray that it will perform well on the private test set and that the quality of labels of the private test set is better than in the public train/test set." }, { "id": 1254635, "postDate": "2021-03-27T22:22:03.780Z", "content": "now most the people on the first places have done hand labeling, but none of them are sharing he DS that they are using, should then by disqualified ?
\nSome of us that been working very hard on this competition are being left behind because we don't know how to hand label, and now one wants to share how and what.
", "rawMarkdown": "now most the people on the first places have done hand labeling, but none of them are sharing he DS that they are using, should then by disqualified ?\n\nSome of us that been working very hard on this competition are being left behind because we don't know how to hand label, and now one wants to share how and what.\n\n" }, { "id": 1253861, "postDate": "2021-03-27T05:31:45.320Z", "content": "Added this to the other discussion post but putting here too.
\nWould like to add this from the RANZCR competitions experience and believe competition rules here should be amended in a similar way but clarifying the test data set since there are now old test that could be considered external data -
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644
\n\"There have been some questions from competitors regarding hand-labelled annotations. To clarify the rules as per section A2:
Publicly, freely available external data is permitted. Entrants may re-annotate images in the training set, however Entrants will (i) ensure the re-annotated data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the re-annotated data for the participants to the official competition forum prior to the Entry Deadline. Entrants may not hand-label predictions in the test data set, including having human observers rate and evaluate the test data set.\"\nThere is no way to know if the private test data also has \"dark glomeruli\" as discovered in the public test data. If so this confers an advantage to those that have these annotations.
", "rawMarkdown": "Added this to the other discussion post but putting here too.\n \nWould like to add this from the RANZCR competitions experience and believe competition rules here should be amended in a similar way but clarifying the test data set since there are now old test that could be considered external data -\n \nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644 \n\"There have been some questions from competitors regarding hand-labelled annotations. To clarify the rules as per section A2:\n\n Publicly, freely available external data is permitted. Entrants may re-annotate images in the training set, however Entrants will (i) ensure the re-annotated data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the re-annotated data for the participants to the official competition forum prior to the Entry Deadline. Entrants may not hand-label predictions in the test data set, including having human observers rate and evaluate the test data set.\"\n\nThere is no way to know if the private test data also has \"dark glomeruli\" as discovered in the public test data. If so this confers an advantage to those that have these annotations. " }, { "id": 1252786, "postDate": "2021-03-26T03:45:03.140Z", "content": "How do you go about doing hand labeling? can you one explain, thanks
", "rawMarkdown": "How do you go about doing hand labeling? can you one explain, thanks" }, { "id": 1252086, "postDate": "2021-03-25T12:18:19.990Z", "content": " it is quite difficult to distinguish it from pseudo labeling, I guess.
\nPublic test images shouldn’t be uncovered.
EDIT: public test hand labeling seems to be permitted now.
", "rawMarkdown": "~~Although public test hand labeling is prohibited as Addison stated,~~ it is quite difficult to distinguish it from pseudo labeling, I guess.\nPublic test images shouldn’t be uncovered.\n\nEDIT: public test hand labeling seems to be permitted now.", "replies": [ { "id": 1252107, "postDate": "2021-03-25T12:30:12.543Z", "content": "@drtausamaru
\nThank you for your comment.
\n\nAlthough public test hand labeling is prohibited as Addison stated
\n
Where can I find this??
", "rawMarkdown": "@drtausamaru \nThank you for your comment.\n> Although public test hand labeling is prohibited as Addison stated\n\nWhere can I find this??" }, { "id": 1252110, "postDate": "2021-03-25T12:33:34.360Z", "content": "Here I guess.
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1249808
\n\"as long as it isn't the test dataset\"
You have to read his comments further. Now it is allowed to hand label the test data! The rule is also changed accordingly. I am now really frustrated after dealing month with broken data, now this.
", "rawMarkdown": "You have to read his comments further. Now it is allowed to hand label the test data! The rule is also changed accordingly. I am now really frustrated after dealing month with broken data, now this.", "votes": 5 }, { "id": 1252114, "postDate": "2021-03-25T12:37:54.830Z", "content": "Thanks, but OMG…
", "rawMarkdown": "Thanks, but OMG...", "votes": 1 } ] }, { "id": 1252238, "postDate": "2021-03-25T14:19:57.540Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1252096, "author_name": "tugstugi", "author_url": "", "post_date": "2021-03-25T12:22:50.893000", "content": "If the hand labeling of the public test data is allowed, we will give up this competition. Makes no sense at all.
", "votes": 7, "replies": [] }, { "id": 1252142, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-25T13:06:27.267000", "content": "I agree with you so much. Public LB is not for evaluating annotation skill.
", "votes": 8, "replies": [] }, { "id": 1252293, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-03-25T15:05:49.067000", "content": "Although I don't like the rule change, I do not entirely agree with you :
\n\n\nSo now, unfortunately, this competition got an annotation skill competition.
\n
Labelling the public LB is the same as labelling external data on this regard. So it was already (sort of) an annotation competition.
\n\n\nPublic leaderboard doesn't work anymore.
\n
Public LB is often unrealiable, which is why you always need to have a reliable CV.
", "votes": 1, "replies": [ { "id": 1252316, "author_name": "Inoichan", "author_url": "", "post_date": "2021-03-25T15:25:56.750000", "content": "@theoviel
\nIf you annotate external data, you don't know how well it was annotated.
\nHowever, in this competition, the quality of the annotations can be checked on the Public leaderboard. Moreover, the quality of the annotations based on LB can be of the quality required by this competition. I think this is also different from using external data.
\n\nPublic LB is often unrealiable,
\n
That's true. But, if you get additional 5 test images validated by LB added to train data, your CV will get more reliable.
", "votes": 3, "replies": [] }, { "id": 1252328, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-03-25T15:33:37.810000", "content": "That's a good point indeed, I didn't consider that you can get feedback on your annotations.
", "votes": 0, "replies": [] } ] }, { "id": 1252088, "author_name": "Shihao Shao", "author_url": "", "post_date": "2021-03-25T12:18:43.183000", "content": "I think prohibition of hand-labeling public test dataset is hard to conducted, for you cannot judge whether one’s model was trained by that. IMO the proper way to get around is giving us new public test dataset and DO NOT make it visible to us.
", "votes": 1, "replies": [] }, { "id": 1255447, "author_name": "Gena", "author_url": "", "post_date": "2021-03-28T20:17:23.173000", "content": "I think the rules clearly state that we should share datasets if we use external data (section B.7.C). The addition of A.3 does not require that. Hand labelling is just a faster way to overfit the public test set and then focus on the development of a robust model. And given that there are still errors present in the public test set (~2-6%?) and the competition is a code competition - it sounds like a fair change to me, LB was of a limited value anyway. This will ensure that we focus more on the generalization of our models faster instead of spending time just on the overfitting to the public test set. Even the best hand-labelled submissions with the LB 99% will be useless on the private test set anyway. So it became kind of two-step competition right now: 1) probe the public test set and develop a good model with that knowledge, and 2) develop a model that will generalize well, and pray that it will perform well on the private test set and that the quality of labels of the private test set is better than in the public train/test set.
", "votes": 0, "replies": [] }, { "id": 1254635, "author_name": "TheStoneMX", "author_url": "", "post_date": "2021-03-27T22:22:03.780000", "content": "now most the people on the first places have done hand labeling, but none of them are sharing he DS that they are using, should then by disqualified ?
\nSome of us that been working very hard on this competition are being left behind because we don't know how to hand label, and now one wants to share how and what.
", "votes": 0, "replies": [] }, { "id": 1253861, "author_name": "something4kag", "author_url": "", "post_date": "2021-03-27T05:31:45.320000", "content": "Added this to the other discussion post but putting here too.
\nWould like to add this from the RANZCR competitions experience and believe competition rules here should be amended in a similar way but clarifying the test data set since there are now old test that could be considered external data -
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222644
\n\"There have been some questions from competitors regarding hand-labelled annotations. To clarify the rules as per section A2:
Publicly, freely available external data is permitted. Entrants may re-annotate images in the training set, however Entrants will (i) ensure the re-annotated data is available to use by all participants of the competition for purposes of the competition at no cost to the other participants and (ii) post such access to the re-annotated data for the participants to the official competition forum prior to the Entry Deadline. Entrants may not hand-label predictions in the test data set, including having human observers rate and evaluate the test data set.\"\nThere is no way to know if the private test data also has \"dark glomeruli\" as discovered in the public test data. If so this confers an advantage to those that have these annotations.
", "votes": 0, "replies": [] }, { "id": 1252786, "author_name": "TheStoneMX", "author_url": "", "post_date": "2021-03-26T03:45:03.140000", "content": "How do you go about doing hand labeling? can you one explain, thanks
", "votes": 0, "replies": [] }, { "id": 1252086, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-25T12:18:19.990000", "content": " it is quite difficult to distinguish it from pseudo labeling, I guess.
\nPublic test images shouldn’t be uncovered.
EDIT: public test hand labeling seems to be permitted now.
", "votes": 0, "replies": [ { "id": 1252107, "author_name": "Inoichan", "author_url": "", "post_date": "2021-03-25T12:30:12.543000", "content": "@drtausamaru
\nThank you for your comment.
\n\nAlthough public test hand labeling is prohibited as Addison stated
\n
Where can I find this??
", "votes": 0, "replies": [] }, { "id": 1252110, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-25T12:33:34.360000", "content": "Here I guess.
\nhttps://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1249808
\n\"as long as it isn't the test dataset\"
You have to read his comments further. Now it is allowed to hand label the test data! The rule is also changed accordingly. I am now really frustrated after dealing month with broken data, now this.
", "votes": 5, "replies": [] }, { "id": 1252114, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-03-25T12:37:54.830000", "content": "Thanks, but OMG…
", "votes": 1, "replies": [] } ] }, { "id": 1252238, "author_name": "", "author_url": "", "post_date": "2021-03-25T14:19:57.540000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1252075": "As discussed in [this post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616), the rule has been changed to allow hand labeling public test data.\n\n**From A. COMPETITION-SPECIFIC RULES in the [rule](https://www.kaggle.com/c/hubmap-kidney-segmentation/rules)**\n> Submissions may use or incorporate information from hand labeling or human prediction of the validation dataset or public test data records.\n\n**However, I am strongly opposed to this rule change.