{ "id": 413000, "title": "Data: Legal Issue", "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/413000", "author_name": "", "post_date": "2023-05-26T09:44:13.128869800Z", "votes": 6, "comment_count": 16, "views": 0, "content": "

Hey!

\n

Is annotating data from the Dataset3 by the external human resources (doctors) legal?
\nIf not, how can we be sure that someone will not pay for additional annotations, gaining a considerable advantage over the rest participants?
\nHow to interpret the following rule?
\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.

", "messages": [ { "id": "2274847", "postDate": "05/26/2023 09:44:13", "content": "

Hey!

\n

Is annotating data from the Dataset3 by the external human resources (doctors) legal?
\nIf not, how can we be sure that someone will not pay for additional annotations, gaining a considerable advantage over the rest participants?
\nHow to interpret the following rule?
\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.

", "rawMarkdown": "Hey!\n\nIs annotating data from the Dataset3 by the `external human resources` (doctors) legal?\nIf not, how can we be sure that someone will not pay for additional annotations, gaining a considerable advantage over the rest participants?\nHow to interpret the following rule?\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.", "votes": null }, { "id": "2276214", "postDate": "05/26/2023 15:23:44", "content": "

Hi there,

\n

Additionally in Rule B.5 is the following:

\n
\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.

\n
\n

External hand annotation is not permitted in this competition.

", "rawMarkdown": "Hi there,\n\nAdditionally in Rule B.5 is the following:\n\n>Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n\nExternal hand annotation is not permitted in this competition.", "votes": null }, { "id": "2276826", "postDate": "05/27/2023 08:30:07", "content": "

I don't know about that.
\nWhen I read \"Submissions\", in my mind it means the final actual predictions.
\nIt is reinforced when it talks about incorporating humans in labeling the validation and test data.

\n

It says nothing about what the model can be trained on.
\nOf course labeling the validation and test data and then training the model on that data is not covered here but it's common sense.

\n

Do I misunderstood something?

", "rawMarkdown": "I don't know about that.\nWhen I read \"Submissions\", in my mind it means the final actual predictions.\nIt is reinforced when it talks about incorporating humans in labeling the validation and test data.\n\nIt says nothing about what the model can be trained on.\nOf course labeling the validation and test data and then training the model on that data is not covered here but it's common sense.\n\nDo I misunderstood something?", "votes": null }, { "id": "2278503", "postDate": "05/28/2023 18:29:02", "content": "

You \"can\", in fact, externally annotate dataset 3, violating that part of the competition rules, and of course, this will very likely boost your score both in the public and private LB. However, at the end of the competition, as a winner, you're going to have to open source your code and model and \"reveal\" how you've reached the results that you've got. And that's where your submission will be invalidated when Kaggle staff and the competition hosts discover your breach of the competition rules.
\nSo, no, I don't think anyone will want to do that.

", "rawMarkdown": "You \"can\", in fact, externally annotate dataset 3, violating that part of the competition rules, and of course, this will very likely boost your score both in the public and private LB. However, at the end of the competition, as a winner, you're going to have to open source your code and model and \"reveal\" how you've reached the results that you've got. And that's where your submission will be invalidated when Kaggle staff and the competition hosts discover your breach of the competition rules.\nSo, no, I don't think anyone will want to do that.", "votes": null }, { "id": "2278527", "postDate": "05/28/2023 18:59:59", "content": "

@momakmd I have doubts about that.

\n
\n

Open Source: You hereby license and will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code. To the extent your Submission makes use of generally commercially available software not owned by you that you used to generate your submission, but that can be procured by the Competition Sponsor without undue expense, you do not grant the license in the preceding sentence to that software.

\n
\n

It is said that you will have to open-source the code to reproduce the submission, not the code to train your model.
\n@addisonhoward can You please tighten the rules? In the present state of the rules, it is not possible to reveal if a model was trained with a human-labeled Dataset3

", "rawMarkdown": "momakmd I have doubts about that.\n>Open Source: You hereby license and will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code. To the extent your Submission makes use of generally commercially available software not owned by you that you used to generate your submission, but that can be procured by the Competition Sponsor without undue expense, you do not grant the license in the preceding sentence to that software.\n\nIt is said that you will have to open-source the code to reproduce the submission, not the code to train your model.\n@addisonhoward can You please tighten the rules? In the present state of the rules, it is not possible to reveal if a model was trained with a human-labeled Dataset3", "votes": null }, { "id": "2279805", "postDate": "05/29/2023 16:47:40", "content": "

Oh sorry, I've missed the part that the question is about dataset 3.

", "rawMarkdown": "Oh sorry, I've missed the part that the question is about dataset 3.", "votes": null }, { "id": "2281069", "postDate": "05/30/2023 14:55:58", "content": "

Hi there. Kaggle is not a legal party to the competition rules, and therefore we cannot provide legal advice or interpretation. However I would also like to direct you to Rule B.10, which includes:

\n
\n

Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.

\n

A disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.

\n
", "rawMarkdown": "Hi there. Kaggle is not a legal party to the competition rules, and therefore we cannot provide legal advice or interpretation. However I would also like to direct you to Rule B.10, which includes:\n\n>Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.\n\n>A disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.", "votes": null }, { "id": "2281300", "postDate": "05/30/2023 18:42:55", "content": "

@yashvrdnjain @katherinegustilo can you please clarify this?

", "rawMarkdown": "yashvrdnjain @katherinegustilo can you please clarify this?", "votes": null }, { "id": "2281423", "postDate": "05/30/2023 21:29:40", "content": "

Hello,
\nHuman annotation of any data is not allowed. On competition end, the winning teams will be asked to submit their training and inference code, trained models, description of methods for reproducing the solution, and all training data used (including external datasets). All these will be used to reproduce and validate the solutions against the leaderboard scores. If we suspect human annotations were used (or any other violation of competition rules), your submission will not be eligible for the prize. Other penalties may follow as per Kaggle rules.

", "rawMarkdown": "Hello,\nHuman annotation of any data is not allowed. On competition end, the winning teams will be asked to submit their training and inference code, trained models, description of methods for reproducing the solution, and all training data used (including external datasets). All these will be used to reproduce and validate the solutions against the leaderboard scores. If we suspect human annotations were used (or any other violation of competition rules), your submission will not be eligible for the prize. Other penalties may follow as per Kaggle rules.", "votes": null }, { "id": "2281765", "postDate": "05/31/2023 05:40:27", "content": "

Glad to hear that, thanks!

", "rawMarkdown": "Glad to hear that, thanks!", "votes": null }, { "id": "2310544", "postDate": "06/20/2023 12:55:03", "content": "

I just entered this competition and I am concerned about this topic.
\nMy understanding from your comment is that it is acceptable to use pseudo-labels by model predictions because they can be reproduced without human annotation, is that correct?

", "rawMarkdown": "I just entered this competition and I am concerned about this topic.\nMy understanding from your comment is that it is acceptable to use pseudo-labels by model predictions because they can be reproduced without human annotation, is that correct?", "votes": null }, { "id": "2340305", "postDate": "07/11/2023 12:05:50", "content": "

@addisonhoward but this does not state that it is forbidden to annotate the train set. Dataset 3 is not validation or test data. There is nothing in the rules that says it is forbidden to annotate the train set manually.

", "rawMarkdown": "addisonhoward but this does not state that it is forbidden to annotate the train set. Dataset 3 is not validation or test data. There is nothing in the rules that says it is forbidden to annotate the train set manually.", "votes": null }, { "id": "2340321", "postDate": "07/11/2023 12:16:48", "content": "

@yashvrdnjain is there a reason this is not in the rules?

\n

Why not change Rule B.5 (which actually allows annotation of train data) from:

\n
\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
\n to:
\n Submissions may not use or incorporate information from hand labeling or human prediction.

\n
\n

What is the reason for not allowing hand labeling? For competitors, it is better when it is not allowed since it requires a lot of manual work. But, by allowing labeling, you will most likely get a better model that is still open-source.

", "rawMarkdown": "yashvrdnjain is there a reason this is not in the rules?\n\nWhy not change Rule B.5 (which actually allows annotation of train data) from:\n>Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\nto:\n>Submissions may not use or incorporate information from hand labeling or human prediction.\n\nWhat is the reason for not allowing hand labeling? For competitors, it is better when it is not allowed since it requires a lot of manual work. But, by allowing labeling, you will most likely get a better model that is still open-source.", "votes": null }, { "id": "2340331", "postDate": "07/11/2023 12:24:55", "content": "

I believe most of the WSIs provided are part of previous HuBMAP competition including WSI 3/4 which are used to evaluate public leaderboard test data. I think it's good that hand labelling is not allowed, should prevent any leakage and unfair competition, also encouraging semi supervised solutions.

", "rawMarkdown": "I believe most of the WSIs provided are part of previous HuBMAP competition including WSI 3/4 which are used to evaluate public leaderboard test data. I think it's good that hand labelling is not allowed, should prevent any leakage and unfair competition, also encouraging semi supervised solutions.", "votes": null }, { "id": "2340632", "postDate": "07/11/2023 15:33:51", "content": "

@crodoc

\n

Hand labeling is never allowed in any of the comps, afaik. That said we select folds, isn't that a sort of hand labeling? I guess there is some blurry line

", "rawMarkdown": "crodoc \n\nHand labeling is never allowed in any of the comps, afaik. That said we select folds, isn't that a sort of hand labeling? I guess there is some blurry line", "votes": null }, { "id": "2340644", "postDate": "07/11/2023 15:41:46", "content": "

actually top solutions in last computer vision competitions used hand labeling on training data

", "rawMarkdown": "actually top solutions in last computer vision competitions used hand labeling on training data", "votes": null }, { "id": "2340891", "postDate": "07/11/2023 18:36:48", "content": "

Hand labeling the train set or external data is usually allowed. You are not allowed to label public or private test data if it is given (sometimes you get the test set and you only submit a csv file that you can generate locally).

", "rawMarkdown": "Hand labeling the train set or external data is usually allowed. You are not allowed to label public or private test data if it is given (sometimes you get the test set and you only submit a csv file that you can generate locally).", "votes": null } ], "comments": [ { "id": 2276214, "author_name": "addisonhoward", "author_url": "", "post_date": "05/26/2023 15:23:44", "content": "

Hi there,

\n

Additionally in Rule B.5 is the following:

\n
\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.

\n
\n

External hand annotation is not permitted in this competition.

", "votes": null, "replies": [ { "id": 2276826, "author_name": "nofreewill", "author_url": "", "post_date": "05/27/2023 08:30:07", "content": "

I don't know about that.
\nWhen I read \"Submissions\", in my mind it means the final actual predictions.
\nIt is reinforced when it talks about incorporating humans in labeling the validation and test data.

\n

It says nothing about what the model can be trained on.
\nOf course labeling the validation and test data and then training the model on that data is not covered here but it's common sense.

\n

Do I misunderstood something?

", "votes": null, "replies": [ { "id": 2278503, "author_name": "momakmd", "author_url": "", "post_date": "05/28/2023 18:29:02", "content": "

You \"can\", in fact, externally annotate dataset 3, violating that part of the competition rules, and of course, this will very likely boost your score both in the public and private LB. However, at the end of the competition, as a winner, you're going to have to open source your code and model and \"reveal\" how you've reached the results that you've got. And that's where your submission will be invalidated when Kaggle staff and the competition hosts discover your breach of the competition rules.
\nSo, no, I don't think anyone will want to do that.

", "votes": null, "replies": [ { "id": 2278527, "author_name": "janglinko2", "author_url": "", "post_date": "05/28/2023 18:59:59", "content": "

@momakmd I have doubts about that.

\n
\n

Open Source: You hereby license and will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code. To the extent your Submission makes use of generally commercially available software not owned by you that you used to generate your submission, but that can be procured by the Competition Sponsor without undue expense, you do not grant the license in the preceding sentence to that software.

\n
\n

It is said that you will have to open-source the code to reproduce the submission, not the code to train your model.
\n@addisonhoward can You please tighten the rules? In the present state of the rules, it is not possible to reveal if a model was trained with a human-labeled Dataset3

", "votes": null, "replies": [ { "id": 2281069, "author_name": "addisonhoward", "author_url": "", "post_date": "05/30/2023 14:55:58", "content": "

Hi there. Kaggle is not a legal party to the competition rules, and therefore we cannot provide legal advice or interpretation. However I would also like to direct you to Rule B.10, which includes:

\n
\n

Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.

\n

A disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.

\n
", "votes": null, "replies": [] } ] }, { "id": 2279805, "author_name": "nofreewill", "author_url": "", "post_date": "05/29/2023 16:47:40", "content": "

Oh sorry, I've missed the part that the question is about dataset 3.

", "votes": null, "replies": [] } ] } ] }, { "id": 2340305, "author_name": "crodoc", "author_url": "", "post_date": "07/11/2023 12:05:50", "content": "

@addisonhoward but this does not state that it is forbidden to annotate the train set. Dataset 3 is not validation or test data. There is nothing in the rules that says it is forbidden to annotate the train set manually.

", "votes": null, "replies": [] } ] }, { "id": 2281300, "author_name": "janglinko2", "author_url": "", "post_date": "05/30/2023 18:42:55", "content": "

@yashvrdnjain @katherinegustilo can you please clarify this?

", "votes": null, "replies": [] }, { "id": 2281423, "author_name": "yashvrdnjain", "author_url": "", "post_date": "05/30/2023 21:29:40", "content": "

Hello,
\nHuman annotation of any data is not allowed. On competition end, the winning teams will be asked to submit their training and inference code, trained models, description of methods for reproducing the solution, and all training data used (including external datasets). All these will be used to reproduce and validate the solutions against the leaderboard scores. If we suspect human annotations were used (or any other violation of competition rules), your submission will not be eligible for the prize. Other penalties may follow as per Kaggle rules.

", "votes": null, "replies": [ { "id": 2281765, "author_name": "janglinko2", "author_url": "", "post_date": "05/31/2023 05:40:27", "content": "

Glad to hear that, thanks!

", "votes": null, "replies": [] }, { "id": 2310544, "author_name": "mrt0933", "author_url": "", "post_date": "06/20/2023 12:55:03", "content": "

I just entered this competition and I am concerned about this topic.
\nMy understanding from your comment is that it is acceptable to use pseudo-labels by model predictions because they can be reproduced without human annotation, is that correct?

", "votes": null, "replies": [] }, { "id": 2340321, "author_name": "crodoc", "author_url": "", "post_date": "07/11/2023 12:16:48", "content": "

@yashvrdnjain is there a reason this is not in the rules?

\n

Why not change Rule B.5 (which actually allows annotation of train data) from:

\n
\n

Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
\n to:
\n Submissions may not use or incorporate information from hand labeling or human prediction.

\n
\n

What is the reason for not allowing hand labeling? For competitors, it is better when it is not allowed since it requires a lot of manual work. But, by allowing labeling, you will most likely get a better model that is still open-source.

", "votes": null, "replies": [ { "id": 2340331, "author_name": "nischaydnk", "author_url": "", "post_date": "07/11/2023 12:24:55", "content": "

I believe most of the WSIs provided are part of previous HuBMAP competition including WSI 3/4 which are used to evaluate public leaderboard test data. I think it's good that hand labelling is not allowed, should prevent any leakage and unfair competition, also encouraging semi supervised solutions.

", "votes": null, "replies": [] }, { "id": 2340632, "author_name": "kaggleqrdl", "author_url": "", "post_date": "07/11/2023 15:33:51", "content": "

@crodoc

\n

Hand labeling is never allowed in any of the comps, afaik. That said we select folds, isn't that a sort of hand labeling? I guess there is some blurry line

", "votes": null, "replies": [ { "id": 2340644, "author_name": "ptran1203", "author_url": "", "post_date": "07/11/2023 15:41:46", "content": "

actually top solutions in last computer vision competitions used hand labeling on training data

", "votes": null, "replies": [] }, { "id": 2340891, "author_name": "crodoc", "author_url": "", "post_date": "07/11/2023 18:36:48", "content": "

Hand labeling the train set or external data is usually allowed. You are not allowed to label public or private test data if it is given (sometimes you get the test set and you only submit a csv file that you can generate locally).

", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2274847": "Hey!\n\nIs annotating data from the Dataset3 by the `external human resources` (doctors) legal?\nIf not, how can we be sure that someone will not pay for additional annotations, gaining a considerable advantage over the rest participants?\nHow to interpret the following rule?\nC. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants.", "2276214": "Hi there,\n\nAdditionally in Rule B.5 is the following:\n\n>Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\n\nExternal hand annotation is not permitted in this competition.", "2276826": "I don't know about that.\nWhen I read \"Submissions\", in my mind it means the final actual predictions.\nIt is reinforced when it talks about incorporating humans in labeling the validation and test data.\n\nIt says nothing about what the model can be trained on.\nOf course labeling the validation and test data and then training the model on that data is not covered here but it's common sense.\n\nDo I misunderstood something?", "2278503": "You \"can\", in fact, externally annotate dataset 3, violating that part of the competition rules, and of course, this will very likely boost your score both in the public and private LB. However, at the end of the competition, as a winner, you're going to have to open source your code and model and \"reveal\" how you've reached the results that you've got. And that's where your submission will be invalidated when Kaggle staff and the competition hosts discover your breach of the competition rules.\nSo, no, I don't think anyone will want to do that.", "2278527": "momakmd I have doubts about that.\n>Open Source: You hereby license and will license your winning Submission and the source code used to generate the Submission under an Open Source Initiative-approved license (see www.opensource.org) that in no event limits commercial use of such code or model containing or depending on such code. To the extent your Submission makes use of generally commercially available software not owned by you that you used to generate your submission, but that can be procured by the Competition Sponsor without undue expense, you do not grant the license in the preceding sentence to that software.\n\nIt is said that you will have to open-source the code to reproduce the submission, not the code to train your model.\n@addisonhoward can You please tighten the rules? In the present state of the rules, it is not possible to reveal if a model was trained with a human-labeled Dataset3", "2279805": "Oh sorry, I've missed the part that the question is about dataset 3.", "2281069": "Hi there. Kaggle is not a legal party to the competition rules, and therefore we cannot provide legal advice or interpretation. However I would also like to direct you to Rule B.10, which includes:\n\n>Competition Sponsor reserves the right to disqualify any participant from the Competition if the Competition Sponsor reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other participants, Competition Sponsor or Kaggle.\n\n>A disqualified participant may be removed from the Competition leaderboard, at Kaggle's sole discretion. If a Participant is removed from the Competition Leaderboard, additional winning features associated with the Kaggle competition platform, for example Kaggle points or medals, may also not be awarded.", "2281300": "yashvrdnjain @katherinegustilo can you please clarify this?", "2281423": "Hello,\nHuman annotation of any data is not allowed. On competition end, the winning teams will be asked to submit their training and inference code, trained models, description of methods for reproducing the solution, and all training data used (including external datasets). All these will be used to reproduce and validate the solutions against the leaderboard scores. If we suspect human annotations were used (or any other violation of competition rules), your submission will not be eligible for the prize. Other penalties may follow as per Kaggle rules.", "2281765": "Glad to hear that, thanks!", "2310544": "I just entered this competition and I am concerned about this topic.\nMy understanding from your comment is that it is acceptable to use pseudo-labels by model predictions because they can be reproduced without human annotation, is that correct?", "2340305": "addisonhoward but this does not state that it is forbidden to annotate the train set. Dataset 3 is not validation or test data. There is nothing in the rules that says it is forbidden to annotate the train set manually.", "2340321": "yashvrdnjain is there a reason this is not in the rules?\n\nWhy not change Rule B.5 (which actually allows annotation of train data) from:\n>Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\nto:\n>Submissions may not use or incorporate information from hand labeling or human prediction.\n\nWhat is the reason for not allowing hand labeling? For competitors, it is better when it is not allowed since it requires a lot of manual work. But, by allowing labeling, you will most likely get a better model that is still open-source.", "2340331": "I believe most of the WSIs provided are part of previous HuBMAP competition including WSI 3/4 which are used to evaluate public leaderboard test data. I think it's good that hand labelling is not allowed, should prevent any leakage and unfair competition, also encouraging semi supervised solutions.", "2340632": "crodoc \n\nHand labeling is never allowed in any of the comps, afaik. That said we select folds, isn't that a sort of hand labeling? I guess there is some blurry line", "2340644": "actually top solutions in last computer vision competitions used hand labeling on training data", "2340891": "Hand labeling the train set or external data is usually allowed. You are not allowed to label public or private test data if it is given (sometimes you get the test set and you only submit a csv file that you can generate locally)." }, "source": "meta" }