@cdcarr @juliaelliott - This might be obvious to others but I don't find it called out anywhere. I hope it's not out of line that I ask this.
\nAre there rules against hand-labelling the public-test dataset for use as a validation dataset? If so, how is this policed and what are the consequences?
\nI'm asking this publicly because there is no way people will not hand-label this dataset. If you did 5 submissions a day to probe… you'd be done in less than 3 weeks (and there are probably faster ways known by those smarter than me).
\nThe only related rule I could find was…
\n\n\n\"Submissions may not use or incorporate information from hand labelling or human prediction of the validation dataset or test data records.\"
\n
Additionally, there is this point about external data (which this, technically, is not… still…)
\n\n\n\"C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).\"
\n
Would this indicate that the data should be shared if hand-labelled? Because \"test your submissions\" would seem pretty similar to \"hand-labelling for validation purposes\"
\nI simply wanted to bring this topic out in the open now… before we start seeing 1.00 on the LB. I hope this makes sense and will make things crystal clear regarding this topic.
\nThanks in advance!
", "messages": [ { "id": 1410185, "postDate": "2021-08-02T17:37:32.143Z", "content": "@cdcarr @juliaelliott - This might be obvious to others but I don't find it called out anywhere. I hope it's not out of line that I ask this.
\nAre there rules against hand-labelling the public-test dataset for use as a validation dataset? If so, how is this policed and what are the consequences?
\nI'm asking this publicly because there is no way people will not hand-label this dataset. If you did 5 submissions a day to probe… you'd be done in less than 3 weeks (and there are probably faster ways known by those smarter than me).
\nThe only related rule I could find was…
\n\n\n\"Submissions may not use or incorporate information from hand labelling or human prediction of the validation dataset or test data records.\"
\n
Additionally, there is this point about external data (which this, technically, is not… still…)
\n\n\n\"C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all participants of the Competition for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).\"
\n
Would this indicate that the data should be shared if hand-labelled? Because \"test your submissions\" would seem pretty similar to \"hand-labelling for validation purposes\"
\nI simply wanted to bring this topic out in the open now… before we start seeing 1.00 on the LB. I hope this makes sense and will make things crystal clear regarding this topic.
\nThanks in advance!
", "rawMarkdown": "@cdcarr @juliaelliott - This might be obvious to others but I don't find it called out anywhere. I hope it's not out of line that I ask this.\n\n**Are there rules against hand-labelling the public-test dataset for use as a validation dataset? If so, how is this policed and what are the consequences?**\n\n---\n\nI'm asking this publicly because there is no way people will not hand-label this dataset. If you did 5 submissions a day to probe... you'd be done in less than 3 weeks (and there are probably faster ways known by those smarter than me).\n\n---\n\nThe only related rule I could find was...\n\n> \"Submissions may not **use** or **incorporate** information from hand labelling or human prediction of the validation dataset or test data records.\"\n\n---\n\nAdditionally, there is this point about external data (which this, technically, is not... still...)\n\n> \"C. External Data. You may use data other than the Competition Data (“External Data”) to develop and test your Submissions. However, you will **ensure the External Data is publicly available and equally accessible to use by all participants of the Competition** for purposes of the competition at no cost to the other participants. The ability to use External Data under this Section 7.C (External Data) does not limit your other obligations under these Competition Rules, including but not limited to Section 11 (Winners Obligations).\"\n\n**Would this indicate that the data should be shared if hand-labelled? Because \"test your submissions\" would seem pretty similar to \"hand-labelling for validation purposes\"**\n\n---\n\nI simply wanted to bring this topic out in the open now... before we start seeing 1.00 on the LB. I hope this makes sense and will make things crystal clear regarding this topic.\n\nThanks in advance!", "votes": 43 }, { "id": 1447662, "postDate": "2021-08-04T15:34:58.277Z", "content": "I think there's a couple of different concepts getting crossed here, which is understandable because the difference is subtle. 1. Leaderboard probing, which is making submissions that specifically give you insight to data on the public LB is generally allowed. Search LB probing on the forums and you'll find some amazingly creative ways this has been done in past competitions. 2. Hand labeling which is using non-programatic techniques (ie human intuition, perception) is generally not allowed unless specifically permitted in the rules.
\nSince this competition has a unique policy that the leaders on the public LB at the end of Aug (before the comp ends) will be invited to present their work, I really hope no one will probe and then submit probed values and thus interfere with the ability identify the real top solutions.
", "rawMarkdown": "I think there's a couple of different concepts getting crossed here, which is understandable because the difference is subtle. 1. ***Leaderboard probing***, which is making submissions that specifically give you insight to data on the public LB is generally allowed. Search LB probing on the forums and you'll find some amazingly creative ways this has been done in past competitions. 2. ***Hand labeling*** which is using non-programatic techniques (ie human intuition, perception) is generally not allowed unless specifically permitted in the rules.\n\nSince this competition has a unique policy that the leaders on the *public* LB at the end of Aug (before the comp ends) will be invited to present their work, I really hope no one will probe and then submit probed values and thus interfere with the ability identify the real top solutions.", "votes": 10, "replies": [ { "id": 1451193, "postDate": "2021-08-05T09:26:31.403Z", "content": "Given the organizers' plan is to invite the top 10 teams to present their work, how accurate is such a selection procedure anyways? To me it seems like the public test set is not representative of the training set at all. I experience CV to LB score differences of roughĺy 20 percent points. What do you think? I triple checked my code already and also saw a similar thread discussing this issue already. Maybe worth it to discuss this with the organizers?
", "rawMarkdown": "Given the organizers' plan is to invite the top 10 teams to present their work, how accurate is such a selection procedure anyways? To me it seems like the public test set is not representative of the training set at all. I experience CV to LB score differences of roughĺy 20 percent points. What do you think? I triple checked my code already and also saw a similar thread discussing this issue already. Maybe worth it to discuss this with the organizers? ", "votes": 1 }, { "id": 1452187, "postDate": "2021-08-05T14:39:31.017Z", "content": "Technically, the technique described above would be viewed as LB probing then… and would be a valid strategy. Interesting take! Thanks for the comment.
\nI second your sentiment and hope that no one probes and submits these values as it will destroy the integrity of the Public LB.
", "rawMarkdown": "Technically, the technique described above would be viewed as LB probing then... and would be a valid strategy. Interesting take! Thanks for the comment.\n\nI second your sentiment and hope that no one probes and submits these values as it will destroy the integrity of the Public LB.", "votes": 3 } ] }, { "id": 1459326, "postDate": "2021-08-08T10:20:59.683Z", "content": "Hi there.
\nI had posted a very similar concern in this thread.
\n👉 https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352#1402817
I guess it depends on what the hosts and Kaggle administration think about this competition, but unfortunately in the last competition I participated in, HuBMAP, which is a kidney tissue segmentation competition, hand labeling was allowed 😭
\nDiscussion about hand labeing in HuBMAP
\n👉 https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616)
More specifically, if a good result was obtained in public by hand labeling, many participants would use that label for training. However, the competition had the drawback of having different annotation methods for Public and Private, so it was ironic that there was a big shake down when hand labeling was introduced.
\nI don't know if hand labeling will be allowed in this competition, but considering the meaning and properness of the competition, I believe it would be better for both the organizer and the participants to ban hand labeling.
I'm gonna do this, because why not? 87 more training samples(~15% of the training set). And I think this will only fit the 87 test samples, not the whole LB set or private test set. But I really don't think it's a good idea to give so much samples in the sample_submission.csv file, especially for such a small train set, 1 or 3 samples is engough for debugging.
Moreover, if this is not against the rule, I think who have done this should make the results public.
", "rawMarkdown": "I'm gonna do this, because why not? 87 more training samples(~15% of the training set). And I think this will only fit the 87 test samples, not the whole LB set or private test set. But I really don't think it's a good idea to give so much samples in the `sample_submission.csv` file, especially for such a small train set, 1 or 3 samples is engough for debugging.\n\nMoreover, if this is not against the rule, I think who have done this should make the results public.", "votes": 1, "replies": [ { "id": 1481498, "postDate": "2021-08-19T14:44:59.623Z", "content": "I did it. I added [HL] to our team name to indicate as such. I’ll remove the [HL] when We use a model that does not incorporate the probed labels in submission/train-set.
\nI’d be cautious about using the public LB to train with as it may violate the rules stated above in my main post. I think using it as a CV set is allowed though?
\nThat being said… this needs clarity from the hosts to put things to rest.
", "rawMarkdown": "I did it. I added [HL] to our team name to indicate as such. I’ll remove the [HL] when We use a model that does not incorporate the probed labels in submission/train-set.\n\nI’d be cautious about using the public LB to train with as it may violate the rules stated above in my main post. I think using it as a CV set is allowed though? \n\nThat being said… this needs clarity from the hosts to put things to rest.", "votes": 1 }, { "id": 1481523, "postDate": "2021-08-19T14:59:18.240Z", "content": "I think there are no differences with how to use these samples(train or validate). Point is we get more labeled samples.
\nI'll do what you have done, the [HL] mark, and waiting for the clarity from the host.
I don't really understand the decision of the organizers to supply the public LB data to the contestants and identify it as such, making LB probing so easy. In other Kaggle contests I have competed in, both Code Competitions and others, the public LB was usually based on a random sample of the total test set. That would have made LB probing in this contest a lot harder, even with the relatively small total number of records.
", "rawMarkdown": "I don't really understand the decision of the organizers to supply the public LB data to the contestants and identify it as such, making LB probing so easy. In other Kaggle contests I have competed in, both Code Competitions and others, the public LB was usually based on a random sample of the total test set. That would have made LB probing in this contest a lot harder, even with the relatively small total number of records.", "votes": 2 } ] }, { "id": 1477911, "postDate": "2021-08-17T17:38:09.010Z", "content": "It's a request that, if someone uses hand labelling they can write this on the side of the name of the team so that we can see them on the leaderboard.
\nI am not saying you shouldn't do handlabeling.
\nI said just to make the leaderboard not lose its usefulness,
\nThanks
Agreed. I submitted a 20 count of hand-labelled predictions to test if it was working today and it shot our team to second place. If I could erase that submission I would.
\nUntil our models have surpassed this level of performance (0.776), I will add/keep \"[HL]\" in the beginning of our team name -> [HL] Quantiphi. Only after we have surpassed this performance with a model(s) trained on the provided training data (not on any hand-labelled data) will we remove the [HL] tag.
\nApologies all for cluttering things up and/or causing confusion.
\n@juliaelliott - You may also remove our latest submission from the public LB if that helps too. Thanks!
", "rawMarkdown": "Agreed. I submitted a 20 count of hand-labelled predictions to test if it was working today and it shot our team to second place. If I could erase that submission I would. \n\n---\n\n**Until our models have surpassed this level of performance (0.776), I will add/keep \"[HL]\" in the beginning of our team name -> *[HL] Quantiphi*. Only after we have surpassed this performance with a model(s) trained on the provided training data (not on any hand-labelled data) will we remove the [HL] tag.**\n\n---\n\nApologies all for cluttering things up and/or causing confusion.\n\n@juliaelliott - You may also remove our latest submission from the public LB if that helps too. Thanks!", "votes": 7 }, { "id": 1477970, "postDate": "2021-08-17T18:04:26.760Z", "content": "Thanks, @dschettler8845, that's the competition spirit!!!
", "rawMarkdown": "Thanks, @dschettler8845, that's the competition spirit!!!", "votes": 1 }, { "id": 1482510, "postDate": "2021-08-20T06:00:12.747Z", "content": "I'm responding on behalf of Julia: We are not planning to take any action at this point, thanks for the context.
", "rawMarkdown": "I'm responding on behalf of Julia: We are not planning to take any action at this point, thanks for the context. ", "votes": 6 }, { "id": 1482516, "postDate": "2021-08-20T06:05:27.707Z", "content": "Glad to hear that.
", "rawMarkdown": "Glad to hear that." } ] }, { "id": 1457967, "postDate": "2021-08-07T16:41:03.693Z", "content": "Can we get a clarification on this?
", "rawMarkdown": "Can we get a clarification on this?", "votes": 1 }, { "id": 1484201, "postDate": "2021-08-21T05:55:09.750Z", "content": "For anyone who has successfully hand-labelled majority of the testing dataset. Would really appreciate it if those values can be made public (as a notebook or dataset) to increase testing data for others.
", "rawMarkdown": "For anyone who has successfully hand-labelled majority of the testing dataset. Would really appreciate it if those values can be made public (as a notebook or dataset) to increase testing data for others. " }, { "id": 1486872, "postDate": "2021-08-23T09:16:59.953Z", "content": "Actually, there is a way of hand labelling without getting to the top of the leaderboard. Instead of going up, one can go down - by incrementally getting labels wrong.
\nAnyway, could someone who has HLed the test set share these findings? 😈
\nMight be useful for others and prevent the cluttering on the LB.
Should they public those hand-labelled data?
", "rawMarkdown": "Should they public those hand-labelled data?" }, { "id": 1503216, "postDate": "2021-09-05T06:38:42.303Z", "content": "I wonder why the test set is so small. Is it really that hard to make a significant big one so that this kind of things are not feasible?
", "rawMarkdown": "I wonder why the test set is so small. Is it really that hard to make a significant big one so that this kind of things are not feasible? " }, { "id": 1498782, "postDate": "2021-09-01T07:37:32.583Z", "content": "\"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\", so we cannot use public test result by hand-labeling for additional training data or validation set.
", "rawMarkdown": "\"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\", so we cannot use public test result by hand-labeling for additional training data or validation set.", "replies": [ { "id": 1499224, "postDate": "2021-09-01T14:04:39.667Z", "content": "This was my thought as well. I had just wanted to hear it directly from the competition organizers as I know many people will take the gray area (lack of response) as permission to do whatever they want.
", "rawMarkdown": "This was my thought as well. I had just wanted to hear it directly from the competition organizers as I know many people will take the gray area (lack of response) as permission to do whatever they want.", "votes": 1 }, { "id": 1499234, "postDate": "2021-09-01T14:11:51.760Z", "content": "Yeah, current status is \"no response\".
", "rawMarkdown": "Yeah, current status is \"no response\".", "votes": 1 } ] }, { "id": 1494005, "postDate": "2021-08-28T10:24:47.130Z", "content": "This is why I find competitions that can not be probed or hand labeled interesting 👀
", "rawMarkdown": "This is why I find competitions that can not be probed or hand labeled interesting 👀" }, { "id": 1490977, "postDate": "2021-08-26T04:09:15.997Z", "content": "I found this part of the rules
\n\n\nA. Data Access and Use. For the purposes of the competition, you may access and use the Competition Data for non-commercial purposes only, including for participating in the Competition and on Kaggle.com forums, and for academic research and education. Following the challenge's completion, there may be different revisions in licensing terms as determined by the contributing institutions at that time. The Competition Sponsor reserves the right to disqualify any participant who uses the Competition Data other than as permitted by the Competition Website and these Rules.
\n
\n Imaging studies used in this Challenge have been contributed by several sources and are de-identified patient data, meaning that the contributing institutions have taken reasonable care to remove from them all personally identifiable information. As a participant in the Competition, you agree 1) not to make copies of, or in any way redistribute, any of the data made available to you, 2) not to attempt to re-identify any personal information based on the data, 3) not to attempt to probe the test set's labels, and 4) to notify the Competition organizers of any personally identifiable information you encounter in the data by posting a message to the Kaggle community discussion forums.
Perhaps we are prohibited from probing the public lb… Can anyone confirm this?
", "rawMarkdown": "I found this part of the rules\n> A. Data Access and Use. For the purposes of the competition, you may access and use the Competition Data for non-commercial purposes only, including for participating in the Competition and on Kaggle.com forums, and for academic research and education. Following the challenge's completion, there may be different revisions in licensing terms as determined by the contributing institutions at that time. The Competition Sponsor reserves the right to disqualify any participant who uses the Competition Data other than as permitted by the Competition Website and these Rules.\nImaging studies used in this Challenge have been contributed by several sources and are de-identified patient data, meaning that the contributing institutions have taken reasonable care to remove from them all personally identifiable information. As a participant in the Competition, you agree 1) not to make copies of, or in any way redistribute, any of the data made available to you, 2) not to attempt to re-identify any personal information based on the data, 3) **not to attempt to probe the test set's labels**, and 4) to notify the Competition organizers of any personally identifiable information you encounter in the data by posting a message to the Kaggle community discussion forums.\n\nPerhaps we are prohibited from probing the public lb... Can anyone confirm this?" }, { "id": 1460552, "postDate": "2021-08-09T00:28:15.307Z", "content": "Has anyone tried? I am trying to upload a submission of a handcrafted radiomics model which I coded with Matlab (Honestly) just to check it against other models, but I am getting submission errors… so I guess they programed something against it?
", "rawMarkdown": "Has anyone tried? I am trying to upload a submission of a handcrafted radiomics model which I coded with Matlab (Honestly) just to check it against other models, but I am getting submission errors... so I guess they programed something against it? " }, { "id": 1436320, "postDate": "2021-08-03T15:03:48.340Z", "content": "Great question @dschettler8845 .. Outside of farming notebook votes, hand labeling to fit the LB seems a poor idea to me. Is there some trick this leads to?
", "rawMarkdown": "Great question @dschettler8845 .. Outside of farming notebook votes, hand labeling to fit the LB seems a poor idea to me. Is there some trick this leads to?", "replies": [ { "id": 1442728, "postDate": "2021-08-04T00:08:19.437Z", "content": "To hand-label the dataset you make a submission with all 0s. (this scores 0.500 AUC)
\nThen you simply change one row at a time and submit.
\nOnce you have this dataset, you could theoretically use it as a perfect replica of the public LB. There really is no way for Kaggle to know that you're doing this outside of looking for a submission pattern as described above.
\nThis is why I wanted to bring it up now so that it doesn't contribute to an unfair advantage (or break the rules) for those who think/want to do it.
", "rawMarkdown": "To hand-label the dataset you make a submission with all 0s. (this scores 0.500 AUC)\n\nThen you simply change one row at a time and submit.\n- If the changed row is correct... the score will change to something like 0.511 ... if incorrect it will change to something like 0.489.\n- With 5 submissions a day and a public dataset of only 87 images... it would take 17 days (with 3 left-over) to completely label the public dataset this way.\n\nOnce you have this dataset, you could theoretically use it as a perfect replica of the public LB. There really is no way for Kaggle to know that you're doing this outside of looking for a submission pattern as described above.\n\nThis is why I wanted to bring it up now so that it doesn't contribute to an unfair advantage (or break the rules) for those who think/want to do it.", "votes": 12 }, { "id": 1442731, "postDate": "2021-08-04T00:09:40.050Z", "content": "As I've heard nothing from the Kaggle Admins or the host (I emailed directly as well)… I'm leaning towards thinking this approach is not against the rules.
\nThat being said I would really like some confirmation from @juliaelliott or @cdcarr one way or another.
", "rawMarkdown": "As I've heard nothing from the Kaggle Admins or the host (I emailed directly as well)... I'm leaning towards thinking this approach is not against the rules.\n\nThat being said I would really like some confirmation from @juliaelliott or @cdcarr one way or another." }, { "id": 1443371, "postDate": "2021-08-04T02:45:29.117Z", "content": "Of course you could get a perfect public LB score doing that, my question is why would anyone want to do it? Is there some advantage to purposefully overfitting the LB?
", "rawMarkdown": "Of course you could get a perfect public LB score doing that, my question is why would anyone want to do it? Is there some advantage to purposefully overfitting the LB?", "votes": 2 }, { "id": 1444991, "postDate": "2021-08-04T07:33:11.227Z", "content": "Using hand-labeled public test as pseudo-labels is generally prohibited, I guess.
", "rawMarkdown": "Using hand-labeled public test as pseudo-labels is generally prohibited, I guess." }, { "id": 1445810, "postDate": "2021-08-04T10:10:06.380Z", "content": "@davidbroberts if someone has access to labels of 22% of whole test data they can use it as train data. This gives them obvious advantage in final LB.
\nWhat if we only use it as a CV set (don't train on it?).
", "rawMarkdown": "What if we only use it as a CV set (don't train on it?).", "votes": 1 }, { "id": 1446652, "postDate": "2021-08-04T12:47:12.593Z", "content": "Thanks @abhijeetptl5 .. but how could someone get access to the 22% to train on? I understand you could hand label and figure out the predictions, but I don't get how you could train on them.
", "rawMarkdown": "Thanks @abhijeetptl5 .. but how could someone get access to the 22% to train on? I understand you could hand label and figure out the predictions, but I don't get how you could train on them.", "votes": 1 }, { "id": 1446848, "postDate": "2021-08-04T13:18:07.423Z", "content": "@davidbroberts The published test data is 22% of test data on which final LB will get calculated (as mentioned in competition rules).
", "rawMarkdown": "@davidbroberts The published test data is 22% of test data on which final LB will get calculated (as mentioned in competition rules). " }, { "id": 1446883, "postDate": "2021-08-04T13:24:06.720Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1446886, "postDate": "2021-08-04T13:25:02.610Z", "content": "@abhijeetptl5 .. I get it now. Brain fart for a minute. Thanks for clarifying.
", "rawMarkdown": "@abhijeetptl5 .. I get it now. Brain fart for a minute. Thanks for clarifying.", "votes": 1 }, { "id": 1489164, "postDate": "2021-08-24T18:47:52.830Z", "content": "Hi @dschettler8845 I tried to follow the approach you shared. When I changed the MGMT value for a single row from 0 to 1, the AUC neither increased or decreased it stayed at 0.5. Any idea whats going on here?
", "rawMarkdown": "Hi @dschettler8845 I tried to follow the approach you shared. When I changed the MGMT value for a single row from 0 to 1, the AUC neither increased or decreased it stayed at 0.5. Any idea whats going on here?" }, { "id": 1489182, "postDate": "2021-08-24T18:58:35.720Z", "content": "I assume you're making a mistake in your submission process. Without seeing the code I wouldn't be able to help.
", "rawMarkdown": "I assume you're making a mistake in your submission process. Without seeing the code I wouldn't be able to help. " }, { "id": 1489226, "postDate": "2021-08-24T19:42:58.473Z", "content": "In my code I read the sample csv file and sort based on BraTS21ID. Then set all values for MGMT as zero and only change the row at index 1 to 1. When I submit all zeros the AUC is 0.5 and when I change the MGMT value for 1st row i.e. for id 00013 the AUC is still 0.5. I'm not sure what is the error.
\nsubmission = pd.read_csv(\"sample_submission.csv\", dtype=\"object\")\nsubmission = submission.sort_values(\"BraTS21ID\")\nsubmission[\"MGMT_value\"] = 0\nsubmission.loc[1, [\"MGMT_value\"]] = 1\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)\nAs of my understanding, this 1st row i.e, for id 00013 doesn't belong to the public test set. I mean it is not in that 22% of test data. change any other id and see if it changes. If any other id changes the AUC score then that id contributes to the public test set.
As of my understanding, this 1st row i.e, for id 00013 doesn't belong to the public test set. I mean it is not in that 22% of test data. change any other id and see if it changes. If any other id changes the AUC score then that id contributes to the public test set.
I think there's a couple of different concepts getting crossed here, which is understandable because the difference is subtle. 1. Leaderboard probing, which is making submissions that specifically give you insight to data on the public LB is generally allowed. Search LB probing on the forums and you'll find some amazingly creative ways this has been done in past competitions. 2. Hand labeling which is using non-programatic techniques (ie human intuition, perception) is generally not allowed unless specifically permitted in the rules.
\nSince this competition has a unique policy that the leaders on the public LB at the end of Aug (before the comp ends) will be invited to present their work, I really hope no one will probe and then submit probed values and thus interfere with the ability identify the real top solutions.
", "votes": 10, "replies": [ { "id": 1451193, "author_name": "Max Lübbering", "author_url": "", "post_date": "2021-08-05T09:26:31.403000", "content": "Given the organizers' plan is to invite the top 10 teams to present their work, how accurate is such a selection procedure anyways? To me it seems like the public test set is not representative of the training set at all. I experience CV to LB score differences of roughĺy 20 percent points. What do you think? I triple checked my code already and also saw a similar thread discussing this issue already. Maybe worth it to discuss this with the organizers?
", "votes": 1, "replies": [] }, { "id": 1452187, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-05T14:39:31.017000", "content": "Technically, the technique described above would be viewed as LB probing then… and would be a valid strategy. Interesting take! Thanks for the comment.
\nI second your sentiment and hope that no one probes and submits these values as it will destroy the integrity of the Public LB.
", "votes": 3, "replies": [] } ] }, { "id": 1459326, "author_name": "Maxwell", "author_url": "", "post_date": "2021-08-08T10:20:59.683000", "content": "Hi there.
\nI had posted a very similar concern in this thread.
\n👉 https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/255352#1402817
I guess it depends on what the hosts and Kaggle administration think about this competition, but unfortunately in the last competition I participated in, HuBMAP, which is a kidney tissue segmentation competition, hand labeling was allowed 😭
\nDiscussion about hand labeing in HuBMAP
\n👉 https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616)
More specifically, if a good result was obtained in public by hand labeling, many participants would use that label for training. However, the competition had the drawback of having different annotation methods for Public and Private, so it was ironic that there was a big shake down when hand labeling was introduced.
\nI don't know if hand labeling will be allowed in this competition, but considering the meaning and properness of the competition, I believe it would be better for both the organizer and the participants to ban hand labeling.
I'm gonna do this, because why not? 87 more training samples(~15% of the training set). And I think this will only fit the 87 test samples, not the whole LB set or private test set. But I really don't think it's a good idea to give so much samples in the sample_submission.csv file, especially for such a small train set, 1 or 3 samples is engough for debugging.
Moreover, if this is not against the rule, I think who have done this should make the results public.
", "votes": 1, "replies": [ { "id": 1481498, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-19T14:44:59.623000", "content": "I did it. I added [HL] to our team name to indicate as such. I’ll remove the [HL] when We use a model that does not incorporate the probed labels in submission/train-set.
\nI’d be cautious about using the public LB to train with as it may violate the rules stated above in my main post. I think using it as a CV set is allowed though?
\nThat being said… this needs clarity from the hosts to put things to rest.
", "votes": 1, "replies": [] }, { "id": 1481523, "author_name": "Chenglu", "author_url": "", "post_date": "2021-08-19T14:59:18.240000", "content": "I think there are no differences with how to use these samples(train or validate). Point is we get more labeled samples.
\nI'll do what you have done, the [HL] mark, and waiting for the clarity from the host.
I don't really understand the decision of the organizers to supply the public LB data to the contestants and identify it as such, making LB probing so easy. In other Kaggle contests I have competed in, both Code Competitions and others, the public LB was usually based on a random sample of the total test set. That would have made LB probing in this contest a lot harder, even with the relatively small total number of records.
", "votes": 2, "replies": [] } ] }, { "id": 1477911, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-17T17:38:09.010000", "content": "It's a request that, if someone uses hand labelling they can write this on the side of the name of the team so that we can see them on the leaderboard.
\nI am not saying you shouldn't do handlabeling.
\nI said just to make the leaderboard not lose its usefulness,
\nThanks
Agreed. I submitted a 20 count of hand-labelled predictions to test if it was working today and it shot our team to second place. If I could erase that submission I would.
\nUntil our models have surpassed this level of performance (0.776), I will add/keep \"[HL]\" in the beginning of our team name -> [HL] Quantiphi. Only after we have surpassed this performance with a model(s) trained on the provided training data (not on any hand-labelled data) will we remove the [HL] tag.
\nApologies all for cluttering things up and/or causing confusion.
\n@juliaelliott - You may also remove our latest submission from the public LB if that helps too. Thanks!
", "votes": 7, "replies": [] }, { "id": 1477970, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2021-08-17T18:04:26.760000", "content": "Thanks, @dschettler8845, that's the competition spirit!!!
", "votes": 1, "replies": [] }, { "id": 1482510, "author_name": "Maggie", "author_url": "", "post_date": "2021-08-20T06:00:12.747000", "content": "I'm responding on behalf of Julia: We are not planning to take any action at this point, thanks for the context.
", "votes": 6, "replies": [] }, { "id": 1482516, "author_name": "Chenglu", "author_url": "", "post_date": "2021-08-20T06:05:27.707000", "content": "Glad to hear that.
", "votes": 0, "replies": [] } ] }, { "id": 1457967, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2021-08-07T16:41:03.693000", "content": "Can we get a clarification on this?
", "votes": 1, "replies": [] }, { "id": 1484201, "author_name": "Aryaman Sharma", "author_url": "", "post_date": "2021-08-21T05:55:09.750000", "content": "For anyone who has successfully hand-labelled majority of the testing dataset. Would really appreciate it if those values can be made public (as a notebook or dataset) to increase testing data for others.
", "votes": 0, "replies": [] }, { "id": 1486872, "author_name": "charzu", "author_url": "", "post_date": "2021-08-23T09:16:59.953000", "content": "Actually, there is a way of hand labelling without getting to the top of the leaderboard. Instead of going up, one can go down - by incrementally getting labels wrong.
\nAnyway, could someone who has HLed the test set share these findings? 😈
\nMight be useful for others and prevent the cluttering on the LB.
Should they public those hand-labelled data?
", "votes": 0, "replies": [] }, { "id": 1503216, "author_name": "Md Zarif Ul Alam", "author_url": "", "post_date": "2021-09-05T06:38:42.303000", "content": "I wonder why the test set is so small. Is it really that hard to make a significant big one so that this kind of things are not feasible?
", "votes": 0, "replies": [] }, { "id": 1498782, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-09-01T07:37:32.583000", "content": "\"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.\", so we cannot use public test result by hand-labeling for additional training data or validation set.
", "votes": 0, "replies": [ { "id": 1499224, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-09-01T14:04:39.667000", "content": "This was my thought as well. I had just wanted to hear it directly from the competition organizers as I know many people will take the gray area (lack of response) as permission to do whatever they want.
", "votes": 1, "replies": [] }, { "id": 1499234, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-09-01T14:11:51.760000", "content": "Yeah, current status is \"no response\".
", "votes": 1, "replies": [] } ] }, { "id": 1494005, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2021-08-28T10:24:47.130000", "content": "This is why I find competitions that can not be probed or hand labeled interesting 👀
", "votes": 0, "replies": [] }, { "id": 1490977, "author_name": "NaN", "author_url": "", "post_date": "2021-08-26T04:09:15.997000", "content": "I found this part of the rules
\n\n\nA. Data Access and Use. For the purposes of the competition, you may access and use the Competition Data for non-commercial purposes only, including for participating in the Competition and on Kaggle.com forums, and for academic research and education. Following the challenge's completion, there may be different revisions in licensing terms as determined by the contributing institutions at that time. The Competition Sponsor reserves the right to disqualify any participant who uses the Competition Data other than as permitted by the Competition Website and these Rules.
\n
\n Imaging studies used in this Challenge have been contributed by several sources and are de-identified patient data, meaning that the contributing institutions have taken reasonable care to remove from them all personally identifiable information. As a participant in the Competition, you agree 1) not to make copies of, or in any way redistribute, any of the data made available to you, 2) not to attempt to re-identify any personal information based on the data, 3) not to attempt to probe the test set's labels, and 4) to notify the Competition organizers of any personally identifiable information you encounter in the data by posting a message to the Kaggle community discussion forums.
Perhaps we are prohibited from probing the public lb… Can anyone confirm this?
", "votes": 0, "replies": [] }, { "id": 1460552, "author_name": "Ranafago", "author_url": "", "post_date": "2021-08-09T00:28:15.307000", "content": "Has anyone tried? I am trying to upload a submission of a handcrafted radiomics model which I coded with Matlab (Honestly) just to check it against other models, but I am getting submission errors… so I guess they programed something against it?
", "votes": 0, "replies": [] }, { "id": 1436320, "author_name": "David Roberts", "author_url": "", "post_date": "2021-08-03T15:03:48.340000", "content": "Great question @dschettler8845 .. Outside of farming notebook votes, hand labeling to fit the LB seems a poor idea to me. Is there some trick this leads to?
", "votes": 0, "replies": [ { "id": 1442728, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-04T00:08:19.437000", "content": "To hand-label the dataset you make a submission with all 0s. (this scores 0.500 AUC)
\nThen you simply change one row at a time and submit.
\nOnce you have this dataset, you could theoretically use it as a perfect replica of the public LB. There really is no way for Kaggle to know that you're doing this outside of looking for a submission pattern as described above.
\nThis is why I wanted to bring it up now so that it doesn't contribute to an unfair advantage (or break the rules) for those who think/want to do it.
", "votes": 12, "replies": [] }, { "id": 1442731, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-04T00:09:40.050000", "content": "As I've heard nothing from the Kaggle Admins or the host (I emailed directly as well)… I'm leaning towards thinking this approach is not against the rules.
\nThat being said I would really like some confirmation from @juliaelliott or @cdcarr one way or another.
", "votes": 0, "replies": [] }, { "id": 1443371, "author_name": "David Roberts", "author_url": "", "post_date": "2021-08-04T02:45:29.117000", "content": "Of course you could get a perfect public LB score doing that, my question is why would anyone want to do it? Is there some advantage to purposefully overfitting the LB?
", "votes": 2, "replies": [] }, { "id": 1444991, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-08-04T07:33:11.227000", "content": "Using hand-labeled public test as pseudo-labels is generally prohibited, I guess.
", "votes": 0, "replies": [] }, { "id": 1445810, "author_name": "Abhijeet Patil", "author_url": "", "post_date": "2021-08-04T10:10:06.380000", "content": "@davidbroberts if someone has access to labels of 22% of whole test data they can use it as train data. This gives them obvious advantage in final LB.
\nWhat if we only use it as a CV set (don't train on it?).
", "votes": 1, "replies": [] }, { "id": 1446652, "author_name": "David Roberts", "author_url": "", "post_date": "2021-08-04T12:47:12.593000", "content": "Thanks @abhijeetptl5 .. but how could someone get access to the 22% to train on? I understand you could hand label and figure out the predictions, but I don't get how you could train on them.
", "votes": 1, "replies": [] }, { "id": 1446848, "author_name": "Abhijeet Patil", "author_url": "", "post_date": "2021-08-04T13:18:07.423000", "content": "@davidbroberts The published test data is 22% of test data on which final LB will get calculated (as mentioned in competition rules).
", "votes": 0, "replies": [] }, { "id": 1446883, "author_name": "", "author_url": "", "post_date": "2021-08-04T13:24:06.720000", "content": "", "votes": 0, "replies": [] }, { "id": 1446886, "author_name": "David Roberts", "author_url": "", "post_date": "2021-08-04T13:25:02.610000", "content": "@abhijeetptl5 .. I get it now. Brain fart for a minute. Thanks for clarifying.
", "votes": 1, "replies": [] }, { "id": 1489164, "author_name": "Sheldon", "author_url": "", "post_date": "2021-08-24T18:47:52.830000", "content": "Hi @dschettler8845 I tried to follow the approach you shared. When I changed the MGMT value for a single row from 0 to 1, the AUC neither increased or decreased it stayed at 0.5. Any idea whats going on here?
", "votes": 0, "replies": [] }, { "id": 1489182, "author_name": "Darien Schettler", "author_url": "", "post_date": "2021-08-24T18:58:35.720000", "content": "I assume you're making a mistake in your submission process. Without seeing the code I wouldn't be able to help.
", "votes": 0, "replies": [] }, { "id": 1489226, "author_name": "Sheldon", "author_url": "", "post_date": "2021-08-24T19:42:58.473000", "content": "In my code I read the sample csv file and sort based on BraTS21ID. Then set all values for MGMT as zero and only change the row at index 1 to 1. When I submit all zeros the AUC is 0.5 and when I change the MGMT value for 1st row i.e. for id 00013 the AUC is still 0.5. I'm not sure what is the error.
\nsubmission = pd.read_csv(\"sample_submission.csv\", dtype=\"object\")\nsubmission = submission.sort_values(\"BraTS21ID\")\nsubmission[\"MGMT_value\"] = 0\nsubmission.loc[1, [\"MGMT_value\"]] = 1\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)\nAs of my understanding, this 1st row i.e, for id 00013 doesn't belong to the public test set. I mean it is not in that 22% of test data. change any other id and see if it changes. If any other id changes the AUC score then that id contributes to the public test set.
As of my understanding, this 1st row i.e, for id 00013 doesn't belong to the public test set. I mean it is not in that 22% of test data. change any other id and see if it changes. If any other id changes the AUC score then that id contributes to the public test set.