{ "id": 237290, "title": "Use of Hand-labelled possible missing grid points is prohibited in this competition ", "url": "/competitions/indoor-location-navigation/discussion/237290", "author_name": "mamas", "post_date": "2021-05-07T23:19:11.257000", "votes": 21, "comment_count": 9, "views": 0, "content": "

The kaggle admin @inversion said

\n
The guiding general principle is this: \"Would this method still be applicable to a completely new, unseen test set?\" If the answer is YES, then it is allowable. If the answer is, NO then it is not allowable.\n\nSo, in this case, hand labeling of grid points that would be used for Test traces would not be acceptable. An automated process to fill in grid points that would work for new locations would be acceptable.\n
\n

in this discussion, so, please don't use the dataset. Of course, our team will never use the subs that use hand-labelled possible missing grid points for the final submissions. The valid sub scores 1.728 now, while the sub with hand-labelled possible missing grid points scores 1.604.

", "messages": [ { "id": 1297323, "postDate": "2021-05-07T23:19:11.257Z", "content": "

The kaggle admin @inversion said

\n
The guiding general principle is this: \"Would this method still be applicable to a completely new, unseen test set?\" If the answer is YES, then it is allowable. If the answer is, NO then it is not allowable.\n\nSo, in this case, hand labeling of grid points that would be used for Test traces would not be acceptable. An automated process to fill in grid points that would work for new locations would be acceptable.\n
\n

in this discussion, so, please don't use the dataset. Of course, our team will never use the subs that use hand-labelled possible missing grid points for the final submissions. The valid sub scores 1.728 now, while the sub with hand-labelled possible missing grid points scores 1.604.

", "rawMarkdown": "The kaggle admin @inversion said \n```\nThe guiding general principle is this: \"Would this method still be applicable to a completely new, unseen test set?\" If the answer is YES, then it is allowable. If the answer is, NO then it is not allowable.\n\nSo, in this case, hand labeling of grid points that would be used for Test traces would not be acceptable. An automated process to fill in grid points that would work for new locations would be acceptable.\n```\n[in this discussion](https://www.kaggle.com/c/indoor-location-navigation/discussion/235813#1297306), so, please don't use the [dataset](https://www.kaggle.com/saitodevel01/indoor-navigation-hand-labeled-waypoints). Of course, our team will never use the subs that use hand-labelled possible missing grid points for the final submissions. The valid sub scores ~~1.704~~ 1.728 now, while the sub with hand-labelled possible missing grid points scores 1.604. ", "votes": 21 }, { "id": 1303310, "postDate": "2021-05-12T02:29:59.697Z", "content": "

Reminder for all the participants: Please never forget to exclude hand-labelled waypoints from your code and NEVER choose the submission that uses hand-labelled waypoints for the final submissions!

", "rawMarkdown": "Reminder for all the participants: Please never forget to exclude hand-labelled waypoints from your code and NEVER choose the submission that uses hand-labelled waypoints for the final submissions!", "votes": 5, "replies": [ { "id": 1303764, "postDate": "2021-05-12T08:23:33.090Z", "content": "

Thanks for the reminder. We already chose submissions w/o hand-labeled waypoints.

", "rawMarkdown": "Thanks for the reminder. We already chose submissions w/o hand-labeled waypoints.", "votes": 4 } ] }, { "id": 1303718, "postDate": "2021-05-12T07:54:29.557Z", "content": "

I think it's impossible to verify whether or not participants have used these hand-labelled waypoints. Unfortunately, I think many will.

", "rawMarkdown": "I think it's impossible to verify whether or not participants have used these hand-labelled waypoints. Unfortunately, I think many will.", "votes": 1, "replies": [ { "id": 1309389, "postDate": "2021-05-15T22:30:40.990Z", "content": "

I agree, and it's unfortunate. Especially for those of us competing for medals, but probably not in the running for cash, since our code will likely not be reviewed. What would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.

\n

Better yet would be reviewing a random sample of all gold/silver/bronze kaggle notebooks. But I'm not sure if this is realistic with kaggle's terms, and would be even more challenging to audit teams not using kaggle notebooks.

", "rawMarkdown": "I agree, and it's unfortunate. Especially for those of us competing for medals, but probably not in the running for cash, since our code will likely not be reviewed. What would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.\n\nBetter yet would be reviewing a random sample of all gold/silver/bronze kaggle notebooks. But I'm not sure if this is realistic with kaggle's terms, and would be even more challenging to audit teams not using kaggle notebooks. ", "votes": 4 }, { "id": 1309399, "postDate": "2021-05-15T22:49:05.487Z", "content": "

also… fwiw, the rules definitely seem to imply that the team itself is actually disqualified if caught, and presumably can't plead ignorance or \"fall back\" on their 2nd submission.

\n

\"Submissions are void if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late. Competition Sponsor reserves the right to disqualify any entrant who does not follow these Rules, including making a Submission that does not meet the Requirements.\"

", "rawMarkdown": "also... fwiw, the rules definitely seem to imply that the team itself is actually disqualified if caught, and presumably can't plead ignorance or \"fall back\" on their 2nd submission. \n\n\"Submissions are void if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late. Competition Sponsor reserves the right to disqualify any entrant who does not follow these Rules, including making **a Submission** that does not meet the Requirements.\"", "votes": 1 }, { "id": 1309461, "postDate": "2021-05-16T01:30:54.907Z", "content": "
What would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.\n
\n

I agree with you. I think if one of the final submissions breaks the rule, both final submissions should be disqualified, because kaggle admin clearly said using hand-labelled missing grid points are breaking the rule.

", "rawMarkdown": "```\nWhat would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.\n```\nI agree with you. I think if one of the final submissions breaks the rule, both final submissions should be disqualified, because kaggle admin clearly said using hand-labelled missing grid points are breaking the rule. ", "votes": 3 }, { "id": 1309472, "postDate": "2021-05-16T01:46:35.297Z", "content": "

From this thread, the effect of hand-labelled waypoint is 0.02 ~ 0.12, so it is a big problem especially for the teams that are in the borderline (the teams which score 2.7x ~ 2.9x and 3.9x ~ 4.1x and 4.2x ~ 4.4x)

", "rawMarkdown": "From [this thread](https://www.kaggle.com/c/indoor-location-navigation/discussion/237293), the effect of hand-labelled waypoint is 0.02 ~ 0.12, so it is a big problem especially for the teams that are in the borderline (the teams which score 2.7x ~ 2.9x and 3.9x ~ 4.1x and 4.2x ~ 4.4x)", "votes": 3 }, { "id": 1309836, "postDate": "2021-05-16T10:40:05.483Z", "content": "

I have just changed my selected submissions.
\nMy best public score now goes from 4.082 to 4.143.
\nI joined the competition to learn from people and not to win any medal so I am going to be coherent with it. However, I must say it feels discouraging once you are in the medal zone to see how easy is to cheat.
\nI believe it is going to be impossible for the organizers to detect people who use some form of manual waypoints. For the public dataset though there are at least two possibilities:

\n
    \n
  1. If the submission comes from a Kernel then just check whether the manual waypoints were used or not.
    2. \n
  2. If the submission does not come from a Kernel then try to match submissions waypoints with waypoints coming from public manual waypoints.
    3. \n
\n

If the submission uses manual waypoints created by the user then I think it is impossible to detect it. The only possibility I see is to submit code instead of CSVs.

", "rawMarkdown": "I have just changed my selected submissions.\nMy best public score now goes from 4.082 to 4.143.\nI joined the competition to learn from people and not to win any medal so I am going to be coherent with it. However, I must say it feels discouraging once you are in the medal zone to see how easy is to cheat.\nI believe it is going to be impossible for the organizers to detect people who use some form of manual waypoints. For the public dataset though there are at least two possibilities: \n1. If the submission comes from a Kernel then just check whether the manual waypoints were used or not.\n2. If the submission does not come from a Kernel then try to match submissions waypoints with waypoints coming from public manual waypoints.\n\nIf the submission uses manual waypoints created by the user then I think it is impossible to detect it. The only possibility I see is to submit code instead of CSVs.", "votes": 5 }, { "id": 1310041, "postDate": "2021-05-16T13:07:37.363Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1303310, "author_name": "mamas", "author_url": "", "post_date": "2021-05-12T02:29:59.697000", "content": "

Reminder for all the participants: Please never forget to exclude hand-labelled waypoints from your code and NEVER choose the submission that uses hand-labelled waypoints for the final submissions!

", "votes": 5, "replies": [ { "id": 1303764, "author_name": "higepon", "author_url": "", "post_date": "2021-05-12T08:23:33.090000", "content": "

Thanks for the reminder. We already chose submissions w/o hand-labeled waypoints.

", "votes": 4, "replies": [] } ] }, { "id": 1303718, "author_name": "rfbr", "author_url": "", "post_date": "2021-05-12T07:54:29.557000", "content": "

I think it's impossible to verify whether or not participants have used these hand-labelled waypoints. Unfortunately, I think many will.

", "votes": 1, "replies": [ { "id": 1309389, "author_name": "Paul Fornia", "author_url": "", "post_date": "2021-05-15T22:30:40.990000", "content": "

I agree, and it's unfortunate. Especially for those of us competing for medals, but probably not in the running for cash, since our code will likely not be reviewed. What would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.

\n

Better yet would be reviewing a random sample of all gold/silver/bronze kaggle notebooks. But I'm not sure if this is realistic with kaggle's terms, and would be even more challenging to audit teams not using kaggle notebooks.

", "votes": 4, "replies": [] }, { "id": 1309399, "author_name": "Paul Fornia", "author_url": "", "post_date": "2021-05-15T22:49:05.487000", "content": "

also… fwiw, the rules definitely seem to imply that the team itself is actually disqualified if caught, and presumably can't plead ignorance or \"fall back\" on their 2nd submission.

\n

\"Submissions are void if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late. Competition Sponsor reserves the right to disqualify any entrant who does not follow these Rules, including making a Submission that does not meet the Requirements.\"

", "votes": 1, "replies": [] }, { "id": 1309461, "author_name": "mamas", "author_url": "", "post_date": "2021-05-16T01:30:54.907000", "content": "
What would help is if the hosts made it clear that anyone caught cheating will have BOTH submissions disqualified. At least this will prevent folks from submitting one illegal submission plus one legal one as a \"backup\" in case they get caught.\n
\n

I agree with you. I think if one of the final submissions breaks the rule, both final submissions should be disqualified, because kaggle admin clearly said using hand-labelled missing grid points are breaking the rule.

", "votes": 3, "replies": [] }, { "id": 1309472, "author_name": "mamas", "author_url": "", "post_date": "2021-05-16T01:46:35.297000", "content": "

From this thread, the effect of hand-labelled waypoint is 0.02 ~ 0.12, so it is a big problem especially for the teams that are in the borderline (the teams which score 2.7x ~ 2.9x and 3.9x ~ 4.1x and 4.2x ~ 4.4x)

", "votes": 3, "replies": [] }, { "id": 1309836, "author_name": "JQ", "author_url": "", "post_date": "2021-05-16T10:40:05.483000", "content": "

I have just changed my selected submissions.
\nMy best public score now goes from 4.082 to 4.143.
\nI joined the competition to learn from people and not to win any medal so I am going to be coherent with it. However, I must say it feels discouraging once you are in the medal zone to see how easy is to cheat.
\nI believe it is going to be impossible for the organizers to detect people who use some form of manual waypoints. For the public dataset though there are at least two possibilities:

\n
    \n
  1. If the submission comes from a Kernel then just check whether the manual waypoints were used or not.
    2. \n
  2. If the submission does not come from a Kernel then try to match submissions waypoints with waypoints coming from public manual waypoints.
    3. \n
\n

If the submission uses manual waypoints created by the user then I think it is impossible to detect it. The only possibility I see is to submit code instead of CSVs.

", "votes": 5, "replies": [] }, { "id": 1310041, "author_name": "", "author_url": "", "post_date": "2021-05-16T13:07:37.363000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1297323": "The kaggle admin @inversion said \n```\nThe guiding general principle is this: \"Would this method still be applicable to a completely new, unseen test set?\" If the answer is YES, then it is allowable. If the answer is, NO then it is not allowable.\n\nSo, in this case, hand labeling of grid points that would be used for Test traces would not be acceptable. An automated process to fill in grid points that would work for new locations would be acceptable.\n```\n[in this discussion](https://www.kaggle.com/c/indoor-location-navigation/discussion/235813#1297306), so, please don't use the [dataset](https://www.kaggle.com/saitodevel01/indoor-navigation-hand-labeled-waypoints). Of course, our team will never use the subs that use hand-labelled possible missing grid points for the final submissions. The valid sub scores ~~1.704~~ 1.728 now, while the sub with hand-labelled possible missing grid points scores 1.604. ", "1303310": "Reminder for all the participants: Please never forget to exclude hand-labelled waypoints from your code and NEVER choose the submission that uses hand-labelled waypoints for the final submissions!", "1303718": "I think it's impossible to verify whether or not participants have used these hand-labelled waypoints. Unfortunately, I think many will." } }