This is a synchronous kernel only competition, there are public test data and private test data.\nAnd as you know many competitors probed private data set distribution by using the technique like the one discussed in this thread.
\n\nBut I also saw someone complains and accuses that such kind of private data proving is cheating.\nOf course someone who created new account to do this should be banned.\nBut in my understanding, as long as using 5 submissions which is allocated to everyone, this does not violate any competition rules.
\n\nI should say it's not competitor's fault, but kaggle team's fault because it's structural problem of synchronous kernel only competition.(a kind of security hole)\nNo one want to submit and probe repeatedly, but unfortunately it can bring very useful\ninformation. I hope it will be fixed in next synchronous kernel only competition.
\n\nPlease let me know your opinion :)
", "messages": [ { "id": "622408", "postDate": "09/09/2019 16:00:39", "content": "This is a synchronous kernel only competition, there are public test data and private test data.\nAnd as you know many competitors probed private data set distribution by using the technique like the one discussed in this thread.
\n\nBut I also saw someone complains and accuses that such kind of private data proving is cheating.\nOf course someone who created new account to do this should be banned.\nBut in my understanding, as long as using 5 submissions which is allocated to everyone, this does not violate any competition rules.
\n\nI should say it's not competitor's fault, but kaggle team's fault because it's structural problem of synchronous kernel only competition.(a kind of security hole)\nNo one want to submit and probe repeatedly, but unfortunately it can bring very useful\ninformation. I hope it will be fixed in next synchronous kernel only competition.
\n\nPlease let me know your opinion :)
", "rawMarkdown": "This is a synchronous kernel only competition, there are public test data and private test data.\nAnd as you know many competitors probed private data set distribution by using the technique like the one discussed in [this thread](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/104692#latest-603729).\n\nBut I also saw someone complains and accuses that such kind of private data proving is cheating.\nOf course someone who created new account to do this should be banned.\nBut in my understanding, as long as using 5 submissions which is allocated to everyone, this does not violate any competition rules.\n\nI should say it's not competitor's fault, but kaggle team's fault because it's structural problem of synchronous kernel only competition.(a kind of security hole)\nNo one want to submit and probe repeatedly, but unfortunately it can bring very useful\ninformation. I hope it will be fixed in next synchronous kernel only competition.\n\nPlease let me know your opinion :)", "votes": null }, { "id": "622413", "postDate": "09/09/2019 16:10:41", "content": "I tried searching for rules regarding private data probing, couldn't find anything. Instead, I found out that many people have done so in the past competitions. In my opinion, it goes against the spirit of the competition and should be curbed.
", "rawMarkdown": "I tried searching for rules regarding private data probing, couldn't find anything. Instead, I found out that many people have done so in the past competitions. In my opinion, it goes against the spirit of the competition and should be curbed.", "votes": null }, { "id": "622416", "postDate": "09/09/2019 16:20:46", "content": "People have been doing this since the creation of kaggle. Don't know how you could enforce a rule forbidding it. I think the only way are old-school kernel only competitions.
", "rawMarkdown": "People have been doing this since the creation of kaggle. Don't know how you could enforce a rule forbidding it. I think the only way are old-school kernel only competitions.", "votes": null }, { "id": "622443", "postDate": "09/09/2019 16:54:49", "content": "I think probing private is important for this kind of competition as long as it is not forbidden, since we can no longer assume the private and public are separated randomly.
\n\nWhile I agree that keeping private test set 'private' is important to provide a rigorous machine learning model, but some basic information for the test set, for example, how the samples are collected, is necessary. I believe we have access to get this kind of information even in production environment.
", "rawMarkdown": "I think probing private is important for this kind of competition as long as it is not forbidden, since we can no longer assume the private and public are separated randomly.\n\nWhile I agree that keeping private test set 'private' is important to provide a rigorous machine learning model, but some basic information for the test set, for example, how the samples are collected, is necessary. I believe we have access to get this kind of information even in production environment.", "votes": null }, { "id": "622444", "postDate": "09/09/2019 16:55:14", "content": "It's a \"dark area\"
", "rawMarkdown": "It's a \"dark area\"", "votes": null }, { "id": "622445", "postDate": "09/09/2019 16:55:45", "content": "Maybe rerunning the two final selected kernels on the private set after the deadline? Don't know if there's any other way.
", "rawMarkdown": "Maybe rerunning the two final selected kernels on the private set after the deadline? Don't know if there's any other way.", "votes": null }, { "id": "622474", "postDate": "09/09/2019 17:44:44", "content": "That's how KO competitions worked before. It has other issues though.
", "rawMarkdown": "That's how KO competitions worked before. It has other issues though.", "votes": null }, { "id": "622480", "postDate": "09/09/2019 17:56:10", "content": "I forgot to write my proposal. One way to deal with this situation is not forbidding but less motivating by clarifying how public/private test set ware split.(ex. public/private test set was split to 50:50 by using stratified K fold). And there should be no meaning to change private and public distribution intentionally like this competition, isn't it??
", "rawMarkdown": "I forgot to write my proposal. One way to deal with this situation is not forbidding but less motivating by clarifying how public/private test set ware split.(ex. public/private test set was split to 50:50 by using stratified K fold). And there should be no meaning to change private and public distribution intentionally like this competition, isn't it??", "votes": null }, { "id": "622484", "postDate": "09/09/2019 17:59:57", "content": "Yes, I agree. It doesn't help at all to solve real problem.
", "rawMarkdown": "Yes, I agree. It doesn't help at all to solve real problem.", "votes": null }, { "id": "622487", "postDate": "09/09/2019 18:03:00", "content": "Yep, at first I thought old style is better because it's waste of computation resources. But after the end of competition, I changed my mind because I realized how good it is for my mental health!
", "rawMarkdown": "Yep, at first I thought old style is better because it's waste of computation resources. But after the end of competition, I changed my mind because I realized how good it is for my mental health!", "votes": null }, { "id": "622490", "postDate": "09/09/2019 18:08:21", "content": "Yes, it's actually very important to know how private test set is especially when train/test split looks crazy like this competition....\nI agree with you, if kaggle provide a little bit more information about public/private split, it will be fairer, I believe.
", "rawMarkdown": "Yes, it's actually very important to know how private test set is especially when train/test split looks crazy like this competition....\nI agree with you, if kaggle provide a little bit more information about public/private split, it will be fairer, I believe.", "votes": null }, { "id": "622491", "postDate": "09/09/2019 18:11:12", "content": "Yes, this kind of effort that does not help to solve real problem should not be need to complete...
", "rawMarkdown": "Yes, this kind of effort that does not help to solve real problem should not be need to complete...", "votes": null }, { "id": "622518", "postDate": "09/09/2019 19:02:48", "content": "As @philippsinger mentioned, probing private has long history in kaggle. Actually people collaborated to probe the private class distribution, for example. I think it is a very useful strategy to prevent shake between public/private.
\n\nBTW, I have no idea why people downvote my original post...
", "rawMarkdown": "As @philippsinger mentioned, probing private has long history in kaggle. Actually people collaborated to probe the private class distribution, for [example](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#396205). I think it is a very useful strategy to prevent shake between public/private. \n\nBTW, I have no idea why people downvote my original post...", "votes": null }, { "id": "622656", "postDate": "09/09/2019 23:55:55", "content": "That is one of the reason why I made this thread. It has been done for a long time and it seems inevitable, but still someone accuses who did it. I think it's very natural behavior for competitor because everyone want to win. I understand if he/she complains to kaggle, but sometimes not...
", "rawMarkdown": "That is one of the reason why I made this thread. It has been done for a long time and it seems inevitable, but still someone accuses who did it. I think it's very natural behavior for competitor because everyone want to win. I understand if he/she complains to kaggle, but sometimes not...", "votes": null }, { "id": "622854", "postDate": "09/10/2019 06:59:34", "content": "Some combination of both would be maybe nice. People did fast submissions here anyways mostly. So for example allow upload-file submissions until like 1 week before deadline and only then switch to kernel submissions. That would also save a lot of unnecessary resources.
", "rawMarkdown": "Some combination of both would be maybe nice. People did fast submissions here anyways mostly. So for example allow upload-file submissions until like 1 week before deadline and only then switch to kernel submissions. That would also save a lot of unnecessary resources.", "votes": null }, { "id": "622875", "postDate": "09/10/2019 07:34:06", "content": "cheat to win?
", "rawMarkdown": "cheat to win?", "votes": null }, { "id": "623194", "postDate": "09/10/2019 15:29:54", "content": "I don't think it's cheating for now, unfortunately. I understand your feeling and I agree it's against the spirit, waste of time and useless for solving real problem, but currently the rule doesn't say anything about that.\nUnlike the private sharing, I think it can be prevented by host because it's somehow due to their design of competition. I hope it will be solved by improving the system itself, not just relying on competitor's dignity.
", "rawMarkdown": "I don't think it's cheating for now, unfortunately. I understand your feeling and I agree it's against the spirit, waste of time and useless for solving real problem, but currently the rule doesn't say anything about that.\nUnlike the private sharing, I think it can be prevented by host because it's somehow due to their design of competition. I hope it will be solved by improving the system itself, not just relying on competitor's dignity.", "votes": null } ], "comments": [ { "id": 622413, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/09/2019 16:10:41", "content": "I tried searching for rules regarding private data probing, couldn't find anything. Instead, I found out that many people have done so in the past competitions. In my opinion, it goes against the spirit of the competition and should be curbed.
", "votes": null, "replies": [ { "id": 622484, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 17:59:57", "content": "Yes, I agree. It doesn't help at all to solve real problem.
", "votes": null, "replies": [] } ] }, { "id": 622416, "author_name": "philippsinger", "author_url": "", "post_date": "09/09/2019 16:20:46", "content": "People have been doing this since the creation of kaggle. Don't know how you could enforce a rule forbidding it. I think the only way are old-school kernel only competitions.
", "votes": null, "replies": [ { "id": 622445, "author_name": "rishabhiitbhu", "author_url": "", "post_date": "09/09/2019 16:55:45", "content": "Maybe rerunning the two final selected kernels on the private set after the deadline? Don't know if there's any other way.
", "votes": null, "replies": [] }, { "id": 622474, "author_name": "philippsinger", "author_url": "", "post_date": "09/09/2019 17:44:44", "content": "That's how KO competitions worked before. It has other issues though.
", "votes": null, "replies": [] }, { "id": 622487, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 18:03:00", "content": "Yep, at first I thought old style is better because it's waste of computation resources. But after the end of competition, I changed my mind because I realized how good it is for my mental health!
", "votes": null, "replies": [] }, { "id": 622854, "author_name": "philippsinger", "author_url": "", "post_date": "09/10/2019 06:59:34", "content": "Some combination of both would be maybe nice. People did fast submissions here anyways mostly. So for example allow upload-file submissions until like 1 week before deadline and only then switch to kernel submissions. That would also save a lot of unnecessary resources.
", "votes": null, "replies": [] } ] }, { "id": 622443, "author_name": "naivelamb", "author_url": "", "post_date": "09/09/2019 16:54:49", "content": "I think probing private is important for this kind of competition as long as it is not forbidden, since we can no longer assume the private and public are separated randomly.
\n\nWhile I agree that keeping private test set 'private' is important to provide a rigorous machine learning model, but some basic information for the test set, for example, how the samples are collected, is necessary. I believe we have access to get this kind of information even in production environment.
", "votes": null, "replies": [ { "id": 622490, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 18:08:21", "content": "Yes, it's actually very important to know how private test set is especially when train/test split looks crazy like this competition....\nI agree with you, if kaggle provide a little bit more information about public/private split, it will be fairer, I believe.
", "votes": null, "replies": [] }, { "id": 622518, "author_name": "naivelamb", "author_url": "", "post_date": "09/09/2019 19:02:48", "content": "As @philippsinger mentioned, probing private has long history in kaggle. Actually people collaborated to probe the private class distribution, for example. I think it is a very useful strategy to prevent shake between public/private.
\n\nBTW, I have no idea why people downvote my original post...
", "votes": null, "replies": [] }, { "id": 622656, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 23:55:55", "content": "That is one of the reason why I made this thread. It has been done for a long time and it seems inevitable, but still someone accuses who did it. I think it's very natural behavior for competitor because everyone want to win. I understand if he/she complains to kaggle, but sometimes not...
", "votes": null, "replies": [] }, { "id": 622875, "author_name": "abhishek", "author_url": "", "post_date": "09/10/2019 07:34:06", "content": "cheat to win?
", "votes": null, "replies": [] }, { "id": 623194, "author_name": "bamps53", "author_url": "", "post_date": "09/10/2019 15:29:54", "content": "I don't think it's cheating for now, unfortunately. I understand your feeling and I agree it's against the spirit, waste of time and useless for solving real problem, but currently the rule doesn't say anything about that.\nUnlike the private sharing, I think it can be prevented by host because it's somehow due to their design of competition. I hope it will be solved by improving the system itself, not just relying on competitor's dignity.
", "votes": null, "replies": [] } ] }, { "id": 622444, "author_name": "abhishek", "author_url": "", "post_date": "09/09/2019 16:55:14", "content": "It's a \"dark area\"
", "votes": null, "replies": [ { "id": 622491, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 18:11:12", "content": "Yes, this kind of effort that does not help to solve real problem should not be need to complete...
", "votes": null, "replies": [] } ] }, { "id": 622480, "author_name": "bamps53", "author_url": "", "post_date": "09/09/2019 17:56:10", "content": "I forgot to write my proposal. One way to deal with this situation is not forbidding but less motivating by clarifying how public/private test set ware split.(ex. public/private test set was split to 50:50 by using stratified K fold). And there should be no meaning to change private and public distribution intentionally like this competition, isn't it??
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "622408": "This is a synchronous kernel only competition, there are public test data and private test data.\nAnd as you know many competitors probed private data set distribution by using the technique like the one discussed in [this thread](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/104692#latest-603729).\n\nBut I also saw someone complains and accuses that such kind of private data proving is cheating.\nOf course someone who created new account to do this should be banned.\nBut in my understanding, as long as using 5 submissions which is allocated to everyone, this does not violate any competition rules.\n\nI should say it's not competitor's fault, but kaggle team's fault because it's structural problem of synchronous kernel only competition.(a kind of security hole)\nNo one want to submit and probe repeatedly, but unfortunately it can bring very useful\ninformation. I hope it will be fixed in next synchronous kernel only competition.\n\nPlease let me know your opinion :)", "622413": "I tried searching for rules regarding private data probing, couldn't find anything. Instead, I found out that many people have done so in the past competitions. In my opinion, it goes against the spirit of the competition and should be curbed.", "622416": "People have been doing this since the creation of kaggle. Don't know how you could enforce a rule forbidding it. I think the only way are old-school kernel only competitions.", "622443": "I think probing private is important for this kind of competition as long as it is not forbidden, since we can no longer assume the private and public are separated randomly.\n\nWhile I agree that keeping private test set 'private' is important to provide a rigorous machine learning model, but some basic information for the test set, for example, how the samples are collected, is necessary. I believe we have access to get this kind of information even in production environment.", "622444": "It's a \"dark area\"", "622445": "Maybe rerunning the two final selected kernels on the private set after the deadline? Don't know if there's any other way.", "622474": "That's how KO competitions worked before. It has other issues though.", "622480": "I forgot to write my proposal. One way to deal with this situation is not forbidding but less motivating by clarifying how public/private test set ware split.(ex. public/private test set was split to 50:50 by using stratified K fold). And there should be no meaning to change private and public distribution intentionally like this competition, isn't it??", "622484": "Yes, I agree. It doesn't help at all to solve real problem.", "622487": "Yep, at first I thought old style is better because it's waste of computation resources. But after the end of competition, I changed my mind because I realized how good it is for my mental health!", "622490": "Yes, it's actually very important to know how private test set is especially when train/test split looks crazy like this competition....\nI agree with you, if kaggle provide a little bit more information about public/private split, it will be fairer, I believe.", "622491": "Yes, this kind of effort that does not help to solve real problem should not be need to complete...", "622518": "As @philippsinger mentioned, probing private has long history in kaggle. Actually people collaborated to probe the private class distribution, for [example](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#396205). I think it is a very useful strategy to prevent shake between public/private. \n\nBTW, I have no idea why people downvote my original post...", "622656": "That is one of the reason why I made this thread. It has been done for a long time and it seems inevitable, but still someone accuses who did it. I think it's very natural behavior for competitor because everyone want to win. I understand if he/she complains to kaggle, but sometimes not...", "622854": "Some combination of both would be maybe nice. People did fast submissions here anyways mostly. So for example allow upload-file submissions until like 1 week before deadline and only then switch to kernel submissions. That would also save a lot of unnecessary resources.", "622875": "cheat to win?", "623194": "I don't think it's cheating for now, unfortunately. I understand your feeling and I agree it's against the spirit, waste of time and useless for solving real problem, but currently the rule doesn't say anything about that.\nUnlike the private sharing, I think it can be prevented by host because it's somehow due to their design of competition. I hope it will be solved by improving the system itself, not just relying on competitor's dignity." }, "source": "meta" }