![\"lower](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F728436%2F0ba9302174257b80c027b61b70a2a63d%2Fprivate_test_prediction_upperbound.png?generation=1567780509125169&alt=media\")
\n![\"upper](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F728436%2F195289b414eadcc0c4dee56310ef9cec%2Fprivate_test_prediction_upperbound.png?generation=1567780525902792&alt=media\")
Hellow kagglers, I'm a researcher working on optical image analyses including macula-centered, wide fundus and OCT. Personally, I'm a newbie to kaggle and this is my first competition in kaggle. But I participated in many other fundus-related competitions with team name VRT (VUNO RETINA TEAM) (IDRiD, REFUGE, PALM)
\n\nIn fact, I made a second account few days ago to probe private test dataset. It was motivated by \nthis kernel. Basically, the idea is to run inference on the private test data with a reliable model and tally the distribution of grades and finally replace labels for public test data with submissions of which kappa values are known based on the distribution.
\n\nThis is the detail. First, you bound the ratio of each grade from 0 to 1 and check whether the ratio for each grade is above or below 0.5. By the first submission, you get to know which half the ratio of each grade falls into through public weighted kappa. In the second round, you start with narrower bounds and keep going. By only 5 submissions, you can bound the distributon within the margin of 0.03125. Of course, in order to probe reliably, you should have a reliable model to inference on the private test dataset. I used a single model with qwk~0.84. You can see the attached codes to completely grasp how I did it. With the second account, I made about 20 submissions, 10 of which evaporated for debugging and 5 of which were successful for probing. For about 5 submissions, I tried to probe the image size but I noticed that making the second account results in dropout from the competition. I got scared and deleted the account right away (no concrete records left for that but I dimly remember that (1536, 2048), (1746, 2416), (1958, 2588), (2136,3216) take up about 25%).
\n\nHere, I share the gist of what I have figured out just to be fairer.
\n\nFollowings are the lower bound and the upper bound of label distributions with a single reliable model (qwk~0.84). I assumed that the number of the private test dataset amounts to 13,000.
\n\n\n
I know it's too late. I have no words for that. I may get kicked out of the competition for making the second account and probing the private test dataset. However, I'm publishing this post hoping for our company VUNO INC. not to get tarnished by my misbehavior.
\n\nBy the time I created a second account, I was overwhelmed by thoughts that others would do probe the private test dataset or have already finished probing the private dataset. I knew it was illegal but couldn't help doing it when I thought I might get fall behind. It's my fault not to follow discussion threads which warn enough about the harsh consequences of the fraudulent behavior.
\n\nHowever, our team did not make many submissions so far (139 submissions). The total amount of submissions is far below 200 even though submissions from the second account were added. :) Also, we share the information we have known with the second account with all kaggler :). We hope to stick to the competition. :)
", "messages": [ { "id": "619769", "postDate": "09/06/2019 14:56:29", "content": "Hellow kagglers, I'm a researcher working on optical image analyses including macula-centered, wide fundus and OCT. Personally, I'm a newbie to kaggle and this is my first competition in kaggle. But I participated in many other fundus-related competitions with team name VRT (VUNO RETINA TEAM) (IDRiD, REFUGE, PALM)
\n\nIn fact, I made a second account few days ago to probe private test dataset. It was motivated by \nthis kernel. Basically, the idea is to run inference on the private test data with a reliable model and tally the distribution of grades and finally replace labels for public test data with submissions of which kappa values are known based on the distribution.
\n\nThis is the detail. First, you bound the ratio of each grade from 0 to 1 and check whether the ratio for each grade is above or below 0.5. By the first submission, you get to know which half the ratio of each grade falls into through public weighted kappa. In the second round, you start with narrower bounds and keep going. By only 5 submissions, you can bound the distributon within the margin of 0.03125. Of course, in order to probe reliably, you should have a reliable model to inference on the private test dataset. I used a single model with qwk~0.84. You can see the attached codes to completely grasp how I did it. With the second account, I made about 20 submissions, 10 of which evaporated for debugging and 5 of which were successful for probing. For about 5 submissions, I tried to probe the image size but I noticed that making the second account results in dropout from the competition. I got scared and deleted the account right away (no concrete records left for that but I dimly remember that (1536, 2048), (1746, 2416), (1958, 2588), (2136,3216) take up about 25%).
\n\nHere, I share the gist of what I have figured out just to be fairer.
\n\nFollowings are the lower bound and the upper bound of label distributions with a single reliable model (qwk~0.84). I assumed that the number of the private test dataset amounts to 13,000.
\n\n\n
I know it's too late. I have no words for that. I may get kicked out of the competition for making the second account and probing the private test dataset. However, I'm publishing this post hoping for our company VUNO INC. not to get tarnished by my misbehavior.
\n\nBy the time I created a second account, I was overwhelmed by thoughts that others would do probe the private test dataset or have already finished probing the private dataset. I knew it was illegal but couldn't help doing it when I thought I might get fall behind. It's my fault not to follow discussion threads which warn enough about the harsh consequences of the fraudulent behavior.
\n\nHowever, our team did not make many submissions so far (139 submissions). The total amount of submissions is far below 200 even though submissions from the second account were added. :) Also, we share the information we have known with the second account with all kaggler :). We hope to stick to the competition. :)
", "rawMarkdown": "Hellow kagglers, I'm a researcher working on optical image analyses including macula-centered, wide fundus and OCT. Personally, I'm a newbie to kaggle and this is my first competition in kaggle. But I participated in many other fundus-related competitions with team name VRT (VUNO RETINA TEAM) ([IDRiD](https://idrid.grand-challenge.org/), [REFUGE](https://refuge.grand-challenge.org/), [PALM](https://palm.grand-challenge.org/))\n\nIn fact, I made a second account few days ago to probe private test dataset. It was motivated by \n[this kernel](https://www.kaggle.com/ren4yu/aptos-2019-probing-private-test-image-sizes). Basically, the idea is to run inference on the private test data with a reliable model and tally the distribution of grades and finally replace labels for public test data with submissions of which kappa values are known based on the distribution. \n\nThis is the detail. First, you bound the ratio of each grade from 0 to 1 and check whether the ratio for each grade is above or below 0.5. By the first submission, you get to know which half the ratio of each grade falls into through public weighted kappa. In the second round, you start with narrower bounds and keep going. By only 5 submissions, you can bound the distributon within the margin of 0.03125. Of course, in order to probe reliably, you should have a reliable model to inference on the private test dataset. I used a single model with qwk~0.84. You can see the attached codes to completely grasp how I did it. With the second account, I made about 20 submissions, 10 of which evaporated for debugging and 5 of which were successful for probing. For about 5 submissions, I tried to probe the image size but I noticed that making the second account results in dropout from the competition. I got scared and deleted the account right away (no concrete records left for that but I dimly remember that (1536, 2048), (1746, 2416), (1958, 2588), (2136,3216) take up about 25%).\n\nHere, I share the gist of what I have figured out just to be fairer. \n### The private test data would be very similar to the public training dataset.\nFollowings are the lower bound and the upper bound of label distributions with a single reliable model (qwk~0.84). I assumed that the number of the private test dataset amounts to 13,000.\n\n\n\n\nI know it's too late. I have no words for that. I may get kicked out of the competition for making the second account and probing the private test dataset. However, I'm publishing this post hoping for our company VUNO INC. not to get tarnished by my misbehavior.\n\nBy the time I created a second account, I was overwhelmed by thoughts that others would do probe the private test dataset or have already finished probing the private dataset. I knew it was illegal but couldn't help doing it when I thought I might get fall behind. It's my fault not to follow discussion threads which warn enough about the harsh consequences of the fraudulent behavior.\n\nHowever, our team did not make many submissions so far (139 submissions). The total amount of submissions is far below 200 even though submissions from the second account were added. :) Also, we share the information we have known with the second account with all kaggler :). We hope to stick to the competition. :)", "votes": null }, { "id": "619772", "postDate": "09/06/2019 14:58:17", "content": "** I made a second account **
\n\nThis is against the rules if i understand correctly...
", "rawMarkdown": "** I made a second account **\n\n\nThis is against the rules if i understand correctly...", "votes": null }, { "id": "619774", "postDate": "09/06/2019 15:01:26", "content": "Yes, you're right. No words for that. I did not know much about the consequences of that behavior. It's a shame to get kicked out from the leader board. To be less shameful, I'm sharing the information I gained from the second account.
", "rawMarkdown": "Yes, you're right. No words for that. I did not know much about the consequences of that behavior. It's a shame to get kicked out from the leader board. To be less shameful, I'm sharing the information I gained from the second account.", "votes": null }, { "id": "619778", "postDate": "09/06/2019 15:03:26", "content": "Also you could have posted this information with python scripts after competition ends...
", "rawMarkdown": "Also you could have posted this information with python scripts after competition ends...", "votes": null }, { "id": "619781", "postDate": "09/06/2019 15:09:25", "content": "You can see the python scripts. I did not know I'm going to get kicked out of the competition if I make two accounts. Before getting deleted, I'm sharing the information. In fact, I'm a bit confused of defition of \"multiple accounts\". If I make 5 accounts from the start and merge the team, is it still legal?
", "rawMarkdown": "You can see the python scripts. I did not know I'm going to get kicked out of the competition if I make two accounts. Before getting deleted, I'm sharing the information. In fact, I'm a bit confused of defition of \"multiple accounts\". If I make 5 accounts from the start and merge the team, is it still legal?", "votes": null }, { "id": "619783", "postDate": "09/06/2019 15:11:24", "content": "@jaeminkkk I appreciate your honesty, but I'm closing this post until after the competition has closed to limit the distribution of the python scripts.
", "rawMarkdown": "jaeminkkk I appreciate your honesty, but I'm closing this post until after the competition has closed to limit the distribution of the python scripts.", "votes": null }, { "id": "620778", "postDate": "09/08/2019 00:46:25", "content": "This is interesting work and I don't think this would have been illegal if you didn't use a second account to make submissions. Anyway, I hope your hard work does not go to waste.
", "rawMarkdown": "This is interesting work and I don't think this would have been illegal if you didn't use a second account to make submissions. Anyway, I hope your hard work does not go to waste.", "votes": null }, { "id": "620932", "postDate": "09/08/2019 04:36:21", "content": "I did not use the second account for testing my models... In fact, I didn't have to make a new account to probe the private test set... I left several daily quotas in the last days LoL So regretful!
", "rawMarkdown": "I did not use the second account for testing my models... In fact, I didn't have to make a new account to probe the private test set... I left several daily quotas in the last days LoL So regretful!", "votes": null }, { "id": "621363", "postDate": "09/08/2019 13:16:48", "content": "@jaeminkkk, you are 43rd on this competition right now before cheaters accounts are removed.
\n\n\n\n\nI knew it was illegal but couldn't help doing it when I thought I might get fall behind.
\n
It is always a good idea to read the rules before joining any competition. And if you think about doing something so obvious as LB probing, perhaps you should pause and ask yourself what if everyone does that? Common sense usually can guide one on the right path.
", "rawMarkdown": "jaeminkkk, you are 43rd on this competition right now before cheaters accounts are removed.\n\n> I knew it was illegal but couldn't help doing it when I thought I might get fall behind. \n\nIt is always a good idea to read the rules before joining any competition. And if you think about doing something so obvious as LB probing, perhaps you should pause and ask yourself what if everyone does that? Common sense usually can guide one on the right path.", "votes": null }, { "id": "621519", "postDate": "09/08/2019 16:03:33", "content": "You're right. I was too blind at that time. Making a second ID was a worse choice. I saw top rankers still make active submissions even though qwk seems to saturate for the public test set which mainly consists of grade 2. With simple simulations, you can simply know that improving kappa when one grade is largely dominant is difficult and it would lead to overfitting (which I suffered). Then, I thought it could be a private test probing and I lost my mind. Or that would be just my misconceptions. I learned some lessons though.
", "rawMarkdown": "You're right. I was too blind at that time. Making a second ID was a worse choice. I saw top rankers still make active submissions even though qwk seems to saturate for the public test set which mainly consists of grade 2. With simple simulations, you can simply know that improving kappa when one grade is largely dominant is difficult and it would lead to overfitting (which I suffered). Then, I thought it could be a private test probing and I lost my mind. Or that would be just my misconceptions. I learned some lessons though.", "votes": null }, { "id": "621613", "postDate": "09/08/2019 18:04:14", "content": "No problem @jaeminkkk, we all make mistakes. Be careful next time even when merging with people, ask question about the rules. One person in a team breaking the rule leads to a painful team disqualification. See you in the next one :)
", "rawMarkdown": "No problem @jaeminkkk, we all make mistakes. Be careful next time even when merging with people, ask question about the rules. One person in a team breaking the rule leads to a painful team disqualification. See you in the next one :)", "votes": null }, { "id": "621652", "postDate": "09/08/2019 19:49:08", "content": "If I understand it correctly, this post was published 2 days before the competition ends, it was seen by some of the competitors. It was closed at that time and then released after the competitions ends by @sohier ?
", "rawMarkdown": "If I understand it correctly, this post was published 2 days before the competition ends, it was seen by some of the competitors. It was closed at that time and then released after the competitions ends by @sohier ?", "votes": null }, { "id": "621661", "postDate": "09/08/2019 20:07:44", "content": "Yes, I saw this too 2 days ago. And at that time we were 5th place and they were 4th, I was really about to open it up.....😈
", "rawMarkdown": "Yes, I saw this too 2 days ago. And at that time we were 5th place and they were 4th, I was really about to open it up.....😈", "votes": null }, { "id": "621694", "postDate": "09/08/2019 20:44:54", "content": "@seesee \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97652\nHave fun reconstructing the private test images one bit at a time! This is no leakage.\nHave you changed your mind? :)
I have a complicated feeling about it now...
", "rawMarkdown": "I have a complicated feeling about it now...", "votes": null }, { "id": "622384", "postDate": "09/09/2019 15:20:00", "content": "Yes, multiple accounts are banned regardless of whether they end up on the same team.
\n\nI just happened to double check the rules page for something else and that turns out to be literally the first rule.
", "rawMarkdown": "Yes, multiple accounts are banned regardless of whether they end up on the same team.\n\nI just happened to double check the rules page for something else and that turns out to be literally the first rule.", "votes": null } ], "comments": [ { "id": 619772, "author_name": "drhabib", "author_url": "", "post_date": "09/06/2019 14:58:17", "content": "** I made a second account **
\n\nThis is against the rules if i understand correctly...
", "votes": null, "replies": [ { "id": 619774, "author_name": "jaeminkkk", "author_url": "", "post_date": "09/06/2019 15:01:26", "content": "Yes, you're right. No words for that. I did not know much about the consequences of that behavior. It's a shame to get kicked out from the leader board. To be less shameful, I'm sharing the information I gained from the second account.
", "votes": null, "replies": [] }, { "id": 619778, "author_name": "drhabib", "author_url": "", "post_date": "09/06/2019 15:03:26", "content": "Also you could have posted this information with python scripts after competition ends...
", "votes": null, "replies": [] }, { "id": 619781, "author_name": "jaeminkkk", "author_url": "", "post_date": "09/06/2019 15:09:25", "content": "You can see the python scripts. I did not know I'm going to get kicked out of the competition if I make two accounts. Before getting deleted, I'm sharing the information. In fact, I'm a bit confused of defition of \"multiple accounts\". If I make 5 accounts from the start and merge the team, is it still legal?
", "votes": null, "replies": [] }, { "id": 622384, "author_name": "sohier", "author_url": "", "post_date": "09/09/2019 15:20:00", "content": "Yes, multiple accounts are banned regardless of whether they end up on the same team.
\n\nI just happened to double check the rules page for something else and that turns out to be literally the first rule.
", "votes": null, "replies": [] } ] }, { "id": 619783, "author_name": "sohier", "author_url": "", "post_date": "09/06/2019 15:11:24", "content": "@jaeminkkk I appreciate your honesty, but I'm closing this post until after the competition has closed to limit the distribution of the python scripts.
", "votes": null, "replies": [] }, { "id": 620778, "author_name": "tanlikesmath", "author_url": "", "post_date": "09/08/2019 00:46:25", "content": "This is interesting work and I don't think this would have been illegal if you didn't use a second account to make submissions. Anyway, I hope your hard work does not go to waste.
", "votes": null, "replies": [ { "id": 620932, "author_name": "jaeminkkk", "author_url": "", "post_date": "09/08/2019 04:36:21", "content": "I did not use the second account for testing my models... In fact, I didn't have to make a new account to probe the private test set... I left several daily quotas in the last days LoL So regretful!
", "votes": null, "replies": [] } ] }, { "id": 621363, "author_name": "sheriytm", "author_url": "", "post_date": "09/08/2019 13:16:48", "content": "@jaeminkkk, you are 43rd on this competition right now before cheaters accounts are removed.
\n\n\n\n\nI knew it was illegal but couldn't help doing it when I thought I might get fall behind.
\n
It is always a good idea to read the rules before joining any competition. And if you think about doing something so obvious as LB probing, perhaps you should pause and ask yourself what if everyone does that? Common sense usually can guide one on the right path.
", "votes": null, "replies": [ { "id": 621519, "author_name": "jaeminkkk", "author_url": "", "post_date": "09/08/2019 16:03:33", "content": "You're right. I was too blind at that time. Making a second ID was a worse choice. I saw top rankers still make active submissions even though qwk seems to saturate for the public test set which mainly consists of grade 2. With simple simulations, you can simply know that improving kappa when one grade is largely dominant is difficult and it would lead to overfitting (which I suffered). Then, I thought it could be a private test probing and I lost my mind. Or that would be just my misconceptions. I learned some lessons though.
", "votes": null, "replies": [] }, { "id": 621613, "author_name": "sheriytm", "author_url": "", "post_date": "09/08/2019 18:04:14", "content": "No problem @jaeminkkk, we all make mistakes. Be careful next time even when merging with people, ask question about the rules. One person in a team breaking the rule leads to a painful team disqualification. See you in the next one :)
", "votes": null, "replies": [] } ] }, { "id": 621652, "author_name": "naivelamb", "author_url": "", "post_date": "09/08/2019 19:49:08", "content": "If I understand it correctly, this post was published 2 days before the competition ends, it was seen by some of the competitors. It was closed at that time and then released after the competitions ends by @sohier ?
", "votes": null, "replies": [ { "id": 621661, "author_name": "bamps53", "author_url": "", "post_date": "09/08/2019 20:07:44", "content": "Yes, I saw this too 2 days ago. And at that time we were 5th place and they were 4th, I was really about to open it up.....😈
", "votes": null, "replies": [] }, { "id": 621788, "author_name": "naivelamb", "author_url": "", "post_date": "09/09/2019 00:32:02", "content": "I have a complicated feeling about it now...
", "votes": null, "replies": [] } ] }, { "id": 621694, "author_name": "artyomp", "author_url": "", "post_date": "09/08/2019 20:44:54", "content": "@seesee \nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/97652\nHave fun reconstructing the private test images one bit at a time! This is no leakage.\nHave you changed your mind? :)