![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F644036%2Ff7fc749a373c3ba12da61593d5af3697%2FScreenshot%20from%202020-08-13%2018-17-40.png?generation=1597357117685509&alt=media\")
I can only speculate, but telling from the number of submissions, they just intensively probed image per image on the LB to label the test images which are part of the public test set… Whats the point? While you can use the probed labels as additional training data and probably get a better score, how does this help anyone but your own ego?
", "messages": [ { "id": 970111, "postDate": "2020-08-14T08:03:07.497Z", "content": "Hi, please let me clarify as much as I can now. That result is based on LB probing, it comprises many probing submissions, each of the probing submission scores is in range 0.4-0.6 that is why it was not visible before. This submission will be very bad for the private LB, it is basically very overfited to the public. I plan to publish more details on the procedure when the competition ends. I submitted it now to showcase what is possible to achieve with the probing. We didn't submit it earlier in order not to disrupt the leaderboard.
\nMy motivation: I like solving tasks with mixed integer programming, I worked with it at IBM many years ago and I practise it when appropriate since then. I try to distill as much information as possible from each submission, it is challenging and interesting.
\nI understand that the area is a gray area, so I try to maintain full transparency, specifically on how we use it in our final solutions, if at all. Will be glad to share more when the competition ends.
\nGood luck to you in the last few days!
", "rawMarkdown": "Hi, please let me clarify as much as I can now. That result is based on LB probing, it comprises many probing submissions, each of the probing submission scores is in range 0.4-0.6 that is why it was not visible before. This submission will be very bad for the private LB, it is basically very overfited to the public. I plan to publish more details on the procedure when the competition ends. I submitted it now to showcase what is possible to achieve with the probing. We didn't submit it earlier in order not to disrupt the leaderboard.\n\nMy motivation: I like solving tasks with mixed integer programming, I worked with it at IBM many years ago and I practise it when appropriate since then. I try to distill as much information as possible from each submission, it is challenging and interesting.\n\nI understand that the area is a gray area, so I try to maintain full transparency, specifically on how we use it in our final solutions, if at all. Will be glad to share more when the competition ends.\n\nGood luck to you in the last few days!", "votes": 33, "replies": [ { "id": 970163, "postDate": "2020-08-14T09:00:23.573Z", "content": "Thanks for clarifying.
", "rawMarkdown": "Thanks for clarifying." }, { "id": 970217, "postDate": "2020-08-14T09:31:13.457Z", "content": "\n\nI like solving tasks with mixed integer programming,
\n
I love it. Thanks for clarifying.
\nDid you use CPLEX while at IBM? That's what I was working on as you probably know.
", "rawMarkdown": "> I like solving tasks with mixed integer programming,\n\nI love it. Thanks for clarifying.\n\nDid you use CPLEX while at IBM? That's what I was working on as you probably know.", "votes": 2 }, { "id": 970300, "postDate": "2020-08-14T10:46:29.843Z", "content": "Yes, I remember you also participated in the latest Santa competition. At IBM we used CPLEX, of course, but during the Santa competition it became clear to us that Gurobi worked better there, as reported also by many other people. So I switched to Gurobi, and here also I used Gurobi.
", "rawMarkdown": "Yes, I remember you also participated in the latest Santa competition. At IBM we used CPLEX, of course, but during the Santa competition it became clear to us that Gurobi worked better there, as reported also by many other people. So I switched to Gurobi, and here also I used Gurobi.", "votes": 1 }, { "id": 970358, "postDate": "2020-08-14T11:36:50.890Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 970359, "postDate": "2020-08-14T11:37:27.910Z", "content": "That's already known. 77 or 78 positive images.
", "rawMarkdown": "[That's already known](https://www.kaggle.com/cpmpml/number-of-public-melanoma-is-78-or-77). 77 or 78 positive images.", "votes": 3 }, { "id": 970374, "postDate": "2020-08-14T11:46:48.433Z", "rawMarkdown": "", "isDeleted": true }, { "id": 970440, "postDate": "2020-08-14T13:06:02.497Z", "content": "It is the best notebook ever.
", "rawMarkdown": "It is the best notebook ever.", "votes": 8 } ] }, { "id": 973382, "postDate": "2020-08-17T09:20:57.800Z", "content": "@zaharch was also the one who had an impressive public LB score in LANL Earthquake Prediction not long ago. He did some smart optimization to know public rows of submissions.
", "rawMarkdown": "@zaharch was also the one who had an impressive public LB score in LANL Earthquake Prediction not long ago. He did some smart optimization to know public rows of submissions. " } ] }, { "id": 969690, "postDate": "2020-08-13T21:13:51.307Z", "content": "I can only speculate, but telling from the number of submissions, they just intensively probed image per image on the LB to label the test images which are part of the public test set… Whats the point? While you can use the probed labels as additional training data and probably get a better score, how does this help anyone but your own ego?
", "rawMarkdown": "I can only speculate, but telling from the number of submissions, they just intensively probed image per image on the LB to label the test images which are part of the public test set... Whats the point? While you can use the probed labels as additional training data and probably get a better score, how does this help anyone but your own ego?", "votes": 37 }, { "id": 969722, "postDate": "2020-08-13T22:09:23.657Z", "content": "I don't think we should make any kind of allegations or comments without knowing their side of the story.
\nIf it's probing it has been done already by a few this competition.
\nAs long as the private solution is clean, who cares.
Totally agreeing with you
", "rawMarkdown": "Totally agreeing with you" } ] }, { "id": 969719, "postDate": "2020-08-13T22:02:13.193Z", "content": "Kaggle should make this one a kernel competition.
", "rawMarkdown": "Kaggle should make this one a kernel competition.", "votes": 13, "replies": [ { "id": 969731, "postDate": "2020-08-13T22:22:27.067Z", "content": "\n\nKaggle should make this one a kernel competition.
\n
Or 2 stages competition with 2nd stage test set available just a week before the end. That would still allow internet use (for TPU for instance)
\nThere used to be many 2 stages CV competitions before on kaggle( in order to prevent handlabelling among others). I don't why that's not the case anymore.
Two of these things are not like the others 🤔
\n
Great plot. In the first month of the comp, Yuval was in 1st place using a single model that scored LB 0.960 explained here. After Sirish took first, Sirish explained what he did to climb LB here. Then NoSound joined Yuval and their LB did not increase for 22 days until today when it jumped up to match Sirish's.
", "rawMarkdown": "Great plot. In the first month of the comp, Yuval was in 1st place using a single model that scored LB 0.960 explained [here][2]. After Sirish took first, Sirish explained what he did to climb LB [here][1]. Then NoSound joined Yuval and their LB did not increase for 22 days until today when it jumped up to match Sirish's.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161497#901185\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168152#939957", "votes": 7 }, { "id": 969760, "postDate": "2020-08-13T23:03:35.020Z", "content": "\n\ntheir LB did not increase for 22 days until today
\n
It shows LB is not required to measure progress.
", "rawMarkdown": "> their LB did not increase for 22 days until today\n\nIt shows LB is not required to measure progress.", "votes": 7 }, { "id": 970024, "postDate": "2020-08-14T06:20:43.227Z", "content": "How many subs during the no-progress period?
", "rawMarkdown": "How many subs during the no-progress period?" }, { "id": 970031, "postDate": "2020-08-14T06:25:21.370Z", "content": "I forget. Can we check that in meta Kaggle dataset? If I remember correctly, i think they did about 5 a day, so about 100 subs.
", "rawMarkdown": "I forget. Can we check that in meta Kaggle dataset? If I remember correctly, i think they did about 5 a day, so about 100 subs.", "votes": 1 }, { "id": 970038, "postDate": "2020-08-14T06:32:28.003Z", "content": "That is probably enough to increase 0.960 to 0.990. I think you could check your model's top 3000 predictions by making 100 subs where you set the top pred[30*k,30*(k+1)]=1 for k in range(100) (and other predictions to zero). Then you order all those groups by their LB score (keeping them above the bottom 7000). I could run a simulation on OOF doing this procedure to see how much it increases AUC.
@cdeotte Could you explain how this is done or could you create a notebook about this one??
", "rawMarkdown": "@cdeotte Could you explain how this is done or could you create a notebook about this one??\n" }, { "id": 970510, "postDate": "2020-08-14T14:15:45.710Z", "content": "I did a quick simulation of what I describe above using OOF. It's not as effective as I thought.
\nIf you do 100 probes (size 32 each) your LB would increase from 0.9600 to 0.9650.
\nIf you do 200 probes (size 16 each) then your LB would increase from 0.9600 to 0.9675.
\nIf you do 400 probes (size 8 each) then your LB would increase from 0.9600 to 0.9715.
\nIf you do 800 probes (size 4 each) then your LB would increase from 0.9600 to 0.9800.
Furthermore, simulation suggests that this procedure would increase public LB but it would simultaneously decrease private LB.
", "rawMarkdown": "I did a quick simulation of what I describe above using OOF. It's not as effective as I thought. \n\nIf you do 100 probes (size 32 each) your LB would increase from 0.9600 to 0.9650. \nIf you do 200 probes (size 16 each) then your LB would increase from 0.9600 to 0.9675. \nIf you do 400 probes (size 8 each) then your LB would increase from 0.9600 to 0.9715. \nIf you do 800 probes (size 4 each) then your LB would increase from 0.9600 to 0.9800.\n\nFurthermore, simulation suggests that this procedure would increase public LB but it would simultaneously decrease private LB.", "votes": 3 } ] }, { "id": 970075, "postDate": "2020-08-14T07:15:46.937Z", "content": "Sounds reasonable @cdeotte. Let's see your simulation results.
", "rawMarkdown": "Sounds reasonable @cdeotte. Let's see your simulation results.", "votes": 1 } ] }, { "id": 969694, "postDate": "2020-08-13T21:26:06.180Z", "content": "I think they jumped from 962 directly to 99x which is somehow weird :)
", "rawMarkdown": "I think they jumped from 962 directly to 99x which is somehow weird :)", "votes": 4, "replies": [ { "id": 969713, "postDate": "2020-08-13T21:54:05.047Z", "content": "I agree, more weird than the 1st.
\nPlus, seeing many GMs like you not in the front part makes me feel a shake will happen.
So big jump is very strange for probing … maybe they found some leak, doctor who labeled image by hand (joke). At least they have good PL and can train model on test dataset which is a big advantage
", "rawMarkdown": "So big jump is very strange for probing ... maybe they found some leak, doctor who labeled image by hand (joke). At least they have good PL and can train model on test dataset which is a big advantage" }, { "id": 969721, "postDate": "2020-08-13T22:07:47.447Z", "content": "Or they probed images separately and then combined the probe.
", "rawMarkdown": "Or they probed images separately and then combined the probe.", "votes": 5 }, { "id": 969725, "postDate": "2020-08-13T22:13:04.863Z", "content": "💯.
", "rawMarkdown": "💯. ", "votes": 1 } ] }, { "id": 969739, "postDate": "2020-08-13T22:31:28.750Z", "content": "\n\nWhile you can use the probed labels as additional training data
\n
No you can't. Hand labeling of test data is prohibited.
", "rawMarkdown": "> While you can use the probed labels as additional training data \n\nNo you can't. Hand labeling of test data is prohibited.", "votes": 1, "replies": [ { "id": 969743, "postDate": "2020-08-13T22:38:01.613Z", "content": "its not hand labeling, its Public LB labeling :D
", "rawMarkdown": "its not hand labeling, its Public LB labeling :D", "votes": 2 }, { "id": 969750, "postDate": "2020-08-13T22:47:36.483Z", "content": "You can completely automate it with a script. A script (without any human intervention) can make a submission to LB and store the result. (A human never sees the result). Then the same script can impart it's knowledge into your model.
", "rawMarkdown": "You can completely automate it with a script. A script (without any human intervention) can make a submission to LB and store the result. (A human never sees the result). Then the same script can impart it's knowledge into your model.", "votes": 3 }, { "id": 969755, "postDate": "2020-08-13T22:58:15.960Z", "content": "Well, if this is how they did it then fine. But I doubt it very much. And I'm curious to see how host interprets it.
", "rawMarkdown": "Well, if this is how they did it then fine. But I doubt it very much. And I'm curious to see how host interprets it." }, { "id": 969758, "postDate": "2020-08-13T23:02:08.807Z", "content": "There is a lot of gray area here. What if you cluster the images unsupervised into groups of 30. Then you make 333 LB submissions. Afterward is it hand labeling or a decision tree?
\n If test image belongs to cluster X then classify as Y\nYou said it : it is a gray area.
\nI interpreted rules to forbid it, but I get that another interpretation is possible.
\nBest would be to have Kaggle or host weigh in.
", "rawMarkdown": "You said it : it is a gray area.\n\nI interpreted rules to forbid it, but I get that another interpretation is possible.\n\nBest would be to have Kaggle or host weigh in.\n", "votes": 1 } ] }, { "id": 970133, "postDate": "2020-08-14T08:30:10.750Z", "content": "I think LB probing is fine if someone is curious. Personally I just could never pass the thought of using up all those submissions, especially toward the end. It would take alot of submissions to probe to get to .99, and they have two in the team, but still, 385 submissions = 77 days of submissions, so burn then however you like. The only issue I can see is if someone has made dummy accounts to do the probing, although Kaggle has done a good job of slow playing people like that and then dealing with it when competitions end.
", "rawMarkdown": "I think LB probing is fine if someone is curious. Personally I just could never pass the thought of using up all those submissions, especially toward the end. It would take alot of submissions to probe to get to .99, and they have two in the team, but still, 385 submissions = 77 days of submissions, so burn then however you like. The only issue I can see is if someone has made dummy accounts to do the probing, although Kaggle has done a good job of slow playing people like that and then dealing with it when competitions end. ", "votes": 2 }, { "id": 970041, "postDate": "2020-08-14T06:35:03.357Z", "content": "Well I’m not assuming that they used probing to get there, but here is a simple set up you can use to probe (or simply check your progress) as much as you want in this competition while staying under the radars:
\nSimply take your submission and do 1 - submission before submitting it, then your AUC will be 1-real_AUC. No one will know about how good your models are performing. Once you are coming close to the end of the competition and want to have a real submission to compete on the private LB then submit the real submission file. IMHO this is one possible explanation for such a sudden jump!
", "rawMarkdown": "Well I’m not assuming that they used probing to get there, but here is a simple set up you can use to probe (or simply check your progress) as much as you want in this competition while staying under the radars:\n\nSimply take your submission and do 1 - submission before submitting it, then your AUC will be 1-real_AUC. No one will know about how good your models are performing. Once you are coming close to the end of the competition and want to have a real submission to compete on the private LB then submit the real submission file. IMHO this is one possible explanation for such a sudden jump!", "votes": 2, "replies": [ { "id": 970048, "postDate": "2020-08-14T06:41:14.570Z", "content": "To avoid this, in AUC metric competitions, Kaggle should probably show LB as max(AUC, 1-AUC)
", "rawMarkdown": "To avoid this, in AUC metric competitions, Kaggle should probably show LB as max(AUC, 1-AUC)" }, { "id": 970084, "postDate": "2020-08-14T07:23:27.613Z", "content": "It could be an explanation but perhaps not for this one.
\nThey DID get a score about 0.96 before their jump. Which means they need to submit 300X scores lower than 0.04 seemed really impossible.
not sure to understand your comment. They had 0.96, wanted to try something new without being visible, created a submission file, took 1 - subfile and submited: score is 0.03. Then they know they can get to 0.97 but you don't. That's all I'm saying, there is no need for 300 submissions.
", "rawMarkdown": "not sure to understand your comment. They had 0.96, wanted to try something new without being visible, created a submission file, took 1 - subfile and submited: score is 0.03. Then they know they can get to 0.97 but you don't. That's all I'm saying, there is no need for 300 submissions.", "votes": 1 }, { "id": 970371, "postDate": "2020-08-14T11:46:04.610Z", "content": "I read somewhere some time ago that kaggle already does this.
", "rawMarkdown": "I read somewhere some time ago that kaggle already does this.", "votes": 1 }, { "id": 970441, "postDate": "2020-08-14T13:06:53.717Z", "content": "\n\nKaggle should probably show LB as max(AUC, 1-AUC)
\n
Kaggle does it.
", "rawMarkdown": "> Kaggle should probably show LB as max(AUC, 1-AUC)\n\nKaggle does it." }, { "id": 970513, "postDate": "2020-08-14T14:17:36.773Z", "content": "I think Kaggle only does it if your AUC is very low. For example if your AUC is 0.49 then Kaggle does not report 0.51. But if your AUC is 0.10 then perhaps Kaggle reports 0.90.
", "rawMarkdown": "I think Kaggle only does it if your AUC is very low. For example if your AUC is 0.49 then Kaggle does not report 0.51. But if your AUC is 0.10 then perhaps Kaggle reports 0.90.", "votes": 1 }, { "id": 970527, "postDate": "2020-08-14T14:34:27.167Z", "content": "nope. Had a small bug and got LB AUC of 0.06 instead of 0.94
", "rawMarkdown": "nope. Had a small bug and got LB AUC of 0.06 instead of 0.94", "votes": 4, "replies": [ { "id": 970698, "postDate": "2020-08-14T17:26:57.243Z", "content": "ok, Kaggle did it in some competitions, I've seen it ;)
", "rawMarkdown": "ok, Kaggle did it in some competitions, I've seen it ;)", "votes": 1 } ] } ] }, { "id": 969753, "postDate": "2020-08-13T22:52:00.617Z", "content": "I never did LB probing but heard many times during many competitions. By its intuition, personally we didn't feel any excitement on this technique, so never wanted to try. But I also read that this technique is not prohibited in competition. So, just wondering why it looks like many participants get bothered about it!! But I must agree with Guanshuo Xu, it would be better if it's a kernel competition.
", "rawMarkdown": "I never did LB probing but heard many times during many competitions. By its intuition, personally we didn't feel any excitement on this technique, so never wanted to try. But I also read that this technique is not prohibited in competition. So, just wondering why it looks like many participants get bothered about it!! But I must agree with Guanshuo Xu, it would be better if it's a kernel competition. ", "votes": 2, "replies": [ { "id": 969754, "postDate": "2020-08-13T22:54:02.933Z", "content": "In real life your \"employer\" tells you where the test images come from. In Kaggle \"LB probing\" tells you where the test images come from. It's important to know if the test images come from the same distribution as train or if the features are shifted (explained here).
", "rawMarkdown": "In real life your \"employer\" tells you where the test images come from. In Kaggle \"LB probing\" tells you where the test images come from. It's important to know if the test images come from the same distribution as train or if the features are shifted (explained [here] [1]).\n\n[1]: https://towardsdatascience.com/understanding-dataset-shift-f2a5a262a766", "votes": 5 }, { "id": 970364, "postDate": "2020-08-14T11:40:16.130Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 970507, "postDate": "2020-08-14T14:14:09.007Z", "content": "Agree. Bad practice! Kaggle should have a simple script to check this and deny such mal practice.
", "rawMarkdown": "Agree. Bad practice! Kaggle should have a simple script to check this and deny such mal practice.", "votes": -4 }, { "id": 969740, "postDate": "2020-08-13T22:32:56.693Z", "content": "Is this the start of a leak hunt?
", "rawMarkdown": "Is this the start of a leak hunt?", "replies": [ { "id": 969751, "postDate": "2020-08-13T22:51:31.990Z", "content": "Quote from Sirish here
\n\n", "rawMarkdown": "Quote from Sirish [here][1]\n> I did not find/exploit any LEAK. I can understand your concern, especially because recently top teams lost their ranking (and prize money), for example during Liverpool competition link1 link2\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161497#900907", "votes": 3 }, { "id": 969759, "postDate": "2020-08-13T23:02:10.283Z", "content": "I did not find/exploit any LEAK. I can understand your concern, especially because recently top teams lost their ranking (and prize money), for example during Liverpool competition link1 link2
\n
I once wrote I was not using a leak then was accused of misleading people on purpose. What is a leak and what is not a leak is not well defined.
", "rawMarkdown": "I once wrote I was not using a leak then was accused of misleading people on purpose. What is a leak and what is not a leak is not well defined.", "votes": 3 } ] }, { "id": 969724, "postDate": "2020-08-13T22:13:02.347Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 970111, "author_name": "nosound", "author_url": "", "post_date": "2020-08-14T08:03:07.497000", "content": "Hi, please let me clarify as much as I can now. That result is based on LB probing, it comprises many probing submissions, each of the probing submission scores is in range 0.4-0.6 that is why it was not visible before. This submission will be very bad for the private LB, it is basically very overfited to the public. I plan to publish more details on the procedure when the competition ends. I submitted it now to showcase what is possible to achieve with the probing. We didn't submit it earlier in order not to disrupt the leaderboard.
\nMy motivation: I like solving tasks with mixed integer programming, I worked with it at IBM many years ago and I practise it when appropriate since then. I try to distill as much information as possible from each submission, it is challenging and interesting.
\nI understand that the area is a gray area, so I try to maintain full transparency, specifically on how we use it in our final solutions, if at all. Will be glad to share more when the competition ends.
\nGood luck to you in the last few days!
", "votes": 33, "replies": [ { "id": 970163, "author_name": "jsyphil", "author_url": "", "post_date": "2020-08-14T09:00:23.573000", "content": "Thanks for clarifying.
", "votes": 0, "replies": [] }, { "id": 970217, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T09:31:13.457000", "content": "\n\nI like solving tasks with mixed integer programming,
\n
I love it. Thanks for clarifying.
\nDid you use CPLEX while at IBM? That's what I was working on as you probably know.
", "votes": 2, "replies": [] }, { "id": 970300, "author_name": "nosound", "author_url": "", "post_date": "2020-08-14T10:46:29.843000", "content": "Yes, I remember you also participated in the latest Santa competition. At IBM we used CPLEX, of course, but during the Santa competition it became clear to us that Gurobi worked better there, as reported also by many other people. So I switched to Gurobi, and here also I used Gurobi.
", "votes": 1, "replies": [] }, { "id": 970358, "author_name": "", "author_url": "", "post_date": "2020-08-14T11:36:50.890000", "content": "", "votes": 0, "replies": [ { "id": 970359, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-14T11:37:27.910000", "content": "That's already known. 77 or 78 positive images.
", "votes": 3, "replies": [] }, { "id": 970374, "author_name": "", "author_url": "", "post_date": "2020-08-14T11:46:48.433000", "content": "", "votes": 0, "replies": [] }, { "id": 970440, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T13:06:02.497000", "content": "It is the best notebook ever.
", "votes": 8, "replies": [] } ] }, { "id": 973382, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-08-17T09:20:57.800000", "content": "@zaharch was also the one who had an impressive public LB score in LANL Earthquake Prediction not long ago. He did some smart optimization to know public rows of submissions.
", "votes": 0, "replies": [] } ] }, { "id": 969722, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2020-08-13T22:09:23.657000", "content": "I don't think we should make any kind of allegations or comments without knowing their side of the story.
\nIf it's probing it has been done already by a few this competition.
\nAs long as the private solution is clean, who cares.
Totally agreeing with you
", "votes": 0, "replies": [] } ] }, { "id": 969719, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2020-08-13T22:02:13.193000", "content": "Kaggle should make this one a kernel competition.
", "votes": 13, "replies": [ { "id": 969731, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-13T22:22:27.067000", "content": "\n\nKaggle should make this one a kernel competition.
\n
Or 2 stages competition with 2nd stage test set available just a week before the end. That would still allow internet use (for TPU for instance)
\nThere used to be many 2 stages CV competitions before on kaggle( in order to prevent handlabelling among others). I don't why that's not the case anymore.
Two of these things are not like the others 🤔
\n
Great plot. In the first month of the comp, Yuval was in 1st place using a single model that scored LB 0.960 explained here. After Sirish took first, Sirish explained what he did to climb LB here. Then NoSound joined Yuval and their LB did not increase for 22 days until today when it jumped up to match Sirish's.
", "votes": 7, "replies": [] }, { "id": 969760, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T23:03:35.020000", "content": "\n\ntheir LB did not increase for 22 days until today
\n
It shows LB is not required to measure progress.
", "votes": 7, "replies": [] }, { "id": 970024, "author_name": "Psi", "author_url": "", "post_date": "2020-08-14T06:20:43.227000", "content": "How many subs during the no-progress period?
", "votes": 0, "replies": [] }, { "id": 970031, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-14T06:25:21.370000", "content": "I forget. Can we check that in meta Kaggle dataset? If I remember correctly, i think they did about 5 a day, so about 100 subs.
", "votes": 1, "replies": [] }, { "id": 970038, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-14T06:32:28.003000", "content": "That is probably enough to increase 0.960 to 0.990. I think you could check your model's top 3000 predictions by making 100 subs where you set the top pred[30*k,30*(k+1)]=1 for k in range(100) (and other predictions to zero). Then you order all those groups by their LB score (keeping them above the bottom 7000). I could run a simulation on OOF doing this procedure to see how much it increases AUC.
@cdeotte Could you explain how this is done or could you create a notebook about this one??
", "votes": 0, "replies": [] }, { "id": 970510, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-14T14:15:45.710000", "content": "I did a quick simulation of what I describe above using OOF. It's not as effective as I thought.
\nIf you do 100 probes (size 32 each) your LB would increase from 0.9600 to 0.9650.
\nIf you do 200 probes (size 16 each) then your LB would increase from 0.9600 to 0.9675.
\nIf you do 400 probes (size 8 each) then your LB would increase from 0.9600 to 0.9715.
\nIf you do 800 probes (size 4 each) then your LB would increase from 0.9600 to 0.9800.
Furthermore, simulation suggests that this procedure would increase public LB but it would simultaneously decrease private LB.
", "votes": 3, "replies": [] } ] }, { "id": 970075, "author_name": "Changyi", "author_url": "", "post_date": "2020-08-14T07:15:46.937000", "content": "Sounds reasonable @cdeotte. Let's see your simulation results.
", "votes": 1, "replies": [] } ] }, { "id": 969694, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T21:26:06.180000", "content": "I think they jumped from 962 directly to 99x which is somehow weird :)
", "votes": 4, "replies": [ { "id": 969713, "author_name": "Changyi", "author_url": "", "post_date": "2020-08-13T21:54:05.047000", "content": "I agree, more weird than the 1st.
\nPlus, seeing many GMs like you not in the front part makes me feel a shake will happen.
So big jump is very strange for probing … maybe they found some leak, doctor who labeled image by hand (joke). At least they have good PL and can train model on test dataset which is a big advantage
", "votes": 0, "replies": [] }, { "id": 969721, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T22:07:47.447000", "content": "Or they probed images separately and then combined the probe.
", "votes": 5, "replies": [] }, { "id": 969725, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-13T22:13:04.863000", "content": "💯.
", "votes": 1, "replies": [] } ] }, { "id": 969739, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T22:31:28.750000", "content": "\n\nWhile you can use the probed labels as additional training data
\n
No you can't. Hand labeling of test data is prohibited.
", "votes": 1, "replies": [ { "id": 969743, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-13T22:38:01.613000", "content": "its not hand labeling, its Public LB labeling :D
", "votes": 2, "replies": [] }, { "id": 969750, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T22:47:36.483000", "content": "You can completely automate it with a script. A script (without any human intervention) can make a submission to LB and store the result. (A human never sees the result). Then the same script can impart it's knowledge into your model.
", "votes": 3, "replies": [] }, { "id": 969755, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T22:58:15.960000", "content": "Well, if this is how they did it then fine. But I doubt it very much. And I'm curious to see how host interprets it.
", "votes": 0, "replies": [] }, { "id": 969758, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T23:02:08.807000", "content": "There is a lot of gray area here. What if you cluster the images unsupervised into groups of 30. Then you make 333 LB submissions. Afterward is it hand labeling or a decision tree?
\n If test image belongs to cluster X then classify as Y\nYou said it : it is a gray area.
\nI interpreted rules to forbid it, but I get that another interpretation is possible.
\nBest would be to have Kaggle or host weigh in.
", "votes": 1, "replies": [] } ] }, { "id": 970133, "author_name": "Signal", "author_url": "", "post_date": "2020-08-14T08:30:10.750000", "content": "I think LB probing is fine if someone is curious. Personally I just could never pass the thought of using up all those submissions, especially toward the end. It would take alot of submissions to probe to get to .99, and they have two in the team, but still, 385 submissions = 77 days of submissions, so burn then however you like. The only issue I can see is if someone has made dummy accounts to do the probing, although Kaggle has done a good job of slow playing people like that and then dealing with it when competitions end.
", "votes": 2, "replies": [] }, { "id": 970041, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-14T06:35:03.357000", "content": "Well I’m not assuming that they used probing to get there, but here is a simple set up you can use to probe (or simply check your progress) as much as you want in this competition while staying under the radars:
\nSimply take your submission and do 1 - submission before submitting it, then your AUC will be 1-real_AUC. No one will know about how good your models are performing. Once you are coming close to the end of the competition and want to have a real submission to compete on the private LB then submit the real submission file. IMHO this is one possible explanation for such a sudden jump!
", "votes": 2, "replies": [ { "id": 970048, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-14T06:41:14.570000", "content": "To avoid this, in AUC metric competitions, Kaggle should probably show LB as max(AUC, 1-AUC)
", "votes": 0, "replies": [] }, { "id": 970084, "author_name": "Changyi", "author_url": "", "post_date": "2020-08-14T07:23:27.613000", "content": "It could be an explanation but perhaps not for this one.
\nThey DID get a score about 0.96 before their jump. Which means they need to submit 300X scores lower than 0.04 seemed really impossible.
not sure to understand your comment. They had 0.96, wanted to try something new without being visible, created a submission file, took 1 - subfile and submited: score is 0.03. Then they know they can get to 0.97 but you don't. That's all I'm saying, there is no need for 300 submissions.
", "votes": 1, "replies": [] }, { "id": 970371, "author_name": "Rohit Agarwal", "author_url": "", "post_date": "2020-08-14T11:46:04.610000", "content": "I read somewhere some time ago that kaggle already does this.
", "votes": 1, "replies": [] }, { "id": 970441, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T13:06:53.717000", "content": "\n\nKaggle should probably show LB as max(AUC, 1-AUC)
\n
Kaggle does it.
", "votes": 0, "replies": [] }, { "id": 970513, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-14T14:17:36.773000", "content": "I think Kaggle only does it if your AUC is very low. For example if your AUC is 0.49 then Kaggle does not report 0.51. But if your AUC is 0.10 then perhaps Kaggle reports 0.90.
", "votes": 1, "replies": [] }, { "id": 970527, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-14T14:34:27.167000", "content": "nope. Had a small bug and got LB AUC of 0.06 instead of 0.94
", "votes": 4, "replies": [ { "id": 970698, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-14T17:26:57.243000", "content": "ok, Kaggle did it in some competitions, I've seen it ;)
", "votes": 1, "replies": [] } ] } ] }, { "id": 969753, "author_name": "Innat", "author_url": "", "post_date": "2020-08-13T22:52:00.617000", "content": "I never did LB probing but heard many times during many competitions. By its intuition, personally we didn't feel any excitement on this technique, so never wanted to try. But I also read that this technique is not prohibited in competition. So, just wondering why it looks like many participants get bothered about it!! But I must agree with Guanshuo Xu, it would be better if it's a kernel competition.
", "votes": 2, "replies": [ { "id": 969754, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T22:54:02.933000", "content": "In real life your \"employer\" tells you where the test images come from. In Kaggle \"LB probing\" tells you where the test images come from. It's important to know if the test images come from the same distribution as train or if the features are shifted (explained here).
", "votes": 5, "replies": [] }, { "id": 970364, "author_name": "", "author_url": "", "post_date": "2020-08-14T11:40:16.130000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 970507, "author_name": "Patrick Chan", "author_url": "", "post_date": "2020-08-14T14:14:09.007000", "content": "Agree. Bad practice! Kaggle should have a simple script to check this and deny such mal practice.
", "votes": -4, "replies": [] }, { "id": 969740, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T22:32:56.693000", "content": "Is this the start of a leak hunt?
", "votes": 0, "replies": [ { "id": 969751, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T22:51:31.990000", "content": "Quote from Sirish here
\n\n", "votes": 3, "replies": [] }, { "id": 969759, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-13T23:02:10.283000", "content": "I did not find/exploit any LEAK. I can understand your concern, especially because recently top teams lost their ranking (and prize money), for example during Liverpool competition link1 link2
\n
I once wrote I was not using a leak then was accused of misleading people on purpose. What is a leak and what is not a leak is not well defined.
", "votes": 3, "replies": [] } ] }, { "id": 969724, "author_name": "", "author_url": "", "post_date": "2020-08-13T22:13:02.347000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "970111": "Hi, please let me clarify as much as I can now. That result is based on LB probing, it comprises many probing submissions, each of the probing submission scores is in range 0.4-0.6 that is why it was not visible before. This submission will be very bad for the private LB, it is basically very overfited to the public. I plan to publish more details on the procedure when the competition ends. I submitted it now to showcase what is possible to achieve with the probing. We didn't submit it earlier in order not to disrupt the leaderboard.\n\nMy motivation: I like solving tasks with mixed integer programming, I worked with it at IBM many years ago and I practise it when appropriate since then. I try to distill as much information as possible from each submission, it is challenging and interesting.\n\nI understand that the area is a gray area, so I try to maintain full transparency, specifically on how we use it in our final solutions, if at all. Will be glad to share more when the competition ends.\n\nGood luck to you in the last few days!", "969690": "I can only speculate, but telling from the number of submissions, they just intensively probed image per image on the LB to label the test images which are part of the public test set... Whats the point? While you can use the probed labels as additional training data and probably get a better score, how does this help anyone but your own ego?", "969722": "I don't think we should make any kind of allegations or comments without knowing their side of the story.\nIf it's probing it has been done already by a few this competition. \nAs long as the private solution is clean, who cares. \n", "969719": "Kaggle should make this one a kernel competition.", "969729": "Two of these things are not like the others 🤔\n\n", "969694": "I think they jumped from 962 directly to 99x which is somehow weird :)", "969739": "> While you can use the probed labels as additional training data \n\nNo you can't. Hand labeling of test data is prohibited.", "970133": "I think LB probing is fine if someone is curious. Personally I just could never pass the thought of using up all those submissions, especially toward the end. It would take alot of submissions to probe to get to .99, and they have two in the team, but still, 385 submissions = 77 days of submissions, so burn then however you like. The only issue I can see is if someone has made dummy accounts to do the probing, although Kaggle has done a good job of slow playing people like that and then dealing with it when competitions end. ", "970041": "Well I’m not assuming that they used probing to get there, but here is a simple set up you can use to probe (or simply check your progress) as much as you want in this competition while staying under the radars:\n\nSimply take your submission and do 1 - submission before submitting it, then your AUC will be 1-real_AUC. No one will know about how good your models are performing. Once you are coming close to the end of the competition and want to have a real submission to compete on the private LB then submit the real submission file. IMHO this is one possible explanation for such a sudden jump!", "969753": "I never did LB probing but heard many times during many competitions. By its intuition, personally we didn't feel any excitement on this technique, so never wanted to try. But I also read that this technique is not prohibited in competition. So, just wondering why it looks like many participants get bothered about it!! But I must agree with Guanshuo Xu, it would be better if it's a kernel competition. ", "970507": "Agree. Bad practice! Kaggle should have a simple script to check this and deny such mal practice.", "969740": "Is this the start of a leak hunt?", "969724": "" } }