The Leak to get to the 0.999 score is explained here .
\nIt is essentially the date the numpy (.npy) files were processed.
\nI have created a kernel that demonstrates how to create and use the leak here that scores 0.991 .
\nI have generated the leak and made it a dataset here. To generate the leak you download the data locally. I.E it does not work with the data from inside the kernel.
\nI hate looking for leaks. I never do it unless I am certain there is one and a score of 0.999 signifies that there is definitely one.
\nI remain loyal to my belief that these leaks should be shared as they can compromise the many good models that other participants may have created. We would hate to win by just using that leak - this is not data science and gives zero value to the organiser.
\nSadly for kaggle (cc @maggiemd )this is (at minimum) the 3rd time( [1],[2]) that this type of leak has occurred in my kaggle tenure. I am bit disappointed by that.
\nI am sure kaggle team tries a lot to keep competitions well-regulated, but the same mistakes cannot happen 3 times! You really need to put some process in place to check for the date the files are processed!!!
\nI am also a bit disappointed by the top ranked members NOT revealing the leak when they clearly use it.
\nHowever I cannot fully blame them because kaggle in the past has shown that leaks are valid and can win competitions.
\nSoooo.. Where do we go from here?! Restart with non-leaky data…. or fight for the 5th-6th digit?
\n1) https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries/discussion/31870
\n2)https://www.kaggle.com/c/dato-native/discussion/16485
The Leak to get to the 0.999 score is explained here .
\nIt is essentially the date the numpy (.npy) files were processed.
\nI have created a kernel that demonstrates how to create and use the leak here that scores 0.991 .
\nI have generated the leak and made it a dataset here. To generate the leak you download the data locally. I.E it does not work with the data from inside the kernel.
\nI hate looking for leaks. I never do it unless I am certain there is one and a score of 0.999 signifies that there is definitely one.
\nI remain loyal to my belief that these leaks should be shared as they can compromise the many good models that other participants may have created. We would hate to win by just using that leak - this is not data science and gives zero value to the organiser.
\nSadly for kaggle (cc @maggiemd )this is (at minimum) the 3rd time( [1],[2]) that this type of leak has occurred in my kaggle tenure. I am bit disappointed by that.
\nI am sure kaggle team tries a lot to keep competitions well-regulated, but the same mistakes cannot happen 3 times! You really need to put some process in place to check for the date the files are processed!!!
\nI am also a bit disappointed by the top ranked members NOT revealing the leak when they clearly use it.
\nHowever I cannot fully blame them because kaggle in the past has shown that leaks are valid and can win competitions.
\nSoooo.. Where do we go from here?! Restart with non-leaky data…. or fight for the 5th-6th digit?
\n1) https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries/discussion/31870
\n2)https://www.kaggle.com/c/dato-native/discussion/16485
I want to share my personal view as well. As a team we have discussed the possibility of leakage yesterday. It was revealing the \"extra\" digit that made @kazanova 99.99% sure that there was a leakage involved. I took a snapshot of the LB before Marios exposes the leakage revealing the gap from 0.995(#6 place) to is 0.989 (#7 place). I believe, from my 3 week-experience in this competition, that any score >=0.995 it is extremely difficult to appear on public LB by pure CNN modelling without \"meta\" or \"leaked\" information.
\n0.995 AUC score was also set by @inversion as a top target at the end of this competition.
\n\n\nUltimately, though, it's not the value of the metric that matters. This competition is about looking for something we don't know what it looks like. So, we could make the data arbitrarily hard. But that may not actually help in the real search for E.T. It's really about figuring out ways to find things that look different (and not just RFI). So, if the competition ends at 0.995, that still might be a major win for science.
\n
Kudos to my teammate @kazanova for resolving the \"leakage\". He did it in less than 20 minutes because as he told me he knew where to look from previous Kaggle competitions with similar cases. He sets the example for all Kagglers for competition fairness. As for the players who new the leak and chose silence, they should change their mindset. Finding such a leak is not done randomly it requires some special effort in this direction! One does not simply randomly discovers a leak!! 😄😄
\nRevealing such a leak is good for the task we are trying to solve and tights the bonds of Kaggle community!
", "rawMarkdown": "I want to share my personal view as well. As a team we have discussed the possibility of leakage yesterday. It was revealing the \"extra\" digit that made @kazanova 99.99% sure that there was a leakage involved. I took a snapshot of the LB before Marios exposes the leakage revealing the gap from 0.995(#6 place) to is 0.989 (#7 place). I believe, from my 3 week-experience in this competition, that any score >=0.995 it is extremely difficult to appear on public LB by pure CNN modelling without \"meta\" or \"leaked\" information. \n\n0.995 AUC score was also set by @inversion as a top target at the end of this competition.\n> Ultimately, though, it's not the value of the metric that matters. This competition is about looking for something we don't know what it looks like. So, we could make the data arbitrarily hard. But that may not actually help in the real search for E.T. It's really about figuring out ways to find things that look different (and not just RFI). So, if the competition ends at 0.995, that still might be a major win for science.\n\nKudos to my teammate @kazanova for resolving the \"leakage\". He did it in less than 20 minutes because as he told me he knew where to look from previous Kaggle competitions with similar cases. He sets the example for all Kagglers for competition fairness. As for the players who new the leak and chose silence, they should change their mindset. Finding such a leak is not done randomly it requires some special effort in this direction! One does not simply randomly discovers a leak!! 😄😄\n\nRevealing such a leak is good for the task we are trying to solve and tights the bonds of Kaggle community!\n", "votes": 27 }, { "id": 1353566, "postDate": "2021-06-17T07:11:32.040Z", "content": "@inversion I hope kaggle swag for @kazanova is already on the way ;)
", "rawMarkdown": "@inversion I hope kaggle swag for @kazanova is already on the way ;)", "votes": 18, "replies": [ { "id": 1353874, "postDate": "2021-06-17T09:24:27.033Z", "content": "A t-shirt with \"Leak detector GM\" written on it. :D
", "rawMarkdown": "A t-shirt with \"Leak detector GM\" written on it. :D ", "votes": 10 }, { "id": 1354418, "postDate": "2021-06-17T15:25:37.703Z", "content": "\n\n@inversion I hope kaggle swag for @kazanova is already on the way ;)
\n
I have them all by now 😛
", "rawMarkdown": ">@inversion I hope kaggle swag for @kazanova is already on the way ;)\n\nI have them all by now 😛", "votes": 4 }, { "id": 1354774, "postDate": "2021-06-17T21:20:09.687Z", "content": "@kazanova I can send you some Berkeley SETI swag if you DM me your mailing address
", "rawMarkdown": "@kazanova I can send you some Berkeley SETI swag if you DM me your mailing address", "votes": 16 } ] }, { "id": 1353189, "postDate": "2021-06-17T01:16:15Z", "content": "Thanks for the post @kazanova. We're looking into this and will give an update shortly.
", "rawMarkdown": "Thanks for the post @kazanova. We're looking into this and will give an update shortly.", "votes": 16 }, { "id": 1353235, "postDate": "2021-06-17T02:11:06.910Z", "content": "Omg. Impressive finding. But interesting fact, I am not using any information about id and time creation. Just really custom nn! Mb nn exploits this leak implicitly, but i don't think so.
We'll find out when we swap out a new Test set. 😄
", "rawMarkdown": "We'll find out when we swap out a new Test set. 😄", "votes": 12 }, { "id": 1353250, "postDate": "2021-06-17T02:35:12.877Z", "content": "It is very good to hear that. We should not be too skeptic about others, just wait for restarting the competition.
", "rawMarkdown": "It is very good to hear that. We should not be too skeptic about others, just wait for restarting the competition.", "votes": 1 }, { "id": 1353256, "postDate": "2021-06-17T02:37:38.617Z", "content": "@tomooinubushi People can continue to build their models with local validation, and even using the Public LB, while we wait for the new Test data.
", "rawMarkdown": "@tomooinubushi People can continue to build their models with local validation, and even using the Public LB, while we wait for the new Test data.", "votes": 9 }, { "id": 1353415, "postDate": "2021-06-17T05:10:49.973Z", "content": "Good decision to reset the test data else this competition is utterly useless.
", "rawMarkdown": "Good decision to reset the test data else this competition is utterly useless.", "votes": 1 }, { "id": 1353427, "postDate": "2021-06-17T05:21:14.947Z", "content": "@inversionspaces How much time will it take for the new test dataset ?
", "rawMarkdown": "@inversionspaces How much time will it take for the new test dataset ?" }, { "id": 1353495, "postDate": "2021-06-17T06:23:05.633Z", "content": "Your CV is also 0.999+ @sggpls ? - if so then thats impressive. We will see in the re-score I guess. I just hope there aren't any more leaks.
", "rawMarkdown": "Your CV is also 0.999+ @sggpls ? - if so then thats impressive. We will see in the re-score I guess. I just hope there aren't any more leaks.", "votes": 7 }, { "id": 1353883, "postDate": "2021-06-17T09:29:31.600Z", "content": "Okay, waiting for the new test dataset.
", "rawMarkdown": "Okay, waiting for the new test dataset.", "votes": 2 }, { "id": 1357027, "postDate": "2021-06-19T12:18:00.350Z", "content": "i want to new test dataset soon.
", "rawMarkdown": "i want to new test dataset soon." } ] }, { "id": 1353239, "postDate": "2021-06-17T02:18:34.527Z", "content": "I responded in this thread:
\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782
", "rawMarkdown": "I responded in this thread:\n\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782", "votes": 7 }, { "id": 1354194, "postDate": "2021-06-17T12:48:21.760Z", "content": "Thanks for sharing @kazanova. I thought Kaggle was already \"vaccinated\" against this kind of silly mistake. Now LB becomes useless, people can hide the real scores.
", "rawMarkdown": "Thanks for sharing @kazanova. I thought Kaggle was already \"vaccinated\" against this kind of silly mistake. Now LB becomes useless, people can hide the real scores.", "votes": 8, "replies": [ { "id": 1354383, "postDate": "2021-06-17T14:54:25.397Z", "content": "The date-in-the-files virus has developed immunity over the years 😛😛
", "rawMarkdown": "The date-in-the-files virus has developed immunity over the years 😛😛", "votes": 5 } ] }, { "id": 1353188, "postDate": "2021-06-17T01:16:05.453Z", "content": "Thank you for reporting the leak.
\nI think someone like you always makes kaggle interesting.
Marios, thanks for explaining the leak. You could have taken advantage of it and keep it for your team.
\nI hope there will be a continuation with new (private) test data.
\nI remember one previous case where processing date (pictures in the 2sigma appartment rental unless mistaken)
\nEdit: @inversion already communicated they will release a new test set. Good move.
", "rawMarkdown": "Marios, thanks for explaining the leak. You could have taken advantage of it and keep it for your team.\n\nI hope there will be a continuation with new (private) test data.\n\nI remember one previous case where processing date (pictures in the 2sigma appartment rental unless mistaken)\n\nEdit: @inversion already communicated they will release a new test set. Good move.", "votes": 5 }, { "id": 1353215, "postDate": "2021-06-17T01:47:49.667Z", "content": "Thanks, @kazanova, for bringing that up.
", "rawMarkdown": "Thanks, @kazanova, for bringing that up. ", "votes": 3 }, { "id": 1353182, "postDate": "2021-06-17T00:55:19.667Z", "content": "thanks for sharing.
\nI was disappointed to know this….
Same here - join the club :(
", "rawMarkdown": "Same here - join the club :(", "votes": 3 } ] }, { "id": 1353196, "postDate": "2021-06-17T01:29:28.953Z", "content": "Much respect @kazanova . Thank you very much for sharing.
\nI thought they are alien, who reached 0.999 before you. :)
@kazanova Thanks for sharing. I hoped that there is need to check metadata for leakage as well. I did not use any metadata to get 0.999 on LB, so there may be another leak in the data itself.
", "rawMarkdown": "@kazanova Thanks for sharing. I hoped that there is need to check metadata for leakage as well. I did not use any metadata to get 0.999 on LB, so there may be another leak in the data itself.", "votes": 1 }, { "id": 1353925, "postDate": "2021-06-17T09:46:03.937Z", "content": "@kazanova Thanks alot for sharing this leakage , Its because of Honest and Seasoned Kagglers like you that Kaggle still remains fair game to some extent . I never thought I would encounter a Leakage at Kaggle Competitions and hence didn't take part explaining Leakage , but next time competing I will look for leaks as well. Thanks alot to Christof Henkel as well his twitter post brought me here.
", "rawMarkdown": "@kazanova Thanks alot for sharing this leakage , Its because of Honest and Seasoned Kagglers like you that Kaggle still remains fair game to some extent . I never thought I would encounter a Leakage at Kaggle Competitions and hence didn't take part explaining Leakage , but next time competing I will look for leaks as well. Thanks alot to Christof Henkel as well his twitter post brought me here.", "votes": 1 }, { "id": 1353329, "postDate": "2021-06-17T03:58:30.877Z", "content": "OMG… thank you for reporting the details of leakage.
\nI hope that updating test set will recover this competition …
", "rawMarkdown": "OMG... thank you for reporting the details of leakage.\n\nI hope that updating test set will recover this competition ...", "votes": 1 }, { "id": 1353324, "postDate": "2021-06-17T03:50:33.163Z", "content": "wow…
\nthis challenge related to operating system ..
\nI never think meta data is a key of this challenge to reach 0.999
If you do not make this issue, after challenge, 1st place user's presentation must be \"extract meta data, is my solution\"
\nBecause, if you (or even other user) were aware of the existence of this metadata, surely someone wouldn't have missed auc 1.0
Hello. I have been thinking deeply about this issue.
\nWhy wasn't there an immediate report of a leak? In the past, it would have been reported immediately.
Because Kaggle has become too famous.
\nToday, we all know that data scientists can expect the highest salaries.
\nAnd every company uses Kaggle rank as a reference.
What if you win the competition? Even if you live in the farthest corner of the world, an American company will hire you at the highest salary and you will be able to drive a Ferrari in Silicon Valley.
\nBut what if you don't get a single medal? Yes, you are a worker who finds traffic lights in Mechanical Turk.
They are desperate. They are in it to change their lives.
\nSo you can't expect them to report leaks…unless you have another reward for them.
So I propose that we create a new, fiveth Kaggle rank.
\nHow about giving the title of [DataSearch Master] to anyone who finds a serious leak like this one? Or even Expert if it's not that serious.
\nThat way, leaks of this kind will be reported in no time.
\nAnyway, if there is no reward of some kind, no one will report it. To maintain a community of good will.
no company is going to hire you based on single competition win.
", "rawMarkdown": "no company is going to hire you based on single competition win.", "votes": 1 }, { "id": 1381927, "postDate": "2021-07-09T11:10:01.987Z", "content": "\n\nIn the past, it would have been reported immediately.
\n
There are many examples of Leak not being reported when found in the past. Many.
\nHere Mario reported the leak within 24 hours of suspicion, which is unusually fast.
", "rawMarkdown": "> In the past, it would have been reported immediately.\n\nThere are many examples of Leak not being reported when found in the past. Many.\n\nHere Mario reported the leak within 24 hours of suspicion, which is unusually fast.", "votes": 4 }, { "id": 1381938, "postDate": "2021-07-09T11:14:57.527Z", "content": "Did I overstate the case?
\nHere's an example. DeNA in Japan hires employees based on their Kaggle rank, and if you make it to the top 3 five times, you get to spend 100% of your work time Kaggling.
\n\nDid I overstate the case?
\n
Which case?
", "rawMarkdown": "> Did I overstate the case?\n\nWhich case?" }, { "id": 1381975, "postDate": "2021-07-09T12:11:04.890Z", "content": "you get to spend 100% of your work time Kaggling
\nFor how long would you be able to give your 100% of your work time Kaggling ? 😄
Thanks @kazanova for highlighting this. I just looked into the competition and suspected something wrong after looking at 0.99 all over the leaderboard.
", "rawMarkdown": "Thanks @kazanova for highlighting this. I just looked into the competition and suspected something wrong after looking at 0.99 all over the leaderboard. ", "votes": 2 }, { "id": 1353243, "postDate": "2021-06-17T02:24:21.637Z", "content": "Will Kaggle get into this and re-organize the dataset?
", "rawMarkdown": "Will Kaggle get into this and re-organize the dataset?", "votes": 2, "replies": [ { "id": 1353247, "postDate": "2021-06-17T02:33:16.980Z", "content": "Yes. We're working on a new Test set. Stay tuned.
", "rawMarkdown": "Yes. We're working on a new Test set. Stay tuned.", "votes": 9 } ] }, { "id": 1395700, "postDate": "2021-07-21T13:24:55.297Z", "content": "crystal clear
", "rawMarkdown": "crystal clear" }, { "id": 1355594, "postDate": "2021-06-18T11:52:37.567Z", "content": "incredible~👀👀👀
", "rawMarkdown": "incredible~👀👀👀" }, { "id": 1355529, "postDate": "2021-06-18T10:47:14.197Z", "content": "Interesting finding! Going forward we may see notebooks like Basic EDA + Leak Check 😁. Anyway, the new data set shouldn't be of much problem for folks not relying on leaks.
", "rawMarkdown": "Interesting finding! Going forward we may see notebooks like Basic EDA + Leak Check 😁. Anyway, the new data set shouldn't be of much problem for folks not relying on leaks. " }, { "id": 1355019, "postDate": "2021-06-18T04:14:37.210Z", "content": "Awesome work to find this leakage. Wow.
", "rawMarkdown": "Awesome work to find this leakage. Wow." }, { "id": 1354079, "postDate": "2021-06-17T11:22:38.920Z", "content": "Thank you for speaking out to the public by bringing that up this topic. @kazanova
", "rawMarkdown": "Thank you for speaking out to the public by bringing that up this topic. @kazanova " }, { "id": 1353472, "postDate": "2021-06-17T06:06:28.843Z", "content": "sorry, I don't fully understand this. leakage means some images in test dataset are the same as train. I don't how to squeeze more info just with timestamp and id if no this kind of leakage. Is this leakage similar to KNN with timestamp info as clue？someone help me.
Thanks for sharing the details about the leak. 👌
", "rawMarkdown": "Thanks for sharing the details about the leak. 👌 " }, { "id": 1355653, "postDate": "2021-06-18T12:59:48Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1353368, "postDate": "2021-06-17T04:33:17.577Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1353447, "author_name": "Costas Voglis", "author_url": "", "post_date": "2021-06-17T05:39:10.880000", "content": "I want to share my personal view as well. As a team we have discussed the possibility of leakage yesterday. It was revealing the \"extra\" digit that made @kazanova 99.99% sure that there was a leakage involved. I took a snapshot of the LB before Marios exposes the leakage revealing the gap from 0.995(#6 place) to is 0.989 (#7 place). I believe, from my 3 week-experience in this competition, that any score >=0.995 it is extremely difficult to appear on public LB by pure CNN modelling without \"meta\" or \"leaked\" information.
\n0.995 AUC score was also set by @inversion as a top target at the end of this competition.
\n\n\nUltimately, though, it's not the value of the metric that matters. This competition is about looking for something we don't know what it looks like. So, we could make the data arbitrarily hard. But that may not actually help in the real search for E.T. It's really about figuring out ways to find things that look different (and not just RFI). So, if the competition ends at 0.995, that still might be a major win for science.
\n
Kudos to my teammate @kazanova for resolving the \"leakage\". He did it in less than 20 minutes because as he told me he knew where to look from previous Kaggle competitions with similar cases. He sets the example for all Kagglers for competition fairness. As for the players who new the leak and chose silence, they should change their mindset. Finding such a leak is not done randomly it requires some special effort in this direction! One does not simply randomly discovers a leak!! 😄😄
\nRevealing such a leak is good for the task we are trying to solve and tights the bonds of Kaggle community!
", "votes": 27, "replies": [] }, { "id": 1353566, "author_name": "Dieter", "author_url": "", "post_date": "2021-06-17T07:11:32.040000", "content": "@inversion I hope kaggle swag for @kazanova is already on the way ;)
", "votes": 18, "replies": [ { "id": 1353874, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-06-17T09:24:27.033000", "content": "A t-shirt with \"Leak detector GM\" written on it. :D
", "votes": 10, "replies": [] }, { "id": 1354418, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2021-06-17T15:25:37.703000", "content": "\n\n@inversion I hope kaggle swag for @kazanova is already on the way ;)
\n
I have them all by now 😛
", "votes": 4, "replies": [] }, { "id": 1354774, "author_name": "Steve Croft", "author_url": "", "post_date": "2021-06-17T21:20:09.687000", "content": "@kazanova I can send you some Berkeley SETI swag if you DM me your mailing address
", "votes": 16, "replies": [] } ] }, { "id": 1353189, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T01:16:15", "content": "Thanks for the post @kazanova. We're looking into this and will give an update shortly.
", "votes": 16, "replies": [] }, { "id": 1353235, "author_name": "Sergey Bryansky", "author_url": "", "post_date": "2021-06-17T02:11:06.910000", "content": "Omg. Impressive finding. But interesting fact, I am not using any information about id and time creation. Just really custom nn! Mb nn exploits this leak implicitly, but i don't think so.
We'll find out when we swap out a new Test set. 😄
", "votes": 12, "replies": [] }, { "id": 1353250, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2021-06-17T02:35:12.877000", "content": "It is very good to hear that. We should not be too skeptic about others, just wait for restarting the competition.
", "votes": 1, "replies": [] }, { "id": 1353256, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T02:37:38.617000", "content": "@tomooinubushi People can continue to build their models with local validation, and even using the Public LB, while we wait for the new Test data.
", "votes": 9, "replies": [] }, { "id": 1353415, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2021-06-17T05:10:49.973000", "content": "Good decision to reset the test data else this competition is utterly useless.
", "votes": 1, "replies": [] }, { "id": 1353427, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-17T05:21:14.947000", "content": "@inversionspaces How much time will it take for the new test dataset ?
", "votes": 0, "replies": [] }, { "id": 1353495, "author_name": "Psi", "author_url": "", "post_date": "2021-06-17T06:23:05.633000", "content": "Your CV is also 0.999+ @sggpls ? - if so then thats impressive. We will see in the re-score I guess. I just hope there aren't any more leaks.
", "votes": 7, "replies": [] }, { "id": 1353883, "author_name": "Sergey Bryansky", "author_url": "", "post_date": "2021-06-17T09:29:31.600000", "content": "Okay, waiting for the new test dataset.
", "votes": 2, "replies": [] }, { "id": 1357027, "author_name": "tensor choko", "author_url": "", "post_date": "2021-06-19T12:18:00.350000", "content": "i want to new test dataset soon.
", "votes": 0, "replies": [] } ] }, { "id": 1353239, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T02:18:34.527000", "content": "I responded in this thread:
\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782
", "votes": 7, "replies": [] }, { "id": 1354194, "author_name": "Giba", "author_url": "", "post_date": "2021-06-17T12:48:21.760000", "content": "Thanks for sharing @kazanova. I thought Kaggle was already \"vaccinated\" against this kind of silly mistake. Now LB becomes useless, people can hide the real scores.
", "votes": 8, "replies": [ { "id": 1354383, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2021-06-17T14:54:25.397000", "content": "The date-in-the-files virus has developed immunity over the years 😛😛
", "votes": 5, "replies": [] } ] }, { "id": 1353188, "author_name": "Yamame🐟", "author_url": "", "post_date": "2021-06-17T01:16:05.453000", "content": "Thank you for reporting the leak.
\nI think someone like you always makes kaggle interesting.
Marios, thanks for explaining the leak. You could have taken advantage of it and keep it for your team.
\nI hope there will be a continuation with new (private) test data.
\nI remember one previous case where processing date (pictures in the 2sigma appartment rental unless mistaken)
\nEdit: @inversion already communicated they will release a new test set. Good move.
", "votes": 5, "replies": [] }, { "id": 1353215, "author_name": "FelipeKitamura, MD, PhD", "author_url": "", "post_date": "2021-06-17T01:47:49.667000", "content": "Thanks, @kazanova, for bringing that up.
", "votes": 3, "replies": [] }, { "id": 1353182, "author_name": "patriot", "author_url": "", "post_date": "2021-06-17T00:55:19.667000", "content": "thanks for sharing.
\nI was disappointed to know this….
Same here - join the club :(
", "votes": 3, "replies": [] } ] }, { "id": 1353196, "author_name": "Fayzur", "author_url": "", "post_date": "2021-06-17T01:29:28.953000", "content": "Much respect @kazanova . Thank you very much for sharing.
\nI thought they are alien, who reached 0.999 before you. :)
@kazanova Thanks for sharing. I hoped that there is need to check metadata for leakage as well. I did not use any metadata to get 0.999 on LB, so there may be another leak in the data itself.
", "votes": 1, "replies": [] }, { "id": 1353925, "author_name": "Athar Sayed", "author_url": "", "post_date": "2021-06-17T09:46:03.937000", "content": "@kazanova Thanks alot for sharing this leakage , Its because of Honest and Seasoned Kagglers like you that Kaggle still remains fair game to some extent . I never thought I would encounter a Leakage at Kaggle Competitions and hence didn't take part explaining Leakage , but next time competing I will look for leaks as well. Thanks alot to Christof Henkel as well his twitter post brought me here.
", "votes": 1, "replies": [] }, { "id": 1353329, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-17T03:58:30.877000", "content": "OMG… thank you for reporting the details of leakage.
\nI hope that updating test set will recover this competition …
", "votes": 1, "replies": [] }, { "id": 1353324, "author_name": "assign", "author_url": "", "post_date": "2021-06-17T03:50:33.163000", "content": "wow…
\nthis challenge related to operating system ..
\nI never think meta data is a key of this challenge to reach 0.999
If you do not make this issue, after challenge, 1st place user's presentation must be \"extract meta data, is my solution\"
\nBecause, if you (or even other user) were aware of the existence of this metadata, surely someone wouldn't have missed auc 1.0
Hello. I have been thinking deeply about this issue.
\nWhy wasn't there an immediate report of a leak? In the past, it would have been reported immediately.
Because Kaggle has become too famous.
\nToday, we all know that data scientists can expect the highest salaries.
\nAnd every company uses Kaggle rank as a reference.
What if you win the competition? Even if you live in the farthest corner of the world, an American company will hire you at the highest salary and you will be able to drive a Ferrari in Silicon Valley.
\nBut what if you don't get a single medal? Yes, you are a worker who finds traffic lights in Mechanical Turk.
They are desperate. They are in it to change their lives.
\nSo you can't expect them to report leaks…unless you have another reward for them.
So I propose that we create a new, fiveth Kaggle rank.
\nHow about giving the title of [DataSearch Master] to anyone who finds a serious leak like this one? Or even Expert if it's not that serious.
\nThat way, leaks of this kind will be reported in no time.
\nAnyway, if there is no reward of some kind, no one will report it. To maintain a community of good will.
no company is going to hire you based on single competition win.
", "votes": 1, "replies": [] }, { "id": 1381927, "author_name": "CPMP", "author_url": "", "post_date": "2021-07-09T11:10:01.987000", "content": "\n\nIn the past, it would have been reported immediately.
\n
There are many examples of Leak not being reported when found in the past. Many.
\nHere Mario reported the leak within 24 hours of suspicion, which is unusually fast.
", "votes": 4, "replies": [] }, { "id": 1381938, "author_name": "Tanrei(nama)", "author_url": "", "post_date": "2021-07-09T11:14:57.527000", "content": "Did I overstate the case?
\nHere's an example. DeNA in Japan hires employees based on their Kaggle rank, and if you make it to the top 3 five times, you get to spend 100% of your work time Kaggling.
\n\nDid I overstate the case?
\n
Which case?
", "votes": 0, "replies": [] }, { "id": 1381975, "author_name": "Alin Cijov", "author_url": "", "post_date": "2021-07-09T12:11:04.890000", "content": "you get to spend 100% of your work time Kaggling
\nFor how long would you be able to give your 100% of your work time Kaggling ? 😄
Thanks @kazanova for highlighting this. I just looked into the competition and suspected something wrong after looking at 0.99 all over the leaderboard.
", "votes": 2, "replies": [] }, { "id": 1353243, "author_name": "Chenglu", "author_url": "", "post_date": "2021-06-17T02:24:21.637000", "content": "Will Kaggle get into this and re-organize the dataset?
", "votes": 2, "replies": [ { "id": 1353247, "author_name": "inversion", "author_url": "", "post_date": "2021-06-17T02:33:16.980000", "content": "Yes. We're working on a new Test set. Stay tuned.
", "votes": 9, "replies": [] } ] }, { "id": 1395700, "author_name": "zizheng", "author_url": "", "post_date": "2021-07-21T13:24:55.297000", "content": "crystal clear
", "votes": 0, "replies": [] }, { "id": 1355594, "author_name": "NicoNiconi", "author_url": "", "post_date": "2021-06-18T11:52:37.567000", "content": "incredible~👀👀👀
", "votes": 0, "replies": [] }, { "id": 1355529, "author_name": "Saurav", "author_url": "", "post_date": "2021-06-18T10:47:14.197000", "content": "Interesting finding! Going forward we may see notebooks like Basic EDA + Leak Check 😁. Anyway, the new data set shouldn't be of much problem for folks not relying on leaks.
", "votes": 0, "replies": [] }, { "id": 1355019, "author_name": "EchoZuluEcho", "author_url": "", "post_date": "2021-06-18T04:14:37.210000", "content": "Awesome work to find this leakage. Wow.
", "votes": 0, "replies": [] }, { "id": 1354079, "author_name": "M.Fauzan Alfariz", "author_url": "", "post_date": "2021-06-17T11:22:38.920000", "content": "Thank you for speaking out to the public by bringing that up this topic. @kazanova
", "votes": 0, "replies": [] }, { "id": 1353472, "author_name": "Yi Wu", "author_url": "", "post_date": "2021-06-17T06:06:28.843000", "content": "sorry, I don't fully understand this. leakage means some images in test dataset are the same as train. I don't how to squeeze more info just with timestamp and id if no this kind of leakage. Is this leakage similar to KNN with timestamp info as clue？someone help me.
Thanks for sharing the details about the leak. 👌
", "votes": 0, "replies": [] }, { "id": 1355653, "author_name": "", "author_url": "", "post_date": "2021-06-18T12:59:48", "content": "", "votes": 0, "replies": [] }, { "id": 1353368, "author_name": "", "author_url": "", "post_date": "2021-06-17T04:33:17.577000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1353175": "The Leak to get to the 0.999 score is explained here .\n\n**It is essentially the date the numpy (.npy) files were processed.**\n\nI have created a kernel that demonstrates how to create and use the leak [here](https://www.kaggle.com/kazanova/leak-submission-lb-0-991) that scores 0.991 . \n\nI have generated the leak and made it a dataset [here](https://www.kaggle.com/kazanova/leak-seti). To generate the leak you download the data locally. I.E it does not work with the data from inside the kernel. \n\nI hate looking for leaks. I never do it unless I am certain there is one and a score of 0.999 signifies that there is definitely one.\n\nI remain loyal to my belief that these leaks should be shared as they can compromise the many good models that other participants may have created. We would hate to win by just using that leak - this is not data science and gives zero value to the organiser. \n\nSadly for kaggle (cc @maggiemd )this is (at minimum) the 3rd time( [1],[2]) that this type of leak has occurred in my kaggle tenure. I am bit disappointed by that. \n\nI am sure kaggle team tries a lot to keep competitions well-regulated, but the same mistakes cannot happen 3 times! **You really need to put some process in place to check for the date the files are processed!!!**\n\nI am also a bit disappointed by the top ranked members NOT revealing the leak when they clearly use it.\n\nHowever I cannot fully blame them because kaggle in the past has shown that leaks are valid and can win competitions.\n\nSoooo.. Where do we go from here?! Restart with non-leaky data.... or fight for the 5th-6th digit?\n\n\n1) https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries/discussion/31870\n2)https://www.kaggle.com/c/dato-native/discussion/16485\n", "1353447": "I want to share my personal view as well. As a team we have discussed the possibility of leakage yesterday. It was revealing the \"extra\" digit that made @kazanova 99.99% sure that there was a leakage involved. I took a snapshot of the LB before Marios exposes the leakage revealing the gap from 0.995(#6 place) to is 0.989 (#7 place). I believe, from my 3 week-experience in this competition, that any score >=0.995 it is extremely difficult to appear on public LB by pure CNN modelling without \"meta\" or \"leaked\" information. \n\n0.995 AUC score was also set by @inversion as a top target at the end of this competition.\n> Ultimately, though, it's not the value of the metric that matters. This competition is about looking for something we don't know what it looks like. So, we could make the data arbitrarily hard. But that may not actually help in the real search for E.T. It's really about figuring out ways to find things that look different (and not just RFI). So, if the competition ends at 0.995, that still might be a major win for science.\n\nKudos to my teammate @kazanova for resolving the \"leakage\". He did it in less than 20 minutes because as he told me he knew where to look from previous Kaggle competitions with similar cases. He sets the example for all Kagglers for competition fairness. As for the players who new the leak and chose silence, they should change their mindset. Finding such a leak is not done randomly it requires some special effort in this direction! One does not simply randomly discovers a leak!! 😄😄\n\nRevealing such a leak is good for the task we are trying to solve and tights the bonds of Kaggle community!\n", "1353566": "@inversion I hope kaggle swag for @kazanova is already on the way ;)", "1353189": "Thanks for the post @kazanova. We're looking into this and will give an update shortly.", "1353235": "Omg. Impressive finding. But interesting fact, I am not using any information about `id` and time creation. Just really custom nn! Mb nn exploits this leak *implicitly*, but i don't think so.", "1353239": "I responded in this thread:\n\nhttps://www.kaggle.com/c/seti-breakthrough-listen/discussion/246782", "1354194": "Thanks for sharing @kazanova. I thought Kaggle was already \"vaccinated\" against this kind of silly mistake. Now LB becomes useless, people can hide the real scores.", "1353188": "Thank you for reporting the leak.\nI think someone like you always makes kaggle interesting.", "1353689": "Marios, thanks for explaining the leak. You could have taken advantage of it and keep it for your team.\n\nI hope there will be a continuation with new (private) test data.\n\nI remember one previous case where processing date (pictures in the 2sigma appartment rental unless mistaken)\n\nEdit: @inversion already communicated they will release a new test set. Good move.", "1353215": "Thanks, @kazanova, for bringing that up. ", "1353182": "thanks for sharing.\nI was disappointed to know this....", "1353196": "Much respect @kazanova . Thank you very much for sharing. \nI thought they are alien, who reached 0.999 before you. :) ", "1354168": "@kazanova Thanks for sharing. I hoped that there is need to check metadata for leakage as well. I did not use any metadata to get 0.999 on LB, so there may be another leak in the data itself.", "1353925": "@kazanova Thanks alot for sharing this leakage , Its because of Honest and Seasoned Kagglers like you that Kaggle still remains fair game to some extent . I never thought I would encounter a Leakage at Kaggle Competitions and hence didn't take part explaining Leakage , but next time competing I will look for leaks as well. Thanks alot to Christof Henkel as well his twitter post brought me here.", "1353329": "OMG... thank you for reporting the details of leakage.\n\nI hope that updating test set will recover this competition ...", "1353324": "wow...\nthis challenge related to operating system ..\nI never think meta data is a key of this challenge to reach 0.999\n\nIf you do not make this issue, after challenge, 1st place user's presentation must be \"extract meta data, is my solution\"\nBecause, if you (or even other user) were aware of the existence of this metadata, surely someone wouldn't have missed auc 1.0", "1381912": "Hello. I have been thinking deeply about this issue.\nWhy wasn't there an immediate report of a leak? In the past, it would have been reported immediately.\n\nBecause Kaggle has become too famous.\n\nToday, we all know that data scientists can expect the highest salaries.\nAnd every company uses Kaggle rank as a reference.\n\nWhat if you win the competition? Even if you live in the farthest corner of the world, an American company will hire you at the highest salary and you will be able to drive a Ferrari in Silicon Valley.\nBut what if you don't get a single medal? Yes, you are a worker who finds traffic lights in Mechanical Turk.\n\nThey are desperate. They are in it to change their lives.\nSo you can't expect them to report leaks...unless you have another reward for them.\n\nSo I propose that we create a new, fiveth Kaggle rank.\nHow about giving the title of [DataSearch Master] to anyone who finds a serious leak like this one? Or even Expert if it's not that serious.\nThat way, leaks of this kind will be reported in no time.\nAnyway, if there is no reward of some kind, no one will report it. To maintain a community of good will.\n", "1353245": "Thanks @kazanova for highlighting this. I just looked into the competition and suspected something wrong after looking at 0.99 all over the leaderboard. ", "1353243": "Will Kaggle get into this and re-organize the dataset?", "1395700": "crystal clear", "1355594": "incredible~👀👀👀", "1355529": "Interesting finding! Going forward we may see notebooks like Basic EDA + Leak Check 😁. Anyway, the new data set shouldn't be of much problem for folks not relying on leaks. ", "1355019": "Awesome work to find this leakage. Wow.", "1354079": "Thank you for speaking out to the public by bringing that up this topic. @kazanova ", "1353472": "sorry, I don't fully understand this. leakage means some images in test dataset are the same as train. I don't how to squeeze more info just with `timestamp` and `id` if no this kind of leakage. Is this leakage similar to KNN with `timestamp` info as clue？someone help me.", "1353353": "Thanks for sharing the details about the leak. 👌 ", "1355653": "", "1353368": "" } }