The near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find a typo in the title, I guess there is no way to fix that lol
\n\nUpdate: I didn't expect people to get so angry (a few at me, I noticed I got some downvote) and disappointed about the issue. I have mixed feelings for this. But I don't remember I got so upset by not knowing the leakage of tube price ID after someone point it out after that competition was over, that one has more than 1000 players, yet only a few complaints, so I guess fewer upset player. So now I understand why people don't want to come out and confirm anything, because they want people to be happy. Sorry if I made people disappointed or upset by posting this topic.
", "messages": [ { "id": "92249", "postDate": "09/12/2015 14:22:22", "content": "The near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find a typo in the title, I guess there is no way to fix that lol
\n\nUpdate: I didn't expect people to get so angry (a few at me, I noticed I got some downvote) and disappointed about the issue. I have mixed feelings for this. But I don't remember I got so upset by not knowing the leakage of tube price ID after someone point it out after that competition was over, that one has more than 1000 players, yet only a few complaints, so I guess fewer upset player. So now I understand why people don't want to come out and confirm anything, because they want people to be happy. Sorry if I made people disappointed or upset by posting this topic.
", "rawMarkdown": "The near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.\r\n\r\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too. \r\n\r\nEdit: sorry, just find a typo in the title, I guess there is no way to fix that lol\r\n\r\nUpdate: I didn't expect people to get so angry (a few at me, I noticed I got some downvote) and disappointed about the issue. I have mixed feelings for this. But I don't remember I got so upset by not knowing the leakage of tube price ID after someone point it out after that competition was **over**, that one has more than 1000 players, yet only a few complaints, so I guess fewer upset player. So now I understand why people don't want to come out and confirm anything, because they want people to be happy. Sorry if I made people disappointed or upset by posting this topic.", "votes": null }, { "id": "92252", "postDate": "09/12/2015 15:00:08", "content": "[quote=SkyLibrary;92249]
\n\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find typo in title, guess this is no way to fix that lol
\n\n[/quote]
\n\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board.
\n\nI hope you can understand that I don't want to be involve in controversy every time :)
", "rawMarkdown": "[quote=SkyLibrary;92249]\r\n\r\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.\r\n\r\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too. \r\n\r\nEdit: sorry, just find typo in title, guess this is no way to fix that lol\r\n\r\n[/quote]\r\n\r\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board. \r\n\r\nI hope you can understand that I don't want to be involve in controversy every time :)", "votes": null }, { "id": "92254", "postDate": "09/12/2015 15:30:35", "content": "Just wait few weeks :)
", "rawMarkdown": "Just wait few weeks :)", "votes": null }, { "id": "92257", "postDate": "09/12/2015 16:21:59", "content": "It is pretty much a yes or no question, if you guys don't want to answer it, then I will assume the answer is yes.
", "rawMarkdown": "It is pretty much a yes or no question, if you guys don't want to answer it, then I will assume the answer is yes.", "votes": null }, { "id": "92276", "postDate": "09/12/2015 19:50:26", "content": "Why would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though.
", "rawMarkdown": "Why would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though.", "votes": null }, { "id": "92280", "postDate": "09/12/2015 20:26:16", "content": "I didn't say they should answer my question, but I have the right to post the question no matter they reply or not. If they do, it will confirm your 'highly likely', if not, then it is still highly likely :)
\n\n[quote=Giulio;92276]
\n\nWhy would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though.
\n\n[/quote]
", "rawMarkdown": "I didn't say they should answer my question, but I have the right to post the question no matter they reply or not. If they do, it will confirm your 'highly likely', if not, then it is still highly likely :)\r\n\r\n[quote=Giulio;92276]\r\n\r\nWhy would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though. \r\n\r\n[/quote]", "votes": null }, { "id": "92285", "postDate": "09/12/2015 22:48:30", "content": "My guess is:
\n\nThey are running an unsupervised algorithm over both training and test that generates relationships between elements in both datasets. Let's call this a vocabulary. This thus creates highly abstract relationships between all documents at word level, but the difference now is that this is across both train and test.
\n\nThen for training, create (meaningful) document vectors from this vocabulary and start training to indicate sponsored / not sponsored. The difference is that you're not training on occurrences of individual features of a document (which can be meaningful or not), but you're training against the known vector space of all documents (with 1/3 of the data).
", "rawMarkdown": "My guess is:\r\n\r\nThey are running an unsupervised algorithm over both training and test that generates relationships between elements in both datasets. Let's call this a vocabulary. This thus creates highly abstract relationships between all documents at word level, but the difference now is that this is across both train and test.\r\n\r\nThen for training, create (meaningful) document vectors from this vocabulary and start training to indicate sponsored / not sponsored. The difference is that you're not training on occurrences of individual features of a document (which can be meaningful or not), but you're training against the known vector space of all documents (with 1/3 of the data).", "votes": null }, { "id": "92294", "postDate": "09/13/2015 01:05:02", "content": "[quote=DataGeek;92252]
\n\n[quote=SkyLibrary;92249]
\n\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find typo in title, guess this is no way to fix that lol
\n\n[/quote]
\n\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board.
\n\nI hope you can understand that I don't want to be involve in controversy every time :)
\n\n[/quote]
\n\nI think more and more people are getting offended by this, and joining the 0.99 club. :)
", "rawMarkdown": "[quote=DataGeek;92252]\r\n\r\n[quote=SkyLibrary;92249]\r\n\r\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.\r\n\r\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too. \r\n\r\nEdit: sorry, just find typo in title, guess this is no way to fix that lol\r\n\r\n[/quote]\r\n\r\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board. \r\n\r\nI hope you can understand that I don't want to be involve in controversy every time :)\r\n\r\n[/quote]\r\n\r\nI think more and more people are getting offended by this, and joining the 0.99 club. :)", "votes": null }, { "id": "92299", "postDate": "09/13/2015 03:07:39", "content": "After download the data and play with it for 5 hours, I am here to confirm the answer is yes.
", "rawMarkdown": "After download the data and play with it for 5 hours, I am here to confirm the answer is yes.", "votes": null }, { "id": "92301", "postDate": "09/13/2015 04:32:35", "content": "Well this is fun, and here I thought this would be the one competition that is actually like the old days since the data is too large for scripts. Silly me.
", "rawMarkdown": "Well this is fun, and here I thought this would be the one competition that is actually like the old days since the data is too large for scripts. Silly me.", "votes": null }, { "id": "92303", "postDate": "09/13/2015 05:37:23", "content": "Interesting. I wonder if Kaggle will put a hold on this or just let it be it.
", "rawMarkdown": "Interesting. I wonder if Kaggle will put a hold on this or just let it be it.", "votes": null }, { "id": "92304", "postDate": "09/13/2015 05:55:33", "content": "I am a noob in machine learning. I played with natural language processing algorithms. I achieved .76 score then I found that beside language processing, we can have better result if we used other features in the HTML files such as: javascript, meta tag attribute...
", "rawMarkdown": "I am a noob in machine learning. I played with natural language processing algorithms. I achieved .76 score then I found that beside language processing, we can have better result if we used other features in the HTML files such as: javascript, meta tag attribute...", "votes": null }, { "id": "92305", "postDate": "09/13/2015 06:04:10", "content": "So there is little space for the improvement after you applying the leakage information into your model.
", "rawMarkdown": "So there is little space for the improvement after you applying the leakage information into your model.", "votes": null }, { "id": "92308", "postDate": "09/13/2015 06:46:58", "content": "For those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging.
", "rawMarkdown": "For those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging.", "votes": null }, { "id": "92313", "postDate": "09/13/2015 09:28:41", "content": "[quote=Dmitry Ulyanov;92308]
\n\nFor those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging.
\n\n[/quote]
\n\nBut now you must know this "perfect predictor", which it's a leakage and not real data, in order to be successful in this competition.
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
", "rawMarkdown": "[quote=Dmitry Ulyanov;92308]\r\n\r\nFor those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging. \r\n\r\n[/quote]\r\n\r\nBut now you must know this \"perfect predictor\", which it's a leakage and not real data, in order to be successful in this competition.\r\n\r\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.", "votes": null }, { "id": "92314", "postDate": "09/13/2015 09:29:01", "content": "This is unfair to WNV or Cat competitors to say that leakage exploitation was the only determinant in victory.
\nMany did exploit it (even myself in WNV, not knowing it was, while in Cat I discarded it believing I would make a methodology mistake), and we did not all achieve top scores.
Leakage exploitation still asks for talent and skills.\nAlso, nothing tells that what you call leakage, as I said before, is just a very strong association rule in the dataset and was present from the start.
", "rawMarkdown": "This is unfair to WNV or Cat competitors to say that leakage exploitation was the only determinant in victory. \r\nMany did exploit it (even myself in WNV, not knowing it was, while in Cat I discarded it believing I would make a methodology mistake), and we did not all achieve top scores.\r\n\r\nLeakage exploitation still asks for talent and skills.\r\nAlso, nothing tells that what you call leakage, as I said before, is just a very strong association rule in the dataset and was present from the start.", "votes": null }, { "id": "92321", "postDate": "09/13/2015 13:18:19", "content": "My code that is looking at what I think is the leak is apparently very slow, lol.
", "rawMarkdown": "My code that is looking at what I think is the leak is apparently very slow, lol.", "votes": null }, { "id": "92322", "postDate": "09/13/2015 13:27:26", "content": "I thought that, since an ad is "sponsored", you'd find some kind of link or script in the files across all the sponsored ones (tracking links), but it isn't as easy as that. Just filtering on _gac for example gave me 0.546 and non-sponsored ones contain the same thing. But I reckon it must be somewhere along those lines...
", "rawMarkdown": "I thought that, since an ad is \"sponsored\", you'd find some kind of link or script in the files across all the sponsored ones (tracking links), but it isn't as easy as that. Just filtering on _gac for example gave me 0.546 and non-sponsored ones contain the same thing. But I reckon it must be somewhere along those lines...", "votes": null }, { "id": "92324", "postDate": "09/13/2015 13:52:50", "content": "[quote=SkyLibrary;92299]
\n\nAfter download the data and play with it for 5 hours, I am here to confirm the answer is yes.
\n\n[/quote]\nSo this is a leakage, and you have found it?
", "rawMarkdown": "[quote=SkyLibrary;92299]\r\n\r\nAfter download the data and play with it for 5 hours, I am here to confirm the answer is yes.\r\n\r\n[/quote]\r\nSo this is a leakage, and you have found it?", "votes": null }, { "id": "92326", "postDate": "09/13/2015 14:33:21", "content": "[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
", "rawMarkdown": "[quote=clustifier;92313]\r\n\r\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.\r\n\r\n[/quote]\r\n\r\nEven with this leak it is still fair for players.\r\n\r\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.", "votes": null }, { "id": "92327", "postDate": "09/13/2015 14:38:01", "content": "I am sure that there is a leakage that can be used as a "golden feature", but only using this can not reach 1.0 on the public leader board (0.99805). Or, as pointed out on the other thread, there are some errors in this dataset so the "golden feature" is perfect but labels are wrong.
\n\nI want to know what the admin is thinking about this since further explorer might be in vain.
", "rawMarkdown": "I am sure that there is a leakage that can be used as a \"golden feature\", but only using this can not reach 1.0 on the public leader board (0.99805). Or, as pointed out on the other thread, there are some errors in this dataset so the \"golden feature\" is perfect but labels are wrong.\r\n\r\nI want to know what the admin is thinking about this since further explorer might be in vain.", "votes": null }, { "id": "92331", "postDate": "09/13/2015 14:59:08", "content": "[quote=rcarson;92326]
\n\n[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
\n\n[/quote]
\n\nDefine "fair". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
", "rawMarkdown": "[quote=rcarson;92326]\r\n\r\n[quote=clustifier;92313]\r\n\r\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.\r\n\r\n[/quote]\r\n\r\nEven with this leak it is still fair for players.\r\n\r\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.\r\n\r\n\r\n[/quote]\r\n\r\nDefine \"fair\". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.", "votes": null }, { "id": "92332", "postDate": "09/13/2015 15:11:02", "content": "I guess this has something to do with scripts inside html files.
\n\nFor example, the function of ads' charging. But I'm not familiar with javascript unfortunately, lol.
", "rawMarkdown": "I guess this has something to do with scripts inside html files.\r\n\r\nFor example, the function of ads' charging. But I'm not familiar with javascript unfortunately, lol.", "votes": null }, { "id": "92333", "postDate": "09/13/2015 15:17:04", "content": "Stumbleupon doesn't require any additions to html page.
", "rawMarkdown": "Stumbleupon doesn't require any additions to html page.", "votes": null }, { "id": "92334", "postDate": "09/13/2015 15:23:01", "content": "[quote=David McGarry;92331]
\n\n[quote=rcarson;92326]
\n\n[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
\n\n[/quote]
\n\nDefine "fair". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm not stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
\n\n[/quote]
\n\nOn the contrary, I found it fun to look for the leak ( I haven't found it yet).
\n\nA little bit off the topic, I think a great part of kaggle competitions is not about machine learning. It is more about data mining. Finding soft or strong leak (good or golden features) is usually more dominant, given there are so many good out-of-box tools like xgboost for the learning part.
\n\nIn my opinion, finding a leak is a great skill even in real world case. But I agree in this case, it is a mess that the leak is too strong..
", "rawMarkdown": "[quote=David McGarry;92331]\r\n\r\n[quote=rcarson;92326]\r\n\r\n[quote=clustifier;92313]\r\n\r\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.\r\n\r\n[/quote]\r\n\r\nEven with this leak it is still fair for players.\r\n\r\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.\r\n\r\n\r\n[/quote]\r\n\r\nDefine \"fair\". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm not stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.\r\n\r\n[/quote]\r\n\r\nOn the contrary, I found it fun to look for the leak ( I haven't found it yet). \r\n\r\nA little bit off the topic, I think a great part of kaggle competitions is not about machine learning. It is more about data mining. Finding soft or strong leak (good or golden features) is usually more dominant, given there are so many good out-of-box tools like xgboost for the learning part.\r\n\r\nIn my opinion, finding a leak is a great skill even in real world case. But I agree in this case, it is a mess that the leak is too strong..", "votes": null }, { "id": "92336", "postDate": "09/13/2015 15:26:43", "content": "I think i found a leakage, but it doesn't improve my score. LOL.
", "rawMarkdown": "I think i found a leakage, but it doesn't improve my score. LOL.", "votes": null }, { "id": "92340", "postDate": "09/13/2015 16:35:16", "content": "Ok, let's play this out as if it was a real project.
\n\nBusiness guy: we think we could really benefit from being able to use ML to classify pages with native add content.
\n\nData Scientist: give me a week to look at the data and I will get back to you with a first evaluation of what I think is possible. I'll work with engineering to get some data.
\n\n...a week later...
\n\n(very excited, yet skeptical) Data Scientist: I think I found something in the data that allows a perfect prediction. This is XYZ feature.
\n\nBusiness Guy: oh, well, that makes sense. Seems kind of obvious. Wonder what "name of a peer/manager/director who is taking the heat" thought when he came up with the idea of using ML for this... You know what, I don't think we need ML anymore.
\n\n(disappointed) Data Scientist: well, I guess I'll go back to building that boring dashboard then. Too bad, I was so excited of finally using some of those ML skills I had learnt on Kaggle...
\n\nMy point, when a perfect predictor exists, there is no reason to use ML because it is not an ML problem. This should have never made it to be an ML project/competition. Now, will the finding of XYZ feature be of value to Dato? Who knows? I would be surprised if it does. Will it still be interesting to try to maximize performance on misslabeld observations? I mean, why not, it's just a problem like any other. But, wow, not the problem I thought I was signing up for.\nGlad I haven't made a submission yet...
", "rawMarkdown": "Ok, let's play this out as if it was a real project.\r\n\r\nBusiness guy: we think we could really benefit from being able to use ML to classify pages with native add content.\r\n\r\nData Scientist: give me a week to look at the data and I will get back to you with a first evaluation of what I think is possible. I'll work with engineering to get some data.\r\n\r\n...a week later...\r\n\r\n(very excited, yet skeptical) Data Scientist: I think I found something in the data that allows a perfect prediction. This is XYZ feature.\r\n\r\nBusiness Guy: oh, well, that makes sense. Seems kind of obvious. Wonder what \"name of a peer/manager/director who is taking the heat\" thought when he came up with the idea of using ML for this... You know what, I don't think we need ML anymore.\r\n\r\n(disappointed) Data Scientist: well, I guess I'll go back to building that boring dashboard then. Too bad, I was so excited of finally using some of those ML skills I had learnt on Kaggle...\r\n\r\n\r\nMy point, when a perfect predictor exists, there is no reason to use ML because it is not an ML problem. This should have never made it to be an ML project/competition. Now, will the finding of XYZ feature be of value to Dato? Who knows? I would be surprised if it does. Will it still be interesting to try to maximize performance on misslabeld observations? I mean, why not, it's just a problem like any other. But, wow, not the problem I thought I was signing up for.\r\nGlad I haven't made a submission yet...", "votes": null }, { "id": "92341", "postDate": "09/13/2015 16:39:06", "content": "[quote=David McGarry;92331]\n I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
\n\n[/quote]
\n\n+1 for (c)
", "rawMarkdown": "[quote=David McGarry;92331]\r\n I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.\r\n\r\n[/quote]\r\n\r\n+1 for (c)", "votes": null }, { "id": "92342", "postDate": "09/13/2015 16:41:24", "content": "Giulio, it's much worse than that... much worse :) one of those things: "soon as you see it..."
\n\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. I had been trying to improve from 0.951 before this without success. That by itself is already a very reasonable score I think.
", "rawMarkdown": "Giulio, it's much worse than that... much worse :) one of those things: \"soon as you see it...\" \r\n\r\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. I had been trying to improve from 0.951 before this without success. That by itself is already a very reasonable score I think.", "votes": null }, { "id": "92343", "postDate": "09/13/2015 16:44:13", "content": "In the end, don't tell me it is a filename leak... ==
", "rawMarkdown": "In the end, don't tell me it is a filename leak... ==", "votes": null }, { "id": "92344", "postDate": "09/13/2015 16:51:18", "content": "[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
", "rawMarkdown": "[quote=Gerard Toonstra;92342]\r\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]\r\n\r\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.", "votes": null }, { "id": "92345", "postDate": "09/13/2015 16:56:13", "content": "[quote=NxGTR;92344]
\n\n[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
\n\n[/quote]
\n\nI'm sad too, especially because of time I spent on it in the last few weeks.
", "rawMarkdown": "[quote=NxGTR;92344]\r\n\r\n[quote=Gerard Toonstra;92342]\r\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]\r\n\r\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.\r\n\r\n[/quote]\r\n\r\nI'm sad too, especially because of time I spent on it in the last few weeks.", "votes": null }, { "id": "92348", "postDate": "09/13/2015 17:19:20", "content": "[quote=NxGTR;92344]
\n\n[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
\n\n[/quote]
\n\nI'm looking forward to an explanation of your approach, parametrizations and feature choices, if you decide to write it up.
", "rawMarkdown": "[quote=NxGTR;92344]\r\n\r\n[quote=Gerard Toonstra;92342]\r\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]\r\n\r\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.\r\n\r\n[/quote]\r\n\r\nI'm looking forward to an explanation of your approach, parametrizations and feature choices, if you decide to write it up.", "votes": null }, { "id": "92349", "postDate": "09/13/2015 19:07:04", "content": "The date in the zip files looks suspect.
", "rawMarkdown": "The date in the zip files looks suspect.", "votes": null }, { "id": "92350", "postDate": "09/13/2015 19:25:14", "content": "yes, it looks like the 1's files were written only on June 24 or July 27th.
", "rawMarkdown": "yes, it looks like the 1's files were written only on June 24 or July 27th.", "votes": null }, { "id": "92353", "postDate": "09/13/2015 19:55:58", "content": "[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
", "rawMarkdown": "[quote=Lawrence Chernin;92350]\r\n\r\nyes, it looks like the 1's files were written only on June 24 or July 27th.\r\n\r\n[/quote]\r\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.", "votes": null }, { "id": "92355", "postDate": "09/13/2015 20:02:11", "content": "[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n", "rawMarkdown": "[quote=Little Boat;92353]\r\n\r\n[quote=Lawrence Chernin;92350]\r\n\r\nyes, it looks like the 1's files were written only on June 24 or July 27th.\r\n\r\n[/quote]\r\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time. \r\n\r\n[/quote]\r\nThere is nothing to remove :)\r\n\r\n> StumbleUpon Ads is a native advertising platform that does not use\r\n> traditional ad units such as banners or text links. This means that on\r\n> StumbleUpon your ad is, quite simply, your URL. Since your web page\r\n> itself is the ad unit, no additional creative assets or copy are\r\n> needed to drive traffic. We drive directly to your page when a user\r\n> clicks the Stumble button.\r\n\r\n\r\n> http://ads.stumbleupon.com/wp-content/uploads/2014/09/CampaignCreationGuide.pdf", "votes": null }, { "id": "92356", "postDate": "09/13/2015 20:05:35", "content": "StumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n \nhttp://ads.stumbleupon.com/wp-content/uploads/2014/09/CampaignCreationGuide.pdf
\n
Hmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)
\n\n[quote=Artem;92355]
\n\n[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n\n\nStumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n
[/quote]
", "rawMarkdown": "Hmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)\r\n\r\n[quote=Artem;92355]\r\n\r\n[quote=Little Boat;92353]\r\n\r\n[quote=Lawrence Chernin;92350]\r\n\r\nyes, it looks like the 1's files were written only on June 24 or July 27th.\r\n\r\n[/quote]\r\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time. \r\n\r\n[/quote]\r\nThere is nothing to remove :)\r\n\r\n> StumbleUpon Ads is a native advertising platform that does not use\r\n> traditional ad units such as banners or text links. This means that on\r\n> StumbleUpon your ad is, quite simply, your URL. Since your web page\r\n> itself is the ad unit, no additional creative assets or copy are\r\n> needed to drive traffic. We drive directly to your page when a user\r\n> clicks the Stumble button.\r\n\r\n[/quote]", "votes": null }, { "id": "92357", "postDate": "09/13/2015 20:11:46", "content": "[quote=Little Boat;92356]
\n\nHmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)
\n\n[quote=Artem;92355]
\n\n[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n\n\nStumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n
[/quote]\n[/quote]\nI checked it locally and it looks like true.
\n\nAlso i found another "leakage" that can give 0.86+ without ml.
", "rawMarkdown": "[quote=Little Boat;92356]\r\n\r\nHmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)\r\n\r\n[quote=Artem;92355]\r\n\r\n[quote=Little Boat;92353]\r\n\r\n[quote=Lawrence Chernin;92350]\r\n\r\nyes, it looks like the 1's files were written only on June 24 or July 27th.\r\n\r\n[/quote]\r\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time. \r\n\r\n[/quote]\r\nThere is nothing to remove :)\r\n\r\n> StumbleUpon Ads is a native advertising platform that does not use\r\n> traditional ad units such as banners or text links. This means that on\r\n> StumbleUpon your ad is, quite simply, your URL. Since your web page\r\n> itself is the ad unit, no additional creative assets or copy are\r\n> needed to drive traffic. We drive directly to your page when a user\r\n> clicks the Stumble button.\r\n\r\n[/quote]\r\n[/quote]\r\nI checked it locally and it looks like true.\r\n\r\n\r\nAlso i found another \"leakage\" that can give 0.86+ without ml.", "votes": null }, { "id": "92358", "postDate": "09/13/2015 20:13:48", "content": "There is definitely leakage and I don't see how this could be solved/repaired, short of giving us a completely new data set...
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
", "rawMarkdown": "There is definitely leakage and I don't see how this could be solved/repaired, short of giving us a completely new data set...\r\n\r\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.", "votes": null }, { "id": "92359", "postDate": "09/13/2015 20:14:50", "content": "[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]
\n\nWell that's fun. Hopefully they clear the LB and release a new dataset (using new HTML files would be ideal, but at least renaming all of the existing HTML files would discourage all but the least honest to ignore the known values in the test set).
", "rawMarkdown": "[quote=Lawrence Chernin;92350]\r\n\r\nyes, it looks like the 1's files were written only on June 24 or July 27th.\r\n\r\n[/quote]\r\n\r\nWell that's fun. Hopefully they clear the LB and release a new dataset (using new HTML files would be ideal, but at least renaming all of the existing HTML files would discourage all but the least honest to ignore the known values in the test set).", "votes": null }, { "id": "92362", "postDate": "09/13/2015 20:45:35", "content": "Now I can confirm that the date is the leakage.
", "rawMarkdown": "Now I can confirm that the date is the leakage.", "votes": null }, { "id": "92363", "postDate": "09/13/2015 20:46:47", "content": "[quote=Triskelion;92358]
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
\n\n[/quote]
\n\nHow about the tiniest of of edges that is NOT worth competing for.
", "rawMarkdown": "[quote=Triskelion;92358]\r\n\r\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.\r\n\r\n[/quote]\r\n\r\nHow about the tiniest of of edges that is NOT worth competing for.", "votes": null }, { "id": "92366", "postDate": "09/13/2015 21:25:58", "content": "The one I was looking at was an unusual amount of the file name numbers showing up in other files, but hadn't looked hard at it yet - the code is slow.\nMaybe just genuinely random, but seems odd if you look at it.
", "rawMarkdown": "The one I was looking at was an unusual amount of the file name numbers showing up in other files, but hadn't looked hard at it yet - the code is slow.\r\nMaybe just genuinely random, but seems odd if you look at it.", "votes": null }, { "id": "92373", "postDate": "09/13/2015 22:42:13", "content": "A small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
", "rawMarkdown": "A small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.", "votes": null }, { "id": "92376", "postDate": "09/13/2015 23:00:46", "content": "[quote=senbong;92373]
\n\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
\n\n[/quote]
\n\nMmm... what about incentives to players whose workflow is not:
\n\n1) Lets find some leaks
\n\n2) Do actual ML
\n\n:D
", "rawMarkdown": "[quote=senbong;92373]\r\n\r\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.\r\n\r\n[/quote]\r\n\r\nMmm... what about incentives to players whose workflow is not:\r\n\r\n1) Lets find some leaks\r\n\r\n2) Do actual ML\r\n\r\n:D", "votes": null }, { "id": "92377", "postDate": "09/13/2015 23:11:06", "content": "[quote=NxGTR;92376]
\n\n[quote=senbong;92373]
\n\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
\n\n[/quote]
\n\nMmm... what about incentives to players whose workflow is not:
\n\n1) Lets find some leaks
\n\n2) Do actual ML
\n\n:D
\n\n[/quote]
\n\nIn my opinion, detecting this type of problem earlier is very important. The worst case is that someone keeps this leak to the later stage and uses it to win a competition.
", "rawMarkdown": "[quote=NxGTR;92376]\r\n\r\n[quote=senbong;92373]\r\n\r\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.\r\n\r\n[/quote]\r\n\r\nMmm... what about incentives to players whose workflow is not:\r\n\r\n1) Lets find some leaks\r\n\r\n2) Do actual ML\r\n\r\n:D\r\n\r\n\r\n[/quote]\r\n\r\n\r\nIn my opinion, detecting this type of problem earlier is very important. The worst case is that someone keeps this leak to the later stage and uses it to win a competition.", "votes": null }, { "id": "92379", "postDate": "09/13/2015 23:43:34", "content": "If the competition is stopped and isn't rerun then what might be nice is the offer of some sort of small reward (swag perhaps?) for those who attained a good score without the use of the "file_modified"
\n\nI only really entered this competition to try large scale analysis for the first time and learn from other people. It would be a shame for other people's work to go to waste simply due to a massive data leak.
", "rawMarkdown": "If the competition is stopped and isn't rerun then what might be nice is the offer of some sort of small reward (swag perhaps?) for those who attained a good score without the use of the \"file_modified\" \r\n\r\nI only really entered this competition to try large scale analysis for the first time and learn from other people. It would be a shame for other people's work to go to waste simply due to a massive data leak.", "votes": null }, { "id": "92380", "postDate": "09/13/2015 23:45:10", "content": "[quote=Giulio;92363]
\n\n[quote=Triskelion;92358]
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
\n\n[/quote]
\n\nHow about the tiniest of of edges that is NOT worth competing for.
\n\n[/quote]
\n\nLooks to me the 10.000$ is still up for the taking. Kinda agree with your sentiment though.
", "rawMarkdown": "[quote=Giulio;92363]\r\n\r\n[quote=Triskelion;92358]\r\n\r\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.\r\n\r\n[/quote]\r\n\r\nHow about the tiniest of of edges that is NOT worth competing for.\r\n\r\n[/quote]\r\n\r\nLooks to me the 10.000$ is still up for the taking. Kinda agree with your sentiment though.", "votes": null }, { "id": "92389", "postDate": "09/14/2015 03:55:03", "content": "If the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid?
\n\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.
\n\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.
\n\nAnother good solution is to replace the original data set with a normal one.
\n\nThe worst solution is that you know it's harmful to the contest, and just let it go.
", "rawMarkdown": "If the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid? \r\n\r\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.\r\n\r\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.\r\n\r\nAnother good solution is to replace the original data set with a normal one.\r\n\r\nThe worst solution is that you know it's harmful to the contest, and just let it go.", "votes": null }, { "id": "92392", "postDate": "09/14/2015 04:38:22", "content": "[quote=YcdoiT;92389]
\n\nIf the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid?
\n\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.
\n\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.
\n\nAnother good solution is to replace the original data set with a normal one.
\n\nThe worst solution is that you know it's harmful to the contest, and just let it go.
\n\n[/quote]
\n\nActually it will be quite difficult to go through everyone's codes to see if they used date variable or not. Alternatively, they can make two leaderboards, one just before leakage and one after and at least award the ranks/points, whichever is maximum of the two LB's.
", "rawMarkdown": "[quote=YcdoiT;92389]\r\n\r\nIf the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid? \r\n\r\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.\r\n\r\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.\r\n\r\nAnother good solution is to replace the original data set with a normal one.\r\n\r\nThe worst solution is that you know it's harmful to the contest, and just let it go. \r\n\r\n[/quote]\r\n\r\nActually it will be quite difficult to go through everyone's codes to see if they used date variable or not. Alternatively, they can make two leaderboards, one just before leakage and one after and at least award the ranks/points, whichever is maximum of the two LB's.", "votes": null }, { "id": "92400", "postDate": "09/14/2015 05:23:38", "content": "@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
", "rawMarkdown": "Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\r\nSo, maybe replacing total data set is a good idea.", "votes": null }, { "id": "92402", "postDate": "09/14/2015 05:28:11", "content": "Yeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.
", "rawMarkdown": "Yeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.", "votes": null }, { "id": "92405", "postDate": "09/14/2015 05:33:26", "content": "[quote=Subhajit Mandal;92402]
\n\nYeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.
\n\n[/quote]
\n\nI think a new test dataset should be enough. We can still use the downloaded data as the training data.
", "rawMarkdown": "[quote=Subhajit Mandal;92402]\r\n\r\nYeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.\r\n\r\n[/quote]\r\n\r\nI think a new test dataset should be enough. We can still use the downloaded data as the training data.", "votes": null }, { "id": "92408", "postDate": "09/14/2015 05:39:15", "content": "It looked like this competition was organized very well: lots of data left for testing - fairer scoring.
\n\nIt is a unfortunate that labels were leaked and now it will have to be cancelled.
\n\nMaybe organizers could save the competition, by coming up with more unseen test data and converting the data given so far to a "train" set?
", "rawMarkdown": "It looked like this competition was organized very well: lots of data left for testing - fairer scoring.\r\n\r\nIt is a unfortunate that labels were leaked and now it will have to be cancelled.\r\n\r\nMaybe organizers could save the competition, by coming up with more unseen test data and converting the data given so far to a \"train\" set?", "votes": null }, { "id": "92424", "postDate": "09/14/2015 06:14:10", "content": "[quote=YcdoiT;92400]
\n\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
\n\nYeah, whatever, at least 0.99805 of the test set was leaked.\n[/quote]
", "rawMarkdown": "[quote=YcdoiT;92400]\r\n\r\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\r\nSo, maybe replacing total data set is a good idea.\r\n\r\nYeah, whatever, at least 0.99805 of the test set was leaked.\r\n[/quote]", "votes": null }, { "id": "92425", "postDate": "09/14/2015 06:14:41", "content": "[quote=YcdoiT;92400]
\n\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
\n\n[/quote]
\n\nYeah, whatever, at least 0.99805 of the test set was leaked.
", "rawMarkdown": "[quote=YcdoiT;92400]\r\n\r\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\r\nSo, maybe replacing total data set is a good idea.\r\n\r\n[/quote]\r\n\r\nYeah, whatever, at least 0.99805 of the test set was leaked.", "votes": null }, { "id": "92476", "postDate": "09/14/2015 13:55:35", "content": "I'll vote for a restart with a new test data set.
", "rawMarkdown": "I'll vote for a restart with a new test data set.", "votes": null }, { "id": "92478", "postDate": "09/14/2015 14:15:46", "content": "Long silence from Dato/Kaggle suggests that they don't have additional data. In any case, I would expect creating datasets of these sizes (especially from 3rd party websites like StumbleUpon) should take a long time.
\n\nI would vote for a 2-stage approach like Netflix, GE (and many other competitions on Kaggle). Keep the end-date of stage 1 on Sept. 10th --- that was the day the leak was discovered. Start stage-2 with much bigger (and cleaner) datasets when they are ready.
", "rawMarkdown": "Long silence from Dato/Kaggle suggests that they don't have additional data. In any case, I would expect creating datasets of these sizes (especially from 3rd party websites like StumbleUpon) should take a long time.\r\n\r\nI would vote for a 2-stage approach like Netflix, GE (and many other competitions on Kaggle). Keep the end-date of stage 1 on Sept. 10th --- that was the day the leak was discovered. Start stage-2 with much bigger (and cleaner) datasets when they are ready.", "votes": null }, { "id": "92486", "postDate": "09/14/2015 14:32:23", "content": "I think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the "ground truth" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat.
", "rawMarkdown": "I think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the \"ground truth\" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat.", "votes": null }, { "id": "92488", "postDate": "09/14/2015 14:37:50", "content": "[quote=David McGarry;92486]
\n\nI think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the "ground truth" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat.
\n\n[/quote]
\n\nFile similarity is one thing, but that's not the only path to cheating. People could treat combined train+test dataset as a giant training dataset, overfit the model to it and achieve high score on LB.
", "rawMarkdown": "[quote=David McGarry;92486]\r\n\r\nI think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the \"ground truth\" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat. \r\n\r\n[/quote]\r\n\r\nFile similarity is one thing, but that's not the only path to cheating. People could treat combined train+test dataset as a giant training dataset, overfit the model to it and achieve high score on LB.", "votes": null }, { "id": "92489", "postDate": "09/14/2015 14:38:55", "content": "Hashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.
", "rawMarkdown": "Hashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.", "votes": null }, { "id": "92492", "postDate": "09/14/2015 15:12:59", "content": "Guys, people work for Kaggle also take weekends off. So I believe if they are gonna do something, they will start doing it today. So we should just be patient.
", "rawMarkdown": "Guys, people work for Kaggle also take weekends off. So I believe if they are gonna do something, they will start doing it today. So we should just be patient.", "votes": null }, { "id": "92497", "postDate": "09/14/2015 15:45:36", "content": "The question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:
\n\n-NxGTR's .974 performance without using the leak (great job!)
\n\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.
\n\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!
", "rawMarkdown": "The question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\r\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:\r\n\r\n-NxGTR's .974 performance without using the leak (great job!)\r\n\r\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.\r\n\r\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!\r\n\r\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.\r\n\r\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!", "votes": null }, { "id": "92500", "postDate": "09/14/2015 15:53:54", "content": "[quote=Ilya Nekhay;92489]
\n\nHashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.
\n\n[/quote]
\n\nTrue, although "a couple of hours" to cheat sounds expensive to me, but perhaps I don't understand the motivations for some joining the competition. Plus introducing some innocuous white noise to files would make hashing more difficult, but again still completely solvable if someone really wanted to cheat. As for feature selection, given the size of the training set I'm not sure there is really that much to gain unless people are developing massively overfit features. I suppose some would, which is unfortunate.
\n\nIt would nice if the competition could be salvaged. I'm not sure how possible that is if more data isn't available given that my idea was more-or-less de-bunked.
", "rawMarkdown": "[quote=Ilya Nekhay;92489]\r\n\r\nHashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.\r\n\r\n[/quote]\r\n\r\nTrue, although \"a couple of hours\" to cheat sounds expensive to me, but perhaps I don't understand the motivations for some joining the competition. Plus introducing some innocuous white noise to files would make hashing more difficult, but again still completely solvable if someone really wanted to cheat. As for feature selection, given the size of the training set I'm not sure there is really that much to gain unless people are developing massively overfit features. I suppose some would, which is unfortunate.\r\n\r\nIt would nice if the competition could be salvaged. I'm not sure how possible that is if more data isn't available given that my idea was more-or-less de-bunked.", "votes": null }, { "id": "92501", "postDate": "09/14/2015 15:54:56", "content": "[quote=Giulio;92497]
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n[/quote]
\n\nWhile I agree in principle, I for one have never so far found nor exploited a leak in a kaggle contest, and for me it will be a nice (albeit small) exercise to do so (if the data is still up until I get a chance to download it all).
", "rawMarkdown": "[quote=Giulio;92497]\r\n\r\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!\r\n\r\n[/quote]\r\n\r\nWhile I agree in principle, I for one have never so far found nor exploited a leak in a kaggle contest, and for me it will be a nice (albeit small) exercise to do so (if the data is still up until I get a chance to download it all).", "votes": null }, { "id": "92503", "postDate": "09/14/2015 16:02:31", "content": "[quote=Giulio;92497]
\n\nThe question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:
\n\n-NxGTR's .974 performance without using the leak (great job!)
\n\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.
\n\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!
\n\n[/quote]
\n\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.
", "rawMarkdown": "[quote=Giulio;92497]\r\n\r\nThe question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\r\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:\r\n\r\n-NxGTR's .974 performance without using the leak (great job!)\r\n\r\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.\r\n\r\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!\r\n\r\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.\r\n\r\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!\r\n\r\n[/quote]\r\n\r\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\r\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.", "votes": null }, { "id": "92505", "postDate": "09/14/2015 16:08:50", "content": "[quote=bekbolatov;92503]
\n\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.
\n\n[/quote]
\n\nApparently NxGTR's score is legit. 1) my dumb model is 0.95 already without the leak. 2) The leak is 0.99805 immediately. It's clear that he didn't know it at all.
", "rawMarkdown": "[quote=bekbolatov;92503]\r\n\r\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\r\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.\r\n\r\n[/quote]\r\n\r\nApparently NxGTR's score is legit. 1) my dumb model is 0.95 already without the leak. 2) The leak is 0.99805 immediately. It's clear that he didn't know it at all.", "votes": null }, { "id": "92506", "postDate": "09/14/2015 16:09:55", "content": "I got to 0.951 only using the content of the files, so I'm pretty sure it's possible. I made some decisions that worsened my model it seems, so I fell short after getting these results.
", "rawMarkdown": "I got to 0.951 only using the content of the files, so I'm pretty sure it's possible. I made some decisions that worsened my model it seems, so I fell short after getting these results.", "votes": null }, { "id": "92507", "postDate": "09/14/2015 16:18:31", "content": "Thanks guys. Totally above my expectations.
", "rawMarkdown": "Thanks guys. Totally above my expectations.", "votes": null }, { "id": "92509", "postDate": "09/14/2015 16:30:22", "content": "Anyone who had actually work in this one will know that my score its just good and legit, no leaks. My second submission got me almost 0.96 already, so 0.97x is not unbelievable.
", "rawMarkdown": "Anyone who had actually work in this one will know that my score its just good and legit, no leaks. My second submission got me almost 0.96 already, so 0.97x is not unbelievable.", "votes": null }, { "id": "92512", "postDate": "09/14/2015 16:43:37", "content": "It's all good, NxGTR. I haven't had time to try it out yet.
\n\nI was just getting familiar with the data and initially it seemed pretty hard to have a good separation like that - turns out it isn't.
\n\nI haven't decided whether I want to put an effort into this competition - given the easy separation.
\n\nThis means that practically all of the Native ads are detectable :)
", "rawMarkdown": "It's all good, NxGTR. I haven't had time to try it out yet.\r\n\r\nI was just getting familiar with the data and initially it seemed pretty hard to have a good separation like that - turns out it isn't.\r\n\r\nI haven't decided whether I want to put an effort into this competition - given the easy separation.\r\n\r\nThis means that practically all of the Native ads are detectable :)", "votes": null }, { "id": "92519", "postDate": "09/14/2015 17:02:44", "content": "I got a 0.97064 submission score with a model based on HTML alone.
\n\nMixing my content based model with the metadata model yields a slightly worse submission score (0.99826) than the metadata model alone (0.99846).
\n\nThe metadata based models worked by injecting tags like <file-meta-###/>, so that I wouldn't have to use a separate algorithm to mix the features.
Hey bro
", "rawMarkdown": "Hey bro", "votes": null } ], "comments": [ { "id": 92252, "author_name": "thakurrajanand", "author_url": "", "post_date": "09/12/2015 15:00:08", "content": "[quote=SkyLibrary;92249]
\n\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find typo in title, guess this is no way to fix that lol
\n\n[/quote]
\n\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board.
\n\nI hope you can understand that I don't want to be involve in controversy every time :)
", "votes": null, "replies": [] }, { "id": 92254, "author_name": "rushter", "author_url": "", "post_date": "09/12/2015 15:30:35", "content": "Just wait few weeks :)
", "votes": null, "replies": [] }, { "id": 92257, "author_name": "skylibrary", "author_url": "", "post_date": "09/12/2015 16:21:59", "content": "It is pretty much a yes or no question, if you guys don't want to answer it, then I will assume the answer is yes.
", "votes": null, "replies": [] }, { "id": 92276, "author_name": "adjgiulio", "author_url": "", "post_date": "09/12/2015 19:50:26", "content": "Why would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though.
", "votes": null, "replies": [] }, { "id": 92280, "author_name": "skylibrary", "author_url": "", "post_date": "09/12/2015 20:26:16", "content": "I didn't say they should answer my question, but I have the right to post the question no matter they reply or not. If they do, it will confirm your 'highly likely', if not, then it is still highly likely :)
\n\n[quote=Giulio;92276]
\n\nWhy would they have to say anything? It is already a HUGE advantage for everybody to know that it is possible to achieve that score. I dropped the competition because I believe it is highly likely that this is an exercise in finding a deterministic rule in the dataset, and that treasure hunt is not worth my time. I could be wrong though.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 92285, "author_name": "remap1", "author_url": "", "post_date": "09/12/2015 22:48:30", "content": "My guess is:
\n\nThey are running an unsupervised algorithm over both training and test that generates relationships between elements in both datasets. Let's call this a vocabulary. This thus creates highly abstract relationships between all documents at word level, but the difference now is that this is across both train and test.
\n\nThen for training, create (meaningful) document vectors from this vocabulary and start training to indicate sponsored / not sponsored. The difference is that you're not training on occurrences of individual features of a document (which can be meaningful or not), but you're training against the known vector space of all documents (with 1/3 of the data).
", "votes": null, "replies": [] }, { "id": 92294, "author_name": "mandalsubhajit", "author_url": "", "post_date": "09/13/2015 01:05:02", "content": "[quote=DataGeek;92252]
\n\n[quote=SkyLibrary;92249]
\n\nThe near perfect result is pretty surprising to me, I am just wondering if top three players are willing to tell it is label/tag leakage or not ? I think there is no harm confirming this, after all, leakage is nothing related to violating competition rules.
\n\nThe dataset is quite interesting, but I don't want to spend time on a competition with leakage again, the west nile virus one is a nightmare for me, and the tube price too.
\n\nEdit: sorry, just find typo in title, guess this is no way to fix that lol
\n\n[/quote]
\n\nIt would be unfair with the 2 above me if I tell you anything. Last time I posted high model in Otto competition and everyone was just behind me. Someone from top 10 even criticized me by messaging me personally that because of me they are losing rank on Leader board.
\n\nI hope you can understand that I don't want to be involve in controversy every time :)
\n\n[/quote]
\n\nI think more and more people are getting offended by this, and joining the 0.99 club. :)
", "votes": null, "replies": [] }, { "id": 92299, "author_name": "skylibrary", "author_url": "", "post_date": "09/13/2015 03:07:39", "content": "After download the data and play with it for 5 hours, I am here to confirm the answer is yes.
", "votes": null, "replies": [] }, { "id": 92301, "author_name": "dmcgarry", "author_url": "", "post_date": "09/13/2015 04:32:35", "content": "Well this is fun, and here I thought this would be the one competition that is actually like the old days since the data is too large for scripts. Silly me.
", "votes": null, "replies": [] }, { "id": 92303, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/13/2015 05:37:23", "content": "Interesting. I wonder if Kaggle will put a hold on this or just let it be it.
", "votes": null, "replies": [] }, { "id": 92304, "author_name": "", "author_url": "", "post_date": "09/13/2015 05:55:33", "content": "I am a noob in machine learning. I played with natural language processing algorithms. I achieved .76 score then I found that beside language processing, we can have better result if we used other features in the HTML files such as: javascript, meta tag attribute...
", "votes": null, "replies": [] }, { "id": 92305, "author_name": "lijunjie", "author_url": "", "post_date": "09/13/2015 06:04:10", "content": "So there is little space for the improvement after you applying the leakage information into your model.
", "votes": null, "replies": [] }, { "id": 92308, "author_name": "dulyanov", "author_url": "", "post_date": "09/13/2015 06:46:58", "content": "For those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging.
", "votes": null, "replies": [] }, { "id": 92313, "author_name": "clustifier", "author_url": "", "post_date": "09/13/2015 09:28:41", "content": "[quote=Dmitry Ulyanov;92308]
\n\nFor those of you who think this competition doesn't make sense now -- it is really a whole different world in tuning your model, when you have one almost perfect predictor. Just try to go beyond the level this predictor gives you, it is challenging.
\n\n[/quote]
\n\nBut now you must know this "perfect predictor", which it's a leakage and not real data, in order to be successful in this competition.
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
", "votes": null, "replies": [] }, { "id": 92314, "author_name": "wildwizard", "author_url": "", "post_date": "09/13/2015 09:29:01", "content": "This is unfair to WNV or Cat competitors to say that leakage exploitation was the only determinant in victory.
\nMany did exploit it (even myself in WNV, not knowing it was, while in Cat I discarded it believing I would make a methodology mistake), and we did not all achieve top scores.
Leakage exploitation still asks for talent and skills.\nAlso, nothing tells that what you call leakage, as I said before, is just a very strong association rule in the dataset and was present from the start.
", "votes": null, "replies": [] }, { "id": 92321, "author_name": "omgponies", "author_url": "", "post_date": "09/13/2015 13:18:19", "content": "My code that is looking at what I think is the leak is apparently very slow, lol.
", "votes": null, "replies": [] }, { "id": 92322, "author_name": "remap1", "author_url": "", "post_date": "09/13/2015 13:27:26", "content": "I thought that, since an ad is "sponsored", you'd find some kind of link or script in the files across all the sponsored ones (tracking links), but it isn't as easy as that. Just filtering on _gac for example gave me 0.546 and non-sponsored ones contain the same thing. But I reckon it must be somewhere along those lines...
", "votes": null, "replies": [] }, { "id": 92324, "author_name": "yejiming", "author_url": "", "post_date": "09/13/2015 13:52:50", "content": "[quote=SkyLibrary;92299]
\n\nAfter download the data and play with it for 5 hours, I am here to confirm the answer is yes.
\n\n[/quote]\nSo this is a leakage, and you have found it?
", "votes": null, "replies": [] }, { "id": 92326, "author_name": "jiweiliu", "author_url": "", "post_date": "09/13/2015 14:33:21", "content": "[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
", "votes": null, "replies": [] }, { "id": 92327, "author_name": "khyh00", "author_url": "", "post_date": "09/13/2015 14:38:01", "content": "I am sure that there is a leakage that can be used as a "golden feature", but only using this can not reach 1.0 on the public leader board (0.99805). Or, as pointed out on the other thread, there are some errors in this dataset so the "golden feature" is perfect but labels are wrong.
\n\nI want to know what the admin is thinking about this since further explorer might be in vain.
", "votes": null, "replies": [] }, { "id": 92331, "author_name": "dmcgarry", "author_url": "", "post_date": "09/13/2015 14:59:08", "content": "[quote=rcarson;92326]
\n\n[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
\n\n[/quote]
\n\nDefine "fair". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
", "votes": null, "replies": [] }, { "id": 92332, "author_name": "yejiming", "author_url": "", "post_date": "09/13/2015 15:11:02", "content": "I guess this has something to do with scripts inside html files.
\n\nFor example, the function of ads' charging. But I'm not familiar with javascript unfortunately, lol.
", "votes": null, "replies": [] }, { "id": 92333, "author_name": "rushter", "author_url": "", "post_date": "09/13/2015 15:17:04", "content": "Stumbleupon doesn't require any additions to html page.
", "votes": null, "replies": [] }, { "id": 92334, "author_name": "jiweiliu", "author_url": "", "post_date": "09/13/2015 15:23:01", "content": "[quote=David McGarry;92331]
\n\n[quote=rcarson;92326]
\n\n[quote=clustifier;92313]
\n\nSo if it's a leakage, and according to @SkyLibrary it is, the fair thing to do is to restart this competition without this leakage.
\n\n[/quote]
\n\nEven with this leak it is still fair for players.
\n\nThe real bad thing is that it is pointless now for anyone trying to win the Dato Creators Award with this leakage existing.
\n\n[/quote]
\n\nDefine "fair". Sure, all players have the same opportunity to find and exploit leakage but this is supposed to be a machine learning competition. I would have never signed up if it was a race to find some mistake in the data. I'm not stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
\n\n[/quote]
\n\nOn the contrary, I found it fun to look for the leak ( I haven't found it yet).
\n\nA little bit off the topic, I think a great part of kaggle competitions is not about machine learning. It is more about data mining. Finding soft or strong leak (good or golden features) is usually more dominant, given there are so many good out-of-box tools like xgboost for the learning part.
\n\nIn my opinion, finding a leak is a great skill even in real world case. But I agree in this case, it is a mess that the leak is too strong..
", "votes": null, "replies": [] }, { "id": 92336, "author_name": "rushter", "author_url": "", "post_date": "09/13/2015 15:26:43", "content": "I think i found a leakage, but it doesn't improve my score. LOL.
", "votes": null, "replies": [] }, { "id": 92340, "author_name": "adjgiulio", "author_url": "", "post_date": "09/13/2015 16:35:16", "content": "Ok, let's play this out as if it was a real project.
\n\nBusiness guy: we think we could really benefit from being able to use ML to classify pages with native add content.
\n\nData Scientist: give me a week to look at the data and I will get back to you with a first evaluation of what I think is possible. I'll work with engineering to get some data.
\n\n...a week later...
\n\n(very excited, yet skeptical) Data Scientist: I think I found something in the data that allows a perfect prediction. This is XYZ feature.
\n\nBusiness Guy: oh, well, that makes sense. Seems kind of obvious. Wonder what "name of a peer/manager/director who is taking the heat" thought when he came up with the idea of using ML for this... You know what, I don't think we need ML anymore.
\n\n(disappointed) Data Scientist: well, I guess I'll go back to building that boring dashboard then. Too bad, I was so excited of finally using some of those ML skills I had learnt on Kaggle...
\n\nMy point, when a perfect predictor exists, there is no reason to use ML because it is not an ML problem. This should have never made it to be an ML project/competition. Now, will the finding of XYZ feature be of value to Dato? Who knows? I would be surprised if it does. Will it still be interesting to try to maximize performance on misslabeld observations? I mean, why not, it's just a problem like any other. But, wow, not the problem I thought I was signing up for.\nGlad I haven't made a submission yet...
", "votes": null, "replies": [] }, { "id": 92341, "author_name": "adjgiulio", "author_url": "", "post_date": "09/13/2015 16:39:06", "content": "[quote=David McGarry;92331]\n I'm now stuck in a demlina where I have to decide if I want to, (a) continue using machine learning to tackle a seemingly interesting problem, (b) give in and find the leakage or (c) say screw it, delete the large data files from my system and feel even worse about the post-scripts world of kaggle.
\n\n[/quote]
\n\n+1 for (c)
", "votes": null, "replies": [] }, { "id": 92342, "author_name": "remap1", "author_url": "", "post_date": "09/13/2015 16:41:24", "content": "Giulio, it's much worse than that... much worse :) one of those things: "soon as you see it..."
\n\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. I had been trying to improve from 0.951 before this without success. That by itself is already a very reasonable score I think.
", "votes": null, "replies": [] }, { "id": 92343, "author_name": "senbong", "author_url": "", "post_date": "09/13/2015 16:44:13", "content": "In the end, don't tell me it is a filename leak... ==
", "votes": null, "replies": [] }, { "id": 92344, "author_name": "carloshuertas", "author_url": "", "post_date": "09/13/2015 16:51:18", "content": "[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
", "votes": null, "replies": [] }, { "id": 92345, "author_name": "chonglinsun", "author_url": "", "post_date": "09/13/2015 16:56:13", "content": "[quote=NxGTR;92344]
\n\n[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
\n\n[/quote]
\n\nI'm sad too, especially because of time I spent on it in the last few weeks.
", "votes": null, "replies": [] }, { "id": 92348, "author_name": "remap1", "author_url": "", "post_date": "09/13/2015 17:19:20", "content": "[quote=NxGTR;92344]
\n\n[quote=Gerard Toonstra;92342]\nWhat I am interested in is what NXGTR was using for his model before this happened. The real submissions up to 0.970 or so. [/quote]
\n\nI am very sad for this competition... I can confirm 0.973x is achievable without losing my own standards.
\n\n[/quote]
\n\nI'm looking forward to an explanation of your approach, parametrizations and feature choices, if you decide to write it up.
", "votes": null, "replies": [] }, { "id": 92349, "author_name": "gkoundry", "author_url": "", "post_date": "09/13/2015 19:07:04", "content": "The date in the zip files looks suspect.
", "votes": null, "replies": [] }, { "id": 92350, "author_name": "lawrencechernin", "author_url": "", "post_date": "09/13/2015 19:25:14", "content": "yes, it looks like the 1's files were written only on June 24 or July 27th.
", "votes": null, "replies": [] }, { "id": 92353, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/13/2015 19:55:58", "content": "[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
", "votes": null, "replies": [] }, { "id": 92355, "author_name": "rushter", "author_url": "", "post_date": "09/13/2015 20:02:11", "content": "[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n", "votes": null, "replies": [] }, { "id": 92356, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/13/2015 20:05:35", "content": "StumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n \nhttp://ads.stumbleupon.com/wp-content/uploads/2014/09/CampaignCreationGuide.pdf
\n
Hmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)
\n\n[quote=Artem;92355]
\n\n[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n\n\nStumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n
[/quote]
", "votes": null, "replies": [] }, { "id": 92357, "author_name": "rushter", "author_url": "", "post_date": "09/13/2015 20:11:46", "content": "[quote=Little Boat;92356]
\n\nHmm. Then I don't see how that would be the leakage. Anyway, if I am bored enough to find out if it is, I will post it here. But I am hoping someone else can do it for us. :-)
\n\n[quote=Artem;92355]
\n\n[quote=Little Boat;92353]
\n\n[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]\nIf that is the leakage, I think it might be because dato removed the 'sponsored' label in original html files or did something similar at the same time.
\n\n[/quote]\nThere is nothing to remove :)
\n\n\n\n\nStumbleUpon Ads is a native advertising platform that does not use\n traditional ad units such as banners or text links. This means that on\n StumbleUpon your ad is, quite simply, your URL. Since your web page\n itself is the ad unit, no additional creative assets or copy are\n needed to drive traffic. We drive directly to your page when a user\n clicks the Stumble button.
\n
[/quote]\n[/quote]\nI checked it locally and it looks like true.
\n\nAlso i found another "leakage" that can give 0.86+ without ml.
", "votes": null, "replies": [] }, { "id": 92358, "author_name": "triskelion", "author_url": "", "post_date": "09/13/2015 20:13:48", "content": "There is definitely leakage and I don't see how this could be solved/repaired, short of giving us a completely new data set...
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
", "votes": null, "replies": [] }, { "id": 92359, "author_name": "dmcgarry", "author_url": "", "post_date": "09/13/2015 20:14:50", "content": "[quote=Lawrence Chernin;92350]
\n\nyes, it looks like the 1's files were written only on June 24 or July 27th.
\n\n[/quote]
\n\nWell that's fun. Hopefully they clear the LB and release a new dataset (using new HTML files would be ideal, but at least renaming all of the existing HTML files would discourage all but the least honest to ignore the known values in the test set).
", "votes": null, "replies": [] }, { "id": 92362, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/13/2015 20:45:35", "content": "Now I can confirm that the date is the leakage.
", "votes": null, "replies": [] }, { "id": 92363, "author_name": "adjgiulio", "author_url": "", "post_date": "09/13/2015 20:46:47", "content": "[quote=Triskelion;92358]
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
\n\n[/quote]
\n\nHow about the tiniest of of edges that is NOT worth competing for.
", "votes": null, "replies": [] }, { "id": 92366, "author_name": "omgponies", "author_url": "", "post_date": "09/13/2015 21:25:58", "content": "The one I was looking at was an unusual amount of the file name numbers showing up in other files, but hadn't looked hard at it yet - the code is slow.\nMaybe just genuinely random, but seems odd if you look at it.
", "votes": null, "replies": [] }, { "id": 92373, "author_name": "senbong", "author_url": "", "post_date": "09/13/2015 22:42:13", "content": "A small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
", "votes": null, "replies": [] }, { "id": 92376, "author_name": "carloshuertas", "author_url": "", "post_date": "09/13/2015 23:00:46", "content": "[quote=senbong;92373]
\n\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
\n\n[/quote]
\n\nMmm... what about incentives to players whose workflow is not:
\n\n1) Lets find some leaks
\n\n2) Do actual ML
\n\n:D
", "votes": null, "replies": [] }, { "id": 92377, "author_name": "senbong", "author_url": "", "post_date": "09/13/2015 23:11:06", "content": "[quote=NxGTR;92376]
\n\n[quote=senbong;92373]
\n\nA small suggestion to Kaggle is that, may be they can give some incentives to the player who first reports this type of leak (like 10% badge in the profile?) to detect this type of problems earlier.
\n\n[/quote]
\n\nMmm... what about incentives to players whose workflow is not:
\n\n1) Lets find some leaks
\n\n2) Do actual ML
\n\n:D
\n\n[/quote]
\n\nIn my opinion, detecting this type of problem earlier is very important. The worst case is that someone keeps this leak to the later stage and uses it to win a competition.
", "votes": null, "replies": [ { "id": 305511, "author_name": "work4diapers", "author_url": "", "post_date": "03/29/2018 01:14:05", "content": "Hey bro
", "votes": null, "replies": [] } ] }, { "id": 92379, "author_name": "hillbert", "author_url": "", "post_date": "09/13/2015 23:43:34", "content": "If the competition is stopped and isn't rerun then what might be nice is the offer of some sort of small reward (swag perhaps?) for those who attained a good score without the use of the "file_modified"
\n\nI only really entered this competition to try large scale analysis for the first time and learn from other people. It would be a shame for other people's work to go to waste simply due to a massive data leak.
", "votes": null, "replies": [] }, { "id": 92380, "author_name": "triskelion", "author_url": "", "post_date": "09/13/2015 23:45:10", "content": "[quote=Giulio;92363]
\n\n[quote=Triskelion;92358]
\n\nWhat the final few % is I have no idea. Could be mislabelling. Could be LB noise. Could be the tiniest of edges we are now competing for.
\n\n[/quote]
\n\nHow about the tiniest of of edges that is NOT worth competing for.
\n\n[/quote]
\n\nLooks to me the 10.000$ is still up for the taking. Kinda agree with your sentiment though.
", "votes": null, "replies": [] }, { "id": 92389, "author_name": "lijunjie", "author_url": "", "post_date": "09/14/2015 03:55:03", "content": "If the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid?
\n\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.
\n\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.
\n\nAnother good solution is to replace the original data set with a normal one.
\n\nThe worst solution is that you know it's harmful to the contest, and just let it go.
", "votes": null, "replies": [] }, { "id": 92392, "author_name": "mandalsubhajit", "author_url": "", "post_date": "09/14/2015 04:38:22", "content": "[quote=YcdoiT;92389]
\n\nIf the date is the leakage, how about the Kaggle admin add an additional restriction just for this contest that using this leakage in the model is invalid?
\n\nActually, I don't think winning with this leakage has any value for the original purpose. It's not even like a data mining technology to discover the leakage.
\n\nAnd we should also pay respect to the participants who has already payed much of time to get a high ranking without using leakage information.
\n\nAnother good solution is to replace the original data set with a normal one.
\n\nThe worst solution is that you know it's harmful to the contest, and just let it go.
\n\n[/quote]
\n\nActually it will be quite difficult to go through everyone's codes to see if they used date variable or not. Alternatively, they can make two leaderboards, one just before leakage and one after and at least award the ranks/points, whichever is maximum of the two LB's.
", "votes": null, "replies": [] }, { "id": 92400, "author_name": "lijunjie", "author_url": "", "post_date": "09/14/2015 05:23:38", "content": "@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
", "votes": null, "replies": [] }, { "id": 92402, "author_name": "mandalsubhajit", "author_url": "", "post_date": "09/14/2015 05:28:11", "content": "Yeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.
", "votes": null, "replies": [] }, { "id": 92405, "author_name": "senbong", "author_url": "", "post_date": "09/14/2015 05:33:26", "content": "[quote=Subhajit Mandal;92402]
\n\nYeah theoretically, that's the best idea. But downloading the whole data, and extracting them, and running all the models which take a large amount of time and resources would be very difficult.
\n\n[/quote]
\n\nI think a new test dataset should be enough. We can still use the downloaded data as the training data.
", "votes": null, "replies": [] }, { "id": 92408, "author_name": "bekbolatov", "author_url": "", "post_date": "09/14/2015 05:39:15", "content": "It looked like this competition was organized very well: lots of data left for testing - fairer scoring.
\n\nIt is a unfortunate that labels were leaked and now it will have to be cancelled.
\n\nMaybe organizers could save the competition, by coming up with more unseen test data and converting the data given so far to a "train" set?
", "votes": null, "replies": [] }, { "id": 92424, "author_name": "chonglinsun", "author_url": "", "post_date": "09/14/2015 06:14:10", "content": "[quote=YcdoiT;92400]
\n\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
\n\nYeah, whatever, at least 0.99805 of the test set was leaked.\n[/quote]
", "votes": null, "replies": [] }, { "id": 92425, "author_name": "chonglinsun", "author_url": "", "post_date": "09/14/2015 06:14:41", "content": "[quote=YcdoiT;92400]
\n\n@Subhajit Mandal, yeah, it's hard to tell if a person using this leakage unless he/she want to get the money(Top 1).\nSo, maybe replacing total data set is a good idea.
\n\n[/quote]
\n\nYeah, whatever, at least 0.99805 of the test set was leaked.
", "votes": null, "replies": [] }, { "id": 92476, "author_name": "jiweiliu", "author_url": "", "post_date": "09/14/2015 13:55:35", "content": "I'll vote for a restart with a new test data set.
", "votes": null, "replies": [] }, { "id": 92478, "author_name": "sjuvekar", "author_url": "", "post_date": "09/14/2015 14:15:46", "content": "Long silence from Dato/Kaggle suggests that they don't have additional data. In any case, I would expect creating datasets of these sizes (especially from 3rd party websites like StumbleUpon) should take a long time.
\n\nI would vote for a 2-stage approach like Netflix, GE (and many other competitions on Kaggle). Keep the end-date of stage 1 on Sept. 10th --- that was the day the leak was discovered. Start stage-2 with much bigger (and cleaner) datasets when they are ready.
", "votes": null, "replies": [] }, { "id": 92486, "author_name": "dmcgarry", "author_url": "", "post_date": "09/14/2015 14:32:23", "content": "I think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the "ground truth" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat.
", "votes": null, "replies": [] }, { "id": 92488, "author_name": "sjuvekar", "author_url": "", "post_date": "09/14/2015 14:37:50", "content": "[quote=David McGarry;92486]
\n\nI think that renaming all of the files and even jumbling the composition of the training/test sets would be sufficient. It's true that someone could still keep both sets of data and do an expensive file similarity to find the "ground truth" and cheat -- but anyone who spend the time to do that would be pathetic. Plus all prize eligible solutions need to be revealed so those couldn't exactly cheat.
\n\n[/quote]
\n\nFile similarity is one thing, but that's not the only path to cheating. People could treat combined train+test dataset as a giant training dataset, overfit the model to it and achieve high score on LB.
", "votes": null, "replies": [] }, { "id": 92489, "author_name": "ilyanekhay", "author_url": "", "post_date": "09/14/2015 14:38:55", "content": "Hashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.
", "votes": null, "replies": [] }, { "id": 92492, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/14/2015 15:12:59", "content": "Guys, people work for Kaggle also take weekends off. So I believe if they are gonna do something, they will start doing it today. So we should just be patient.
", "votes": null, "replies": [] }, { "id": 92497, "author_name": "adjgiulio", "author_url": "", "post_date": "09/14/2015 15:45:36", "content": "The question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:
\n\n-NxGTR's .974 performance without using the leak (great job!)
\n\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.
\n\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!
", "votes": null, "replies": [] }, { "id": 92500, "author_name": "dmcgarry", "author_url": "", "post_date": "09/14/2015 15:53:54", "content": "[quote=Ilya Nekhay;92489]
\n\nHashing 300k files and then matching by hash doesn't seem expensive to me. Actually, they can even be beautifulsoupped in a couple of hours on a single 8-core machine. At the same time, feature selection done on ground truth might be quite effective and can't be proven to be a cheat.
\n\n[/quote]
\n\nTrue, although "a couple of hours" to cheat sounds expensive to me, but perhaps I don't understand the motivations for some joining the competition. Plus introducing some innocuous white noise to files would make hashing more difficult, but again still completely solvable if someone really wanted to cheat. As for feature selection, given the size of the training set I'm not sure there is really that much to gain unless people are developing massively overfit features. I suppose some would, which is unfortunate.
\n\nIt would nice if the competition could be salvaged. I'm not sure how possible that is if more data isn't available given that my idea was more-or-less de-bunked.
", "votes": null, "replies": [] }, { "id": 92501, "author_name": "tobycheese", "author_url": "", "post_date": "09/14/2015 15:54:56", "content": "[quote=Giulio;92497]
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n[/quote]
\n\nWhile I agree in principle, I for one have never so far found nor exploited a leak in a kaggle contest, and for me it will be a nice (albeit small) exercise to do so (if the data is still up until I get a chance to download it all).
", "votes": null, "replies": [] }, { "id": 92503, "author_name": "bekbolatov", "author_url": "", "post_date": "09/14/2015 16:02:31", "content": "[quote=Giulio;92497]
\n\nThe question is whether there is any advantage for Kaggle and Dato to reset the competition. Dato hasn't put a lot of money on the the plate, which is indicative that this wasn't a must solve problem for them. Maybe more of a relatively cheap way to get some PR for their tool, but here I'm just speculating.\nAnyways, here are the few things that stood out to me so far, probably the only things I will remember about yet another flawed competition:
\n\n-NxGTR's .974 performance without using the leak (great job!)
\n\n-Dimitry's being the first finding the leak. I think, without him finding the problem, this might have gone unnoticed till the end. Great eye and I appreciate the honesty to put this out immediately instead of using it as a last day submission.
\n\n-The sadness of seeing the LB full of new entrants as soon as the leak was made public. Like sharks attracted by blood in the water. Come on guys, there is more to life than a few Kaggle points!
\n\n-Yet another flawed competition. I mean, I know that life sometimes throws curve balls, but it is interesting to observe how even companies that do DS & ML for a living, can overlook huge flaws like the one we're now dealing with.
\n\nAnyhow, however this goes, I hope everybody gets to enjoy the remainder of the competition. Good luck to all!
\n\n[/quote]
\n\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.
", "votes": null, "replies": [] }, { "id": 92505, "author_name": "jiweiliu", "author_url": "", "post_date": "09/14/2015 16:08:50", "content": "[quote=bekbolatov;92503]
\n\nI don't think that .974 is possible without using the leak - these webpages are well-disguised.\n@NxGTR - can you confirm that you didn't use the leak somehow/accidentally? I find it hard to believe, since these could be any webpages.
\n\n[/quote]
\n\nApparently NxGTR's score is legit. 1) my dumb model is 0.95 already without the leak. 2) The leak is 0.99805 immediately. It's clear that he didn't know it at all.
", "votes": null, "replies": [] }, { "id": 92506, "author_name": "remap1", "author_url": "", "post_date": "09/14/2015 16:09:55", "content": "I got to 0.951 only using the content of the files, so I'm pretty sure it's possible. I made some decisions that worsened my model it seems, so I fell short after getting these results.
", "votes": null, "replies": [] }, { "id": 92507, "author_name": "bekbolatov", "author_url": "", "post_date": "09/14/2015 16:18:31", "content": "Thanks guys. Totally above my expectations.
", "votes": null, "replies": [] }, { "id": 92509, "author_name": "carloshuertas", "author_url": "", "post_date": "09/14/2015 16:30:22", "content": "Anyone who had actually work in this one will know that my score its just good and legit, no leaks. My second submission got me almost 0.96 already, so 0.97x is not unbelievable.
", "votes": null, "replies": [] }, { "id": 92512, "author_name": "bekbolatov", "author_url": "", "post_date": "09/14/2015 16:43:37", "content": "It's all good, NxGTR. I haven't had time to try it out yet.
\n\nI was just getting familiar with the data and initially it seemed pretty hard to have a good separation like that - turns out it isn't.
\n\nI haven't decided whether I want to put an effort into this competition - given the easy separation.
\n\nThis means that practically all of the Native ads are detectable :)
", "votes": null, "replies": [] }, { "id": 92519, "author_name": "mortehu", "author_url": "", "post_date": "09/14/2015 17:02:44", "content": "I got a 0.97064 submission score with a model based on HTML alone.
\n\nMixing my content based model with the metadata model yields a slightly worse submission score (0.99826) than the metadata model alone (0.99846).
\n\nThe metadata based models worked by injecting tags like <file-meta-###/>, so that I wouldn't have to use a separate algorithm to mix the features.