{
  "id": 20282,
  "title": "A leak in the data ?",
  "url": "/competitions/expedia-hotel-recommendations/discussion/20282",
  "author_name": "",
  "post_date": "2016-04-20T13:02:52.697Z",
  "votes": 44,
  "comment_count": 31,
  "views": 7858,
  "content": "<p>Hi all,</p>\n\n<p>I am happy to be the first one at the moment, but after re-checking my model I think there is a leak in the data. </p>\n\n<p>@admin, feel free to email me for more info ;)</p>",
  "messages": [
    {
      "id": "115847",
      "postDate": "04/20/2016 13:02:52",
      "content": "<p>Hi all,</p>\n\n<p>I am happy to be the first one at the moment, but after re-checking my model I think there is a leak in the data. </p>\n\n<p>@admin, feel free to email me for more info ;)</p>",
      "rawMarkdown": "Hi all,\r\n\r\nI am happy to be the first one at the moment, but after re-checking my model I think there is a leak in the data. \r\n\r\n@admin, feel free to email me for more info ;)",
      "votes": null
    },
    {
      "id": "115853",
      "postDate": "04/20/2016 13:26:57",
      "content": "<p>email sent :-)</p>",
      "rawMarkdown": "email sent :-)",
      "votes": null
    },
    {
      "id": "115858",
      "postDate": "04/20/2016 14:10:24",
      "content": "<p>So, is it a leak?</p>",
      "rawMarkdown": "So, is it a leak?",
      "votes": null
    },
    {
      "id": "115891",
      "postDate": "04/20/2016 16:13:12",
      "content": "<p>Abhishek, do not be impatient. It takes time.</p>\n\n<p>So, is there a leak?</p>",
      "rawMarkdown": "Abhishek, do not be impatient. It takes time.\r\n\r\nSo, is there a leak?",
      "votes": null
    },
    {
      "id": "115904",
      "postDate": "04/20/2016 18:39:31",
      "content": "<p>Note sure yet, still investigating but very likely. If it's confirmed I will describe it here ;-)</p>",
      "rawMarkdown": "Note sure yet, still investigating but very likely. If it's confirmed I will describe it here ;-)",
      "votes": null
    },
    {
      "id": "115911",
      "postDate": "04/20/2016 19:00:35",
      "content": "<p>We are investigating. Since we (Expedia and Kaggle) haven't received the details of @arnaud's method yet, we are not yet able to verify. </p>",
      "rawMarkdown": "We are investigating. Since we (Expedia and Kaggle) haven't received the details of @arnaud's method yet, we are not yet able to verify.",
      "votes": null
    },
    {
      "id": "115922",
      "postDate": "04/20/2016 20:08:05",
      "content": "<p>edit: deleted</p>",
      "rawMarkdown": "edit: deleted",
      "votes": null
    },
    {
      "id": "116045",
      "postDate": "04/21/2016 17:11:09",
      "content": "<p>@arnaud demytt, no matter what it is, you are so awesome to say it this early!!</p>\n\n<p>I wanna upvote you 1 thousand times.</p>\n\n<p>I am humbled as both a 'victim' and 'convict' with regard to leak : P</p>",
      "rawMarkdown": "arnaud demytt, no matter what it is, you are so awesome to say it this early!!\r\n\r\nI wanna upvote you 1 thousand times.\r\n\r\nI am humbled as both a 'victim' and 'convict' with regard to leak : P",
      "votes": null
    },
    {
      "id": "116049",
      "postDate": "04/21/2016 17:34:24",
      "content": "<p>Someone please upvote me 1000 times to ask about the leak :P</p>",
      "rawMarkdown": "Someone please upvote me 1000 times to ask about the leak :P",
      "votes": null
    },
    {
      "id": "116103",
      "postDate": "04/22/2016 03:00:32",
      "content": "<p>[quote=Abhishek;116049]</p>\n\n<p>Someone please upvote me 1000 times to ask about the leak :P</p>\n\n<p>[/quote]\nThis is what happens when I publish a <a href=\"https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma\">leaderboard of users by forum karma</a> ;)</p>",
      "rawMarkdown": "[quote=Abhishek;116049]\r\n\r\nSomeone please upvote me 1000 times to ask about the leak :P\r\n\r\n[/quote]\r\nThis is what happens when I publish a [leaderboard of users by forum karma](https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma) ;)",
      "votes": null
    },
    {
      "id": "116122",
      "postDate": "04/22/2016 05:52:28",
      "content": "<p>[quote=Ben Hamner;116103]\nThis is what happens when I publish a <a href=\"https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma\">leaderboard of users by forum karma</a> ;)\n[/quote]</p>\n\n<p>It seems like if I am not good at ML, at least I have a very outstanding Karma at 22/520k+ ;)</p>",
      "rawMarkdown": "[quote=Ben Hamner;116103]\r\nThis is what happens when I publish a [leaderboard of users by forum karma](https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma) ;)\r\n[/quote]\r\n\r\nIt seems like if I am not good at ML, at least I have a very outstanding Karma at 22/520k+ ;)",
      "votes": null
    },
    {
      "id": "116131",
      "postDate": "04/22/2016 07:05:54",
      "content": "<p>I think it is a bit early to share my approach here, but I am sure many of you are going to find it soon ! </p>\n\n<p>Expedia agrees there is more information than expected... so at least now you know there is something magic in the data ;-)</p>",
      "rawMarkdown": "I think it is a bit early to share my approach here, but I am sure many of you are going to find it soon ! \r\n\r\nExpedia agrees there is more information than expected... so at least now you know there is something magic in the data ;-)",
      "votes": null
    },
    {
      "id": "116137",
      "postDate": "04/22/2016 07:38:34",
      "content": "<p>So the competition will become in a race to find a 'soft' leak...</p>",
      "rawMarkdown": "So the competition will become in a race to find a 'soft' leak...",
      "votes": null
    },
    {
      "id": "116139",
      "postDate": "04/22/2016 07:52:38",
      "content": "<p>Doing kaggle competitions has started to feel like I'm in a sitcom - have I made the right choice for my sanity?</p>\n\n<p>I'm training a classifier to find out, brb</p>",
      "rawMarkdown": "Doing kaggle competitions has started to feel like I'm in a sitcom - have I made the right choice for my sanity?\r\n\r\nI'm training a classifier to find out, brb",
      "votes": null
    },
    {
      "id": "116142",
      "postDate": "04/22/2016 08:03:08",
      "content": "<p>One more competition to find the magical feature or leak . Oh not again please.! :| </p>",
      "rawMarkdown": "One more competition to find the magical feature or leak . Oh not again please.! :|",
      "votes": null
    },
    {
      "id": "116153",
      "postDate": "04/22/2016 09:06:55",
      "content": "<p>Newbie here.</p>\n\n<p>What exactly is a data leak?</p>",
      "rawMarkdown": "Newbie here.\r\n\r\nWhat exactly is a data leak?",
      "votes": null
    },
    {
      "id": "116171",
      "postDate": "04/22/2016 11:32:24",
      "content": "<p>I guess most obvious definition is an information in the train set that can be matched with test set that would provide you with an estimated value almost with no cost.</p>\n\n<p>So it is a leak of predicted value from train to test.</p>",
      "rawMarkdown": "I guess most obvious definition is an information in the train set that can be matched with test set that would provide you with an estimated value almost with no cost.\r\n\r\nSo it is a leak of predicted value from train to test.",
      "votes": null
    },
    {
      "id": "116174",
      "postDate": "04/22/2016 11:59:46",
      "content": "<p>Thanks, Marcin. In the interim, another member posted a few plots hinting at just such a possibility.</p>",
      "rawMarkdown": "Thanks, Marcin. In the interim, another member posted a few plots hinting at just such a possibility.",
      "votes": null
    },
    {
      "id": "116178",
      "postDate": "04/22/2016 12:42:29",
      "content": "<p>So.....there is a leak, but not sufficient to recall the data?</p>",
      "rawMarkdown": "So.....there is a leak, but not sufficient to recall the data?",
      "votes": null
    },
    {
      "id": "116197",
      "postDate": "04/22/2016 13:46:50",
      "content": "<p>I think the reason of leakage is &quot;orig_destination_distance&quot; in the test dataset.</p>",
      "rawMarkdown": "I think the reason of leakage is \"orig_destination_distance\" in the test dataset.",
      "votes": null
    },
    {
      "id": "116227",
      "postDate": "04/22/2016 15:45:10",
      "content": "<p>[quote=Olexandr Topchylo;116197]</p>\n\n<p>I think the reason of leakage is &quot;orig_destination_distance&quot; in the test dataset.</p>\n\n<p>[/quote]</p>\n\n<p>Yeah, this seems to be it. Though, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. But I guess it might be also possible to figure out distance between users etc. making it much worse.</p>",
      "rawMarkdown": "[quote=Olexandr Topchylo;116197]\r\n\r\nI think the reason of leakage is \"orig_destination_distance\" in the test dataset.\r\n\r\n[/quote]\r\n\r\nYeah, this seems to be it. Though, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. But I guess it might be also possible to figure out distance between users etc. making it much worse.",
      "votes": null
    },
    {
      "id": "116242",
      "postDate": "04/22/2016 16:57:22",
      "content": "<p>Olexandr Topchylo  you're right, orig_destination_distance is the key of the problem...</p>\n\n<p>So from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)</p>",
      "rawMarkdown": "Olexandr Topchylo  you're right, orig_destination_distance is the key of the problem...\r\n\r\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)",
      "votes": null
    },
    {
      "id": "116243",
      "postDate": "04/22/2016 17:07:45",
      "content": "<p>[quote=Herra Huu;116227]\nThough, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. \n[/quote]</p>\n\n<p>It seems that distances are so accurate in the data that one distance is most of the time related only to one hotel no matter what is the user (maybe there are many users in one location). So you can actually merge only by distance instead of user_id and distance.  </p>",
      "rawMarkdown": "[quote=Herra Huu;116227]\r\nThough, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. \r\n[/quote]\r\n\r\nIt seems that distances are so accurate in the data that one distance is most of the time related only to one hotel no matter what is the user (maybe there are many users in one location). So you can actually merge only by distance instead of user_id and distance.",
      "votes": null
    },
    {
      "id": "116253",
      "postDate": "04/22/2016 17:57:05",
      "content": "<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>",
      "rawMarkdown": "admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.",
      "votes": null
    },
    {
      "id": "116255",
      "postDate": "04/22/2016 18:02:02",
      "content": "<p>[quote=Sergey Yurgenson;116253]</p>\n\n<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>\n\n<p>[/quote]</p>\n\n<p>Around 1/3 of the distances are missing, and also I think 60% of the distance in test are not in train. Otherwise, yeah...</p>",
      "rawMarkdown": "[quote=Sergey Yurgenson;116253]\r\n\r\n@admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.\r\n\r\n[/quote]\r\n\r\nAround 1/3 of the distances are missing, and also I think 60% of the distance in test are not in train. Otherwise, yeah...",
      "votes": null
    },
    {
      "id": "116262",
      "postDate": "04/22/2016 18:37:47",
      "content": "<p>[quote=Sergey Yurgenson;116253]</p>\n\n<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>\n\n<p>[/quote]</p>\n\n<p>I think one reasonable solution would be to keep only rows where orig_destination_distance==NA. It would still leave test set of size 847461 rows (more than enough) and we still have user_location_country and hotel_country features containing most of that distance info anyway.</p>",
      "rawMarkdown": "[quote=Sergey Yurgenson;116253]\r\n\r\n@admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.\r\n\r\n[/quote]\r\n\r\nI think one reasonable solution would be to keep only rows where orig_destination_distance==NA. It would still leave test set of size 847461 rows (more than enough) and we still have user_location_country and hotel_country features containing most of that distance info anyway.",
      "votes": null
    },
    {
      "id": "116263",
      "postDate": "04/22/2016 18:39:31",
      "content": "<p>So, Expedia is not going to reboot the competition?</p>",
      "rawMarkdown": "So, Expedia is not going to reboot the competition?",
      "votes": null
    },
    {
      "id": "116269",
      "postDate": "04/22/2016 19:33:54",
      "content": "<p>Hi all, </p>\n\n<p>Please give us a bit more time. We (Kaggle/Expedia) are evaluating different ways to handle this.</p>",
      "rawMarkdown": "Hi all, \r\n\r\nPlease give us a bit more time. We (Kaggle/Expedia) are evaluating different ways to handle this.",
      "votes": null
    },
    {
      "id": "119225",
      "postDate": "05/08/2016 08:28:22",
      "content": "<p>[quote=arnaud demytt;116242]\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)\n[/quote]\nIs such approach allowed?\nDid anybody try it?</p>",
      "rawMarkdown": "[quote=arnaud demytt;116242]\r\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)\r\n[/quote]\r\nIs such approach allowed?\r\nDid anybody try it?",
      "votes": null
    },
    {
      "id": "119760",
      "postDate": "05/12/2016 17:00:55",
      "content": "<p>Looks plausible..</p>",
      "rawMarkdown": "Looks plausible..",
      "votes": null
    },
    {
      "id": "119768",
      "postDate": "05/12/2016 17:40:36",
      "content": "<p>Yes it is doable ;) although I assume these coords have at least few hundreds of meter average error. It could be used to create additional features (eg. New regions). It is not that powerful as the raw leakage. Just try to check rounded versions of distance matching.</p>",
      "rawMarkdown": "Yes it is doable ;) although I assume these coords have at least few hundreds of meter average error. It could be used to create additional features (eg. New regions). It is not that powerful as the raw leakage. Just try to check rounded versions of distance matching.",
      "votes": null
    },
    {
      "id": "119769",
      "postDate": "05/12/2016 17:42:41",
      "content": "<p>Arnaud, how did you realise there was a data leakage. How do you detect data leakage?</p>\n\n<p>thanx </p>",
      "rawMarkdown": "Arnaud, how did you realise there was a data leakage. How do you detect data leakage?\r\n\r\nthanx",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 115853,
      "author_name": "adamwoz",
      "author_url": "",
      "post_date": "04/20/2016 13:26:57",
      "content": "<p>email sent :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115858,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "04/20/2016 14:10:24",
      "content": "<p>So, is it a leak?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115891,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "04/20/2016 16:13:12",
      "content": "<p>Abhishek, do not be impatient. It takes time.</p>\n\n<p>So, is there a leak?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115904,
      "author_name": "ademyttenaere",
      "author_url": "",
      "post_date": "04/20/2016 18:39:31",
      "content": "<p>Note sure yet, still investigating but very likely. If it's confirmed I will describe it here ;-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115911,
      "author_name": "wendykan",
      "author_url": "",
      "post_date": "04/20/2016 19:00:35",
      "content": "<p>We are investigating. Since we (Expedia and Kaggle) haven't received the details of @arnaud's method yet, we are not yet able to verify. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 115922,
      "author_name": "herrahuu",
      "author_url": "",
      "post_date": "04/20/2016 20:08:05",
      "content": "<p>edit: deleted</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116045,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "04/21/2016 17:11:09",
      "content": "<p>@arnaud demytt, no matter what it is, you are so awesome to say it this early!!</p>\n\n<p>I wanna upvote you 1 thousand times.</p>\n\n<p>I am humbled as both a 'victim' and 'convict' with regard to leak : P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116049,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "04/21/2016 17:34:24",
      "content": "<p>Someone please upvote me 1000 times to ask about the leak :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116103,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "04/22/2016 03:00:32",
      "content": "<p>[quote=Abhishek;116049]</p>\n\n<p>Someone please upvote me 1000 times to ask about the leak :P</p>\n\n<p>[/quote]\nThis is what happens when I publish a <a href=\"https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma\">leaderboard of users by forum karma</a> ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116122,
      "author_name": "carloshuertas",
      "author_url": "",
      "post_date": "04/22/2016 05:52:28",
      "content": "<p>[quote=Ben Hamner;116103]\nThis is what happens when I publish a <a href=\"https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma\">leaderboard of users by forum karma</a> ;)\n[/quote]</p>\n\n<p>It seems like if I am not good at ML, at least I have a very outstanding Karma at 22/520k+ ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116131,
      "author_name": "ademyttenaere",
      "author_url": "",
      "post_date": "04/22/2016 07:05:54",
      "content": "<p>I think it is a bit early to share my approach here, but I am sure many of you are going to find it soon ! </p>\n\n<p>Expedia agrees there is more information than expected... so at least now you know there is something magic in the data ;-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116137,
      "author_name": "blindape",
      "author_url": "",
      "post_date": "04/22/2016 07:38:34",
      "content": "<p>So the competition will become in a race to find a 'soft' leak...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116139,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "04/22/2016 07:52:38",
      "content": "<p>Doing kaggle competitions has started to feel like I'm in a sitcom - have I made the right choice for my sanity?</p>\n\n<p>I'm training a classifier to find out, brb</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116142,
      "author_name": "sudalairajkumar",
      "author_url": "",
      "post_date": "04/22/2016 08:03:08",
      "content": "<p>One more competition to find the magical feature or leak . Oh not again please.! :| </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116153,
      "author_name": "navneethc",
      "author_url": "",
      "post_date": "04/22/2016 09:06:55",
      "content": "<p>Newbie here.</p>\n\n<p>What exactly is a data leak?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116171,
      "author_name": "mpekalski",
      "author_url": "",
      "post_date": "04/22/2016 11:32:24",
      "content": "<p>I guess most obvious definition is an information in the train set that can be matched with test set that would provide you with an estimated value almost with no cost.</p>\n\n<p>So it is a leak of predicted value from train to test.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116174,
      "author_name": "navneethc",
      "author_url": "",
      "post_date": "04/22/2016 11:59:46",
      "content": "<p>Thanks, Marcin. In the interim, another member posted a few plots hinting at just such a possibility.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116178,
      "author_name": "inspector",
      "author_url": "",
      "post_date": "04/22/2016 12:42:29",
      "content": "<p>So.....there is a leak, but not sufficient to recall the data?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116197,
      "author_name": "topchylo",
      "author_url": "",
      "post_date": "04/22/2016 13:46:50",
      "content": "<p>I think the reason of leakage is &quot;orig_destination_distance&quot; in the test dataset.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116227,
      "author_name": "herrahuu",
      "author_url": "",
      "post_date": "04/22/2016 15:45:10",
      "content": "<p>[quote=Olexandr Topchylo;116197]</p>\n\n<p>I think the reason of leakage is &quot;orig_destination_distance&quot; in the test dataset.</p>\n\n<p>[/quote]</p>\n\n<p>Yeah, this seems to be it. Though, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. But I guess it might be also possible to figure out distance between users etc. making it much worse.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116242,
      "author_name": "ademyttenaere",
      "author_url": "",
      "post_date": "04/22/2016 16:57:22",
      "content": "<p>Olexandr Topchylo  you're right, orig_destination_distance is the key of the problem...</p>\n\n<p>So from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116243,
      "author_name": "pekkaa",
      "author_url": "",
      "post_date": "04/22/2016 17:07:45",
      "content": "<p>[quote=Herra Huu;116227]\nThough, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. \n[/quote]</p>\n\n<p>It seems that distances are so accurate in the data that one distance is most of the time related only to one hotel no matter what is the user (maybe there are many users in one location). So you can actually merge only by distance instead of user_id and distance.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116253,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "04/22/2016 17:57:05",
      "content": "<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116255,
      "author_name": "sangxia",
      "author_url": "",
      "post_date": "04/22/2016 18:02:02",
      "content": "<p>[quote=Sergey Yurgenson;116253]</p>\n\n<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>\n\n<p>[/quote]</p>\n\n<p>Around 1/3 of the distances are missing, and also I think 60% of the distance in test are not in train. Otherwise, yeah...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116262,
      "author_name": "herrahuu",
      "author_url": "",
      "post_date": "04/22/2016 18:37:47",
      "content": "<p>[quote=Sergey Yurgenson;116253]</p>\n\n<p>@admin Are you considering updating datasets/rules or other changes in competition?\nIf I understand correctly the nature of the leak it may result in perfect score at the end.</p>\n\n<p>[/quote]</p>\n\n<p>I think one reasonable solution would be to keep only rows where orig_destination_distance==NA. It would still leave test set of size 847461 rows (more than enough) and we still have user_location_country and hotel_country features containing most of that distance info anyway.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116263,
      "author_name": "gospursgo",
      "author_url": "",
      "post_date": "04/22/2016 18:39:31",
      "content": "<p>So, Expedia is not going to reboot the competition?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 116269,
      "author_name": "wendykan",
      "author_url": "",
      "post_date": "04/22/2016 19:33:54",
      "content": "<p>Hi all, </p>\n\n<p>Please give us a bit more time. We (Kaggle/Expedia) are evaluating different ways to handle this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119225,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "05/08/2016 08:28:22",
      "content": "<p>[quote=arnaud demytt;116242]\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)\n[/quote]\nIs such approach allowed?\nDid anybody try it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119760,
      "author_name": "nedelko",
      "author_url": "",
      "post_date": "05/12/2016 17:00:55",
      "content": "<p>Looks plausible..</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119768,
      "author_name": "gaborfodor",
      "author_url": "",
      "post_date": "05/12/2016 17:40:36",
      "content": "<p>Yes it is doable ;) although I assume these coords have at least few hundreds of meter average error. It could be used to create additional features (eg. New regions). It is not that powerful as the raw leakage. Just try to check rounded versions of distance matching.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 119769,
      "author_name": "valerioorfano",
      "author_url": "",
      "post_date": "05/12/2016 17:42:41",
      "content": "<p>Arnaud, how did you realise there was a data leakage. How do you detect data leakage?</p>\n\n<p>thanx </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "115847": "Hi all,\r\n\r\nI am happy to be the first one at the moment, but after re-checking my model I think there is a leak in the data. \r\n\r\n@admin, feel free to email me for more info ;)",
    "115853": "email sent :-)",
    "115858": "So, is it a leak?",
    "115891": "Abhishek, do not be impatient. It takes time.\r\n\r\nSo, is there a leak?",
    "115904": "Note sure yet, still investigating but very likely. If it's confirmed I will describe it here ;-)",
    "115911": "We are investigating. Since we (Expedia and Kaggle) haven't received the details of @arnaud's method yet, we are not yet able to verify.",
    "115922": "edit: deleted",
    "116045": "arnaud demytt, no matter what it is, you are so awesome to say it this early!!\r\n\r\nI wanna upvote you 1 thousand times.\r\n\r\nI am humbled as both a 'victim' and 'convict' with regard to leak : P",
    "116049": "Someone please upvote me 1000 times to ask about the leak :P",
    "116103": "[quote=Abhishek;116049]\r\n\r\nSomeone please upvote me 1000 times to ask about the leak :P\r\n\r\n[/quote]\r\nThis is what happens when I publish a [leaderboard of users by forum karma](https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma) ;)",
    "116122": "[quote=Ben Hamner;116103]\r\nThis is what happens when I publish a [leaderboard of users by forum karma](https://www.kaggle.com/benhamner/d/kaggle/meta-kaggle/kaggle-users-with-most-forum-karma) ;)\r\n[/quote]\r\n\r\nIt seems like if I am not good at ML, at least I have a very outstanding Karma at 22/520k+ ;)",
    "116131": "I think it is a bit early to share my approach here, but I am sure many of you are going to find it soon ! \r\n\r\nExpedia agrees there is more information than expected... so at least now you know there is something magic in the data ;-)",
    "116137": "So the competition will become in a race to find a 'soft' leak...",
    "116139": "Doing kaggle competitions has started to feel like I'm in a sitcom - have I made the right choice for my sanity?\r\n\r\nI'm training a classifier to find out, brb",
    "116142": "One more competition to find the magical feature or leak . Oh not again please.! :|",
    "116153": "Newbie here.\r\n\r\nWhat exactly is a data leak?",
    "116171": "I guess most obvious definition is an information in the train set that can be matched with test set that would provide you with an estimated value almost with no cost.\r\n\r\nSo it is a leak of predicted value from train to test.",
    "116174": "Thanks, Marcin. In the interim, another member posted a few plots hinting at just such a possibility.",
    "116178": "So.....there is a leak, but not sufficient to recall the data?",
    "116197": "I think the reason of leakage is \"orig_destination_distance\" in the test dataset.",
    "116227": "[quote=Olexandr Topchylo;116197]\r\n\r\nI think the reason of leakage is \"orig_destination_distance\" in the test dataset.\r\n\r\n[/quote]\r\n\r\nYeah, this seems to be it. Though, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. But I guess it might be also possible to figure out distance between users etc. making it much worse.",
    "116242": "Olexandr Topchylo  you're right, orig_destination_distance is the key of the problem...\r\n\r\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)",
    "116243": "[quote=Herra Huu;116227]\r\nThough, there isn't huge number of users with same orig_destination_distance in both train and test sets. So maybe it isn't too bad for the contest. \r\n[/quote]\r\n\r\nIt seems that distances are so accurate in the data that one distance is most of the time related only to one hotel no matter what is the user (maybe there are many users in one location). So you can actually merge only by distance instead of user_id and distance.",
    "116253": "admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.",
    "116255": "[quote=Sergey Yurgenson;116253]\r\n\r\n@admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.\r\n\r\n[/quote]\r\n\r\nAround 1/3 of the distances are missing, and also I think 60% of the distance in test are not in train. Otherwise, yeah...",
    "116262": "[quote=Sergey Yurgenson;116253]\r\n\r\n@admin Are you considering updating datasets/rules or other changes in competition?\r\nIf I understand correctly the nature of the leak it may result in perfect score at the end.\r\n\r\n[/quote]\r\n\r\nI think one reasonable solution would be to keep only rows where orig_destination_distance==NA. It would still leave test set of size 847461 rows (more than enough) and we still have user_location_country and hotel_country features containing most of that distance info anyway.",
    "116263": "So, Expedia is not going to reboot the competition?",
    "116269": "Hi all, \r\n\r\nPlease give us a bit more time. We (Kaggle/Expedia) are evaluating different ways to handle this.",
    "119225": "[quote=arnaud demytt;116242]\r\nSo from now the challenge may look like a distance matrix completion problem in order to find the position of every user and hotel. Enjoy ! ;-)\r\n[/quote]\r\nIs such approach allowed?\r\nDid anybody try it?",
    "119760": "Looks plausible..",
    "119768": "Yes it is doable ;) although I assume these coords have at least few hundreds of meter average error. It could be used to create additional features (eg. New regions). It is not that powerful as the raw leakage. Just try to check rounded versions of distance matching.",
    "119769": "Arnaud, how did you realise there was a data leakage. How do you detect data leakage?\r\n\r\nthanx"
  },
  "source": "meta"
}