{ "id": 20806, "title": "date_time and srch_ci ", "url": "/competitions/expedia-hotel-recommendations/discussion/20806", "author_name": "", "post_date": "2016-05-08T12:55:59.120Z", "votes": 2, "comment_count": 16, "views": 2022, "content": "

How come that some srch_ci have a date anterior to date_time?\nCan somebody help me understand?\n8457 events are concerned and 617 of them are actual bookings (facts from the training data).

", "messages": [ { "id": "119249", "postDate": "05/08/2016 12:55:59", "content": "

How come that some srch_ci have a date anterior to date_time?\nCan somebody help me understand?\n8457 events are concerned and 617 of them are actual bookings (facts from the training data).

", "rawMarkdown": "How come that some srch_ci have a date anterior to date_time?\r\nCan somebody help me understand?\r\n8457 events are concerned and 617 of them are actual bookings (facts from the training data).", "votes": null }, { "id": "119261", "postDate": "05/08/2016 16:38:28", "content": "

that sounds wrong but I guess it is safe to ignore them due to the small amount of data affected.

", "rawMarkdown": "that sounds wrong but I guess it is safe to ignore them due to the small amount of data affected.", "votes": null }, { "id": "119262", "postDate": "05/08/2016 16:53:09", "content": "

thanks rcarson

", "rawMarkdown": "thanks rcarson", "votes": null }, { "id": "119323", "postDate": "05/09/2016 08:08:20", "content": "

It's likely a system error. If you use stay dates in your model it's safe to ignore these records.

", "rawMarkdown": "It's likely a system error. If you use stay dates in your model it's safe to ignore these records.", "votes": null }, { "id": "119596", "postDate": "05/11/2016 17:28:31", "content": "

some of them are errors.\nsome are not.\nit's depends on how you calculate the day difference. if you, like what I did, convert them to datetime type, system treat sch_ci as 0:00 of check in day. For same-day booking, booking time is behind CI time.

", "rawMarkdown": "some of them are errors.\r\nsome are not.\r\nit's depends on how you calculate the day difference. if you, like what I did, convert them to datetime type, system treat sch_ci as 0:00 of check in day. For same-day booking, booking time is behind CI time.", "votes": null }, { "id": "120442", "postDate": "05/18/2016 07:57:04", "content": "

@ShenHuang, Can you elaborate on how you treated this issue, it's a bit unclear. What do you mean by "system treat sch_ci as 0:00 of check in day" ?

", "rawMarkdown": "ShenHuang, Can you elaborate on how you treated this issue, it's a bit unclear. What do you mean by \"system treat sch_ci as 0:00 of check in day\" ?", "votes": null }, { "id": "120480", "postDate": "05/18/2016 15:35:25", "content": "

@dot277 \nFor example, when a guest arrives Las Vegas at 5pm today, he started looking for a room for tonight.\nExpedia gets the record:

\n\n

datetime = "2016-05-18 17:09:54"

\n\n

srch_ci = "2016-05-18"

\n\n

if you convert both string to datetime, you get:

\n\n

datetime = 2016-05-18 17:09:54

\n\n

srch_ci = 2016-05-18 00:00:00

\n\n

The time difference:

\n\n

srch_ci - datetime = -1 days +06:50:06

\n\n

So, I treat 0 day to -2 days as same-day booking.\nIt is just my case. There could be other reasons. I hope it can help you.

", "rawMarkdown": "dot277 \r\nFor example, when a guest arrives Las Vegas at 5pm today, he started looking for a room for tonight.\r\nExpedia gets the record:\r\n\r\ndatetime = \"2016-05-18 17:09:54\"\r\n\r\nsrch_ci = \"2016-05-18\"\r\n\r\nif you convert both string to **datetime**, you get:\r\n\r\ndatetime = 2016-05-18 17:09:54\r\n\r\nsrch_ci = 2016-05-18 00:00:00\r\n\r\nThe time difference:\r\n\r\nsrch_ci - datetime = -1 days +06:50:06\r\n\r\nSo, I treat 0 day to -2 days as same-day booking.\r\nIt is just my case. There could be other reasons. I hope it can help you.", "votes": null }, { "id": "120656", "postDate": "05/19/2016 19:10:22", "content": "

I treat negative days the same way, and that's been my observation of those specific cases. Of note, I haven't got any improvement on my modeling approach using CI-DT or CO-CI, at least not yet, whether binned (by quartiles, etc.) or not. Has anyone got a noticeable bump on the LB using date diffs between date_time and CI/CO?

", "rawMarkdown": "I treat negative days the same way, and that's been my observation of those specific cases. Of note, I haven't got any improvement on my modeling approach using CI-DT or CO-CI, at least not yet, whether binned (by quartiles, etc.) or not. Has anyone got a noticeable bump on the LB using date diffs between date_time and CI/CO?", "votes": null }, { "id": "120760", "postDate": "05/20/2016 11:53:45", "content": "

I also tried CI-DT and CP-CI without much success.

", "rawMarkdown": "I also tried CI-DT and CP-CI without much success.", "votes": null }, { "id": "120773", "postDate": "05/20/2016 13:29:25", "content": "

@eipiplus1\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.

\n\n

Good catch to use ID-DT, I will try to add it.

", "rawMarkdown": "eipiplus1\r\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.\r\n\r\nGood catch to use ID-DT, I will try to add it.", "votes": null }, { "id": "120804", "postDate": "05/20/2016 15:58:07", "content": "

@YijunMa\ndid you bin it or use it as it?

", "rawMarkdown": "YijunMa\r\ndid you bin it or use it as it?", "votes": null }, { "id": "120842", "postDate": "05/20/2016 21:50:50", "content": "

[quote=YijunMa;120773]

\n\n

@eipiplus1\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.

\n\n

Good catch to use ID-DT, I will try to add it.

\n\n

[/quote]

\n\n

I'll try it again. It seems reasonable to think that hotel stay duration would matter in search results, so perhaps I did not use it correctly. By the way, did you mean CI-DT (not ID-DT - unless that was a sneaky little hint!). Replying to Shen, I applied a max only, so I ended up with 1..7 where 7 is 7+ days duration. I wonder if binning 1, 2-6, 7+ may be useful - or even 1, 2+.

", "rawMarkdown": "[quote=YijunMa;120773]\r\n\r\n@eipiplus1\r\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.\r\n\r\nGood catch to use ID-DT, I will try to add it.\r\n\r\n[/quote]\r\n\r\nI'll try it again. It seems reasonable to think that hotel stay duration would matter in search results, so perhaps I did not use it correctly. By the way, did you mean CI-DT (not ID-DT - unless that was a sneaky little hint!). Replying to Shen, I applied a max only, so I ended up with 1..7 where 7 is 7+ days duration. I wonder if binning 1, 2-6, 7+ may be useful - or even 1, 2+.", "votes": null }, { "id": "121239", "postDate": "05/25/2016 04:59:05", "content": "

Hello all, there is an error in one of the srch_ci (2161-10-00)

\n\n

test_recom[312921,] id date_time site_name posa_continent user_location_country user_location_region 312920 2015-09-27 12:17:51 13 1 46 244 user_location_city orig_destination_distance user_id is_mobile is_package channel srch_ci 33092 NA 1041293 0 0 10 2161-10-00 srch_co srch_adults_cnt srch_children_cnt srch_rm_cnt srch_destination_id 2016-01-22 2 0 1 6387 srch_destination_type_id hotel_continent hotel_country hotel_market 1 6 170 1894

", "rawMarkdown": "Hello all, there is an error in one of the srch_ci (2161-10-00)\r\n\r\ntest_recom[312921,] id date_time site_name posa_continent user_location_country user_location_region 312920 2015-09-27 12:17:51 13 1 46 244 user_location_city orig_destination_distance user_id is_mobile is_package channel srch_ci 33092 NA 1041293 0 0 10 2161-10-00 srch_co srch_adults_cnt srch_children_cnt srch_rm_cnt srch_destination_id 2016-01-22 2 0 1 6387 srch_destination_type_id hotel_continent hotel_country hotel_market 1 6 170 1894", "votes": null }, { "id": "121265", "postDate": "05/25/2016 08:11:57", "content": "

Parmod, if you mean there are outliers, then yes. Some are even further in the future.

", "rawMarkdown": "Parmod, if you mean there are outliers, then yes. Some are even further in the future.", "votes": null }, { "id": "121285", "postDate": "05/25/2016 11:36:32", "content": "

@ CPMP - In test.csv[312921,] "srch_ci" is 2161-10-00 , which is not correct for a test data from 2015. "2161" and "00" do not make any sense to me.

", "rawMarkdown": "CPMP - In test.csv[312921,] \"srch_ci\" is 2161-10-00 , which is not correct for a test data from 2015. \"2161\" and \"00\" do not make any sense to me.", "votes": null }, { "id": "121456", "postDate": "05/26/2016 14:12:49", "content": "

@CPMP - How did you treat dates that does not make sense?

", "rawMarkdown": "CPMP - How did you treat dates that does not make sense?", "votes": null }, { "id": "121459", "postDate": "05/26/2016 14:28:29", "content": "

I clip to 2012-01-01 and 2020-01-01

", "rawMarkdown": "I clip to 2012-01-01 and 2020-01-01", "votes": null } ], "comments": [ { "id": 119261, "author_name": "jiweiliu", "author_url": "", "post_date": "05/08/2016 16:38:28", "content": "

that sounds wrong but I guess it is safe to ignore them due to the small amount of data affected.

", "votes": null, "replies": [] }, { "id": 119262, "author_name": "amgmails", "author_url": "", "post_date": "05/08/2016 16:53:09", "content": "

thanks rcarson

", "votes": null, "replies": [] }, { "id": 119323, "author_name": "adamwoz", "author_url": "", "post_date": "05/09/2016 08:08:20", "content": "

It's likely a system error. If you use stay dates in your model it's safe to ignore these records.

", "votes": null, "replies": [] }, { "id": 119596, "author_name": "hswater1", "author_url": "", "post_date": "05/11/2016 17:28:31", "content": "

some of them are errors.\nsome are not.\nit's depends on how you calculate the day difference. if you, like what I did, convert them to datetime type, system treat sch_ci as 0:00 of check in day. For same-day booking, booking time is behind CI time.

", "votes": null, "replies": [] }, { "id": 120442, "author_name": "dot277", "author_url": "", "post_date": "05/18/2016 07:57:04", "content": "

@ShenHuang, Can you elaborate on how you treated this issue, it's a bit unclear. What do you mean by "system treat sch_ci as 0:00 of check in day" ?

", "votes": null, "replies": [] }, { "id": 120480, "author_name": "hswater1", "author_url": "", "post_date": "05/18/2016 15:35:25", "content": "

@dot277 \nFor example, when a guest arrives Las Vegas at 5pm today, he started looking for a room for tonight.\nExpedia gets the record:

\n\n

datetime = "2016-05-18 17:09:54"

\n\n

srch_ci = "2016-05-18"

\n\n

if you convert both string to datetime, you get:

\n\n

datetime = 2016-05-18 17:09:54

\n\n

srch_ci = 2016-05-18 00:00:00

\n\n

The time difference:

\n\n

srch_ci - datetime = -1 days +06:50:06

\n\n

So, I treat 0 day to -2 days as same-day booking.\nIt is just my case. There could be other reasons. I hope it can help you.

", "votes": null, "replies": [] }, { "id": 120656, "author_name": "siliconvalley", "author_url": "", "post_date": "05/19/2016 19:10:22", "content": "

I treat negative days the same way, and that's been my observation of those specific cases. Of note, I haven't got any improvement on my modeling approach using CI-DT or CO-CI, at least not yet, whether binned (by quartiles, etc.) or not. Has anyone got a noticeable bump on the LB using date diffs between date_time and CI/CO?

", "votes": null, "replies": [] }, { "id": 120760, "author_name": "cpmpml", "author_url": "", "post_date": "05/20/2016 11:53:45", "content": "

I also tried CI-DT and CP-CI without much success.

", "votes": null, "replies": [] }, { "id": 120773, "author_name": "ma350365879", "author_url": "", "post_date": "05/20/2016 13:29:25", "content": "

@eipiplus1\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.

\n\n

Good catch to use ID-DT, I will try to add it.

", "votes": null, "replies": [] }, { "id": 120804, "author_name": "hswater1", "author_url": "", "post_date": "05/20/2016 15:58:07", "content": "

@YijunMa\ndid you bin it or use it as it?

", "votes": null, "replies": [] }, { "id": 120842, "author_name": "siliconvalley", "author_url": "", "post_date": "05/20/2016 21:50:50", "content": "

[quote=YijunMa;120773]

\n\n

@eipiplus1\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.

\n\n

Good catch to use ID-DT, I will try to add it.

\n\n

[/quote]

\n\n

I'll try it again. It seems reasonable to think that hotel stay duration would matter in search results, so perhaps I did not use it correctly. By the way, did you mean CI-DT (not ID-DT - unless that was a sneaky little hint!). Replying to Shen, I applied a max only, so I ended up with 1..7 where 7 is 7+ days duration. I wonder if binning 1, 2-6, 7+ may be useful - or even 1, 2+.

", "votes": null, "replies": [] }, { "id": 121239, "author_name": "caldmet70", "author_url": "", "post_date": "05/25/2016 04:59:05", "content": "

Hello all, there is an error in one of the srch_ci (2161-10-00)

\n\n

test_recom[312921,] id date_time site_name posa_continent user_location_country user_location_region 312920 2015-09-27 12:17:51 13 1 46 244 user_location_city orig_destination_distance user_id is_mobile is_package channel srch_ci 33092 NA 1041293 0 0 10 2161-10-00 srch_co srch_adults_cnt srch_children_cnt srch_rm_cnt srch_destination_id 2016-01-22 2 0 1 6387 srch_destination_type_id hotel_continent hotel_country hotel_market 1 6 170 1894

", "votes": null, "replies": [] }, { "id": 121265, "author_name": "cpmpml", "author_url": "", "post_date": "05/25/2016 08:11:57", "content": "

Parmod, if you mean there are outliers, then yes. Some are even further in the future.

", "votes": null, "replies": [] }, { "id": 121285, "author_name": "caldmet70", "author_url": "", "post_date": "05/25/2016 11:36:32", "content": "

@ CPMP - In test.csv[312921,] "srch_ci" is 2161-10-00 , which is not correct for a test data from 2015. "2161" and "00" do not make any sense to me.

", "votes": null, "replies": [] }, { "id": 121456, "author_name": "dot277", "author_url": "", "post_date": "05/26/2016 14:12:49", "content": "

@CPMP - How did you treat dates that does not make sense?

", "votes": null, "replies": [] }, { "id": 121459, "author_name": "cpmpml", "author_url": "", "post_date": "05/26/2016 14:28:29", "content": "

I clip to 2012-01-01 and 2020-01-01

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "119249": "How come that some srch_ci have a date anterior to date_time?\r\nCan somebody help me understand?\r\n8457 events are concerned and 617 of them are actual bookings (facts from the training data).", "119261": "that sounds wrong but I guess it is safe to ignore them due to the small amount of data affected.", "119262": "thanks rcarson", "119323": "It's likely a system error. If you use stay dates in your model it's safe to ignore these records.", "119596": "some of them are errors.\r\nsome are not.\r\nit's depends on how you calculate the day difference. if you, like what I did, convert them to datetime type, system treat sch_ci as 0:00 of check in day. For same-day booking, booking time is behind CI time.", "120442": "ShenHuang, Can you elaborate on how you treated this issue, it's a bit unclear. What do you mean by \"system treat sch_ci as 0:00 of check in day\" ?", "120480": "dot277 \r\nFor example, when a guest arrives Las Vegas at 5pm today, he started looking for a room for tonight.\r\nExpedia gets the record:\r\n\r\ndatetime = \"2016-05-18 17:09:54\"\r\n\r\nsrch_ci = \"2016-05-18\"\r\n\r\nif you convert both string to **datetime**, you get:\r\n\r\ndatetime = 2016-05-18 17:09:54\r\n\r\nsrch_ci = 2016-05-18 00:00:00\r\n\r\nThe time difference:\r\n\r\nsrch_ci - datetime = -1 days +06:50:06\r\n\r\nSo, I treat 0 day to -2 days as same-day booking.\r\nIt is just my case. There could be other reasons. I hope it can help you.", "120656": "I treat negative days the same way, and that's been my observation of those specific cases. Of note, I haven't got any improvement on my modeling approach using CI-DT or CO-CI, at least not yet, whether binned (by quartiles, etc.) or not. Has anyone got a noticeable bump on the LB using date diffs between date_time and CI/CO?", "120760": "I also tried CI-DT and CP-CI without much success.", "120773": "eipiplus1\r\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.\r\n\r\nGood catch to use ID-DT, I will try to add it.", "120804": "YijunMa\r\ndid you bin it or use it as it?", "120842": "[quote=YijunMa;120773]\r\n\r\n@eipiplus1\r\nCO-CI is useful for me. I use random forest, and group by destination to train, i remember the feature importance function rank this feature pretty high. Be aware of feature selection, it might cause over fitting to the leakage data.\r\n\r\nGood catch to use ID-DT, I will try to add it.\r\n\r\n[/quote]\r\n\r\nI'll try it again. It seems reasonable to think that hotel stay duration would matter in search results, so perhaps I did not use it correctly. By the way, did you mean CI-DT (not ID-DT - unless that was a sneaky little hint!). Replying to Shen, I applied a max only, so I ended up with 1..7 where 7 is 7+ days duration. I wonder if binning 1, 2-6, 7+ may be useful - or even 1, 2+.", "121239": "Hello all, there is an error in one of the srch_ci (2161-10-00)\r\n\r\ntest_recom[312921,] id date_time site_name posa_continent user_location_country user_location_region 312920 2015-09-27 12:17:51 13 1 46 244 user_location_city orig_destination_distance user_id is_mobile is_package channel srch_ci 33092 NA 1041293 0 0 10 2161-10-00 srch_co srch_adults_cnt srch_children_cnt srch_rm_cnt srch_destination_id 2016-01-22 2 0 1 6387 srch_destination_type_id hotel_continent hotel_country hotel_market 1 6 170 1894", "121265": "Parmod, if you mean there are outliers, then yes. Some are even further in the future.", "121285": "CPMP - In test.csv[312921,] \"srch_ci\" is 2161-10-00 , which is not correct for a test data from 2015. \"2161\" and \"00\" do not make any sense to me.", "121456": "CPMP - How did you treat dates that does not make sense?", "121459": "I clip to 2012-01-01 and 2020-01-01" }, "source": "meta" }