Here is the code - call it a reward for CPMP saving me loads of time
\n\nDon't forget to use all the correct data and time order it ;)
", "messages": [ { "id": "306646", "postDate": "03/30/2018 19:32:45", "content": "Here is the code - call it a reward for CPMP saving me loads of time
\n\nDon't forget to use all the correct data and time order it ;)
", "rawMarkdown": "Here is the code - call it a reward for CPMP saving me loads of time\n\nDon't forget to use all the ***correct*** data and time order it ;)", "votes": null }, { "id": "308620", "postDate": "04/03/2018 19:54:28", "content": "Don't forget to use all the correct data and time order it ;)\nmeans to sort train and test file by click_time and the find delta, am I right?
", "rawMarkdown": "Don't forget to use all the correct data and time order it ;)\n\n\n\nmeans to sort train and test file by click_time and the find delta, am I right?", "votes": null }, { "id": "308627", "postDate": "04/03/2018 20:13:42", "content": "Indeed you are measure times between successive clicks
", "rawMarkdown": "Indeed you are measure times between *successive* clicks", "votes": null }, { "id": "308636", "postDate": "04/03/2018 20:36:02", "content": "\n\n\ncall it a reward for CPMP saving me loads of time
\n
@Scirpus, thank you. Wish I had your ability to solve all my programming problems in C++.
\n\n@CPMP, better post something here so we can reward you as well.
", "rawMarkdown": "> call it a reward for CPMP saving me loads of time\n\n@Scirpus, thank you. Wish I had your ability to solve all my programming problems in C++.\n\n@CPMP, better post something here so we can reward you as well.", "votes": null }, { "id": "308653", "postDate": "04/03/2018 21:20:30", "content": "I posted there: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53361#306625
", "rawMarkdown": "I posted there: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53361#306625", "votes": null }, { "id": "308859", "postDate": "04/04/2018 08:02:22", "content": "How to run this on linux(For c/c++ noobs like me ):
\ndownload CMAKE file by @Scirpus from here here and copy it in to the directory with main.cpp and CMakeCache file. And do the:
$ sudo apt-get install cmake \n$ cmake\n$ make\n$ ./GP\nNote: You might need to change directory path in main.cpp and CMakeCache files if you have different directory structure.
\nThanks
Hi Scirpus, I'm totally noob in C. Can you tell me against what columns are you grouping click-time to calculate the delta? \nFor eg; for every ip and app, you sort clicktime and calculate the delta? Or its something else?
\n\nSelf edit : Saw the older Time Delta thread. Got it.
", "rawMarkdown": "Hi Scirpus, I'm totally noob in C. Can you tell me against what columns are you grouping click-time to calculate the delta? \nFor eg; for every ip and app, you sort clicktime and calculate the delta? Or its something else?\n\nSelf edit : Saw the older Time Delta thread. Got it.", "votes": null }, { "id": "308958", "postDate": "04/04/2018 12:37:02", "content": "Using R data.table goes pretty fast as well (on the whole train+test approx 10 min).
\n\nIf I get it right (let me know otherwise), the time delta @Scirpus is computing is equivalent to the following:
\n\n dt[,tt:=as.numeric(click_time)]\n setorder(dt, tt)\n\n dt[,delta_ip :=tt-shift(tt), by=.(ip)]\n dt[,delta_ip_dev :=tt-shift(tt), by=.(ip,device)]\n dt[,delta_ip_app :=tt-shift(tt), by=.(ip,app)]\n dt[,delta_ip_chan:=tt-shift(tt), by=.(ip,channel)]\n\n dt[,tt:=NULL]\n\n vars <- dt %>% names %gv% \"^delta_\" \n for(jj in vars){\n set(dt,i= which(is.na(dt[[jj]])), j=jj, value=-9999)\n }\nWhere %gv% is a custom function to grep values
", "rawMarkdown": "Using R data.table goes pretty fast as well (on the whole train+test approx 10 min).\n \nIf I get it right (let me know otherwise), the time delta @Scirpus is computing is equivalent to the following:\n\n\n dt[,tt:=as.numeric(click_time)]\n setorder(dt, tt)\n \n dt[,delta_ip :=tt-shift(tt), by=.(ip)]\n dt[,delta_ip_dev :=tt-shift(tt), by=.(ip,device)]\n dt[,delta_ip_app :=tt-shift(tt), by=.(ip,app)]\n dt[,delta_ip_chan:=tt-shift(tt), by=.(ip,channel)]\n \n dt[,tt:=NULL]\n \n vars <- dt %>% names %gv% \"^delta_\" \n for(jj in vars){\n set(dt,i= which(is.na(dt[[jj]])), j=jj, value=-9999)\n }\n\n Where %gv% is a custom function to grep values", "votes": null }, { "id": "309038", "postDate": "04/04/2018 14:40:05", "content": "So it would be
\n\ndf['time_delta'] = df.sort_values(['click_time']).groupby('ip', 'app', 'device', os', 'channel')['click_time'].diff()\nIn pandas right?
", "rawMarkdown": "So it would be \n\n df['time_delta'] = df.sort_values(['click_time']).groupby('ip', 'app', 'device', os', 'channel')['click_time'].diff()\n\nIn pandas right?", "votes": null }, { "id": "309041", "postDate": "04/04/2018 14:50:00", "content": "Pandas one is very very very slow...
", "rawMarkdown": "Pandas one is very very very slow...", "votes": null }, { "id": "309083", "postDate": "04/04/2018 16:04:14", "content": "Yeah just wanted to see if I understood
", "rawMarkdown": "Yeah just wanted to see if I understood", "votes": null }, { "id": "309153", "postDate": "04/04/2018 18:19:13", "content": "You only need to sort by click_time, that's faster.
", "rawMarkdown": "You only need to sort by click_time, that's faster.", "votes": null }, { "id": "309398", "postDate": "04/05/2018 08:24:33", "content": "Could someone upload the results to kaggle? Would like to test it but it keeps crashing after 53million rows for me.
", "rawMarkdown": "Could someone upload the results to kaggle? Would like to test it but it keeps crashing after 53million rows for me.", "votes": null }, { "id": "309513", "postDate": "04/05/2018 13:54:10", "content": "Pandas diff requires a ridiculous amount of memory (probably why it's crashing). We could do it in chunks, I guess.
", "rawMarkdown": "Pandas diff requires a ridiculous amount of memory (probably why it's crashing). We could do it in chunks, I guess.", "votes": null }, { "id": "309681", "postDate": "04/05/2018 19:48:56", "content": "OK, I've tried to do it in Python, rather kludgily, I'm afraid, but it runs in reasonable time, and (after a great deal of difficulty) doesn't breach the memory limit. I haven't tested the result, so if someone wants to check it against what Scirpus has, that would be useful. (The output is in pickle format with NaN values for the first occurrence of each combined category. Obviously, assuming it's correct, one could reuse the code for different aggregations and so on.)
", "rawMarkdown": "OK, I've tried to [do it in Python][1], rather kludgily, I'm afraid, but it runs in reasonable time, and (after a great deal of difficulty) doesn't breach the memory limit. I haven't tested the result, so if someone wants to check it against what Scirpus has, that would be useful. (The output is in pickle format with NaN values for the first occurrence of each combined category. Obviously, assuming it's correct, one could reuse the code for different aggregations and so on.)\n\n [1]: https://www.kaggle.com/aharless/time-deltas", "votes": null }, { "id": "316772", "postDate": "04/19/2018 21:18:43", "content": "Here's a Makefile for a simple build without CMake. @Scirpus Did you intend to attach CMakeLists.txt instead of CMakeCache.txt?
", "rawMarkdown": "Here's a Makefile for a simple build without CMake. @Scirpus Did you intend to attach CMakeLists.txt instead of CMakeCache.txt?", "votes": null }, { "id": "316857", "postDate": "04/20/2018 04:52:01", "content": "Quote:Don't forget to use all the correct data and time order it ;)\nYou mean to say data is not already sorted?
", "rawMarkdown": "Quote:Don't forget to use all the correct data and time order it ;)\nYou mean to say data is not already sorted?", "votes": null }, { "id": "316866", "postDate": "04/20/2018 05:10:49", "content": "Perhaps it has to do if you plan to do it with merged train and test files.
", "rawMarkdown": "Perhaps it has to do if you plan to do it with merged train and test files.", "votes": null }, { "id": "316902", "postDate": "04/20/2018 07:01:47", "content": "I just added this as a reminder to check - especially when using the test supplement
", "rawMarkdown": "I just added this as a reminder to check - especially when using the test supplement", "votes": null }, { "id": "316903", "postDate": "04/20/2018 07:02:11", "content": "Add CMake File - thanks
", "rawMarkdown": "Add CMake File - thanks", "votes": null } ], "comments": [ { "id": 308620, "author_name": "sohaibomar", "author_url": "", "post_date": "04/03/2018 19:54:28", "content": "Don't forget to use all the correct data and time order it ;)\nmeans to sort train and test file by click_time and the find delta, am I right?
", "votes": null, "replies": [ { "id": 308627, "author_name": "scirpus", "author_url": "", "post_date": "04/03/2018 20:13:42", "content": "Indeed you are measure times between successive clicks
", "votes": null, "replies": [] }, { "id": 308953, "author_name": "nuhsikander", "author_url": "", "post_date": "04/04/2018 12:27:27", "content": "Hi Scirpus, I'm totally noob in C. Can you tell me against what columns are you grouping click-time to calculate the delta? \nFor eg; for every ip and app, you sort clicktime and calculate the delta? Or its something else?
\n\nSelf edit : Saw the older Time Delta thread. Got it.
", "votes": null, "replies": [] } ] }, { "id": 308636, "author_name": "tilii7", "author_url": "", "post_date": "04/03/2018 20:36:02", "content": "\n\n\ncall it a reward for CPMP saving me loads of time
\n
@Scirpus, thank you. Wish I had your ability to solve all my programming problems in C++.
\n\n@CPMP, better post something here so we can reward you as well.
", "votes": null, "replies": [ { "id": 308653, "author_name": "cpmpml", "author_url": "", "post_date": "04/03/2018 21:20:30", "content": "I posted there: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53361#306625
", "votes": null, "replies": [] } ] }, { "id": 308859, "author_name": "sohaibomar", "author_url": "", "post_date": "04/04/2018 08:02:22", "content": "How to run this on linux(For c/c++ noobs like me ):
\ndownload CMAKE file by @Scirpus from here here and copy it in to the directory with main.cpp and CMakeCache file. And do the:
$ sudo apt-get install cmake \n$ cmake\n$ make\n$ ./GP\nNote: You might need to change directory path in main.cpp and CMakeCache files if you have different directory structure.
\nThanks
Using R data.table goes pretty fast as well (on the whole train+test approx 10 min).
\n\nIf I get it right (let me know otherwise), the time delta @Scirpus is computing is equivalent to the following:
\n\n dt[,tt:=as.numeric(click_time)]\n setorder(dt, tt)\n\n dt[,delta_ip :=tt-shift(tt), by=.(ip)]\n dt[,delta_ip_dev :=tt-shift(tt), by=.(ip,device)]\n dt[,delta_ip_app :=tt-shift(tt), by=.(ip,app)]\n dt[,delta_ip_chan:=tt-shift(tt), by=.(ip,channel)]\n\n dt[,tt:=NULL]\n\n vars <- dt %>% names %gv% \"^delta_\" \n for(jj in vars){\n set(dt,i= which(is.na(dt[[jj]])), j=jj, value=-9999)\n }\nWhere %gv% is a custom function to grep values
", "votes": null, "replies": [] }, { "id": 309038, "author_name": "michaelsnell", "author_url": "", "post_date": "04/04/2018 14:40:05", "content": "So it would be
\n\ndf['time_delta'] = df.sort_values(['click_time']).groupby('ip', 'app', 'device', os', 'channel')['click_time'].diff()\nIn pandas right?
", "votes": null, "replies": [ { "id": 309041, "author_name": "cczaixian", "author_url": "", "post_date": "04/04/2018 14:50:00", "content": "Pandas one is very very very slow...
", "votes": null, "replies": [] }, { "id": 309083, "author_name": "michaelsnell", "author_url": "", "post_date": "04/04/2018 16:04:14", "content": "Yeah just wanted to see if I understood
", "votes": null, "replies": [] }, { "id": 309153, "author_name": "cpmpml", "author_url": "", "post_date": "04/04/2018 18:19:13", "content": "You only need to sort by click_time, that's faster.
", "votes": null, "replies": [] }, { "id": 309513, "author_name": "aharless", "author_url": "", "post_date": "04/05/2018 13:54:10", "content": "Pandas diff requires a ridiculous amount of memory (probably why it's crashing). We could do it in chunks, I guess.
", "votes": null, "replies": [] } ] }, { "id": 309398, "author_name": "michaelsnell", "author_url": "", "post_date": "04/05/2018 08:24:33", "content": "Could someone upload the results to kaggle? Would like to test it but it keeps crashing after 53million rows for me.
", "votes": null, "replies": [] }, { "id": 309681, "author_name": "aharless", "author_url": "", "post_date": "04/05/2018 19:48:56", "content": "OK, I've tried to do it in Python, rather kludgily, I'm afraid, but it runs in reasonable time, and (after a great deal of difficulty) doesn't breach the memory limit. I haven't tested the result, so if someone wants to check it against what Scirpus has, that would be useful. (The output is in pickle format with NaN values for the first occurrence of each combined category. Obviously, assuming it's correct, one could reuse the code for different aggregations and so on.)
", "votes": null, "replies": [] }, { "id": 316772, "author_name": "pliptor", "author_url": "", "post_date": "04/19/2018 21:18:43", "content": "Here's a Makefile for a simple build without CMake. @Scirpus Did you intend to attach CMakeLists.txt instead of CMakeCache.txt?
", "votes": null, "replies": [ { "id": 316903, "author_name": "scirpus", "author_url": "", "post_date": "04/20/2018 07:02:11", "content": "Add CMake File - thanks
", "votes": null, "replies": [] } ] }, { "id": 316857, "author_name": "adityakumarsinha", "author_url": "", "post_date": "04/20/2018 04:52:01", "content": "Quote:Don't forget to use all the correct data and time order it ;)\nYou mean to say data is not already sorted?
", "votes": null, "replies": [ { "id": 316866, "author_name": "pliptor", "author_url": "", "post_date": "04/20/2018 05:10:49", "content": "Perhaps it has to do if you plan to do it with merged train and test files.
", "votes": null, "replies": [] }, { "id": 316902, "author_name": "scirpus", "author_url": "", "post_date": "04/20/2018 07:01:47", "content": "I just added this as a reminder to check - especially when using the test supplement
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "306646": "Here is the code - call it a reward for CPMP saving me loads of time\n\nDon't forget to use all the ***correct*** data and time order it ;)", "308620": "Don't forget to use all the correct data and time order it ;)\n\n\n\nmeans to sort train and test file by click_time and the find delta, am I right?", "308627": "Indeed you are measure times between *successive* clicks", "308636": "> call it a reward for CPMP saving me loads of time\n\n@Scirpus, thank you. Wish I had your ability to solve all my programming problems in C++.\n\n@CPMP, better post something here so we can reward you as well.", "308653": "I posted there: https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/53361#306625", "308859": "How to run this on linux(For c/c++ noobs like me ):