Hello, everyone.
\nI am recently started to learn R and I find it really interesting. I need your help on creating 'Next Click' feature in R. The code below is what I made for it.
tdiff <- function(tdiff) {\n vc <- NULL\n for (x in 1:length(tdiff)) {\n interval <- interval(tdiff[x], tdiff[x+1])\n diffsecond <- as.integer(seconds(interval))\n vc <- append(vc, diffsecond)\n }\n vc <- ifelse(is.na(vc), 0, vc)\n vc <- append(vc, 0)\n return(vc)\n}\n\nadt[, clicker_Next := tdiff(click_time), by = list(ip, device, os)]\nWhen I try to run this code through Train_Sample, it is working but takes too long about more than 15 mins. But when it comes to Train CSV, this code doesn't work at all. If you could share your experty on this, it would be much appreciated.
\n\nThank you for your help in advance.
", "messages": [ { "id": 323580, "postDate": "2018-05-05T15:33:27.970Z", "content": "shift is known to be slow when too many groups are involved in data.table (because the := operator is not gforce-optimized yet in data.table)
You can do like the following for even faster lag/lead deltas:
\n\nadt[, deltaUApp_nextclick1 := c(click_time_num[-1], NA), by = list(ip, app, device, os)]\nadt[, deltaUApp_nextclick1 := deltaUApp_nextclick1 - click_time_num]\nadt[is.na(deltaUApp_nextclick1), deltaUApp_nextclick1 := 0]\n\nadt[, deltaUApp_prevclick1 := c(NA, click_time_num[-.N]), by = list(ip, app, device, os)]\nadt[, deltaUApp_prevclick1 := deltaUApp_prevclick1 - click_time_num]\nadt[is.na(deltaUApp_prevclick1), deltaUApp_prevclick1 := 0]\nIt assumes you have a integer/numeric time called click_time_num.
Takes less than 5 minutes to do it (i7-7700K, 5.0 GHz) for:
\n\ntext
\n\ndata[, click_time_num := as.numeric(click_time)]\n\ndata[, by_ipappdeviceoschannel_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"app\", \"device\", \"os\", \"channel\")]\ndata[, by_ipappdeviceoschannel_nextclick1 := by_ipappdeviceoschannel_nextclick1 - click_time_num]\ndata[is.na(by_ipappdeviceoschannel_nextclick1), by_ipappdeviceoschannel_nextclick1 := 0]\n\ndata[, by_iposdevice_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\")]\ndata[, by_iposdevice_nextclick1 := by_iposdevice_nextclick1 - click_time_num]\ndata[is.na(by_iposdevice_nextclick1), by_iposdevice_nextclick1 := 0]\n\ndata[, by_iposdeviceapp_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\", \"app\")]\ndata[, by_iposdeviceapp_nextclick1 := by_iposdeviceapp_nextclick1 - click_time_num]\ndata[is.na(by_iposdeviceapp_nextclick1), by_iposdeviceapp_nextclick1 := 0]\n\ndata[, by_ipchannel_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"channel\")]\ndata[, by_ipchannel_prevclick1 := click_time_num - by_ipchannel_prevclick1]\ndata[is.na(by_ipchannel_prevclick1), by_ipchannel_prevclick1 := 0]\n\ndata[, by_ipos_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"os\")]\ndata[, by_ipos_prevclick1 := click_time_num - by_ipos_prevclick1]\ndata[is.na(by_ipos_prevclick1), by_ipos_prevclick1 := 0]\n\n user system elapsed \n 281.92 6.17 288.11 \nIf computed in a parallel fashion (requires 80GB), it takes approx 80 seconds for all 5. In my code, NA conversion is unnecessary, better use 0 instead of NA in that combine (c) function to speed it even more (will require approx 1GB less RAM in that case).
", "rawMarkdown": "`shift` is known to be slow when too many groups are involved in data.table (because the `:=` operator is not `gforce`-optimized yet in data.table)\n\nYou can do like the following for even faster lag/lead deltas:\n\n adt[, deltaUApp_nextclick1 := c(click_time_num[-1], NA), by = list(ip, app, device, os)]\n adt[, deltaUApp_nextclick1 := deltaUApp_nextclick1 - click_time_num]\n adt[is.na(deltaUApp_nextclick1), deltaUApp_nextclick1 := 0]\n \n adt[, deltaUApp_prevclick1 := c(NA, click_time_num[-.N]), by = list(ip, app, device, os)]\n adt[, deltaUApp_prevclick1 := deltaUApp_prevclick1 - click_time_num]\n adt[is.na(deltaUApp_prevclick1), deltaUApp_prevclick1 := 0]\n\nIt assumes you have a integer/numeric time called `click_time_num`.\n\nTakes less than 5 minutes to do it (i7-7700K, 5.0 GHz) for:\n\n* all the 5 groups used on the top kernels\n* with ALL data (train + test_supplement)\n* under 16GB peak (OS not included)\n\ntext\n\n data[, click_time_num := as.numeric(click_time)]\n\n data[, by_ipappdeviceoschannel_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"app\", \"device\", \"os\", \"channel\")]\n data[, by_ipappdeviceoschannel_nextclick1 := by_ipappdeviceoschannel_nextclick1 - click_time_num]\n data[is.na(by_ipappdeviceoschannel_nextclick1), by_ipappdeviceoschannel_nextclick1 := 0]\n \n data[, by_iposdevice_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\")]\n data[, by_iposdevice_nextclick1 := by_iposdevice_nextclick1 - click_time_num]\n data[is.na(by_iposdevice_nextclick1), by_iposdevice_nextclick1 := 0]\n \n data[, by_iposdeviceapp_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\", \"app\")]\n data[, by_iposdeviceapp_nextclick1 := by_iposdeviceapp_nextclick1 - click_time_num]\n data[is.na(by_iposdeviceapp_nextclick1), by_iposdeviceapp_nextclick1 := 0]\n \n data[, by_ipchannel_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"channel\")]\n data[, by_ipchannel_prevclick1 := click_time_num - by_ipchannel_prevclick1]\n data[is.na(by_ipchannel_prevclick1), by_ipchannel_prevclick1 := 0]\n \n data[, by_ipos_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"os\")]\n data[, by_ipos_prevclick1 := click_time_num - by_ipos_prevclick1]\n data[is.na(by_ipos_prevclick1), by_ipos_prevclick1 := 0]\n\n user system elapsed \n 281.92 6.17 288.11 \n\nIf computed in a parallel fashion (requires 80GB), it takes approx 80 seconds for all 5. In my code, NA conversion is unnecessary, better use 0 instead of NA in that combine (c) function to speed it even more (will require approx 1GB less RAM in that case).", "votes": 7, "replies": [ { "id": 323585, "postDate": "2018-05-05T15:58:29.173Z", "content": "I didn't exptect to get this much of help from people and I really feel that I am lucky enough to have a great adviser. Thank you so much for your help. Have a good evening sir. :)
", "rawMarkdown": "I didn't exptect to get this much of help from people and I really feel that I am lucky enough to have a great adviser. Thank you so much for your help. Have a good evening sir. :)", "votes": 1 }, { "id": 323596, "postDate": "2018-05-05T16:34:37.037Z", "content": "I use below syntax (similarly for other grouping) after converting converting click_time to fastPOSIXct (GMT)\ndf[, ip_os_dev_prevclick:=click_time - shift(click_time, 1, fill = 0, \"lag\"),by = list(ip, os, device)]
\n\nLet me know if this works as fast as suggestion given above.
", "rawMarkdown": "I use below syntax (similarly for other grouping) after converting converting click_time to fastPOSIXct (GMT)\ndf[, ip_os_dev_prevclick:=click_time - shift(click_time, 1, fill = 0, \"lag\"),by = list(ip, os, device)]\n\nLet me know if this works as fast as suggestion given above. " }, { "id": 323597, "postDate": "2018-05-05T16:39:36.183Z", "content": "@laurae2, is this parallel implementation available out of the box. The only time I see multiple cores being used from data.table is fwrite (I may be wong). From my experience, fread and other calculations using data.table always use single thread only even if I set setDTthreads() to change the numbers
", "rawMarkdown": "@laurae2, is this parallel implementation available out of the box. The only time I see multiple cores being used from data.table is fwrite (I may be wong). From my experience, fread and other calculations using data.table always use single thread only even if I set setDTthreads() to change the numbers" }, { "id": 323602, "postDate": "2018-05-05T16:53:02.130Z", "content": "@lalthan You need to fork threads (Linux) or create a cluster (Windows), and create specific code to parallelize either asynchronously (futures/promises) or synchronously for the group of operation you want to do.
\n\nUsing fork it will take significantly less than 80GB (Windows requires copies which explodes RAM quickly).
", "rawMarkdown": "@lalthan You need to fork threads (Linux) or create a cluster (Windows), and create specific code to parallelize either asynchronously (futures/promises) or synchronously for the group of operation you want to do.\n\nUsing fork it will take significantly less than 80GB (Windows requires copies which explodes RAM quickly).", "votes": 2 } ] }, { "id": 323532, "postDate": "2018-05-05T11:53:42.400Z", "content": "Convert click_time to an integer and use the shift function. Assuming adt is a data.table it's quite fast, less than 5 minutes for the whole train+test dataset.
\n\ntime<-fast_strptime(adt$click_time, format = \"%Y-%m-%d %H:%M:%S\")\ni_time<-as.numeric(time)\nadt$i_time<-i_time\nadt[, deltaUApp:=shift(i_time,1,type = 'lead',fill=0)-i_time,by=list(ip, app, device, os)]) \nHope it works for you
", "rawMarkdown": "Convert click_time to an integer and use the *shift* function. Assuming adt is a data.table it's quite fast, less than 5 minutes for the whole train+test dataset.\n\n time<-fast_strptime(adt$click_time, format = \"%Y-%m-%d %H:%M:%S\")\n i_time<-as.numeric(time)\n adt$i_time<-i_time\n adt[, deltaUApp:=shift(i_time,1,type = 'lead',fill=0)-i_time,by=list(ip, app, device, os)]) \n\n\nHope it works for you\n\n \n \n", "votes": 3, "replies": [ { "id": 323539, "postDate": "2018-05-05T12:30:44.237Z", "content": "Thank you for your advise. I think it is really valid one, So I want to play with this tonight. I wish I could share some advise with other new comers in the future just like you did for me. Thanks a lot. :)
", "rawMarkdown": "Thank you for your advise. I think it is really valid one, So I want to play with this tonight. I wish I could share some advise with other new comers in the future just like you did for me. Thanks a lot. :)", "votes": 2 } ] }, { "id": 323583, "postDate": "2018-05-05T15:55:50.673Z", "content": "If you want to check multiple combos, the following does work for me on 16gb mbp...
\n\nxtr <- fread(paste('../input/train.csv', sep = ''))\nxtr[ ,click_time := parse_date_time(click_time, orders = 'YmdHMS')]\nxcols <- c('ip', 'app', 'device', 'os', 'channel')\n\nfor (nelem in 1:4)\n{\n xcomb <- combn(xcols, nelem)\n\n for (ii in 1:ncol(xcomb))\n {\n\n # next \n xnam <- paste('nxt', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lead'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(next_time - click_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x)); \n xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n\n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n\n rm(xmat)\n\n # last \n xnam <- paste('lst', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lag'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(click_time - next_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x),2); xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n\n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n\n rm(xmat)\n\n }\n print(paste(nelem,'way combos:done', sep = '-'))\n}\nHello, everyone.
\nI am recently started to learn R and I find it really interesting. I need your help on creating 'Next Click' feature in R. The code below is what I made for it.
tdiff <- function(tdiff) {\n vc <- NULL\n for (x in 1:length(tdiff)) {\n interval <- interval(tdiff[x], tdiff[x+1])\n diffsecond <- as.integer(seconds(interval))\n vc <- append(vc, diffsecond)\n }\n vc <- ifelse(is.na(vc), 0, vc)\n vc <- append(vc, 0)\n return(vc)\n}\n\nadt[, clicker_Next := tdiff(click_time), by = list(ip, device, os)]\nWhen I try to run this code through Train_Sample, it is working but takes too long about more than 15 mins. But when it comes to Train CSV, this code doesn't work at all. If you could share your experty on this, it would be much appreciated.
\n\nThank you for your help in advance.
", "rawMarkdown": "Hello, everyone. \nI am recently started to learn R and I find it really interesting. I need your help on creating 'Next Click' feature in R. The code below is what I made for it.\n\n tdiff <- function(tdiff) {\n vc <- NULL\n for (x in 1:length(tdiff)) {\n interval <- interval(tdiff[x], tdiff[x+1])\n diffsecond <- as.integer(seconds(interval))\n vc <- append(vc, diffsecond)\n }\n vc <- ifelse(is.na(vc), 0, vc)\n vc <- append(vc, 0)\n return(vc)\n }\n \n adt[, clicker_Next := tdiff(click_time), by = list(ip, device, os)]\n\n\nWhen I try to run this code through Train_Sample, it is working but takes too long about more than 15 mins. But when it comes to Train CSV, this code doesn't work at all. If you could share your experty on this, it would be much appreciated. \n\nThank you for your help in advance.\n\n", "votes": 1 }, { "id": 323531, "postDate": "2018-05-05T11:40:19.207Z", "content": "Computing this is also quite long with pandas in Python. You may just want to let your code run longer. If your code breaks, which is worse, then I'm afraid you'll need someone more expert than me on R to help.
", "rawMarkdown": "Computing this is also quite long with pandas in Python. You may just want to let your code run longer. If your code breaks, which is worse, then I'm afraid you'll need someone more expert than me on R to help.", "replies": [ { "id": 323537, "postDate": "2018-05-05T12:29:18.907Z", "content": "Thank you for your attention on my question. Have a great day!
", "rawMarkdown": "Thank you for your attention on my question. Have a great day!", "votes": 2 } ] }, { "id": 323606, "postDate": "2018-05-05T17:02:04.527Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 323580, "author_name": "Laurae", "author_url": "", "post_date": "2018-05-05T15:33:27.970000", "content": "shift is known to be slow when too many groups are involved in data.table (because the := operator is not gforce-optimized yet in data.table)
You can do like the following for even faster lag/lead deltas:
\n\nadt[, deltaUApp_nextclick1 := c(click_time_num[-1], NA), by = list(ip, app, device, os)]\nadt[, deltaUApp_nextclick1 := deltaUApp_nextclick1 - click_time_num]\nadt[is.na(deltaUApp_nextclick1), deltaUApp_nextclick1 := 0]\n\nadt[, deltaUApp_prevclick1 := c(NA, click_time_num[-.N]), by = list(ip, app, device, os)]\nadt[, deltaUApp_prevclick1 := deltaUApp_prevclick1 - click_time_num]\nadt[is.na(deltaUApp_prevclick1), deltaUApp_prevclick1 := 0]\nIt assumes you have a integer/numeric time called click_time_num.
Takes less than 5 minutes to do it (i7-7700K, 5.0 GHz) for:
\n\ntext
\n\ndata[, click_time_num := as.numeric(click_time)]\n\ndata[, by_ipappdeviceoschannel_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"app\", \"device\", \"os\", \"channel\")]\ndata[, by_ipappdeviceoschannel_nextclick1 := by_ipappdeviceoschannel_nextclick1 - click_time_num]\ndata[is.na(by_ipappdeviceoschannel_nextclick1), by_ipappdeviceoschannel_nextclick1 := 0]\n\ndata[, by_iposdevice_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\")]\ndata[, by_iposdevice_nextclick1 := by_iposdevice_nextclick1 - click_time_num]\ndata[is.na(by_iposdevice_nextclick1), by_iposdevice_nextclick1 := 0]\n\ndata[, by_iposdeviceapp_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\", \"app\")]\ndata[, by_iposdeviceapp_nextclick1 := by_iposdeviceapp_nextclick1 - click_time_num]\ndata[is.na(by_iposdeviceapp_nextclick1), by_iposdeviceapp_nextclick1 := 0]\n\ndata[, by_ipchannel_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"channel\")]\ndata[, by_ipchannel_prevclick1 := click_time_num - by_ipchannel_prevclick1]\ndata[is.na(by_ipchannel_prevclick1), by_ipchannel_prevclick1 := 0]\n\ndata[, by_ipos_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"os\")]\ndata[, by_ipos_prevclick1 := click_time_num - by_ipos_prevclick1]\ndata[is.na(by_ipos_prevclick1), by_ipos_prevclick1 := 0]\n\n user system elapsed \n 281.92 6.17 288.11 \nIf computed in a parallel fashion (requires 80GB), it takes approx 80 seconds for all 5. In my code, NA conversion is unnecessary, better use 0 instead of NA in that combine (c) function to speed it even more (will require approx 1GB less RAM in that case).
", "votes": 7, "replies": [ { "id": 323585, "author_name": "LegenDaD", "author_url": "", "post_date": "2018-05-05T15:58:29.173000", "content": "I didn't exptect to get this much of help from people and I really feel that I am lucky enough to have a great adviser. Thank you so much for your help. Have a good evening sir. :)
", "votes": 1, "replies": [] }, { "id": 323596, "author_name": "lalthan", "author_url": "", "post_date": "2018-05-05T16:34:37.037000", "content": "I use below syntax (similarly for other grouping) after converting converting click_time to fastPOSIXct (GMT)\ndf[, ip_os_dev_prevclick:=click_time - shift(click_time, 1, fill = 0, \"lag\"),by = list(ip, os, device)]
\n\nLet me know if this works as fast as suggestion given above.
", "votes": 0, "replies": [] }, { "id": 323597, "author_name": "lalthan", "author_url": "", "post_date": "2018-05-05T16:39:36.183000", "content": "@laurae2, is this parallel implementation available out of the box. The only time I see multiple cores being used from data.table is fwrite (I may be wong). From my experience, fread and other calculations using data.table always use single thread only even if I set setDTthreads() to change the numbers
", "votes": 0, "replies": [] }, { "id": 323602, "author_name": "Laurae", "author_url": "", "post_date": "2018-05-05T16:53:02.130000", "content": "@lalthan You need to fork threads (Linux) or create a cluster (Windows), and create specific code to parallelize either asynchronously (futures/promises) or synchronously for the group of operation you want to do.
\n\nUsing fork it will take significantly less than 80GB (Windows requires copies which explodes RAM quickly).
", "votes": 2, "replies": [] } ] }, { "id": 323532, "author_name": "Vicens Gaitan", "author_url": "", "post_date": "2018-05-05T11:53:42.400000", "content": "Convert click_time to an integer and use the shift function. Assuming adt is a data.table it's quite fast, less than 5 minutes for the whole train+test dataset.
\n\ntime<-fast_strptime(adt$click_time, format = \"%Y-%m-%d %H:%M:%S\")\ni_time<-as.numeric(time)\nadt$i_time<-i_time\nadt[, deltaUApp:=shift(i_time,1,type = 'lead',fill=0)-i_time,by=list(ip, app, device, os)]) \nHope it works for you
", "votes": 3, "replies": [ { "id": 323539, "author_name": "LegenDaD", "author_url": "", "post_date": "2018-05-05T12:30:44.237000", "content": "Thank you for your advise. I think it is really valid one, So I want to play with this tonight. I wish I could share some advise with other new comers in the future just like you did for me. Thanks a lot. :)
", "votes": 2, "replies": [] } ] }, { "id": 323583, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-05-05T15:55:50.673000", "content": "If you want to check multiple combos, the following does work for me on 16gb mbp...
\n\nxtr <- fread(paste('../input/train.csv', sep = ''))\nxtr[ ,click_time := parse_date_time(click_time, orders = 'YmdHMS')]\nxcols <- c('ip', 'app', 'device', 'os', 'channel')\n\nfor (nelem in 1:4)\n{\n xcomb <- combn(xcols, nelem)\n\n for (ii in 1:ncol(xcomb))\n {\n\n # next \n xnam <- paste('nxt', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lead'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(next_time - click_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x)); \n xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n\n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n\n rm(xmat)\n\n # last \n xnam <- paste('lst', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lag'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(click_time - next_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x),2); xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n\n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n\n rm(xmat)\n\n }\n print(paste(nelem,'way combos:done', sep = '-'))\n}\nComputing this is also quite long with pandas in Python. You may just want to let your code run longer. If your code breaks, which is worse, then I'm afraid you'll need someone more expert than me on R to help.
", "votes": 0, "replies": [ { "id": 323537, "author_name": "LegenDaD", "author_url": "", "post_date": "2018-05-05T12:29:18.907000", "content": "Thank you for your attention on my question. Have a great day!
", "votes": 2, "replies": [] } ] }, { "id": 323606, "author_name": "", "author_url": "", "post_date": "2018-05-05T17:02:04.527000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "323580": "`shift` is known to be slow when too many groups are involved in data.table (because the `:=` operator is not `gforce`-optimized yet in data.table)\n\nYou can do like the following for even faster lag/lead deltas:\n\n adt[, deltaUApp_nextclick1 := c(click_time_num[-1], NA), by = list(ip, app, device, os)]\n adt[, deltaUApp_nextclick1 := deltaUApp_nextclick1 - click_time_num]\n adt[is.na(deltaUApp_nextclick1), deltaUApp_nextclick1 := 0]\n \n adt[, deltaUApp_prevclick1 := c(NA, click_time_num[-.N]), by = list(ip, app, device, os)]\n adt[, deltaUApp_prevclick1 := deltaUApp_prevclick1 - click_time_num]\n adt[is.na(deltaUApp_prevclick1), deltaUApp_prevclick1 := 0]\n\nIt assumes you have a integer/numeric time called `click_time_num`.\n\nTakes less than 5 minutes to do it (i7-7700K, 5.0 GHz) for:\n\n* all the 5 groups used on the top kernels\n* with ALL data (train + test_supplement)\n* under 16GB peak (OS not included)\n\ntext\n\n data[, click_time_num := as.numeric(click_time)]\n\n data[, by_ipappdeviceoschannel_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"app\", \"device\", \"os\", \"channel\")]\n data[, by_ipappdeviceoschannel_nextclick1 := by_ipappdeviceoschannel_nextclick1 - click_time_num]\n data[is.na(by_ipappdeviceoschannel_nextclick1), by_ipappdeviceoschannel_nextclick1 := 0]\n \n data[, by_iposdevice_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\")]\n data[, by_iposdevice_nextclick1 := by_iposdevice_nextclick1 - click_time_num]\n data[is.na(by_iposdevice_nextclick1), by_iposdevice_nextclick1 := 0]\n \n data[, by_iposdeviceapp_nextclick1 := c(click_time_num[-1], NA), by = c(\"ip\", \"os\", \"device\", \"app\")]\n data[, by_iposdeviceapp_nextclick1 := by_iposdeviceapp_nextclick1 - click_time_num]\n data[is.na(by_iposdeviceapp_nextclick1), by_iposdeviceapp_nextclick1 := 0]\n \n data[, by_ipchannel_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"channel\")]\n data[, by_ipchannel_prevclick1 := click_time_num - by_ipchannel_prevclick1]\n data[is.na(by_ipchannel_prevclick1), by_ipchannel_prevclick1 := 0]\n \n data[, by_ipos_prevclick1 := c(NA, click_time_num[-.N]), by = c(\"ip\", \"os\")]\n data[, by_ipos_prevclick1 := click_time_num - by_ipos_prevclick1]\n data[is.na(by_ipos_prevclick1), by_ipos_prevclick1 := 0]\n\n user system elapsed \n 281.92 6.17 288.11 \n\nIf computed in a parallel fashion (requires 80GB), it takes approx 80 seconds for all 5. In my code, NA conversion is unnecessary, better use 0 instead of NA in that combine (c) function to speed it even more (will require approx 1GB less RAM in that case).", "323532": "Convert click_time to an integer and use the *shift* function. Assuming adt is a data.table it's quite fast, less than 5 minutes for the whole train+test dataset.\n\n time<-fast_strptime(adt$click_time, format = \"%Y-%m-%d %H:%M:%S\")\n i_time<-as.numeric(time)\n adt$i_time<-i_time\n adt[, deltaUApp:=shift(i_time,1,type = 'lead',fill=0)-i_time,by=list(ip, app, device, os)]) \n\n\nHope it works for you\n\n \n \n", "323583": "If you want to check multiple combos, the following does work for me on 16gb mbp...\n\n xtr <- fread(paste('../input/train.csv', sep = ''))\n xtr[ ,click_time := parse_date_time(click_time, orders = 'YmdHMS')]\n xcols <- c('ip', 'app', 'device', 'os', 'channel')\n \n for (nelem in 1:4)\n {\n xcomb <- combn(xcols, nelem)\n \n for (ii in 1:ncol(xcomb))\n {\n \n # next \n xnam <- paste('nxt', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lead'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(next_time - click_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x)); \n xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n \n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n \n rm(xmat)\n \n # last \n xnam <- paste('lst', paste(str_sub(xcomb[,ii],1,1), sep = '', collapse = ''), sep = '_')\n xtr[ ,next_time := shift(click_time, n= 1, type = 'lag'), by = c(xcomb[,ii])]\n xtr[ ,x := as.numeric(click_time - next_time, units= 'secs')]\n xtr[ ,next_time := NULL, with = TRUE]\n xc <- c('click_id', 'x'); xmat <- xtr[, ..xc, with = TRUE]; xtr[, x:= NULL, with = TRUE]\n xmat$x <- round(log1p(xmat$x),2); xmat$x[is.na(xmat$x)] <- -1; setnames(xmat, 'x', xnam)\n \n save(xmat, file = paste('../input/', paste(xnam, '_f',fnam, sep = ''), '.RData', sep = ''))\n \n rm(xmat)\n \n }\n print(paste(nelem,'way combos:done', sep = '-'))\n }\n\n", "323529": "Hello, everyone. \nI am recently started to learn R and I find it really interesting. I need your help on creating 'Next Click' feature in R. The code below is what I made for it.\n\n tdiff <- function(tdiff) {\n vc <- NULL\n for (x in 1:length(tdiff)) {\n interval <- interval(tdiff[x], tdiff[x+1])\n diffsecond <- as.integer(seconds(interval))\n vc <- append(vc, diffsecond)\n }\n vc <- ifelse(is.na(vc), 0, vc)\n vc <- append(vc, 0)\n return(vc)\n }\n \n adt[, clicker_Next := tdiff(click_time), by = list(ip, device, os)]\n\n\nWhen I try to run this code through Train_Sample, it is working but takes too long about more than 15 mins. But when it comes to Train CSV, this code doesn't work at all. If you could share your experty on this, it would be much appreciated. \n\nThank you for your help in advance.\n\n", "323531": "Computing this is also quite long with pandas in Python. You may just want to let your code run longer. If your code breaks, which is worse, then I'm afraid you'll need someone more expert than me on R to help.", "323606": "" } }