--- title: "TalkingData: Count Features' Analysis with Animation 🕓🕒▶️" author: "Pranav Pandya" output: html_document: number_sections: false toc: true toc_depth: 6 fig_width: 9 fig_height: 7 code_folding: hide highlight: tango theme: spacelab smart: true editor_options: chunk_output_type: console --- ![I just love relating kernels with my favorite tv shows :)](https://78.media.tumblr.com/ed86ef96d7aa17f80335f93e898eec87/tumblr_njdzshkajc1svefdfo1_500.gif) # Introduction In this kernel, I am attempting to visualize whether app was downloaded or not by associating main features with ip_count and app_count feature. Inspiration for app_count feature came after seeing [Alexey's kernel](https://www.kaggle.com/graf10a/pranav-s-lgbm-lb-0-9698?scriptVersionId=3073843) which scores highest on public LB at the moment. Feature importance in that model shows that app_count has second highest gain after app feature. To avoid freezing animation, I am using train_sample which has randomly selected observation from all days (6, 7, 8 and 9). I hope that this animations may be useful for feature engineering by means of adding some sort of constrint on freuency count features. ---------------------------------------------------------------------------------------------------- # Data preparation (train_sample.csv) ```{r message=FALSE} if (!require("pacman")) install.packages("pacman") pacman::p_load(knitr, tidyverse, data.table, DT, lubridate, DescTools, chron, viridis, plotly) options(scipen = 9999, warn = -1) train <- fread("../input/train_sample.csv") %>% mutate(day = day(click_time), hour = hour(click_time)) %>% add_count(ip) %>% rename("ip_count" = n) %>% add_count(app, day, hour) %>% rename("app_count" = n) %>% select(-c(attributed_time)) datatable(head(train)) ``` ## Sample size - train_sample.csv file is used which has randomly selected observations from all days. - Data is filtered for the 9th day (visualization purpose). Last section uses data from all days. - ip_count and app_count (app-day-hour) feature has been used as frequncy count variables. # Animations {.tabset .tabset-fade .tabset-pills} **Important notes: ** (useful for animation interpretation and interaction) - There are total 3 **main tabs** and **5 sub tabs** within last two main tabs. Please navigate through tabs to view each animations. - Animation will run after clicking **play** button on each plot. Data points will be plotted by hour on each interation. - Some of the plots (i.e device) have densed data points on single location. Kindly **drag and select area** for closer look. - Although I am unable to configure pause button at the moment but **slider** can be used to view data points for specific hours. - Hover over data points to view detailed information associated with it. - Label 0 and grey color means app was not downloaded (is_attributed == 0) - Label 1 and **red color** means app was downloaded (focus of this kernel) - app, os and device are used as factors to observe specific value by color in legned. Variables with large number of categories are treated as numeric. Update: - lowered animation speed - removing ~10 plots due to report being cut in half. (Kernel bug due to size limit). ## 1 Animation by all days ip ~ app_count (all days) to visualize result of download status ```{r message=FALSE} train %>% plot_ly(x = ~ip, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "All days | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "ip ~ app_count by ip categories for all days | Press play button on bottom left") ``` ## 2 ip_count ~ download {.tabset .tabset-fade} ### ip ~ ip_count Note that **126414** is the maximumn value of IP address in test data. ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~ip, y = ~ip_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~ip_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

ip_count : ', ip_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "ip ~ ip_count by is_attributed for day 9") ``` ### app ~ ip_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~app, y = ~ip_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~ip_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

ip_count : ', ip_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "app ~ ip_count by is_attributed for day 9") ``` ### device ~ ip_count Note: You can drag and select the region to view observations in particular device i.e device 0 and 1 which are the most used devices. ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~device, y = ~ip_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~ip_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

ip_count : ', ip_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "device ~ ip_count by is_attributed for day 9") ``` ### os ~ ip_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~os, y = ~ip_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~ip_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

ip_count : ', ip_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "os ~ ip_count by is_attributed for day 9") ``` ### channel ~ ip_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~channel, y = ~ip_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~ip_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

ip_count : ', ip_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "channel ~ ip_count by is_attributed for day 9") ``` ## 3 app_count ~ download {.tabset .tabset-fade} ### ip ~ app_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~ip, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "ip ~ app_count by is_attributed for day 9") ``` ### app ~ app_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~app, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "app ~ app_count by is_attributed for day 9") ``` ### device ~ app_count Note: You can drag and select the region to view observations in particular device i.e device 0 and 1 which are the most used devices. ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~device, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "device ~ app_count by is_attributed for day 9") ``` ### os ~ app_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~os, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "os ~ app_count by is_attributed for day 9") ``` ### channel ~ app_count ```{r message=FALSE} train %>% filter(day == 9) %>% plot_ly(x = ~channel, y = ~app_count, color = ~as.factor(is_attributed), colors = c("#9b9ea0", "#911d1d"), frame = ~hour, size = ~app_count, hoverinfo = 'text', text = ~paste('IP : ', ip, '

app_count : ', app_count, '
Day : ', day, '
Hour : ', hour, '
App : ', app, '
Device : ', device, '
OS : ', os, '
Channel : ', channel, '
App downloaded? : ', is_attributed), type = 'scatter', mode = 'markers') %>% animation_opts(frame = 2000) %>% animation_slider(currentvalue = list(prefix = "Day 9 | Hour: ", font = list(color="#104e8b"))) %>% layout(title = "channel ~ app_count by is_attributed for day 9") ``` Note: The one we are interested in is red data points which represents app was downloaded. # Final thoughts - Animations use comparatively small amount of data (randomly sampled for each days) from whole training set. More data points would certainly add more context to the whole picture. - Observations from the plots with ip feature can be useful to target data points after specific ip addresse that are not in test set. An approach could be to utilize those dynamic ips with some sort of grouping in my opinion. # Related work ---------------------------------------------------------------------------------------------------- - **R**: - [TalkingData: EDA to Model Evaluation](https://www.kaggle.com/pranav84/talkingdata-eda-to-model-evaluation-lb-0-9683/) - [TalkingData with Breaking Bad Feature Engineering (°ロ°)☝](https://www.kaggle.com/pranav84/talkingdata-with-breaking-bad-feature-engg) - [LightGBM in R with 75 mln rows](https://www.kaggle.com/pranav84/single-lightgbm-in-r-with-75-mln-rows-lb-0-9690?scriptVersionId=2989011) - [Single Histogram optimised XGBoost in R](https://www.kaggle.com/pranav84/single-xgboost-hist-hitting-0-9686-on-lb?scriptVersionId=2968554) - **Python**: - [LightGBM (Fixing unbalanced data)](https://www.kaggle.com/pranav84/lightgbm-fixing-unbalanced-data-lb-0-9680) - [XGBoost : Histogram Optimized Version](https://www.kaggle.com/pranav84/xgboost-histogram-optimized-version?scriptVersionId=2794247)