{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\nThis notebook was created for data understanding.  \nEDA for datasets can be done efficiently by R, I think.  \nI think that this content may be useful for understanding the data at the start.","metadata":{}},{"cell_type":"code","source":"options(warn=-1)\nlibrary(tidyverse) # metapackage of all tidyverse packages\nsuppressPackageStartupMessages(library(data.table))\nsuppressPackageStartupMessages(library(scales))\nsuppressPackageStartupMessages(library(patchwork)) # lemon is also usefull!\nsuppressPackageStartupMessages(library(lubridate))\nsuppressPackageStartupMessages(library(magick))\n\n#ggplot setting\ntheme_set(theme_minimal() +\n         theme(plot.title = element_text(size = 17, face = \"bold\"),\n               plot.subtitle = element_text(size = 15, face = \"bold\"),\n               axis.text.x = element_text(size = 15),\n               axis.text.y = element_text(size = 15),\n               axis.title.x = element_text(size = 18),\n               axis.title.y = element_text(size = 18),\n               legend.text = element_text(size = 15),\n               legend.title = element_text(size = 15),\n               strip.text.x = element_text(size = 15)\n              )\n          )\n\n#Choose your favorite color\nmycol = c(\"hotpink1\", \"turquoise\", \"lightslateblue\", \n          \"chartreuse2\",\"lightskyblue\", \"palegreen2\", \"grey60\", \"mistyrose2\")\n\n#show_col(mycol, borders = NA)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:00.957314Z","iopub.execute_input":"2022-03-12T04:42:00.959652Z","iopub.status.idle":"2022-03-12T04:42:01.126714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Glance at each dataset**","metadata":{}},{"cell_type":"markdown","source":"Glance at features in dataset and check na.","metadata":{}},{"cell_type":"code","source":"data_path <- \"../input/h-and-m-personalized-fashion-recommendations/\"\nimage_path <- paste0(data_path, \"images/\")\nlist.files(data_path)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:01.129269Z","iopub.execute_input":"2022-03-12T04:42:01.131016Z","iopub.status.idle":"2022-03-12T04:42:01.155805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles <- fread(paste0(data_path, \"articles.csv\"),\n               na.strings=c(\"\", \"NULL\"))\n\ncustomers <- fread(paste0(data_path, \"customers.csv\"),\n                  na.strings=c(\"\", \"NULL\"))\n\ntransactions_train <- fread(paste0(data_path, \"transactions_train.csv\"),\n               na.strings=c(\"\", \"NULL\"))\n\nsub<- fread(paste0(data_path, \"sample_submission.csv\"),\n                                  na.strings=c(\"\", \"NULL\"))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:01.159054Z","iopub.execute_input":"2022-03-12T04:42:01.160765Z","iopub.status.idle":"2022-03-12T04:42:15.743662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glimpse(articles)\ncat(\"\\nCheck NA\")\nchk_na <- colSums(sapply(articles, is.na)) \nchk_na[chk_na>0]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:15.747558Z","iopub.execute_input":"2022-03-12T04:42:15.749548Z","iopub.status.idle":"2022-03-12T04:42:15.811039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glimpse(customers)\ncat(\"\\nCheck NA\")\nchk_na <- colSums(sapply(customers, is.na)) \nchk_na[chk_na>0]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:15.814609Z","iopub.execute_input":"2022-03-12T04:42:15.816463Z","iopub.status.idle":"2022-03-12T04:42:16.478008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glimpse(transactions_train)\ncat(\"\\nCheck NA\")\nchk_na <- colSums(sapply(transactions_train, is.na)) \nchk_na[chk_na>0]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:16.481024Z","iopub.execute_input":"2022-03-12T04:42:16.482683Z","iopub.status.idle":"2022-03-12T04:42:23.423656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"glimpse(sub)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-03-12T04:42:23.426994Z","iopub.execute_input":"2022-03-12T04:42:23.428714Z","iopub.status.idle":"2022-03-12T04:42:23.454452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Check customer_id**","metadata":{}},{"cell_type":"code","source":"cat(paste0(\"\\nCompair with customer_id in sample submission: \", \n           all.equal(customers$customer_id, sub$customer_id)))\n\ncat(paste0(\"\\nDuplicate check: \",\n          length(customers$customer_id)-length(unique(customers$customer_id)),\n          \" duplication.\\n\"))\n\ncus_id <- transactions_train$customer_id %>% unique()\nd = length(customers$customer_id)-length(cus_id)\ncat(paste0(\"\\n\", d, \" id has no transaction data.\"))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:23.457177Z","iopub.execute_input":"2022-03-12T04:42:23.458682Z","iopub.status.idle":"2022-03-12T04:42:27.890788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"str_detect(customers$customer_id,\"[g-z]\") %>% sum()\nstr_detect(customers$customer_id,\"[a-f]\") %>% sum()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:27.895093Z","iopub.execute_input":"2022-03-12T04:42:27.89708Z","iopub.status.idle":"2022-03-12T04:42:30.174467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Customer ID may be hexadecimal.  ","metadata":{}},{"cell_type":"markdown","source":"# **articles**\nLet's examine the data in detail !","metadata":{}},{"cell_type":"markdown","source":"**Unique values**","metadata":{}},{"cell_type":"code","source":"sapply(articles, function(x){length(unique(x))}) %>%\ndata.table(feature = names(.), unique_values =.)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:30.178666Z","iopub.execute_input":"2022-03-12T04:42:30.180463Z","iopub.status.idle":"2022-03-12T04:42:30.284542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Article_id**  \n* The first 6 digits of article_id are the product_code.  \n* The article_id is a string, but this EDA reads it as an integer.","metadata":{}},{"cell_type":"code","source":"art_id <- as.integer(articles$article_id/1000)\nall.equal(art_id, articles$product_code)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:30.291532Z","iopub.execute_input":"2022-03-12T04:42:30.295394Z","iopub.status.idle":"2022-03-12T04:42:30.31842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**article_id, product_code**  \n* The last three digits of article_id seem to represent different colors and patterns.","metadata":{}},{"cell_type":"code","source":"show_image <- function(st1, st2){\n\n    image_list1 <- list.files(paste0(image_path, st1))\n    images1 <- image_list1[str_detect(image_list1, st2)]\n\n    for (i in 1:length(images1)){\n\n      img1 <- image_read(paste0(image_path, st1, images1[i]))\n      img1 <- image_resize(img1, geometry = \"255x\")\n      img1 <- image_annotate(img1, \n                     text = images1[i],\n                     gravity = \"north\",\n                     location = \"+0+15\",\n                     size = 30,\n                     #font = \"Noto Serif\",\n                     #style = \"Italic\",\n                     #weight = 700,\n                     color = \"black\")\n\n        if (i == 1 ){\n          imgs <- img1\n        }else{\n          imgs <- image_append(c(imgs, img1))  \n        }\n    }\n    print(imgs)\n\n}","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:30.32239Z","iopub.execute_input":"2022-03-12T04:42:30.324146Z","iopub.status.idle":"2022-03-12T04:42:30.33836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat(\"-example1-\")\n\nst1 = \"073/\"\nst2 = \"0730365\"\nshow_image(st1, st2)\n#\ncat(\"\\n\")\ncat(\"-example2-\")\nst1 = \"070/\"\nst2 = \"0704148\"\nshow_image(st1, st2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:30.342293Z","iopub.execute_input":"2022-03-12T04:42:30.34405Z","iopub.status.idle":"2022-03-12T04:42:32.016285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**product_code, prod_name**  \n* There are cases where multiple product_names correspond to one product_code, but the difference in product_name does not seem to make sense.","metadata":{}},{"cell_type":"code","source":"dt_tmp <- articles[, .(n = .N), by = .(product_code, prod_name)]%>% rename(n_name = n)\ndt_tmp2 <- articles %>% count(product_code) %>% arrange(-n) %>% rename(n_code = n)\n\ndt_tmp3 <- dt_tmp2 %>% left_join(dt_tmp, by = \"product_code\")\ndt_tmp3 %>% head(30)\ndt_tmp3 %>% tail(10)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:32.019833Z","iopub.execute_input":"2022-03-12T04:42:32.0218Z","iopub.status.idle":"2022-03-12T04:42:34.264427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**product_type_no, product_type_name duplication check**  \n* One product_type_name was used in duplicate in product_type_no. ","metadata":{}},{"cell_type":"code","source":"dt_tmp <- articles[, .(n = .N), by = .(product_type_no, product_type_name)]\ns <- dt_tmp$product_type_name\ncat(paste0(s[duplicated(s)], \" is duplicated.\"))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:34.267045Z","iopub.execute_input":"2022-03-12T04:42:34.268796Z","iopub.status.idle":"2022-03-12T04:42:34.296726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**department_no, department_name duplication check**  \n* 22 department_names was used in duplicate in department_no. ","metadata":{}},{"cell_type":"code","source":"dt_tmp <- articles[, .(n = .N), by = .(department_no, department_name)]\ns <- dt_tmp$department_name\ns[duplicated(s)]%>% unique() ","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:34.300502Z","iopub.execute_input":"2022-03-12T04:42:34.302319Z","iopub.status.idle":"2022-03-12T04:42:34.331424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Visualizations**  \nFirst, let's start with the place with the fewest unique values.  \nindex_group_name seems to be the largest classification.","metadata":{}},{"cell_type":"code","source":"articles$index_group_name <-  factor(articles$index_group_name, \n                                     levels = c(\"Sport\", \"Menswear\", \n                                                \"Divided\", \"Baby/Children\", \n                                                \"Ladieswear\"))\narticles[, n := .N, by = .(product_group_name)]\narticles[, N:=1]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:34.334032Z","iopub.execute_input":"2022-03-12T04:42:34.335613Z","iopub.status.idle":"2022-03-12T04:42:34.36558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"options(repr.plot.width = 10, repr.plot.height = 4)\narticles %>% count(index_group_name) %>% \n  ggplot(aes(x = index_group_name, y = n, fill = fct_rev(index_group_name))) +\n  geom_bar(stat = \"identity\", alpha = 0.5) +\n  #scale_fill_brewer(palette = \"Set3\") +\n  scale_fill_manual(values = mycol) +\n  theme(legend.position = \"none\")+\n  coord_flip()+\n  labs(title = \"- index_group_name -\",x = \"\",  y= \"count\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:34.369067Z","iopub.execute_input":"2022-03-12T04:42:34.370779Z","iopub.status.idle":"2022-03-12T04:42:34.996967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**How the contents of product_group_name are composed of index_group?**","metadata":{}},{"cell_type":"code","source":"lg <-\narticles %>% count(index_group_name) %>% \n  ggplot(aes(x = index_group_name, y = n, fill = fct_rev(index_group_name))) +\n  geom_bar(stat = \"identity\", alpha = 0.5) +\n  scale_fill_manual(values = mycol) +\n  labs(fill = \"index_group_name\")\n\n#legnedだけ表示\noptions(repr.plot.width = 8, repr.plot.height = 1.5)\nlg_legend <- lemon::g_legend(lg)\ngrid::grid.draw(lg_legend)\n\noptions(repr.plot.width = 12, repr.plot.height = 7)\narticles %>%\n  ggplot(aes(x =reorder(product_group_name, n), y = N, fill = index_group_name)) +\n  stat_summary(position = \"stack\", fun = \"sum\", geom = \"bar\", alpha = 0.5) +\n  scale_fill_manual(values = mycol[c(5,4,3,2,1)]) +\n  theme(legend.position = \"none\")+\n  coord_flip()+\n  labs(title = \"- product_group_name -\",x = \"\",  y= \"count\")\n\n#ここはstat_summary()で描画しないとなぜかalpha = 0.5 がきかなくなる。\n#geom_bar(stat = \"identity\", alpha = 0.5)だと描画はするが薄くならない。","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:34.999658Z","iopub.execute_input":"2022-03-12T04:42:35.001202Z","iopub.status.idle":"2022-03-12T04:42:36.300177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Next, take a look at product_type_name.**   \nTry to find out what kind of products are on sale.","metadata":{}},{"cell_type":"code","source":"options(repr.plot.width = 6, repr.plot.height = 1.5)\nlg_legend <- lemon::g_legend(lg)\ngrid::grid.draw(lg_legend)\n#\np_g_n <- articles %>% count(product_group_name) %>% \n  arrange(-n) %>% .[, product_group_name]\narticles[, n:= .N, by = .(product_group_name,product_type_name)]\n\noptions(repr.plot.width = 15, repr.plot.height = 20)\ng <- list()\n\nfor(i in 1:9){\n  dt_tmp <- articles[product_group_name == p_g_n[i]]\n  g_lev <- dt_tmp[,.N, by = index_group_name][order(-N)][, index_group_name]\n  g_lev <- as.integer(fct_rev(g_lev))\n  g_lev <-  sort(g_lev, decreasing = T)\n    \ng[[i]] <-\n  dt_tmp %>% \n  ggplot(aes(x = reorder(product_type_name, n), y= N, fill = (index_group_name)))+\n  stat_summary(position = \"stack\", fun = \"sum\", geom = \"bar\", alpha = 0.5)+\n  #scale_fill_manual(values = mycol[c(5,4,3,2,1)]) +\n  scale_fill_manual(values = mycol[g_lev]) +\n  theme(legend.position = \"none\")+\n  coord_flip()+\n  labs(title = p_g_n[i], x = \"\")\n}\n\ng[[1]] + g[[2]] + g[[3]] + g[[4]] + g[[5]] + g[[6]] + g[[7]] + g[[8]]+ g[[9]]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:36.303086Z","iopub.execute_input":"2022-03-12T04:42:36.304991Z","iopub.status.idle":"2022-03-12T04:42:40.347666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#legnedだけ表示\noptions(repr.plot.width = 6, repr.plot.height = 1.5)\nlg_legend <- lemon::g_legend(lg)\ngrid::grid.draw(lg_legend)\n\noptions(repr.plot.width = 15, repr.plot.height = 10)\nfor(i in 11:19){\n  dt_tmp <- articles[product_group_name == p_g_n[i]]\n  g_lev <- dt_tmp[,.N, by = index_group_name][, index_group_name]\n  g_lev <- as.integer(fct_rev(g_lev))\n  g_lev <-  sort(g_lev, decreasing = T)\n    \ng[[i]] <-\n  dt_tmp %>% \n    ggplot(aes(x = reorder(product_type_name, n), y= N, fill = (index_group_name)))+\n    stat_summary(position = \"stack\", fun = \"sum\", geom = \"bar\", alpha = 0.5)+\n    scale_fill_manual(values = mycol[g_lev]) +\n    theme(legend.position = \"none\")+\n    coord_flip()+\n    labs(title = p_g_n[i], x = \"\")\n}\n\ng[[11]] + g[[12]] + g[[13]] + g[[14]] + g[[15]] + g[[16]] + g[[17]] + g[[18]]+ g[[19]]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:40.350408Z","iopub.execute_input":"2022-03-12T04:42:40.352025Z","iopub.status.idle":"2022-03-12T04:42:42.572221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles[, \":=\"(n = NULL, N = NULL)]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:42.574929Z","iopub.execute_input":"2022-03-12T04:42:42.576878Z","iopub.status.idle":"2022-03-12T04:42:42.590985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm(dt_tmp)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:42.59345Z","iopub.execute_input":"2022-03-12T04:42:42.594996Z","iopub.status.idle":"2022-03-12T04:42:44.159278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **customers**","metadata":{}},{"cell_type":"markdown","source":"**Unique values**","metadata":{}},{"cell_type":"code","source":"sapply(customers, function(x){length(unique(x))}) %>%\ndata.table(feature = names(.), unique_values =.)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:44.161799Z","iopub.execute_input":"2022-03-12T04:42:44.163289Z","iopub.status.idle":"2022-03-12T04:42:44.666609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**postal_code: 10 in order of frequency**  \n* This also seems to be a hexadecimal number.  \n  \nTip  \ncustomers %>% count(postal_code)\n-->This is more than 10 times slower than the data table function below.","metadata":{}},{"cell_type":"code","source":"customers[, .N, by = postal_code][order(-N)][1:10]","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:44.669129Z","iopub.execute_input":"2022-03-12T04:42:44.670605Z","iopub.status.idle":"2022-03-12T04:42:44.90889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Extract with the most frequent postal_code.","metadata":{}},{"cell_type":"code","source":"ps_code <- customers[, .N, by = postal_code][order(-N)][,postal_code][1]\ndt_tmp <- customers[postal_code == ps_code]\ndt_tmp %>% head(5)\nsapply(dt_tmp, function(x){length(unique(x))}) %>%\ndata.table(feature = names(.), unique_values =.)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:42:44.911495Z","iopub.execute_input":"2022-03-12T04:42:44.913019Z","iopub.status.idle":"2022-03-12T04:42:45.665591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There seems to be no other feature that seems to correlate with the postal code.  \n* Information about where the customer lives?","metadata":{}},{"cell_type":"markdown","source":"# **transactions_train**","metadata":{}},{"cell_type":"markdown","source":"**Unique values**","metadata":{}},{"cell_type":"code","source":"sapply(transactions_train, function(x){length(unique(x))}) %>%\ndata.table(feature = names(.), unique_values =.)","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:45.66805Z","iopub.execute_input":"2022-03-12T04:42:45.669459Z","iopub.status.idle":"2022-03-12T04:42:55.747772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_conv <- transactions_train %>% left_join(articles, by = \"article_id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:42:55.750279Z","iopub.execute_input":"2022-03-12T04:42:55.751775Z","iopub.status.idle":"2022-03-12T04:43:15.785341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm(transactions_train)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:15.787883Z","iopub.execute_input":"2022-03-12T04:43:15.789344Z","iopub.status.idle":"2022-03-12T04:43:18.416401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Number of transactions per indedex_group_name**\n* Mainly sales of Ladieswear and Divided.  \n* Baby has a lot of Airticles but not a lot of sales.","metadata":{}},{"cell_type":"code","source":"options(repr.plot.width = 10, repr.plot.height = 4)\n\ndt_conv %>% count(index_group_name) %>% \n  ggplot(aes(x = index_group_name, y = n, fill = fct_rev(index_group_name))) +\n  geom_bar(stat = \"identity\", alpha = 0.5) +\n  #scale_fill_brewer(palette = \"Set3\") +\n  scale_fill_manual(values = mycol) +\n  scale_y_continuous(labels = comma)+\n  theme(legend.position = \"none\")+\n  coord_flip()+\n  labs(title = \"- index_group_name -\",x = \"\",  y= \"count of transaction\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:18.418869Z","iopub.execute_input":"2022-03-12T04:43:18.4203Z","iopub.status.idle":"2022-03-12T04:43:19.465021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The number of transactions for each customer_id.**  \n* Looking at the number of purchases for each customer_id, there are extreme outliers.","metadata":{}},{"cell_type":"code","source":"dt_tmp <- dt_conv[, .(n = .N), by = .(customer_id)][order(-n)] %>% \nmutate(category = \"all user_id\")\n\ng1 <-\n  dt_tmp %>% \n  ggplot(aes(x = category, y = n))+\n  scale_y_continuous(breaks = seq(0, 2000, 200))+\n  geom_boxplot(color = mycol[3], fill = mycol[5], alpha = 0.3)+\n  labs(subtitle = \"There are large outliers.\" ,\n       x =\"\", y = \"num of transactions\")\n\ng2 <- \n  dt_tmp %>% \n  ggplot(aes(x = category, y = n))+ \n  geom_boxplot(color = mycol[3], fill = mycol[5], alpha = 0.3)+\n  scale_y_continuous(breaks = seq(0, 2000, 20))+\n  coord_cartesian(ylim = c(0,100))+\n  labs(subtitle = \"Most customers make less than 30 purchases.\" \n       ,x =\"\", y = \"num of transactions\")\n\noptions(repr.plot.width = 12, repr.plot.height = 5)\ng1 + g2","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:19.468362Z","iopub.execute_input":"2022-03-12T04:43:19.471517Z","iopub.status.idle":"2022-03-12T04:43:34.000877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm(dt_tmp, g1, g2)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:34.005079Z","iopub.execute_input":"2022-03-12T04:43:34.007471Z","iopub.status.idle":"2022-03-12T04:43:36.605345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sales time series data-1**  \nFirst, let's roughly summarize by month.","metadata":{}},{"cell_type":"code","source":"dt_conv[, month := month(t_dat)]\ndt_conv[, year := year(t_dat)]\n\ndt_tmp <- dt_conv[, .(n = .N), by = .(year, month, sales_channel_id)] \nd1 <- as.character(dt_tmp$year)\nd2 <- str_sub(dt_tmp$month + 100, start = 2)\nd3 <- ymd(paste0(d1,d2,\"01\"))\nd3 <- as.POSIXct(d3)\ndt_tmp <- dt_tmp %>% mutate(DATE = d3)\n\noptions(repr.plot.width = 12, repr.plot.height = 4)\ndt_tmp %>% \n  ggplot(aes(x = DATE, y = n, color = factor(sales_channel_id)))+\n  geom_point(alpha = 0.8, size = 3)+\n  geom_line(alpha = 0.3)+\n  scale_x_datetime(breaks = \"2 months\", date_labels = \"%Y-%m\")+\n  scale_y_continuous(labels = comma)+\n  theme(axis.text.x = element_text(angle = 90, hjust = 1))+\n  labs(title = \"All customer_id\",x = \"\",y = \"num of transaction by month\",\n      color = \"1:Store, 2:Online\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:36.60907Z","iopub.execute_input":"2022-03-12T04:43:36.610771Z","iopub.status.idle":"2022-03-12T04:43:48.872741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm(dt_tmp)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:48.875875Z","iopub.execute_input":"2022-03-12T04:43:48.878Z","iopub.status.idle":"2022-03-12T04:43:51.375502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_tmp2 <- dt_conv[, .(n = .N), by = .(customer_id)][order(-n)]\nid1 <- dt_tmp2[1:10][, customer_id]\n\ndt_tmp <- dt_conv[customer_id %chin% id1] %>%\ncount(year, month, sales_channel_id)\nd1 <- as.character(dt_tmp$year)\nd2 <- str_sub(dt_tmp$month + 100, start = 2)\nd3 <- ymd(paste0(d1,d2,\"01\"))\nd3 <- as.POSIXct(d3)\ndt_tmp <- dt_tmp %>% mutate(DATE = d3)\n\ndt_tmp %>% \n  ggplot(aes(x = DATE, y = n, color = factor(sales_channel_id)))+\n  geom_point(alpha = 0.8, size = 3)+\n  geom_line(alpha = 0.3)+\n  scale_x_datetime(breaks = \"2 months\", date_labels = \"%Y-%m\")+\n  scale_y_continuous(labels = comma)+\n  theme(axis.text.x = element_text(angle = 90, hjust = 1))+\n  labs(title = \"Top 10 customer_id\",x = \"\",y = \"num of transaction by month\",\n      color = \"1:Store, 2:Online\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:51.379086Z","iopub.execute_input":"2022-03-12T04:43:51.380671Z","iopub.status.idle":"2022-03-12T04:43:56.766248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* It seems that the number of transactions in June is large.  \n* Customers with high purchases have a high percentage of online transactions.","metadata":{}},{"cell_type":"code","source":"rm(dt_tmp, dt_tmp2)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:56.770353Z","iopub.execute_input":"2022-03-12T04:43:56.772348Z","iopub.status.idle":"2022-03-12T04:43:59.229454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Sales time series data-2**  \nAfter all it seems better to see it on a daily basis.","metadata":{}},{"cell_type":"code","source":"d_line <-\n  c(as.POSIXct(max(dt_conv$t_dat))-years(2),\n    as.POSIXct(max(dt_conv$t_dat))-years(1))\nd_line2 <- \n  c(as.POSIXct(max(dt_conv$t_dat))+days(7),\n    as.POSIXct(max(dt_conv$t_dat))) \n\ng1 <- \ndt_conv[, .N, by =.(t_dat,sales_channel_id)] %>% \n  mutate(t_dat = as.POSIXct(t_dat)) %>%\n  ggplot(aes(x = t_dat, y = N, color = factor(sales_channel_id))) +\n  geom_line(alpha = 0.7)+\n  geom_vline(xintercept = d_line, linetype = \"dashed\", color =\"blue\")+\n  geom_vline(xintercept = d_line2, linetype = \"dashed\", color =\"red\")+\n  scale_x_datetime(breaks = \"2 months\", date_labels = \"%Y-%m\")+\n  scale_y_continuous(labels = comma)+\n  theme(axis.text.x = element_text(angle = 90, hjust = 1))+\n  labs(title = \"All customer_id\",x = \"\",y = \"num of transaction by day\",\n      color = \"1:Store, 2:Online\")+\n  geom_label(aes(x = d_line2[2], y = 160000, label = \"Prediction\\nperiod\"),\n             color = \"black\", size = 4)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:43:59.233133Z","iopub.execute_input":"2022-03-12T04:43:59.235159Z","iopub.status.idle":"2022-03-12T04:44:01.79716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_tmp2 <- dt_conv[, .(n = .N), by = .(customer_id)][order(-n)]\nid1 <- dt_tmp2[1:10][, customer_id]\n\ng2 <- \ndt_conv[customer_id %chin% id1][, .N, by =.(t_dat,sales_channel_id)] %>%\n  mutate(t_dat = as.POSIXct(t_dat)) %>%\n  ggplot(aes(x = t_dat, y = N, color = factor(sales_channel_id))) +\n  geom_line(alpha = 0.7)+\n  geom_vline(xintercept = d_line, linetype = \"dashed\", color =\"blue\")+\n  geom_vline(xintercept = d_line2, linetype = \"dashed\", color =\"red\")+\n  scale_x_datetime(breaks = \"2 months\", date_labels = \"%Y-%m\")+\n  theme(axis.text.x = element_text(angle = 90, hjust = 1))+\n  labs(title = \"Top 10 customer_id\",x = \"\",y = \"num of transaction by day\",\n      color = \"1:Store, 2:Online\")+\n  geom_label(aes(x = d_line2[2], y = 110, label = \"Prediction\\nperiod\"),\n             color = \"black\", size = 4)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:01.799897Z","iopub.execute_input":"2022-03-12T04:44:01.801461Z","iopub.status.idle":"2022-03-12T04:44:04.266316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"options(repr.plot.width = 15, repr.plot.height = 10)\ng1/g2","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:44:04.270095Z","iopub.execute_input":"2022-03-12T04:44:04.272071Z","iopub.status.idle":"2022-03-12T04:44:23.399202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The forecast period seems to be a date with large past fluctuations.","metadata":{}},{"cell_type":"code","source":"rm(dt_tmp2, g1, g2)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:23.402468Z","iopub.execute_input":"2022-03-12T04:44:23.404287Z","iopub.status.idle":"2022-03-12T04:44:26.064337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Combine Customers**","metadata":{}},{"cell_type":"code","source":"ps_c <- customers[, .N, by = postal_code][order(-N)][, postal_code]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:26.068189Z","iopub.execute_input":"2022-03-12T04:44:26.06997Z","iopub.status.idle":"2022-03-12T04:44:26.289483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt_conv <- dt_conv %>% left_join(customers, by = \"customer_id\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T04:44:26.293416Z","iopub.execute_input":"2022-03-12T04:44:26.295435Z","iopub.status.idle":"2022-03-12T04:44:56.279095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rm(customers)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:56.282978Z","iopub.execute_input":"2022-03-12T04:44:56.284751Z","iopub.status.idle":"2022-03-12T04:44:59.681957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#For making legend\nlg <- \ndt_conv[1:10, .N, by = .(age, sales_channel_id)] %>%\n  ggplot(aes(x = \" \", y = N, color = factor(sales_channel_id)))+\n  geom_point(alpha = 0.7)+\n  theme(axis.text.x = element_text(angle = 90, hjust = 1))+\n  labs(color = \"1:Store, 2:Online\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:59.684568Z","iopub.execute_input":"2022-03-12T04:44:59.686119Z","iopub.status.idle":"2022-03-12T04:44:59.710134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g <- list()\nfor(i in 1:6){\n  \n g[[i]]<-\n   dt_conv[postal_code %chin% ps_c[i]][, .N, by =.(t_dat,sales_channel_id)] %>% \n    mutate(t_dat = as.POSIXct(t_dat)) %>%\n    ggplot(aes(x = t_dat, y = N, color = factor(sales_channel_id))) +\n    geom_line(alpha = 0.7)+\n    geom_vline(xintercept = d_line, linetype = \"dashed\", color =\"blue\")+\n    geom_vline(xintercept = d_line2, linetype = \"dashed\", color =\"red\")+\n    geom_vline(xintercept = d_line, linetype = \"dashed\", color =\"blue\")+\n    scale_x_datetime(breaks = \"2 months\", date_labels = \"%Y-%m\")+\n    theme(legend.position = \"none\", axis.text.x = element_text(angle = 90, hjust = 1))+\n    labs(x =\"\", y = \"num of transaction by day\")\n}","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:44:59.713421Z","iopub.execute_input":"2022-03-12T04:44:59.715071Z","iopub.status.idle":"2022-03-12T04:45:02.028661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Draws the number of transactions in order of the number of elements in the postal_code.**","metadata":{}},{"cell_type":"code","source":"#描画用の文字列を作成\nfor(i in 1:6){\n  st1 <- paste0(\"g[[\",i,\"]]\")\n  if(i ==1){\n    st = st1}\n  else{\n    st <- paste(st, st1, sep = \"+\")}\n}\n\n#legendだけ表示\noptions(repr.plot.width =2, repr.plot.height = 1.5)\nlg_legend <- lemon::g_legend(lg)\ngrid::grid.draw(lg_legend)\n\noptions(repr.plot.width = 15, repr.plot.height = 8)\n\n#文字列をコマンドとして実行\neval(parse(text = st))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:02.031684Z","iopub.execute_input":"2022-03-12T04:45:02.033517Z","iopub.status.idle":"2022-03-12T04:45:03.840442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The majority of the top three postal_code purchases are made in stores.","metadata":{}},{"cell_type":"code","source":"rm(g)\ngc()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:03.843199Z","iopub.execute_input":"2022-03-12T04:45:03.844934Z","iopub.status.idle":"2022-03-12T04:45:06.546942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Online-only and shop-only products**  \n* There are likely to be many online-only products that shop-only users will not be able to purchase.  \n* In terms of transaction results, the products that sell the most are those that are handled both online and in the stores..","metadata":{}},{"cell_type":"code","source":"# Find products that are sold only online or only in the store.\n\np1<- dt_conv[sales_channel_id == 1, product_code] %>% unique()\np2<- dt_conv[sales_channel_id == 2, product_code] %>% unique()\n\nboth <- intersect(p1, p2)\nshop <- setdiff(p1, p2)\nnet <- setdiff(p2, p1)\n\ndt_conv[, sales_channel := \"1:both_online_shop\"]\ndt_conv[product_code %chin% shop, sales_channel := \"3:shop_only\"]\ndt_conv[product_code %chin% net, sales_channel := \"2:online_only\"]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:06.5495Z","iopub.execute_input":"2022-03-12T04:45:06.551011Z","iopub.status.idle":"2022-03-12T04:45:10.115759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g1 <-\ndt_conv[, .N, by = .(product_code, sales_channel)\n        ][, .N, by =sales_channel] %>%\n  ggplot(aes(x = \"\", y = N, fill = fct_rev(sales_channel)))+\n  geom_bar(stat = \"identity\", position = \"fill\", color = \"white\", alpha = 0.5)+\n  scale_fill_manual(values = mycol[c(4,1,3)])+\n  coord_polar('y')+\n  geom_text(aes(1.5, label=N), \n            position = position_fill(vjust=0.5), size= 8)+\n  labs(title = \"Number of prodcts\" ,fill = \"sales channel\")+\n  theme_void()\n\ng2 <-\ndt_conv[, .N, by = .(sales_channel)] %>%\n  ggplot(aes(x = \"\", y = N, fill = fct_rev(sales_channel)))+\n  geom_bar(stat = \"identity\", position = \"fill\", color = \"white\", alpha = 0.5)+\n  scale_fill_manual(values = mycol[c(4,1,3)])+\n  coord_polar('y')+\n  geom_text(aes(1.5, label=  ifelse(N > 30000000, paste0(as.integer(N/nrow(dt_conv)*100), \"%\") ,\"\")), \n            position = position_fill(vjust=0.5), size= 8)+\n  labs(title = \"Number of transactions\" ,fill = \"sales channel\")+\n  theme_void()\n\ng1 + g2","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:10.118355Z","iopub.execute_input":"2022-03-12T04:45:10.119837Z","iopub.status.idle":"2022-03-12T04:45:13.201645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**age**  \n* Looking at sales channels broken down by age. I thought that the online ratio would decrease with age, but this does not appear to be the case.  ","metadata":{}},{"cell_type":"code","source":"dt_conv[, generation := \"10s\"]\ndt_conv[age >= 20, generation := \"20s\"]\ndt_conv[age >= 30, generation := \"30s\"]\ndt_conv[age >= 40, generation := \"40s\"]\ndt_conv[age >= 50, generation := \"50s\"]\ndt_conv[age >= 60, generation := \"60s\"]\ndt_conv[age >= 70, generation := \"70s\"]\n\noptions(repr.plot.width = 10, repr.plot.height = 5)\n\ndt_conv[, .N, by=.(generation, sales_channel_id)] %>%\n  ggplot(aes(x = generation, y = N, fill = factor(sales_channel_id)))+\n  geom_bar(position = \"stack\", stat = \"identity\", alpha = 0.5)+\n  scale_y_continuous(labels = comma)+\n  labs(y = \"number of transctions\", fill = \"1:Store, 2:Online\")\n\n###","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:13.204352Z","iopub.execute_input":"2022-03-12T04:45:13.206003Z","iopub.status.idle":"2022-03-12T04:45:18.417462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Women in their 20s appear to be the primary users. \n* I thought \"Devided\" was for young people, but that doesn't seem to be the case.","metadata":{}},{"cell_type":"code","source":"dt_conv[, .N, by=.(generation, index_group_name)] %>%\n  ggplot(aes(x = generation, y = N, fill = (index_group_name)))+\n  geom_bar(position = \"stack\", stat = \"identity\", alpha = 0.5)+\n  scale_y_continuous(labels = comma)+\n  scale_fill_manual(values = mycol[c(5,4,3,2,1)]) +\n  labs(y = \"number of transctions\",fill = \"1:Store, 2:Online\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-12T04:45:18.420212Z","iopub.execute_input":"2022-03-12T04:45:18.421828Z","iopub.status.idle":"2022-03-12T04:45:19.735238Z"},"trusted":true},"execution_count":null,"outputs":[]}]}