{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EDA TOC\n1. Shape of Ratings - how many are users and items?\n2. Outlier buyers - these may be test buyers\n2. Behavior in terms of time - how many users buy items in the next 7 days?\n3. Behavior in terms of tops - what are the most bought items? item types? What are the items most bought one time only?\n4. Buyer archetypes - what do they buy?\n    - Loyal customer - >24 items (75 pct), what are the items being bought?\n    - See table:\n|                 | 8 items (50 pct)  | 3 items (25 pct) |   |   |\n|-----------------|-------------------|------------------|---|---|\n| Short Intervals | Obsessed Customer | Surge Customer   |   |   |\n| Long Intervals  | Regular Customer  | Repeat Customer  |   |   |\n|                 |                   |                  |   |   |\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom recommender_utils import RecommenderUtils\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\n\npd.set_option('display.max_colwidth', None)\n\n\nINPUT_DIR = \"/kaggle/input/h-and-m-personalized-fashion-recommendations\"\nMETADATA_ITEMS_FILE = f\"{INPUT_DIR}/articles.csv\"\nMETADATA_USERS_FILE = f\"{INPUT_DIR}/customers.csv\"\nMETADATA_TRANS_FILE = f\"{INPUT_DIR}/transactions_train.csv\"\nIMAGES_DIR = f\"{INPUT_DIR}/images\"\nSUBMISSIONS_SAMPLE_FILE = f\"{INPUT_DIR}/sample_submission.csv\"\n\n# IMPORTANT COLUMNS / GROUPS\nUSER_ID = \"customer_id\"\nITEM_ID = \"article_id\"\nRATING=\"price\"\nITEM_CATEGORICAL_COLS = [\"product_group_name\", \"graphical_appearance_name\", \"colour_group_name\", \"perceived_colour_value_name\",\n                        \"perceived_colour_master_name\", \"index_name\", \"index_group_name\", \"section_name\", \"garment_group_name\"]\nITEM_TEXT_COLS = [\"product_type_name\", \"prod_name\", \"department_name\", \"detail_desc\"]\nUSER_CATEGORICAL_COLS = [\"club_member_status\", \"fashion_news_frequency\", \"postal_code\"]\nUSER_BOOLEAN_COLS = [\"FN\", \"Active\"]\nUSER_NUMERICAL_COLS = [\"age\", \"min_purchase_interval\", \"median_purchase_interval\", \"max_purchase_interval\"]","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-10T10:19:30.337477Z","iopub.execute_input":"2022-02-10T10:19:30.337767Z","iopub.status.idle":"2022-02-10T10:19:31.574128Z","shell.execute_reply.started":"2022-02-10T10:19:30.337737Z","shell.execute_reply":"2022-02-10T10:19:31.573137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_items = pd.read_csv(METADATA_ITEMS_FILE)\ndf_users = pd.read_csv(METADATA_USERS_FILE)\ndf_txn = pd.read_csv(METADATA_TRANS_FILE)\n\nprint(f\"Item shape: {df_items.shape}, User shape: {df_users.shape}, Transaction Shape: {df_txn.shape}\")\n\n# one can transform the ids to categoricals, to save space\nprint(\"Items\")\nprint(df_items.columns)\nprint(\"Users\")\nprint(df_users.columns)\nprint(\"Ratings\")\nprint(df_txn.columns)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:19:31.575815Z","iopub.execute_input":"2022-02-10T10:19:31.576042Z","iopub.status.idle":"2022-02-10T10:20:44.499978Z","shell.execute_reply.started":"2022-02-10T10:19:31.576015Z","shell.execute_reply":"2022-02-10T10:20:44.498668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_items[:3])\ndisplay(df_users[:3])\ndisplay(df_txn[:3])","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:20:44.502707Z","iopub.execute_input":"2022-02-10T10:20:44.503001Z","iopub.status.idle":"2022-02-10T10:20:44.568893Z","shell.execute_reply.started":"2022-02-10T10:20:44.502965Z","shell.execute_reply":"2022-02-10T10:20:44.567992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Shape of Ratings\n- There are 1M+ users, 100k items, 31M ratings, with a sparsity of 2e-4\n- Price seems to be from >0.01 to 0.5. Bit weird.\n- Median of the # of items to users and vice versa seem healthy. As always, one should take care of sparsity. How many users have only one transaction? This can determine the importance of the side information (item and user metadata).\n    - One time purchase users are 11% while on the other end, items are 4%. It's not the worst I've seen.\n    - These sparse buying users can probably be saved by the item metadata. But it's just a small fraction anyway.\n- **[MODELING NOTES]Outliers: 99 PCT of items to users is 153, and the max is 1346 purchases! Perhaps it's good to remove these users in the modeling stage.**","metadata":{}},{"cell_type":"code","source":"utils = RecommenderUtils(user_id = \"customer_id\", item_id = \"article_id\", rating=\"price\")\nutils.print_ratings_shape(df_txn)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:20:44.570209Z","iopub.execute_input":"2022-02-10T10:20:44.570456Z","iopub.status.idle":"2022-02-10T10:20:53.524034Z","shell.execute_reply.started":"2022-02-10T10:20:44.570424Z","shell.execute_reply":"2022-02-10T10:20:53.522984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note:** For quicker EDA, I'll subset only 10% of the transactions for some of the graphs","metadata":{}},{"cell_type":"code","source":"ratings = df_txn.sample(frac = 0.1, random_state=42)\nutils.print_ratings_shape(ratings)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:20:53.525831Z","iopub.execute_input":"2022-02-10T10:20:53.526052Z","iopub.status.idle":"2022-02-10T10:21:00.93624Z","shell.execute_reply.started":"2022-02-10T10:20:53.526025Z","shell.execute_reply":"2022-02-10T10:21:00.934534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(ratings[\"price\"].describe())\nsns.distplot(ratings[\"price\"])","metadata":{"execution":{"iopub.status.busy":"2022-02-10T08:50:28.262313Z","iopub.execute_input":"2022-02-10T08:50:28.262602Z","iopub.status.idle":"2022-02-10T08:50:39.328158Z","shell.execute_reply.started":"2022-02-10T08:50:28.262573Z","shell.execute_reply":"2022-02-10T08:50:39.327033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# average number of items per user\nitem_per_user = df_txn.groupby(USER_ID)[ITEM_ID].nunique()\n\nfig = plt.figure(figsize=(10, 8))\nax = fig.add_subplot(211)\ndisplay(item_per_user.describe().to_frame(\"Number of items per user\"))\nsns.distplot(item_per_user, kde=False, ax=ax)\nax.set_title(\"Median number of items per user: {:.2f}\".format(item_per_user.median()))\n\n# # average number of users per item\nuser_per_item = df_txn.groupby(ITEM_ID)[USER_ID].nunique()\nax = fig.add_subplot(212)\ndisplay(user_per_item.describe().to_frame(\"Number of users per item\"))\nsns.distplot(user_per_item, kde=False, ax=ax)\nax.set_title(\"Median number of users per item: {:.2f}\".format(user_per_item.median()))\n\nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-02-10T08:49:13.820945Z","iopub.execute_input":"2022-02-10T08:49:13.82128Z","iopub.status.idle":"2022-02-10T08:50:11.456863Z","shell.execute_reply.started":"2022-02-10T08:49:13.821249Z","shell.execute_reply":"2022-02-10T08:50:11.456107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_time_buyers = item_per_user[item_per_user == 1]\none_time_purchases = user_per_item[user_per_item == 1]\n\npct_one_time_buyers = len(one_time_buyers) / len(item_per_user)\npct_one_time_purchases = len(one_time_purchases) / len(user_per_item)\n\nprint(f\"(users) One time buyers: {len(one_time_buyers)} ({pct_one_time_buyers:.4f})\")\nprint(f\"(items) One time purchases: {len(one_time_purchases)} ({pct_one_time_purchases:.4f})\")","metadata":{"execution":{"iopub.status.busy":"2022-02-10T08:50:11.458185Z","iopub.execute_input":"2022-02-10T08:50:11.458497Z","iopub.status.idle":"2022-02-10T08:50:11.493367Z","shell.execute_reply.started":"2022-02-10T08:50:11.458469Z","shell.execute_reply":"2022-02-10T08:50:11.492761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Outlier customers (99 pct): {item_per_user.quantile(0.99)}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:00:33.3047Z","iopub.execute_input":"2022-02-10T09:00:33.305063Z","iopub.status.idle":"2022-02-10T09:00:33.326353Z","shell.execute_reply.started":"2022-02-10T09:00:33.304983Z","shell.execute_reply":"2022-02-10T09:00:33.325326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cold start users\nWarning, cold start. Hence, the side info is REALLY IMPORTANT!","metadata":{}},{"cell_type":"code","source":"# there are cold-start users??\nnum_users_with_txn = set(df_users[USER_ID]).intersection(set(df_txn[USER_ID]))\nnum_cold_start = len(df_users) - len(num_users_with_txn)\nprint(f\"Num cold start users!: {num_cold_start}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Behavior in terms of time\n- The data has two years of purchasing behavior.\n- ~25% of the users buy items within 7 days. Almost 60% buy within 32 days (a complete month cycle). This is a frequent buying pattern! Fast fashion?\n    - **Recommenders can really boost the bottom line!**\n- **[MODELING NOTES] Include average purchasing interval for customers**","metadata":{}},{"cell_type":"code","source":"df_txn[\"t_dat\"] = pd.to_datetime(df_txn[\"t_dat\"])","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:03:03.470163Z","iopub.execute_input":"2022-02-10T09:03:03.470875Z","iopub.status.idle":"2022-02-10T09:03:05.508599Z","shell.execute_reply.started":"2022-02-10T09:03:03.470823Z","shell.execute_reply":"2022-02-10T09:03:05.5075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots(figsize=(15,10))\ndf_txn[\"t_dat\"].value_counts().plot()\ndisplay(df_txn[\"t_dat\"].describe())","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:05:18.615139Z","iopub.execute_input":"2022-02-10T09:05:18.615486Z","iopub.status.idle":"2022-02-10T09:05:19.818195Z","shell.execute_reply.started":"2022-02-10T09:05:18.615454Z","shell.execute_reply":"2022-02-10T09:05:19.817198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vc_year_months = df_txn[\"t_dat\"].dt.strftime(\"%Y-%m\").value_counts()\n\nvc_year_months = vc_year_months.sort_index()\ndisplay(vc_year_months)\n_, ax = plt.subplots(figsize=(15,10))\nvc_year_months.plot()","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:18:00.049813Z","iopub.execute_input":"2022-02-10T09:18:00.050157Z","iopub.status.idle":"2022-02-10T09:18:04.525408Z","shell.execute_reply.started":"2022-02-10T09:18:00.050126Z","shell.execute_reply":"2022-02-10T09:18:04.524316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# computing purchase lags\nstart_of_observation = df_txn[\"t_dat\"].min()\ndf_txn[\"offset_purchase_dat\"] = df_txn[\"t_dat\"] - start_of_observation\ndf_txn['prev_purchase_offset'] = df_txn.groupby(USER_ID)['offset_purchase_dat'].shift()\n\n# deduplicate same day purchases per customer\ndf_removed_same_day_purchases =  df_txn[df_txn[\"offset_purchase_dat\"] != df_txn[\"prev_purchase_offset\"]]\ndf_removed_same_day_purchases[\"purchase_lag\"] = df_removed_same_day_purchases[\"offset_purchase_dat\"] - df_removed_same_day_purchases[\"prev_purchase_offset\"]","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:44:26.136625Z","iopub.execute_input":"2022-02-10T09:44:26.136974Z","iopub.status.idle":"2022-02-10T09:45:46.089114Z","shell.execute_reply.started":"2022-02-10T09:44:26.13692Z","shell.execute_reply":"2022-02-10T09:45:46.087459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# looks right, yes?\ndf_removed_same_day_purchases[df_removed_same_day_purchases[USER_ID] == \"fffef3b6b73545df065b521e19f64bf6fe93bfd450ab20e02ce5d1e58a8f700b\"]","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:51:31.532639Z","iopub.execute_input":"2022-02-10T09:51:31.533143Z","iopub.status.idle":"2022-02-10T09:51:31.548414Z","shell.execute_reply.started":"2022-02-10T09:51:31.53311Z","shell.execute_reply":"2022-02-10T09:51:31.547749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is the unaveraged lag\ndisplay(df_removed_same_day_purchases[\"purchase_lag\"].describe(percentiles = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.]))\nplt.figure(figsize=(15,10))\nsns.distplot(df_removed_same_day_purchases[\"purchase_lag\"].dt.days)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:57:33.403974Z","iopub.execute_input":"2022-02-10T09:57:33.404686Z","iopub.status.idle":"2022-02-10T09:57:33.765783Z","shell.execute_reply.started":"2022-02-10T09:57:33.404638Z","shell.execute_reply":"2022-02-10T09:57:33.764898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is the averaged lag\naverage_purchase_lag_per_user = df_removed_same_day_purchases.groupby(USER_ID)[\"purchase_lag\"].mean()\naverage_purchase_lag_per_user.describe(percentiles = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.])\nplt.figure(figsize=(15,10))\nsns.distplot(average_purchase_lag_per_user.dt.days)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T09:57:18.331545Z","iopub.execute_input":"2022-02-10T09:57:18.332424Z","iopub.status.idle":"2022-02-10T09:57:18.751746Z","shell.execute_reply.started":"2022-02-10T09:57:18.332373Z","shell.execute_reply":"2022-02-10T09:57:18.751043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Behavior in terms of tops \n- what are the most bought items? item types? What are the items most bought one time only?\n    - these are ladieswear mostly\n- are there any differences with the once-bought items?\n    - Kids' items are mostly bought once. Specials?\n- commonalities?\n    - dark, trousers, upper garments","metadata":{}},{"cell_type":"code","source":"vc_item_id = df_txn[ITEM_ID].value_counts()\nmost_bought_items = vc_item_id[vc_item_id > vc_item_id.quantile(0.99)]\nmost_bought_items = most_bought_items.to_frame(\"count\").reset_index()\nmost_bought_items.rename(columns={\"index\" : ITEM_ID}, inplace=True)\n\nonce_bought_items = vc_item_id[vc_item_id == 1]\nonce_bought_items = once_bought_items.to_frame(\"count\").reset_index()\nonce_bought_items.rename(columns={\"index\" : ITEM_ID}, inplace=True)\n\ndf_items_popular = df_items.merge(most_bought_items)\ndf_items_rare = df_items.merge(once_bought_items)","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:30:02.121154Z","iopub.execute_input":"2022-02-10T10:30:02.121465Z","iopub.status.idle":"2022-02-10T10:30:03.975271Z","shell.execute_reply.started":"2022-02-10T10:30:02.121431Z","shell.execute_reply":"2022-02-10T10:30:03.974477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Most bought items and their categories","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(10, 30))\nfor idx, cat_col in enumerate(ITEM_CATEGORICAL_COLS):\n    ax = fig.add_subplot(len(ITEM_CATEGORICAL_COLS), 1, idx+1)\n    df_items_popular[cat_col].value_counts()[:5][::-1].plot.barh(ax=ax)\n    \n    ax.set_title(cat_col)\n    ax.spines['right'].set_visible(False)\n    ax.spines['top'].set_visible(False)\n    \nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:28:37.68574Z","iopub.execute_input":"2022-02-10T10:28:37.686609Z","iopub.status.idle":"2022-02-10T10:28:39.945306Z","shell.execute_reply.started":"2022-02-10T10:28:37.686548Z","shell.execute_reply":"2022-02-10T10:28:39.944354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Once bought items and their categories","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(10, 30))\nfor idx, cat_col in enumerate(ITEM_CATEGORICAL_COLS):\n    ax = fig.add_subplot(len(ITEM_CATEGORICAL_COLS), 1, idx+1)\n    df_items_rare[cat_col].value_counts()[:5][::-1].plot.barh(ax=ax)\n    \n    ax.set_title(cat_col)\n    ax.spines['right'].set_visible(False)\n    ax.spines['top'].set_visible(False)\n    \nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:30:35.358361Z","iopub.execute_input":"2022-02-10T10:30:35.358718Z","iopub.status.idle":"2022-02-10T10:30:37.565333Z","shell.execute_reply.started":"2022-02-10T10:30:35.358683Z","shell.execute_reply":"2022-02-10T10:30:37.564441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Buyer Archetypes\n- WIP!\n\n- what do they buy?\n    - Loyal customer - >24 items (75 pct), what are the items being bought?\n    - See table:\n|                 | 8 items (50 pct)  | 3 items (25 pct) |   |   |\n|-----------------|-------------------|------------------|---|---|\n| Short Intervals (1-7 days) | Obsessed Customer | Surge Customer   |   |   |\n| Long Intervals (30-120 days) | Regular Customer  | Repeat Customer  |   |   |\n|                 |                   |                  |   |   |","metadata":{"execution":{"iopub.status.busy":"2022-02-10T10:22:19.471162Z","iopub.execute_input":"2022-02-10T10:22:19.471493Z","iopub.status.idle":"2022-02-10T10:22:19.480979Z","shell.execute_reply.started":"2022-02-10T10:22:19.471457Z","shell.execute_reply":"2022-02-10T10:22:19.479884Z"}}},{"cell_type":"code","source":"# define short interval","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Features\n## Age is bimodal. How to impute this?\n- It's a toughie since, short of MICE, there is no single variable that can separate age.","metadata":{}},{"cell_type":"code","source":"\nsns.distplot(df_users[\"age\"].sample(frac=0.01))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(df_users.sample(frac=0.01), x=\"age\", col=\"club_member_status\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(df_users.sample(frac=0.01), x=\"age\", col=\"FN\", row=\"Active\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Items","metadata":{}},{"cell_type":"code","source":"for col in ITEM_CATEGORICAL_COLS:\n    display(df_items[col].value_counts())","metadata":{},"execution_count":null,"outputs":[]}]}