{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Do customers buy the SAME products again? \n\nTo someone who already read this(https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2)\n\nYou've probably wondered if customers would really buy the exact same thing again (!?)  \n\nAt least I (as a one of the cosumer) **couldn't imagine buying the same product within 1 month**!, \n\nSo I changed the above NOTEBOOK and adopted the strategy of \"not recommending the same product\". Then my score **DROPPED** from 0.02 to 0.006:(\n\nOkay... I have to accept that fact. Some customers will purchase exact the same products again within 1 week. But for making sure, I will check this in a data driven way.\n\n\nThis notebook is my survey report about **'Do customers buy the SAME products again?'**  \nConclusion first. Yes they will. And there are clear features in the characteristics of **customers** and **products** that they buy again.\n","metadata":{}},{"cell_type":"markdown","source":"# **Please upvote! :)**\n\n### If you haven't seen this notebook yet, take a look at this one too.\nhttps://www.kaggle.com/lichtlab/h-m-data-deep-dive-chap-1-understand-article","metadata":{}},{"cell_type":"markdown","source":"# Agenda\n1. Do customers buy the same product multiple times?\n2. Do customers buy different colors and sizes of the same product?\n3. Do customers buy the same product type?\n4. What kind of customer feature or product feature lead to 'one more time purchase'? (will be updated soon)","metadata":{}},{"cell_type":"markdown","source":"# 1. Do customers buy the same product multiple times?","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport plotly.express as px","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:17:29.179368Z","iopub.execute_input":"2022-02-21T14:17:29.179838Z","iopub.status.idle":"2022-02-21T14:17:30.858440Z","shell.execute_reply.started":"2022-02-21T14:17:29.179807Z","shell.execute_reply":"2022-02-21T14:17:30.857466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data prep","metadata":{}},{"cell_type":"code","source":"df_trans = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv',dtype={'article_id': str})\ndf_trans['t_dat'] = pd.to_datetime(df_trans['t_dat'])\ndf_trans = df_trans[df_trans['t_dat'] >= pd.to_datetime('2020-07-01')]\ndf_article = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv',dtype={'article_id': str})\n\ndf_article['idxgrp_idx_prdtyp'] = df_article['index_group_name'] + '_' + df_article['index_name'] + '_' + df_article['product_type_name']\n\ndf = pd.merge(\n    df_trans,\n    df_article,\n    on='article_id',\n    how='left'\n)\ndf['product_code'] = df['product_code'].astype(str)\ndf['num_week'] = df['t_dat'].dt.isocalendar().week\ndf['product_code'] = df['product_code'].astype(str)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-21T14:17:30.860431Z","iopub.execute_input":"2022-02-21T14:17:30.860732Z","iopub.status.idle":"2022-02-21T14:19:02.769264Z","shell.execute_reply.started":"2022-02-21T14:17:30.860699Z","shell.execute_reply":"2022-02-21T14:19:02.768439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Compare a set of products purchased in any given week with a set of products purchased 1, 2, or 3 weeks ago to see if they contain the SAME products.","metadata":{}},{"cell_type":"code","source":"def do_customers_purchase_same_AGGKEY(df, agg_key):\n    dfagg = df.groupby(['num_week','customer_id'])[[agg_key]].agg({\n            agg_key: lambda x: ','.join(x)\n    }).reset_index().rename(columns={agg_key: 'purchased_set'})\n    dfagg['num_2wk_before'] = dfagg['num_week'] + 2\n    dfagg = pd.merge(\n        dfagg[['num_week','customer_id','purchased_set']],\n        dfagg.rename(columns={'purchased_set': '2wk_before_purchased_set'})[['num_2wk_before','customer_id','2wk_before_purchased_set']],\n        left_on=['num_week', 'customer_id'],\n        right_on=['num_2wk_before', 'customer_id'],\n        how='left'\n    )\n    dfagg['num_1wk_before'] = dfagg['num_week'] + 1\n    dfagg = pd.merge(\n        dfagg,\n        dfagg.rename(columns={'purchased_set': '1wk_before_purchased_set'})[['num_1wk_before','customer_id','1wk_before_purchased_set']],\n        left_on=['num_week', 'customer_id'],\n        right_on=['num_1wk_before', 'customer_id'],\n        how='left'\n    )\n    dfagg['num_3wk_before'] = dfagg['num_week'] + 3\n    dfagg = pd.merge(\n        dfagg,\n        dfagg.rename(columns={'purchased_set': '3wk_before_purchased_set'})[['num_3wk_before','customer_id','3wk_before_purchased_set']],\n        left_on=['num_week', 'customer_id'],\n        right_on=['num_3wk_before', 'customer_id'],\n        how='left'\n\n    )\n    dfagg = dfagg[['num_week','customer_id','purchased_set','1wk_before_purchased_set','2wk_before_purchased_set','3wk_before_purchased_set']]\n    for col in ['purchased_set','1wk_before_purchased_set', '2wk_before_purchased_set', '3wk_before_purchased_set']:\n        dfagg[col] = dfagg[col].fillna('')\n        dfagg[col] = dfagg[col].str.split(',')\n    dfagg['2wk_before_purchased_set'] = dfagg['2wk_before_purchased_set'] + dfagg['1wk_before_purchased_set']\n    dfagg['3wk_before_purchased_set'] = dfagg['3wk_before_purchased_set'] + dfagg['2wk_before_purchased_set']\n    for col in ['purchased_set','1wk_before_purchased_set', '2wk_before_purchased_set', '3wk_before_purchased_set']:\n        dfagg[col] = dfagg[col].map(set)\n\n    dfagg['is_purchased_same_within_1wk'] = (dfagg['purchased_set'] & dfagg['1wk_before_purchased_set']).astype(int)\n    dfagg['is_purchased_same_within_2wk'] = (dfagg['purchased_set'] & dfagg['2wk_before_purchased_set']).astype(int)\n    dfagg['is_purchased_same_within_3wk'] = (dfagg['purchased_set'] & dfagg['3wk_before_purchased_set']).astype(int)\n    print(\n        len(dfagg[dfagg['is_purchased_same_within_3wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100,\n        len(dfagg[dfagg['is_purchased_same_within_2wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100,\n        len(dfagg[dfagg['is_purchased_same_within_1wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100\n    )\n    df_vis = pd.DataFrame({\n        'Pediod': ['Within_1wk', 'Within_2wk', 'Within_3wk'],\n        'Ratio': [len(dfagg[dfagg['is_purchased_same_within_1wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100,\n                  len(dfagg[dfagg['is_purchased_same_within_2wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100,\n                  len(dfagg[dfagg['is_purchased_same_within_3wk'] == 1]['customer_id'].unique()) / len(dfagg['customer_id'].unique()) * 100]\n    })\n    fig = px.bar(df_vis, x='Pediod', y='Ratio')\n    fig.show()\n    return dfagg","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:19:02.770978Z","iopub.execute_input":"2022-02-21T14:19:02.771210Z","iopub.status.idle":"2022-02-21T14:19:02.793167Z","shell.execute_reply.started":"2022-02-21T14:19:02.771182Z","shell.execute_reply":"2022-02-21T14:19:02.792203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Result","metadata":{}},{"cell_type":"code","source":"dfagg_article = do_customers_purchase_same_AGGKEY(df, 'article_id')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:19:02.795205Z","iopub.execute_input":"2022-02-21T14:19:02.795455Z","iopub.status.idle":"2022-02-21T14:19:42.672512Z","shell.execute_reply.started":"2022-02-21T14:19:02.795425Z","shell.execute_reply":"2022-02-21T14:19:42.671648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Conclusion - Do customers buy the same product multiple times?\n- **5.1%** of customers will buy the same product in one week   \n- **6.2%** will buy the same product within two weeks  \n- **6.6%** will buy the same product within three weeks  \n\nIn other words, most customers who purchase a product again purchase the same product within **two weeks**.","metadata":{}},{"cell_type":"markdown","source":"# 2. Do customers buy different colors and sizes of the same product?\nIn this analysis I aggregate data at 'product code'-granularity instead of 'article_id'-granularity.\n\n### Result","metadata":{}},{"cell_type":"code","source":"dfagg_prdcd = do_customers_purchase_same_AGGKEY(df, 'product_code')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:19:42.674032Z","iopub.execute_input":"2022-02-21T14:19:42.674440Z","iopub.status.idle":"2022-02-21T14:20:22.571477Z","shell.execute_reply.started":"2022-02-21T14:19:42.674394Z","shell.execute_reply":"2022-02-21T14:20:22.570155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Conclusion - Do customers buy different colors and sizes of the same product?\n- **6.8%** of customers will buy the same product code item in one week   \n- **8.5%** will buy the same product code item within two weeks  \n- **9.4%** will buy the same product code item within three weeks  \n\nCustomers seem to need a little more time when buying products with the same product code but different colors and sizes.","metadata":{}},{"cell_type":"markdown","source":"# 3. Do customers buy the same product type?\nIn this analysis I aggregate data at 'idxgrp_idx_prdtyp'-granularity instead of 'product code'-granularity.\n\n*idxgrp_idx_prdtyp explanation  \nhttps://www.kaggle.com/lichtlab/h-m-data-deep-dive-chap-1-understand-article\n\n### Result","metadata":{}},{"cell_type":"code","source":"dfagg_idxgrp_idx_prdtyp = do_customers_purchase_same_AGGKEY(df, 'idxgrp_idx_prdtyp')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:20:22.572400Z","iopub.execute_input":"2022-02-21T14:20:22.572918Z","iopub.status.idle":"2022-02-21T14:21:06.657430Z","shell.execute_reply.started":"2022-02-21T14:20:22.572885Z","shell.execute_reply":"2022-02-21T14:21:06.656428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Conclusion - Do customers buy the same product type?\n\n- **10.8%** of customers will buy the same idxgrp_idx_prdtyp item in one week   \n- **14.3%** will buy the same idxgrp_idx_prdtyp item within two weeks  \n- **16.5%** will buy the same idxgrp_idx_prdtyp item within three weeks  \n\n**16.5%** is huge! If we observe the purchasing behavior of customers over a short period of time, they seem to buy multiple similar products.","metadata":{}},{"cell_type":"markdown","source":"# 4. What kind of customer feature or product feature lead to 'one more time purchase'?\n\n## 4-1 Product feature\n## Data prep\nFor each customer, set a binary flag to indicate whether the customer has purchased a product with the same product_code in the last three weeks.","metadata":{}},{"cell_type":"code","source":"dfagg = df.sort_values(\"t_dat\")\\\n        .set_index('t_dat')\\\n        .groupby(['customer_id','product_code'])\\\n        .rolling(\"21d\")[[\"price\"]]\\\n        .count()\\\n        .reset_index()\\\n        .rename(columns={'price': 'num_purchased_same_article'})\ndfagg['is_purchased_same_prdcd_within_3wk'] = (dfagg['num_purchased_same_article'] > 1).astype(int)\ndfagg = dfagg[dfagg['t_dat'] > '2020-09-01']\ndfagg = pd.merge(\n    dfagg,\n    df.groupby(['idxgrp_idx_prdtyp','product_code'])[[]].count().reset_index(),\n    on='product_code'\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:21:06.658854Z","iopub.execute_input":"2022-02-21T14:21:06.659365Z","iopub.status.idle":"2022-02-21T14:22:50.802637Z","shell.execute_reply.started":"2022-02-21T14:21:06.659320Z","shell.execute_reply":"2022-02-21T14:22:50.801829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model design\nCreate a logistic regression model with the idxgrp_idx_prdtyp as the explanatory variable to see if it can explain the binary flags I have created earlier.","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\ndftrain = pd.get_dummies(dfagg[['idxgrp_idx_prdtyp']], drop_first=True)\ndftrain['is_purchased_same_prdcd_within_3wk'] = dfagg['is_purchased_same_prdcd_within_3wk']\ndftrain_partial = dftrain.sample(frac=0.01, random_state=0)\ndftrain_partial = dftrain_partial.dropna()\nfeature_cols = [col for col in dftrain.columns if col not in ['is_purchased_same_prdcd_within_3wk']]\nmodel = LogisticRegression(C=5.0, penalty=\"l1\", tol=0.01, solver=\"saga\")\nmodel.fit(dftrain_partial[feature_cols], dftrain_partial['is_purchased_same_prdcd_within_3wk'])\ndf_coef = pd.DataFrame(\n        model.coef_,\n        index=['coefficient'],\n        columns=feature_cols).T.reset_index()\ndf_coef['index'] = df_coef['index'].str.replace('idxgrp_idx_prdtyp_', '')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:24:40.997612Z","iopub.execute_input":"2022-02-21T14:24:40.998301Z","iopub.status.idle":"2022-02-21T14:24:44.498902Z","shell.execute_reply.started":"2022-02-21T14:24:40.998259Z","shell.execute_reply":"2022-02-21T14:24:44.497997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Types of products that are easy to purchase multiple times","metadata":{}},{"cell_type":"code","source":"px.bar(df_coef.sort_values(by='coefficient', ascending=False)[:15], x='index', y='coefficient')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:26:55.496735Z","iopub.execute_input":"2022-02-21T14:26:55.496990Z","iopub.status.idle":"2022-02-21T14:26:55.564145Z","shell.execute_reply.started":"2022-02-21T14:26:55.496963Z","shell.execute_reply":"2022-02-21T14:26:55.563246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Types of products that are not purchased multiple times","metadata":{}},{"cell_type":"code","source":"px.bar(df_coef.sort_values(by='coefficient', ascending=False)[-15:], x='index', y='coefficient')","metadata":{"execution":{"iopub.status.busy":"2022-02-21T14:27:36.298803Z","iopub.execute_input":"2022-02-21T14:27:36.299102Z","iopub.status.idle":"2022-02-21T14:27:36.365766Z","shell.execute_reply.started":"2022-02-21T14:27:36.299070Z","shell.execute_reply":"2022-02-21T14:27:36.365187Z"},"trusted":true},"execution_count":null,"outputs":[]}]}