{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M: Overview of Tabular Part\nThis competition provides three CSV files (customers, articles, and transactions) and the article images. To begin with, I'd like to share this notebook to see what the tabular part of the data look like.","metadata":{}},{"cell_type":"code","source":"import glob\n\nimport numpy as np\nimport pandas as pd\npd.options.display.max_columns = None\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set()\n\nfrom PIL import Image","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-23T05:55:24.867796Z","iopub.execute_input":"2022-02-23T05:55:24.868647Z","iopub.status.idle":"2022-02-23T05:55:25.900780Z","shell.execute_reply.started":"2022-02-23T05:55:24.868544Z","shell.execute_reply":"2022-02-23T05:55:25.899863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Customers","metadata":{}},{"cell_type":"code","source":"df_customers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ndisplay(df_customers)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:26:45.31611Z","iopub.execute_input":"2022-02-17T09:26:45.316388Z","iopub.status.idle":"2022-02-17T09:26:49.637469Z","shell.execute_reply.started":"2022-02-17T09:26:45.316358Z","shell.execute_reply":"2022-02-17T09:26:49.63657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- `FN`: subscribe fashion news (1) or not (NaN)\n- `Active`: \"Active is if the customer is active for communication\"\n\n[source](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1683754)\n\n## Cleansing\nNaNs of these columns should be filled by 0s.","metadata":{}},{"cell_type":"code","source":"df_customers[[\"FN\", \"Active\"]] = df_customers[[\"FN\", \"Active\"]].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:26:49.639535Z","iopub.execute_input":"2022-02-17T09:26:49.640045Z","iopub.status.idle":"2022-02-17T09:26:49.68322Z","shell.execute_reply.started":"2022-02-17T09:26:49.639998Z","shell.execute_reply":"2022-02-17T09:26:49.6821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These columns are basically aligned.","metadata":{}},{"cell_type":"code","source":"# Jaccard similarity\n(df_customers[\"FN\"] * df_customers[\"Active\"]).sum() / (df_customers[\"FN\"] + df_customers[\"Active\"]).clip(0,1).sum()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:26:49.686838Z","iopub.execute_input":"2022-02-17T09:26:49.687112Z","iopub.status.idle":"2022-02-17T09:26:49.722195Z","shell.execute_reply.started":"2022-02-17T09:26:49.687079Z","shell.execute_reply":"2022-02-17T09:26:49.721193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`fashion_news_frequency` seems to contain some errors: `None` and `NONE`","metadata":{}},{"cell_type":"code","source":"df_customers[\"fashion_news_frequency\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:26:49.723948Z","iopub.execute_input":"2022-02-17T09:26:49.72432Z","iopub.status.idle":"2022-02-17T09:26:49.943696Z","shell.execute_reply.started":"2022-02-17T09:26:49.724288Z","shell.execute_reply":"2022-02-17T09:26:49.943071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's merge `NONE` with `None`","metadata":{}},{"cell_type":"code","source":"df_customers[\"fashion_news_frequency\"] = df_customers[\"fashion_news_frequency\"].str.replace(\"NONE\", \"None\")","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:40.782321Z","iopub.execute_input":"2022-02-17T09:27:40.783051Z","iopub.status.idle":"2022-02-17T09:27:41.717261Z","shell.execute_reply.started":"2022-02-17T09:27:40.782991Z","shell.execute_reply":"2022-02-17T09:27:41.716335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are still some missing values, but we leave it for now.","metadata":{}},{"cell_type":"code","source":"df_customers.isna().sum(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:43.558295Z","iopub.execute_input":"2022-02-17T09:27:43.559179Z","iopub.status.idle":"2022-02-17T09:27:44.189382Z","shell.execute_reply.started":"2022-02-17T09:27:43.559138Z","shell.execute_reply":"2022-02-17T09:27:44.188502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization\nLet's visualize the distribution of the numerical column.","metadata":{}},{"cell_type":"code","source":"sns.histplot(x=\"age\", data=df_customers, bins=20);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:46.011841Z","iopub.execute_input":"2022-02-17T09:27:46.012428Z","iopub.status.idle":"2022-02-17T09:27:46.559878Z","shell.execute_reply.started":"2022-02-17T09:27:46.012383Z","shell.execute_reply":"2022-02-17T09:27:46.558936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then the categorical columns.","metadata":{}},{"cell_type":"code","source":"sns.countplot(x='FN', data=df_customers);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:48.229684Z","iopub.execute_input":"2022-02-17T09:27:48.22996Z","iopub.status.idle":"2022-02-17T09:27:48.50834Z","shell.execute_reply.started":"2022-02-17T09:27:48.229933Z","shell.execute_reply":"2022-02-17T09:27:48.507559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='Active', data=df_customers);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:49.182475Z","iopub.execute_input":"2022-02-17T09:27:49.183268Z","iopub.status.idle":"2022-02-17T09:27:49.461221Z","shell.execute_reply.started":"2022-02-17T09:27:49.183231Z","shell.execute_reply":"2022-02-17T09:27:49.460378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='fashion_news_frequency', data=df_customers);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:27:50.373315Z","iopub.execute_input":"2022-02-17T09:27:50.373868Z","iopub.status.idle":"2022-02-17T09:27:52.142703Z","shell.execute_reply.started":"2022-02-17T09:27:50.373832Z","shell.execute_reply":"2022-02-17T09:27:52.14179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='club_member_status', data=df_customers);","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:28:08.217579Z","iopub.execute_input":"2022-02-17T09:28:08.218123Z","iopub.status.idle":"2022-02-17T09:28:10.086236Z","shell.execute_reply.started":"2022-02-17T09:28:08.218081Z","shell.execute_reply":"2022-02-17T09:28:10.085436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`postal_code` has 353k unique values and is heavy-tailed. The top area `2c29...` has by far the most customers. Is there some place where H&M stores are densly located?","metadata":{}},{"cell_type":"code","source":"df_customers[\"postal_code\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T09:29:13.456768Z","iopub.execute_input":"2022-02-17T09:29:13.457047Z","iopub.status.idle":"2022-02-17T09:29:14.333409Z","shell.execute_reply.started":"2022-02-17T09:29:13.457006Z","shell.execute_reply":"2022-02-17T09:29:14.332633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Articles","metadata":{}},{"cell_type":"code","source":"df_articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ndisplay(df_articles)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T12:25:23.177101Z","iopub.execute_input":"2022-02-17T12:25:23.1774Z","iopub.status.idle":"2022-02-17T12:25:24.134798Z","shell.execute_reply.started":"2022-02-17T12:25:23.177366Z","shell.execute_reply":"2022-02-17T12:25:24.133964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`article_id` is the primary key to join with the images.","metadata":{}},{"cell_type":"code","source":"df_articles[df_articles[\"article_id\"] // 10000000 == 10][[\"article_id\", \"product_code\", \"prod_name\", \"graphical_appearance_no\", \"graphical_appearance_name\", \"colour_group_code\", \"colour_group_name\", \"detail_desc\"]]","metadata":{"execution":{"iopub.status.busy":"2022-02-17T12:37:51.915743Z","iopub.execute_input":"2022-02-17T12:37:51.916309Z","iopub.status.idle":"2022-02-17T12:37:51.933181Z","shell.execute_reply.started":"2022-02-17T12:37:51.916246Z","shell.execute_reply":"2022-02-17T12:37:51.932605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure()\nfor i, image_path in enumerate(glob.glob(\"../input/h-and-m-personalized-fashion-recommendations/images/010/*\")):\n    article_id = image_path.split(\"/\")[-1]\n    plt.subplot(1, 3, i + 1)\n    plt.axis('off')\n    plt.title(article_id)\n    image = Image.open(image_path)\n    plt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T12:43:54.546176Z","iopub.execute_input":"2022-02-17T12:43:54.546474Z","iopub.status.idle":"2022-02-17T12:43:55.930075Z","shell.execute_reply.started":"2022-02-17T12:43:54.546446Z","shell.execute_reply":"2022-02-17T12:43:55.929463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are many other columns but most of them are self-explanatory. The columns from `product_code` to `garment_group_name` indicate the property of the product, such as the category, color, garment type, etc.","metadata":{}},{"cell_type":"markdown","source":"## Cleansing","metadata":{}},{"cell_type":"markdown","source":"Apart from `detail_desc`, there are no missing values.","metadata":{}},{"cell_type":"code","source":"df_articles.isna().sum(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T12:25:48.500359Z","iopub.execute_input":"2022-02-17T12:25:48.501173Z","iopub.status.idle":"2022-02-17T12:25:48.575223Z","shell.execute_reply.started":"2022-02-17T12:25:48.501122Z","shell.execute_reply":"2022-02-17T12:25:48.574215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization\nFirst, let's visualize the distribution of high cardinality columns.","metadata":{}},{"cell_type":"code","source":"cols = [\"product_code\", \"product_type_no\", \"department_no\"]\nfor col in cols:\n    plt.plot(df_articles[col].value_counts().values)\n    plt.title(f\"{col} distribution\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:02:00.908357Z","iopub.execute_input":"2022-02-17T13:02:00.908574Z","iopub.status.idle":"2022-02-17T13:02:01.887259Z","shell.execute_reply.started":"2022-02-17T13:02:00.908541Z","shell.execute_reply":"2022-02-17T13:02:01.886634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, low cardinality columns.","metadata":{}},{"cell_type":"code","source":"cols = [\"product_group_name\", \"garment_group_name\", \"graphical_appearance_name\", \"colour_group_name\",\n       \"perceived_colour_value_name\", \"perceived_colour_master_name\", \"index_name\", \"index_group_name\",\n       \"section_name\", \"garment_group_name\"]\nfor col in cols:\n    plt.figure(figsize=(10, 4))\n    ax = sns.countplot(x=col, data=df_articles)\n    ax.set_xticklabels(ax.get_xticklabels(), rotation=90)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:01:54.85605Z","iopub.execute_input":"2022-02-17T13:01:54.856368Z","iopub.status.idle":"2022-02-17T13:02:00.906958Z","shell.execute_reply.started":"2022-02-17T13:01:54.856331Z","shell.execute_reply":"2022-02-17T13:02:00.906061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Transactions","metadata":{}},{"cell_type":"code","source":"df_transactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\",\n                              dtype={\"t_dat\": \"object\", \"customer_id\": \"object\", \"article_id\": \"object\", \"price\": float, \"sales_channel_id\": int})\ndisplay(df_transactions)","metadata":{"execution":{"iopub.status.busy":"2022-02-23T05:55:37.242332Z","iopub.execute_input":"2022-02-23T05:55:37.242989Z","iopub.status.idle":"2022-02-23T05:56:52.623063Z","shell.execute_reply.started":"2022-02-23T05:55:37.242949Z","shell.execute_reply":"2022-02-23T05:56:52.622194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- `price`: scaled somehow for a privacy reason\n- `sales_channel_id`: `1` means offline and `2` means online\n\n[source](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306016#1680549)\n\n## Cleansing\nNo need for cleansing:","metadata":{}},{"cell_type":"code","source":"df_transactions.isna().sum(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:16:56.484955Z","iopub.execute_input":"2022-02-17T13:16:56.486137Z","iopub.status.idle":"2022-02-17T13:16:59.405117Z","shell.execute_reply.started":"2022-02-17T13:16:56.48607Z","shell.execute_reply":"2022-02-17T13:16:59.404295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualization\n\n`price` ranges between 0 and 0.6 and its frequency decays exponentially.","metadata":{}},{"cell_type":"code","source":"sns.histplot(x=\"price\", data=df_transactions, bins=20, log_scale=(False, True));","metadata":{"execution":{"iopub.status.busy":"2022-02-23T05:56:52.624907Z","iopub.execute_input":"2022-02-23T05:56:52.625429Z","iopub.status.idle":"2022-02-23T05:57:01.677124Z","shell.execute_reply.started":"2022-02-23T05:56:52.625382Z","shell.execute_reply":"2022-02-23T05:57:01.676315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As reported in [this discussion](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306016), the channel 1's transactions in April 2020 are missing.","metadata":{}},{"cell_type":"code","source":"df_transactions[\"t_month\"] = df_transactions[\"t_dat\"].str.rpartition(\"-\")[0]\ngr = df_transactions.groupby([\"sales_channel_id\", \"t_month\"]).count()['customer_id']\ngr","metadata":{"execution":{"iopub.status.busy":"2022-02-23T05:57:01.678736Z","iopub.execute_input":"2022-02-23T05:57:01.679309Z","iopub.status.idle":"2022-02-23T05:58:19.362669Z","shell.execute_reply.started":"2022-02-23T05:57:01.679224Z","shell.execute_reply":"2022-02-23T05:58:19.361818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This should by treated as anomaly, but here I impute it with 0 for visualization.","metadata":{}},{"cell_type":"code","source":"tmp = gr[1].reset_index()\ntmp = tmp.append({\"t_month\": \"2020-04\", \"customer_id\":0}, ignore_index=True)\ntmp = tmp.sort_values(\"t_month\")\nplt.plot(tmp['t_month'], tmp['customer_id'], label=\"offline\")\nplt.xticks(rotation=90)\n\ntmp = gr[2].reset_index()\nplt.plot(tmp['t_month'], tmp['customer_id'], label=\"online\")\nplt.legend()\nplt.title(\"Monthly transactions\");","metadata":{"execution":{"iopub.status.busy":"2022-02-23T05:58:19.364668Z","iopub.execute_input":"2022-02-23T05:58:19.364990Z","iopub.status.idle":"2022-02-23T05:58:19.824324Z","shell.execute_reply.started":"2022-02-23T05:58:19.364945Z","shell.execute_reply":"2022-02-23T05:58:19.823362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This result agree with what happened during April 2020 (due to Covid-19 lockdown, stores were closed and online shopping was dominant).","metadata":{}},{"cell_type":"markdown","source":"As one might expect, the customers and the articles are both heavy-tailed.","metadata":{}},{"cell_type":"code","source":"cols = [\"customer_id\", \"article_id\"]\nfor col in cols:\n    plt.hist(df_transactions[col].value_counts().values)\n    plt.yscale('log')\n    plt.title(f\"{col}'s appearance distribution\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T06:04:57.859523Z","iopub.execute_input":"2022-02-23T06:04:57.860217Z","iopub.status.idle":"2022-02-23T06:05:16.599762Z","shell.execute_reply.started":"2022-02-23T06:04:57.860178Z","shell.execute_reply":"2022-02-23T06:05:16.599129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It should be noted that the top customers purchased more than 1,000 times in two years, i.e. more than once a day.","metadata":{}},{"cell_type":"markdown","source":"Are those customers purchase regularly? If so, they are loyal ones who are highly likely to buy again in the test period. If not, it could be harmful when training a recommendation model.","metadata":{}},{"cell_type":"code","source":"tmp = df_transactions[\"customer_id\"].value_counts()\ncustomers = tmp[tmp >= 1000].keys().tolist()\ndf_transactions.query(\"customer_id in @customers\").groupby([\"customer_id\"])[\"t_month\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T14:39:49.440106Z","iopub.execute_input":"2022-02-17T14:39:49.440431Z","iopub.status.idle":"2022-02-17T14:39:58.35393Z","shell.execute_reply.started":"2022-02-17T14:39:49.440392Z","shell.execute_reply":"2022-02-17T14:39:58.35334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The training period is 25 months, so they are constantly buying items.","metadata":{}},{"cell_type":"markdown","source":"# Whant's Next?\nIn this notebook, I just scratched the surface of the tabular part of the data. Here are the possible next directions:\n- analyze customer-article interactions\n- predict whether each user will come back in the test period\n- use images\n- use text descriptions","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}