{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **HnM RecSys Notebook 9417**","metadata":{}},{"cell_type":"markdown","source":"## **Table of Contents**\n* [**1. Imports**](#Imports) \n* [**2. Pre-Processing**](#Pre-Processing)\n* [**3. Exploratory Data Analysis**](#Exploratory-Data-Analysis) \n * [**3.1 Articles**](#EDA::Articles) \n * [**3.2 Customers**](#EDA::Customers)\n * [**3.3 Transactions**](#EDA::Transactions)","metadata":{"_kg_hide-output":false,"tags":[]}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib\nimport seaborn as sns\nimport os\nimport re\nimport warnings\n# import cudf # switch on P100 GPU for this to work in Kaggle\n# import cupy as cp\n\n# Importing data\narticles = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv')\nprint(articles.head())\nprint(\"--\")\ncustomers = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/customers.csv')\nprint(customers.head())\nprint(\"--\")\ntransactions = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")\nprint(transactions.head())\nprint(\"--\")","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:11:18.128752Z","iopub.execute_input":"2023-04-16T05:11:18.129400Z","iopub.status.idle":"2023-04-16T05:11:53.818380Z","shell.execute_reply.started":"2023-04-16T05:11:18.129359Z","shell.execute_reply":"2023-04-16T05:11:53.817235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pre-Processing","metadata":{}},{"cell_type":"code","source":"# ----- empty value stats -------------\nprint(\"Missing values: \")\nprint(customers.isnull().sum())\nprint(\"--\\n\")\n\nprint(\"FN Newsletter vals: \", customers['FN'].unique())\nprint(\"Active communication vals: \",customers['Active'].unique())\nprint(\"Club member status vals: \", customers['club_member_status'].unique())\nprint(\"Fashion News frequency vals: \", customers['fashion_news_frequency'].unique())\nprint(\"--\\n\")\n\n# ---- data cleaning -------------\n\ncustomers['FN'] = customers['FN'].fillna(0)\ncustomers['Active'] = customers['Active'].fillna(0)\n\n# replace club_member_status missing values with 'LEFT CLUB' --> no members with LEFT CLUB status in data\ncustomers['club_member_status'] = customers['club_member_status'].fillna('LEFT CLUB')\ncustomers['fashion_news_frequency'] = customers['fashion_news_frequency'].fillna('None')\ncustomers['fashion_news_frequency'] = customers['fashion_news_frequency'].replace('NONE', 'None')\ncustomers['age'] = customers['age'].fillna(customers['age'].mean())\ncustomers['age'] = customers['age'].astype(int)\narticles['detail_desc'] = articles['detail_desc'].fillna('None')\n\n\nprint(\"Customers' Missing values: \")\nprint(customers.isnull().sum())\nprint(\"--\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:11:53.820634Z","iopub.execute_input":"2023-04-16T05:11:53.821012Z","iopub.status.idle":"2023-04-16T05:11:54.687525Z","shell.execute_reply.started":"2023-04-16T05:11:53.820972Z","shell.execute_reply":"2023-04-16T05:11:54.686518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---- memory optimizations -------------\n\n# reference: https://www.kaggle.com/arjanso/reducing-dataframe-memory-size-by-65\n\n# iterate through all the columns of a dataframe and reduce the int and float data types to the smallest possible size, ex. customer_id should not be reduced from int64 to a samller value as it would have collisions\nimport numpy as np\nimport pandas as pd\n\ndef reduce_mem_usage(df):\n \"\"\"Iterate over all the columns of a DataFrame and modify the data type\n to reduce memory usage, handling ordered Categoricals\"\"\"\n \n # check the memory usage of the DataFrame\n start_mem = df.memory_usage().sum() / 1024**2\n print(\"Memory usage of dataframe is {:.2f} MB\".format(start_mem))\n \n for col in df.columns:\n col_type = df[col].dtype\n \n if col_type == 'category':\n if df[col].cat.ordered:\n # Convert ordered Categorical to an integer\n df[col] = df[col].cat.codes.astype('int16')\n else:\n # Convert unordered Categorical to a string\n df[col] = df[col].astype('str')\n \n elif col_type != object:\n c_min = df[col].min()\n c_max = df[col].max()\n if str(col_type)[:3] == 'int':\n if c_min >= np.iinfo(np.int8).min and c_max <= np.iinfo(np.int8).max:\n df[col] = df[col].astype(np.int8)\n elif c_min >= np.iinfo(np.int16).min and c_max <= np.iinfo(np.int16).max:\n df[col] = df[col].astype(np.int16)\n elif c_min >= np.iinfo(np.int32).min and c_max <= np.iinfo(np.int32).max:\n df[col] = df[col].astype(np.int32)\n elif c_min >= np.iinfo(np.int64).min and c_max <= np.iinfo(np.int64).max:\n df[col] = df[col].astype(np.int64) \n else:\n if c_min >= np.finfo(np.float16).min and c_max <= np.finfo(np.float16).max:\n df[col] = df[col].astype(np.float16)\n elif c_min >= np.finfo(np.float32).min and c_max <= np.finfo(np.float32).max:\n df[col] = df[col].astype(np.float32)\n else:\n df[col] = df[col].astype(np.float64)\n \n # check the memory usage after optimization\n end_mem = df.memory_usage().sum() / 1024**2\n print(\"Memory usage after optimization is: {:.2f} MB\".format(end_mem))\n\n # calculate the percentage of the memory usage reduction\n mem_reduction = 100 * (start_mem - end_mem) / start_mem\n print(\"Memory usage decreased by {:.1f}%\".format(mem_reduction))\n \n return df\n\n ","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:11:54.691529Z","iopub.execute_input":"2023-04-16T05:11:54.691829Z","iopub.status.idle":"2023-04-16T05:11:54.707193Z","shell.execute_reply.started":"2023-04-16T05:11:54.691799Z","shell.execute_reply":"2023-04-16T05:11:54.706205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Articles Info: \")\nprint(articles.info())\nprint(\"Customer Info: \")\nprint(customers.info())\nprint(\"Transactions Info: \")\nprint(transactions.info())","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:11:54.710070Z","iopub.execute_input":"2023-04-16T05:11:54.710659Z","iopub.status.idle":"2023-04-16T05:11:55.016909Z","shell.execute_reply.started":"2023-04-16T05:11:54.710621Z","shell.execute_reply":"2023-04-16T05:11:55.015697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ---- memory optimizations -------------\n\n# uses 8 bytes instead of given 64 byte string, reduces mem by 8x, \n# !!!! have to convert back before merging w/ sample_submissions.csv\n# convert transactions['customer_id'] to 8 bytes int\n# transactions['customer_id'] = transactions['customer_id'].astype('int64')\ntransactions['customer_id'] = transactions['customer_id'].apply(lambda x: int(x[-16:], 16)).astype('int64')\ncustomers['customer_id'] = customers['customer_id'].apply(lambda x: int(x[-16:], 16)).astype('int64')\n\narticles = reduce_mem_usage(articles)\ncustomers = reduce_mem_usage(customers)\ntransactions = reduce_mem_usage(transactions)\n\n# articles['article_id'] = articles['article_id'].astype('int32')\n# transactions['article_id'] = transactions['article_id'].astype('int32') \n# # !!!! ADD LEADING ZERO BACK BEFORE SUBMISSION OF PREDICTIONS TO KAGGLE: \n# # Ex.: transactions['article_id'] = '0' + transactions.article_id.astype('str')\n\nprint(\"Articles Info: \")\nprint(articles.info())\nprint(\"Customer Info: \")\nprint(customers.info())\nprint(\"Transactions Info: \")\nprint(transactions.info())","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:11:55.019086Z","iopub.execute_input":"2023-04-16T05:11:55.020272Z","iopub.status.idle":"2023-04-16T05:12:19.888080Z","shell.execute_reply.started":"2023-04-16T05:11:55.020209Z","shell.execute_reply":"2023-04-16T05:12:19.886918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory-Data-Analysis","metadata":{}},{"cell_type":"markdown","source":"### EDA::Articles","metadata":{}},{"cell_type":"markdown","source":"Article data:\n\n`article_id` : Unique id for every article of clothing
\n\nObserving the structure of the column info, this indentation structure of features satisfied article identification:
\n\n- `` and `` :: (clothing categories)\n\t- `` and `` :: (clothing categories' sub-groups) -- same as index group if no subgroups for a category\n\t\t- `` and `` :: (clothing collections)\n\t\t\t- `` and `` :: (garment groups)\n\t\t\t\t- `` and `` :: (product groups)\n\t\t\t\t\t- `` and `` :: (product types)\n\t\t\t\t\t\t- `` and `` :: (product names)\n\nOther data:
\n`colour_*`: colour info of each article \n`perceived_colour_*`: colour info of each article
\n`department_*`: department info
\n`detail_desc`: article description
\n(we're ignoring `graphical_*` features since we are not going to use the image data)\n","metadata":{}},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:19.889601Z","iopub.execute_input":"2023-04-16T05:12:19.890163Z","iopub.status.idle":"2023-04-16T05:12:19.920213Z","shell.execute_reply.started":"2023-04-16T05:12:19.890123Z","shell.execute_reply":"2023-04-16T05:12:19.919040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing the most popular clothing categories (indices)\n\n# Convert index_name to ordered categorical for ordered histplot\nordered_index_names = articles['index_name'].value_counts().index\narticles['index_name'] = pd.Categorical(articles['index_name'], categories=ordered_index_names, ordered=True)\n\n# Plot histogram\nf, ax = plt.subplots(figsize=(10, 6))\nsns.histplot(data=articles, y='index_name')\nax.set_xlabel('count of articles')\nax.set_ylabel('index_name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:19.922194Z","iopub.execute_input":"2023-04-16T05:12:19.922605Z","iopub.status.idle":"2023-04-16T05:12:20.283962Z","shell.execute_reply.started":"2023-04-16T05:12:19.922567Z","shell.execute_reply":"2023-04-16T05:12:20.282954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Ladieswear category and Children (aggregated) category have the most articles. The Sport category has the least articles.","metadata":{}},{"cell_type":"code","source":"# Observing the most popular clothing collections (sections)\n\nordered_section_names = articles['section_name'].value_counts().index\narticles['section_name'] = pd.Categorical(articles['section_name'], categories=ordered_section_names, ordered=True)\n\nf, ax = plt.subplots(figsize=(14, 14))\nsns.histplot(data=articles, y='section_name', bins=len(ordered_section_names))\nax.set_xlabel('count of articles by clothing collection')\nax.set_ylabel('section_name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:20.285406Z","iopub.execute_input":"2023-04-16T05:12:20.286310Z","iopub.status.idle":"2023-04-16T05:12:21.310991Z","shell.execute_reply.started":"2023-04-16T05:12:20.286239Z","shell.execute_reply":"2023-04-16T05:12:21.309909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Women's Everyday Collection, followed by the miscellaneous section Divided Collection, and Baby Essentials & Complements.\nLadies Other has the least number of articles.","metadata":{}},{"cell_type":"code","source":"# Observing garments grouped by their clothing category (index_group)\n\nordered_garment_group_names = articles['garment_group_name'].value_counts().index\nordered_index_group_names = articles['index_group_name'].value_counts().index\narticles['garment_group_name'] = pd.Categorical(articles['garment_group_name'], categories=ordered_garment_group_names, ordered=True)\narticles['index_group_name'] = pd.Categorical(articles['index_group_name'], categories=ordered_index_group_names, ordered=True)\n\nf, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=articles, y='garment_group_name', hue='index_group_name', multiple=\"stack\")\nax.set_xlabel('count of articles by garment group')\nax.set_ylabel('garment_group_name')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:21.312294Z","iopub.execute_input":"2023-04-16T05:12:21.312600Z","iopub.status.idle":"2023-04-16T05:12:21.972940Z","shell.execute_reply.started":"2023-04-16T05:12:21.312570Z","shell.execute_reply":"2023-04-16T05:12:21.971895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Jersey Fancy and Accessories are the most popular garment groups; a large part of the Ladieswear and Children categories contribute to the garment group counts.","metadata":{}},{"cell_type":"code","source":"# Observing number of articles per clothing category\n\narticles.groupby(['index_group_name']).count()['article_id']","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:21.977426Z","iopub.execute_input":"2023-04-16T05:12:21.977724Z","iopub.status.idle":"2023-04-16T05:12:22.050774Z","shell.execute_reply.started":"2023-04-16T05:12:21.977689Z","shell.execute_reply":"2023-04-16T05:12:22.049609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since some clothing categories (index_group_name) have sub-categories (index_name):\n# Observing number of articles per sub-category\n\ngrouped_counts = articles.groupby(['index_group_name', 'index_name']).count()['article_id']\ngrouped_counts = grouped_counts[grouped_counts != 0]\ngrouped_counts","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:22.052635Z","iopub.execute_input":"2023-04-16T05:12:22.053081Z","iopub.status.idle":"2023-04-16T05:12:22.131016Z","shell.execute_reply.started":"2023-04-16T05:12:22.053043Z","shell.execute_reply":"2023-04-16T05:12:22.129201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The clothing sub-catgeory of Ladieswear in the Ladieswear category has the most articles.
\nThe clothing sub-catgeory of Children Sizes 92-140 in the Baby/Children category has the most articles in the category.","metadata":{}},{"cell_type":"code","source":"# Observing number of articles by product group\ngrouped_counts = articles.groupby(['garment_group_name', 'product_group_name']).count()['article_id']\ngrouped_counts = grouped_counts[grouped_counts != 0]\ngrouped_counts","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:22.132565Z","iopub.execute_input":"2023-04-16T05:12:22.135528Z","iopub.status.idle":"2023-04-16T05:12:22.215967Z","shell.execute_reply.started":"2023-04-16T05:12:22.135487Z","shell.execute_reply":"2023-04-16T05:12:22.214816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing number of articles by product groups\n\ngrouped_counts = articles.groupby(['product_group_name', 'product_type_name']).count()['article_id']\ngrouped_counts = grouped_counts[grouped_counts != 0]\ngrouped_counts","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:22.217870Z","iopub.execute_input":"2023-04-16T05:12:22.218280Z","iopub.status.idle":"2023-04-16T05:12:22.296733Z","shell.execute_reply.started":"2023-04-16T05:12:22.218220Z","shell.execute_reply":"2023-04-16T05:12:22.295601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing the most popular colours for articles\n\nordered_colour_names = articles['colour_group_name'].value_counts().index\narticles['colour_group_name'] = pd.Categorical(articles['colour_group_name'], categories=ordered_colour_names, ordered=True)\n\nf, ax = plt.subplots(figsize=(15, 10))\nsns.countplot(y='colour_group_name', data=articles, order=ordered_colour_names)\nax.set_xlabel('count of articles')\nax.set_ylabel('colour_group_name')\nax.set_title('Count of articles by Colour Group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:22.298541Z","iopub.execute_input":"2023-04-16T05:12:22.298955Z","iopub.status.idle":"2023-04-16T05:12:23.079842Z","shell.execute_reply.started":"2023-04-16T05:12:22.298917Z","shell.execute_reply":"2023-04-16T05:12:23.078674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Black, dark blue and white are the most popular colours overall.","metadata":{}},{"cell_type":"code","source":"# Observing the most popular graphics for articles\n\ncount_by_graphical_appearance = articles['graphical_appearance_name'].value_counts().sort_values(ascending=True)\n\nfig, ax = plt.subplots(figsize=(10, 8))\nax.barh(count_by_graphical_appearance.index, count_by_graphical_appearance.values)\nax.set_title('Count of Articles by Graphical Appearance')\nax.set_xlabel('count of articles')\nax.set_ylabel('graphical_appearance_name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:23.081494Z","iopub.execute_input":"2023-04-16T05:12:23.082124Z","iopub.status.idle":"2023-04-16T05:12:23.556264Z","shell.execute_reply.started":"2023-04-16T05:12:23.082086Z","shell.execute_reply":"2023-04-16T05:12:23.555271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A Solid pattern on articles is most popular.","metadata":{}},{"cell_type":"markdown","source":"### EDA::Customers","metadata":{}},{"cell_type":"markdown","source":"Customer data:\n\n`customer_id` : Unique id for every customer
\n`FN` (Does the customer receive fashion news): 1 or 0
\n`Active` (Is the customer active for communication): 1 or 0
\n`club_member_status` (Customer's club status): 'ACTIVE' or 'PRE-CREATE' or 'LEFT CLUB'
\n`fashion_news_frequency` (How often H&M may send news to customer): 'Regularly' or 'Monthly' or 'None'
\n`age` : Customer's age
\n`postal_code` : Customer's postal code
","metadata":{}},{"cell_type":"code","source":"customers.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:23.557782Z","iopub.execute_input":"2023-04-16T05:12:23.558144Z","iopub.status.idle":"2023-04-16T05:12:23.572541Z","shell.execute_reply.started":"2023-04-16T05:12:23.558108Z","shell.execute_reply":"2023-04-16T05:12:23.571287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing postal code counts\n\ntop_5_postal_codes = customers['postal_code'].value_counts().head(5)\nprint(top_5_postal_codes)","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:23.574186Z","iopub.execute_input":"2023-04-16T05:12:23.575376Z","iopub.status.idle":"2023-04-16T05:12:24.122281Z","shell.execute_reply.started":"2023-04-16T05:12:23.575335Z","shell.execute_reply":"2023-04-16T05:12:24.121138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clearly, the most common postal code is either some default postal code or a centralized delivery location.","metadata":{}},{"cell_type":"code","source":"# Observing the customer age distribution\n\nf, ax = plt.subplots(figsize=(15,5))\nsns.histplot(data=customers, x='age')\nax.set_ylabel('number of customers')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:24.123898Z","iopub.execute_input":"2023-04-16T05:12:24.124727Z","iopub.status.idle":"2023-04-16T05:12:24.916339Z","shell.execute_reply.started":"2023-04-16T05:12:24.124683Z","shell.execute_reply":"2023-04-16T05:12:24.915268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_5_ages = customers['age'].value_counts().head(5)\nprint(top_5_ages)","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:24.917774Z","iopub.execute_input":"2023-04-16T05:12:24.918833Z","iopub.status.idle":"2023-04-16T05:12:24.934150Z","shell.execute_reply.started":"2023-04-16T05:12:24.918793Z","shell.execute_reply":"2023-04-16T05:12:24.932968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clearly, the age range of 20-25 has the most customers.","metadata":{}},{"cell_type":"code","source":"# Observing the club member status of customers\n\n# Group the customers by club member status and count the number of customers in each group\nclub_member_counts = customers.groupby('club_member_status')['customer_id'].count()\n\n# Pie chart\nplt.pie(club_member_counts.values, labels=club_member_counts.index, autopct='%1.1f%%')\nplt.title('Club Member Statuses')\nplt.axis('equal')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:24.935949Z","iopub.execute_input":"2023-04-16T05:12:24.936431Z","iopub.status.idle":"2023-04-16T05:12:25.170470Z","shell.execute_reply.started":"2023-04-16T05:12:24.936393Z","shell.execute_reply":"2023-04-16T05:12:25.169019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"An overwhelming majority of customers currently have an active club status.","metadata":{}},{"cell_type":"code","source":"# Observing the FN subscription of customers\n\nnews_frequency_counts = customers.groupby('fashion_news_frequency')['customer_id'].count()\n\n# create a pie chart\nplt.pie(news_frequency_counts, labels=news_frequency_counts.index, autopct='%1.1f%%')\nplt.title('Fashion News Newsletter Frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:25.172486Z","iopub.execute_input":"2023-04-16T05:12:25.172880Z","iopub.status.idle":"2023-04-16T05:12:25.413721Z","shell.execute_reply.started":"2023-04-16T05:12:25.172841Z","shell.execute_reply":"2023-04-16T05:12:25.412197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A majority of customers don't subsribe to the fashion newsletter.","metadata":{}},{"cell_type":"markdown","source":"### EDA::Transactions","metadata":{}},{"cell_type":"markdown","source":"Transaction data:\n\n`t_dat`: date the transaction occured in yyyy-mm-dd format
\n`customer_id`: in customers df
\n`article_id` in articles df
\n`price`: geneneralized price (not a currency or unit)
\n`sales_channel_id`: 1 = in-store or 2 = online
","metadata":{}},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:25.415807Z","iopub.execute_input":"2023-04-16T05:12:25.416323Z","iopub.status.idle":"2023-04-16T05:12:25.438169Z","shell.execute_reply.started":"2023-04-16T05:12:25.416270Z","shell.execute_reply":"2023-04-16T05:12:25.436649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15,1))\nax.set_title('Price distribution of all articles')\nsns.boxplot(x='price', data=transactions)\nplt.show()\npd.set_option('display.float_format', '{:.4f}'.format)\ntransactions.describe()['price']","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:25.439729Z","iopub.execute_input":"2023-04-16T05:12:25.440365Z","iopub.status.idle":"2023-04-16T05:12:36.743600Z","shell.execute_reply.started":"2023-04-16T05:12:25.440322Z","shell.execute_reply":"2023-04-16T05:12:36.742404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The prices seem to vary a lot across all articles, so the above plot doesn't give us useful information.
\nThe total transaction count is ~31 million, so we'll use a 100,000 sample from the transaction data as needed.","metadata":{}},{"cell_type":"code","source":"# merging transactions and artciles df on aritcle_id\n\narticles_product_columns = articles[['article_id', 'index_name', 'product_group_name', 'product_type_name', 'prod_name']]\n# merged_table_ta --> merged_table_transactions_articles\nmerged_table_ta = transactions[['customer_id', 'article_id', 'price', 'sales_channel_id','t_dat']].merge(articles_product_columns, on='article_id', how='left')\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:36.745221Z","iopub.execute_input":"2023-04-16T05:12:36.745663Z","iopub.status.idle":"2023-04-16T05:12:44.834505Z","shell.execute_reply.started":"2023-04-16T05:12:36.745620Z","shell.execute_reply":"2023-04-16T05:12:44.833434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing the mean price of each sub-clothing catgeory (index_name)\n\n# Group the data by index_name and calculate the mean price for each group\nprice_by_index_name = merged_table_ta.groupby('index_name')['price'].mean()\nprice_by_index_name = price_by_index_name.sort_values(ascending=True)\n\n# Plot\nfig, ax = plt.subplots(figsize=(10, 5))\nax.barh(price_by_index_name.index, price_by_index_name.values)\nax.set_title('Average prices by index_name (all clothing categories)')\nax.set_xlabel('mean price')\nax.set_ylabel('index_name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:44.835992Z","iopub.execute_input":"2023-04-16T05:12:44.836373Z","iopub.status.idle":"2023-04-16T05:12:45.461374Z","shell.execute_reply.started":"2023-04-16T05:12:44.836335Z","shell.execute_reply":"2023-04-16T05:12:45.460224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Ladieswear sub-category in the Ladieswear category has the largest mean price.","metadata":{}},{"cell_type":"code","source":"# Observing the mean price of each product group\n\n# Group the data by product_group_name and calculate the mean price for each group\nprice_by_product_group = merged_table_ta[['product_group_name', 'price']].groupby('product_group_name').mean()\nprice_by_product_group = price_by_product_group.sort_values(by='price', ascending=True)\n\n# Plot\nfig, ax = plt.subplots(figsize=(10, 5))\nax.barh(price_by_product_group.index, price_by_product_group['price'])\nax.set_title('Average prices by product group')\nax.set_xlabel('mean price')\nax.set_ylabel('product_group_name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:45.462986Z","iopub.execute_input":"2023-04-16T05:12:45.463479Z","iopub.status.idle":"2023-04-16T05:12:53.541248Z","shell.execute_reply.started":"2023-04-16T05:12:45.463436Z","shell.execute_reply":"2023-04-16T05:12:53.540212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing price distributions by product group \n\n# plot all boxplots\nf, ax = plt.subplots(figsize=(25,18))\nax = sns.boxplot(data=merged_table_ta, x='price', y='product_group_name')\nax.set_xlabel('price by product group', fontsize=18)\nax.set_ylabel('product_group_name', fontsize=18)\nax.xaxis.set_tick_params(labelsize=16)\nax.yaxis.set_tick_params(labelsize=16)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:12:53.542724Z","iopub.execute_input":"2023-04-16T05:12:53.543788Z","iopub.status.idle":"2023-04-16T05:13:11.320046Z","shell.execute_reply.started":"2023-04-16T05:12:53.543733Z","shell.execute_reply":"2023-04-16T05:13:11.318911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The prices for product groups Garment upper/lower/full and Shoes have a large variance in prices. The opposite is true for Cosmetic, Stationery and Fun product groups.
This is reasonable since the price can vary between different clothing collections of the same product group (Ex. premium full garmnet collection vs an on-sale garment collection).","metadata":{}},{"cell_type":"code","source":"# Observing the distribution of price frequency in samples\n\n# Sample 100,000 observations\nmerged_sample = merged_table_ta.sample(n=100000)\n\n# using kdeplots to plot probability dist. of price values\nfig, ax = plt.subplots(1, 1, figsize=(14, 5))\nsns.kdeplot(np.log(merged_sample.loc[merged_sample[\"sales_channel_id\"]==1].price.value_counts()))\nsns.kdeplot(np.log(merged_sample.loc[merged_sample[\"sales_channel_id\"]==2].price.value_counts()))\nax.legend(labels=['In-store: Sales channel 1', 'Online: Sales channel 2'])\nplt.title(\"Logarithmic distribution of price frequency in transactions, grouped by sales channel (100k sample)\")\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:11.325747Z","iopub.execute_input":"2023-04-16T05:13:11.326151Z","iopub.status.idle":"2023-04-16T05:13:13.205278Z","shell.execute_reply.started":"2023-04-16T05:13:11.326112Z","shell.execute_reply":"2023-04-16T05:13:13.204212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a slightly larger tendency for customers to purchase more expensive items online.","metadata":{}},{"cell_type":"code","source":"# Observing the top 10 customers by number of transactions\n\ntransactions['customer_id'].value_counts().head(10)","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:13.206801Z","iopub.execute_input":"2023-04-16T05:13:13.207129Z","iopub.status.idle":"2023-04-16T05:13:14.780601Z","shell.execute_reply.started":"2023-04-16T05:13:13.207099Z","shell.execute_reply":"2023-04-16T05:13:14.779627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observing the number of transaction per day by sales channel\n\n# Sample 100,000 observations\nmerged_table_ta_sample = merged_table_ta.sample(n=100000, random_state=69)\n\n# Convert t_dat column to datetime format\nmerged_table_ta_sample['t_dat'] = pd.to_datetime(merged_table_ta_sample['t_dat'])\n\n# Group the data by sales channel and date, and count the number of transactions for each group\ntransactions_by_day = merged_table_ta_sample.groupby(['sales_channel_id', pd.Grouper(key='t_dat', freq='D')])['article_id'].count()\n\n# Create a line plot\nfig, ax = plt.subplots(figsize=(10, 6))\nfor channel in transactions_by_day.index.levels[0]:\n if channel == 1:\n ax.plot(transactions_by_day[channel], label=f'In-Store: Sales channel 1')\n else:\n ax.plot(transactions_by_day[channel], label=f'Online: Sales channel 2')\nax.legend()\nax.set_title('Number of Transactions per Day by Sales Channel')\nax.set_xlabel('Date')\nax.set_ylabel('Number of Transactions')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:14.782236Z","iopub.execute_input":"2023-04-16T05:13:14.782640Z","iopub.status.idle":"2023-04-16T05:13:16.646134Z","shell.execute_reply.started":"2023-04-16T05:13:14.782602Z","shell.execute_reply":"2023-04-16T05:13:16.645192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is worth noting here that for April 2020, the number of in-store transactions is virtually nonexistent, while there is a sharp spike in online transactions for the same month; this is highly likely due to in-person stores possibly closing in April due to Covid-19.","metadata":{}},{"cell_type":"code","source":"# Purchase stats\n\nnum_unique_customers = len(transactions['customer_id'].unique())\nnum_unique_articles = len(transactions['article_id'].unique())\ntotal_transactions = len(transactions)\n\nprint(\"Total H&M customers:\", len(customers))\nprint(\"Total H&M articles:\", len(articles))\nprint(\"Number of unique customers that purchased at least 1 article:\", num_unique_customers)\nprint(\"Number of unique articles purchased:\", num_unique_articles)\nprint(\"Number of customers that didn't make any transactions:\", len(customers) - num_unique_customers)\nprint(\"Number of unique articles that weren't purchased:\", len(articles) - num_unique_articles)\nprint(\"Total number of transactions:\", total_transactions)","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:16.647792Z","iopub.execute_input":"2023-04-16T05:13:16.648509Z","iopub.status.idle":"2023-04-16T05:13:17.231372Z","shell.execute_reply.started":"2023-04-16T05:13:16.648467Z","shell.execute_reply":"2023-04-16T05:13:17.230145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get % of customers that made at least 1 transaction in the last 3 months\n\ntransactions['t_dat'] = pd.to_datetime(transactions['t_dat'])\n\n# get the last transaction date for each customer\nlast_transaction_date = transactions.groupby('customer_id')['t_dat'].max()\n\nthree_months_ago = last_transaction_date.max() - pd.Timedelta(days=90)\n\n# get the customers who made at least 1 transaction in the last 3 months\nactive_customers = last_transaction_date[last_transaction_date >= three_months_ago].index\npercent_active_customers = len(active_customers) / len(last_transaction_date) * 100\n\nprint(f\"Percentage of customers that made at least 1 transaction in the last 3 months: {percent_active_customers:.2f}%\")\n\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:17.232792Z","iopub.execute_input":"2023-04-16T05:13:17.233713Z","iopub.status.idle":"2023-04-16T05:13:23.620983Z","shell.execute_reply.started":"2023-04-16T05:13:17.233682Z","shell.execute_reply":"2023-04-16T05:13:23.619777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Validating that there are no duplicate customer or article IDs\n\nduplicate_customers = customers[customers.duplicated(subset=['customer_id'], keep=False)]\nduplicate_customers = duplicate_customers.sort_values(by=['customer_id'])\nprint(\"Number of non-unique customer IDs:\", len(duplicate_customers))\n\nduplicate_articles = articles[articles.duplicated(subset=['article_id'], keep=False)]\nduplicate_articles = duplicate_articles.sort_values(by=['article_id'])\nprint(\"Number of non-unique article IDs:\", len(duplicate_customers))\n","metadata":{"execution":{"iopub.status.busy":"2023-04-16T05:13:23.622567Z","iopub.execute_input":"2023-04-16T05:13:23.623141Z","iopub.status.idle":"2023-04-16T05:13:23.853419Z","shell.execute_reply.started":"2023-04-16T05:13:23.623102Z","shell.execute_reply":"2023-04-16T05:13:23.852186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 105 weeks of transaction data, with 31 million transactions. This is a lot of data to process - instead, we will use the the transaction data of the top 200 customers, based on the number of items they have bought in total. This will also help reduce the sparsity of any user-item interaction matrices we use.","metadata":{}},{"cell_type":"markdown","source":"We will use an 80/20 time-based split for testing our data subset.","metadata":{}},{"cell_type":"markdown","source":"## Models\n\nNotes from the problem description:\n\n\"Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.\"\n => `Predictions on 7 day period after the latest date found in the training data. The test week is the same for all customers, not one individual week per customer based on their latest training sample.`\n\n- You will be making purchase predictions for all customer_id values provided, regardless of whether these customers made purchases in the training data.\n- Customer that did not make any purchase during test period are excluded from the scoring.\n- There is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.\n\n-------------------------------------------------------------------------------------------------------------------------\nAs per the given evaluation metric `MAP@12`, since the rank = 12, we have to recommend the top 12 products (sorted by recommendation score), each customer would likely purchase.\n\nWe will be using 1 baseline popularity model and 5 different models that utilize `collaborative filtering` to make recommendations, and evaluate each model's performance using the MAP@12 metric. \n\nThe 5 different models are: ALS, SGD, LightGBM, CatBoost and a GNN. \n\nSince the dataset has no `explicit feedback` from customers (ex. ratings, reviews), we can consider the quantity of each item a customer purchased as an indicator of purchase preference, or a binary indicator of whether an item was purchased - either of these strategies will act as our `implicit feedback`. \n\nThe primary goal of this recommender system, and product-based recommender systems in general, is to `recommend articles the user is likely to be interested in`. Quantity of an article may not always be indicator of interest in an item, especially if the item bought usually requires multiple amounts of the item purchased, or if the item purchased is a gift. However, a normalized purchase quantity will weight the user-item interaction matrix, and thus could potentially provide more information to the recommender system. \n\nAdditionally, one could also incorporate the price of an item and purchase recency as part of the implicit feedback, as buying an expensive item could potentially be a stronger indicator of item preferences vs. buying large quantities of cheaper items. ","metadata":{}}]}