{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Topic: H&M Personalized Fashion Recommendations\n\nFor this challenge you are given the purchase history of customers across time, along with supporting metadata. Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.","metadata":{}},{"cell_type":"code","source":"## Imports\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\nimport cudf\nfrom mlxtend.preprocessing import TransactionEncoder\nfrom mlxtend.frequent_patterns import apriori, association_rules, fpgrowth","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:56:52.502722Z","iopub.execute_input":"2022-11-27T05:56:52.503437Z","iopub.status.idle":"2022-11-27T05:56:55.749008Z","shell.execute_reply.started":"2022-11-27T05:56:52.503397Z","shell.execute_reply":"2022-11-27T05:56:55.747971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read data\n\n- As the dataset is very large ~13 gigs we are using cudf on a gpu P100 provided by kaggle, cudf uses gpu clusters to effiiciently load chunks of dataframe into memory.\n- We are also converting the article_id column to int32 type by defining appropriate types to variables reduces the space occupied by the dataframe.\n- Only filtering by 2020 year as we are only interested in the most recent year. Also, removing unnecesaary column like sales_channel_id.","metadata":{}},{"cell_type":"code","source":"## Read data\ntranscation_data = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\ntranscation_data['customer_id'] = transcation_data['customer_id'].str[-16:].str.hex_to_int().astype('int64')\ntranscation_data['article_id'] = transcation_data.article_id.astype('int32')\ntranscation_data.t_dat = cudf.to_datetime(transcation_data.t_dat)\ntranscation_data = transcation_data[['t_dat','customer_id','article_id', 'price']]\nprint( transcation_data.shape )\ntranscation_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:56:55.751140Z","iopub.execute_input":"2022-11-27T05:56:55.751522Z","iopub.status.idle":"2022-11-27T05:57:34.147288Z","shell.execute_reply.started":"2022-11-27T05:56:55.751486Z","shell.execute_reply":"2022-11-27T05:57:34.146100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Preprocess \n \n - Converting t_dat column to datetime column so as to extract year, month and day into separate columns","metadata":{}},{"cell_type":"code","source":"## Data preprocess\ntranscation_data.loc[:,'t_dat'] = cudf.to_datetime(transcation_data['t_dat'])\ntranscation_data.loc[:, 'year'] = transcation_data['t_dat'].dt.year\ntranscation_data.loc[:, 'month'] = transcation_data['t_dat'].dt.month\ntranscation_data.loc[:, 'day'] = transcation_data['t_dat'].dt.day\ntranscation_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:34.149565Z","iopub.execute_input":"2022-11-27T05:57:34.150347Z","iopub.status.idle":"2022-11-27T05:57:34.197849Z","shell.execute_reply.started":"2022-11-27T05:57:34.150304Z","shell.execute_reply":"2022-11-27T05:57:34.196461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Converting into parquet files as parquet files are faster to load into memory","metadata":{}},{"cell_type":"code","source":"transcation_data.to_parquet('transcation_data_hm.pqt',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:34.201365Z","iopub.execute_input":"2022-11-27T05:57:34.201804Z","iopub.status.idle":"2022-11-27T05:57:35.026854Z","shell.execute_reply.started":"2022-11-27T05:57:34.201762Z","shell.execute_reply":"2022-11-27T05:57:35.025409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transcation_data = pd.read_parquet(\"./transcation_data_hm.pqt\")","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:35.032578Z","iopub.execute_input":"2022-11-27T05:57:35.032892Z","iopub.status.idle":"2022-11-27T05:57:36.811166Z","shell.execute_reply.started":"2022-11-27T05:57:35.032861Z","shell.execute_reply":"2022-11-27T05:57:36.810121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### How many years data do we have?\n\nObservation: Seems like we made lot more transcation in 2019 than the previous 2 years","metadata":{}},{"cell_type":"code","source":"sns.countplot(x=transcation_data['year']);","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:36.812956Z","iopub.execute_input":"2022-11-27T05:57:36.813337Z","iopub.status.idle":"2022-11-27T05:57:39.485130Z","shell.execute_reply.started":"2022-11-27T05:57:36.813297Z","shell.execute_reply":"2022-11-27T05:57:39.484058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Do we have data for all the 12 months?","metadata":{}},{"cell_type":"code","source":"plt.xlabel(\"count of transcations\")\nplt.ylabel(\"months\")\ntranscation_data.month.value_counts().plot(kind=\"barh\");","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:39.486667Z","iopub.execute_input":"2022-11-27T05:57:39.487031Z","iopub.status.idle":"2022-11-27T05:57:39.911941Z","shell.execute_reply.started":"2022-11-27T05:57:39.486994Z","shell.execute_reply":"2022-11-27T05:57:39.910882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Is there a particular day which has highest transactions?\n\nObservation: \n\nUsually more transcation happening during the month end except for the very last day of the month. Probably as people usually recieve their monthly salaries during the month end.","metadata":{}},{"cell_type":"code","source":"plt.ylabel(\"count of transcations\")\nplt.xlabel(\"days\")\nsns.countplot(x=\"day\", data=transcation_data);","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:39.913463Z","iopub.execute_input":"2022-11-27T05:57:39.914027Z","iopub.status.idle":"2022-11-27T05:57:42.813351Z","shell.execute_reply.started":"2022-11-27T05:57:39.913989Z","shell.execute_reply":"2022-11-27T05:57:42.812321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Price fluctuation over the years for 2020","metadata":{}},{"cell_type":"markdown","source":"Obervations:\n\n- We see a gradual increase from the start of the year and then a decline in transaction during the mid of the year and then a very steep increase by the end of the year.\n- This might be due to the fact that there is thanksgiving and christmas holidays not to forget black friday sales during year end in whiich people buy the most.","metadata":{}},{"cell_type":"code","source":"transcations_2020 =  transcation_data[transcation_data['year'] == 2020]\nplt.xticks(rotation=90)\nplt.xlabel(\"2020 data\")\nsns.lineplot(x=\"t_dat\", y=\"price\", data=transcations_2020);","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:57:42.814816Z","iopub.execute_input":"2022-11-27T05:57:42.815124Z","iopub.status.idle":"2022-11-27T05:59:41.604286Z","shell.execute_reply.started":"2022-11-27T05:57:42.815097Z","shell.execute_reply":"2022-11-27T05:59:41.603076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets analyse the latest year -- 2020!!","metadata":{}},{"cell_type":"code","source":"transcation_2020 = transcation_data[transcation_data['year'] == 2020]","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:41.611281Z","iopub.execute_input":"2022-11-27T05:59:41.614296Z","iopub.status.idle":"2022-11-27T05:59:42.114448Z","shell.execute_reply.started":"2022-11-27T05:59:41.614244Z","shell.execute_reply":"2022-11-27T05:59:42.113341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which articles have the max price and the most cheap article?","metadata":{}},{"cell_type":"code","source":"max(transcation_2020.price), min(transcation_2020.price)","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:42.116145Z","iopub.execute_input":"2022-11-27T05:59:42.116540Z","iopub.status.idle":"2022-11-27T05:59:43.737413Z","shell.execute_reply.started":"2022-11-27T05:59:42.116498Z","shell.execute_reply":"2022-11-27T05:59:43.736269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Most frequently bought articles","metadata":{}},{"cell_type":"code","source":"transcations_article_ids_value_counts = transcation_2020.article_id.value_counts()\ntranscations_article_ids_top_20 = transcations_article_ids_value_counts[:20].index.to_list()\narticles_data = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\", index_col = \"article_id\")\ntop_20_articles = articles_data.filter(items=transcations_article_ids_top_20, axis=0)\ntop_20_articles","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:43.739366Z","iopub.execute_input":"2022-11-27T05:59:43.739829Z","iopub.status.idle":"2022-11-27T05:59:44.882243Z","shell.execute_reply.started":"2022-11-27T05:59:43.739784Z","shell.execute_reply":"2022-11-27T05:59:44.881297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Seems like trousers are the most bought item!!","metadata":{}},{"cell_type":"code","source":"plt.xticks(rotation=45)\nsns.countplot(x = top_20_articles.product_type_name);","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:44.883449Z","iopub.execute_input":"2022-11-27T05:59:44.884637Z","iopub.status.idle":"2022-11-27T05:59:45.111101Z","shell.execute_reply.started":"2022-11-27T05:59:44.884600Z","shell.execute_reply":"2022-11-27T05:59:45.110206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which group dominates in terms of sales?\n\nSeems like peeople buy much more lower body clothes than others, we can also see swimwear, socks and underwear in the top category.","metadata":{}},{"cell_type":"code","source":"names, values = top_20_articles.product_group_name.value_counts().index, top_20_articles.product_group_name.value_counts().values\nplt.pie(values, labels = names, autopct='%1.1f%%');","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:45.112345Z","iopub.execute_input":"2022-11-27T05:59:45.112699Z","iopub.status.idle":"2022-11-27T05:59:45.237096Z","shell.execute_reply.started":"2022-11-27T05:59:45.112664Z","shell.execute_reply":"2022-11-27T05:59:45.235884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### How much sales did we make selling top 20 items?","metadata":{}},{"cell_type":"code","source":"count_top_20_transcation_articles = transcations_article_ids_value_counts[transcations_article_ids_top_20[:10]].values\nprice_top_20_transcation_articles = transcation_2020[transcation_2020.article_id.isin(transcations_article_ids_top_20)]\nprice_top_20_transcation_articles = price_top_20_transcation_articles.groupby('article_id').agg({\"price\": \"sum\"}).sort_values('price',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:45.238690Z","iopub.execute_input":"2022-11-27T05:59:45.239013Z","iopub.status.idle":"2022-11-27T05:59:45.418570Z","shell.execute_reply.started":"2022-11-27T05:59:45.238978Z","shell.execute_reply":"2022-11-27T05:59:45.417556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_20_articles = price_top_20_transcation_articles.merge(articles_data, how = 'left', left_index = True, right_index = True)\nsns.barplot(y = \"prod_name\", x = \"price\", data = top_20_articles);","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:45.420148Z","iopub.execute_input":"2022-11-27T05:59:45.420562Z","iopub.status.idle":"2022-11-27T05:59:45.794353Z","shell.execute_reply.started":"2022-11-27T05:59:45.420511Z","shell.execute_reply":"2022-11-27T05:59:45.793318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets have a look at some of these popular items","metadata":{}},{"cell_type":"code","source":"path = \"../input/h-and-m-personalized-fashion-recommendations/images\"\nf, ax = plt.subplots(1, 5, figsize=(20,10))\ni = 0\nfor idx, data in top_20_articles[:5].iterrows():\n    file_name = \"0\" + str(idx) + \".jpg\"\n    dir_name = \"0\" + str(idx)[:2]\n    image = mpimg.imread(path + \"/\" + dir_name + \"/\" + file_name)\n    ax[i].imshow(image)\n    ax[i].set_title(f'price: {data.price:.2f}')\n    ax[i].set_xticks([], [])\n    ax[i].set_yticks([], [])\n    ax[i].grid(False)\n    ax[i].set_xlabel(data['prod_name'], fontsize=10)\n    i += 1\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:45.795691Z","iopub.execute_input":"2022-11-27T05:59:45.796046Z","iopub.status.idle":"2022-11-27T05:59:47.311252Z","shell.execute_reply.started":"2022-11-27T05:59:45.796010Z","shell.execute_reply":"2022-11-27T05:59:47.310335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Items purchased together by customers?","metadata":{}},{"cell_type":"code","source":"customers_data = cudf.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ncustomers_data['customer_id'] = customers_data['customer_id'].str[-16:].str.hex_to_int().astype('int64')\narticles_data = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ntranscations_data_2020 = transcation_data[(transcation_data['year'] == 2020) & (transcation_data['month'] > 6) & (transcation_data['day'] > 20)]\ntranscations_customers_all_articles = transcations_data_2020.groupby('customer_id')['article_id'].unique().reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:47.312322Z","iopub.execute_input":"2022-11-27T05:59:47.312727Z","iopub.status.idle":"2022-11-27T05:59:59.667358Z","shell.execute_reply.started":"2022-11-27T05:59:47.312681Z","shell.execute_reply":"2022-11-27T05:59:59.666323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Articles that customers purchased together in single transcation: \")\nfor _, data in transcations_customers_all_articles.head(5).iterrows():\n    print(\"Customer id\", data['customer_id'])\n    f, ax = plt.subplots(1, len(data['article_id']), figsize=(6, 6))\n    path = \"../input/h-and-m-personalized-fashion-recommendations/images\"\n    for i, article in enumerate(data['article_id']):\n        file_name = \"0\" + str(article) + \".jpg\"\n        dir_name = \"0\" + str(article)[:2]\n        image = mpimg.imread(path + \"/\" + dir_name + \"/\" + file_name)\n        ax[i].imshow(image)\n        ax[i].set_xticks([], [])\n        ax[i].set_yticks([], [])\n        ax[i].grid(False)\n        i += 1\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T05:59:59.668904Z","iopub.execute_input":"2022-11-27T05:59:59.669285Z","iopub.status.idle":"2022-11-27T06:00:04.035282Z","shell.execute_reply.started":"2022-11-27T05:59:59.669248Z","shell.execute_reply":"2022-11-27T06:00:04.034291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which are the most common itemsets purchased by customers?","metadata":{}},{"cell_type":"code","source":"#transcations_customers_all_articles['article_id'].value_counts()[:10]","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:04.036986Z","iopub.execute_input":"2022-11-27T06:00:04.037666Z","iopub.status.idle":"2022-11-27T06:00:04.042534Z","shell.execute_reply.started":"2022-11-27T06:00:04.037627Z","shell.execute_reply":"2022-11-27T06:00:04.041228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Which customers purchased these top items?","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Can we summarise the customers who purchase a lot?","metadata":{}},{"cell_type":"code","source":"top_customers_id = transcation_2020['customer_id'].value_counts().index.to_list()\ntop_10_customers = customers_data[customers_data['customer_id'].isin(top_customers_id[:10])]\ntop_10_customers","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:04.043856Z","iopub.execute_input":"2022-11-27T06:00:04.046246Z","iopub.status.idle":"2022-11-27T06:00:04.721399Z","shell.execute_reply.started":"2022-11-27T06:00:04.046215Z","shell.execute_reply":"2022-11-27T06:00:04.720351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Price of popular items over the year","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### All 3 years top performing departments images","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Aprori\n\nRef: https://towardsdatascience.com/the-frequently-bought-together-recommendation-system-b4ed076b24e5\n\nLets try and apply the aprori algorithm that tries to generate association rules by pruning itemsets based on their support values, we will use the mlextend library to impliment this.","metadata":{}},{"cell_type":"code","source":"## Generate all article ids purchased by every customer\ntranscations_data_2020 = transcation_data[(transcation_data['year'] == 2020) & (transcation_data['month'] > 6) & (transcation_data['day'] > 20)]\ntranscation_aprori = transcations_data_2020.groupby('customer_id')['article_id'].unique().reset_index()\ntranscation_aprori","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:04.722995Z","iopub.execute_input":"2022-11-27T06:00:04.723364Z","iopub.status.idle":"2022-11-27T06:00:14.928569Z","shell.execute_reply.started":"2022-11-27T06:00:04.723326Z","shell.execute_reply":"2022-11-27T06:00:14.927431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"te = TransactionEncoder()\nte.fit(transcation_aprori['article_id'])\norders_one_hot_encoded = te.transform(transcation_aprori['article_id'])\n\norders_one_hot_encoded = pd.DataFrame(orders_one_hot_encoded, columns =te.columns_)\norders_one_hot_encoded.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:14.930397Z","iopub.execute_input":"2022-11-27T06:00:14.930799Z","iopub.status.idle":"2022-11-27T06:00:17.196430Z","shell.execute_reply.started":"2022-11-27T06:00:14.930759Z","shell.execute_reply":"2022-11-27T06:00:17.195409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = apriori(orders_one_hot_encoded, min_support=0.003, max_len=2, use_colnames=True)\nresults.sort_values(by=['support'])","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:17.197990Z","iopub.execute_input":"2022-11-27T06:00:17.198351Z","iopub.status.idle":"2022-11-27T06:00:29.834432Z","shell.execute_reply.started":"2022-11-27T06:00:17.198315Z","shell.execute_reply":"2022-11-27T06:00:29.833338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Oops we dont see any association rules using aprori might be due to very less support values for each itemset!!","metadata":{}},{"cell_type":"code","source":"association_rules = association_rules(results, metric=\"lift\")\nassociation_rules.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:29.836037Z","iopub.execute_input":"2022-11-27T06:00:29.836406Z","iopub.status.idle":"2022-11-27T06:00:29.848689Z","shell.execute_reply.started":"2022-11-27T06:00:29.836354Z","shell.execute_reply":"2022-11-27T06:00:29.847393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### FPGrowth","metadata":{}},{"cell_type":"markdown","source":"FPGrowth is an alternatiive algorithm to aporori but needs less computation space and runs faster than the aprori which takes O(n^2) time using the mlextend library.","metadata":{}},{"cell_type":"code","source":"transcations_data_2020 = transcation_data[(transcation_data['year'] == 2020) & (transcation_data['month'] > 6)]\ntranscation_apri = transcations_data_2020.groupby('customer_id')['article_id'].unique().reset_index()\nte = TransactionEncoder()\nte.fit(transcation_apri['article_id'])\norders_1hot = te.transform(transcation_apri['article_id'])\norders_1hot = pd.DataFrame(orders_1hot, columns =te.columns_)\nfpgrowth(orders_1hot, min_support=0.6, use_colnames=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-27T06:00:29.850041Z","iopub.execute_input":"2022-11-27T06:00:29.850361Z","iopub.status.idle":"2022-11-27T06:01:25.683559Z","shell.execute_reply.started":"2022-11-27T06:00:29.850333Z","shell.execute_reply":"2022-11-27T06:01:25.682391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Next Steps??\n\n- Run classiification algorithms to classify product categories given different product attributes.\n- Run Neural Networks to predict the prices of articles using only their images.\n- Train model to generate labels for articles using their images.","metadata":{}}]}