{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **If you find \"My Notebook\" impressive 💖 and useful 👍🏻, Please Upvote it.**\n### **(Your vote and feedbacks are always welcome and valuable for me 😊✌🏻)**","metadata":{}},{"cell_type":"markdown","source":"# **Introduction**\n\nThe dataset contains 4 csv files and one folder with several subfolders.\n\nIn this Exploratory Data Analysis Notebook we will look to the data, will analyze the content of each csv file, understand the data distribution, see what are the relations between data in various files.\n\n**Aim is to show insightful plots and simple prediction.**","metadata":{}},{"cell_type":"markdown","source":"# **Table of Content**\n\n1).  Data Exploration and Visualization\n\n2).  Articles\n\n3).  Customers\n\n4).  Transactions\n\n5).  Simple Prediction\n\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":" ## **Data Exploration**","metadata":{}},{"cell_type":"markdown","source":"### **Import Libraries**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport plotly.express as px\nfrom matplotlib import pyplot as plt\nfrom tqdm.notebook import tqdm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Reading the files**","metadata":{}},{"cell_type":"code","source":"df_articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ndf_customers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ndf_transactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Article data description:**\n\n* **article_id :** A unique identifier of every article.\n\n* **product_code, prod_name :** A unique identifier of every product and its name (not the same).\n\n* **product_type, product_type_name :** The group of product_code and its name\n\n* **graphical_appearance_no, graphical_appearance_name :** The group of graphics and its name\n\n* **colour_group_code, colour_group_name :** The group of color and its name\n\n* **graphical_appearance_no, graphical_appearance_name :** The group of graphics and its name\n\n* **perceived_colour_value_id, perceived_colour_value_name, perceived_colour_master_id, perceived_colour_master_name :** The added color info\n\n* **department_no, department_name: :** A unique identifier of every dep and its name\n\n* **index_code, index_name: :** A unique identifier of every index and its name\n\n* **index_group_no, index_group_name: :** A group of indeces and its name\n\n* **section_no, section_name: :** A unique identifier of every section and its name\n\n* **garment_group_no, garment_group_name: :** A unique identifier of every garment and its name\n\n* **detail_desc: :** Details","metadata":{}},{"cell_type":"markdown","source":"## **Customers data description:**\n\n* **customer_id :** A unique identifier of every customer\n\n* **FN :** 1 or missed\n\n* **Active :** 1 or missed\n\n* **club_member_status :** Status in club\n\n* **fashion_news_frequency :** How often H&M may send news to customer\n\n* **age :** The current age\n\n* **postal_code :** Postal code of customer","metadata":{}},{"cell_type":"markdown","source":"## **Transactions data description:**\n\n* **t_dat :** A unique identifier of every customer\n\n* **customer_id :** A unique identifier of every customer (in customers table)\n\n* **article_id :** A unique identifier of every article (in articles table)\n\n* **price :** Price of purchase\n\n* **sales_channel_id :** 1 or 2","metadata":{}},{"cell_type":"code","source":"def plot_distribution(x, data, title):\n        fig = px.histogram(\n        data, \n        x = x,\n        width = 800,\n        height = 500,\n        title = title\n        )\n\n        fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Checking tables of articles, customers and transactions**","metadata":{}},{"cell_type":"code","source":"df_articles.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_articles.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_customers.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_customers.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_transactions.shape","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_transactions.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_articles.iloc[:, :-1].describe().T.sort_values(by='std' , ascending = False)\\\n                     .style.background_gradient(cmap='GnBu')\\\n                     .bar(subset=[\"max\"], color='#F8766D')\\\n                     .bar(subset=[\"mean\",], color='#00BFC4')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Insightful Plots : Articles**","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=df_articles, y='index_name', color='cyan')\nax.set_xlabel('count by index name')\nax.set_ylabel('index name')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From above we observed that Ladieswear accounts for a significant part of all dresses. Sportswear has the least portion.**","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=df_articles, y='garment_group_name', color='tomato', hue='index_group_name', multiple=\"stack\")\nax.set_xlabel('count by garment group')\nax.set_ylabel('garment group')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From above it seems that the garments grouped by index: Jersey fancy is the most frequent garment, especially for women and children. The next by number is accessories, many various accessories with low price.**","metadata":{}},{"cell_type":"markdown","source":"#### **And the table with number of unique values in columns:**","metadata":{}},{"cell_type":"code","source":"for col in df_articles.columns:\n    if not 'no' in col and not 'code' in col and not 'id' in col:\n        un_n = df_articles[col].nunique()\n        print(f'n of unique {col}: {un_n}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Insightful Plots : Customers**","metadata":{}},{"cell_type":"code","source":"df_customers.iloc[:, :-1].describe().T.sort_values(by='std' , ascending = False)\\\n                     .style.background_gradient(cmap='GnBu')\\\n                     .bar(subset=[\"max\"], color='#F8766D')\\\n                     .bar(subset=[\"mean\",], color='#00BFC4')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_distribution('age', df_customers, 'Age distribution')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From above plot we observed that the most common age is about 21-24**","metadata":{}},{"cell_type":"code","source":"df_customers.fashion_news_frequency.value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_distribution('fashion_news_frequency', df_customers, 'Fasion News Frequency')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From it seems that customers prefer not to get any messages about the current news.**","metadata":{}},{"cell_type":"code","source":"df_customers.club_member_status.value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_distribution('club_member_status', df_customers, 'Club Member Status')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From above we sae that Status in H&M club. Almost every customer has an active club status, some of them begin to activate it (pre-create). A tiny part of customers abandoned the club.**","metadata":{}},{"cell_type":"markdown","source":"## **Insightful Plots : Transactions**","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.float_format', '{:.4f}'.format)\ndf_transactions.describe()['price']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.boxplot(data=df_transactions, x='price', color='darkorange')\nax.set_xlabel('Price outliers')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From above we saw that the outliers for price**","metadata":{}},{"cell_type":"markdown","source":"### **Top 10 customers by num of transactions.**","metadata":{}},{"cell_type":"code","source":"df_transactions_byid = df_transactions.groupby('customer_id').count()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_transactions_byid.sort_values(by='price', ascending=False)['price'][:10]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Get subset from articles and merge it to transactions.**","metadata":{}},{"cell_type":"code","source":"articles_for_merge = df_articles[['article_id', 'prod_name', 'product_type_name', 'product_group_name', 'index_name']]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_for_merge = df_transactions[['customer_id', 'article_id', 'price', 't_dat']].merge(articles_for_merge, on='article_id', how='left')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The index with the highest mean price is Ladieswear. With the lowest - Children.**","metadata":{}},{"cell_type":"code","source":"articles_index = articles_for_merge[['index_name', 'price']].groupby('index_name').mean()\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=articles_index.price, y=articles_index.index, color='gold', alpha=0.8)\nax.set_xlabel('Price by index')\nax.set_ylabel('Index')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Stationery has the lowest mean price, the highest - Shoes.**","metadata":{}},{"cell_type":"code","source":"articles_index = articles_for_merge[['product_group_name', 'price']].groupby('product_group_name').mean()\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=articles_index.price, y=articles_index.index, color='orangered', alpha=0.8)\nax.set_xlabel('Price by product group')\nax.set_ylabel('Product group')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now check the mean price change in time for top 5 product groups by mean price:**","metadata":{}},{"cell_type":"code","source":"articles_for_merge['t_dat'] = pd.to_datetime(articles_for_merge['t_dat'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_list = ['Shoes', 'Garment Full body', 'Bags', 'Garment Lower body', 'Underwear/nightwear']\ncolors = ['cadetblue', 'orange', 'mediumspringgreen', 'tomato', 'lightseagreen']\nk = 0\nf, ax = plt.subplots(3, 2, figsize=(20, 15))\nfor i in range(3):\n    for j in range(2):\n        try:\n            product = product_list[k]\n            articles_for_merge_product = articles_for_merge[articles_for_merge.product_group_name == product_list[k]]\n            series_mean = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).mean().fillna(0)\n            series_std = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).std().fillna(0)\n            ax[i, j].plot(series_mean, linewidth=4, color=colors[k])\n            ax[i, j].fill_between(series_mean.index, (series_mean.values-2*series_std.values).ravel(), \n                             (series_mean.values+2*series_std.values).ravel(), color=colors[k], alpha=.1)\n            ax[i, j].set_title(f'Mean {product_list[k]} price in time')\n            ax[i, j].set_xlabel('month')\n            ax[i, j].set_xlabel(f'{product_list[k]}')\n            k += 1\n        except IndexError:\n            ax[i, j].set_visible(False)\n            \nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **Simple Prediction** ","metadata":{}},{"cell_type":"markdown","source":"**For this initial submission, we apply the following simplified logic:**\n* If there are articles for a certain client, pick the most recent buys;\n\n* If there are not articles for a certain client, just pick the most frequently buyed  articles.","metadata":{}},{"cell_type":"code","source":"df_transactions = df_transactions.sort_values([\"customer_id\", \"t_dat\"], ascending=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_transactions.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's capture first what are the most frequent recently bought articles.","metadata":{}},{"cell_type":"code","source":"last_date = df_transactions.t_dat.max()\nprint(last_date)\nprint(df_transactions.loc[df_transactions.t_dat==last_date].shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"most_frequent_articles = list(df_transactions.loc[df_transactions.t_dat==last_date].article_id.value_counts()[0:12].index)\nart_list = []\nfor art in most_frequent_articles:\n    art = \"0\"+str(art)\n    art_list.append(art)\nart_str = \" \".join(art_list)\nprint(\"Frequent articles bought recently: \", art_str)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"agg_df = df_transactions.groupby([\"customer_id\"])[\"article_id\"].agg(lambda x: str(x.values[0:12])[1:-1]).reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def padding_articles(x):\n    if x:\n        xl = x.split()\n        x = []\n        for xi in xl:\n            x.append(\"0\"+xi)\n        dimm_x = len(x)\n        if dimm_x < 12:\n            x.extend(art_list[:12-dimm_x])\n        return(\" \".join(x))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"agg_df[\"article_id\"] = agg_df[\"article_id\"].apply(lambda x: padding_articles(x))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_submission = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Aggregated transaction history: \", agg_df.customer_id.nunique())\nprint(\"Submission sample: \", df_sample_submission.customer_id.nunique())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will replace the values in sample submission with the existent in aggregated transactions data and just let the default one otherwise.","metadata":{}},{"cell_type":"code","source":"print(df_sample_submission.shape)\ndf_sample_submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the customers with missing articles, we simply replace with most frequent buyed articles in most recent day(s).","metadata":{}},{"cell_type":"code","source":"My_Final_Submission = agg_df.merge(df_sample_submission[[\"customer_id\"]], how=\"right\")\nMy_Final_Submission.columns = [\"customer_id\", \"prediction\"]\nprint(My_Final_Submission.shape)\nMy_Final_Submission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Rows with missing data in submission:**","metadata":{}},{"cell_type":"code","source":"My_Final_Submission.loc[My_Final_Submission.prediction.isna()].shape[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We replace the missing data with the most frequently bought articles, from recent days. We calculated it before.","metadata":{}},{"cell_type":"code","source":"My_Final_Submission.loc[My_Final_Submission.prediction.isna(), [\"prediction\"]] = art_str","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Rows with missing data in submission: \", My_Final_Submission.loc[My_Final_Submission.prediction.isna()].shape[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"My_Final_Submission.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}