{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This is very sample EDA for H&M Personalized Fashion Recommendations. I just created this EDA for quick jump into competition. Hope you find it useful for your own competition start. Enjoy and have fun in competiton!\n\n
\n\n# COMPETITION GOAL\n\n🛍️ Competition Goal: In this competition, H&M Group invites you to develop product recommendations based on data from previous transactions, as well as from customer and product meta data. The available meta data spans from simple data, such as garment type and customer age, to text data from product descriptions, to image data from garment images. \n\n## Files\n- `images/` - a folder of images corresponding to each article_id; images are placed in subfolders starting with the first three digits of the article_id; note, not all article_id values have a corresponding image.\n- `articles.csv` - detailed metadata for each article_id available for purchase\n- `customers.csv` - metadata for each customer_id in dataset\n- `sample_submission.csv` - a sample submission file in the correct format\n- `transactions_train.csv` - the training data, consisting of the purchases each customer for each date, as well as additional information. Duplicate rows correspond to multiple purchases of the same item. Your task is to predict the article_ids each customer will purchase during the 7-day period immediately after the training data period.\n","metadata":{}},{"cell_type":"markdown","source":"\n#### Column Descriptions [(source)](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/overview)\n\n\n
\n\n| Column Name - customers.csv | Description |\n|:--|:--|\n| customer_id | A unique identifier of every customer | \n| FN | if a customer get Fashion News newsletter | \n| Active | if the customer is active for communication |\n| club_member_status | Status in club |\n| fashion_news_frequency | How often H&M may send news to customer |\n| age | The current age |\n| postal_code | Postal code of customer|\n\n
\n\n| Column Name - transactions_train.csv | Description |\n|:--|:--|\n| t_dat | transaction date | \n| customer_id | A unique identifier of every customer (in customers table) | \n| article_id | A unique identifier of every article (in articles table) |\n| price | Price of purchase |\n| sales_channel_id | 2 is online and 1 store |\n\n
\n\n| Article.csv - Column Name | Description |\n|:--|:--|\n| article_id | A unique identifier of every article | \n| Date | Date of Order | \n| product_code, prod_name | A unique identifier of every product and its name (not the same) |\n| Sproduct_type, product_type_name | The group of product_code and its name |\n| graphical_appearance_no, graphical_appearance_name | The group of graphics and its name |\n| colour_group_code, colour_group_name | The group of color and its name |\n| graphical_appearance_no, graphical_appearance_name | The group of graphics and its name |\n| perceived_colour_value_id, perceived_colour_value_name, perceived_colour_master_id, perceived_colour_master_name | The added color info |\n| department_no, department_name | A unique identifier of every dep and its name |\n| index_code, index_name | A unique identifier of every index and its name |\n| index_group_no, index_group_name | A group of indeces and its name | \n| section_no, section_name | A unique identifier of every section and its name | \n| garment_group_no, garment_group_name | A unique identifier of every garment and its name |\n| detail_desc | Details | \n\n
\n","metadata":{}},{"cell_type":"markdown","source":"### Import Libraries","metadata":{}},{"cell_type":"code","source":"from IPython.display import display_html\n\n\nimport sys\nimport pandas as pd\nimport numpy as np\nimport plotly.express as px\nimport seaborn as sns\n\nfrom tqdm import tqdm\nimport glob\nfrom collections import Counter\nfrom PIL import Image\n\nimport pandas as pd\nfrom pathlib import Path\n\nimport matplotlib as mpl\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nimport calendar\nfrom termcolor import colored\nfrom IPython.display import HTML\n\nimport time","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Environment check\nimport warnings\nimport os\n\nwarnings.filterwarnings(\"ignore\")\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCONFIG = {'competition': 'HandM', '_wandb_kernel': 'aot'}\n\n# Custom colors\nclass clr:\n S = '\\033[1m' + '\\033[95m'\n E = '\\033[0m'\n \nmy_colors = [\"#003f5c\", \"#2f4b7c\", \"#665191\", \"#a05195\", \"#d45087\", \"#f95d6a\", \"#ff7c43\", \"#ffa600\"]\nprint(clr.S+\"Notebook Color Scheme:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Dataset\n\n🛍️ **There are 3 metadata .csv files and 1 image file:**\n* `images` - folder containing the photo of *almost* all `article_ids`\n* `articles.csv` - description features of all `article_ids` **(105,542 datapoints)**\n* `customers.csv` - description features of the customer profiles **(1,371,980 datapoints)**\n* `transactions_train.csv` - file containing the `customer_id`, the article that was bought and at what price **(31,788,324 datapoints)**","metadata":{}},{"cell_type":"code","source":"%%time\npath = Path(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/\")\n\narticles_df = pd.read_csv(path / \"articles.csv\", dtype = {'article_id': str})\ncust_df = pd.read_csv(path / \"customers.csv\", dtype = {'customer_id': str})\ntrans_df = pd.read_csv(path / \"transactions_train.csv\", dtype = {'article_id': str,'customer_id': str})\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-25T00:37:36.938169Z","iopub.execute_input":"2022-05-25T00:37:36.939012Z","iopub.status.idle":"2022-05-25T00:38:47.580141Z","shell.execute_reply.started":"2022-05-25T00:37:36.938959Z","shell.execute_reply":"2022-05-25T00:38:47.579215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Getting an overview of the datasets\n#### ---- Start the data cleaning----","metadata":{}},{"cell_type":"markdown","source":"#### **Article dataset**","metadata":{}},{"cell_type":"code","source":"# number_of_rows = len(articles_df)\n# number_of_col = len(articles_df.columns)\n# print(f'Number of rows in articles.csv: {number_of_rows}')\n# print(f'Number of rows in articles.csv: {number_of_col}')\n\n\nprint(clr.S+\"ARTICLES:\"+clr.E, articles_df.shape)\ndisplay_html(articles_df.head(3).T)\nprint(\"\\n\", clr.S+\"CUSTOMERS:\"+clr.E, cust_df.shape)\ndisplay_html(cust_df.head(3).T)\nprint(\"\\n\", clr.S+\"TRANSACTIONS:\"+clr.E, trans_df.shape)\ndisplay_html(trans_df.head(3).T)\n# print(\"\\n\", clr.S+\"SAMPLE_SUBMISSION:\"+clr.E, ss.shape)\n# display_html(ss.head(3))","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:38:47.58165Z","iopub.execute_input":"2022-05-25T00:38:47.582133Z","iopub.status.idle":"2022-05-25T00:38:47.610531Z","shell.execute_reply.started":"2022-05-25T00:38:47.582072Z","shell.execute_reply":"2022-05-25T00:38:47.60972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#### Duplicate Records\n##### How many duplicate transaction records are there?\ndup_rows = articles_df.duplicated().sum()\nprint(f'Number of duplicate record in articles.csv:{dup_rows}')\ndup_rows = cust_df.duplicated().sum()\nprint(f'Number of duplicate record in customers.csv:{dup_rows}')\ndup_rows = trans_df.duplicated().sum()\nprint(f'Number of duplicate record in transactions_train.csv:{dup_rows}')\n#### Drop Duplicate Records\n##### Drop the duplicated records.\n\n# articles_df = articles_df.drop_duplicates()\n# cust_df= cust_df.drop_duplicates()\n# cust_df = cust_df.drop_duplicates()\n#### Missing Values\n##### How many missing values are there?\n# articles_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:38:47.612879Z","iopub.execute_input":"2022-05-25T00:38:47.613612Z","iopub.status.idle":"2022-05-25T00:39:15.222356Z","shell.execute_reply.started":"2022-05-25T00:38:47.613582Z","shell.execute_reply":"2022-05-25T00:39:15.220923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Functions","metadata":{}},{"cell_type":"code","source":"def adjust_id(x):\n '''Adjusts article ID code.'''\n x = str(x)\n if len(x) == 9:\n x = \"0\"+x\n \n return x\n\n\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:15.225091Z","iopub.execute_input":"2022-05-25T00:39:15.225366Z","iopub.status.idle":"2022-05-25T00:39:15.236529Z","shell.execute_reply.started":"2022-05-25T00:39:15.225334Z","shell.execute_reply":"2022-05-25T00:39:15.235696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Articles\n\n## I. Preprocessing\n\n🛍️ **Important Notes**:\n* There are *more* `article_ids` than actual images:\n * unique article ids: 105,542\n * unique images: 105,100\n* The `path` processing was taking too long, so the fastest (takes 1 second) way to do it was to create a variable that contains all article ids within the `images` folder (remember, `set()` is faster than a `list`), and then to correct any path that was invalid within the `articles.csv` file.\n* There are only 416 missing values within the `desc` column - product description","metadata":{}},{"cell_type":"code","source":"# Get all paths from the image folder\n\nall_image_paths = glob.glob(f\"/kaggle/input/h-and-m-personalized-fashion-recommendations/images/*/*\")\n\nprint(clr.S+\"Number of unique article_ids within articles.csv:\"+clr.E, len(articles_df), \"\\n\"+\n clr.S+\"Number of unique images within the image folder:\"+clr.E, len(all_image_paths), \"\\n\"+\n clr.S+\"=> not all article_ids have a corresponding image!!!\"+clr.E, \"\\n\")\n\n\n# Get all valid article ids\n# Create a set() - as it moves faster than a list\nall_image_ids = set()\n\nfor path in tqdm(all_image_paths):\n article_id = path.split('/')[-1].split('.')[0]\n all_image_ids.add(article_id)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:15.238376Z","iopub.execute_input":"2022-05-25T00:39:15.238857Z","iopub.status.idle":"2022-05-25T00:39:30.264889Z","shell.execute_reply.started":"2022-05-25T00:39:15.238827Z","shell.execute_reply":"2022-05-25T00:39:30.264081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(clr.S+\"There are no missing values in any columns but 'Detail Description':\"+clr.E,\n articles_df.isna().sum()[-1], \"total missing values\")\n\n# Replace missing values\narticles_df.fillna(value=\"No Description\", inplace=True)\n\n# Adjust the article ID and product code to be string & add \"0\"\narticles_df[\"article_id\"] = articles_df[\"article_id\"].apply(lambda x: adjust_id(x))\narticles_df[\"product_code\"] = articles_df[\"article_id\"].apply(lambda x: x[:3])","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:30.267221Z","iopub.execute_input":"2022-05-25T00:39:30.267501Z","iopub.status.idle":"2022-05-25T00:39:30.516885Z","shell.execute_reply.started":"2022-05-25T00:39:30.267473Z","shell.execute_reply":"2022-05-25T00:39:30.516045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# An image path example: ../input/h-and-m-personalized-fashion-recommendations/images/010/0108775015.jpg\n\n# Create full path to the article image\nimages_path = \"../input/h-and-m-personalized-fashion-recommendations/images/\"\narticles_df [\"path\"] = images_path + articles_df[\"product_code\"] + \"/\" + articles_df[\"article_id\"] + \".jpg\"\n\n# Adjust the incorrect paths and set them to None\nfor k, article_id in tqdm(enumerate(articles_df[\"article_id\"])):\n if article_id not in all_image_ids:\n articles_df.loc[k, \"path\"] = None","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:30.518235Z","iopub.execute_input":"2022-05-25T00:39:30.518427Z","iopub.status.idle":"2022-05-25T00:39:32.166101Z","shell.execute_reply.started":"2022-05-25T00:39:30.518403Z","shell.execute_reply":"2022-05-25T00:39:32.165264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## II. Explore","metadata":{}},{"cell_type":"code","source":"import matplotlib as mpl\nmpl.rcParams.update(mpl.rcParamsDefault)\n\ndata = articles_df\nart_dtypes = articles_df.dtypes.value_counts()\n\nfig = plt.figure(figsize=(8,2),facecolor='white')\n\nax0 = fig.add_subplot(1,1,1)\nfont = 'monospace'\nax0.text(1, 0.8, \"Key figures\",color='black',fontsize=28, fontweight='bold', fontfamily=font, ha='center')\n\nax0.text(0, 0.4, \"{:,d}\".format(len(articles_df)), color='#fcba03', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax0.text(0, 0.001, \"# of unique article_id \\nin articles.csv\",color='dimgrey',fontsize=15, fontweight='light', fontfamily=font,ha='center')\n\nax0.text(0.6, 0.4, \"{:,d}\".format(len(cust_df)), color='#fcba03', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax0.text(0.6, 0.001, \"# of unique customer_id \\nin customers.csv\",color='dimgrey',fontsize=15, fontweight='light', fontfamily=font,ha='center')\n\nax0.text(1.2, 0.4, \"{:,d}\".format(len(trans_df)), color='#fcba03', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax0.text(1.2, 0.001, \"# of transaction \\nin the transaction_train\",color='dimgrey',fontsize=15, fontweight='light', fontfamily=font, ha='center')\n\n\nax0.text(1.9, 0.4,\"{:,d}\".format(len(trans_df.groupby(['customer_id'])['customer_id'].count())), color='#fcba03', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax0.text(1.9, 0.001,\"# of numeric columns \\nin the dataset\",color='dimgrey',fontsize=15, fontweight='light', fontfamily=font,ha='center')\n\nax0.set_yticklabels('')\nax0.tick_params(axis='y',length=0)\nax0.tick_params(axis='x',length=0)\nax0.set_xticklabels('')\n\nfor direction in ['top','right','left','bottom']:\n ax0.spines[direction].set_visible(False)\n\nfig.subplots_adjust(top=0.9, bottom=0.2, left=0, hspace=1)\n\nfig.patch.set_linewidth(5)\nfig.patch.set_edgecolor('#8c8c8c')\nfig.patch.set_facecolor('#f6f6f6')\nax0.set_facecolor('#f6f6f6')\n \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:32.167406Z","iopub.execute_input":"2022-05-25T00:39:32.167617Z","iopub.status.idle":"2022-05-25T00:39:44.678629Z","shell.execute_reply.started":"2022-05-25T00:39:32.167592Z","shell.execute_reply":"2022-05-25T00:39:44.677851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Articles dataset**\n#### Descriptive of Articles dataset","metadata":{}},{"cell_type":"code","source":"print(clr.S+\"Number of unique article_ids within articles.csv:\"+clr.E, len(articles_df), \"\\n\"+\n clr.S+\"Number of unique images within the image folder:\"+clr.E, len(all_image_paths), \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:44.680062Z","iopub.execute_input":"2022-05-25T00:39:44.680369Z","iopub.status.idle":"2022-05-25T00:39:44.685777Z","shell.execute_reply.started":"2022-05-25T00:39:44.680329Z","shell.execute_reply":"2022-05-25T00:39:44.685165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Customer dataset**\n#### Descriptive of customer dataset","metadata":{}},{"cell_type":"code","source":"n = cust_df.customer_id.nunique()\nprint(f'Number of customer: {n}')\nn = cust_df.age.mean()\nprint(f'Average age of customer: {n}')","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:44.686727Z","iopub.execute_input":"2022-05-25T00:39:44.687374Z","iopub.status.idle":"2022-05-25T00:39:45.307348Z","shell.execute_reply.started":"2022-05-25T00:39:44.687343Z","shell.execute_reply":"2022-05-25T00:39:45.306429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_df.iloc[:, :-1].describe().T.sort_values(by='std' , ascending = False)\\\n .style.background_gradient(cmap='GnBu')\\\n .bar(subset=[\"max\"], color='#F8766D')\\\n .bar(subset=[\"mean\",], color='#00BFC4')","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:45.308588Z","iopub.execute_input":"2022-05-25T00:39:45.308891Z","iopub.status.idle":"2022-05-25T00:39:45.623348Z","shell.execute_reply.started":"2022-05-25T00:39:45.308852Z","shell.execute_reply":"2022-05-25T00:39:45.62256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### **Transaction dataset**","metadata":{}},{"cell_type":"code","source":"number_of_rows = len(trans_df)\nnumber_of_col = len(trans_df.columns)\nprint(f\"Number of rows in transactions_train.csv: {colored(number_of_rows, 'yellow')}\")\nprint(f'Number of rows in transactions_train.csv:{number_of_col}')","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:45.625797Z","iopub.execute_input":"2022-05-25T00:39:45.625994Z","iopub.status.idle":"2022-05-25T00:39:45.630485Z","shell.execute_reply.started":"2022-05-25T00:39:45.62597Z","shell.execute_reply":"2022-05-25T00:39:45.629578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Join tables","metadata":{}},{"cell_type":"code","source":"#Extract only the column needed\narticles_df = articles_df[['article_id', 'product_type_name','product_group_name','colour_group_name','prod_name','path','graphical_appearance_name']]\ncust_df = cust_df[['customer_id', 'club_member_status','fashion_news_frequency','age']]\n","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:45.631729Z","iopub.execute_input":"2022-05-25T00:39:45.632158Z","iopub.status.idle":"2022-05-25T00:39:45.78481Z","shell.execute_reply.started":"2022-05-25T00:39:45.632103Z","shell.execute_reply":"2022-05-25T00:39:45.784107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.merge(trans_df, cust_df, on='customer_id', how='left')\ndel trans_df,cust_df","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:39:45.78603Z","iopub.execute_input":"2022-05-25T00:39:45.786262Z","iopub.status.idle":"2022-05-25T00:40:00.548036Z","shell.execute_reply.started":"2022-05-25T00:39:45.786236Z","shell.execute_reply":"2022-05-25T00:40:00.547285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.merge(df, articles_df, on='article_id', how='left')\n# del articles_df","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:40:00.549043Z","iopub.execute_input":"2022-05-25T00:40:00.549702Z","iopub.status.idle":"2022-05-25T00:40:13.683653Z","shell.execute_reply.started":"2022-05-25T00:40:00.549667Z","shell.execute_reply":"2022-05-25T00:40:13.682604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:40:13.684928Z","iopub.execute_input":"2022-05-25T00:40:13.685184Z","iopub.status.idle":"2022-05-25T00:40:13.699229Z","shell.execute_reply.started":"2022-05-25T00:40:13.685154Z","shell.execute_reply":"2022-05-25T00:40:13.698245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"time.sleep(30)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:46:15.595489Z","iopub.execute_input":"2022-05-25T00:46:15.596279Z","iopub.status.idle":"2022-05-25T00:46:45.63148Z","shell.execute_reply.started":"2022-05-25T00:46:15.596225Z","shell.execute_reply":"2022-05-25T00:46:45.630566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Consistent formatting - Date","metadata":{}},{"cell_type":"code","source":"import calendar\ndf['t_dat'] = pd.to_datetime(df['t_dat'])\ndf['YYYY_MM'] = df['t_dat'].dt.year.astype(str) + '_' + df['t_dat'].dt.month.astype(str)\ndf['year'] = df['t_dat'].dt.year\n# df['month'] = df['t_dat'].dt.month\n# df['month_'] = df['month'].apply(lambda x: calendar.month_abbr[x])","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:46:45.63526Z","iopub.execute_input":"2022-05-25T00:46:45.635586Z","iopub.status.idle":"2022-05-25T00:47:40.700348Z","shell.execute_reply.started":"2022-05-25T00:46:45.635554Z","shell.execute_reply":"2022-05-25T00:47:40.699487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_df = df[['t_dat', 'customer_id','article_id']]\ncount_df = count_df.groupby(['t_dat', 'customer_id']).size().rename('quantity').reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:47:40.701528Z","iopub.execute_input":"2022-05-25T00:47:40.701727Z","iopub.status.idle":"2022-05-25T00:47:57.841767Z","shell.execute_reply.started":"2022-05-25T00:47:40.701703Z","shell.execute_reply":"2022-05-25T00:47:57.841165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Printing minimum and the maximum date from dataset.\nprint(f'The time range of the transaction csv: From {df.t_dat.min():%Y-%m-%d} To {df.t_dat.max():%Y-%m-%d}')\n# print(f'Number of unique customers: {df.customer_id.nunique()}')\n# print(f'Number of unique items: {df.article_id.nunique()}')\n\nprint(f'Average purchase quantity per interaction: {int(count_df.quantity.mean())}')\nprint(f'Minimum purchase quantity per interaction: {count_df.quantity.min()}')\nprint(f'Maximum purchase quantity per interaction: {count_df.quantity.max()}')","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:47:57.843338Z","iopub.execute_input":"2022-05-25T00:47:57.843712Z","iopub.status.idle":"2022-05-25T00:47:58.077706Z","shell.execute_reply.started":"2022-05-25T00:47:57.843668Z","shell.execute_reply":"2022-05-25T00:47:58.076805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Sale of H&M per month(Sep2018 - Sep2020)","metadata":{}},{"cell_type":"code","source":"# import plotly.express as px\n# dfg = df[['t_dat','price']]\n# dfg = df.groupby(df['t_dat']).agg({'price':sum}).reset_index()\n# dfg\n# fig = px.line(dfg, x=\"t_dat\", y=\"price\"\n# ,hover_data={\"t_dat\": \"|%B %d, %Y\"}\n# ,template = \"plotly_white\"\n# )\n# fig.update_layout(\n# title=\"Sale per month (Sep2018-Sep2022)\"\n# ,xaxis_title=\"year_month\"\n# ,yaxis_title=\"Sale\"\n\n# )\n\n# fig.update_xaxes(\n# dtick=\"M1\",\n# tickformat=\"%b\\n%Y\",\n# ticklabelmode=\"period\")\n\n# fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import plotly.express as px\n# dfg = df.groupby(['t_dat','sales_channel_id']).agg({'price':sum}).reset_index()\n# dfg\n# fig = px.line(dfg, x=\"t_dat\", y=\"price\",\n# color='sales_channel_id'\n# ,hover_data={\"t_dat\": \"|%B %d, %Y\"}\n# ,template = \"plotly_white\"\n# )\n# fig.update_layout(\n# title=\"Sale per month by sale channel(Sep2018-Sep2022)\"\n# ,xaxis_title=\"year_month\"\n# ,yaxis_title=\"Sale\"\n\n# )\n\n# fig.update_xaxes(\n# dtick=\"M1\",\n# tickformat=\"%b\\n%Y\",\n# ticklabelmode=\"period\")\n\n# fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dfg = df[['t_dat','price']]\n# dfg = df.groupby(df['t_dat'].dt.strftime('%Y_%b')).agg({'price':sum}).reset_index()\n# dfg\n# fig = px.line(dfg, x=\"t_dat\", y=\"price\"\n# ,hover_data={\"t_dat\": \"|%B, %Y\"}\n# ,markers=True\n# # ,color_discrete_sequence=px.colors.diverging.PRGn\n# ,template = \"plotly_white\"\n# )\n# fig.update_layout(\n# title=\"Sale per month (Sep2018-Sep2022)\"\n# ,xaxis_title=\"year_month\"\n# ,yaxis_title=\"Sale\"\n\n# )\n\n# fig.update_xaxes(\n# dtick=\"M1\",\n# tickformat=\"%b\\n%Y\",\n# ticklabelmode=\"period\")\n\n# fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Top Prod Category","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# #Top 10 Prod Category\n# taba = pd.crosstab(df.product_group_name, df.YYYY_MM, values=df.price, aggfunc='sum').round(0)\n# taba = taba.sort_values(by='2020_9',ascending=False)\n# tabb = pd.crosstab(df.product_group_name, df.YYYY_MM, values=df.price, aggfunc='sum',normalize='columns').round(4)*100\n\n# tab = (\n# pd.concat([taba,tabb],axis = 1, keys = ['sum','%'])\n# .swaplevel(axis = 1)\n# .sort_index(axis = 1, ascending=[True, False])\n# .rename_axis(['YYYY_MM', 'product_group_name'], axis = 1)\n# )\n\n# tab\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Compare the % of total sale by month.","metadata":{}},{"cell_type":"code","source":"# cross_tab_prop = df.loc[df['year'] >= 2019]\n# cross_tab_prop = cross_tab_prop[['product_group_name', 'year','price']]\n\n# cross_tab_prop = cross_tab_prop.groupby(['year','product_group_name'])['price'].sum()\n# cross_tab_prop = cross_tab_prop.groupby(level=0).apply(lambda x:100 * x / float(x.sum())).rename('percentage')\n\n# cross_tab_prop = cross_tab_prop.to_frame().reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig=px.bar(cross_tab_prop\n# ,x='percentage'\n# ,y='year'\n# ,color = 'product_group_name'\n# , orientation='h'\n# , barmode = 'stack'\n# ,color_discrete_sequence=px.colors.diverging.PRGn\n# ,text=cross_tab_prop['percentage'].map('{:,.2f}%'.format)\n# )\n\n# fig.update_layout(title = \"Percentage share of product category\", \n# template = 'simple_white', xaxis_title = '%', \n# yaxis_title = 'year',\n# legend_title_text='product_group_name')\n\n# fig\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# cross_tab_prop = df.groupby([df['t_dat'].dt.strftime('%Y_%b'),'product_group_name']).agg({'price':sum}).reset_index()\n# fig = px.line(cross_tab_prop, x=\"t_dat\", y=\"price\"\n# ,hover_data={\"t_dat\": \"|%Y_%b\"}\n# # ,markers=True\n# ,color='product_group_name'\n# ,color_discrete_sequence=px.colors.diverging.PRGn\n# ,template = \"plotly_white\"\n# )\n# fig.update_layout(\n# title=\"Sale per month (Sep2018-Sep2020)\"\n# ,xaxis_title=\"year_month\"\n# ,yaxis_title=\"Sale\"\n# )\n\n# fig.update_xaxes(\n# dtick=\"M1\",\n# tickformat=\"%b\\n%Y\",\n# ticklabelmode=\"period\")\n\n# fig.show(\"\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# fig = px.bar(cross_tab_prop, x=\"t_dat\", y=\"price\",color='product_group_name'\n# ,hover_data={\"t_dat\": \"|%Y_%b\"}\n# # ,markers=True\n# ,color_discrete_sequence=px.colors.diverging.PRGn\n# ,template = \"plotly_white\"\n# )\n# fig.update_layout(\n# title=\"Sale per month by product category(Sep2018-Sep2020)\"\n# ,xaxis_title=\"Year\"\n# ,yaxis_title=\"Sale\"\n# ,legend_title_text='product_group_name'\n# )\n\n\n# fig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Most Freq Product Names","metadata":{}},{"cell_type":"code","source":"# print(clr.S+\"Total Number of unique Product Names:\"+clr.E, df[\"prod_name\"].nunique())\n\n# # Data\n# prod_name = df[\"prod_name\"].value_counts().reset_index().head(15)\n# total_prod_names = df[\"prod_name\"].nunique()\n# clrs = [\"#CB2170\" if x==max(prod_name[\"prod_name\"]) else '#954E93' for x in prod_name[\"prod_name\"]]\n\n# # Get images\n# prod_name_images = articles[articles[\"prod_name\"].isin(prod_name[\"index\"].tolist())].groupby(\"prod_name\")[\"path\"].first().reset_index()\n# image_paths = prod_name_images[\"path\"].tolist()\n# image_names = prod_name_images[\"prod_name\"].tolist()\n\n# # Plot\n# fig, ax = plt.subplots(figsize=(25, 13))\n# plt.title('- Most Frequent Product Names -', size=22, weight=\"bold\")\n\n# sns.barplot(data=prod_name, x=\"prod_name\", y=\"index\", ax=ax,\n# palette=clrs)\n# x0,x1 = ax.get_xlim()\n# y0,y1 = ax.get_ylim()\n# plt.imshow(bk_image, zorder=0, extent=[x0, x1, y0, y1], alpha=0.35, aspect='auto')\n\n# show_values_on_bars(axs=ax, h_v=\"h\", space=0.4)\n# plt.ylabel(\"Product Name\", size = 16, weight=\"bold\")\n# plt.xlabel(\"\")\n# plt.xticks([])\n# plt.yticks(size=16)\n# plt.tick_params(size=16)\n\n# insert_image(path='../input/hm-fashion-recommender-dataset/pics/dragonfly.jpg', zoom=0.45, xybox=(92, 11), ax=ax)\n\n# sns.despine(left=True, bottom=True)\n# plt.show();\n\n# print(\"\\n\")\n\n# # Plot\n# fig, axs = plt.subplots(3, 5, figsize=(23, 8))\n# fig.suptitle('- Example Images -', size=22, weight=\"bold\")\n# axs = axs.flatten()\n\n# for k, (path, name) in enumerate(zip(image_paths, image_names)):\n# axs[k].set_title(f\"{name}\", size = 16)\n# img = plt.imread(path)\n# axs[k].imshow(img)\n# axs[k].axis(\"off\")\n\n# plt.tight_layout()\n# plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Find the Monthly Top 12 Articles\nI would recommend the latest monthly Top 10 items to the customer who does not have transaction(that I can not learn) idea from https://www.kaggle.com/negoto/best-selling-items-catalog-like-eda-of-articles","metadata":{}},{"cell_type":"code","source":"monthly_df = df.query(\"'2020-9-1' <= t_dat\")\nweekly_df = df.query(\"'2020-9-16' <= t_dat\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"time.sleep(30)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:47:58.078749Z","iopub.execute_input":"2022-05-25T00:47:58.078965Z","iopub.status.idle":"2022-05-25T00:48:28.112275Z","shell.execute_reply.started":"2022-05-25T00:47:58.078937Z","shell.execute_reply":"2022-05-25T00:48:28.111511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom collections import Counter\nfrom PIL import Image\nfrom pathlib import Path\n\n\ndef show_images(article_ids, cols=1, rows=-1):\n if isinstance(article_ids, int) or isinstance(article_ids, str):\n article_ids = [article_ids]\n article_count = len(article_ids)\n if rows < 0: rows = (article_count // cols) + 1\n plt.figure(figsize=(3 + 3.5 * cols, 3 + 5 * rows))\n for i in range(article_count):\n article_id = (\"0\" + str(article_ids[i]))[-10:]\n plt.subplot(rows, cols, i + 1)\n plt.axis('off')\n plt.title(article_id)\n try:\n image = Image.open(f\"/kaggle/input/h-and-m-personalized-fashion-recommendations/images/{article_id[:3]}/{article_id}.jpg\")\n plt.imshow(image)\n except:\n pass\n\n\nsales_counts = Counter(df.article_id)\nfor i in range(len(articles_df)):\n articles_df.at[i, \"sales_count\"] = sales_counts[articles_df.at[i, \"article_id\"]]\n\n# monthly_sales_counts = Counter(monthly_df.article_id)\n# for i in range(len(df)):\n# df.at[i, \"monthly_sales_count\"] = monthly_sales_counts[df.at[i, \"article_id\"]]\n \n# weekly_sales_counts = Counter(weekly_df.article_id)\n# for i in range(len(df)):\n# df.at[i, \"weekly_sales_count\"] = weekly_sales_counts[df.at[i, \"article_id\"]]","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:48:28.113323Z","iopub.execute_input":"2022-05-25T00:48:28.114012Z","iopub.status.idle":"2022-05-25T00:48:37.137307Z","shell.execute_reply.started":"2022-05-25T00:48:28.113959Z","shell.execute_reply":"2022-05-25T00:48:37.136557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_df = articles_df.sort_values(by=\"sales_count\", ascending=False)\ntemp = articles_df.article_id[:12]\nshow_images(list(temp), 6)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:48:37.138468Z","iopub.execute_input":"2022-05-25T00:48:37.13876Z","iopub.status.idle":"2022-05-25T00:48:40.443512Z","shell.execute_reply.started":"2022-05-25T00:48:37.138724Z","shell.execute_reply":"2022-05-25T00:48:40.44168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(clr.S+\"Total Number of unique Product Types:\"+clr.E, articles_df[\"product_type_name\"].nunique())\n\n# Data\nprod_type = articles_df[\"product_type_name\"].value_counts().reset_index().head(15)\ntotal_prod_types = articles_df[\"product_type_name\"].nunique()\nclrs = [\"#00BDE3\" if x==max(prod_type[\"product_type_name\"]) else '#398BBB' for x in prod_type[\"product_type_name\"]]\n\n# Get images\nprod_type_images = articles_df[articles_df[\"product_type_name\"].isin(prod_type[\"index\"].tolist())].groupby(\"product_type_name\")[\"path\"].first().reset_index()\nimage_paths = prod_type_images[\"path\"].tolist()\nimage_names = prod_type_images[\"product_type_name\"].tolist()\n\n# Plot\nfig, ax = plt.subplots(figsize=(25, 13))\nplt.title('- Most Frequent Product Types -', size=22, weight=\"bold\")\n\nsns.barplot(data=prod_type, x=\"product_type_name\", y=\"index\", ax=ax,\n palette=clrs)\nx0,x1 = ax.get_xlim()\ny0,y1 = ax.get_ylim()\n# plt.imshow(bk_image, zorder=0, extent=[x0, x1, y0, y1], alpha=0.35, aspect='auto')\n\nshow_values_on_bars(axs=ax, h_v=\"h\", space=0.4)\nplt.ylabel(\"Product Type\", size = 16, weight=\"bold\")\nplt.xlabel(\"\")\nplt.xticks([])\nplt.yticks(size=16)\nplt.tick_params(size=16)\n\n# insert_image(path='../input/hm-fashion-recommender-dataset/pics/blue.jpg', zoom=0.45, xybox=(11000, 11), ax=ax)\n\nsns.despine(left=True, bottom=True)\nplt.show();\n\nprint(\"\\n\")\n\n# Plot\nfig, axs = plt.subplots(3, 5, figsize=(23, 8))\nfig.suptitle('- Example Images -', size=22, weight=\"bold\")\naxs = axs.flatten()\n\nfor k, (path, name) in enumerate(zip(image_paths, image_names)):\n axs[k].set_title(f\"{name}\", size = 16)\n img = plt.imread(path)\n axs[k].imshow(img)\n axs[k].axis(\"off\")\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:50:48.481814Z","iopub.execute_input":"2022-05-25T00:50:48.482742Z","iopub.status.idle":"2022-05-25T00:50:53.142579Z","shell.execute_reply.started":"2022-05-25T00:50:48.482702Z","shell.execute_reply":"2022-05-25T00:50:53.141735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(clr.S+\"Total Number of unique Product Group:\"+clr.E, articles_df[\"product_group_name\"].nunique())\n\n# Data\nprod_group = articles_df[\"product_group_name\"].value_counts().reset_index()\ntotal_prod_groups = articles_df[\"product_group_name\"].nunique()\nclrs = [\"#E90B60\" if x==max(prod_group[\"product_group_name\"]) else '#AF0848' for x in prod_group[\"product_group_name\"]]\n\n# Get images\nprod_group_images = articles_df[articles_df[\"product_group_name\"].isin(prod_group[\"index\"].tolist())].groupby(\"product_group_name\")[\"path\"].first().reset_index()\nimage_paths = prod_group_images[\"path\"].tolist()\nimage_names = prod_group_images[\"product_group_name\"].tolist()\n\n# Plot\nfig, ax = plt.subplots(figsize=(25, 13))\nplt.title('- Most Frequent Product Groups -', size=22, weight=\"bold\")\n\nsns.barplot(data=prod_group, x=\"product_group_name\", y=\"index\", ax=ax,\n palette=clrs)\nx0,x1 = ax.get_xlim()\ny0,y1 = ax.get_ylim()\n# plt.imshow(bk_image, zorder=0, extent=[x0, x1, y0, y1], alpha=0.35, aspect='auto')\n\nshow_values_on_bars(axs=ax, h_v=\"h\", space=0.4)\nplt.ylabel(\"Product Group\", size = 16, weight=\"bold\")\nplt.xlabel(\"\")\nplt.xticks([])\nplt.yticks(size=16)\nplt.tick_params(size=16)\n\n# insert_image(path='../input/hm-fashion-recommender-dataset/pics/chloe.jpg', zoom=0.45, xybox=(40000, 14), ax=ax)\n\nsns.despine(left=True, bottom=True)\nplt.show();\n\nprint(\"\\n\")\n\n# Plot\nfig, axs = plt.subplots(4, 6, figsize=(23, 10))\nfig.suptitle('- Example Images -', size=22, weight=\"bold\")\naxs = axs.flatten()\n\nfor k, (path, name) in enumerate(zip(image_paths, image_names)):\n axs[k].set_title(f\"{name}\", size = 16)\n img = plt.imread(path)\n axs[k].imshow(img)\n axs[k].axis(\"off\")\n\nfor a in [-1, -2, -3, -4, -5]: axs[a].set_visible(False)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:50:55.944914Z","iopub.execute_input":"2022-05-25T00:50:55.945219Z","iopub.status.idle":"2022-05-25T00:51:01.926774Z","shell.execute_reply.started":"2022-05-25T00:50:55.945187Z","shell.execute_reply":"2022-05-25T00:51:01.926025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def change_color(x):\n '''Change color name.'''\n if (\"light\" in x.lower().strip()) or \\\n (\"dark\" in x.lower().strip()) or \\\n (\"greyish\" in x.lower().strip()) or \\\n (\"yellowish\" in x.lower().strip()) or \\\n (\"greenish\" in x.lower().strip()) or \\\n (\"off\" in x.lower().strip()) or \\\n (\"other\" in x.lower().strip()):\n x = x.split(\" \")[-1]\n \n return x\n\narticles_df[\"colour_group_name\"] = articles_df[\"colour_group_name\"].apply(lambda x: change_color(x))","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:51:01.928044Z","iopub.execute_input":"2022-05-25T00:51:01.928394Z","iopub.status.idle":"2022-05-25T00:51:02.033015Z","shell.execute_reply.started":"2022-05-25T00:51:01.928364Z","shell.execute_reply":"2022-05-25T00:51:02.032292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Appearance and color\nprint(clr.S+\"Total Number of unique Product Appearances:\"+clr.E, articles_df[\"graphical_appearance_name\"].nunique())\nprint(clr.S+\"Total Number of unique Product Colors (after preprocess):\"+clr.E, articles_df[\"colour_group_name\"].nunique())\n\n# --- Data 1 ---\nprod_appearance = articles_df[\"graphical_appearance_name\"].value_counts().reset_index().head(15)\ntotal_prod_appearances = articles_df[\"graphical_appearance_name\"].nunique()\nclrs1 = [\"#AF0848\" if x==max(prod_appearance[\"graphical_appearance_name\"]) else '#E90B60' for x in prod_appearance[\"graphical_appearance_name\"]]\n\n\n# Get images\nprod_appearance_images = articles_df[articles_df[\"graphical_appearance_name\"].isin(prod_appearance[\"index\"].tolist())].groupby(\"graphical_appearance_name\")[\"path\"].first().reset_index()\nimage_paths1 = prod_appearance_images[\"path\"].tolist()\nimage_names1 = prod_appearance_images[\"graphical_appearance_name\"].tolist()\n\n# --- Data 2 ---\nprod_color = articles_df[\"colour_group_name\"].value_counts().reset_index().head(15)\ntotal_prod_color = articles_df[\"colour_group_name\"].nunique()\nclrs2 = [\"#CB2170\" if x==max(prod_color[\"colour_group_name\"]) else '#954E93' for x in prod_color[\"colour_group_name\"]]\n\n# Get images\nprod_color_images = articles_df[articles_df[\"colour_group_name\"].isin(prod_color[\"index\"].tolist())].groupby(\"colour_group_name\")[\"path\"].first().reset_index()\nimage_paths2 = prod_color_images[\"path\"].tolist()\nimage_names2 = prod_color_images[\"colour_group_name\"].tolist()\n\n# Plot\nfig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(25, 13))\n\nax1.set_title('- Most Frequent Product Appearances -', size=22, weight=\"bold\")\nsns.barplot(data=prod_appearance, x=\"graphical_appearance_name\", y=\"index\", ax=ax1,\n palette=clrs2)\nx0,x1 = ax1.get_xlim()\ny0,y1 = ax1.get_ylim()\n# ax1.imshow(bk_image, zorder=0, extent=[x0, x1, y0, y1], alpha=0.35, aspect='auto')\n\nshow_values_on_bars(axs=ax1, h_v=\"h\", space=0.4)\nax1.set_ylabel(\"Product Appearance\", size = 16, weight=\"bold\")\nax1.set_xlabel(\"\")\nax1.set_xticks([])\n# ax1.set_yticks(size=16)\n# ax1.set_tick_params(size=16)\n\n# insert_image(path='../input/hm-fashion-recommender-dataset/pics/blue.jpg', zoom=0.45, xybox=(11000, 11), ax=ax1)\n\n\nax2.set_title('- Most Frequent Product Colors -', size=22, weight=\"bold\")\nsns.barplot(data=prod_color, x=\"colour_group_name\", y=\"index\", ax=ax2,\n palette=clrs2)\nx0,x1 = ax2.get_xlim()\ny0,y1 = ax2.get_ylim()\n# ax2.imshow(bk_image, zorder=0, extent=[x0, x1, y0, y1], alpha=0.35, aspect='auto')\n\nshow_values_on_bars(axs=ax2, h_v=\"h\", space=0.4)\nax2.set_ylabel(\"Product Colors\", size = 16, weight=\"bold\")\nax2.set_xlabel(\"\")\nax2.set_xticks([])\n# ax1.set_yticks(size=16)\n# ax1.set_tick_params(size=16)\n\n# insert_image(path='../input/hm-fashion-recommender-dataset/pics/blue.jpg', zoom=0.45, xybox=(11000, 11), ax=ax1)\n\nsns.despine(left=True, bottom=True)\nplt.show();\n\nprint(\"\\n\")\n\n# Plot\nfig, axs = plt.subplots(3, 5, figsize=(23, 8))\nfig.suptitle('- Example Images [Appearance] -', size=22, weight=\"bold\")\naxs = axs.flatten()\n\nfor k, (path, name) in enumerate(zip(image_paths1, image_names1)):\n axs[k].set_title(f\"{name}\", size = 16)\n img = plt.imread(path)\n axs[k].imshow(img)\n axs[k].axis(\"off\")\n\nplt.tight_layout()\nplt.show()\n\n# Plot\nfig, axs = plt.subplots(3, 5, figsize=(23, 8))\nfig.suptitle('- Example Images [Color] -', size=22, weight=\"bold\")\naxs = axs.flatten()\n\nfor k, (path, name) in enumerate(zip(image_paths2, image_names2)):\n axs[k].set_title(f\"{name}\", size = 16)\n img = plt.imread(path)\n axs[k].imshow(img)\n axs[k].axis(\"off\")\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:51:02.034407Z","iopub.execute_input":"2022-05-25T00:51:02.034625Z","iopub.status.idle":"2022-05-25T00:51:11.590977Z","shell.execute_reply.started":"2022-05-25T00:51:02.0346Z","shell.execute_reply":"2022-05-25T00:51:11.59015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Mean price by Cat\nNow check the mean price change in time for top 5 product groups by mean price:\n\n-Shoes\n\n-Garment Full body\n\n-Bags\n\n-Garment Lower body\n\n-Underwear/nightwear","metadata":{}},{"cell_type":"code","source":"time.sleep(30)","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:51:25.798379Z","iopub.execute_input":"2022-05-25T00:51:25.798644Z","iopub.status.idle":"2022-05-25T00:51:55.832316Z","shell.execute_reply.started":"2022-05-25T00:51:25.798617Z","shell.execute_reply":"2022-05-25T00:51:55.831502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_list = ['Shoes', 'Garment Full body', 'Bags', 'Garment Lower body', 'Underwear/nightwear']\ncolors = ['cadetblue', 'orange', 'mediumspringgreen', 'tomato', 'lightseagreen']\nk = 0\nf, ax = plt.subplots(3, 2, figsize=(20, 15))\nfor i in range(3):\n for j in range(2):\n try:\n product = product_list[k]\n articles_for_merge_product = df[df.product_group_name == product_list[k]]\n series_mean = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).mean().fillna(0)\n series_std = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).std().fillna(0)\n ax[i, j].plot(series_mean, linewidth=4, color=colors[k])\n ax[i, j].fill_between(series_mean.index, (series_mean.values-2*series_std.values).ravel(), \n (series_mean.values+2*series_std.values).ravel(), color=colors[k], alpha=.1)\n ax[i, j].set_title(f'Mean {product_list[k]} price in time')\n ax[i, j].set_xlabel('month')\n ax[i, j].set_xlabel(f'{product_list[k]}')\n k += 1\n except IndexError:\n ax[i, j].set_visible(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:51:55.833873Z","iopub.execute_input":"2022-05-25T00:51:55.834079Z","iopub.status.idle":"2022-05-25T00:52:10.260625Z","shell.execute_reply.started":"2022-05-25T00:51:55.834056Z","shell.execute_reply":"2022-05-25T00:52:10.260039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### SHOW MOST COMMON WORDS IN DESCRIPTION","metadata":{}},{"cell_type":"code","source":"time.sleep(30)\ndel articles_df\ndel df","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:54:50.075922Z","iopub.execute_input":"2022-05-25T00:54:50.07621Z","iopub.status.idle":"2022-05-25T00:55:21.909129Z","shell.execute_reply.started":"2022-05-25T00:54:50.07618Z","shell.execute_reply":"2022-05-25T00:55:21.908315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\npath = Path(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/\")\n\narticles_df = pd.read_csv(path / \"articles.csv\", dtype = {'article_id': str})\ntrans_df = pd.read_csv(path / \"transactions_train.csv\", dtype = {'article_id': str,'customer_id': str})\n\n#Extract only the column needed\narticles_df = articles_df[['article_id','detail_desc']]\n\ndf = pd.merge(trans_df, articles_df, on='article_id', how='left')\n# del articles_df","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:57:43.798791Z","iopub.execute_input":"2022-05-25T00:57:43.799103Z","iopub.status.idle":"2022-05-25T00:58:50.662554Z","shell.execute_reply.started":"2022-05-25T00:57:43.799073Z","shell.execute_reply":"2022-05-25T00:58:50.661725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prod_desc = articles_df[articles_df.detail_desc.notnull()].detail_desc.sample(5000).values","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:58:50.664254Z","iopub.execute_input":"2022-05-25T00:58:50.664885Z","iopub.status.idle":"2022-05-25T00:58:50.688573Z","shell.execute_reply.started":"2022-05-25T00:58:50.664836Z","shell.execute_reply":"2022-05-25T00:58:50.687782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\n\nstopwords = set(STOPWORDS) \nwordcloud = WordCloud(width = 800, \n height = 800,\n background_color ='white',\n min_font_size = 10,\n stopwords = stopwords,).generate(' '.join(prod_desc)) \n\n# plot the WordCloud image \nplt.figure(figsize = (8, 8), facecolor = None) \nplt.imshow(wordcloud) \nplt.axis(\"off\") \nplt.tight_layout(pad = 0) \n\nplt.show() ","metadata":{"execution":{"iopub.status.busy":"2022-05-25T00:59:30.630076Z","iopub.execute_input":"2022-05-25T00:59:30.630406Z","iopub.status.idle":"2022-05-25T00:59:32.824703Z","shell.execute_reply.started":"2022-05-25T00:59:30.630371Z","shell.execute_reply":"2022-05-25T00:59:32.823877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"time.sleep(30)\ndel articles_df\ndel trans_df\ndel df","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Customer Portfolio Analysis","metadata":{}},{"cell_type":"code","source":"%%time\npath = Path(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/\")\n\narticles_df = pd.read_csv(path / \"articles.csv\", dtype = {'article_id': str})\ncust_df = pd.read_csv(path / \"customers.csv\", dtype = {'customer_id': str})\ntrans_df = pd.read_csv(path / \"transactions_train.csv\", dtype = {'article_id': str,'customer_id': str})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"time.sleep(30)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Extract only the column needed\narticles_df = articles_df[['article_id', 'product_type_name','product_group_name','colour_group_name','prod_name','graphical_appearance_name']]\ncust_df = cust_df[['customer_id', 'club_member_status','fashion_news_frequency','age']]\n\n\ndf = pd.merge(trans_df, cust_df, on='customer_id', how='left')\ndel trans_df,cust_df\n\n# df = pd.merge(df, articles_df, on='article_id', how='left')\n# del articles_df\n\ntime.sleep(30)\n\nimport calendar\ndf['t_dat'] = pd.to_datetime(df['t_dat'])\ndf['YYYY_MM'] = df['t_dat'].dt.year.astype(str) + '_' + df['t_dat'].dt.month.astype(str)\ndf['year'] = df['t_dat'].dt.year\n# df['month'] = df['t_dat'].dt.month\n# df['month_'] = df['month'].apply(lambda x: calendar.month_abbr[x])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfg = df[['age','fashion_news_frequency','customer_id']]\ndfg = dfg.groupby(['age','fashion_news_frequency']).count().reset_index()\ndfg.rename(columns = {\"customer_id\": \"count\"}, inplace=True)\ndfg\nfig = px.bar(dfg, x=\"age\", y=\"count\",color='fashion_news_frequency'\n# ,markers=True\n ,color_discrete_sequence=px.colors.diverging.PRGn\n ,template = \"plotly_white\"\n ) \nfig.update_layout(\n title=\"Number of customer by age\"\n ,xaxis_title=\"Age\"\n ,yaxis_title=\"Count\"\n ,legend_title_text='fashion_news_frequency'\n)\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:31:33.065532Z","iopub.execute_input":"2022-05-25T01:31:33.066398Z","iopub.status.idle":"2022-05-25T01:31:39.589439Z","shell.execute_reply.started":"2022-05-25T01:31:33.066351Z","shell.execute_reply":"2022-05-25T01:31:39.588598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reference：　https://www.kaggle.com/code/melodyyiphoiching/h-m-deep-sales-and-customers-analysis/","metadata":{}},{"cell_type":"markdown","source":"### Which Age Group purchase more products?","metadata":{}},{"cell_type":"code","source":"df['age_groups'] = pd.cut(df['age'], bins=[16, 20, 30, 40,50, 60, 70, float('Inf')], labels=['16-20', '20-30','30-40','40-50','50-60','60-70' , '70+'])","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:32:25.262262Z","iopub.execute_input":"2022-05-25T01:32:25.262881Z","iopub.status.idle":"2022-05-25T01:32:26.040262Z","shell.execute_reply.started":"2022-05-25T01:32:25.262829Z","shell.execute_reply":"2022-05-25T01:32:26.039554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8,5))\nplt.title(\"Purchased quantity by age group\\n\", fontweight=\"bold\", size=28)\ng = sns.barplot(x=\"age_groups\", y=\"Purchased Quantity(%)\", data=df.groupby(\"age_groups\")[\"article_id\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('Purchased Quantity(%)').reset_index(), palette=\"icefire\", edgecolor=\"black\")\nplt.xlabel(\"Age Group\",fontweight=\"bold\", size=22)\nplt.ylabel(\"Purchased Quantity (%)\",fontweight=\"bold\", size=19)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=18, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-25T01:32:43.891266Z","iopub.execute_input":"2022-05-25T01:32:43.892011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights:\n\n- Customers in the range 20-30 are responsible for more than 42% of the total purchased products.\n- Customers in the range 16-20. 60-70 and 70+ are responsible for the 8% of the total purchased products\n- Customers in the range 30-40, 40-50 and 50-60 are responsible for 16% of purchased quantity each.\n\nAfter analyzing the purchases quantity, it could be interesting to analyze the earnings provided to the company by each customer.","metadata":{}},{"cell_type":"markdown","source":"### Which Age Group generate more earnings for the company?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(8,5))\nplt.title(\"Company Earnings by age group\\n\", fontweight=\"bold\", size=28)\ng = sns.barplot(x=\"age_groups\", y=\"earning(%)\", data=df.groupby(\"age_groups\")[\"price\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index(), palette=\"icefire\",edgecolor=\"black\")\nplt.xlabel(\"Age Group\",fontweight=\"bold\", size=22)\nplt.ylabel(\"Earnings (%)\",fontweight=\"bold\", size=25)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=18, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed a very similar situation to the purchases quantity can be found in the earnings analysis, since customers who buys more, on average leads to higher earnings for the company.\nThe age group 20-30 is by far responsible for the highest earnings for the company (41.9% of total earnings).","metadata":{}},{"cell_type":"markdown","source":"### Do active customers on the fashion news purchase more products?","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(9,5))\nplt.title(\"Purchased quantity by Fashion News Frequency\\n\", fontweight=\"bold\", size=20)\ng = sns.barplot(x=\"fashion_news_frequency\", y=\"Purchased Quantity(%)\", data=df.groupby(\"fashion_news_frequency\")[\"article_id\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('Purchased Quantity(%)').reset_index(), palette=\"Spectral\", edgecolor=\"black\")\nplt.xlabel(\"Fashion News Frequency\",fontweight=\"bold\", size=22)\nplt.ylabel(\"Purchased Quantity (%)\",fontweight=\"bold\", size=25)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.3f', fontsize=18, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Active customers on the fashion news are responsible for 43% of the total purchases, while the remaining 57% of purchased quantity comes from customer not registed in the fashion news.\nThe other 2 categories \"Monthly\" and \"None\" can be ignored and won't be considered for the further analysis.\n\nSo then it could be interesting to check the fashion news frequency by age group, to find more useful insights,","metadata":{}},{"cell_type":"code","source":"x, y = 'age_groups', 'fashion_news_frequency'\ndf_age_news = df.groupby(x)[y].value_counts(normalize=True)\ndf_age_news = df_age_news.mul(100)\ndf_age_news = df_age_news.rename('percent(%)').reset_index()\ndf_age_news = df_age_news[df_age_news[\"fashion_news_frequency\"].isin([\"Regularly\",\"NONE\"])]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"palette1 = {\"Regularly\":'#46C646', \"NONE\":'#FF0000'}\n\nplt.figure(figsize=(13,6))\nplt.title(\"Fashion News Frequency by age group\\n\",fontweight=\"bold\", size=33)\ng=sns.barplot(x=\"age_groups\", y=\"percent(%)\",data=df_age_news, hue=\"fashion_news_frequency\", palette=palette1)\nplt.xlabel(\"Age group\",fontweight=\"bold\", size=22)\nplt.ylabel(\"Percentage (%)\",fontweight=\"bold\", size=25)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=16, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.legend(title='News\\nFrequency',bbox_to_anchor=(1.0, 1.0), ncol=1, fancybox=True, shadow=True, fontsize=17,title_fontsize=22)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that customers in the range 20-30 and 30-40 have the lowest percentage of fashion news frequency, while being the groups which buy the most.\nMoreover, the frequency of customer that regulary check fashion news starts increasing from the range 40-50, with a peak value of 43.7% of regular/active users for customers in the range 70+ years old. This means that checking fashion news seems to be more effective for older customers, who still represent a small percentage of total sold products, while younger customers do not need to check the news to buy new products.\nIt could be effective for the company to invite younger customers (range 20-40) to check the news more frequently in order to increase the sold items.","metadata":{}},{"cell_type":"markdown","source":"### Does the club member status influence the purchased quantity?","metadata":{}},{"cell_type":"code","source":"df[\"club_member_status\"].value_counts(normalize=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that:\n\n- More than 93% of the customers belong to the ACTIVE category\n- 6.8% of the customers belong to the PRE-CREATE cateory\n- 0.3% of the customers belong to the LEFT CLUB category\n\nThis shows a very high imbalance among the classes: if we consider the sum of purchased products per each category, this will likely show that the most part of Purchased products belongs to the ACTIVE members.","metadata":{}},{"cell_type":"code","source":"df.groupby(\"club_member_status\")[\"article_id\"].sum()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, more customers in a group leads to higher purchases. For this reason, it is more wise to consider a mean Purchased quantity instead of a sum:","metadata":{}},{"cell_type":"code","source":"print(\"The average quantity of purchased products by the customers is {:.0f} products \".format(df[\"article_id\"].mean()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"The average quantity of purchased products by the ACTIVE customers is {:.0f} products \".format(df.groupby(\"club_member_status\")[\"article_id\"].mean()[\"ACTIVE\"]))\nprint(\"The average quantity of purchased products by the LEFT-CLUB customers is {:.0f} products \".format(df.groupby(\"club_member_status\")[\"article_id\"].mean()[\"LEFT CLUB\"]))\nprint(\"The average quantity of purchased products by the PRE-CREATE customers is {:.0f} products \".format(df.groupby(\"club_member_status\")[\"article_id\"].mean()[\"PRE-CREATE\"]))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By considering the mean, we can see a very different situation, which will be shown as percentages in the following plot:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(9,5))\nplt.title(\"Average Purchased Quantity by Club Member Status\\n\", fontweight=\"bold\", size=22)\ng = sns.barplot(x=\"club_member_status\", y=\"article_id\", data=df.groupby(\"club_member_status\")[\"article_id\"].mean().astype(int).reset_index(), palette=\"viridis\", edgecolor=\"black\")\nplt.axhline(y = cust_details[\"article_id\"].mean(), color = 'r', linestyle = '--')\nplt.text(0.76, 23.7, 'Mean Purchased Quantity: {:.0f}'.format(df[\"article_id\"].mean()), size=16, color=\"red\",fontweight=\"bold\")\nplt.xlabel(\"Club Member Status\",fontweight=\"bold\", size=20)\nplt.ylabel(\"Average Purchased Quantity\",fontweight=\"bold\", size=16)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.0f', fontsize=23, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This plots shows that the average purchased quantity differs a lot among the categories.\nIn particular, customers belonging to the ACTIVE clubs, purchase more products than other categories, while those in the \"pre-create\" category purchaes on average less than a third of third of active customers.\n\nFinally, since the distribution of the purchased quantity is heavily right skewed, it could be interesting to check out also the median purhcased quantity.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(9,5))\nplt.title(\"Median Purchased Quantity by Club Member Status\\n\", fontweight=\"bold\", size=22)\ng = sns.barplot(x=\"club_member_status\", y=\"article_id\", data=df.groupby(\"club_member_status\")[\"article_id\"].median().reset_index(), palette=\"viridis\", edgecolor=\"black\")\nplt.axhline(y = cust_details[\"article_id\"].median(), color = 'r', linestyle = '--')\nplt.text(0.76, 9.3, 'Median Purchased Quantity: {:.2f}'.format(df[\"article_id\"].median()), size=16, color=\"red\",fontweight=\"bold\")\nplt.xlabel(\"Club Member Status\",fontweight=\"bold\", size=20)\nplt.ylabel(\"Median Purchaed Quantity\",fontweight=\"bold\", size=16)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.0f', fontsize=23, color=\"black\")\nplt.grid(axis=\"y\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, even if the Median is quite different for the Mean due to high skeweness of the data, a very similar situation situation to the mean purchases quantity can be observed, where ACTIVE customers buys more product on average.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"work in progress.\nIf you find my work impressive 💖 and useful 👍🏻, Please Upvote it.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}