{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

⬆️⬆️⬆️ If you find this note book helpful. please upvote! ⬆️⬆️⬆️

","metadata":{}},{"cell_type":"markdown","source":"

H&M Exploratory Data Analysis

","metadata":{}},{"cell_type":"markdown","source":"

\n\n

","metadata":{}},{"cell_type":"markdown","source":"# Introduction to the Problem statement...\n* H&M Group is a family of brands and businesses with 53 online markets and approximately 4,850 stores. Our online store offers shoppers an extensive selection of products to browse through. But with too many choices, customers might not quickly find what interests them or what they are looking for, and ultimately, they might not make a purchase. To enhance the shopping experience, product recommendations are key. More importantly, helping customers make the right choices also has a positive implications for sustainability, as it reduces returns, and thereby minimizes emissions from transportation.\n\n* In this competition, H&M Group invites you to develop product recommendations based on data from previous transactions, as well as from customer and product meta data. The available meta data spans from simple data, such as garment type and customer age, to text data from product descriptions, to image data from garment images.\n\n* There are no preconceptions on what information that may be useful – that is for you to find out. If you want to investigate a categorical data type algorithm, or dive into NLP and image processing deep learning, that is up to you.","metadata":{}},{"cell_type":"markdown","source":"# Data Description\n* For this challenge you are given the purchase history of customers across time, along with supporting metadata. Your challenge is to predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customer who did not make any purchase during that time are excluded from the scoring.\n\n# Files Includes\n* images - a folder of images corresponding to each article_id; images are placed in subfolders starting with the first three digits of the article_id; note, not all article_id values have a corresponding image.\n* articles.csv - detailed metadata for each article_id available for purchase\n* customers.csv - metadata for each customer_id in dataset\n* sample_submission.csv - a sample submission file in the correct format\n* transactions_train.csv - the training data, consisting of the purchases each customer for each date, as well as additional information. Duplicate rows correspond to multiple purchases of the same item. Your task is to predict the article_ids each customer will purchase during the 7-day period immediately after the training data period.\n* NOTE: You must make predictions for all customer_id values found in the sample submission. All customers who made purchases during the test period are scored, regardless of whether they had purchase history in the training data.","metadata":{}},{"cell_type":"markdown","source":"## import libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nfrom pandasql import sqldf\nfrom matplotlib_venn import venn2\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.style.use('seaborn-white')\nsns.set_style(\"whitegrid\")\nsns.despine()\nplt.rc(\"figure\", autolayout=True)\nplt.rc(\"axes\", labelweight=\"bold\", labelsize=\"large\", titleweight=\"bold\", titlesize=14, titlepad=10)\n\nimport matplotlib as mpl\n\nmpl.rcParams['axes.spines.left'] = False\nmpl.rcParams['axes.spines.right'] = False\nmpl.rcParams['axes.spines.top'] = False\nmpl.rcParams['axes.spines.bottom'] = False\nplt.rcParams[\"font.weight\"] = \"bold\"\nplt.rcParams[\"axes.labelweight\"] = \"bold\"","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:06:18.476826Z","iopub.execute_input":"2022-05-16T09:06:18.477202Z","iopub.status.idle":"2022-05-16T09:06:19.815153Z","shell.execute_reply.started":"2022-05-16T09:06:18.477086Z","shell.execute_reply":"2022-05-16T09:06:19.813202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data import","metadata":{}},{"cell_type":"code","source":"df_a = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ndf_t = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")\ndf_c = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/customers.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:06:19.817748Z","iopub.execute_input":"2022-05-16T09:06:19.818120Z","iopub.status.idle":"2022-05-16T09:07:33.595537Z","shell.execute_reply.started":"2022-05-16T09:06:19.818072Z","shell.execute_reply":"2022-05-16T09:07:33.594561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_t.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:33.596743Z","iopub.execute_input":"2022-05-16T09:07:33.596966Z","iopub.status.idle":"2022-05-16T09:07:33.606443Z","shell.execute_reply.started":"2022-05-16T09:07:33.596940Z","shell.execute_reply":"2022-05-16T09:07:33.605387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 3.Data Preparation\n","metadata":{}},{"cell_type":"markdown","source":"## Articles dataframe ","metadata":{}},{"cell_type":"code","source":"print(f\"The dataframe articles has {len(df_a)} rows\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:33.608801Z","iopub.execute_input":"2022-05-16T09:07:33.609070Z","iopub.status.idle":"2022-05-16T09:07:33.630331Z","shell.execute_reply.started":"2022-05-16T09:07:33.609036Z","shell.execute_reply":"2022-05-16T09:07:33.629171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_a.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:33.634762Z","iopub.execute_input":"2022-05-16T09:07:33.634974Z","iopub.status.idle":"2022-05-16T09:07:33.789045Z","shell.execute_reply.started":"2022-05-16T09:07:33.634948Z","shell.execute_reply":"2022-05-16T09:07:33.788186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The \"articles\" dataframe has 25 columns and more than 100k rows.
\nFor our our analysis we will just select the following columns since the names of columns corresponding with another ones carry names-codes such as : \n\n- product_type_no ----------> product_type_name \n- graphical_appearance_no ----------> graphical_appearance_name \n- colour_group_code---------->colour_group_name \n- perceived_colour_value_id ---------->perceived_colour_value_name \n- perceived_colour_master_id----------> perceived_colour_master_name \n- department_no ---------->department_name \n- index_code----------> index_name \n- index_group_no ----------> index_group_name \n- section_no---------->section_name \n- garment_group_no---------->garment_group_name \n\n\nfor that reason we will select only the following rows : \n- article_id\n- prod_name\n- product_type_name\n- product_group_name\n- colour_group_name\n- index_name\n- index_group_name\n- section_name \n- garment_group_name \n- graphical_appearance_name\n- perceived_colour_value_name\n- perceived_colour_master_name \n- department_name\n\nBy considering only these columns we can also save lots of memory.","metadata":{}},{"cell_type":"markdown","source":"To filter the columns, we will use SQL like code through SQL-DF library.","metadata":{}},{"cell_type":"code","source":"df_a = sqldf(\"\"\"SELECT article_id\n,prod_name\n, product_type_name\n,product_group_name\n, colour_group_name\n,index_name\n, index_group_name\n, section_name \n, garment_group_name \n,graphical_appearance_name\n, perceived_colour_value_name\n,perceived_colour_master_name \n, department_name\n FROM df_a\n \"\"\")\ndf_a","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:33.790299Z","iopub.execute_input":"2022-05-16T09:07:33.790882Z","iopub.status.idle":"2022-05-16T09:07:39.248006Z","shell.execute_reply.started":"2022-05-16T09:07:33.790848Z","shell.execute_reply":"2022-05-16T09:07:39.247108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**first i m going to use InteractiveShell function in order to display mutiple outputs in the same cell, i think it will be more readable and understandable for the code**","metadata":{}},{"cell_type":"code","source":"from IPython.core.interactiveshell import InteractiveShell \nInteractiveShell.ast_node_interactivity = \"all\"","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.249552Z","iopub.execute_input":"2022-05-16T09:07:39.250097Z","iopub.status.idle":"2022-05-16T09:07:39.254961Z","shell.execute_reply.started":"2022-05-16T09:07:39.250037Z","shell.execute_reply":"2022-05-16T09:07:39.253666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_a.head()\ndf_a.columns","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.256930Z","iopub.execute_input":"2022-05-16T09:07:39.257439Z","iopub.status.idle":"2022-05-16T09:07:39.293400Z","shell.execute_reply.started":"2022-05-16T09:07:39.257391Z","shell.execute_reply":"2022-05-16T09:07:39.292049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\na = df_a.isna().values\ntype(a)\n# go from ndarrays to map object then to pandas series in order to apply value_counts function \na = map(lambda x: x[0], a)\ntype(a)\na = pd.Series(a)\ntype(a)\n\na.value_counts()\n\n# to check for null values , go with the same steps above \n#a = df_a.isnull().values\n\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.296470Z","iopub.execute_input":"2022-05-16T09:07:39.296781Z","iopub.status.idle":"2022-05-16T09:07:39.504741Z","shell.execute_reply.started":"2022-05-16T09:07:39.296751Z","shell.execute_reply":"2022-05-16T09:07:39.503759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- ***since we have only the boolen False as output from value_counts function which applied on checking null values in our dataframe , that 's mean that there is no null values***\n- ***same as is_na check gunction since we have only the boolen False as output from value_counts function which applied on checking null values in our dataframe , that 's mean that there is no NaN values***","metadata":{}},{"cell_type":"markdown","source":"**next step is to show common values between columns , that's would help us in the next aggregation**","metadata":{}},{"cell_type":"code","source":"\na = df_a.index_name.unique()\nb = df_a.index_group_name.unique()\nvenn2([set(df_a['index_name'].to_list()), \n set(df_a['index_group_name'].to_list()) \n ],\n set_labels=('index_name', 'index_group_name'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.506305Z","iopub.execute_input":"2022-05-16T09:07:39.506663Z","iopub.status.idle":"2022-05-16T09:07:39.812313Z","shell.execute_reply.started":"2022-05-16T09:07:39.506574Z","shell.execute_reply":"2022-05-16T09:07:39.811198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"as plotted above there is a relation between index_name and index_group_name, seems that a index_group_name is a subcolumn for index_name \n- similarily we do the same thing for next couple of columns to discover more ","metadata":{}},{"cell_type":"code","source":"\nvenn2([set(df_a['garment_group_name'].to_list()), \n set(df_a['graphical_appearance_name'].to_list()) \n ],\n set_labels=('garment_group_name', 'graphical_appearance_name'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.820029Z","iopub.execute_input":"2022-05-16T09:07:39.820416Z","iopub.status.idle":"2022-05-16T09:07:39.993789Z","shell.execute_reply.started":"2022-05-16T09:07:39.820374Z","shell.execute_reply":"2022-05-16T09:07:39.992685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"venn2([set(df_a['perceived_colour_value_name'].to_list()), \n set(df_a['perceived_colour_master_name'].to_list()) \n ],\n set_labels=('garment_group_name', 'graphical_appearance_name'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:39.995719Z","iopub.execute_input":"2022-05-16T09:07:39.996383Z","iopub.status.idle":"2022-05-16T09:07:40.178003Z","shell.execute_reply.started":"2022-05-16T09:07:39.996326Z","shell.execute_reply":"2022-05-16T09:07:40.177044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_a.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:40.179820Z","iopub.execute_input":"2022-05-16T09:07:40.180425Z","iopub.status.idle":"2022-05-16T09:07:40.207592Z","shell.execute_reply.started":"2022-05-16T09:07:40.180376Z","shell.execute_reply":"2022-05-16T09:07:40.206319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"venn2([set(df_a['department_name'].to_list()), \n set(df_a['garment_group_name'].to_list()) \n ],\n set_labels=('department_name', 'garment_group_name'))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:40.209604Z","iopub.execute_input":"2022-05-16T09:07:40.210246Z","iopub.status.idle":"2022-05-16T09:07:40.382522Z","shell.execute_reply.started":"2022-05-16T09:07:40.210193Z","shell.execute_reply":"2022-05-16T09:07:40.381196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**interestingly from plotting some couple of columns above , we notice two sets that would give us some inspirations after aggregations which are :** \n- 'department_name', 'garment_group_name' \n- 'index_name', 'index_group_name' ","metadata":{}},{"cell_type":"markdown","source":"\n# 4.df_articles Analysis","metadata":{}},{"cell_type":"code","source":"df_a.head(2)\nprod_name =df_a[\"prod_name\"].value_counts() \np = pd.DataFrame(prod_name)\np.rename(columns = {\"index\":\"prod_name\",\"prod_name\":\"qty\"}, inplace=True)\np = p.sort_values(by='qty', ascending=False)[:10]\np\n\nplot = p.plot.pie(y='qty', figsize=(9,9))\n\n\n\n\n\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:40.384927Z","iopub.execute_input":"2022-05-16T09:07:40.385344Z","iopub.status.idle":"2022-05-16T09:07:41.021157Z","shell.execute_reply.started":"2022-05-16T09:07:40.385299Z","shell.execute_reply":"2022-05-16T09:07:41.020228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**we notice that there is an equivalent dividend for the top 10 products existing in the HM repositories , further we will discuss if these top 10 are deserved to be on the top of repositories or not by discussing number of sells after**","metadata":{}},{"cell_type":"code","source":"df_a.head(2)\nprod_name =df_a[\"product_type_name\"].value_counts() \np = pd.DataFrame(prod_name)\np.rename(columns = {\"index\":\"prod_name\",\"product_type_name\":\"qty\"}, inplace=True)\np = p.sort_values(by='qty', ascending=False)[:10]\np\n\nplot = p.plot.pie(y='qty', figsize=(9,9))\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.022478Z","iopub.execute_input":"2022-05-16T09:07:41.022732Z","iopub.status.idle":"2022-05-16T09:07:41.558667Z","shell.execute_reply.started":"2022-05-16T09:07:41.022701Z","shell.execute_reply":"2022-05-16T09:07:41.557265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Trousers, Dress,Sweater are taking the half of the top 10 product_type_name \n- let's conserve those numbers in order to see if the HM making the right decision to store more than a half in the top10 ","metadata":{}},{"cell_type":"markdown","source":"**let's automate a little bit the operation to get more general view** ","metadata":{}},{"cell_type":"code","source":"df_a.head(2)\n\n# define a function to plot top 10 values from each column(variable) \ndef top_ten(variable) : \n\n top_ten_df =df_a[variable].value_counts() \n top_ten_df\n p = pd.DataFrame(top_ten_df)\n \n p.rename(columns = {\"index\":variable,variable:\"qty\"}, inplace=True)\n p = p.sort_values(by='qty', ascending=False)[:10]\n \n p.plot(kind='barh',y=\"qty\") \n \n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.560258Z","iopub.execute_input":"2022-05-16T09:07:41.560617Z","iopub.status.idle":"2022-05-16T09:07:41.583751Z","shell.execute_reply.started":"2022-05-16T09:07:41.560567Z","shell.execute_reply":"2022-05-16T09:07:41.582489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# a = df_a.columns\n# # note : we pass the article_id variable \n# for x in a : \n# if x == \"article_id\" : \n# continue \n# print(x)\n# top_ten(x)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.585399Z","iopub.execute_input":"2022-05-16T09:07:41.585711Z","iopub.status.idle":"2022-05-16T09:07:41.596889Z","shell.execute_reply.started":"2022-05-16T09:07:41.585682Z","shell.execute_reply":"2022-05-16T09:07:41.596149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**note : i came to comment some cells because of the leak memory , i m working to find a solution n it is a large data size especially for transaction , till then you can uncomment and test the code separately**","metadata":{}},{"cell_type":"code","source":"# def top_three(variable) : \n\n# top_ten_df =df_a[variable].value_counts() \n# top_ten_df\n# p = pd.DataFrame(top_ten_df)\n \n# p.rename(columns = {\"index\":variable,variable:\"qty\"}, inplace=True)\n# p = p.sort_values(by='qty', ascending=False)[:10]\n# return p.index[:3].tolist()\n# top_three(\"prod_name\")\n# for x in a : \n# if x == \"article_id\" : \n# continue \n# print(f\"the head repository list of {x} we have {top_three(x)}\")\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.598216Z","iopub.execute_input":"2022-05-16T09:07:41.598560Z","iopub.status.idle":"2022-05-16T09:07:41.612605Z","shell.execute_reply.started":"2022-05-16T09:07:41.598531Z","shell.execute_reply":"2022-05-16T09:07:41.611819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***after checking what we have in our HM repositories , let's make a check if that compatible with our sales from the transaction table***\n\n- to do that we have to merge between dataframes first , clean new data and discover some trends \n- let's check for NAN and null values in transactions DataFrame ","metadata":{}},{"cell_type":"code","source":"\n# a = df_t.isna().values\n# type(a)\n# # go from ndarrays to map object then to pandas series in order to apply value_counts function \n# a = map(lambda x: x[0], a)\n# type(a)\n# a = pd.Series(a)\n# type(a)\n# a.value_counts()\n# # to check for null values , go with the same steps above \n# #a = df_a.isnull().values\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.613770Z","iopub.execute_input":"2022-05-16T09:07:41.614138Z","iopub.status.idle":"2022-05-16T09:07:41.633620Z","shell.execute_reply.started":"2022-05-16T09:07:41.614092Z","shell.execute_reply":"2022-05-16T09:07:41.632373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***since we have only false as an output from value_counts() function , this means there is no NAN and the same for NULL values***","metadata":{}},{"cell_type":"code","source":"# merge articles with transactions \n# drop customer_id\tprice\tsales_channel_id columns to avoid memory satisfaction since we won't use them \n# in our further analysis \n\ndf_a.head(2)\ndf_t.head(2)\ndf_t = df_t.drop(['customer_id','sales_channel_id','price'],axis=1)\n\nart_trans_merged = df_a.merge(df_t, how='inner',on=\"article_id\" )\nart_trans_merged.head(2)\n\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:41.635421Z","iopub.execute_input":"2022-05-16T09:07:41.636778Z","iopub.status.idle":"2022-05-16T09:07:55.010435Z","shell.execute_reply.started":"2022-05-16T09:07:41.636716Z","shell.execute_reply":"2022-05-16T09:07:55.009784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# using the garbage collection to avoid memory saturation \nimport gc\ndel df_t\ndel df_a\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:55.012063Z","iopub.execute_input":"2022-05-16T09:07:55.012928Z","iopub.status.idle":"2022-05-16T09:07:55.228339Z","shell.execute_reply.started":"2022-05-16T09:07:55.012866Z","shell.execute_reply":"2022-05-16T09:07:55.227250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # rearranging columns for more readable dataframe \n# art_trans_merged.head()\n# cols = art_trans_merged.columns.tolist()\n# cols = cols[-1:] + cols[:-1]\n\n# art_trans_merged = art_trans_merged.reindex(columns=cols)\n# art_trans_merged.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:55.229839Z","iopub.execute_input":"2022-05-16T09:07:55.230173Z","iopub.status.idle":"2022-05-16T09:07:55.236077Z","shell.execute_reply.started":"2022-05-16T09:07:55.230111Z","shell.execute_reply":"2022-05-16T09:07:55.234642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**let's convert t_dat to a date_time object for more flexibility**","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:55.238764Z","iopub.execute_input":"2022-05-16T09:07:55.239511Z","iopub.status.idle":"2022-05-16T09:07:55.884745Z","shell.execute_reply.started":"2022-05-16T09:07:55.239460Z","shell.execute_reply":"2022-05-16T09:07:55.883878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"art_trans_merged.info()\nart_trans_merged['t_dat'] = pd.to_datetime(art_trans_merged['t_dat'],format='%Y-%m-%d')\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:55.885804Z","iopub.execute_input":"2022-05-16T09:07:55.887483Z","iopub.status.idle":"2022-05-16T09:07:59.950443Z","shell.execute_reply.started":"2022-05-16T09:07:55.887415Z","shell.execute_reply":"2022-05-16T09:07:59.949516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nart_trans_merged.dtypes\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:07:59.951788Z","iopub.execute_input":"2022-05-16T09:07:59.952026Z","iopub.status.idle":"2022-05-16T09:08:00.094895Z","shell.execute_reply.started":"2022-05-16T09:07:59.951997Z","shell.execute_reply":"2022-05-16T09:08:00.093983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# adding day of weeks \n# art_trans_merged['DayOfWeek'] = art_trans_merged['t_dat'].dt.day_name()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:00.100061Z","iopub.execute_input":"2022-05-16T09:08:00.100370Z","iopub.status.idle":"2022-05-16T09:08:00.108095Z","shell.execute_reply.started":"2022-05-16T09:08:00.100338Z","shell.execute_reply":"2022-05-16T09:08:00.107012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()\nart_trans_merged.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:09:56.755030Z","iopub.execute_input":"2022-05-16T09:09:56.755356Z","iopub.status.idle":"2022-05-16T09:09:56.898332Z","shell.execute_reply.started":"2022-05-16T09:09:56.755320Z","shell.execute_reply":"2022-05-16T09:09:56.897692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"art_trans_merged.set_index('t_dat', inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:00.254469Z","iopub.execute_input":"2022-05-16T09:08:00.254667Z","iopub.status.idle":"2022-05-16T09:08:00.260772Z","shell.execute_reply.started":"2022-05-16T09:08:00.254644Z","shell.execute_reply":"2022-05-16T09:08:00.259619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nart_trans_merged.index.min()\nart_trans_merged.index.max()\nart_trans_merged.index.max() - art_trans_merged.index.min()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:00.261987Z","iopub.execute_input":"2022-05-16T09:08:00.262400Z","iopub.status.idle":"2022-05-16T09:08:00.448801Z","shell.execute_reply.started":"2022-05-16T09:08:00.262364Z","shell.execute_reply":"2022-05-16T09:08:00.448165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***what are the top 5 prod_names sold for 2019 and 2020?***","metadata":{}},{"cell_type":"code","source":"art_trans_merged.head(2)\ntop_prod_name_2019 =art_trans_merged['2018-09-20':'2019-09-20']['prod_name'].value_counts()[:5].to_frame()\ntop_prod_name_2020 =art_trans_merged['2019-09-21':'2020-09-22']['prod_name'].value_counts()[:5].to_frame()\n\n\n\npd.concat({'2019':top_prod_name_2019, '2020':top_prod_name_2020}, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:00.449827Z","iopub.execute_input":"2022-05-16T09:08:00.450158Z","iopub.status.idle":"2022-05-16T09:08:09.064904Z","shell.execute_reply.started":"2022-05-16T09:08:00.450118Z","shell.execute_reply":"2022-05-16T09:08:09.062609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**interesting almost the same prod_name for each year , let's check for other variables**","metadata":{}},{"cell_type":"code","source":"art_trans_merged.head(2)\ntop_prod_name_2019 =art_trans_merged['2018-09-20':'2019-09-20']['department_name'].value_counts()[:5].to_frame()\ntop_prod_name_2020 =art_trans_merged['2019-09-21':'2020-09-22']['department_name'].value_counts()[:5].to_frame()\n\n\n\npd.concat({'2019':top_prod_name_2019, '2020':top_prod_name_2020}, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:09.066895Z","iopub.execute_input":"2022-05-16T09:08:09.067226Z","iopub.status.idle":"2022-05-16T09:08:17.673907Z","shell.execute_reply.started":"2022-05-16T09:08:09.067181Z","shell.execute_reply":"2022-05-16T09:08:17.672627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**as proved above the HM sales continue to maintain top 5 sold items from department_name** ","metadata":{}},{"cell_type":"code","source":"art_trans_merged.head(2)\ntop_prod_name_2019 =art_trans_merged['2018-09-20':'2019-09-20']['colour_group_name'].value_counts()[:5].to_frame()\ntop_prod_name_2020 =art_trans_merged['2019-09-21':'2020-09-22']['colour_group_name'].value_counts()[:5].to_frame()\n\npd.concat({'2019':top_prod_name_2019, '2020':top_prod_name_2020}, axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:08:17.675540Z","iopub.execute_input":"2022-05-16T09:08:17.675840Z","iopub.status.idle":"2022-05-16T09:08:26.268747Z","shell.execute_reply.started":"2022-05-16T09:08:17.675808Z","shell.execute_reply":"2022-05-16T09:08:26.267867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"art_trans_merged.head(2)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:10:46.507562Z","iopub.execute_input":"2022-05-16T09:10:46.508042Z","iopub.status.idle":"2022-05-16T09:10:46.523927Z","shell.execute_reply.started":"2022-05-16T09:10:46.508010Z","shell.execute_reply":"2022-05-16T09:10:46.522564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**it seems that sales have the same behaviour for each year ...even for colors** \n","metadata":{}},{"cell_type":"code","source":"df_t = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:24:42.018685Z","iopub.execute_input":"2022-05-16T09:24:42.019257Z","iopub.status.idle":"2022-05-16T09:25:41.621169Z","shell.execute_reply.started":"2022-05-16T09:24:42.019220Z","shell.execute_reply":"2022-05-16T09:25:41.620235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_a = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv\")\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:31:26.599860Z","iopub.execute_input":"2022-05-16T09:31:26.600149Z","iopub.status.idle":"2022-05-16T09:31:27.659104Z","shell.execute_reply.started":"2022-05-16T09:31:26.600100Z","shell.execute_reply":"2022-05-16T09:31:27.658093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices = df_t[[\"price\",\"article_id\"]].groupby(\"article_id\").sum().sort_values(by=\"price\", ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:27:40.899978Z","iopub.execute_input":"2022-05-16T09:27:40.900278Z","iopub.status.idle":"2022-05-16T09:27:42.239355Z","shell.execute_reply.started":"2022-05-16T09:27:40.900249Z","shell.execute_reply":"2022-05-16T09:27:42.238489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices.rename(columns={\"price\":\"earning\"}, inplace=True)\ndf_prices = df_prices.reset_index()\ndf_prices.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:27:49.642944Z","iopub.execute_input":"2022-05-16T09:27:49.643836Z","iopub.status.idle":"2022-05-16T09:27:49.655634Z","shell.execute_reply.started":"2022-05-16T09:27:49.643773Z","shell.execute_reply":"2022-05-16T09:27:49.654961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of different sold articles:\",len(df_prices[\"earning\"]))\nprint(\"Total Earnings:\",df_prices[\"earning\"].sum())","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:28:34.567693Z","iopub.execute_input":"2022-05-16T09:28:34.568003Z","iopub.status.idle":"2022-05-16T09:28:34.575373Z","shell.execute_reply.started":"2022-05-16T09:28:34.567973Z","shell.execute_reply":"2022-05-16T09:28:34.574655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in [10,50,100,200,300,400,1000]:\n print(\"The TOP {} of products that generate most earnings, account for the {:.2f} % of total earnings\".format(i, df_prices[\"earning\"].iloc[:i].sum() / df_prices[\"earning\"].iloc[:].sum() * 100) ) ","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:29:21.512935Z","iopub.execute_input":"2022-05-16T09:29:21.513229Z","iopub.status.idle":"2022-05-16T09:29:21.526509Z","shell.execute_reply.started":"2022-05-16T09:29:21.513198Z","shell.execute_reply":"2022-05-16T09:29:21.525544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The TOP 100 of over 100000 products, generates around 5% of the total earnings. It can be interesting to check these products names and characteristics.**","metadata":{}},{"cell_type":"code","source":"top_100_prices=df_prices.iloc[:100]","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:30:12.971354Z","iopub.execute_input":"2022-05-16T09:30:12.971663Z","iopub.status.idle":"2022-05-16T09:30:12.977062Z","shell.execute_reply.started":"2022-05-16T09:30:12.971634Z","shell.execute_reply":"2022-05-16T09:30:12.976109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_100_price_details = sqldf(\"\"\"SELECT *\n FROM top_100_prices t\n INNER JOIN df_a a\n on t.article_id = a.article_id\"\"\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:31:35.029812Z","iopub.execute_input":"2022-05-16T09:31:35.030109Z","iopub.status.idle":"2022-05-16T09:31:39.740037Z","shell.execute_reply.started":"2022-05-16T09:31:35.030075Z","shell.execute_reply":"2022-05-16T09:31:39.739135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_100_price_details.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:34:48.960709Z","iopub.execute_input":"2022-05-16T09:34:48.961429Z","iopub.status.idle":"2022-05-16T09:34:48.986319Z","shell.execute_reply.started":"2022-05-16T09:34:48.961375Z","shell.execute_reply":"2022-05-16T09:34:48.985459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,11))\nplt.title(\"TOP 50 most profitable products\", size=40, fontweight=\"bold\")\nno=50\ng = sns.barplot(y=\"prod_name\", x=\"earning(%)\", data=top_100_price_details.iloc[:no].groupby(\"prod_name\")[\"earning\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index().sort_values(by=\"earning(%)\", ascending=False), \\\n palette=\"mako\", ci=False)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=15)\nplt.xlabel(\"Earnings (%)\", size=25, fontweight=\"bold\")\nplt.ylabel(\"\")\nplt.grid(axis=\"x\",color = 'grey', linestyle = '--', linewidth = 1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:43:08.612359Z","iopub.execute_input":"2022-05-16T09:43:08.612638Z","iopub.status.idle":"2022-05-16T09:43:09.462608Z","shell.execute_reply.started":"2022-05-16T09:43:08.612609Z","shell.execute_reply":"2022-05-16T09:43:09.461639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2, figsize=(13,9))\nplt.suptitle(\"TOP 100 most profitable products characteristics\", fontweight=\"bold\", fontsize=30)\n\nno=100\n\ng = sns.barplot(y=\"product_type_name\", x=\"earning(%)\", data=top_100_price_details.iloc[:no].groupby(\"product_type_name\")[\"earning\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index().sort_values(by=\"earning(%)\", ascending=False), \\\n ax=ax[0,0],palette=\"Blues_r\", ci=False)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=14, color=\"black\")\nax[0,0].set_ylabel(\"\")\nax[0,0].set_xlabel(\"Earnings (%)\", size=20,fontweight=\"bold\")\nax[0,0].set_title(\"Product Type\", size=25,fontweight=\"bold\")\nax[0,0].grid(axis=\"x\",color = 'grey', linestyle = '--', linewidth = 1.5)\n\n\ng = sns.barplot(y=\"index_name\", x=\"earning(%)\", data=top_100_price_details.iloc[:no].groupby(\"index_name\")[\"earning\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index().sort_values(by=\"earning(%)\", ascending=False), \\\n ax=ax[0,1],palette=\"viridis\", ci=False)\nfor container in g.containers:\n g.bar_label(container, fmt='%.1f', padding = 5, fontsize=18, color=\"black\")\nax[0,1].set_ylabel(\"\")\nax[0,1].set_xlabel(\"Earnings (%)\", size=20,fontweight=\"bold\")\nax[0,1].set_title(\"Index\", size=25,fontweight=\"bold\")\nax[0,1].grid(axis=\"x\",color = 'grey', linestyle = '--', linewidth = 1.5)\n\n\ng = sns.barplot(y=\"colour_group_name\", x=\"earning(%)\", data=top_100_price_details.iloc[:no].groupby(\"colour_group_name\")[\"earning\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index().sort_values(by=\"earning(%)\", ascending=False), \\\n ax=ax[1,0],palette=\"mako\", ci=False)\nfor container in g.containers:\n g.bar_label(container, padding = 5, fmt='%.1f', fontsize=18, color=\"black\")\nax[1,0].set_ylabel(\"\")\nax[1,0].set_xlabel(\"Earnings (%)\", size=20,fontweight=\"bold\")\nax[1,0].set_title(\"Colour Group\", size=25,fontweight=\"bold\")\nax[1,0].grid(axis=\"x\",color = 'grey', linestyle = '--', linewidth = 1.5)\n\ng = sns.barplot(y=\"product_group_name\", x=\"earning(%)\", data=top_100_price_details.iloc[:no].groupby(\"product_group_name\")[\"earning\"].sum() \\\n .transform(lambda x: (x / x.sum() * 100)).rename('earning(%)').reset_index().sort_values(by=\"earning(%)\", ascending=False), \\\n ax=ax[1,1],palette=\"Reds_r\", ci=False)\nfor container in g.containers:\n g.bar_label(container, fmt='%.1f', padding=5, fontsize=18, color=\"black\")\nax[1,1].set_ylabel(\"\")\nax[1,1].set_xlabel(\"Earnings (%)\", size=20,fontweight=\"bold\")\nax[1,1].set_title(\"Product Group\", size=25,fontweight=\"bold\")\nax[1,1].grid(axis=\"x\",color = 'grey', linestyle = '--', linewidth = 1.5)\nfig.tight_layout()\n\nplt.show() ","metadata":{"execution":{"iopub.status.busy":"2022-05-16T09:32:30.203493Z","iopub.execute_input":"2022-05-16T09:32:30.203798Z","iopub.status.idle":"2022-05-16T09:32:31.729530Z","shell.execute_reply.started":"2022-05-16T09:32:30.203764Z","shell.execute_reply":"2022-05-16T09:32:31.728835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights:\n- Over 60% of the TOP 100 products in terms of earnings are generated by selling trousers\n- Around 50% of these products are divided (a H&M teenage collection)\n- 37% of the products are from the Ladieswear line\n- 55% of the products are black\n- 66.2% of the products are related to lower body\n\n**NOTE: It is also important to notice that the TOP 100 most profitable products list do not exactly match the TOP 100 most sold products one, since lots of products that sells a lot in quantity are cheap, and so generate less earnings.**","metadata":{}},{"cell_type":"markdown","source":"

⬆️⬆️⬆️ This Notebook is still a W.I.P., I will update it please upvote if you enjoy the first steps! ⬆️⬆️⬆️

","metadata":{}}]}