{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\nThis note is an EDA focusing on the relationship between the number of units sold and the price of an articles, and shows the correlation coefficient between the number of units sold per month and the average price per month.   \nMany of the other notes do not use \"price\" in the train data, which makes me wonder how price affects sales. (I am most likely missing it).  \nHere is what this note indicates    \n・Almost no correlation between the number of articles and the price of the articles in-store sales.  \n・Online shopping shows a slight correlation between the number of units sold and the price of the product.    \n・Overall, there is a slight correlation between the number of units sold and article prices.   \nThe code is partially based on the code at https://www.kaggle.com/code/negoto/h-m-sales-period-of-fashion-items-with-k-means/notebook.  ","metadata":{"papermill":{"duration":0.019425,"end_time":"2022-03-28T22:57:50.505324","exception":false,"start_time":"2022-03-28T22:57:50.485899","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set()\nimport tqdm\nfrom datetime import datetime as dt\nfrom collections import Counter\nfrom pathlib import Path\npath = Path(\"../input/h-and-m-personalized-fashion-recommendations/\")\narticles = pd.read_csv(path / \"articles.csv\", dtype = {'article_id': str})\ntrain = pd.read_csv(path / \"transactions_train.csv\", dtype = {'article_id': str})\ntrain = train[[\"t_dat\", \"article_id\", \"sales_channel_id\",\"price\"]]\ntrain[\"t_dat\"] = pd.to_datetime(train[\"t_dat\"])\n# Uncomment the following if you want to limit channel_id\n# train = train.query(\"sales_channel_id == 1\") \n# train = train.query(\"sales_channel_id == 2\")\ntrain = train.sort_values([\"article_id\", \"t_dat\"], ascending=False)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2022-03-28T22:57:50.552403Z","iopub.status.busy":"2022-03-28T22:57:50.551424Z","iopub.status.idle":"2022-03-28T22:59:31.324006Z","shell.execute_reply":"2022-03-28T22:59:31.324585Z","shell.execute_reply.started":"2022-03-28T12:17:06.191046Z"},"papermill":{"duration":100.800571,"end_time":"2022-03-28T22:59:31.324922","exception":false,"start_time":"2022-03-28T22:57:50.524351","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data processing","metadata":{"papermill":{"duration":0.017878,"end_time":"2022-03-28T22:59:31.364816","exception":false,"start_time":"2022-03-28T22:59:31.346938","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Add columns for average number of units purchased and average price for each article over all time periods.\nsales_counts = Counter(train.article_id)\nfor i in articles.index:\n    articles.at[i, \"sales_count\"] = sales_counts[articles.at[i, \"article_id\"]]/24 # 24:num of month\nsales_price_ave = {}\nsales_price_ave = train.groupby(\"article_id\").price.mean().to_dict()\nfor i in articles.index:\n    if articles.at[i, \"article_id\"] in sales_price_ave:\n        articles.at[i, \"sales_price_ave\"] = sales_price_ave[articles.at[i, \"article_id\"]]","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2022-03-28T13:59:41.528032Z","iopub.status.busy":"2022-03-28T13:59:41.527355Z","iopub.status.idle":"2022-03-28T13:59:49.401232Z","shell.execute_reply":"2022-03-28T13:59:49.401734Z","shell.execute_reply.started":"2022-03-28T12:18:25.963343Z"},"papermill":{"duration":null,"end_time":null,"exception":false,"start_time":"2022-03-28T22:59:31.383112","status":"running"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Add the number of units purchased and average price for each month\nYM = [201809, 201810]\nwhile YM[0] < 202010:\n    start, end = \"-\".join(map(str, [YM[0] // 100, YM[0] % 100, 1])), \"-\".join(map(str, [YM[1] // 100, YM[1] % 100, 1]))\n    monthly_sales = Counter(train.query(f\"'{start}' <= t_dat < '{end}'\").article_id)\n    # Sales num\n    articles[YM[0]] = 0\n    for i in articles.index:\n        articles.at[i, YM[0]]= monthly_sales[articles.at[i, \"article_id\"]]\n    YM[0] = YM[1]\n    YM[1] = (YM[1] + 100 - 11) if YM[1] % 100 == 12 else (YM[1] + 1)\n\nYM = [201809, 201810]\nwhile YM[0] < 202010:\n    start, end = \"-\".join(map(str, [YM[0] // 100, YM[0] % 100, 1])), \"-\".join(map(str, [YM[1] // 100, YM[1] % 100, 1]))\n    # Sales price\n    monthly_price_ave = train.query(f\"'{start}' <= t_dat < '{end}'\").groupby(\"article_id\").price.mean().to_dict()\n    articles[YM[0]+100000000] = 0\n    for i in articles.index:\n        if articles.at[i, \"article_id\"] in monthly_price_ave:\n            articles.at[i, YM[0]+100000000] =  monthly_price_ave[articles.at[i, \"article_id\"]]\n        # No purchase stores None\n        if articles.at[i,YM[0]+100000000] < 1e-4: \n            articles.at[i,YM[0]+100000000] = None\n    YM[0] = YM[1]\n    YM[1] = (YM[1] + 100 - 11) if YM[1] % 100 == 12 else (YM[1] + 1)\n\narticles.head()","metadata":{"execution":{"iopub.execute_input":"2022-03-28T13:59:49.438769Z","iopub.status.busy":"2022-03-28T13:59:49.437926Z","iopub.status.idle":"2022-03-28T14:01:50.00375Z","shell.execute_reply":"2022-03-28T14:01:50.004409Z","shell.execute_reply.started":"2022-03-28T12:18:32.386288Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Standardize the number of units sold and selling price\nfor i in tqdm.tqdm(articles.index):\n    count_std = articles.iloc[i,27:52].std()\n    price_std = articles.iloc[i,52:].std()\n    articles.iloc[i,27:52] = (articles.iloc[i,27:52]-articles.iloc[i,25]) / count_std\n    articles.iloc[i,52:] = (articles.iloc[i,52:]-articles.iloc[i,26]) / price_std","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:01:50.092169Z","iopub.status.busy":"2022-03-28T14:01:50.091393Z","iopub.status.idle":"2022-03-28T14:56:44.896997Z","shell.execute_reply":"2022-03-28T14:56:44.897466Z","shell.execute_reply.started":"2022-03-28T12:20:04.700895Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.head()\n","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:56:57.605764Z","iopub.status.busy":"2022-03-28T14:56:57.604837Z","iopub.status.idle":"2022-03-28T14:56:57.631392Z","shell.execute_reply":"2022-03-28T14:56:57.631994Z","shell.execute_reply.started":"2022-03-28T13:10:21.791554Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Transision of article sales and price","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"def plot_sales_num_and_price(article_id):\n    plt.figure(figsize=(24, 1.5))\n    plot_df = articles.query(f\"article_id == '{article_id}'\")\n    sns.lineplot(x=plot_df.columns[27:52].map(lambda x: dt.strptime(str(x),'%Y%m')), y=list(*plot_df.values)[27:52], palette=sns.husl_palette(12),linestyle='None',marker=\"o\", markersize=5,color='r')\n    sns.lineplot(x=plot_df.columns[27:52].map(lambda x: dt.strptime(str(x),'%Y%m')), y=list(*plot_df.values)[52:77], palette=sns.husl_palette(12),linestyle='None',marker=\"o\", markersize=5,color='b')\n    plt.legend(['stand_count','stand_price'])\n    plt.title(\" \".join([\"Monthly Sales of ID :\", article_id, \"sales_count:\", str(plot_df.iloc[0, 25])[:10], \"    price_average :\", str(plot_df.iloc[0, 26])[:10]]))","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:57:23.113305Z","iopub.status.busy":"2022-03-28T14:57:23.112696Z","iopub.status.idle":"2022-03-28T14:57:23.114808Z","shell.execute_reply":"2022-03-28T14:57:23.115346Z","shell.execute_reply.started":"2022-03-28T13:10:21.833166Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check three samples of monthly sales quantity and sales price","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Sample 1\nplot_sales_num_and_price(articles.loc[0,\"article_id\"])","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:57:48.231033Z","iopub.status.busy":"2022-03-28T14:57:48.230101Z","iopub.status.idle":"2022-03-28T14:57:48.880168Z","shell.execute_reply":"2022-03-28T14:57:48.88059Z","shell.execute_reply.started":"2022-03-28T13:10:21.843194Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample 2\nplot_sales_num_and_price(articles.loc[1,\"article_id\"])","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:58:01.496612Z","iopub.status.busy":"2022-03-28T14:58:01.495994Z","iopub.status.idle":"2022-03-28T14:58:01.992653Z","shell.execute_reply":"2022-03-28T14:58:01.992157Z","shell.execute_reply.started":"2022-03-28T13:10:22.464118Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample 3\nprint(articles.shape)\nplot_sales_num_and_price(articles.loc[3,\"article_id\"])","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:58:14.530536Z","iopub.status.busy":"2022-03-28T14:58:14.529863Z","iopub.status.idle":"2022-03-28T14:58:15.03579Z","shell.execute_reply":"2022-03-28T14:58:15.035222Z","shell.execute_reply.started":"2022-03-28T13:10:22.924739Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Calc. correlation coefficient between the number of units sold and the average price","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Correlation coefficients between price and quantity are calculated for each product.\ncorrcoef_array = []\narticles[\"count_price_coef\"] = 0\narticles[\"valid_coef_num\"] = 0\nfor i in tqdm.tqdm(articles.index):\n    stand_count = articles.iloc[i,27:52].values\n    stand_price = articles.iloc[i,52:77].values\n    new_a = []\n    new_b = []\n    for item1,item2 in zip(stand_count,stand_price):\n        if not (np.isnan(item1) or np.isnan(item2)):\n            new_a.append(item1)\n            new_b.append(item2)\n    stand_count = new_a\n    stand_price = new_b\n    tmp = np.corrcoef(stand_count,stand_price)\n    articles.loc[i,\"count_price_coef\"] = tmp[0,1]\n    articles.loc[i,\"valid_coef_num\"] = len(stand_count)\n    \n# Exclude products that have many months in which not a single unit is sold.(Here, n=10)\narticles = articles[articles[\"valid_coef_num\"]>=10]\n","metadata":{"execution":{"iopub.execute_input":"2022-03-28T14:58:40.485914Z","iopub.status.busy":"2022-03-28T14:58:40.485271Z","iopub.status.idle":"2022-03-28T15:02:39.58521Z","shell.execute_reply":"2022-03-28T15:02:39.584591Z","shell.execute_reply.started":"2022-03-28T13:10:23.385657Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a histogram of correlation coefficients\nfig = plt.figure()\nax = fig.add_subplot(1, 1, 1)\nax.hist(articles[\"count_price_coef\"],bins=10)\nplt.xlabel('Correlation coefficient[-]')\nplt.ylabel('Freaquency[-]')","metadata":{"execution":{"iopub.execute_input":"2022-03-28T15:02:53.457497Z","iopub.status.busy":"2022-03-28T15:02:53.456521Z","iopub.status.idle":"2022-03-28T15:02:53.771131Z","shell.execute_reply":"2022-03-28T15:02:53.770357Z","shell.execute_reply.started":"2022-03-28T13:13:33.469103Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check one of the scatter and transition graphs for products with large correlation coefficients as a sample.\nlarge_corr_id = articles[articles[\"count_price_coef\"]>0.8].iloc[0].article_id\ni = articles[articles[\"article_id\"] == large_corr_id].index[0]\nstand_count = articles.iloc[i,27:52].values\nstand_price = articles.iloc[i,52:77].values\nnew_a = []\nnew_b = []\nfor item1,item2 in zip(stand_count,stand_price):\n    if not (np.isnan(item1) or np.isnan(item2)):\n        new_a.append(item1)\n        new_b.append(item2)\nstand_count = new_a\nstand_price = new_b\ntmp = np.corrcoef(stand_count,stand_price)\narticles.loc[i,\"count_price_coef\"] = tmp[0,1]\narticles.loc[i,\"valid_coef_num\"] = len(stand_count)\nplt.plot(stand_count,stand_price,'x')\nplot_sales_num_and_price(articles.loc[i,\"article_id\"])","metadata":{"execution":{"iopub.execute_input":"2022-03-28T15:03:07.787683Z","iopub.status.busy":"2022-03-28T15:03:07.78699Z","iopub.status.idle":"2022-03-28T15:03:08.554054Z","shell.execute_reply":"2022-03-28T15:03:08.553241Z","shell.execute_reply.started":"2022-03-28T13:13:33.801082Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Restricted to channel=1 and channel = 2\nUncomment the code at the beginning to get the result when limited.  \n- channel = 1  \nhttps://cdn.discordapp.com/attachments/957917652846796830/958133553940557824/channel1.png  \n- channel = 2  \nhttps://cdn.discordapp.com/attachments/957917652846796830/958133554154459227/channel2.png","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"# Summary and Discussion\n・I would have expected sales to increase when a sale occurs, but perhaps the trend is more toward lower prices when sales go down.    \n・The correlation between the number of units sold and price is more apparent online. It is difficult to imagine that a lower price would result in fewer sales, and it is reasonable to assume that a lower price would result in fewer sales. We considered the following.    \noffline：Customers will buy at a lower price for the purpose of inventory clearance if they buy at a lower price.  \nonline: Even if it's cheaper, don't buy things that are out of season or out of style (perhaps they can buy them when you need them when online). \n・If a correlation was found, we thought that if we knew the cycle of price reductions, we could determine when the number of units sold would increase, but two years of data were not sufficient to do so.    \n・From the graph of fluctuations: overall, many clothing prices are falling (if it is natural to say so).    \n・I still think it would be better to treat the number of units sold as the feature quantity rather than the price.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}}]}