{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Let's analize price distribution.\nFor simplicity price has been scaled to a range between 0 and 1.","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-15T13:13:31.800710Z","iopub.execute_input":"2022-02-15T13:13:31.801163Z","iopub.status.idle":"2022-02-15T13:13:31.827616Z","shell.execute_reply.started":"2022-02-15T13:13:31.801018Z","shell.execute_reply":"2022-02-15T13:13:31.826876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read and prepare Normalized Price","metadata":{}},{"cell_type":"code","source":"df=pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\",dtype={\"article_id\":str})","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:13:31.829270Z","iopub.execute_input":"2022-02-15T13:13:31.830023Z","iopub.status.idle":"2022-02-15T13:14:51.266311Z","shell.execute_reply.started":"2022-02-15T13:13:31.829982Z","shell.execute_reply":"2022-02-15T13:14:51.265074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"price_normalized\"]=df[\"price\"]/np.max(df[\"price\"])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:14:51.268824Z","iopub.execute_input":"2022-02-15T13:14:51.269214Z","iopub.status.idle":"2022-02-15T13:14:51.538686Z","shell.execute_reply.started":"2022-02-15T13:14:51.269169Z","shell.execute_reply":"2022-02-15T13:14:51.537931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plots of price frequency general and differentiated for channel id ","metadata":{}},{"cell_type":"markdown","source":"From these plots as we could expect we can see that most frequently bought items lies in the lowest price range (price < 0.2).","metadata":{}},{"cell_type":"code","source":"df[\"price_normalized\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:23:58.371196Z","iopub.execute_input":"2022-02-15T13:23:58.371552Z","iopub.status.idle":"2022-02-15T13:24:03.528266Z","shell.execute_reply.started":"2022-02-15T13:23:58.371516Z","shell.execute_reply":"2022-02-15T13:24:03.526977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df[\"sales_channel_id\"]==1][\"price_normalized\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:23:47.761175Z","iopub.execute_input":"2022-02-15T13:23:47.761828Z","iopub.status.idle":"2022-02-15T13:23:50.370836Z","shell.execute_reply.started":"2022-02-15T13:23:47.761787Z","shell.execute_reply":"2022-02-15T13:23:50.369896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df[\"sales_channel_id\"]==2][\"price_normalized\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:03.529994Z","iopub.execute_input":"2022-02-15T13:24:03.530217Z","iopub.status.idle":"2022-02-15T13:24:09.407051Z","shell.execute_reply.started":"2022-02-15T13:24:03.530190Z","shell.execute_reply":"2022-02-15T13:24:09.405962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Log scale for Y axis","metadata":{}},{"cell_type":"markdown","source":"We can see once we introduce logarithmic scale that products with highest prices are still being bought but with a lower frequency (around 10^4 articles for each bin compared to 10^7 for each bin for lowest prices).\nWe have a clear example of a Long Tail, One of the main tasks in this challenge will be to correctly use data about the long tail to customize user Recommendation.","metadata":{}},{"cell_type":"code","source":"df[\"price_normalized\"].plot.hist(bins=50, log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:09.409176Z","iopub.execute_input":"2022-02-15T13:24:09.409640Z","iopub.status.idle":"2022-02-15T13:24:15.718948Z","shell.execute_reply.started":"2022-02-15T13:24:09.409590Z","shell.execute_reply":"2022-02-15T13:24:15.718053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df[\"sales_channel_id\"]==1][\"price_normalized\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:15.720361Z","iopub.execute_input":"2022-02-15T13:24:15.720595Z","iopub.status.idle":"2022-02-15T13:24:18.879949Z","shell.execute_reply.started":"2022-02-15T13:24:15.720568Z","shell.execute_reply":"2022-02-15T13:24:18.878838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[df[\"sales_channel_id\"]==2][\"price_normalized\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:18.882125Z","iopub.execute_input":"2022-02-15T13:24:18.882389Z","iopub.status.idle":"2022-02-15T13:24:29.339690Z","shell.execute_reply.started":"2022-02-15T13:24:18.882360Z","shell.execute_reply":"2022-02-15T13:24:29.338718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# User Price range","metadata":{}},{"cell_type":"code","source":"df_users=df.copy()[[\"customer_id\",\"price_normalized\"]]","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:17:06.426149Z","iopub.execute_input":"2022-02-15T13:17:06.426480Z","iopub.status.idle":"2022-02-15T13:17:10.000913Z","shell.execute_reply.started":"2022-02-15T13:17:06.426446Z","shell.execute_reply":"2022-02-15T13:17:09.999932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_users=df_users.groupby(by=[\"customer_id\"]).agg([\"mean\",\"std\"]).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:17:10.002817Z","iopub.execute_input":"2022-02-15T13:17:10.003315Z","iopub.status.idle":"2022-02-15T13:17:26.686835Z","shell.execute_reply.started":"2022-02-15T13:17:10.003267Z","shell.execute_reply":"2022-02-15T13:17:26.685724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_users.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:19:16.189631Z","iopub.execute_input":"2022-02-15T13:19:16.190483Z","iopub.status.idle":"2022-02-15T13:19:16.206197Z","shell.execute_reply.started":"2022-02-15T13:19:16.190436Z","shell.execute_reply":"2022-02-15T13:19:16.205120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plots User price range (Mean and Std)","metadata":{}},{"cell_type":"markdown","source":"From these plots we can see a similar behaviour compared with the price ranges of transactions in the dataset,\nIn this case we can see that most of the users buy products with an average price of less than 0.1.\nFrom the std plot we can also see that users tends to buy products within a certain price range and rarely buy products with a price too far from their usual range.","metadata":{}},{"cell_type":"code","source":"df_users[\"price_normalized\"][\"mean\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:29.341214Z","iopub.execute_input":"2022-02-15T13:24:29.341472Z","iopub.status.idle":"2022-02-15T13:24:29.968476Z","shell.execute_reply.started":"2022-02-15T13:24:29.341438Z","shell.execute_reply":"2022-02-15T13:24:29.967241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_users[\"price_normalized\"][\"std\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:31.157718Z","iopub.execute_input":"2022-02-15T13:24:31.158170Z","iopub.status.idle":"2022-02-15T13:24:31.661866Z","shell.execute_reply.started":"2022-02-15T13:24:31.158137Z","shell.execute_reply":"2022-02-15T13:24:31.660921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Log scale for Y axys ","metadata":{}},{"cell_type":"markdown","source":"We can see from the log scaled version of the plot that there are users which have a different buying pattern.We can also see some outlier users which buy products with a really high average price compared with a generic user.\nThose users are really important for a business case, they tend to have different patterns than general users and for them usually a top Popular approach doesn't work.\nWe can also see from the std graph that some users buy products from a broader range of price.\n\nFor the users in the Long Tail there could be different approaches, clustering them and using a user-user similarity approach combined with an item-item similarity approach to combine similar users data and similar items data, but also their buying history is important. Combining these 3 elements could lead to better recommendation the further we move from a generic user to the end of the Long Tail.\nAlso an additional analysis of the popularity bias which is introduced by item-item and user-user similarity approaches is important to reduce too generic recommendations.","metadata":{}},{"cell_type":"code","source":"df_users[\"price_normalized\"][\"mean\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:29.970499Z","iopub.execute_input":"2022-02-15T13:24:29.971057Z","iopub.status.idle":"2022-02-15T13:24:31.155810Z","shell.execute_reply.started":"2022-02-15T13:24:29.971010Z","shell.execute_reply":"2022-02-15T13:24:31.155152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_users[\"price_normalized\"][\"std\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:24:31.663109Z","iopub.execute_input":"2022-02-15T13:24:31.663343Z","iopub.status.idle":"2022-02-15T13:24:32.957626Z","shell.execute_reply.started":"2022-02-15T13:24:31.663314Z","shell.execute_reply":"2022-02-15T13:24:32.956640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Products Price range","metadata":{}},{"cell_type":"code","source":"df_products=df.copy()[[\"article_id\",\"price_normalized\"]]","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:25.905435Z","iopub.execute_input":"2022-02-15T13:27:25.906198Z","iopub.status.idle":"2022-02-15T13:27:30.046057Z","shell.execute_reply.started":"2022-02-15T13:27:25.906149Z","shell.execute_reply":"2022-02-15T13:27:30.044861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_products=df_products.groupby(by=[\"article_id\"]).agg([\"mean\",\"std\"]).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:38.309699Z","iopub.execute_input":"2022-02-15T13:27:38.310052Z","iopub.status.idle":"2022-02-15T13:27:46.002202Z","shell.execute_reply.started":"2022-02-15T13:27:38.310013Z","shell.execute_reply":"2022-02-15T13:27:46.001096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_products.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:46.004292Z","iopub.execute_input":"2022-02-15T13:27:46.004577Z","iopub.status.idle":"2022-02-15T13:27:46.021367Z","shell.execute_reply.started":"2022-02-15T13:27:46.004543Z","shell.execute_reply":"2022-02-15T13:27:46.018344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plots products price range (Mean and Std)","metadata":{}},{"cell_type":"markdown","source":"From these plots we can see a similar behaviour compared with the price ranges of transactions in the dataset,\nIn this case we can see that most of the average price for a product tends to be in the lowest price range (<0.1).\nAnd from the std plot we can also see that most products have a small range of variation of their price.","metadata":{}},{"cell_type":"code","source":"df_products[\"price_normalized\"][\"mean\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:46.023508Z","iopub.execute_input":"2022-02-15T13:27:46.024881Z","iopub.status.idle":"2022-02-15T13:27:46.745130Z","shell.execute_reply.started":"2022-02-15T13:27:46.024819Z","shell.execute_reply":"2022-02-15T13:27:46.742897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_products[\"price_normalized\"][\"std\"].plot.hist(bins=50, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:47.427163Z","iopub.execute_input":"2022-02-15T13:27:47.427568Z","iopub.status.idle":"2022-02-15T13:27:47.724494Z","shell.execute_reply.started":"2022-02-15T13:27:47.427533Z","shell.execute_reply":"2022-02-15T13:27:47.723764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Log scale for Y axys ","metadata":{}},{"cell_type":"markdown","source":"The log scaled versions of the graph resembles the ones seen for user price range, an interesting approach could be to segment these ranges and use a binned top popularity approach so for a user who buys product with an average price of 0.15 suggest the most popular item with a price of 0.15 or close to that price. One could divide the price range in an arbitrary amount of bins and than place each user and item in those bins and than use the bins to create recommendations.","metadata":{}},{"cell_type":"code","source":"df_products[\"price_normalized\"][\"mean\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:46.747389Z","iopub.execute_input":"2022-02-15T13:27:46.747672Z","iopub.status.idle":"2022-02-15T13:27:47.425770Z","shell.execute_reply.started":"2022-02-15T13:27:46.747626Z","shell.execute_reply":"2022-02-15T13:27:47.424832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_products[\"price_normalized\"][\"std\"].plot.hist(bins=50,log=True, range=[-0.1, 1.0])","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:27:48.467749Z","iopub.execute_input":"2022-02-15T13:27:48.467973Z","iopub.status.idle":"2022-02-15T13:27:49.193593Z","shell.execute_reply.started":"2022-02-15T13:27:48.467945Z","shell.execute_reply":"2022-02-15T13:27:49.192573Z"},"trusted":true},"execution_count":null,"outputs":[]}]}