{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"The dataset contains 4 csv files and one folder with several subfolders, each with a different number of images.\n\nIn this Exploratory Data Analysis Notebook we will look to the data, will analyze the content of each csv file, check for missing data, understand the data distribution, see what are the relations between data in various files.\n\nWe will also explore the image data, understand how images are indexed in the csv files, if there are articles in the dataset without images. We will also explore image additional information, like image width and height.\n\nWe also investigate a very simple baseline model and create an initial submission.\n\n","metadata":{"execution":{"iopub.status.busy":"2022-03-24T04:56:55.076807Z","iopub.execute_input":"2022-03-24T04:56:55.077509Z","iopub.status.idle":"2022-03-24T04:56:55.105121Z","shell.execute_reply.started":"2022-03-24T04:56:55.0774Z","shell.execute_reply":"2022-03-24T04:56:55.103795Z"}}},{"cell_type":"markdown","source":"![](http://images.unsplash.com/photo-1578983662508-41895226ebfb?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=1211&q=80)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Intro**\n\n**The competition is dedicated to the product recomendations (H&M)**\n\n**Here we have different kinds of data that help us to get good recomendations:**\n\n**📸 images - images of every article_id**\n\n**🙋 articles - detailed metadata of every article_id**\n\n**👔 customers - detailed metadata of every customer_id**\n\n**🧾 transactions_train - purchases with details**","metadata":{}},{"cell_type":"markdown","source":"# Analysis preparation","metadata":{}},{"cell_type":"markdown","source":"We will include here the required packages for reading, parsing, filtering, processing, visualizing the data, both tabular and image.","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:04:25.886113Z","iopub.execute_input":"2022-03-24T05:04:25.886476Z","iopub.status.idle":"2022-03-24T05:04:25.901433Z","shell.execute_reply.started":"2022-03-24T05:04:25.886385Z","shell.execute_reply":"2022-03-24T05:04:25.900243Z"}}},{"cell_type":"markdown","source":"![](https://lp2.hm.com/hmgoepprod?set=quality%5B79%5D%2Csource%5B%2F55%2F6e%2F556e6853e5cf8986017f40e5fcf356016a599e41.jpg%5D%2Corigin%5Bdam%5D%2Ccategory%5Bmen_tshirtstanks_shortsleeve%5D%2Ctype%5BDESCRIPTIVEDETAIL%5D%2Cres%5Bm%5D%2Chmver%5B2%5D&call=url[file:/product/main])","metadata":{}},{"cell_type":"markdown","source":"# Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom wordcloud import WordCloud, STOPWORDS\nfrom datetime import datetime\nfrom PIL import Image","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:06:43.548935Z","iopub.execute_input":"2022-03-24T05:06:43.549204Z","iopub.status.idle":"2022-03-24T05:06:44.739391Z","shell.execute_reply.started":"2022-03-24T05:06:43.549178Z","shell.execute_reply":"2022-03-24T05:06:44.738494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ntransactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:06:54.903519Z","iopub.execute_input":"2022-03-24T05:06:54.903903Z","iopub.status.idle":"2022-03-24T05:08:15.690801Z","shell.execute_reply.started":"2022-03-24T05:06:54.903867Z","shell.execute_reply":"2022-03-24T05:08:15.689475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Let's look at the tables and try to get some outcomes about data inside.**","metadata":{}},{"cell_type":"markdown","source":"# Articles","metadata":{}},{"cell_type":"markdown","source":"This table contains all h&m articles with details such as a type of product, a color, a product group and other features.\nArticle data description:\n\narticle_id : A unique identifier of every article.\n\nproduct_code, prod_name : A unique identifier of every product and its name (not the same).\n\nproduct_type, product_type_name : The group of product_code and its name\n\ngraphical_appearance_no, graphical_appearance_name : The group of graphics and its name\n\ncolour_group_code, colour_group_name : The group of color and its name\n\nperceived_colour_value_id, perceived_colour_value_name, perceived_colour_master_id, perceived_colour_master_name : The added color info\n\ndepartment_no, department_name: : A unique identifier of every dep and its name\n\nindex_code, index_name: : A unique identifier of every index and its name\n\nindex_group_no, index_group_name: : A group of indeces and its name\n\nsection_no, section_name: : A unique identifier of every section and its name\n\ngarment_group_no, garment_group_name: : A unique identifier of every garment and its name\n\ndetail_desc: : Details","metadata":{}},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:09:33.630927Z","iopub.execute_input":"2022-03-24T05:09:33.631260Z","iopub.status.idle":"2022-03-24T05:09:33.682528Z","shell.execute_reply.started":"2022-03-24T05:09:33.631229Z","shell.execute_reply":"2022-03-24T05:09:33.681510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****Ladieswear accounts for a significant part of all dresses. Sportswear has the least portion.****","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=articles, y='index_name', color='orange')\nax.set_xlabel('count by index name')\nax.set_ylabel('index name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:10:11.639814Z","iopub.execute_input":"2022-03-24T05:10:11.640138Z","iopub.status.idle":"2022-03-24T05:10:12.134354Z","shell.execute_reply.started":"2022-03-24T05:10:11.640106Z","shell.execute_reply":"2022-03-24T05:10:12.132944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****The garments grouped by index: Jersey fancy is the most frequent garment, especially for women and children. The next by number is accessories, many various accessories with low price.****","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(15, 7))\nax = sns.histplot(data=articles, y='garment_group_name', color='orange', hue='index_group_name', multiple=\"stack\")\nax.set_xlabel('count by garment group')\nax.set_ylabel('garment group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:10:44.339974Z","iopub.execute_input":"2022-03-24T05:10:44.340273Z","iopub.status.idle":"2022-03-24T05:10:45.296350Z","shell.execute_reply.started":"2022-03-24T05:10:44.340244Z","shell.execute_reply":"2022-03-24T05:10:45.295444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Now,the index group-index structure. Ladieswear and Children/Baby have subgroups.**","metadata":{}},{"cell_type":"code","source":"articles.groupby(['index_group_name', 'index_name']).count()['article_id']","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:11:46.437276Z","iopub.execute_input":"2022-03-24T05:11:46.437583Z","iopub.status.idle":"2022-03-24T05:11:46.634072Z","shell.execute_reply.started":"2022-03-24T05:11:46.437553Z","shell.execute_reply":"2022-03-24T05:11:46.633462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Accessories are really various, the most numerious: bags, earrings and hats. However, trousers prevail.**","metadata":{}},{"cell_type":"code","source":"pd.options.display.max_rows = None\narticles.groupby(['product_group_name', 'product_type_name']).count()['article_id']","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:12:16.513883Z","iopub.execute_input":"2022-03-24T05:12:16.514681Z","iopub.status.idle":"2022-03-24T05:12:16.709467Z","shell.execute_reply.started":"2022-03-24T05:12:16.514622Z","shell.execute_reply":"2022-03-24T05:12:16.708303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Table with number of unique values in columns:**","metadata":{}},{"cell_type":"code","source":"for col in articles.columns:\n    if not 'no' in col and not 'code' in col and not 'id' in col:\n        un_n = articles[col].nunique()\n        print(f'n of unique {col}: {un_n}')","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:12:48.568910Z","iopub.execute_input":"2022-03-24T05:12:48.569702Z","iopub.status.idle":"2022-03-24T05:12:48.729264Z","shell.execute_reply.started":"2022-03-24T05:12:48.569657Z","shell.execute_reply":"2022-03-24T05:12:48.728152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Customers","metadata":{}},{"cell_type":"markdown","source":"Customers data description:\n\ncustomer_id : A unique identifier of every customer\n    \nFN : 1 or missed\n    \nActive : 1 or missed\n    \nclub_member_status : Status in club\n    \nfashion_news_frequency : How often H&M may send news to customer\n    \nage : The current age\n    \npostal_code : Postal code of customer","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:13:26.390566Z","iopub.execute_input":"2022-03-24T05:13:26.390860Z","iopub.status.idle":"2022-03-24T05:13:26.398777Z","shell.execute_reply.started":"2022-03-24T05:13:26.390830Z","shell.execute_reply":"2022-03-24T05:13:26.397389Z"}}},{"cell_type":"code","source":"pd.options.display.max_rows = 50\ncustomers.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:14:28.673094Z","iopub.execute_input":"2022-03-24T05:14:28.673623Z","iopub.status.idle":"2022-03-24T05:14:28.688844Z","shell.execute_reply.started":"2022-03-24T05:14:28.673589Z","shell.execute_reply":"2022-03-24T05:14:28.688067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.shape[0] - customers['customer_id'].nunique()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:14:36.556740Z","iopub.execute_input":"2022-03-24T05:14:36.557392Z","iopub.status.idle":"2022-03-24T05:14:37.338063Z","shell.execute_reply.started":"2022-03-24T05:14:36.557331Z","shell.execute_reply":"2022-03-24T05:14:37.336776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"****Here we have abnormal number of customers by one postal code. One has 120303, it might be encoded nan adress or smth like a huge distribution center, or pickup****","metadata":{}},{"cell_type":"markdown","source":"**Ages, club_member_status are different, like customer_ids.**","metadata":{}},{"cell_type":"code","source":"customers[customers['postal_code']=='2c29ae653a9282cce4151bd87643c907644e09541abc28ae87dea0d1f6603b1c'].head(5)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:15:24.101028Z","iopub.execute_input":"2022-03-24T05:15:24.101381Z","iopub.status.idle":"2022-03-24T05:15:24.408921Z","shell.execute_reply.started":"2022-03-24T05:15:24.101331Z","shell.execute_reply":"2022-03-24T05:15:24.407959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nfrom matplotlib import pyplot as plt\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=customers, x='age', bins=50, color='orange')\nax.set_xlabel('Distribution of the customers age')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:15:32.875213Z","iopub.execute_input":"2022-03-24T05:15:32.875508Z","iopub.status.idle":"2022-03-24T05:15:33.431153Z","shell.execute_reply.started":"2022-03-24T05:15:32.875476Z","shell.execute_reply":"2022-03-24T05:15:33.430124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Status in H&M club.**","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=customers, x='club_member_status', color='orange')\nax.set_xlabel('Distribution of club member status')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:16:20.336933Z","iopub.execute_input":"2022-03-24T05:16:20.337592Z","iopub.status.idle":"2022-03-24T05:16:22.363958Z","shell.execute_reply.started":"2022-03-24T05:16:20.337527Z","shell.execute_reply":"2022-03-24T05:16:22.362770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers['fashion_news_frequency'].unique()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:16:32.204073Z","iopub.execute_input":"2022-03-24T05:16:32.204401Z","iopub.status.idle":"2022-03-24T05:16:32.366922Z","shell.execute_reply.started":"2022-03-24T05:16:32.204354Z","shell.execute_reply":"2022-03-24T05:16:32.365755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.loc[~customers['fashion_news_frequency'].isin(['Regularly', 'Monthly']), 'fashion_news_frequency'] = 'None'\ncustomers['fashion_news_frequency'].unique()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:16:38.707564Z","iopub.execute_input":"2022-03-24T05:16:38.708650Z","iopub.status.idle":"2022-03-24T05:16:38.920071Z","shell.execute_reply.started":"2022-03-24T05:16:38.708572Z","shell.execute_reply":"2022-03-24T05:16:38.918827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pie_data = customers[['customer_id', 'fashion_news_frequency']].groupby('fashion_news_frequency').count()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:16:44.530344Z","iopub.execute_input":"2022-03-24T05:16:44.530676Z","iopub.status.idle":"2022-03-24T05:16:44.899790Z","shell.execute_reply.started":"2022-03-24T05:16:44.530641Z","shell.execute_reply":"2022-03-24T05:16:44.898486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\n# ax = sns.histplot(data=customers, x='fashion_news_frequency', color='orange')\n# ax = sns.pie(data=customers, x='fashion_news_frequency', color='orange')\ncolors = sns.color_palette('pastel')\nax.pie(pie_data.customer_id, labels=pie_data.index, colors = colors)\nax.set_facecolor('lightgrey')\nax.set_xlabel('Distribution of fashion news frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:16:51.347227Z","iopub.execute_input":"2022-03-24T05:16:51.347558Z","iopub.status.idle":"2022-03-24T05:16:51.482270Z","shell.execute_reply.started":"2022-03-24T05:16:51.347528Z","shell.execute_reply":"2022-03-24T05:16:51.480771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Transactions","metadata":{}},{"cell_type":"markdown","source":"Transactions data description:\n\nt_dat : A unique identifier of every customer\n    \ncustomer_id : A unique identifier of every customer (in customers table)\n    \narticle_id : A unique identifier of every article (in articles table)\n    \nprice : Price of purchase\n    \nsales_channel_id : 1 or 2","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:17:16.211092Z","iopub.execute_input":"2022-03-24T05:17:16.211842Z","iopub.status.idle":"2022-03-24T05:17:16.219991Z","shell.execute_reply.started":"2022-03-24T05:17:16.211805Z","shell.execute_reply":"2022-03-24T05:17:16.218564Z"}}},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:17:31.822179Z","iopub.execute_input":"2022-03-24T05:17:31.822504Z","iopub.status.idle":"2022-03-24T05:17:31.838379Z","shell.execute_reply.started":"2022-03-24T05:17:31.822469Z","shell.execute_reply":"2022-03-24T05:17:31.837460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**we see outliers for price.**","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.float_format', '{:.4f}'.format)\ntransactions.describe()['price']","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:17:53.612314Z","iopub.execute_input":"2022-03-24T05:17:53.612610Z","iopub.status.idle":"2022-03-24T05:17:57.001656Z","shell.execute_reply.started":"2022-03-24T05:17:53.612580Z","shell.execute_reply":"2022-03-24T05:17:57.000796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:18:01.102003Z","iopub.execute_input":"2022-03-24T05:18:01.102265Z","iopub.status.idle":"2022-03-24T05:18:01.113711Z","shell.execute_reply.started":"2022-03-24T05:18:01.102237Z","shell.execute_reply":"2022-03-24T05:18:01.112906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.boxplot(data=transactions, x='price', color='orange')\nax.set_xlabel('Price outliers')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:18:07.005658Z","iopub.execute_input":"2022-03-24T05:18:07.005997Z","iopub.status.idle":"2022-03-24T05:18:11.766088Z","shell.execute_reply.started":"2022-03-24T05:18:07.005950Z","shell.execute_reply":"2022-03-24T05:18:11.765328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_byid = transactions.groupby('customer_id').count()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:18:15.127647Z","iopub.execute_input":"2022-03-24T05:18:15.128581Z","iopub.status.idle":"2022-03-24T05:18:34.736958Z","shell.execute_reply.started":"2022-03-24T05:18:15.128541Z","shell.execute_reply":"2022-03-24T05:18:34.735897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_byid.sort_values(by='price', ascending=False)['price'][:10]","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:18:34.738684Z","iopub.execute_input":"2022-03-24T05:18:34.739006Z","iopub.status.idle":"2022-03-24T05:18:35.341110Z","shell.execute_reply.started":"2022-03-24T05:18:34.738964Z","shell.execute_reply":"2022-03-24T05:18:35.340106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**However, comparing prices inside groups is more accurate, because accessories and trousers prices may vary largerly.**\n\n\n\n**Get subset from articles and merge it to transactions.**","metadata":{}},{"cell_type":"code","source":"articles_for_merge = articles[['article_id', 'prod_name', 'product_type_name', 'product_group_name', 'index_name']]","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:19:58.966734Z","iopub.execute_input":"2022-03-24T05:19:58.967062Z","iopub.status.idle":"2022-03-24T05:19:58.975760Z","shell.execute_reply.started":"2022-03-24T05:19:58.967027Z","shell.execute_reply":"2022-03-24T05:19:58.974870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_for_merge = transactions[['customer_id', 'article_id', 'price', 't_dat']].merge(articles_for_merge, on='article_id', how='left')","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:20:04.369305Z","iopub.execute_input":"2022-03-24T05:20:04.370537Z","iopub.status.idle":"2022-03-24T05:20:17.077684Z","shell.execute_reply.started":"2022-03-24T05:20:04.370475Z","shell.execute_reply":"2022-03-24T05:20:17.075985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(25,18))\nax = sns.boxplot(data=articles_for_merge, x='price', y='product_group_name')\nax.set_xlabel('Price outliers', fontsize=22)\nax.set_ylabel('Index names', fontsize=22)\nax.xaxis.set_tick_params(labelsize=22)\nax.yaxis.set_tick_params(labelsize=22)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:20:51.792961Z","iopub.execute_input":"2022-03-24T05:20:51.793288Z","iopub.status.idle":"2022-03-24T05:21:22.068059Z","shell.execute_reply.started":"2022-03-24T05:20:51.793248Z","shell.execute_reply":"2022-03-24T05:21:22.066701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Then look at boxplot prices according to accessories product group and find the reasons of high prices inside group.**\n\n**The largest outliers can be found among bags, which is logical enough. In addition, scarves and other accessories have articles with prices highly contrasting to the rest of garments.**","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(25,18))\n_ = articles_for_merge[articles_for_merge['product_group_name'] == 'Accessories']\nax = sns.boxplot(data=_, x='price', y='product_type_name')\nax.set_xlabel('Price outliers', fontsize=22)\nax.set_ylabel('Index names', fontsize=22)\nax.xaxis.set_tick_params(labelsize=22)\nax.yaxis.set_tick_params(labelsize=22)\ndel _\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:21:29.977876Z","iopub.execute_input":"2022-03-24T05:21:29.978749Z","iopub.status.idle":"2022-03-24T05:21:44.373097Z","shell.execute_reply.started":"2022-03-24T05:21:29.978697Z","shell.execute_reply":"2022-03-24T05:21:44.372004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_index = articles_for_merge[['index_name', 'price']].groupby('index_name').mean()\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=articles_index.price, y=articles_index.index, color='orange', alpha=0.8)\nax.set_xlabel('Price by index')\nax.set_ylabel('Index')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:21:53.926617Z","iopub.execute_input":"2022-03-24T05:21:53.926891Z","iopub.status.idle":"2022-03-24T05:21:58.475164Z","shell.execute_reply.started":"2022-03-24T05:21:53.926862Z","shell.execute_reply":"2022-03-24T05:21:58.474247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_index = articles_for_merge[['product_group_name', 'price']].groupby('product_group_name').mean()\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=articles_index.price, y=articles_index.index, color='orange', alpha=0.8)\nax.set_xlabel('Price by product group')\nax.set_ylabel('Product group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:22:13.251052Z","iopub.execute_input":"2022-03-24T05:22:13.251429Z","iopub.status.idle":"2022-03-24T05:22:17.775515Z","shell.execute_reply.started":"2022-03-24T05:22:13.251394Z","shell.execute_reply":"2022-03-24T05:22:17.774556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Check the mean price change in time for top 5 product groups by mean price:**\n\n* Shoes\n* Garment Full body\n* Bags\n* Garment Lower body\n* Underwear/nightwear","metadata":{}},{"cell_type":"code","source":"articles_for_merge['t_dat'] = pd.to_datetime(articles_for_merge['t_dat'])","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:23:09.738576Z","iopub.execute_input":"2022-03-24T05:23:09.738874Z","iopub.status.idle":"2022-03-24T05:23:18.293459Z","shell.execute_reply.started":"2022-03-24T05:23:09.738836Z","shell.execute_reply":"2022-03-24T05:23:18.292527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"product_list = ['Shoes', 'Garment Full body', 'Bags', 'Garment Lower body', 'Underwear/nightwear']\ncolors = ['cadetblue', 'orange', 'mediumspringgreen', 'tomato', 'lightseagreen']\nk = 0\nf, ax = plt.subplots(3, 2, figsize=(20, 15))\nfor i in range(3):\n    for j in range(2):\n        try:\n            product = product_list[k]\n            articles_for_merge_product = articles_for_merge[articles_for_merge.product_group_name == product_list[k]]\n            series_mean = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).mean().fillna(0)\n            series_std = articles_for_merge_product[['t_dat', 'price']].groupby(pd.Grouper(key=\"t_dat\", freq='M')).std().fillna(0)\n            ax[i, j].plot(series_mean, linewidth=4, color=colors[k])\n            ax[i, j].fill_between(series_mean.index, (series_mean.values-2*series_std.values).ravel(), \n                             (series_mean.values+2*series_std.values).ravel(), color=colors[k], alpha=.1)\n            ax[i, j].set_title(f'Mean {product_list[k]} price in time')\n            ax[i, j].set_xlabel('month')\n            ax[i, j].set_xlabel(f'{product_list[k]}')\n            k += 1\n        except IndexError:\n            ax[i, j].set_visible(False)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:23:25.863649Z","iopub.execute_input":"2022-03-24T05:23:25.863944Z","iopub.status.idle":"2022-03-24T05:23:53.877237Z","shell.execute_reply.started":"2022-03-24T05:23:25.863909Z","shell.execute_reply":"2022-03-24T05:23:53.876017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Images with description and price","metadata":{}},{"cell_type":"markdown","source":"**Check the last purchases by max price and by min price**","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib.image as mpimg","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:25:00.438421Z","iopub.execute_input":"2022-03-24T05:25:00.438724Z","iopub.status.idle":"2022-03-24T05:25:00.443187Z","shell.execute_reply.started":"2022-03-24T05:25:00.438692Z","shell.execute_reply":"2022-03-24T05:25:00.442462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_price_ids = transactions[transactions.t_dat==transactions.t_dat.max()].sort_values('price', ascending=False).iloc[:5][['article_id', 'price']]\nmin_price_ids = transactions[transactions.t_dat==transactions.t_dat.min()].sort_values('price', ascending=True).iloc[:5][['article_id', 'price']]","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:25:06.507802Z","iopub.execute_input":"2022-03-24T05:25:06.508558Z","iopub.status.idle":"2022-03-24T05:25:26.005918Z","shell.execute_reply.started":"2022-03-24T05:25:06.508518Z","shell.execute_reply":"2022-03-24T05:25:26.005195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Photos with description and price (top 5 max)**","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(1, 5, figsize=(20,10))\ni = 0\nfor _, data in max_price_ids.iterrows():\n    desc = articles[articles['article_id'] == data['article_id']]['detail_desc'].iloc[0]\n    desc_list = desc.split(' ')\n    for j, elem in enumerate(desc_list):\n        if j > 0 and j % 5 == 0:\n            desc_list[j] = desc_list[j] + '\\n'\n    desc = ' '.join(desc_list)\n    img = mpimg.imread(f'../input/h-and-m-personalized-fashion-recommendations/images/0{str(data.article_id)[:2]}/0{int(data.article_id)}.jpg')\n    ax[i].imshow(img)\n    ax[i].set_title(f'price: {data.price:.2f}')\n    ax[i].set_xticks([], [])\n    ax[i].set_yticks([], [])\n    ax[i].grid(False)\n    ax[i].set_xlabel(desc, fontsize=10)\n    i += 1\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:25:26.007460Z","iopub.execute_input":"2022-03-24T05:25:26.007830Z","iopub.status.idle":"2022-03-24T05:25:27.722107Z","shell.execute_reply.started":"2022-03-24T05:25:26.007801Z","shell.execute_reply":"2022-03-24T05:25:27.721442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Photos with description and price (top 5 min)**","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(1, 5, figsize=(20,10))\ni = 0\nfor _, data in min_price_ids.iterrows():\n    desc = articles[articles['article_id'] == data['article_id']]['detail_desc'].iloc[0]\n    desc_list = desc.split(' ')\n    for j, elem in enumerate(desc_list):\n        if j > 0 and j % 4 == 0:\n            desc_list[j] = desc_list[j] + '\\n'\n    desc = ' '.join(desc_list)\n    img = mpimg.imread(f'../input/h-and-m-personalized-fashion-recommendations/images/0{str(data.article_id)[:2]}/0{int(data.article_id)}.jpg')\n    ax[i].imshow(img)\n    ax[i].set_title(f'price: {data.price:.4f}')\n    ax[i].set_xlabel(desc, fontsize=10)\n    ax[i].set_xticks([], [])\n    ax[i].set_yticks([], [])\n    ax[i].grid(False)\n    i += 1\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:26:44.409976Z","iopub.execute_input":"2022-03-24T05:26:44.410283Z","iopub.status.idle":"2022-03-24T05:26:46.146510Z","shell.execute_reply.started":"2022-03-24T05:26:44.410248Z","shell.execute_reply":"2022-03-24T05:26:46.145399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read the data","metadata":{}},{"cell_type":"code","source":"print(f\"files and folders: {os.listdir('/kaggle/input/h-and-m-personalized-fashion-recommendations/')}\")\nprint(\"Subfolders in images folder: \", len(list(os.listdir(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/images\"))))","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:28:10.902078Z","iopub.execute_input":"2022-03-24T05:28:10.902412Z","iopub.status.idle":"2022-03-24T05:28:10.920864Z","shell.execute_reply.started":"2022-03-24T05:28:10.902379Z","shell.execute_reply":"2022-03-24T05:28:10.919703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"total_folders = total_files = 0\nfolder_info = []\nimages_names = []\nfor base, dirs, files in tqdm(os.walk('/kaggle/input/h-and-m-personalized-fashion-recommendations/')):\n    for directories in dirs:\n        folder_info.append((directories, len(os.listdir(os.path.join(base, directories)))))\n        total_folders += 1\n    for _files in files:\n        total_files += 1\n        if len(_files.split(\".jpg\"))==2:\n            images_names.append(_files.split(\".jpg\")[0])","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:28:18.615677Z","iopub.execute_input":"2022-03-24T05:28:18.615959Z","iopub.status.idle":"2022-03-24T05:28:49.242598Z","shell.execute_reply.started":"2022-03-24T05:28:18.615931Z","shell.execute_reply":"2022-03-24T05:28:49.241882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Total number of folders: {total_folders}\\nTotal number of files: {total_files}\")\nfolder_info_df = pd.DataFrame(folder_info, columns=[\"folder\", \"files count\"])\nfolder_info_df.sort_values([\"files count\"], ascending=False).head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:28:49.244081Z","iopub.execute_input":"2022-03-24T05:28:49.244712Z","iopub.status.idle":"2022-03-24T05:28:49.258897Z","shell.execute_reply.started":"2022-03-24T05:28:49.244673Z","shell.execute_reply":"2022-03-24T05:28:49.258175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"folder names: \", list(folder_info_df.folder.unique()))","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:28:56.639665Z","iopub.execute_input":"2022-03-24T05:28:56.639955Z","iopub.status.idle":"2022-03-24T05:28:56.646782Z","shell.execute_reply.started":"2022-03-24T05:28:56.639919Z","shell.execute_reply":"2022-03-24T05:28:56.646114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_df = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers_df = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/customers.csv\")\nsample_submission_df = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:28:59.593836Z","iopub.execute_input":"2022-03-24T05:28:59.594240Z","iopub.status.idle":"2022-03-24T05:29:12.284216Z","shell.execute_reply.started":"2022-03-24T05:28:59.594210Z","shell.execute_reply":"2022-03-24T05:29:12.282940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_train_df = pd.read_csv(\"/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:29:13.398780Z","iopub.execute_input":"2022-03-24T05:29:13.399085Z","iopub.status.idle":"2022-03-24T05:30:05.292325Z","shell.execute_reply.started":"2022-03-24T05:29:13.399051Z","shell.execute_reply":"2022-03-24T05:30:05.290559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:30:05.295304Z","iopub.execute_input":"2022-03-24T05:30:05.295817Z","iopub.status.idle":"2022-03-24T05:30:05.326540Z","shell.execute_reply.started":"2022-03-24T05:30:05.295680Z","shell.execute_reply":"2022-03-24T05:30:05.324851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:30:05.336572Z","iopub.execute_input":"2022-03-24T05:30:05.336900Z","iopub.status.idle":"2022-03-24T05:30:05.357081Z","shell.execute_reply.started":"2022-03-24T05:30:05.336864Z","shell.execute_reply":"2022-03-24T05:30:05.355455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:30:05.359104Z","iopub.execute_input":"2022-03-24T05:30:05.359869Z","iopub.status.idle":"2022-03-24T05:30:05.373407Z","shell.execute_reply.started":"2022-03-24T05:30:05.359818Z","shell.execute_reply":"2022-03-24T05:30:05.372193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prediction in sample submission is a sequence of article ids (max 12 article ids).","metadata":{}},{"cell_type":"code","source":"transactions_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:30:05.374905Z","iopub.execute_input":"2022-03-24T05:30:05.375818Z","iopub.status.idle":"2022-03-24T05:30:05.389815Z","shell.execute_reply.started":"2022-03-24T05:30:05.375765Z","shell.execute_reply":"2022-03-24T05:30:05.388766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 3 main tables:\n\n* articles - contains informations about each article (like product code, name, product group code, name ...)\n* customers - contains informations about each customer (fidelity card membership, age, postal code)\n* transactions (train)\n\nTransactions have customer_id and article_id, which are foreign keys for the customer and articles tables. Beside this, transaction also contains sales_channel_id.\n\nTransaction train data has entries for the date of the transaction, the customer id, the article id, a price (per transaction) and a sales channel id.","metadata":{}},{"cell_type":"code","source":"def missing_data(data):\n    total = data.isnull().sum().sort_values(ascending = False)\n    percent = (data.isnull().sum()/data.isnull().count()*100).sort_values(ascending = False)\n    return pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:30:53.664062Z","iopub.execute_input":"2022-03-24T05:30:53.664548Z","iopub.status.idle":"2022-03-24T05:30:53.671411Z","shell.execute_reply.started":"2022-03-24T05:30:53.664504Z","shell.execute_reply":"2022-03-24T05:30:53.670663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def unique_values(data):\n    total = data.count()\n    tt = pd.DataFrame(total)\n    tt.columns = ['Total']\n    uniques = []\n    for col in data.columns:\n        unique = data[col].nunique()\n        uniques.append(unique)\n    tt['Uniques'] = uniques\n    return tt","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:31:09.657962Z","iopub.execute_input":"2022-03-24T05:31:09.658407Z","iopub.status.idle":"2022-03-24T05:31:09.665067Z","shell.execute_reply.started":"2022-03-24T05:31:09.658350Z","shell.execute_reply":"2022-03-24T05:31:09.664452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:31:16.205279Z","iopub.execute_input":"2022-03-24T05:31:16.205603Z","iopub.status.idle":"2022-03-24T05:31:16.392262Z","shell.execute_reply.started":"2022-03-24T05:31:16.205572Z","shell.execute_reply":"2022-03-24T05:31:16.390794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_data(articles_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:31:25.237096Z","iopub.execute_input":"2022-03-24T05:31:25.237433Z","iopub.status.idle":"2022-03-24T05:31:25.734305Z","shell.execute_reply.started":"2022-03-24T05:31:25.237398Z","shell.execute_reply":"2022-03-24T05:31:25.733627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the article data, the only missing data is for the detailed description of the article (0.4% missing data).","metadata":{}},{"cell_type":"code","source":"customers_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:31:47.347437Z","iopub.execute_input":"2022-03-24T05:31:47.347714Z","iopub.status.idle":"2022-03-24T05:31:48.010893Z","shell.execute_reply.started":"2022-03-24T05:31:47.347686Z","shell.execute_reply":"2022-03-24T05:31:48.010021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_data(customers_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:31:55.131666Z","iopub.execute_input":"2022-03-24T05:31:55.132144Z","iopub.status.idle":"2022-03-24T05:31:56.987827Z","shell.execute_reply.started":"2022-03-24T05:31:55.132095Z","shell.execute_reply":"2022-03-24T05:31:56.986444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only customer id and postal code are completely filled. Age, fashion news frequency have arounfd 1% misssing data, FN has 65% missing and Active has 66% missing data.","metadata":{}},{"cell_type":"code","source":"sample_submission_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:32:26.277420Z","iopub.execute_input":"2022-03-24T05:32:26.277994Z","iopub.status.idle":"2022-03-24T05:32:26.588405Z","shell.execute_reply.started":"2022-03-24T05:32:26.277958Z","shell.execute_reply":"2022-03-24T05:32:26.587513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntransactions_train_df.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:32:32.426659Z","iopub.execute_input":"2022-03-24T05:32:32.427127Z","iopub.status.idle":"2022-03-24T05:32:32.437125Z","shell.execute_reply.started":"2022-03-24T05:32:32.427077Z","shell.execute_reply":"2022-03-24T05:32:32.436137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_data(transactions_train_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:32:40.302424Z","iopub.execute_input":"2022-03-24T05:32:40.302708Z","iopub.status.idle":"2022-03-24T05:33:01.409180Z","shell.execute_reply.started":"2022-03-24T05:32:40.302679Z","shell.execute_reply":"2022-03-24T05:33:01.408378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_values(articles_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:33:01.410978Z","iopub.execute_input":"2022-03-24T05:33:01.411825Z","iopub.status.idle":"2022-03-24T05:33:01.752091Z","shell.execute_reply.started":"2022-03-24T05:33:01.411779Z","shell.execute_reply":"2022-03-24T05:33:01.751436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nunique_values(customers_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:33:01.753155Z","iopub.execute_input":"2022-03-24T05:33:01.753654Z","iopub.status.idle":"2022-03-24T05:33:04.043819Z","shell.execute_reply.started":"2022-03-24T05:33:01.753609Z","shell.execute_reply":"2022-03-24T05:33:04.042870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nunique_values(transactions_train_df)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:33:15.762424Z","iopub.execute_input":"2022-03-24T05:33:15.762730Z","iopub.status.idle":"2022-03-24T05:33:34.873655Z","shell.execute_reply.started":"2022-03-24T05:33:15.762698Z","shell.execute_reply":"2022-03-24T05:33:34.872672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Percent of articles present in transactions: {round(104547/105542,3)*100}%\")\nprint(f\"Percent of articles present in transactions: {round(1362281/1371980,3)*100}%\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:33:47.325614Z","iopub.execute_input":"2022-03-24T05:33:47.325910Z","iopub.status.idle":"2022-03-24T05:33:47.332259Z","shell.execute_reply.started":"2022-03-24T05:33:47.325879Z","shell.execute_reply":"2022-03-24T05:33:47.331175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stopwords = set(STOPWORDS)\n\ndef show_wordcloud(data, title = None):\n    wordcloud = WordCloud(\n        background_color='white',\n        stopwords=stopwords,\n        max_words=200,\n        max_font_size=40, \n        scale=5,\n        random_state=1\n    ).generate(str(data))\n\n    fig = plt.figure(1, figsize=(10,10))\n    plt.axis('off')\n    if title: \n        fig.suptitle(title, fontsize=14)\n        fig.subplots_adjust(top=2.3)\n\n    plt.imshow(wordcloud)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:34:16.015001Z","iopub.execute_input":"2022-03-24T05:34:16.015276Z","iopub.status.idle":"2022-03-24T05:34:16.023433Z","shell.execute_reply.started":"2022-03-24T05:34:16.015247Z","shell.execute_reply":"2022-03-24T05:34:16.021980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_wordcloud(articles_df[\"prod_name\"], \"Wordcloud from product name\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:34:22.191136Z","iopub.execute_input":"2022-03-24T05:34:22.191797Z","iopub.status.idle":"2022-03-24T05:34:22.952526Z","shell.execute_reply.started":"2022-03-24T05:34:22.191752Z","shell.execute_reply":"2022-03-24T05:34:22.951496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_wordcloud(articles_df[\"detail_desc\"], \"Wordcloud from detailed description of articles\")","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:36:02.167521Z","iopub.execute_input":"2022-03-24T05:36:02.168416Z","iopub.status.idle":"2022-03-24T05:36:03.021064Z","shell.execute_reply.started":"2022-03-24T05:36:02.168369Z","shell.execute_reply":"2022-03-24T05:36:03.019744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Some Analysis of Image data","metadata":{}},{"cell_type":"markdown","source":"![](http://images.unsplash.com/photo-1575729312527-1bdecaae271e?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=687&q=80)","metadata":{}},{"cell_type":"markdown","source":"There are 105542 articles and 105100 different images. Let's check first which articles does not have corresponding images.\n\nThe article_id corresponds to digits from 2nd to the last of the image name. The digits from 2nd to 7th of image name correspond to product code (product_code).","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:37:48.110987Z","iopub.execute_input":"2022-03-24T05:37:48.111433Z","iopub.status.idle":"2022-03-24T05:37:48.119419Z","shell.execute_reply.started":"2022-03-24T05:37:48.111395Z","shell.execute_reply":"2022-03-24T05:37:48.118012Z"}}},{"cell_type":"code","source":"image_name_df = pd.DataFrame(images_names, columns = [\"image_name\"])\nimage_name_df[\"article_id\"] = image_name_df[\"image_name\"].apply(lambda x: int(x[1:]))","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:38:05.513285Z","iopub.execute_input":"2022-03-24T05:38:05.513589Z","iopub.status.idle":"2022-03-24T05:38:05.621931Z","shell.execute_reply.started":"2022-03-24T05:38:05.513560Z","shell.execute_reply":"2022-03-24T05:38:05.621153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_name_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:38:59.527417Z","iopub.execute_input":"2022-03-24T05:38:59.527860Z","iopub.status.idle":"2022-03-24T05:38:59.537729Z","shell.execute_reply.started":"2022-03-24T05:38:59.527829Z","shell.execute_reply":"2022-03-24T05:38:59.536961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_article_df = articles_df[[\"article_id\", \"product_code\", \"product_group_name\", \"product_type_name\"]].merge(image_name_df, on=[\"article_id\"], how=\"left\")\nprint(image_article_df.shape)\nimage_article_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:39:23.779737Z","iopub.execute_input":"2022-03-24T05:39:23.780022Z","iopub.status.idle":"2022-03-24T05:39:23.844318Z","shell.execute_reply.started":"2022-03-24T05:39:23.779994Z","shell.execute_reply":"2022-03-24T05:39:23.843274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Products without images.","metadata":{}},{"cell_type":"code","source":"article_no_image_df = image_article_df.loc[image_article_df.image_name.isna()]\nprint(article_no_image_df.shape)\narticle_no_image_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:40:06.888228Z","iopub.execute_input":"2022-03-24T05:40:06.888645Z","iopub.status.idle":"2022-03-24T05:40:06.924111Z","shell.execute_reply.started":"2022-03-24T05:40:06.888604Z","shell.execute_reply":"2022-03-24T05:40:06.923122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Product codes with some missing images: \", article_no_image_df.product_code.nunique())\nprint(\"Product groups with some missing images: \", list(article_no_image_df.product_group_name.unique()))","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:40:20.604094Z","iopub.execute_input":"2022-03-24T05:40:20.604756Z","iopub.status.idle":"2022-03-24T05:40:20.611645Z","shell.execute_reply.started":"2022-03-24T05:40:20.604718Z","shell.execute_reply":"2022-03-24T05:40:20.610749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize few images.","metadata":{}},{"cell_type":"code","source":"def plot_image_samples(image_article_df, product_group_name, cols=1, rows=-1):\n    image_path = \"/kaggle/input/h-and-m-personalized-fashion-recommendations/images/\"\n    _df = image_article_df.loc[image_article_df.product_group_name==product_group_name]\n    article_ids = _df.article_id.values[0:cols*rows]\n    plt.figure(figsize=(2 + 3 * cols, 2 + 4 * rows))\n    for i in range(cols * rows):\n        article_id = (\"0\" + str(article_ids[i]))[-10:]\n        plt.subplot(rows, cols, i + 1)\n        plt.axis('off')\n        plt.title(f\"{product_group_name} {article_id[:3]}\\n{article_id}.jpg\")\n        image = Image.open(f\"{image_path}{article_id[:3]}/{article_id}.jpg\")\n        plt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:40:42.446340Z","iopub.execute_input":"2022-03-24T05:40:42.447201Z","iopub.status.idle":"2022-03-24T05:40:42.455099Z","shell.execute_reply.started":"2022-03-24T05:40:42.447146Z","shell.execute_reply":"2022-03-24T05:40:42.453953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(image_article_df.product_group_name.unique())","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:40:52.891772Z","iopub.execute_input":"2022-03-24T05:40:52.892474Z","iopub.status.idle":"2022-03-24T05:40:52.906398Z","shell.execute_reply.started":"2022-03-24T05:40:52.892438Z","shell.execute_reply":"2022-03-24T05:40:52.905587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We will represent images grouped on product group name.**","metadata":{}},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Garment Lower body\", 4, 2)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:41:04.920483Z","iopub.execute_input":"2022-03-24T05:41:04.920891Z","iopub.status.idle":"2022-03-24T05:41:07.787577Z","shell.execute_reply.started":"2022-03-24T05:41:04.920860Z","shell.execute_reply":"2022-03-24T05:41:07.786866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Stationery\", 4, 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:41:15.242424Z","iopub.execute_input":"2022-03-24T05:41:15.242834Z","iopub.status.idle":"2022-03-24T05:41:17.083498Z","shell.execute_reply.started":"2022-03-24T05:41:15.242795Z","shell.execute_reply":"2022-03-24T05:41:17.082608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Fun\", 2, 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:41:44.780331Z","iopub.execute_input":"2022-03-24T05:41:44.780623Z","iopub.status.idle":"2022-03-24T05:41:45.790977Z","shell.execute_reply.started":"2022-03-24T05:41:44.780594Z","shell.execute_reply":"2022-03-24T05:41:45.790182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Accessories\", 4, 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:41:54.464501Z","iopub.execute_input":"2022-03-24T05:41:54.464999Z","iopub.status.idle":"2022-03-24T05:41:56.809043Z","shell.execute_reply.started":"2022-03-24T05:41:54.464966Z","shell.execute_reply":"2022-03-24T05:41:56.807988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Swimwear\", 4, 2)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:42:04.466385Z","iopub.execute_input":"2022-03-24T05:42:04.466657Z","iopub.status.idle":"2022-03-24T05:42:07.960491Z","shell.execute_reply.started":"2022-03-24T05:42:04.466628Z","shell.execute_reply":"2022-03-24T05:42:07.959449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Furniture\", 4, 2)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:42:17.007768Z","iopub.execute_input":"2022-03-24T05:42:17.008043Z","iopub.status.idle":"2022-03-24T05:42:20.548741Z","shell.execute_reply.started":"2022-03-24T05:42:17.008014Z","shell.execute_reply":"2022-03-24T05:42:20.547987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Cosmetic\", 4, 1)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:42:23.946028Z","iopub.execute_input":"2022-03-24T05:42:23.946889Z","iopub.status.idle":"2022-03-24T05:42:26.119498Z","shell.execute_reply.started":"2022-03-24T05:42:23.946838Z","shell.execute_reply":"2022-03-24T05:42:26.118739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_image_samples(image_article_df, \"Bags\", 4, 3)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:42:33.565111Z","iopub.execute_input":"2022-03-24T05:42:33.565429Z","iopub.status.idle":"2022-03-24T05:42:38.772155Z","shell.execute_reply.started":"2022-03-24T05:42:33.565393Z","shell.execute_reply":"2022-03-24T05:42:38.771476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"markdown","source":"For this submission, we apply the following simplified logic:\n\n* if there are articles for a certain client, pick the most recent buys;\n* if there are not articles for a certain client, just pick the most frequently buyed articles.","metadata":{}},{"cell_type":"code","source":"transactions_train_df = transactions_train_df.sort_values([\"customer_id\", \"t_dat\"], ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:46:56.975134Z","iopub.execute_input":"2022-03-24T05:46:56.975460Z","iopub.status.idle":"2022-03-24T05:47:26.980814Z","shell.execute_reply.started":"2022-03-24T05:46:56.975429Z","shell.execute_reply":"2022-03-24T05:47:26.979673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions_train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:47:26.982931Z","iopub.execute_input":"2022-03-24T05:47:26.983220Z","iopub.status.idle":"2022-03-24T05:47:26.997951Z","shell.execute_reply.started":"2022-03-24T05:47:26.983187Z","shell.execute_reply":"2022-03-24T05:47:26.996832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Find first what are the most frequent recently bought articles.","metadata":{}},{"cell_type":"code","source":"last_date = transactions_train_df.t_dat.max()\nprint(last_date)\nprint(transactions_train_df.loc[transactions_train_df.t_dat==last_date].shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:50:30.922386Z","iopub.execute_input":"2022-03-24T05:50:30.922903Z","iopub.status.idle":"2022-03-24T05:50:41.714728Z","shell.execute_reply.started":"2022-03-24T05:50:30.922860Z","shell.execute_reply":"2022-03-24T05:50:41.713742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"most_frequent_articles = list(transactions_train_df.loc[transactions_train_df.t_dat==last_date].article_id.value_counts()[0:12].index)\nart_list = []\nfor art in most_frequent_articles:\n    art = \"0\"+str(art)\n    art_list.append(art)\nart_str = \" \".join(art_list)\nprint(\"Frequent articles bought recently: \", art_str)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:50:42.432718Z","iopub.execute_input":"2022-03-24T05:50:42.433426Z","iopub.status.idle":"2022-03-24T05:50:47.547331Z","shell.execute_reply.started":"2022-03-24T05:50:42.433223Z","shell.execute_reply":"2022-03-24T05:50:47.546597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"agg_df = transactions_train_df.groupby([\"customer_id\"])[\"article_id\"].agg(lambda x: str(x.values[0:12])[1:-1]).reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:50:50.625021Z","iopub.execute_input":"2022-03-24T05:50:50.625305Z","iopub.status.idle":"2022-03-24T05:52:39.078998Z","shell.execute_reply.started":"2022-03-24T05:50:50.625276Z","shell.execute_reply":"2022-03-24T05:52:39.077780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def padding_articles(x):\n    if x:\n        xl = x.split()\n        x = []\n        for xi in xl:\n            x.append(\"0\"+xi)\n        dimm_x = len(x)\n        if dimm_x < 12:\n            x.extend(art_list[:12-dimm_x])\n        return(\" \".join(x))","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:39.082216Z","iopub.execute_input":"2022-03-24T05:52:39.082635Z","iopub.status.idle":"2022-03-24T05:52:39.089238Z","shell.execute_reply.started":"2022-03-24T05:52:39.082589Z","shell.execute_reply":"2022-03-24T05:52:39.088038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"agg_df[\"article_id\"] = agg_df[\"article_id\"].apply(lambda x: padding_articles(x))\n","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:39.091203Z","iopub.execute_input":"2022-03-24T05:52:39.091600Z","iopub.status.idle":"2022-03-24T05:52:43.564625Z","shell.execute_reply.started":"2022-03-24T05:52:39.091558Z","shell.execute_reply":"2022-03-24T05:52:43.563340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Aggregated transaction history: \", agg_df.customer_id.nunique())\nprint(\"Submission sample: \", sample_submission_df.customer_id.nunique())","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:43.567255Z","iopub.execute_input":"2022-03-24T05:52:43.567508Z","iopub.status.idle":"2022-03-24T05:52:45.559322Z","shell.execute_reply.started":"2022-03-24T05:52:43.567479Z","shell.execute_reply":"2022-03-24T05:52:45.558448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sample_submission_df.shape)\nsample_submission_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:45.560839Z","iopub.execute_input":"2022-03-24T05:52:45.561119Z","iopub.status.idle":"2022-03-24T05:52:45.779957Z","shell.execute_reply.started":"2022-03-24T05:52:45.561089Z","shell.execute_reply":"2022-03-24T05:52:45.778850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df = agg_df.merge(sample_submission_df[[\"customer_id\"]], how=\"right\")\nsubmission_df.columns = [\"customer_id\", \"prediction\"]\nprint(submission_df.shape)\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:45.781376Z","iopub.execute_input":"2022-03-24T05:52:45.781616Z","iopub.status.idle":"2022-03-24T05:52:48.124494Z","shell.execute_reply.started":"2022-03-24T05:52:45.781588Z","shell.execute_reply":"2022-03-24T05:52:48.123572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Rows with missing data in submission: \", submission_df.loc[submission_df.prediction.isna()].shape[0])","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:48.126142Z","iopub.execute_input":"2022-03-24T05:52:48.126585Z","iopub.status.idle":"2022-03-24T05:52:48.445295Z","shell.execute_reply.started":"2022-03-24T05:52:48.126553Z","shell.execute_reply":"2022-03-24T05:52:48.444448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.loc[submission_df.prediction.isna(), [\"prediction\"]] = art_str","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:52:48.446549Z","iopub.execute_input":"2022-03-24T05:52:48.446784Z","iopub.status.idle":"2022-03-24T05:52:48.613379Z","shell.execute_reply.started":"2022-03-24T05:52:48.446755Z","shell.execute_reply":"2022-03-24T05:52:48.612315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Rows with missing data in submission: \", submission_df.loc[submission_df.prediction.isna()].shape[0])","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:54:07.089960Z","iopub.execute_input":"2022-03-24T05:54:07.090414Z","iopub.status.idle":"2022-03-24T05:54:07.266083Z","shell.execute_reply.started":"2022-03-24T05:54:07.090355Z","shell.execute_reply":"2022-03-24T05:54:07.264876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-24T05:54:18.865620Z","iopub.execute_input":"2022-03-24T05:54:18.866272Z","iopub.status.idle":"2022-03-24T05:54:32.321287Z","shell.execute_reply.started":"2022-03-24T05:54:18.866232Z","shell.execute_reply":"2022-03-24T05:54:32.320093Z"},"trusted":true},"execution_count":null,"outputs":[]}]}