{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M Data Exploration\n\nIn this notebook we investigate the categorical data in articles table. The main objective is to get some info and prepare the categorical (ex. encode product name) data to be processed by ML technique. Since the categorical data needs to be encoded, here we encode the product name of articles (`prod_name` column) in five numerical columns.","metadata":{"execution":{"iopub.execute_input":"2022-02-11T16:30:42.175208Z","iopub.status.busy":"2022-02-11T16:30:42.174423Z","iopub.status.idle":"2022-02-11T16:30:42.178615Z","shell.execute_reply":"2022-02-11T16:30:42.178041Z","shell.execute_reply.started":"2022-02-11T16:30:42.17516Z"}}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:09:13.804320Z","iopub.execute_input":"2022-02-19T22:09:13.805116Z","iopub.status.idle":"2022-02-19T22:09:13.828777Z","shell.execute_reply.started":"2022-02-19T22:09:13.804987Z","shell.execute_reply":"2022-02-19T22:09:13.828146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/articles.csv')\ntrans = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\ncustomer  = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/customers.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:09:13.829901Z","iopub.execute_input":"2022-02-19T22:09:13.830519Z","iopub.status.idle":"2022-02-19T22:10:31.150934Z","shell.execute_reply.started":"2022-02-19T22:09:13.830486Z","shell.execute_reply":"2022-02-19T22:10:31.149616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data info\n\nAn overview of the data we are dealing with. We are interested in the string columns of article table.","metadata":{}},{"cell_type":"code","source":"# Data size\nprint(f\"articles.shape = {articles.shape}\")\nprint(f\"trans.shape = {trans.shape}\")\nprint(f\"customer.shape = {customer.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.153494Z","iopub.execute_input":"2022-02-19T22:10:31.153752Z","iopub.status.idle":"2022-02-19T22:10:31.162090Z","shell.execute_reply.started":"2022-02-19T22:10:31.153722Z","shell.execute_reply":"2022-02-19T22:10:31.161034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Some info\narticles.head().T","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.163586Z","iopub.execute_input":"2022-02-19T22:10:31.163819Z","iopub.status.idle":"2022-02-19T22:10:31.204669Z","shell.execute_reply.started":"2022-02-19T22:10:31.163790Z","shell.execute_reply":"2022-02-19T22:10:31.203673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# object type tell us the column is a string data\narticles.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.206816Z","iopub.execute_input":"2022-02-19T22:10:31.207066Z","iopub.status.idle":"2022-02-19T22:10:31.218067Z","shell.execute_reply.started":"2022-02-19T22:10:31.207036Z","shell.execute_reply":"2022-02-19T22:10:31.217059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can notice that product names repeat. Lets see how many unique names are there.","metadata":{"execution":{"iopub.status.busy":"2022-02-15T22:27:42.442168Z","iopub.execute_input":"2022-02-15T22:27:42.442828Z","iopub.status.idle":"2022-02-15T22:27:42.449848Z","shell.execute_reply.started":"2022-02-15T22:27:42.44278Z","shell.execute_reply":"2022-02-15T22:27:42.448524Z"}}},{"cell_type":"code","source":"# Unique product names\nuniqProdNameCnt = len(articles.loc[:, 'prod_name'].unique())\n# Percent of unique product names\nuniqProdNamePercent = 100*uniqProdNameCnt/articles.shape[0]\nprint(f\"Percent of unique product names: {uniqProdNamePercent:.3f}%\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.219579Z","iopub.execute_input":"2022-02-19T22:10:31.219833Z","iopub.status.idle":"2022-02-19T22:10:31.252989Z","shell.execute_reply.started":"2022-02-19T22:10:31.219805Z","shell.execute_reply":"2022-02-19T22:10:31.252033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Relation between `product_code` and `prod_name`\n\nIf there is correspondence between `product_code` and `prod_name` columns, then we already have the product name encoded. Lets check how many different names have the same product code.","metadata":{}},{"cell_type":"code","source":"print(f\"Number of unique product codes: {len(articles.loc[:, 'product_code'].unique())}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.254247Z","iopub.execute_input":"2022-02-19T22:10:31.254569Z","iopub.status.idle":"2022-02-19T22:10:31.265687Z","shell.execute_reply.started":"2022-02-19T22:10:31.254525Z","shell.execute_reply":"2022-02-19T22:10:31.264610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Number of unique product codes: {len(articles.loc[:, 'prod_name'].unique())}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.267734Z","iopub.execute_input":"2022-02-19T22:10:31.268949Z","iopub.status.idle":"2022-02-19T22:10:31.290925Z","shell.execute_reply.started":"2022-02-19T22:10:31.268902Z","shell.execute_reply":"2022-02-19T22:10:31.290073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets see which different names have the same code, and what are the names that have the same code.","metadata":{}},{"cell_type":"code","source":"diffNameList = []\nfor code in articles['product_code'].unique():\n    apnlist = articles[articles.loc[:, 'product_code'] == code]['prod_name']\n    if len(apnlist.unique()) > 1:\n        diffNameList.append(apnlist.unique())\nprint(f\"There are : {len(diffNameList)} codes that encode more than one prod_name\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:10:31.292383Z","iopub.execute_input":"2022-02-19T22:10:31.292837Z","iopub.status.idle":"2022-02-19T22:11:01.724316Z","shell.execute_reply.started":"2022-02-19T22:10:31.292806Z","shell.execute_reply":"2022-02-19T22:11:01.723382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Different names encoded with the same product_code:\\n\")\nfor i in range(20):\n    print(diffNameList[i])","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:01.725406Z","iopub.execute_input":"2022-02-19T22:11:01.725631Z","iopub.status.idle":"2022-02-19T22:11:01.736355Z","shell.execute_reply.started":"2022-02-19T22:11:01.725604Z","shell.execute_reply":"2022-02-19T22:11:01.735404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, we can conclude that we can use the `product_code` for encoding the `prod_name` categorical data. Next we move to some data exploration to see if we can extract some more data from the product name before discarding it.","metadata":{"execution":{"iopub.status.busy":"2022-02-15T22:40:26.729696Z","iopub.execute_input":"2022-02-15T22:40:26.730771Z","iopub.status.idle":"2022-02-15T22:40:26.737956Z","shell.execute_reply.started":"2022-02-15T22:40:26.730667Z","shell.execute_reply":"2022-02-15T22:40:26.736638Z"}}},{"cell_type":"markdown","source":"# Word processing\n\nWe want to create lists of most used words for column of product name. First we want to lowercase all\nof the strings. Generally we can apply the lowercase only to columns of 'object' type since these are the columns\nwith string data.","metadata":{}},{"cell_type":"code","source":"# Lowercase all of the following articles columns\nartStrCols = articles.dtypes[articles.dtypes == np.dtype(object)]\nprint(artStrCols.keys())\nfor col in artStrCols.keys():\n    articles.loc[:, col] = articles.loc[:, col].str.lower()    ","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:01.739388Z","iopub.execute_input":"2022-02-19T22:11:01.739643Z","iopub.status.idle":"2022-02-19T22:11:02.623411Z","shell.execute_reply.started":"2022-02-19T22:11:01.739608Z","shell.execute_reply":"2022-02-19T22:11:02.622436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.head().T","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:02.624653Z","iopub.execute_input":"2022-02-19T22:11:02.624908Z","iopub.status.idle":"2022-02-19T22:11:02.641829Z","shell.execute_reply.started":"2022-02-19T22:11:02.624879Z","shell.execute_reply":"2022-02-19T22:11:02.641220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the number of unique columns reduced","metadata":{}},{"cell_type":"code","source":"# Unique product names\nuniqProdNameCnt = len(articles.loc[:, 'prod_name'].unique())\n# Percent of unique product names\nuniqProdNamePercent = 100*uniqProdNameCnt/articles.shape[0]\nprint(f\"Percent of unique product names: {uniqProdNamePercent:.3f}%\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:02.642950Z","iopub.execute_input":"2022-02-19T22:11:02.643225Z","iopub.status.idle":"2022-02-19T22:11:02.673211Z","shell.execute_reply.started":"2022-02-19T22:11:02.643196Z","shell.execute_reply":"2022-02-19T22:11:02.672438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets check some regex expressions for extracting words from the strings.","metadata":{}},{"cell_type":"code","source":"import re\n\n# Lets take a speciffic word\nstrOne = articles.loc[3, 'prod_name']\nprint(f\"word: '{strOne}'\")\n# using regex( findall() ) to extract words from string\nres = re.findall(r'\\w+', strOne)\nprint(f\"findall regex1: {res}\")\nres = re.findall(r'\\b[0-9A-Za-z\\-]+', strOne)\nprint(f\"findall regex2: {res}\")\n# Regex 2 is better since it treats '-' as part of words\nregexStr = r'\\b[0-9A-Za-z\\-]+'","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:02.674425Z","iopub.execute_input":"2022-02-19T22:11:02.674765Z","iopub.status.idle":"2022-02-19T22:11:02.688012Z","shell.execute_reply.started":"2022-02-19T22:11:02.674734Z","shell.execute_reply":"2022-02-19T22:11:02.687182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets check how does our chosen regex splits strings.","metadata":{}},{"cell_type":"code","source":"prodNameStrList = articles.loc[:, 'prod_name']\nprodNameWordListList = prodNameStrList.str.findall(regexStr)\nprint(f\"string series:\\n{prodNameStrList.head()}\\n\")\nprint(f\"word series:\\n{prodNameWordListList.head()}\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:02.689473Z","iopub.execute_input":"2022-02-19T22:11:02.689859Z","iopub.status.idle":"2022-02-19T22:11:03.193537Z","shell.execute_reply.started":"2022-02-19T22:11:02.689817Z","shell.execute_reply":"2022-02-19T22:11:03.192708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we make histogram of most used words for prod_name, since we have a Series of lists for the \nword, first we need to flat that out. After flatting we can draw the histogram.","metadata":{}},{"cell_type":"code","source":"prodNameWordList = prodNameWordListList.apply(pd.Series).stack().reset_index(drop=True)\nprodNameWordList.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:03.194708Z","iopub.execute_input":"2022-02-19T22:11:03.195089Z","iopub.status.idle":"2022-02-19T22:11:26.872504Z","shell.execute_reply.started":"2022-02-19T22:11:03.195055Z","shell.execute_reply":"2022-02-19T22:11:26.871539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prodNameWordCount = pd.value_counts(prodNameWordList)\nprint(prodNameWordCount[0:40])\nprodNameWordCount[0:40].plot(kind='bar', figsize=(20,10))","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:31:04.312744Z","iopub.execute_input":"2022-02-19T22:31:04.313054Z","iopub.status.idle":"2022-02-19T22:31:04.880736Z","shell.execute_reply.started":"2022-02-19T22:31:04.313010Z","shell.execute_reply":"2022-02-19T22:31:04.880015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Histogram of word count in a string column\n\nFor later use it will be good to know what is the distribution of word count in a string column. This will help us to choose the best number of columns to add for encoding the string.","metadata":{}},{"cell_type":"code","source":"wordCount = pd.Series(data=0, dtype=int, index=[i for i in range(len(prodNameWordListList))])\nfor i in range(len(prodNameWordListList)):\n    wordCount.loc[i] = len(prodNameWordListList[i])","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:34:20.989018Z","iopub.execute_input":"2022-02-19T22:34:20.989611Z","iopub.status.idle":"2022-02-19T22:34:27.151874Z","shell.execute_reply.started":"2022-02-19T22:34:20.989573Z","shell.execute_reply":"2022-02-19T22:34:27.150843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordDist = pd.value_counts(wordCount, normalize=True).sort_index()\nwordDist","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:46:34.111576Z","iopub.execute_input":"2022-02-19T22:46:34.111952Z","iopub.status.idle":"2022-02-19T22:46:34.122795Z","shell.execute_reply.started":"2022-02-19T22:46:34.111912Z","shell.execute_reply":"2022-02-19T22:46:34.122091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordDist.plot(kind='bar', figsize=(20,10))","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:48:02.989693Z","iopub.execute_input":"2022-02-19T22:48:02.990034Z","iopub.status.idle":"2022-02-19T22:48:03.249139Z","shell.execute_reply.started":"2022-02-19T22:48:02.989998Z","shell.execute_reply":"2022-02-19T22:48:03.247999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We use the cumulative sum of word distribution to select how many columns we will use for encoding. For example, if we want to cover more than 95% of string lengths we should use 5 columns for encoding as is shown in the following example.","metadata":{}},{"cell_type":"code","source":"wordDistCumsum = wordDist.cumsum()\nwordDistCumsum","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:52:06.257328Z","iopub.execute_input":"2022-02-19T22:52:06.258226Z","iopub.status.idle":"2022-02-19T22:52:06.265875Z","shell.execute_reply.started":"2022-02-19T22:52:06.258173Z","shell.execute_reply":"2022-02-19T22:52:06.265138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wordDistCumsum[wordDistCumsum > 0.95].index.min()","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:54:23.701687Z","iopub.execute_input":"2022-02-19T22:54:23.702167Z","iopub.status.idle":"2022-02-19T22:54:23.709212Z","shell.execute_reply.started":"2022-02-19T22:54:23.702109Z","shell.execute_reply":"2022-02-19T22:54:23.708496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Encode string data\n\nHere we construct new articles table consisted only of numerical data. We show how you can encode the product name and additionaly extract some data that is not present in the `product_code`.","metadata":{}},{"cell_type":"code","source":"print(\"Create new table with data from numerical columns:\\n\")\nartNumCols = articles.dtypes[articles.dtypes == np.dtype(np.int64)]\narticlesCoded = pd.DataFrame({key:articles.loc[:, key] for key in artNumCols.keys()})\narticlesCoded.head().T","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:27.998750Z","iopub.execute_input":"2022-02-19T22:11:27.999240Z","iopub.status.idle":"2022-02-19T22:11:28.018074Z","shell.execute_reply.started":"2022-02-19T22:11:27.999193Z","shell.execute_reply":"2022-02-19T22:11:28.017069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create DataFrame for encoding all words that appear in the prod_column","metadata":{}},{"cell_type":"code","source":"code = [i for i in range(len(prodNameWordCount))]\nprodName = pd.DataFrame({'code': code, 'count': prodNameWordCount.values}, index=prodNameWordCount.index)\nprodName.iloc[0:10, :]","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:28.019369Z","iopub.execute_input":"2022-02-19T22:11:28.019767Z","iopub.status.idle":"2022-02-19T22:11:28.038845Z","shell.execute_reply.started":"2022-02-19T22:11:28.019733Z","shell.execute_reply":"2022-02-19T22:11:28.038132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We add five more columns to the numeric articles table in wich we encode words from the `prod_name`. If there are no words, we put `None` and we take care to first sort words in order of frequency of appearance in the `prod_name` column. When we sort words by frequency we favor words that are more common (if the prod_name cosists of more than five words). If we use inverse sorting we will favor words that are more speciffic to the product name.\n\nLets see how words are ordered by the code value (smaller the code value - word is more frequent)","metadata":{}},{"cell_type":"code","source":"alists = articles.loc[0:9, 'prod_name'].str.findall(regexStr)\nprint(f\"alists:\\n{alists}\\n\")\nprint(f\"alists[0].code.sort:\\n{prodName.loc[alists[0], 'code'].sort_values()[0:5]}\\n\")\nprint(f\"alists[8].code.sort:\\n{prodName.loc[alists[8], 'code'].sort_values()[0:5]}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:28.040177Z","iopub.execute_input":"2022-02-19T22:11:28.040414Z","iopub.status.idle":"2022-02-19T22:11:28.060170Z","shell.execute_reply.started":"2022-02-19T22:11:28.040385Z","shell.execute_reply":"2022-02-19T22:11:28.059047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we use the above encoding to add the prod_name words to the articlesCoded dataframe","metadata":{}},{"cell_type":"code","source":"alists = articles.loc[:, 'prod_name'].str.findall(regexStr)\nfor i in range(5):\n    colName = f'prod_name_{i}'\n    valList = [] \n    for j in range(len(alists)):\n        value = prodName.loc[alists[j], 'code'].sort_values()[i] if i < len(alists[j]) else None \n        valList.append(value)\n    articlesCoded[colName] = valList","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:11:28.061571Z","iopub.execute_input":"2022-02-19T22:11:28.061828Z","iopub.status.idle":"2022-02-19T22:14:29.084659Z","shell.execute_reply.started":"2022-02-19T22:11:28.061798Z","shell.execute_reply":"2022-02-19T22:14:29.083398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Voila! We created additional five columns of meaningful data, encoded and based on a categorical column that we can now dicard.","metadata":{}},{"cell_type":"code","source":"articlesCoded.loc[0:8].T","metadata":{"execution":{"iopub.status.busy":"2022-02-19T22:14:29.086177Z","iopub.execute_input":"2022-02-19T22:14:29.086433Z","iopub.status.idle":"2022-02-19T22:14:29.118873Z","shell.execute_reply.started":"2022-02-19T22:14:29.086406Z","shell.execute_reply":"2022-02-19T22:14:29.117862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next you can use the same method for all categorical columns with strings and prepare table for XGBoost, for example.","metadata":{}}]}