{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nimport matplotlib.ticker as ticker\nfrom tqdm.notebook import tqdm\nfrom datetime import datetime","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:10:52.593806Z","iopub.execute_input":"2023-05-18T22:10:52.594159Z","iopub.status.idle":"2023-05-18T22:10:53.237082Z","shell.execute_reply.started":"2023-05-18T22:10:52.594133Z","shell.execute_reply":"2023-05-18T22:10:53.235969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', None)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:10:53.239398Z","iopub.execute_input":"2023-05-18T22:10:53.239827Z","iopub.status.idle":"2023-05-18T22:10:53.245929Z","shell.execute_reply.started":"2023-05-18T22:10:53.239790Z","shell.execute_reply":"2023-05-18T22:10:53.244644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Functions\n# TODO: Add functions for plotting\n\ndef convert_to_category(df: pd.DataFrame, columns:list):\n    # Convert specified columns to category data type\n    df_new = df[columns].astype('category')\n    return df_new\n\ndef select_object_columns(df: pd.DataFrame, col_type:list):\n    # Select columns of specific type\n    object_columns = df.select_dtypes(include=col_type)\n    return object_columns\n\n\ndef drop_columns(df: pd.DataFrame, columns:list):\n    # Drop specified columns\n    return df.drop(columns=columns)\n\n\ndef count_unique_values(df: pd.DataFrame):\n    # Count unique values in each column\n    unique_counts = df.nunique()\n    return unique_counts","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Task: Predict what articles each customer will purchase in the 7-day period immediately after the training data ends. Customers who did not make any purchase during that time are excluded from the scoring.\n\n# Business case\n\nWe need to answer several questions in order to understand purchasing habits and to identify the key factors that will drive short-term purchasing decision. This will help us identify more important features for prediction. \n\n1. How interested in fashion are customers who purchase products?\n1. Do specific product fabric patterns and colors affect purchasing decision?\n1. What are the most purchased products?\n1. What is the effect of price on purchasing?\n1. Is club membership more likely to lead to purchase?\n1. Which sales chanel lead to more purchases?","metadata":{}},{"cell_type":"markdown","source":"# Data understanding\n\nAt this stage we will identify key information about the data we have at our disposal. We will explore the size and structure of the dataset by answering the following questions:\n\n- How big dataset is, in volume? and How many rows and columns?\n- How much (if any) data is missing?\n- How many columns have text, numeric or categorical values?\n- Are there columns that can be removed (for example, if they hold irrelevant information)?\n\nIn **Data preparation** stage we will deal with all these elements:\n - check if due to missing data we need to remove rows\n - convert categorical column values to numbers\n - preprocess text columns if needed\n   - vectorize text columns\n - remove columns that have no useful value for the prediction process","metadata":{}},{"cell_type":"code","source":"articles = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ntransactions = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:10:53.247060Z","iopub.execute_input":"2023-05-18T22:10:53.247644Z","iopub.status.idle":"2023-05-18T22:12:00.239323Z","shell.execute_reply.started":"2023-05-18T22:10:53.247607Z","shell.execute_reply":"2023-05-18T22:12:00.238437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Articles\n\n> * **article_id** : A unique identifier of every article.\n> * **product_code**, **prod_name** : A unique identifier of every product and its name\n> * **product_type**, **product_type_name** : The group of product_code and its name\n> * **graphical_appearance_no**, **graphical_appearance_name** : The group of graphics and its name\n> * **colour_group_code**, **colour_group_name** : The group of color and its name\n> * **perceived_colour_value_id**, **perceived_colour_value_name**, **perceived_colour_master_id**, **perceived_colour_master_name** : The added color info\n> * **department_no**, **department_name**: : A unique identifier of every dep and its name\n> * **index_code**, **index_name**: : A unique identifier of every index and its name\n> * **index_group_no**, **index_group_name**: : A group of indices and its name\n> * **section_no**, **section_name**: : A unique identifier of every section and its name\n> * **garment_group_no**, **garment_group_name**: : A unique identifier of every garment and its name\n> * **detail_desc**: : Details","metadata":{}},{"cell_type":"code","source":"articles.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.240339Z","iopub.execute_input":"2023-05-18T22:12:00.240865Z","iopub.status.idle":"2023-05-18T22:12:00.278342Z","shell.execute_reply.started":"2023-05-18T22:12:00.240835Z","shell.execute_reply":"2023-05-18T22:12:00.277278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.280789Z","iopub.execute_input":"2023-05-18T22:12:00.281080Z","iopub.status.idle":"2023-05-18T22:12:00.287874Z","shell.execute_reply.started":"2023-05-18T22:12:00.281056Z","shell.execute_reply":"2023-05-18T22:12:00.286216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.289558Z","iopub.execute_input":"2023-05-18T22:12:00.289956Z","iopub.status.idle":"2023-05-18T22:12:00.562285Z","shell.execute_reply.started":"2023-05-18T22:12:00.289920Z","shell.execute_reply":"2023-05-18T22:12:00.560908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Percentage of null values in a column: \", (100 * articles['detail_desc'].isna().sum() / articles.shape[0]).round(4))","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.563556Z","iopub.execute_input":"2023-05-18T22:12:00.563994Z","iopub.status.idle":"2023-05-18T22:12:00.577450Z","shell.execute_reply.started":"2023-05-18T22:12:00.563966Z","shell.execute_reply":"2023-05-18T22:12:00.576390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Out of 105 542 records there are 416 missing values in detail description, which is less than a half percent of all records. We would be able to remove these rows.","metadata":{}},{"cell_type":"code","source":"articles.apply(lambda x: x.nunique())","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.578780Z","iopub.execute_input":"2023-05-18T22:12:00.579090Z","iopub.status.idle":"2023-05-18T22:12:00.726181Z","shell.execute_reply.started":"2023-05-18T22:12:00.579064Z","shell.execute_reply":"2023-05-18T22:12:00.725187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# groups and returns count of appearance\ndef find_values_with_multiple_references(df, column2, column3):\n    grouped = df.groupby(column2)[column3].nunique()\n    result = df[df[column2]\n                .isin(grouped[grouped > 1].index)] \\\n        .groupby([column2, column3]) \\\n        .size() \\\n        .reset_index(name='count')\n    \n    return result","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.727359Z","iopub.execute_input":"2023-05-18T22:12:00.727677Z","iopub.status.idle":"2023-05-18T22:12:00.732658Z","shell.execute_reply.started":"2023-05-18T22:12:00.727651Z","shell.execute_reply":"2023-05-18T22:12:00.731914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now let's check some columns\nprint(find_values_with_multiple_references(articles, 'product_type_name', 'product_type_no'))","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.733617Z","iopub.execute_input":"2023-05-18T22:12:00.734262Z","iopub.status.idle":"2023-05-18T22:12:00.773302Z","shell.execute_reply.started":"2023-05-18T22:12:00.734235Z","shell.execute_reply":"2023-05-18T22:12:00.772352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles[articles['product_type_name'] == 'Umbrella'].head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.774470Z","iopub.execute_input":"2023-05-18T22:12:00.774868Z","iopub.status.idle":"2023-05-18T22:12:00.806363Z","shell.execute_reply.started":"2023-05-18T22:12:00.774841Z","shell.execute_reply":"2023-05-18T22:12:00.805177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles[(articles['department_name'] == 'Accessories') & (articles['department_no'] == 3510)].head(1)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.807955Z","iopub.execute_input":"2023-05-18T22:12:00.808532Z","iopub.status.idle":"2023-05-18T22:12:00.835035Z","shell.execute_reply.started":"2023-05-18T22:12:00.808492Z","shell.execute_reply":"2023-05-18T22:12:00.834079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles[(articles['department_name'] == 'Accessories') & (articles['department_no'] == 3941)].head(1)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.836182Z","iopub.execute_input":"2023-05-18T22:12:00.836546Z","iopub.status.idle":"2023-05-18T22:12:00.864432Z","shell.execute_reply.started":"2023-05-18T22:12:00.836522Z","shell.execute_reply":"2023-05-18T22:12:00.863709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(find_values_with_multiple_references(articles, 'section_name', 'section_no'))","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.868839Z","iopub.execute_input":"2023-05-18T22:12:00.869328Z","iopub.status.idle":"2023-05-18T22:12:00.897411Z","shell.execute_reply.started":"2023-05-18T22:12:00.869302Z","shell.execute_reply":"2023-05-18T22:12:00.896360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles[articles['section_name'] == 'Ladies Other']","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.898577Z","iopub.execute_input":"2023-05-18T22:12:00.898893Z","iopub.status.idle":"2023-05-18T22:12:00.927444Z","shell.execute_reply.started":"2023-05-18T22:12:00.898868Z","shell.execute_reply":"2023-05-18T22:12:00.926395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the cases above we see that the difference between the count of values in the following group of pairs is not due to any error but because of differences in other features:\n- _'product_type_name', 'product_type_no'_ - effect here has the different department the product is listed in\n- _'department_name', 'department_no'_ - in this case the section is different\n- _'section_name', 'section_no'_ - and here we have different product type and garment group, etc.\n\nThe biggest difference between columns `product_code` and `product_name` is also due to the fact that there are various other conditions that affect the code number, not just the name.\n\nIn all other cases we can consider the id/no as numerical representation of the category.","metadata":{}},{"cell_type":"code","source":"# Convert multiple columns to category type\ncolumns_to_convert = ['graphical_appearance_name', 'colour_group_name', 'perceived_colour_value_name', 'perceived_colour_master_name',\n                     'index_name', 'index_group_name', 'garment_group_name']\n# Assuming 'df' is your DataFrame and 'columns' is a list of column names\narticles[columns_to_convert] = convert_to_category(articles, columns=columns_to_convert)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:00.934435Z","iopub.execute_input":"2023-05-18T22:12:00.934701Z","iopub.status.idle":"2023-05-18T22:12:01.085982Z","shell.execute_reply.started":"2023-05-18T22:12:00.934679Z","shell.execute_reply":"2023-05-18T22:12:01.084871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since index code is string we will enumerate it so that index_name column has its enumerated ids\narticles['index_id'] = (articles['index_code'].astype('category').cat.codes + 100).astype(int)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:01.087566Z","iopub.execute_input":"2023-05-18T22:12:01.087900Z","iopub.status.idle":"2023-05-18T22:12:01.101427Z","shell.execute_reply.started":"2023-05-18T22:12:01.087874Z","shell.execute_reply":"2023-05-18T22:12:01.100382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the number of numerical, text, and categorical columns\ndef count_numerical_text_categorical_columns(df):\n    numerical_cols = []\n    text_cols = []\n    categorical_cols = []\n\n    for column in df.columns:\n        if pd.api.types.is_numeric_dtype(df[column]):\n            numerical_cols.append(column)\n        elif pd.api.types.is_string_dtype(df[column]):\n            text_cols.append(column)\n        elif pd.api.types.is_categorical_dtype(df[column]):\n            categorical_cols.append(column)\n\n    return (\n        len(numerical_cols),\n        len(text_cols),\n        len(categorical_cols),\n        numerical_cols,\n        text_cols,\n        categorical_cols\n    )\n\n# Example usage\n(\n    num_numerical_cols,\n    num_text_cols,\n    num_categorical_cols,\n    numerical_cols,\n    text_cols,\n    categorical_cols\n) = count_numerical_text_categorical_columns(articles)\n\nprint(f\"Number of numerical columns: {num_numerical_cols}\")\nprint(f\"Numerical columns: {numerical_cols}\")\nprint('\\n' + 100 * \"=\" + '\\n')\nprint(f\"Number of text columns: {num_text_cols}\")\nprint(f\"Text columns: {text_cols}\")\nprint('\\n' + 100 * \"=\" + '\\n')\nprint(f\"Number of categorical columns: {num_categorical_cols}\")\nprint(f\"Categorical columns: {categorical_cols}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:01.102956Z","iopub.execute_input":"2023-05-18T22:12:01.103239Z","iopub.status.idle":"2023-05-18T22:12:01.113743Z","shell.execute_reply.started":"2023-05-18T22:12:01.103216Z","shell.execute_reply":"2023-05-18T22:12:01.112772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before we move on to the next tables let's explore more the practical side of the data.","metadata":{}},{"cell_type":"code","source":"# Count the occurrences of each category\ncategory_counts = articles['index_name'].value_counts()\n\n# Sort the categories by count in descending order\nsorted_categories = category_counts.sort_values(ascending=False).index\n\n# Set the color palette\ncolors = sns.color_palette('Set3', len(sorted_categories))\n\n# Plot the histogram\nsns.set(style='ticks')\nplt.figure(figsize=(8, 6))\nsns.countplot(x='index_name', data=articles, order=sorted_categories, palette=colors)\nplt.xlabel('Category')\nplt.ylabel('Count')\nplt.title('Histogram of Categories (Sorted by Count)')\n\n# Rotate x-axis labels\nplt.xticks(rotation=90, size=9)\nplt.tight_layout()\nplt.show()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-05-18T22:12:01.115050Z","iopub.execute_input":"2023-05-18T22:12:01.115515Z","iopub.status.idle":"2023-05-18T22:12:01.442457Z","shell.execute_reply.started":"2023-05-18T22:12:01.115480Z","shell.execute_reply":"2023-05-18T22:12:01.441430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate the counts for each combination\ncounts = articles.groupby(['garment_group_name', 'index_group_name']).size().unstack(fill_value=0)\n\n# Calculate the total count for each garment group\ntotal_counts = counts.sum(axis=1)\n\n# Sort the values in descending order by the total count\nsorted_counts = counts.loc[total_counts.sort_values(ascending=True).index]\n\n# Create the plot\nplt.figure(figsize=(18, 12))\nsorted_counts.plot(kind='barh', stacked=True, color=sns.color_palette('Set3'))\nplt.xlabel('Index Group')\nplt.ylabel('Count')\nplt.title('Relationship between index_group_name and garment_group_name')\n\n# Display the plot\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:01.443566Z","iopub.execute_input":"2023-05-18T22:12:01.443850Z","iopub.status.idle":"2023-05-18T22:12:02.064815Z","shell.execute_reply.started":"2023-05-18T22:12:01.443827Z","shell.execute_reply":"2023-05-18T22:12:02.063679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"counts = articles.groupby(['index_group_name', 'index_name']).count()['article_id']\nnon_zero_counts = counts[counts > 0]\nprint(non_zero_counts)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:02.066231Z","iopub.execute_input":"2023-05-18T22:12:02.067120Z","iopub.status.idle":"2023-05-18T22:12:02.209297Z","shell.execute_reply.started":"2023-05-18T22:12:02.067079Z","shell.execute_reply":"2023-05-18T22:12:02.208281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.options.display.max_rows = None\n\n# Group by 'product_group_name' and 'product_type_name' and count 'article_id'\ncounts = articles.groupby(['product_group_name', 'product_type_name'])['article_id'].count()\n\n# Sort within each main group\nsorted_counts = counts.groupby(level=0, group_keys=False).apply(lambda x: x.sort_values(ascending=False))\n\n# Print the resulting counts\nprint(sorted_counts)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:02.210567Z","iopub.execute_input":"2023-05-18T22:12:02.210909Z","iopub.status.idle":"2023-05-18T22:12:02.250130Z","shell.execute_reply.started":"2023-05-18T22:12:02.210882Z","shell.execute_reply":"2023-05-18T22:12:02.249007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plots above we can see that the biggest product groups are related to women and children. The most frequent garments are 'Jersey Fancy' and 'Accessories'. We can also see which product types have the highest variability, such as 'Socks' from the 'Socks & Tights' group and 'Underwear bottom' from 'Underwear' product group. The information we get from this table gives us a good idea of the products that are being sold and the possible target groups of buyers, which in our case is the women and girls.\n\nBut we need more information about the customers and further we will try to link it with the purchasing habits.\n\nSo, lets explore the customers table and the information it adds to our business understanding.\n\n## Customers data description\n\n> * **customer_id** : A unique identifier of every customer\n> * **FN** : 1 or missed (not clear what it is)\n> * **Active** : 1 or missed (need to check if it is linked to making purchases)\n> * **club_member_status** : Status in club\n> * **fashion_news_frequency** : How often H&M may send news to customer\n> * **age** : The current age\n> * **postal_code** : Postal code of customer","metadata":{}},{"cell_type":"code","source":"customers.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:02.251305Z","iopub.execute_input":"2023-05-18T22:12:02.251618Z","iopub.status.idle":"2023-05-18T22:12:02.267106Z","shell.execute_reply.started":"2023-05-18T22:12:02.251573Z","shell.execute_reply":"2023-05-18T22:12:02.265887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:02.268430Z","iopub.execute_input":"2023-05-18T22:12:02.269086Z","iopub.status.idle":"2023-05-18T22:12:02.276837Z","shell.execute_reply.started":"2023-05-18T22:12:02.269048Z","shell.execute_reply":"2023-05-18T22:12:02.275905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"customers.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:02.277939Z","iopub.execute_input":"2023-05-18T22:12:02.278225Z","iopub.status.idle":"2023-05-18T22:12:03.235814Z","shell.execute_reply.started":"2023-05-18T22:12:02.278202Z","shell.execute_reply":"2023-05-18T22:12:03.234665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if customer_ids are unique\ncustomers.shape[0] - customers['customer_id'].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:03.237437Z","iopub.execute_input":"2023-05-18T22:12:03.237835Z","iopub.status.idle":"2023-05-18T22:12:03.862169Z","shell.execute_reply.started":"2023-05-18T22:12:03.237799Z","shell.execute_reply":"2023-05-18T22:12:03.860892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Customer club status: \", customers['club_member_status'].unique())\nprint(\"Customer fashion news receiver frequency: \", customers['fashion_news_frequency'].unique())\nprint(\"Customer age: \", customers['age'].unique()) # TODO: put age as category and in age brackets\nprint(\"Customer location: \", customers['postal_code'].nunique()) #TODO: Check the postal codes where purchases are the most","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:03.863301Z","iopub.execute_input":"2023-05-18T22:12:03.863582Z","iopub.status.idle":"2023-05-18T22:12:04.546127Z","shell.execute_reply.started":"2023-05-18T22:12:03.863559Z","shell.execute_reply":"2023-05-18T22:12:04.545137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In the fashion_news_frequency column we have several values that mean the same: NONE, None, nan. Let's fix this\ncustomers.loc[~customers['fashion_news_frequency'].isin(['Regularly', 'Monthly']), 'fashion_news_frequency'] = 'None'\ncustomers['fashion_news_frequency'].unique()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:04.547610Z","iopub.execute_input":"2023-05-18T22:12:04.548000Z","iopub.status.idle":"2023-05-18T22:12:04.732977Z","shell.execute_reply.started":"2023-05-18T22:12:04.547964Z","shell.execute_reply":"2023-05-18T22:12:04.731846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check where most customers come from\ncust_location = customers[['postal_code', 'customer_id']] \\\n    .groupby('postal_code', as_index=False) \\\n    .count() \\\n    .sort_values('customer_id', ascending=False)\ncust_location.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:04.734451Z","iopub.execute_input":"2023-05-18T22:12:04.734768Z","iopub.status.idle":"2023-05-18T22:12:06.451069Z","shell.execute_reply.started":"2023-05-18T22:12:04.734741Z","shell.execute_reply":"2023-05-18T22:12:06.449892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"ticks\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=customers, x='age', bins=50, color='#82cbb2')\nax.set_xlabel('Distribution of the customers age')\n\n# Set x-axis label format for every 10 years\nax.xaxis.set_major_locator(ticker.MultipleLocator(base=10))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:06.452499Z","iopub.execute_input":"2023-05-18T22:12:06.452829Z","iopub.status.idle":"2023-05-18T22:12:07.388576Z","shell.execute_reply.started":"2023-05-18T22:12:06.452802Z","shell.execute_reply":"2023-05-18T22:12:07.387789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that most customers are in the age groups from 20 to 30 and around their 50s. No information is provided about customers sex but it can be assumed that they are mostly women based on the product type ranges.","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(10,5))\nax = sns.histplot(data=customers, x='club_member_status', color='#8e82fe')\nax.set_xlabel('Distribution of club member status')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:07.389770Z","iopub.execute_input":"2023-05-18T22:12:07.390239Z","iopub.status.idle":"2023-05-18T22:12:09.490745Z","shell.execute_reply.started":"2023-05-18T22:12:07.390212Z","shell.execute_reply":"2023-05-18T22:12:09.489635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Looks like very tiny percentage of the members left the club and many more are in the process of becoming members\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10, 5))\ncolors = sns.color_palette('Set3')\nid_news = customers[['customer_id', 'club_member_status']].groupby('club_member_status')['customer_id'].count()\n\n# Calculate percentages\npercentages = id_news / id_news.sum() * 100\n\n# Create the pie chart with percentage labels\nax.pie(id_news, labels=id_news.index, colors=colors, autopct='%1.2f%%')\nax.set_facecolor('lightgrey')\nax.set_xlabel('Distribution of fashion news frequency')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:09.492183Z","iopub.execute_input":"2023-05-18T22:12:09.492578Z","iopub.status.idle":"2023-05-18T22:12:09.893245Z","shell.execute_reply.started":"2023-05-18T22:12:09.492542Z","shell.execute_reply":"2023-05-18T22:12:09.892394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# When it comes to fashion news interest people are either not interested at all or they want a regular update\n# with two-thirds choosing not to receive any news\n\nsns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(10, 5))\ncolors = sns.color_palette('Set3')\nid_news = customers[['customer_id', 'fashion_news_frequency']].groupby('fashion_news_frequency')['customer_id'].count()\n\n# Calculate percentages\npercentages = id_news / id_news.sum() * 100\n\n# Create the pie chart with percentage labels\nax.pie(id_news, labels=id_news.index, colors=colors, autopct='%1.2f%%')\nax.set_facecolor('lightgrey')\nax.set_xlabel('Distribution of fashion news frequency')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:09.894747Z","iopub.execute_input":"2023-05-18T22:12:09.895441Z","iopub.status.idle":"2023-05-18T22:12:10.392611Z","shell.execute_reply.started":"2023-05-18T22:12:09.895404Z","shell.execute_reply":"2023-05-18T22:12:10.391567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After exploring the customer profiles it is time to move to the actual purchase behavior. This information is in the transactions table. Again we will explore the data and then add up to our business case.\n\nTransactions data description:\n\n> * **t_dat** : Purchase date\n> * **customer_id** : A unique identifier of every customer (in customers table)\n> * **article_id** : A unique identifier of every article (in articles table)\n> * **price** : Price of purchase\n> * **sales_channel_id** : 1 or 2","metadata":{}},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:10.394168Z","iopub.execute_input":"2023-05-18T22:12:10.394799Z","iopub.status.idle":"2023-05-18T22:12:10.407606Z","shell.execute_reply.started":"2023-05-18T22:12:10.394764Z","shell.execute_reply":"2023-05-18T22:12:10.406348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.shape","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:10.409213Z","iopub.execute_input":"2023-05-18T22:12:10.409912Z","iopub.status.idle":"2023-05-18T22:12:10.421086Z","shell.execute_reply.started":"2023-05-18T22:12:10.409871Z","shell.execute_reply":"2023-05-18T22:12:10.420045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since count is not shown it means there are no null values in the table\ntransactions.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:10.422760Z","iopub.execute_input":"2023-05-18T22:12:10.423178Z","iopub.status.idle":"2023-05-18T22:12:10.439654Z","shell.execute_reply.started":"2023-05-18T22:12:10.423141Z","shell.execute_reply":"2023-05-18T22:12:10.438751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the minimum and maximum values in a column\ncolumn_min = transactions['price'].min()\ncolumn_max = transactions['price'].max()\n\n# Convert numbers to strings and format without scientific notation\nformatted_min = '{:.6f}'.format(column_min)\nformatted_max = '{:.6f}'.format(column_max)\n\nprint(f\"Minimum value: {formatted_min}\")\nprint(f\"Maximum value: {formatted_max}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:10.441092Z","iopub.execute_input":"2023-05-18T22:12:10.443066Z","iopub.status.idle":"2023-05-18T22:12:10.597504Z","shell.execute_reply.started":"2023-05-18T22:12:10.443022Z","shell.execute_reply":"2023-05-18T22:12:10.596684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.float_format', '{:.5f}'.format)\ntransactions.describe()['price']","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:10.598735Z","iopub.execute_input":"2023-05-18T22:12:10.599987Z","iopub.status.idle":"2023-05-18T22:12:14.539171Z","shell.execute_reply.started":"2023-05-18T22:12:10.599955Z","shell.execute_reply":"2023-05-18T22:12:14.538109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Observe outliers\nplt.figure(figsize=(10, 5))\nsns.violinplot(data=transactions['price'])\nplt.xlabel('Column')\nplt.ylabel('Values')\nplt.title('Violin Plot of price')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:12:14.540399Z","iopub.execute_input":"2023-05-18T22:12:14.541060Z","iopub.status.idle":"2023-05-18T22:13:21.208672Z","shell.execute_reply.started":"2023-05-18T22:12:14.541025Z","shell.execute_reply":"2023-05-18T22:13:21.206942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the start, end, and width of the range\nstart = 0.0\nend = 0.6\nwidth = 0.05\n\n# Generate the range labels\nlabels = [f\"< {start+width:.3f}\"]\nlabels.extend([f\"{i:.3f} - {(i+width):.3f}\" for i in np.arange(start+width, end, width)])\nlabels.append(f\"> {end:.3f}\")\n\n# Create the bins with labels\nbins = [start] + [i+width for i in np.arange(start, end, width)] + [float('inf')]\n\n# Create a new column with the price ranges\ntransactions['price_range'] = pd.cut(transactions['price'], bins=bins, labels=labels, right=False)\n\n# Count the number of prices in each range\nprice_counts = transactions['price_range'].value_counts().sort_index()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:21.217682Z","iopub.execute_input":"2023-05-18T22:13:21.218745Z","iopub.status.idle":"2023-05-18T22:13:22.189410Z","shell.execute_reply.started":"2023-05-18T22:13:21.218703Z","shell.execute_reply":"2023-05-18T22:13:22.188157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate percentages\ntotal_count = price_counts.sum()\npercentages = (price_counts / total_count) * 100\n\n# Format the count and percentage columns\nformatted_counts = price_counts.map(\"{:,}\".format)\nformatted_percentages = percentages.map(\"{:.2f}%\".format)\n\nresult_df = pd.DataFrame({'Count': formatted_counts, 'Percentage': formatted_percentages})\nprint(result_df)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:22.190715Z","iopub.execute_input":"2023-05-18T22:13:22.191032Z","iopub.status.idle":"2023-05-18T22:13:22.201049Z","shell.execute_reply.started":"2023-05-18T22:13:22.191005Z","shell.execute_reply":"2023-05-18T22:13:22.200013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Group customer IDs and count purchases\npurchase_counts = transactions['customer_id'].value_counts()\n\n# Plot the histogram of purchase counts\nplt.figure(figsize=(10, 6))\nplt.hist(purchase_counts, bins=range(1, purchase_counts.max()+2), edgecolor='black')\nplt.xlabel('Number of Purchases')\nplt.ylabel('Frequency')\nplt.title('Distribution of Purchases')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:22.202450Z","iopub.execute_input":"2023-05-18T22:13:22.202799Z","iopub.status.idle":"2023-05-18T22:13:32.470098Z","shell.execute_reply.started":"2023-05-18T22:13:22.202764Z","shell.execute_reply":"2023-05-18T22:13:32.468941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Since it is very skewed distribution, we can make it a bit more clear with log scale.\n# Plot the histogram of purchase counts with logarithmic x-axis scale\nplt.figure(figsize=(10, 6))\nplt.hist(purchase_counts, bins=range(1, purchase_counts.max()+2), color='#464196', edgecolor='#8f8ce7')\nplt.xscale('log')  # Set x-axis scale to logarithmic\nplt.xlabel('Number of Purchases')\nplt.ylabel('Frequency')\nplt.title('Distribution of Purchases (Log Scale)')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:32.471325Z","iopub.execute_input":"2023-05-18T22:13:32.471677Z","iopub.status.idle":"2023-05-18T22:13:36.241765Z","shell.execute_reply.started":"2023-05-18T22:13:32.471649Z","shell.execute_reply":"2023-05-18T22:13:36.240639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"period_len = transactions['t_dat'].nunique()\nstart_date = transactions['t_dat'].min()\nend_date = transactions['t_dat'].max()\nprint(\"Purchase history period length: \", period_len)\nprint(\"Purchase history start date: \", start_date)\nprint(\"Purchase history end date: \", end_date)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:36.243372Z","iopub.execute_input":"2023-05-18T22:13:36.243824Z","iopub.status.idle":"2023-05-18T22:13:42.791751Z","shell.execute_reply.started":"2023-05-18T22:13:36.243785Z","shell.execute_reply":"2023-05-18T22:13:42.790649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert string dates to date objects\ndate1 = datetime.strptime(start_date, '%Y-%m-%d').date()\ndate2 = datetime.strptime(end_date, '%Y-%m-%d').date()\n\n# Calculate the number of days between the dates\ndays_between = (date2 - date1).days + 1\n\nprint(f\"Number of days between {start_date} and {end_date}: {days_between}\")","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:42.793066Z","iopub.execute_input":"2023-05-18T22:13:42.793357Z","iopub.status.idle":"2023-05-18T22:13:42.800479Z","shell.execute_reply.started":"2023-05-18T22:13:42.793333Z","shell.execute_reply":"2023-05-18T22:13:42.799355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert date column to datetime type\ntransactions['t_dat'] = pd.to_datetime(transactions['t_dat'])\n\n# Determine weekday or weekend using vectorized operations\ntransactions['day_type'] = np.where(transactions['t_dat'].dt.dayofweek < 5, 'Weekday', 'Weekend')\n","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:42.801777Z","iopub.execute_input":"2023-05-18T22:13:42.802073Z","iopub.status.idle":"2023-05-18T22:13:57.884822Z","shell.execute_reply.started":"2023-05-18T22:13:42.802049Z","shell.execute_reply":"2023-05-18T22:13:57.883686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:57.886510Z","iopub.execute_input":"2023-05-18T22:13:57.886988Z","iopub.status.idle":"2023-05-18T22:13:57.901007Z","shell.execute_reply.started":"2023-05-18T22:13:57.886930Z","shell.execute_reply":"2023-05-18T22:13:57.900043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Count the number of purchases by day_type\npurchase_counts = transactions['day_type'].value_counts()\n\n# Plot the bar plot\nplt.figure(figsize=(10, 5))\npurchase_counts.plot(kind='bar', color='#c292a1')\nplt.xlabel('Day Type')\nplt.ylabel('Number of Purchases')\nplt.title('Number of Purchases by Day Type')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:13:57.902450Z","iopub.execute_input":"2023-05-18T22:13:57.902791Z","iopub.status.idle":"2023-05-18T22:14:00.896968Z","shell.execute_reply.started":"2023-05-18T22:13:57.902763Z","shell.execute_reply":"2023-05-18T22:14:00.896122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles.columns","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:00.898011Z","iopub.execute_input":"2023-05-18T22:14:00.898469Z","iopub.status.idle":"2023-05-18T22:14:00.905414Z","shell.execute_reply.started":"2023-05-18T22:14:00.898443Z","shell.execute_reply":"2023-05-18T22:14:00.904468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop rows where values in specific column are null\narticles = articles.dropna(subset=['detail_desc'])","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:00.906729Z","iopub.execute_input":"2023-05-18T22:14:00.907153Z","iopub.status.idle":"2023-05-18T22:14:00.975795Z","shell.execute_reply.started":"2023-05-18T22:14:00.907126Z","shell.execute_reply":"2023-05-18T22:14:00.974657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Time to find relationships between tables features\n\narticles_for_merge = articles[['article_id', 'product_code', 'prod_name', 'product_type_no',\n       'product_type_name', 'product_group_name', 'graphical_appearance_name', 'colour_group_name',\n       'perceived_colour_value_name', 'perceived_colour_master_name', 'department_no', \n        'department_name', 'index_name', 'index_group_name', 'section_no', 'section_name',\n       'garment_group_name', 'detail_desc']]\ncustomers_for_merge = customers[['customer_id', 'club_member_status', 'fashion_news_frequency', 'age', 'postal_code']]","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:00.977168Z","iopub.execute_input":"2023-05-18T22:14:00.977452Z","iopub.status.idle":"2023-05-18T22:14:01.110107Z","shell.execute_reply.started":"2023-05-18T22:14:00.977429Z","shell.execute_reply":"2023-05-18T22:14:01.109271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define mapping for day_type\nday_type_map = {'Weekend': 0, 'Weekday': 1}\n\n# Apply mapping to day_type column\ntransactions['day_type_num'] = transactions['day_type'].map(day_type_map)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:01.111177Z","iopub.execute_input":"2023-05-18T22:14:01.112042Z","iopub.status.idle":"2023-05-18T22:14:03.074799Z","shell.execute_reply.started":"2023-05-18T22:14:01.112014Z","shell.execute_reply":"2023-05-18T22:14:03.073678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"combined = transactions[['customer_id', 'article_id', 'price', 'price_range', 't_dat', 'day_type_num']] \\\n.merge(articles_for_merge, on='article_id', how='left') \\\n.merge(customers_for_merge, on='customer_id', how='left')","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:03.076285Z","iopub.execute_input":"2023-05-18T22:14:03.076633Z","iopub.status.idle":"2023-05-18T22:14:53.210055Z","shell.execute_reply.started":"2023-05-18T22:14:03.076587Z","shell.execute_reply":"2023-05-18T22:14:53.208797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Columns to check\ncols = ['product_group_name', 'colour_group_name', 'perceived_colour_value_name',\n'perceived_colour_master_name', 'index_name', 'index_group_name', 'garment_group_name']","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:53.212654Z","iopub.execute_input":"2023-05-18T22:14:53.212976Z","iopub.status.idle":"2023-05-18T22:14:53.218550Z","shell.execute_reply.started":"2023-05-18T22:14:53.212951Z","shell.execute_reply":"2023-05-18T22:14:53.217325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find unique values for all columns excluding specified columns\nunique_values = {}\nfor col in cols:\n    unique_values[col] = combined[col].unique()\n\n# Print the unique values\nfor col, values in unique_values.items():\n    print(f\"Unique values for {col}: {values}\")\n    print(150 * '-')","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:53.220271Z","iopub.execute_input":"2023-05-18T22:14:53.220985Z","iopub.status.idle":"2023-05-18T22:14:55.829459Z","shell.execute_reply.started":"2023-05-18T22:14:53.220944Z","shell.execute_reply":"2023-05-18T22:14:55.828320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate the percentage of null values in each column\n\n# isnull() to create a boolean mask that identifies the null values in each column. \n# mean() function to calculate the proportion of True values (null values) in each column. \n# multiplying by 100 gives the percentage of null values.\n\nnull_percentage = combined.isnull().mean() * 100\n\n# Get the total number of non-null values in each column\nnon_null_count = combined.count()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:14:55.830809Z","iopub.execute_input":"2023-05-18T22:14:55.831214Z","iopub.status.idle":"2023-05-18T22:16:44.600156Z","shell.execute_reply.started":"2023-05-18T22:14:55.831178Z","shell.execute_reply":"2023-05-18T22:16:44.599303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combine the results\nresult = pd.DataFrame({\n    'Null Percentage': null_percentage,\n    'Non-Null Count': non_null_count\n})\n\nprint(result)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:16:44.601499Z","iopub.execute_input":"2023-05-18T22:16:44.602039Z","iopub.status.idle":"2023-05-18T22:16:44.608881Z","shell.execute_reply.started":"2023-05-18T22:16:44.602009Z","shell.execute_reply":"2023-05-18T22:16:44.607992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop all rows where we find values = nan in any of the columns. \n# If we do more thorough exploration we can check each of these columns and try to figure out \n### why these values are nan and instead of just dropping them we can give them another name \n### and leave them in the dataset. Since they represent very small percentage for the sake of \n### simplicity we drop them.\ncombined = combined.dropna()\ncombined.shape\n# combined.to_csv('combined_clean.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:16:44.610227Z","iopub.execute_input":"2023-05-18T22:16:44.610497Z","iopub.status.idle":"2023-05-18T22:18:00.906612Z","shell.execute_reply.started":"2023-05-18T22:16:44.610474Z","shell.execute_reply":"2023-05-18T22:18:00.905517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(20,16))\nax = sns.boxplot(data=combined, x='price', y='product_group_name')\nax.set_xlabel('Price outliers', fontsize=16)\nax.set_ylabel('Index names', fontsize=16)\nax.xaxis.set_tick_params(labelsize=16)\nax.yaxis.set_tick_params(labelsize=16)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:00.908121Z","iopub.execute_input":"2023-05-18T22:18:00.909367Z","iopub.status.idle":"2023-05-18T22:18:19.123795Z","shell.execute_reply.started":"2023-05-18T22:18:00.909334Z","shell.execute_reply":"2023-05-18T22:18:19.122644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some very high prices are observed in the categories 'Garment Upper/Lower/Full body' as well as 'Accessories' and 'Shoes'. We can go deeper into the 'Accessories' category to find out the main 'contributor' for the pricey items in it.","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"darkgrid\")\nf, ax = plt.subplots(figsize=(20,16))\n_ = combined[combined['product_group_name'] == 'Accessories']\nax = sns.boxplot(data=_, x='price', y='product_type_name')\nax.set_xlabel('Price outliers', fontsize=16)\nax.set_ylabel('Index names', fontsize=16)\nax.xaxis.set_tick_params(labelsize=16)\nax.yaxis.set_tick_params(labelsize=16)\ndel _\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:19.125394Z","iopub.execute_input":"2023-05-18T22:18:19.125791Z","iopub.status.idle":"2023-05-18T22:18:23.900029Z","shell.execute_reply.started":"2023-05-18T22:18:19.125759Z","shell.execute_reply":"2023-05-18T22:18:23.898785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Product 'Bags' has items with the highest prices. We could not say it is a surprise.\n\nAdditionaly, we explore the indexes and product groups with higher and lower mean price.","metadata":{}},{"cell_type":"code","source":"combined_index = combined[['index_name', 'price']].groupby('index_name').mean()\nsorted_val = combined_index.sort_values(by='price', ascending=False).index\n\nsns.set_style(\"ticks\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=combined_index.price, y=combined_index.index, order = sorted_val, color='#d5869d')\nax.set_xlabel('Price by index')\nax.set_ylabel('Index')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:23.901272Z","iopub.execute_input":"2023-05-18T22:18:23.902334Z","iopub.status.idle":"2023-05-18T22:18:24.569744Z","shell.execute_reply.started":"2023-05-18T22:18:23.902302Z","shell.execute_reply":"2023-05-18T22:18:24.568471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"combined_index = combined[['product_group_name', 'price']].groupby('product_group_name').mean()\nsorted_val = combined_index.sort_values(by='price', ascending=False).index\n\nsns.set_style(\"ticks\")\nf, ax = plt.subplots(figsize=(10,5))\nax = sns.barplot(x=combined_index.price, y=combined_index.index, order = sorted_val, color='#8e82fe')\nax.set_xlabel('Price by product group')\nax.set_ylabel('Product group')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:24.571289Z","iopub.execute_input":"2023-05-18T22:18:24.571639Z","iopub.status.idle":"2023-05-18T22:18:28.400634Z","shell.execute_reply.started":"2023-05-18T22:18:24.571611Z","shell.execute_reply":"2023-05-18T22:18:28.399568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_colors = combined.groupby(['perceived_colour_master_name', 'perceived_colour_value_name']).agg({'customer_id': 'count'}).reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:28.402215Z","iopub.execute_input":"2023-05-18T22:18:28.403036Z","iopub.status.idle":"2023-05-18T22:18:30.764964Z","shell.execute_reply.started":"2023-05-18T22:18:28.402996Z","shell.execute_reply":"2023-05-18T22:18:30.763825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Have a view on the most popular colours\ncount_colors = count_colors[count_colors['customer_id'] > 0]\ncount_colors.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:30.766431Z","iopub.execute_input":"2023-05-18T22:18:30.767428Z","iopub.status.idle":"2023-05-18T22:18:30.778645Z","shell.execute_reply.started":"2023-05-18T22:18:30.767394Z","shell.execute_reply":"2023-05-18T22:18:30.777840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The purpose of this digging is to get more general idea of the main preferences of the customers. Suggesting a product that a customer is more likely to associate with its style or needs increases the probability of buying. \n\nBefore moving on to Modeling section we need to make some more data conversions in order to be able to work more easily with it.","metadata":{}},{"cell_type":"code","source":"combined.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:30.779900Z","iopub.execute_input":"2023-05-18T22:18:30.780631Z","iopub.status.idle":"2023-05-18T22:18:30.800581Z","shell.execute_reply.started":"2023-05-18T22:18:30.780576Z","shell.execute_reply":"2023-05-18T22:18:30.799408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exclude_columns=['customer_id', 'prod_name', 'detail_desc', 'postal_code']\n\nobject_columns = select_object_columns(combined, ['object'])\ndf_dropped = drop_columns(object_columns, columns=exclude_columns)\nunique_counts = count_unique_values(df_dropped)\nprint(unique_counts)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:30.813213Z","iopub.execute_input":"2023-05-18T22:18:30.813496Z","iopub.status.idle":"2023-05-18T22:18:50.789149Z","shell.execute_reply.started":"2023-05-18T22:18:30.813474Z","shell.execute_reply":"2023-05-18T22:18:50.787983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"combined[object_columns] = convert_to_category(combined, columns=object_columns)","metadata":{"execution":{"iopub.status.busy":"2023-05-18T22:18:50.790675Z","iopub.execute_input":"2023-05-18T22:18:50.791207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"combined.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # This Python 3 environment comes with many helpful analytics libraries installed\n# # It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# # For example, here's several helpful packages to load\n\n# import numpy as np # linear algebra\n# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# # Input data files are available in the read-only \"../input/\" directory\n# # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# # You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}