{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":31254,"databundleVersionId":3103714,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\nH&M recsys challenge \n\nMain page of the challenge: \\\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/overview\n\nH&M Group is a family of brands and businesses with 53 online markets and approximately 4,850 stores. Our online store offers shoppers an extensive selection of products to browse through. But with too many choices, customers might not quickly find what interests them or what they are looking for, and ultimately, they might not make a purchase. To enhance the shopping experience, product recommendations are key. More importantly, helping customers make the right choices also has a positive implications for sustainability, as it reduces returns, and thereby minimizes emissions from transportation.\n\nIn this competition, H&M Group invites you to develop product recommendations based on data from previous transactions, as well as from customer and product meta data. The available meta data spans from simple data, such as garment type and customer age, to text data from product descriptions, to image data from garment images.\n\nThere are no preconceptions on what information that may be useful – that is for you to find out. If you want to investigate a categorical data type algorithm, or dive into NLP and image processing deep learning, that is up to you.\n\nSubmissions are evaluated according to the Mean Average Precision @ 12 (MAP@12)\nI can share a great tutorial about this metric. \\\nhttps://www.kaggle.com/code/debarshichanda/understanding-mean-average-precision\n\n\n# Общее описание \nH&M recsys challenge \n\nГлавная страница задачи: \\\nhttps://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/overview\n\nH&M Group - это семейство брендов и предприятий, включающее 53 онлайн-маркета и около 4850 магазинов. Наш интернет-магазин предлагает покупателям широкий выбор товаров. Но при слишком большом выборе покупатели могут не сразу найти то, что их интересует или что они ищут, и в итоге могут не совершить покупку. Чтобы улучшить впечатления от покупок, рекомендации по товарам играют ключевую роль. Более того, помощь покупателям в правильном выборе также положительно сказывается на экологичности, поскольку сокращает количество возвратов и тем самым минимизирует выбросы при транспортировке.\n\nВ этом конкурсе H&amp;M Group предлагает вам разработать товарные рекомендации на основе данных о предыдущих транзакциях, а также метаданных о покупателях и товарах. Доступные метаданные включают в себя как простые данные, такие как тип одежды и возраст покупателя, так и текстовые данные из описаний товаров, а также данные из изображений одежды.\n\nНет никаких предубеждений относительно того, какая информация может быть полезной - это вы должны выяснить сами. Если вы хотите исследовать алгоритм категориального типа данных или погрузиться в глубокое обучение НЛП и обработке изображений - дело ваше.\n\nПрисылаемые материалы оцениваются по показателю средней точности @ 12 (MAP@12)\nЯ могу поделиться отличным руководством по этой метрике.\nhttps://www.kaggle.com/code/debarshichanda/understanding-mean-average-precision\n","metadata":{}},{"cell_type":"markdown","source":"# Libraries","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade implicit","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T11:55:57.278780Z","iopub.execute_input":"2025-04-08T11:55:57.279141Z","iopub.status.idle":"2025-04-08T11:56:03.198857Z","shell.execute_reply.started":"2025-04-08T11:55:57.279112Z","shell.execute_reply":"2025-04-08T11:56:03.197614Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom datetime import datetime, timedelta\nfrom collections import defaultdict\nimport warnings\nwarnings.filterwarnings('ignore')\n\nfrom sklearn.metrics import mean_squared_error\n\nfrom scipy.sparse import coo_matrix, csr_matrix\nimport implicit\nimport time\nfrom tqdm import tqdm\nfrom implicit.als import AlternatingLeastSquares\n\n\npd.set_option('display.max_columns', None)\npd.set_option('display.max_rows', 50)\npd.set_option('display.float_format', '{:.4f}'.format)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T11:56:03.200494Z","iopub.execute_input":"2025-04-08T11:56:03.200928Z","iopub.status.idle":"2025-04-08T11:56:04.646021Z","shell.execute_reply.started":"2025-04-08T11:56:03.200884Z","shell.execute_reply":"2025-04-08T11:56:04.645202Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Loading dataset","metadata":{}},{"cell_type":"code","source":"base_path = '/kaggle/input/h-and-m-personalized-fashion-recommendations/'\ncsv_transactions = f'{base_path}transactions_train.csv'\ncsv_sample_submission = f'{base_path}sample_submission.csv'\ncsv_customers = f'{base_path}customers.csv'\ncsv_articles = f'{base_path}articles.csv'\n\ndf_transactions = pd.read_csv(csv_transactions, dtype={'article_id': str}, parse_dates=['t_dat'])\ndf_sample_submission = pd.read_csv(csv_sample_submission)\ndf_customers = pd.read_csv(csv_customers)\ndf_articles = pd.read_csv(csv_articles, dtype={'article_id': str})","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T11:56:04.647626Z","iopub.execute_input":"2025-04-08T11:56:04.648162Z","iopub.status.idle":"2025-04-08T11:59:11.734422Z","shell.execute_reply.started":"2025-04-08T11:56:04.648133Z","shell.execute_reply":"2025-04-08T11:59:11.733397Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"code","source":"df_transactions.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.722760Z","iopub.execute_input":"2025-04-08T08:32:45.723131Z","iopub.status.idle":"2025-04-08T08:32:45.754693Z","shell.execute_reply.started":"2025-04-08T08:32:45.723098Z","shell.execute_reply":"2025-04-08T08:32:45.753468Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_sample_submission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.755840Z","iopub.execute_input":"2025-04-08T08:32:45.756285Z","iopub.status.idle":"2025-04-08T08:32:45.766852Z","shell.execute_reply.started":"2025-04-08T08:32:45.756238Z","shell.execute_reply":"2025-04-08T08:32:45.765506Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_customers.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.768068Z","iopub.execute_input":"2025-04-08T08:32:45.768496Z","iopub.status.idle":"2025-04-08T08:32:45.797579Z","shell.execute_reply.started":"2025-04-08T08:32:45.768468Z","shell.execute_reply":"2025-04-08T08:32:45.796350Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_articles.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.798666Z","iopub.execute_input":"2025-04-08T08:32:45.799131Z","iopub.status.idle":"2025-04-08T08:32:45.839162Z","shell.execute_reply.started":"2025-04-08T08:32:45.799099Z","shell.execute_reply":"2025-04-08T08:32:45.837960Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"First, I suggest a better understanding of the data and past buying behavior. How data is distributed in general and what characterizes their behavior. \n\nImportant tip\nsales_channel_id=1: Sales of ONLINE, 2: Sales of OFFLINE\n\nLet's deal with the following parts \\\nCustomer \\\nArticle \\\nTransaction\n\n\nСначала предлагаю лучше понять данные и прошлое поведение покупателей. Как вообще распределены данные и что характеризует их поведение. \n\nВажное замечание\nsales_channel_id=1: Продажи ONLINE, 2: Продажи OFFLINE\n\nРазберемся со следующими частями \\\nКлиент \\\nСтатья \\\nТранзакция","metadata":{}},{"cell_type":"code","source":"print(f\"Number of rows and columns of sample submission: {df_customers.shape}\")\nprint(f\"Number of rows and columns of articles: {df_articles.shape,}\")\nprint(f\"Number of rows and columns of users: {df_customers.shape}\")\nprint(f\"Number of rows and columns of users: {df_transactions.shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.840428Z","iopub.execute_input":"2025-04-08T08:32:45.840855Z","iopub.status.idle":"2025-04-08T08:32:45.862697Z","shell.execute_reply.started":"2025-04-08T08:32:45.840812Z","shell.execute_reply":"2025-04-08T08:32:45.861306Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(f\"Start date: {df_transactions['t_dat'].min()}\")\nprint(f\"End date: {df_transactions['t_dat'].max()}\")\nprint(f\"Timespan: {(df_transactions['t_dat'].max() - df_transactions['t_dat'].min()).days} days\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:45.867069Z","iopub.execute_input":"2025-04-08T08:32:45.867423Z","iopub.status.idle":"2025-04-08T08:32:46.306805Z","shell.execute_reply.started":"2025-04-08T08:32:45.867395Z","shell.execute_reply":"2025-04-08T08:32:46.304562Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_transactions['month_year'] = df_transactions['t_dat'].dt.to_period('M')\nmonthly_sales = df_transactions.groupby('month_year').size().reset_index(name='sales_count')\n\nplt.figure(figsize=(12, 6))\nsns.lineplot(x=[str(p) for p in monthly_sales['month_year']], y=monthly_sales['sales_count'])\nplt.title('monthly sales')\nplt.xlabel('month')\nplt.ylabel('transactions')\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:46.309536Z","iopub.execute_input":"2025-04-08T08:32:46.310002Z","iopub.status.idle":"2025-04-08T08:32:49.415695Z","shell.execute_reply.started":"2025-04-08T08:32:46.309941Z","shell.execute_reply":"2025-04-08T08:32:49.414470Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sales_channels = df_transactions['sales_channel_id'].value_counts().reset_index()\nsales_channels.columns = ['channel', 'count']\n\nplt.figure(figsize=(8, 6))\nsns.barplot(x='channel', y='count', data=sales_channels, palette='viridis')\nplt.title('Channel sales distribution')\nplt.xlabel('Sales Channel')\nplt.ylabel('transactions')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:49.416827Z","iopub.execute_input":"2025-04-08T08:32:49.417152Z","iopub.status.idle":"2025-04-08T08:32:49.779486Z","shell.execute_reply.started":"2025-04-08T08:32:49.417124Z","shell.execute_reply":"2025-04-08T08:32:49.778163Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df_transactions['week'] = df_transactions['t_dat'].dt.isocalendar().week\nweekly_avg_price = df_transactions.groupby(['week'])['price'].mean().reset_index()\n\nplt.figure(figsize=(12, 6))\nsns.lineplot(x='week', y='price', data=weekly_avg_price, color='green')\nplt.title('avg week price trend')\nplt.xlabel('week')\nplt.ylabel('avg price')\nplt.grid(True, alpha=0.3)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:49.780644Z","iopub.execute_input":"2025-04-08T08:32:49.781049Z","iopub.status.idle":"2025-04-08T08:32:52.646510Z","shell.execute_reply.started":"2025-04-08T08:32:49.780988Z","shell.execute_reply":"2025-04-08T08:32:52.644778Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nsns.histplot(df_customers['age'].dropna(), bins=70, kde=True, color='orange')\nplt.title('age distribution')\nplt.xlabel('age')\nplt.ylabel('num of customers')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:52.648708Z","iopub.execute_input":"2025-04-08T08:32:52.649199Z","iopub.status.idle":"2025-04-08T08:32:59.200686Z","shell.execute_reply.started":"2025-04-08T08:32:52.649153Z","shell.execute_reply":"2025-04-08T08:32:59.199213Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fashion_news = df_customers['fashion_news_frequency'].value_counts().reset_index()\nfashion_news.columns = ['frequency', 'count']\n\nplt.figure(figsize=(10, 6))\nsns.barplot(x='frequency', y='count', data=fashion_news, palette='Set2')\nplt.title('fashion news distribution')\nplt.xlabel('frequency')\nplt.ylabel('num of news')\nplt.xticks(rotation=45)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:59.201778Z","iopub.execute_input":"2025-04-08T08:32:59.202162Z","iopub.status.idle":"2025-04-08T08:32:59.515400Z","shell.execute_reply.started":"2025-04-08T08:32:59.202132Z","shell.execute_reply":"2025-04-08T08:32:59.513608Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"club_status = df_customers['club_member_status'].value_counts().reset_index()\nclub_status.columns = ['status', 'count']\n\nplt.figure(figsize=(10, 6))\nplt.pie(club_status['count'], labels=club_status['status'], shadow=True, startangle=90, colors=sns.color_palette('pastel'))\nplt.title('club status distribution')\nplt.axis('equal')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:59.517850Z","iopub.execute_input":"2025-04-08T08:32:59.518252Z","iopub.status.idle":"2025-04-08T08:32:59.815459Z","shell.execute_reply.started":"2025-04-08T08:32:59.518218Z","shell.execute_reply":"2025-04-08T08:32:59.814230Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"product_types = df_articles['product_type_name'].value_counts().head(15).reset_index()\nproduct_types.columns = ['product_type', 'count']\n\nplt.figure(figsize=(12, 8))\nsns.barplot(x='count', y='product_type', data=product_types)\nplt.title('top 15 Products')\nplt.xlabel('num of each type')\nplt.ylabel('type')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:32:59.816611Z","iopub.execute_input":"2025-04-08T08:32:59.816990Z","iopub.status.idle":"2025-04-08T08:33:00.134121Z","shell.execute_reply.started":"2025-04-08T08:32:59.816944Z","shell.execute_reply":"2025-04-08T08:33:00.132877Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"product_groups = df_articles['product_group_name'].value_counts().reset_index()\nproduct_groups.columns = ['product_group', 'count']\n\nplt.figure(figsize=(12, 8))\nsns.barplot(x='count', y='product_group', data=product_groups)\nplt.title('groups distribution')\nplt.xlabel('num of groups')\nplt.ylabel('group')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:00.135409Z","iopub.execute_input":"2025-04-08T08:33:00.135781Z","iopub.status.idle":"2025-04-08T08:33:00.512690Z","shell.execute_reply.started":"2025-04-08T08:33:00.135738Z","shell.execute_reply":"2025-04-08T08:33:00.511410Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"colors = df_articles['perceived_colour_master_name'].value_counts().reset_index()\ncolors.columns = ['color', 'count']\n\nplt.figure(figsize=(12, 8))\nsns.barplot(x='count', y='color', data=colors)\nplt.title('colour distribution')\nplt.xlabel('num of colour')\nplt.ylabel('colour')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:00.513855Z","iopub.execute_input":"2025-04-08T08:33:00.514200Z","iopub.status.idle":"2025-04-08T08:33:00.872264Z","shell.execute_reply.started":"2025-04-08T08:33:00.514171Z","shell.execute_reply":"2025-04-08T08:33:00.870691Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# distribution is highly scewed. Some customres make way too many transactions.\ncustomer_purchase_counts = df_transactions.groupby('customer_id').size().reset_index(name='purchase_count')\n\nplt.figure(figsize=(10, 6))\nsns.histplot(customer_purchase_counts['purchase_count'].clip(upper=50), bins=50, kde=True)\nplt.title('number of transations per customer')\nplt.xlabel('num of purchases')\nplt.ylabel('num of customers')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:00.873304Z","iopub.execute_input":"2025-04-08T08:33:00.873612Z","iopub.status.idle":"2025-04-08T08:33:20.309054Z","shell.execute_reply.started":"2025-04-08T08:33:00.873586Z","shell.execute_reply":"2025-04-08T08:33:20.307829Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# top 10 customers\ncustomer_purchase_counts.sort_values('purchase_count', ascending=False).head(10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:20.310429Z","iopub.execute_input":"2025-04-08T08:33:20.310911Z","iopub.status.idle":"2025-04-08T08:33:20.719240Z","shell.execute_reply.started":"2025-04-08T08:33:20.310817Z","shell.execute_reply":"2025-04-08T08:33:20.718008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# analogously top 10 items\npopular_items = df_transactions.groupby('article_id').size().reset_index(name='purchase_count')\npopular_items = popular_items.sort_values('purchase_count', ascending=False).head(10)\n\npopular_items","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:20.720412Z","iopub.execute_input":"2025-04-08T08:33:20.720739Z","iopub.status.idle":"2025-04-08T08:33:26.107603Z","shell.execute_reply.started":"2025-04-08T08:33:20.720711Z","shell.execute_reply":"2025-04-08T08:33:26.106391Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# tyoe of products and name for top items. Mostly a little purchases, like trousers and socks\ntop_items_details = pd.merge(popular_items, df_articles[['article_id', 'prod_name', 'product_type_name']], on='article_id', how='left')\n\ntop_items_details","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:26.108744Z","iopub.execute_input":"2025-04-08T08:33:26.109115Z","iopub.status.idle":"2025-04-08T08:33:26.166336Z","shell.execute_reply.started":"2025-04-08T08:33:26.109076Z","shell.execute_reply":"2025-04-08T08:33:26.165006Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# What's the most and least popular in terms of price? \n# Wow women tops and bra among the most popular choices. Some of the most unpopular are \nitem_price_popularity = df_transactions.groupby('article_id').agg({'price': 'mean', 'article_id': 'count'})\nitem_price_popularity.columns = ['avg_price', 'popularity']\nitem_price_popularity = item_price_popularity.reset_index().merge(df_articles[['article_id', 'prod_name', 'product_type_name']], on='article_id', how='left')\nitem_price_popularity\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:26.167574Z","iopub.execute_input":"2025-04-08T08:33:26.167992Z","iopub.status.idle":"2025-04-08T08:33:33.708378Z","shell.execute_reply.started":"2025-04-08T08:33:26.167952Z","shell.execute_reply":"2025-04-08T08:33:33.706740Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(10, 6))\nsns.scatterplot(x='avg_price', y='popularity', data=item_price_popularity.sample(10000), alpha=0.5)\nplt.xscale('log')\nplt.yscale('log')\nplt.title('log of Price vs Popularity')\nplt.xlabel('avg price')\nplt.ylabel('num of purchases')\nplt.grid(True, alpha=0.3)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:33.709691Z","iopub.execute_input":"2025-04-08T08:33:33.710122Z","iopub.status.idle":"2025-04-08T08:33:34.580350Z","shell.execute_reply.started":"2025-04-08T08:33:33.710089Z","shell.execute_reply":"2025-04-08T08:33:34.578972Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# price analysis by product category\nmerged_data = pd.merge(df_transactions, df_articles[['article_id', 'product_group_name', 'product_type_name']], \n                      on='article_id', how='left')\n\navg_price_by_group = merged_data.groupby('product_group_name')['price'].mean().reset_index()\navg_price_by_group = avg_price_by_group.sort_values('price', ascending=False)\n\nplt.figure(figsize=(12, 8))\nsns.barplot(x='price', y='product_group_name', data=avg_price_by_group)\nplt.title('avg price by product group')\nplt.xlabel('avg price')\nplt.ylabel('group')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:34.581472Z","iopub.execute_input":"2025-04-08T08:33:34.581763Z","iopub.status.idle":"2025-04-08T08:33:51.562765Z","shell.execute_reply.started":"2025-04-08T08:33:34.581739Z","shell.execute_reply":"2025-04-08T08:33:51.561275Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# price distributions within 5 top groups\ntop_5_groups = avg_price_by_group.head(5)['product_group_name'].tolist()\ntop_groups_data = merged_data[merged_data['product_group_name'].isin(top_5_groups)]\n\nplt.figure(figsize=(14, 8))\nsns.boxplot(x='product_group_name', y='price', data=top_groups_data)\nplt.title('price distribution for 5 top groups')\nplt.xlabel('product group')\nplt.ylabel('price')\nplt.grid()\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:33:51.564114Z","iopub.execute_input":"2025-04-08T08:33:51.564577Z","iopub.status.idle":"2025-04-08T08:34:00.925421Z","shell.execute_reply.started":"2025-04-08T08:33:51.564542Z","shell.execute_reply":"2025-04-08T08:34:00.924324Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# purchase patterns search\n\nmerged_cust_data = pd.merge(df_transactions, df_customers[['customer_id', 'age']], on='customer_id', how='left')\nmerged_cust_data = merged_cust_data.dropna(subset=['age'])\n\nbins_list = [0, 20, 30, 40, 50, 60, 100]\nnames_list = ['<20', '20-30', '30-40', '40-50', '50-60', '60+']\n\nmerged_cust_data['age_bin'] = pd.cut(merged_cust_data['age'], bins=bins_list, labels=names_list)\n\nage_group_spending = merged_cust_data.groupby('age_bin')['price'].mean().reset_index()\n\nplt.figure(figsize=(10, 6))\nsns.barplot(x='age_bin', y='price', data=age_group_spending, palette='viridis')\nplt.title('avg purchise Price per age group')\nplt.xlabel('age group')\nplt.ylabel('avg price')\nplt.grid(True, alpha=0.3)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:34:00.931518Z","iopub.execute_input":"2025-04-08T08:34:00.931872Z","iopub.status.idle":"2025-04-08T08:34:21.625652Z","shell.execute_reply.started":"2025-04-08T08:34:00.931845Z","shell.execute_reply":"2025-04-08T08:34:21.624312Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Most selling product per age group\n\nkey_age_groups = ['20-30', '30-40', '50-60']\nage_product_data = pd.merge(merged_cust_data, df_articles[['article_id', 'product_type_name']], on='article_id', how='left')\n\nfor age_group in key_age_groups:\n    group_data = age_product_data[age_product_data['age_bin'] == age_group]\n    top_products = group_data['product_type_name'].value_counts().head(5)\n    \n    plt.figure(figsize=(10, 6))\n    sns.barplot(x=top_products.values, y=top_products.index)\n    plt.title(f'top 5 products per age group: {age_group}')\n    plt.xlabel('how many purchases')\n    plt.ylabel('type')\n    plt.tight_layout()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:34:21.628259Z","iopub.execute_input":"2025-04-08T08:34:21.628581Z","iopub.status.idle":"2025-04-08T08:34:41.506844Z","shell.execute_reply.started":"2025-04-08T08:34:21.628555Z","shell.execute_reply":"2025-04-08T08:34:41.505462Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Transactions per customer\ntransactions_per_customer = df_transactions.groupby('customer_id').size().reset_index(name='transaction_count')\n\nplt.figure(figsize=(12, 6))\nsns.histplot(data=transactions_per_customer, x='transaction_count', bins=50, kde=True, color='green')\nplt.title('distribution of transactions for customer', fontsize=14)\nplt.xlabel('num of transactions', fontsize=12)\nplt.ylabel('num of customers', fontsize=12)\nplt.xlim(0, transactions_per_customer['transaction_count'].quantile(0.99))\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:34:41.508568Z","iopub.execute_input":"2025-04-08T08:34:41.509065Z","iopub.status.idle":"2025-04-08T08:34:59.833649Z","shell.execute_reply.started":"2025-04-08T08:34:41.508994Z","shell.execute_reply":"2025-04-08T08:34:59.832382Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data preprocessing","metadata":{}},{"cell_type":"code","source":"# convert customer_id to numeric format\ndf_transactions['customer_id_int'] = df_transactions['customer_id'].apply(lambda x: int(x[-16:], 16))\n\n# let's filter unnecesary columns\ndf_transactions = df_transactions[['t_dat', 'customer_id', 'customer_id_int', 'article_id']]\n\ndf_sample_submission['customer_id_int'] = df_sample_submission['customer_id'].apply(lambda x: int(x[-16:], 16)) # same for submission dataframe","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T12:14:34.816673Z","iopub.execute_input":"2025-04-08T12:14:34.817067Z","iopub.status.idle":"2025-04-08T12:15:00.668064Z","shell.execute_reply.started":"2025-04-08T12:14:34.817037Z","shell.execute_reply":"2025-04-08T12:15:00.666899Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Split data into train and test based on time","metadata":{}},{"cell_type":"code","source":"# get last date from all transactions\nlast_date = df_transactions['t_dat'].max()\nprint(f'Last date in all transactions: {last_date}')\n\n# define test start date as 7 days before the last date\ntest_start_date = last_date - timedelta(days=7)\nprint(f'Test start date: {test_start_date}')\n\n# test transactions are transactions from the last week\ndf_transactions_test = df_transactions[df_transactions['t_dat'] >= test_start_date].copy()\nprint(f'Test transactions dates from {df_transactions_test[\"t_dat\"].min()} to {df_transactions_test[\"t_dat\"].max()}, shape: {df_transactions_test.shape}')\n\n# train transactions are all transactions before the test week\ndf_transactions_train = df_transactions[df_transactions['t_dat'] < test_start_date].copy()\nprint(f'Train transactions dates from {df_transactions_train[\"t_dat\"].min()} to {df_transactions_train[\"t_dat\"].max()}, shape: {df_transactions_train.shape}')\n\n# get the last date of the training period\nlast_date_train = df_transactions_train['t_dat'].max()\nprint(f'Last date in train transactions: {last_date_train}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T12:15:03.547912Z","iopub.execute_input":"2025-04-08T12:15:03.548329Z","iopub.status.idle":"2025-04-08T12:15:07.446169Z","shell.execute_reply.started":"2025-04-08T12:15:03.548293Z","shell.execute_reply":"2025-04-08T12:15:07.445150Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Evalation metrics function","metadata":{}},{"cell_type":"code","source":"def evaluate_recommendations(actual_df, pred_dfs, k=12):\n    \"\"\"\n    Main metrics evaluation function. Can receive list of predictions\n\n    Input: actual_df - dataframe with actual purchases\n    pred_dfs : list of dataframes with predicted recommendations\n    k : num of K at MAP\n    Returns: metrics dataframe\n    \"\"\"\n\n    # Actual purchases to dict\n    actual_purchases = defaultdict(set)\n    for _, row in actual_df.iterrows():\n        actual_purchases[row['customer_id_int']].add(row['article_id']) # Используем 'customer_id_int'\n\n    results = {\n        f'map@{k}': [],\n        'precision': [],\n        'recall': [],\n        'f1_score': [],\n        'rmse': [],\n    }\n\n    total_items = len(actual_df['article_id'].unique())\n\n    for pred_df in pred_dfs:\n        # track metrics\n        ap_scores = []\n        precision_scores = []\n        recall_scores = []\n        f1_scores = []\n        rmse_scores = []\n        predicted_items_set = set()\n        recommendation_count = 0\n        pop_scores = []\n\n        # loop for all predictions\n        for _, row in pred_df.iterrows():\n            customer_id = row['customer_id_int']\n\n            # no purchases\n            if customer_id not in actual_purchases:\n                continue\n\n            # Get items\n            if isinstance(row['prediction'], str):\n                pred_items = row['prediction'].strip().split()[:k]\n            else:\n                pred_items = []\n\n            for item in pred_items:\n                predicted_items_set.add(item)\n            recommendation_count += len(pred_items)\n\n            # no predictions\n            if not pred_items:\n                continue\n\n            # get items for customer\n            actual_items = actual_purchases[customer_id]\n\n            hits = 0\n            sum_precisions = 0\n\n            for i, item in enumerate(pred_items):\n                if item in actual_items:\n                    hits += 1\n                    precision_at_i = hits / (i + 1)\n                    sum_precisions += precision_at_i\n\n            if hits > 0:\n                ap = sum_precisions / min(len(actual_items), k)\n            else:\n                ap = 0\n            ap_scores.append(ap)\n\n            # Calculate metrics\n            true_positives = len(set(pred_items) & actual_items)\n            precision = true_positives / len(pred_items) if pred_items else 0\n            recall = true_positives / len(actual_items) if actual_items else 0\n            f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0\n\n            precision_scores.append(precision)\n            recall_scores.append(recall)\n            f1_scores.append(f1)\n\n            # RMSE for binary variant, when 1 if there is a hit, 0 otherwise\n            actual_vector = np.zeros(total_items)\n            pred_vector = np.zeros(total_items)\n\n            for item in actual_items:\n                try:\n                    item_idx = int(item) % total_items\n                except ValueError:\n                    item_idx = hash(item) % total_items\n                actual_vector[item_idx] = 1\n\n            for item in pred_items:\n                try:\n                    item_idx = int(item) % total_items\n                except ValueError:\n                    item_idx = hash(item) % total_items\n                pred_vector[item_idx] = 1\n\n            rmse = np.sqrt(mean_squared_error(actual_vector, pred_vector))\n            rmse_scores.append(rmse)\n\n        results[f'map@{k}'].append(np.mean(ap_scores) if ap_scores else 0)\n        results['precision'].append(np.mean(precision_scores) if precision_scores else 0)\n        results['recall'].append(np.mean(recall_scores) if recall_scores else 0)\n        results['f1_score'].append(np.mean(f1_scores) if f1_scores else 0)\n        results['rmse'].append(np.mean(rmse_scores) if rmse_scores else 0)\n\n    metrics_df = pd.DataFrame(results)\n    return metrics_df","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T12:16:31.492008Z","iopub.execute_input":"2025-04-08T12:16:31.492366Z","iopub.status.idle":"2025-04-08T12:16:31.506635Z","shell.execute_reply.started":"2025-04-08T12:16:31.492339Z","shell.execute_reply":"2025-04-08T12:16:31.505456Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# function to ensure, we have exactly 12 predictions for each customer\n\ndef ensure_12_items(predicted_items, popular_items):\n    if not isinstance(predicted_items, list):\n        predicted_items = []  \n\n    if len(predicted_items) >= 12:\n        return predicted_items[:12]\n    else:\n        remaining = 12 - len(predicted_items)\n        popular_to_add = [item for item in popular_items if item not in predicted_items] \n        return predicted_items + popular_to_add[:remaining]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T12:16:34.857567Z","iopub.execute_input":"2025-04-08T12:16:34.858098Z","iopub.status.idle":"2025-04-08T12:16:34.863486Z","shell.execute_reply.started":"2025-04-08T12:16:34.858063Z","shell.execute_reply":"2025-04-08T12:16:34.862276Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Baselines \n## Last month purchase\nI suggest buiding a couple of baselines before implementing final recommendation model. First baselines will be about recommending most recently purchased item for last month","metadata":{}},{"cell_type":"code","source":"cutoff_date = last_date_train - timedelta(days=30)\nprint(f'Cutoff date for recent transactions: {cutoff_date}')\n\n# get transactions for last 30 days\ndf_recent_transactions_train = df_transactions_train[df_transactions_train['t_dat'] >= cutoff_date].copy()\nprint(f\"Transactions for recent period from {cutoff_date} to {last_date_train}: {df_recent_transactions_train.shape}\")\n\n# how many days ago each purchase was made\ndf_recent_transactions_train['recency'] = (last_date_train - df_recent_transactions_train['t_dat']).dt.days\n\ndf_recent_transactions_train.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T11:35:45.640279Z","iopub.execute_input":"2025-04-08T11:35:45.640769Z","iopub.status.idle":"2025-04-08T11:35:45.996658Z","shell.execute_reply.started":"2025-04-08T11:35:45.640735Z","shell.execute_reply":"2025-04-08T11:35:45.995362Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# we need to know the latest min recency for each cusomter_id and article id pairs\ncustomer_article_metrics_baseline_1 = (df_recent_transactions_train\n                                    .groupby(['customer_id_int', 'article_id'])\n                                    .agg(purchase_count=('t_dat', 'count'),\n                                         last_purchase=('t_dat', 'max'),\n                                         min_recency=('recency', 'min'))\n                                    .reset_index()\n                                   )\n\n\ncustomer_article_metrics_baseline_1.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:35:32.687015Z","iopub.execute_input":"2025-04-08T08:35:32.687472Z","iopub.status.idle":"2025-04-08T08:35:33.445176Z","shell.execute_reply.started":"2025-04-08T08:35:32.687432Z","shell.execute_reply":"2025-04-08T08:35:33.443980Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"customer_article_metrics_baseline_1.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:52:35.490636Z","iopub.execute_input":"2025-04-08T08:52:35.491364Z","iopub.status.idle":"2025-04-08T08:52:35.506445Z","shell.execute_reply.started":"2025-04-08T08:52:35.491325Z","shell.execute_reply":"2025-04-08T08:52:35.505125Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# sorting the metrics. Purchase sorting is irrelevant. We want to know only customer id and it's most recent number of days\ncustomer_article_metrics_baseline_1 = customer_article_metrics_baseline_1.sort_values(\n    ['customer_id_int', 'purchase_count', 'min_recency'],\n    ascending=[True, False, True]\n)\n\n# previous sorting was neeeded for this drop of duplicates. Only top recensy remain\ncustomer_article_metrics_baseline_1 = customer_article_metrics_baseline_1.drop_duplicates(['customer_id_int', 'article_id'])\n\n\ncustomer_article_metrics_baseline_1.head()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:52:43.115259Z","iopub.execute_input":"2025-04-08T08:52:43.115638Z","iopub.status.idle":"2025-04-08T08:52:43.625327Z","shell.execute_reply.started":"2025-04-08T08:52:43.115607Z","shell.execute_reply":"2025-04-08T08:52:43.624108Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Convert customer_article_metrics to dictionary for faster search- hashtable is fast\n\ncustomer_items_baseline_1 = {}\nfor _, row in customer_article_metrics_baseline_1.iterrows():\n    customer_id = row['customer_id_int']\n    article_id = row['article_id']\n\n    if customer_id not in customer_items_baseline_1:\n        customer_items_baseline_1[customer_id] = []\n\n    if len(customer_items_baseline_1[customer_id]) < 12:  # need 12 items for customer\n        customer_items_baseline_1[customer_id].append(article_id)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:52:46.486072Z","iopub.execute_input":"2025-04-08T08:52:46.486603Z","iopub.status.idle":"2025-04-08T08:53:47.171739Z","shell.execute_reply.started":"2025-04-08T08:52:46.486567Z","shell.execute_reply":"2025-04-08T08:53:47.170592Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# solving issues with cold start and filling missing predictions\n# popular items from last 2 weeks\n\nlast_2weeks_date = last_date_train - timedelta(days=14)\nlast_2weeks_transactions_train = df_transactions_train[df_transactions_train['t_dat'] >= last_2weeks_date] \npopular_items_baseline_1 = last_2weeks_transactions_train['article_id'].value_counts().head(12).index.tolist()\n\nprint(popular_items_baseline_1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T08:53:47.173309Z","iopub.execute_input":"2025-04-08T08:53:47.173693Z","iopub.status.idle":"2025-04-08T08:53:47.458501Z","shell.execute_reply.started":"2025-04-08T08:53:47.173640Z","shell.execute_reply":"2025-04-08T08:53:47.457219Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create predictions dataframe\npredictions_baseline_1 = []\nfor customer_id, items in customer_items_baseline_1.items():\n    items_str = ' '.join(items)\n    predictions_baseline_1.append((customer_id, items_str))\n\ndf_predictions_baseline_1 = pd.DataFrame(predictions_baseline_1, columns=['customer_id_int', 'prediction'])\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:15:17.044300Z","iopub.execute_input":"2025-04-08T09:15:17.044821Z","iopub.status.idle":"2025-04-08T09:15:17.421806Z","shell.execute_reply.started":"2025-04-08T09:15:17.044787Z","shell.execute_reply":"2025-04-08T09:15:17.420466Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictions_list_baseline_1 = []\nfor customer_id, items in customer_items_baseline_1.items():\n    items_str = ' '.join(map(str, items)) # items to strings\n    predictions_list_baseline_1.append((customer_id, items_str))\n\ndf_predictions_baseline_1 = pd.DataFrame(predictions_list_baseline_1, columns=['customer_id_int', 'prediction'])\n\nprint(df_predictions_baseline_1.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:15:25.099511Z","iopub.execute_input":"2025-04-08T09:15:25.099871Z","iopub.status.idle":"2025-04-08T09:15:25.875675Z","shell.execute_reply.started":"2025-04-08T09:15:25.099844Z","shell.execute_reply":"2025-04-08T09:15:25.874343Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # submission template\n# submission_baseline_1 = df_sample_submission.merge(\n#     df_predictions_baseline_1, on='customer_id_int', how='left'\n# ).fillna('')\n\n# # filling na\n# submission_baseline_1['prediction'] = submission_baseline_1['prediction'].apply(\n#     lambda x: x if x != '' else ' '.join(map(str, popular_items_baseline_1[:12])) \n# )\n\n# submission_baseline_1['prediction'] = submission_baseline_1['prediction'].apply(lambda x: ensure_12_items(x, popular_items_baseline_1))\n\n# submission_baseline_1_final = submission_baseline_1[['customer_id', 'prediction']]\n\n# print(submission_baseline_1_final.head())","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-04-08T08:35:33.628856Z","iopub.status.idle":"2025-04-08T08:35:33.629459Z","shell.execute_reply":"2025-04-08T08:35:33.629239Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"metrics_baseline_one = evaluate_recommendations(\n    df_transactions_test, \n    [df_predictions_baseline_1],\n    k=12\n)\n\nmetrics_baseline_one","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:15:29.881843Z","iopub.execute_input":"2025-04-08T09:15:29.882357Z","iopub.status.idle":"2025-04-08T09:16:18.774661Z","shell.execute_reply.started":"2025-04-08T09:15:29.882325Z","shell.execute_reply":"2025-04-08T09:16:18.773490Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Thoughts** \\\nmap@12\t0.0414\t- means items are appearing in real data pretty rarely. Metric is obviosly low, but that's the part of the way. Just to get started we need some baseline to compare it to something. Ranking is not high for this one.\n\nPrecision - 0.0371 - it means that only about 3 percent of items being predicted correctly. That's low.\n\nF1 score is also very low. That's it for first baseline. \\\n\nTime to shine for the second baseline.","metadata":{}},{"cell_type":"markdown","source":"## Second baseline\n\nThis one will be about to match items being bought together. ","metadata":{}},{"cell_type":"code","source":"# like for first baseline\ncutoff_date_baseline_2 = last_date_train - timedelta(days=30)\ndf_transactions_train_baseline_2 = df_transactions[df_transactions['t_dat'] < test_start_date].copy()\nlast_date_train_baseline_2 = df_transactions_train_baseline_2['t_dat'].max() \ndf_recent_transactions_train_baseline_2 = df_transactions_train_baseline_2[df_transactions_train_baseline_2['t_dat'] >= cutoff_date_baseline_2].copy() \ndf_recent_transactions_train_baseline_2['recency'] = (last_date_train_baseline_2 - df_recent_transactions_train_baseline_2['t_dat']).dt.days\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:18:04.218549Z","iopub.execute_input":"2025-04-08T09:18:04.219003Z","iopub.status.idle":"2025-04-08T09:18:10.529226Z","shell.execute_reply.started":"2025-04-08T09:18:04.218973Z","shell.execute_reply":"2025-04-08T09:18:10.528236Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"customer_article_metrics_baseline_2 = (df_recent_transactions_train_baseline_2 # Specific name for baseline 2\n                                    .groupby(['customer_id_int', 'article_id'])\n                                    .agg(purchase_count=('t_dat', 'count'),\n                                         last_purchase=('t_dat', 'max'),\n                                         min_recency=('recency', 'min'))\n                                    .reset_index()\n                                   )\n\ncustomer_article_metrics_baseline_2.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:18:10.530607Z","iopub.execute_input":"2025-04-08T09:18:10.530993Z","iopub.status.idle":"2025-04-08T09:18:11.367450Z","shell.execute_reply.started":"2025-04-08T09:18:10.530963Z","shell.execute_reply":"2025-04-08T09:18:11.366173Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# remove duplicates - like for first baseline\n\ncustomer_article_metrics_baseline_2 = customer_article_metrics_baseline_2.sort_values(['customer_id_int', 'purchase_count', 'min_recency'],\n    ascending=[True, False, True])\n\ncustomer_article_metrics_baseline_2 = customer_article_metrics_baseline_2.drop_duplicates(['customer_id_int', 'article_id']) \n\ncustomer_article_metrics_baseline_2.head()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:18:11.369505Z","iopub.execute_input":"2025-04-08T09:18:11.369967Z","iopub.status.idle":"2025-04-08T09:18:11.864737Z","shell.execute_reply.started":"2025-04-08T09:18:11.369933Z","shell.execute_reply":"2025-04-08T09:18:11.863606Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# convert for fast search\n\ncustomer_items_baseline_2_strategy_1 = {} \nfor _, row in customer_article_metrics_baseline_2.iterrows(): \n    customer_id = row['customer_id_int']\n    article_id = row['article_id']\n\n    if customer_id not in customer_items_baseline_2_strategy_1:\n        customer_items_baseline_2_strategy_1[customer_id] = [] \n\n    if len(customer_items_baseline_2_strategy_1[customer_id]) < 12: \n        customer_items_baseline_2_strategy_1[customer_id].append(article_id) ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:18:12.774591Z","iopub.execute_input":"2025-04-08T09:18:12.774961Z","iopub.status.idle":"2025-04-08T09:19:12.145927Z","shell.execute_reply.started":"2025-04-08T09:18:12.774925Z","shell.execute_reply":"2025-04-08T09:19:12.144680Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"temp_df_baseline_2 = df_transactions_train_baseline_2.sort_values(['customer_id_int', 't_dat'])\n\npairs_dict_baseline_2 = {} \nwindow_days = 7\n\nsample_size = min(10000, len(temp_df_baseline_2['customer_id_int'].unique())) \ncustomer_sample = np.random.choice(temp_df_baseline_2['customer_id_int'].unique(), size=sample_size, replace=False) \n\nfor customer_id in customer_sample:\n    customer_purchases = temp_df_baseline_2[temp_df_baseline_2['customer_id_int'] == customer_id] \n\n    for i, row1 in customer_purchases.iterrows():\n        item1 = row1['article_id']\n        purchase_date = row1['t_dat']\n\n        window_purchases = customer_purchases[\n            (customer_purchases['t_dat'] >= purchase_date) &\n            (customer_purchases['t_dat'] <= purchase_date + timedelta(days=window_days)) &\n            (customer_purchases['article_id'] != item1)\n        ]\n\n        for item2 in window_purchases['article_id'].unique():\n            pair = (item1, item2)\n            if pair not in pairs_dict_baseline_2: \n                pairs_dict_baseline_2[pair] = 0 \n            pairs_dict_baseline_2[pair] += 1 \n\n# Convert dict to dataframe\npairs_df_baseline_2 = pd.DataFrame([(k[0], k[1], v) for k, v in pairs_dict_baseline_2.items()], columns=['item1', 'item2', 'count'])\npairs_df_baseline_2 = pairs_df_baseline_2.sort_values('count', ascending=False)  \n\nitem_to_pair_baseline_2 = {} \nfor _, row in pairs_df_baseline_2.iterrows(): \n    item1 = row['item1']\n    item2 = row['item2']\n    if item1 not in item_to_pair_baseline_2: \n        item_to_pair_baseline_2[item1] = item2 \n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:19:12.147551Z","iopub.execute_input":"2025-04-08T09:19:12.147864Z","iopub.status.idle":"2025-04-08T09:31:53.560893Z","shell.execute_reply.started":"2025-04-08T09:19:12.147827Z","shell.execute_reply":"2025-04-08T09:31:53.559392Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"customer_items_baseline_2 = customer_items_baseline_2_strategy_1.copy() \n\nfor customer_id, items in customer_items_baseline_2.items():\n    paired_items = []\n    for item in items:\n        if item in item_to_pair_baseline_2: \n            paired_items.append(item_to_pair_baseline_2[item]) \n\n    for item in paired_items:\n        if item not in customer_items_baseline_2[customer_id] and len(customer_items_baseline_2[customer_id]) < 12:\n            customer_items_baseline_2[customer_id].append(item)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:31:53.563150Z","iopub.execute_input":"2025-04-08T09:31:53.563521Z","iopub.status.idle":"2025-04-08T09:31:54.804550Z","shell.execute_reply.started":"2025-04-08T09:31:53.563478Z","shell.execute_reply":"2025-04-08T09:31:54.803642Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# predictions \npredictions_list_baseline_2 = []\nfor customer_id, items in customer_items_baseline_2.items():\n    items_str = ' '.join(map(str, items)) \n    predictions_list_baseline_2.append((customer_id, items_str))\n\ndf_predictions_baseline_2 = pd.DataFrame(predictions_list_baseline_2, columns=['customer_id_int', 'prediction'])\n\ndf_predictions_baseline_2.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:31:54.805821Z","iopub.execute_input":"2025-04-08T09:31:54.806383Z","iopub.status.idle":"2025-04-08T09:31:55.416472Z","shell.execute_reply.started":"2025-04-08T09:31:54.806346Z","shell.execute_reply":"2025-04-08T09:31:55.415303Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# submission_baseline_2 = df_sample_submission.merge(df_predictions_baseline_2, on='customer_id_int', how='left').fillna('')\n\n# submission_baseline_2['prediction'] = submission_baseline_2['prediction'].apply(lambda x: x if x != '' else ' '.join(map(str, popular_items_baseline_2[:12])) )\n\n# submission_baseline_2['prediction'] = submission_baseline_2['prediction'].apply(lambda x: ensure_12_items(x, popular_items_baseline_2)) \n\n# submission_baseline_2_final = submission_baseline_2[['customer_id', 'prediction']]\n\n# print(submission_baseline_2_final.head())","metadata":{"trusted":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2025-04-08T08:35:33.642460Z","iopub.status.idle":"2025-04-08T08:35:33.642951Z","shell.execute_reply":"2025-04-08T08:35:33.642731Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"metrics_baseline_two = evaluate_recommendations(\n    df_transactions_test,\n    [df_predictions_baseline_2],\n    k=12\n)\n\nmetrics_baseline_two","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T09:31:55.417596Z","iopub.execute_input":"2025-04-08T09:31:55.417909Z","iopub.status.idle":"2025-04-08T09:32:43.988871Z","shell.execute_reply.started":"2025-04-08T09:31:55.417877Z","shell.execute_reply":"2025-04-08T09:32:43.987759Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Thoughts**\n\nVery much mixed picture on metrics. \n\nmap@12 increased slightly, but still low. We miss predictions too much. Relevant items in top 12 ranking a little better with baseline 2. Improvement very slight. \n\nIn the same time there is precision drop compare to first baseline, so actually odd of seeing right recommendation dropped a lot. \n\nf1 score also droppped due to high drop in precision. \n\nConclusion - both baselines are weak and not going to any production :)","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# ALS model","metadata":{}},{"cell_type":"code","source":"# This function will create sparse matrix - need this for collaborative filtering\ndef create_user_item_matrix(df, all_users, all_items):\n    # Need integer indices for sparse matrix\n    row = df['user_idx'].astype(int).values\n    col = df['item_idx'].astype(int).values\n    data = np.ones(df.shape[0]) \n    \n    coo = coo_matrix((data, (row, col)), shape=(len(all_users), len(all_items)))\n    return coo","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:40.479701Z","iopub.execute_input":"2025-04-08T13:31:40.480220Z","iopub.status.idle":"2025-04-08T13:31:40.485438Z","shell.execute_reply.started":"2025-04-08T13:31:40.480176Z","shell.execute_reply":"2025-04-08T13:31:40.484375Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# function returning recommendations. There is several datasets, like submission and test datasets\ndef generate_recommendations(model, user_indices, csr_train, item_ids, num_recommendations=12):\n    recommendations = {}\n    \n    # Loop through users to get recommendations\n    for user_idx in tqdm(user_indices):\n        user_idx = int(user_idx)  # Convert to int just to be safe\n        \n        user_items = csr_train[user_idx] if user_idx < csr_train.shape[0] else None\n        \n        try:\n            recommended_items, _ = model.recommend(\n                user_idx, \n                user_items,\n                N=num_recommendations,\n                filter_already_liked_items=True  # do not recommend items already existed\n            )\n            \n            # Need indexes for further metrics evaluation\n            rec_items = [item_ids[int(item_idx)] for item_idx in recommended_items]\n            recommendations[user_idx] = rec_items\n        except Exception as e: # if error encountered. In some tests before fixes has been a case\n            recommendations[user_idx] = []\n    \n    return recommendations","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:41.715487Z","iopub.execute_input":"2025-04-08T13:31:41.715850Z","iopub.status.idle":"2025-04-08T13:31:41.721757Z","shell.execute_reply.started":"2025-04-08T13:31:41.715819Z","shell.execute_reply":"2025-04-08T13:31:41.720749Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def prepare_submission(recommendations, user_id_mapping, submission_df):\n    submission = submission_df.copy()\n    \n    id_to_recs = {user_id_mapping[user_idx]: recs \n                  for user_idx, recs in recommendations.items() \n                  if user_idx in user_id_mapping}\n    \n    submission['prediction'] = submission['customer_id_int'].map(\n        lambda x: ' '.join(id_to_recs.get(x, [])) if x in id_to_recs else ''\n    )\n    \n    return submission","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:44.441593Z","iopub.execute_input":"2025-04-08T13:31:44.441989Z","iopub.status.idle":"2025-04-08T13:31:44.447197Z","shell.execute_reply.started":"2025-04-08T13:31:44.441950Z","shell.execute_reply":"2025-04-08T13:31:44.446183Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def train_model(matrices, factors=100, iterations=15, regularization=0.01):\n    coo_train = matrices['coo_train']\n    \n    model_als = AlternatingLeastSquares(\n        factors=factors,\n        iterations=iterations,\n        regularization=regularization,\n        random_state=42\n    )\n    \n    model_als.fit(coo_train)\n    \n    return model_als","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:45.442585Z","iopub.execute_input":"2025-04-08T13:31:45.442946Z","iopub.status.idle":"2025-04-08T13:31:45.447844Z","shell.execute_reply.started":"2025-04-08T13:31:45.442911Z","shell.execute_reply":"2025-04-08T13:31:45.446933Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Data preparation - let's focus on recent purchases only\nprint(\"Checking original data size:\", df_transactions.shape)\n\n# Filter to recent data (last 60 days) - older data isn't that useful anyway\ncutoff_date = df_transactions['t_dat'].max() - timedelta(days=60)\nfiltered_transactions = df_transactions[df_transactions['t_dat'] > cutoff_date].copy()\nprint(\"Using only recent transactions:\", filtered_transactions.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:46.909512Z","iopub.execute_input":"2025-04-08T13:31:46.909837Z","iopub.status.idle":"2025-04-08T13:31:47.286160Z","shell.execute_reply.started":"2025-04-08T13:31:46.909809Z","shell.execute_reply":"2025-04-08T13:31:47.285243Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# get unique users and items\nals_users = filtered_transactions['customer_id_int'].unique().tolist()\nals_items = filtered_transactions['article_id'].unique().tolist()\n\n# ids to indexes\nals_user_map = {user_id: idx for idx, user_id in enumerate(als_users)}\nals_item_map = {item_id: idx for idx, item_id in enumerate(als_items)}\n\n# Add index columns to the dataframe\nals_df = filtered_transactions.copy()\nals_df['user_idx'] = als_df['customer_id_int'].map(als_user_map).astype(int)\nals_df['item_idx'] = als_df['article_id'].map(als_item_map).astype(int)\n\nals_user_ids = {idx: user_id for user_id, idx in als_user_map.items()}\nals_item_ids = {idx: item_id for item_id, idx in als_item_map.items()}\n\nprint(len(als_users), len(als_items))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:48.972367Z","iopub.execute_input":"2025-04-08T13:31:48.972676Z","iopub.status.idle":"2025-04-08T13:31:50.015405Z","shell.execute_reply.started":"2025-04-08T13:31:48.972651Z","shell.execute_reply":"2025-04-08T13:31:50.014289Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# train test split for als model\nvalidation_days = 7\nvalidation_cutoff = als_df['t_dat'].max() - pd.Timedelta(days=validation_days)\ndf_train = als_df[als_df['t_dat'] < validation_cutoff]\ndf_val = als_df[als_df['t_dat'] >= validation_cutoff]\n\n# training matrix for als\ncoo_train = create_user_item_matrix(df_train, als_users, als_items)\ncsr_train = coo_train.tocsr()  # CSR format is faster for some operations\n\n# test matrix for als\ncoo_val = create_user_item_matrix(df_val, als_users, als_items)\ncsr_val = coo_val.tocsr()\n\n# dict of metrics\nals_matrices = {\n    'coo_train': coo_train,\n    'csr_train': csr_train,\n    'csr_val': csr_val,\n    'df_train': df_train,\n    'df_val': df_val\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:31:51.459936Z","iopub.execute_input":"2025-04-08T13:31:51.460307Z","iopub.status.idle":"2025-04-08T13:31:51.745836Z","shell.execute_reply.started":"2025-04-08T13:31:51.460278Z","shell.execute_reply":"2025-04-08T13:31:51.744727Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Search for best latent factors\nfactors_to_test = list(range(10, 211, 20)) \niterations = 20  # will leave as it is iterations and regularization\nregularization = 0.01  \n\nresults = []\n\n# loop over factors\nfor factors in factors_to_test:\n    print(f\"current factors ={factors}...\")\n    try:\n        model = train_model(\n            als_matrices,\n            factors=factors,\n            iterations=iterations,\n            regularization=regularization\n        )\n        \n        # predictions\n        val_users = df_val['user_idx'].unique()[:100] \n        \n        val_recommendations = {}\n        for user_idx in val_users:\n            user_idx = int(user_idx)\n            try:\n                rec_items, _ = model.recommend(\n                    user_idx,\n                    csr_train[user_idx],\n                    N=12,\n                    filter_already_liked_items=True\n                )\n                val_recommendations[als_user_ids[user_idx]] = [als_item_ids[int(item)] for item in rec_items]\n            except:\n                val_recommendations[als_user_ids[user_idx]] = []\n        \n        # put predictions to dataframe\n        val_pred_df = pd.DataFrame({\n            'customer_id_int': list(val_recommendations.keys()),\n            'prediction': [' '.join(items) for items in val_recommendations.values()]\n        })\n        \n        # get metrics\n        metrics = evaluate_recommendations(df_val, [val_pred_df], k=12)\n        \n        print(f\"Factors: {factors} - MAP@12: {metrics['map@12'][0]:.4f} - \"\n              f\"Precision: {metrics['precision'][0]:.4f} - Recall: {metrics['recall'][0]:.4f}\")\n        \n        results.append({\n            'factors': factors,\n            'map': metrics['map@12'][0],\n            'precision': metrics['precision'][0],\n            'recall': metrics['recall'][0],\n            'f1_score': metrics['f1_score'][0],\n            'rmse': metrics['rmse'][0]\n        })\n    except Exception as e:\n        print(f\"there is error\")\n        print(e)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:33:49.451128Z","iopub.execute_input":"2025-04-08T13:33:49.451443Z","iopub.status.idle":"2025-04-08T13:41:53.977304Z","shell.execute_reply.started":"2025-04-08T13:33:49.451418Z","shell.execute_reply":"2025-04-08T13:41:53.976364Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Find best parameters\nresults_df = pd.DataFrame(results)\nprint(results_df)\n\n# best model and MAP@12\nbest_params = results_df.loc[results_df['map'].idxmax()].to_dict()\nbest_params","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:42:15.482708Z","iopub.execute_input":"2025-04-08T13:42:15.483058Z","iopub.status.idle":"2025-04-08T13:42:15.493248Z","shell.execute_reply.started":"2025-04-08T13:42:15.483031Z","shell.execute_reply":"2025-04-08T13:42:15.492201Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"По показателям MAP и точности не самая высокая точность, чуть больше 1% рекомендаций попадают в реальные покупки пользователей. Модель и проект имеет широкое окно для улучшений.","metadata":{}},{"cell_type":"code","source":"# train best model on full dataset\nbest_factors = int(best_params['factors'])\n\nfull_coo_train = create_user_item_matrix(als_df, als_users, als_items)\nfull_csr_train = full_coo_train.tocsr()\n\nfinal_model = train_model(\n    {'coo_train': full_coo_train},\n    factors=best_factors,\n    iterations=iterations,\n    regularization=regularization\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:42:21.489863Z","iopub.execute_input":"2025-04-08T13:42:21.490273Z","iopub.status.idle":"2025-04-08T13:42:53.783964Z","shell.execute_reply.started":"2025-04-08T13:42:21.490238Z","shell.execute_reply":"2025-04-08T13:42:53.782972Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Generate recommendations for all users in the submission dataframe\nsubmission_users = df_sample_submission['customer_id_int'].unique()\n\nsubmission_user_indices = []\nfor user_id in submission_users:\n    if user_id in als_user_map:\n        submission_user_indices.append(als_user_map[user_id])\n\nrecommendations = generate_recommendations(\n    final_model,\n    submission_user_indices,\n    full_csr_train,\n    als_item_ids,\n    num_recommendations=12\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:42:53.785272Z","iopub.execute_input":"2025-04-08T13:42:53.785563Z","iopub.status.idle":"2025-04-08T13:48:20.330419Z","shell.execute_reply.started":"2025-04-08T13:42:53.785537Z","shell.execute_reply":"2025-04-08T13:48:20.329488Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Conclusions and implications.\n- The ALS model showed a significant improvement over baseline models, although the MAP@12, Precision and Recall metrics are still relatively low. This may indicate the complexity of the problem and the need for more advanced methods.\n- **The following improvements**:\n    - **Data expansion - images, metadata**\n - **More architectures and approaches**\n - **Shier hyperparameter search**\n - **Accounting for session activity**\n\n**Conclusion**\nThis pet project represents a good start to developing a recommendation system for H&M.\n\n\nЗаключения и выводы.\n- Модель ALS показала значительное улучшение по сравнению с baseline-моделями, хотя метрики MAP@12, Precision и Recall все еще остаются относительно низкими. Это может указывать на сложность задачи и необходимость более продвинутых методов.\n- **Следующие улучшения**:\n    - **Расширение данных - изображения, метаданные**\n    - **Больше архитектур и подходов**\n    - **Шире поиск по гиперпараметрам**\n    - **Учет сессионной активности**\n\n**Заключение**\nЭтот pet-проект представляет собой хороший старт для разработки рекомендательной системы для H&M.","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Prepare submission\n# als_submission = prepare_submission(\n#     recommendations,\n#     als_user_ids,\n#     df_sample_submission\n# )\n\n# # handling missings\n# popular_items = als_df['article_id'].value_counts().index.tolist()[:12]\n\n# empty_recs = als_submission[als_submission['prediction'] == ''] # no recommendations\n# print(len(empty_recs))\n\n# for idx in empty_recs.index:\n#     als_submission.loc[idx, 'prediction'] = ' '.join(ensure_12_items([], popular_items))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:48:20.332206Z","iopub.execute_input":"2025-04-08T13:48:20.332561Z","iopub.status.idle":"2025-04-08T13:50:47.283497Z","shell.execute_reply.started":"2025-04-08T13:48:20.332534Z","shell.execute_reply":"2025-04-08T13:50:47.282537Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Save submission file\n# als_submission[['customer_id', 'prediction']].to_csv('als_recommendations.csv', index=False)\n# print(\"Saved submission file as 'als_recommendations.csv'\")\n\n# # Run full evaluation on our validation set\n# # Create predictions dataframe for validation set\n# val_users = df_val['customer_id_int'].unique()\n\n# # Match format for evaluation\n# val_preds = []\n# for user_id in val_users:\n#     if user_id in als_user_map:\n#         user_idx = als_user_map[user_id]\n#         try:\n#             rec_items, _ = final_model.recommend(\n#                 user_idx,\n#                 full_csr_train[user_idx],\n#                 N=12,\n#                 filter_already_liked_items=True\n#             )\n#             val_preds.append({\n#                 'customer_id_int': user_id,\n#                 'prediction': ' '.join([als_item_ids[int(item)] for item in rec_items])\n#             })\n#         except:\n#             val_preds.append({\n#                 'customer_id_int': user_id,\n#                 'prediction': ' '.join(popular_items)\n#             })\n#     else:\n#         val_preds.append({\n#             'customer_id_int': user_id,\n#             'prediction': ' '.join(popular_items)\n#         })\n\n# val_pred_df = pd.DataFrame(val_preds)\n\n# # Calculate final metrics\n# final_metrics = evaluate_recommendations(df_val, [val_pred_df], k=12)\n# print(\"\\nFinal model performance:\")\n# print(f\"MAP@12: {final_metrics['map@12'][0]:.4f}\")\n# print(f\"Precision: {final_metrics['precision'][0]:.4f}\")\n# print(f\"Recall: {final_metrics['recall'][0]:.4f}\")\n# print(f\"F1 Score: {final_metrics['f1_score'][0]:.4f}\")\n# print(f\"RMSE: {final_metrics['rmse'][0]:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T13:50:47.284515Z","iopub.execute_input":"2025-04-08T13:50:47.284779Z","iopub.status.idle":"2025-04-08T13:52:42.345084Z","shell.execute_reply.started":"2025-04-08T13:50:47.284755Z","shell.execute_reply":"2025-04-08T13:52:42.344184Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}