{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\nimport gc\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-03T17:40:48.516581Z","iopub.execute_input":"2022-05-03T17:40:48.516971Z","iopub.status.idle":"2022-05-03T17:40:48.546115Z","shell.execute_reply.started":"2022-05-03T17:40:48.516874Z","shell.execute_reply":"2022-05-03T17:40:48.545438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# About my pipeline and notebook","metadata":{}},{"cell_type":"markdown","source":"**General idea about my pipeline:**\n1) Preprocessing input tables in pyspark\n\n2) Build features related to last week of transactions in a rolling way, based on the idea that my ranker should be reactive: consider trends, top sold items, other ideas derivable from Transactions table\n\n3) For customers and items tables I will build simple numerical and categorical features (some features could be one hot encoded)\n\n4) Generate negative observations: the strategy to generate those is the most important step of the challenge. At the moment I leverage on the analysis I performed on the transactions table. The final train table will contain *x customers times n test rows*, therefore the dataset becomes quickly heavy. The same can be said about application table: with 20 tests per user, the table becomes around 25mil rows.\n\n5) At the end of the pipeline I want to save locally a train dataset and the application dataset in order to leave the Ram as free as possible for model development. I will do this with pyspark to csv command because it is not possible to perform toPandas() with big tables.\n\n6) My model will be a LightGBM ranker.\n\n**Current attention points**\n\n1) Need to generate new strategies for candidate items\n2) Though the pipeline runs, Ram is becoming a problem again","metadata":{}},{"cell_type":"markdown","source":"# This section contains prints of descriptive information about the input datasets","metadata":{}},{"cell_type":"code","source":"# articles: (105542, 25)\n#######################################\n# unique for each col: article_id                      105542\n# product_code                     47224\n# prod_name                        45875\n# product_type_no                    132\n# product_type_name                  131\n# product_group_name                  19\n# graphical_appearance_no             30\n# graphical_appearance_name           30\n# colour_group_code                   50\n# colour_group_name                   50\n# perceived_colour_value_id            8\n# perceived_colour_value_name          8\n# perceived_colour_master_id          20\n# perceived_colour_master_name        20\n# department_no                      299\n# department_name                    250\n# index_code                          10\n# index_name                          10\n# index_group_no                       5\n#index_group_name                     5\n# section_no                          57\n# section_name                        56\n# garment_group_no                    21\n# garment_group_name                  21\n# detail_desc                      43404\n# dtype: int64\n#######################################\n# null count: article_id                        0\n# detail_desc                     416\n# dtype: int64\n#######################################","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:40:48.57914Z","iopub.execute_input":"2022-05-03T17:40:48.579651Z","iopub.status.idle":"2022-05-03T17:40:48.584224Z","shell.execute_reply.started":"2022-05-03T17:40:48.579617Z","shell.execute_reply":"2022-05-03T17:40:48.583352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# customers: (1371980, 7)\n#######################################\n# unique for each col: customer_id               1371980\n# FN                              1\n# Active                          1\n# club_member_status              3\n# fashion_news_frequency          4\n# age                            84\n# postal_code                352899\n#dtype: int64\n#######################################\n# null count: customer_id                    0\n# FN                        895050\n# Active                    907576\n# club_member_status          6062\n# fashion_news_frequency     16009\n# age                        15861\n# postal_code                    0\n# dtype: int64\n#######################################\n","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:40:48.626709Z","iopub.execute_input":"2022-05-03T17:40:48.627602Z","iopub.status.idle":"2022-05-03T17:40:48.631724Z","shell.execute_reply.started":"2022-05-03T17:40:48.62753Z","shell.execute_reply":"2022-05-03T17:40:48.631153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# transactions: (31788324, 5)\n#######################################\n# unique for each col: t_dat                   734\n# customer_id         1362281\n# article_id           104547\n# price                  9857\n# sales_channel_id          2\n# dtype: int64\n#######################################\n# null count: t_dat               0\n#######################################\n","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:40:48.673265Z","iopub.execute_input":"2022-05-03T17:40:48.673683Z","iopub.status.idle":"2022-05-03T17:40:48.67691Z","shell.execute_reply.started":"2022-05-03T17:40:48.673653Z","shell.execute_reply":"2022-05-03T17:40:48.676369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#application: (1371980, 2)\n#######################################\n# unique for each col: customer_id    1371980\n# prediction           1\n# dtype: int64\n#######################################\n# null count: customer_id    0\n# prediction     0\n# dtype: int64\n#######################################\n","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:40:48.712556Z","iopub.execute_input":"2022-05-03T17:40:48.713324Z","iopub.status.idle":"2022-05-03T17:40:48.719725Z","shell.execute_reply.started":"2022-05-03T17:40:48.713282Z","shell.execute_reply":"2022-05-03T17:40:48.718913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Pyspark","metadata":{}},{"cell_type":"markdown","source":"pyspark will be used for feature engineering and preprocessing","metadata":{}},{"cell_type":"code","source":"!pip install pyspark -q\nimport pyspark\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql import SQLContext\nfrom pyspark.sql import functions as F\nfrom pyspark.sql import Window\nfrom pyspark.sql.types import StructType,StructField, StringType, IntegerType, ArrayType, DoubleType, BooleanType\n\nsc = SparkSession.builder.appName(\"Recommendations\").config(\"spark.sql.files.maxPartitionBytes\", 5000000).getOrCreate()\nspark = SparkSession(sc)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:40:48.753401Z","iopub.execute_input":"2022-05-03T17:40:48.753908Z","iopub.status.idle":"2022-05-03T17:41:41.40528Z","shell.execute_reply.started":"2022-05-03T17:40:48.753874Z","shell.execute_reply":"2022-05-03T17:41:41.404299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"articles = spark.read.option(\"header\",True) \\\n                .csv(\"../input/h-and-m-personalized-fashion-recommendations/articles.csv\")\ncustomers = spark.read.option(\"header\",True) \\\n                .csv(\"../input/h-and-m-personalized-fashion-recommendations/customers.csv\")\ntransactions = spark.read.option(\"header\",True) \\\n                .csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:41:41.407295Z","iopub.execute_input":"2022-05-03T17:41:41.412634Z","iopub.status.idle":"2022-05-03T17:41:46.912181Z","shell.execute_reply.started":"2022-05-03T17:41:41.412567Z","shell.execute_reply":"2022-05-03T17:41:46.911314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Articles table, simplified","metadata":{}},{"cell_type":"code","source":"articles = articles\\\n    .selectExpr('cast (article_id as int) article_id', 'cast (product_type_no as int) product_type_no', 'cast (graphical_appearance_no as int) graphical_appearance_no',\n                'cast (colour_group_code as int) colour_group_code ','cast (perceived_colour_value_id as int) perceived_colour_value_id',\n                'cast (department_no as int) department_no',  'cast (index_group_no as int) index_group_no',\n                'cast (section_no as int) section_no', 'cast (garment_group_no as int) garment_group_no')\\\n    .dropDuplicates()\n\narticles.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:41:46.91334Z","iopub.execute_input":"2022-05-03T17:41:46.91362Z","iopub.status.idle":"2022-05-03T17:41:50.252818Z","shell.execute_reply.started":"2022-05-03T17:41:46.913584Z","shell.execute_reply":"2022-05-03T17:41:50.251896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Customer table","metadata":{}},{"cell_type":"code","source":"customers = customers\\\n    .fillna({'age': '25'})\\\n    .drop('FN', 'Active', 'club_member_status', 'fashion_news_frequency', 'postal_code')","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:41:50.254661Z","iopub.execute_input":"2022-05-03T17:41:50.255636Z","iopub.status.idle":"2022-05-03T17:41:50.377207Z","shell.execute_reply.started":"2022-05-03T17:41:50.255586Z","shell.execute_reply":"2022-05-03T17:41:50.376314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Weeks preprocessing","metadata":{}},{"cell_type":"code","source":"start_date = '2020-07-22'\n\nmin_week = 1\nmax_week = 9\napplication_week = max_week + 1\n#week: changed to tuesday\nprint(application_week)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:41:50.381481Z","iopub.execute_input":"2022-05-03T17:41:50.381836Z","iopub.status.idle":"2022-05-03T17:41:50.388137Z","shell.execute_reply.started":"2022-05-03T17:41:50.381788Z","shell.execute_reply":"2022-05-03T17:41:50.387349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transactions = transactions\\\n    .withColumn('article_id', transactions['article_id'].cast(IntegerType()))\\\n    .filter(F.col('t_dat') >= start_date)\\\n    .withColumn('week', F.when((F.col('t_dat') >= '2020-09-16') & (F.col('t_dat') <= '2020-09-22'), 9)\n                         .when((F.col('t_dat') >= '2020-09-09') & (F.col('t_dat') <= '2020-09-15'), 8)\n                         .when((F.col('t_dat') >= '2020-09-02') & (F.col('t_dat') <= '2020-09-08'), 7)\n                         .when((F.col('t_dat') >= '2020-08-26') & (F.col('t_dat') <= '2020-09-01'), 6)\n                         .when((F.col('t_dat') >= '2020-08-19') & (F.col('t_dat') <= '2020-08-25'), 5)\n                         .when((F.col('t_dat') >= '2020-08-12') & (F.col('t_dat') <= '2020-08-18'), 4)\n                         .when((F.col('t_dat') >= '2020-08-05') & (F.col('t_dat') <= '2020-08-11'), 3)\n                         .when((F.col('t_dat') >= '2020-07-29') & (F.col('t_dat') <= '2020-08-04'), 2)\n                         .when((F.col('t_dat') >= '2020-07-22') & (F.col('t_dat') <= '2020-07-28'), 1)\n                        .otherwise(999))\\\n    .drop('t_dat', 'price', 'sales_channel_id')\\\n    .orderBy(['week', 'customer_id'], ascending=True)\n\ntransactions.show(10)\n# code to generate unique transaction id:\n# .withColumn('t_id', F.concat_ws('_',transactions.t_dat, transactions.customer_id))\\","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:41:50.389837Z","iopub.execute_input":"2022-05-03T17:41:50.390396Z","iopub.status.idle":"2022-05-03T17:42:25.571115Z","shell.execute_reply.started":"2022-05-03T17:41:50.390337Z","shell.execute_reply":"2022-05-03T17:42:25.569259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A bit of useful statistics","metadata":{}},{"cell_type":"code","source":"transactions_per_week = transactions\\\n    .groupBy('week').count().orderBy('week', ascending=True)\\\n\ntransactions_per_week.show(10)\ntransactions_per_week.unpersist()\n\n# check transactions loaded\n# remove data from memory","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:42:25.575183Z","iopub.execute_input":"2022-05-03T17:42:25.575621Z","iopub.status.idle":"2022-05-03T17:42:54.501299Z","shell.execute_reply.started":"2022-05-03T17:42:25.575559Z","shell.execute_reply":"2022-05-03T17:42:54.500439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pyspark.sql.functions import countDistinct\n\nunique_customers = transactions\\\n    .select(countDistinct('customer_id'))\n\nprint(\"customer_id in current perimeter : \"+ str(unique_customers.collect()[0][0]))\nunique_customers.unpersist()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:42:54.503245Z","iopub.execute_input":"2022-05-03T17:42:54.503839Z","iopub.status.idle":"2022-05-03T17:43:19.792133Z","shell.execute_reply.started":"2022-05-03T17:42:54.503796Z","shell.execute_reply":"2022-05-03T17:43:19.791306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# number of orders for each customer each week\n# shift week to +1 to make it a feature for next week\n\ncustomers_orders_lw = transactions\\\n    .groupBy('customer_id', 'week').count().orderBy('count', ascending=False)\\\n    .withColumnRenamed('count', 'lw_orders_count')\\\n    .withColumn('week', F.col('week')+1)\n\ncustomers_orders_lw.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:43:19.793753Z","iopub.execute_input":"2022-05-03T17:43:19.794052Z","iopub.status.idle":"2022-05-03T17:43:44.919691Z","shell.execute_reply.started":"2022-05-03T17:43:19.794011Z","shell.execute_reply":"2022-05-03T17:43:44.918729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generate list of most sold items and top 12 rank for each week","metadata":{}},{"cell_type":"code","source":"#rank articles for each week\n\narticles_rank = transactions\\\n    .groupBy('article_id', 'week').count().orderBy('count', ascending=False)\\\n    .withColumnRenamed('count', 'articles_order_count')\n\nw_articles = Window.partitionBy(['week']).orderBy(articles_rank.articles_order_count.desc())\n\narticles_rank = articles_rank\\\n    .withColumn('rank', F.row_number().over(w_articles))\\\n    .filter(F.col('rank') <= 12)\\\n    .drop('articles_order_count')\\\n    .orderBy(['rank', 'week'])\n\narticles_top12 = articles_rank\\\n    .filter(F.col('rank') <= 12)\\\n    .drop('articles_order_count')\\\n    .orderBy(['rank', 'week'])\n\narticles_rank.show(20)\narticles_top12.show(25)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:43:44.924205Z","iopub.execute_input":"2022-05-03T17:43:44.924871Z","iopub.status.idle":"2022-05-03T17:44:34.548264Z","shell.execute_reply.started":"2022-05-03T17:43:44.924822Z","shell.execute_reply":"2022-05-03T17:44:34.547272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# after counting rows, I can drop duplicates\ntransactions = transactions\\\n    .dropDuplicates()\\\n\ntransactions.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:44:34.549717Z","iopub.execute_input":"2022-05-03T17:44:34.550038Z","iopub.status.idle":"2022-05-03T17:44:57.837991Z","shell.execute_reply.started":"2022-05-03T17:44:34.549994Z","shell.execute_reply":"2022-05-03T17:44:57.837016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Previously bought items","metadata":{}},{"cell_type":"code","source":"# the goal of this transfrom is to shift last bought basket to next week in which the customer bought something\n# this is done to create negative observations in \"next week\" (from a copy of last purchased basked)\n# because the customer could've skipped some weeks, I need to put a number to each week partition and shift it +1\n\n# add a reference number for customers who bought something in a certain week (somewhat of a transaction identifier)\nrn_transactions = transactions\\\n    .select('customer_id', 'week')\\\n    .dropDuplicates()\n\nw_transactions = Window.partitionBy('customer_id').orderBy(rn_transactions.week.asc())\n\n# enumerate rows by week partition\nrn_transactions = rn_transactions\\\n    .withColumn('week_rn', F.row_number().over(w_transactions))\\\n    .select('customer_id', 'week', 'week_rn')\\\n    .orderBy(['customer_id', 'week'], ascending = True)\n\nrn_transactions.show(10)\n\nrn_transactions0 = rn_transactions.drop('article_id')\\\n    .withColumnRenamed('week', 'new_week')\\\n    .dropDuplicates()\n\n# shift rn to next row\n# keep week info from rn_transactions0 and join on shifted week_rn\n\nlast_purchase = rn_transactions\\\n    .withColumn('week_rn', F.col('week_rn')+1)\\\n    .join(rn_transactions0, ['customer_id', 'week_rn'], 'inner')\\\n    .join(transactions, ['customer_id', 'week'], 'left')\\\n\nlast_purchase.show(10)\n\nlast_purchase = last_purchase\\\n    .drop('week')\\\n    .withColumnRenamed('new_week', 'week')\\\n    .orderBy(['customer_id', 'week'], ascending = True)\\\n    .select('customer_id', 'article_id', 'week')\n\nlast_purchase.show(20)\n\nlp_per_week = last_purchase\\\n    .groupBy('week').count().orderBy('week', ascending=True)\\\n\nlp_per_week.show(10)\nlp_per_week.unpersist()\nrn_transactions0.unpersist()","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-05-03T17:44:57.839222Z","iopub.execute_input":"2022-05-03T17:44:57.839558Z","iopub.status.idle":"2022-05-03T17:47:44.959766Z","shell.execute_reply.started":"2022-05-03T17:44:57.839472Z","shell.execute_reply":"2022-05-03T17:47:44.958833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fix the uncertain Y","metadata":{}},{"cell_type":"code","source":"transactions_pos = transactions\\\n    .select('customer_id', 'article_id', 'week')\\\n    .withColumn('y', F.lit(1))","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:47:44.963421Z","iopub.execute_input":"2022-05-03T17:47:44.964135Z","iopub.status.idle":"2022-05-03T17:47:45.024554Z","shell.execute_reply.started":"2022-05-03T17:47:44.964094Z","shell.execute_reply":"2022-05-03T17:47:45.023819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# keep only negative obs by excluding stuff that the customer bought in next week (true y)\nlast_purchase = last_purchase\\\n    .join(transactions_pos, ['customer_id', 'article_id', 'week'], 'left')\\\n    .fillna({'y': 0})\\\n    .filter(F.col('y').isin(0))\\\n    .select('customer_id', 'article_id', 'week', 'y')\n\nlast_purchase.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:47:45.025467Z","iopub.execute_input":"2022-05-03T17:47:45.02567Z","iopub.status.idle":"2022-05-03T17:48:53.752857Z","shell.execute_reply.started":"2022-05-03T17:47:45.025638Z","shell.execute_reply":"2022-05-03T17:48:53.751966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add top 12 items for each week","metadata":{}},{"cell_type":"code","source":"# create negative observations based on top 12 items for each week\n\narticles_top12_pw = articles_top12\\\n    .withColumn('week', F.col('week')+1)\n\nlistona = transactions\\\n    .select('customer_id', 'week')\\\n    .dropDuplicates()\\\n    .join(articles_top12_pw, ['week'], 'left')\\\n    .join(transactions_pos, ['customer_id', 'article_id', 'week'], 'left')\\\n    .fillna({'y': 0})\\\n    .filter(F.col('y').isin(0))\\\n    .select('customer_id', 'article_id', 'week', 'y')\\\n    .orderBy('customer_id', 'week')\\\n\nlistona.show(10)\narticles_top12_pw.unpersist()\ntransactions_pos.unpersist()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:48:53.754213Z","iopub.execute_input":"2022-05-03T17:48:53.754521Z","iopub.status.idle":"2022-05-03T17:50:07.734992Z","shell.execute_reply.started":"2022-05-03T17:48:53.75448Z","shell.execute_reply":"2022-05-03T17:50:07.733742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Put everything together and join features to create train df","metadata":{}},{"cell_type":"code","source":"# exclude from train the most remote week of observation since I don't generate a strategy for it\n# join all features\n\ntrain = transactions\\\n    .select('customer_id', 'article_id', 'week')\\\n    .withColumn('y', F.lit(1))\\\n    .unionByName(listona)\\\n    .unionByName(last_purchase)\\\n    .join(customers, 'customer_id', 'left')\\\n    .join(articles_rank, ['article_id', 'week'], 'left')\\\n    .join(articles, 'article_id', 'left')\\\n    .join(customers_orders_lw, ['customer_id', 'week'], 'left')\\\n    .orderBy(['week', 'customer_id'])\\\n    .filter(~F.col('week').isin(min_week))\\\n    .fillna({'rank': 999})\\\n    .fillna({'lw_orders_count': 0})\\\n    .orderBy(['week', 'customer_id'], ascending=True)\n\ntrain.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:50:07.736932Z","iopub.execute_input":"2022-05-03T17:50:07.737512Z","iopub.status.idle":"2022-05-03T17:54:01.488638Z","shell.execute_reply.started":"2022-05-03T17:50:07.737467Z","shell.execute_reply":"2022-05-03T17:54:01.486036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Replicate the same schema on Application table\n\nNote that by exploding this table the result will always be 1.3 Mil rows times the number of tests.","metadata":{}},{"cell_type":"code","source":"application = spark.read.option(\"header\",True) \\\n                .csv(\"../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:01.49039Z","iopub.execute_input":"2022-05-03T17:54:01.491803Z","iopub.status.idle":"2022-05-03T17:54:01.760323Z","shell.execute_reply.started":"2022-05-03T17:54:01.491745Z","shell.execute_reply":"2022-05-03T17:54:01.759388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Candidate observations for Application","metadata":{}},{"cell_type":"code","source":"# top 12 sold from last week\narticles_top12_app = articles_top12\\\n    .filter(F.col('week').isin(max_week))\\\n    .drop('week')\\\n    .withColumn('week', F.lit(application_week))\\\n    .dropDuplicates()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:01.76156Z","iopub.execute_input":"2022-05-03T17:54:01.76185Z","iopub.status.idle":"2022-05-03T17:54:01.90435Z","shell.execute_reply.started":"2022-05-03T17:54:01.761813Z","shell.execute_reply":"2022-05-03T17:54:01.903411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# top 12 items last week list\n\ntop_12 = articles_top12_app.toPandas()\ntop_12_lw = top_12['article_id'].tolist()\n\nprint(top_12_lw)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:01.905669Z","iopub.execute_input":"2022-05-03T17:54:01.905957Z","iopub.status.idle":"2022-05-03T17:54:28.561228Z","shell.execute_reply.started":"2022-05-03T17:54:01.905917Z","shell.execute_reply":"2022-05-03T17:54:28.560623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# articles rank feature\nlast_week_rank = articles_rank\\\n    .filter(F.col('week').isin(max_week))\\\n    .drop('week')\\\n    .withColumn('week', F.lit(application_week))\\\n    .dropDuplicates()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:28.562172Z","iopub.execute_input":"2022-05-03T17:54:28.562521Z","iopub.status.idle":"2022-05-03T17:54:28.627422Z","shell.execute_reply.started":"2022-05-03T17:54:28.562492Z","shell.execute_reply":"2022-05-03T17:54:28.626433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#last purchased basket\n\nw_lp = Window.partitionBy('customer_id').orderBy(transactions.week.desc())\n\nlast_purchased_app = transactions\\\n    .select('customer_id', 'week')\\\n    .dropDuplicates()\\\n    .withColumn('rn', F.row_number().over(w_lp))\\\n    .filter(F.col('rn').isin(1))\\\n    .drop('rn')\\\n    .join(transactions, ['customer_id', 'week'], 'left')","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:28.629021Z","iopub.execute_input":"2022-05-03T17:54:28.631764Z","iopub.status.idle":"2022-05-03T17:54:28.692454Z","shell.execute_reply.started":"2022-05-03T17:54:28.63171Z","shell.execute_reply":"2022-05-03T17:54:28.691462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"application = application\\\n    .select('customer_id')\\\n    .join(customers, 'customer_id', 'left')\\\n    .dropDuplicates()\\\n    .withColumn('week', F.lit(application_week))\\\n    .join(articles_top12_app, 'week', 'left')\\\n    .select('customer_id', 'article_id', 'week')\\\n    .unionByName(last_purchased_app)\\\n    .dropDuplicates()\\\n    .join(customers, 'customer_id', 'left')\\\n    .join(last_week_rank, 'article_id', 'left')\\\n    .join(articles, 'article_id', 'left')\\\n    .join(customers_orders_lw, ['customer_id', 'week'], 'left')\\\n    .fillna({'rank': 999})\\\n    .fillna({'lw_orders_count': 0})\\\n    .drop('week')\\\n    .dropDuplicates()\n\napplication.show(10)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:54:28.693901Z","iopub.execute_input":"2022-05-03T17:54:28.694196Z","iopub.status.idle":"2022-05-03T17:58:00.143479Z","shell.execute_reply.started":"2022-05-03T17:54:28.694158Z","shell.execute_reply":"2022-05-03T17:58:00.142488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clear memory as much as possible before writing csv","metadata":{}},{"cell_type":"code","source":"import gc\nlast_week_rank.unpersist()\narticles_top12_app.unpersist()\nlast_purchased_app.unpersist()\n\ntransactions.unpersist()\ncustomers.unpersist()\narticles.unpersist()\n\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:58:00.144845Z","iopub.execute_input":"2022-05-03T17:58:00.145115Z","iopub.status.idle":"2022-05-03T17:58:00.308901Z","shell.execute_reply.started":"2022-05-03T17:58:00.14508Z","shell.execute_reply":"2022-05-03T17:58:00.308112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save datasets from pyspark to csv","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport gc \n\ntrain.repartition(1).write.csv('/kaggle/working/train_df', header = 'true')\n\ntrain.unpersist()\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T17:58:00.310173Z","iopub.execute_input":"2022-05-03T17:58:00.310854Z","iopub.status.idle":"2022-05-03T18:02:45.93822Z","shell.execute_reply.started":"2022-05-03T17:58:00.310821Z","shell.execute_reply":"2022-05-03T18:02:45.937397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"save train save path location","metadata":{}},{"cell_type":"code","source":"print(os.listdir(\"../\"))\nprint(os.listdir(\"../working/train_df\"))","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:02:45.939666Z","iopub.execute_input":"2022-05-03T18:02:45.940309Z","iopub.status.idle":"2022-05-03T18:02:45.94761Z","shell.execute_reply.started":"2022-05-03T18:02:45.940263Z","shell.execute_reply":"2022-05-03T18:02:45.946457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_t = os.listdir(\"../working/train_df\")\ntrim_t = [x for x in path_t if x.startswith('part')]\nstringa_t = ''.join(trim_t)\ntrain_path = '../working/train_df/'+stringa_t\n\nprint(train_path)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:02:45.9495Z","iopub.execute_input":"2022-05-03T18:02:45.949984Z","iopub.status.idle":"2022-05-03T18:02:45.958266Z","shell.execute_reply.started":"2022-05-03T18:02:45.94994Z","shell.execute_reply":"2022-05-03T18:02:45.957611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save application\napplication.repartition(1).write.csv('/kaggle/working/application', header = 'true')\n\napplication.unpersist()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:02:45.959868Z","iopub.execute_input":"2022-05-03T18:02:45.9602Z","iopub.status.idle":"2022-05-03T18:07:18.918615Z","shell.execute_reply.started":"2022-05-03T18:02:45.960158Z","shell.execute_reply":"2022-05-03T18:07:18.917427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom lightgbm.sklearn import LGBMRanker\n\ntrain = pd.read_csv(train_path)\ntrain.sort_values(['week', 'customer_id'], inplace=True)\ntrain.reset_index(drop=True, inplace=True)\nprint('train:', train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:07:18.923803Z","iopub.execute_input":"2022-05-03T18:07:18.924066Z","iopub.status.idle":"2022-05-03T18:07:39.034258Z","shell.execute_reply.started":"2022-05-03T18:07:18.924031Z","shell.execute_reply":"2022-05-03T18:07:39.033125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import re\n#train = train.rename(columns = lambda x:re.sub('[^A-Za-z0-9_]+', '', x))\n# columns renamed because for some reason one hot encoding creates invalid characters\ntrain_cols = list(train.columns)\nprint(train_cols)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:07:39.035745Z","iopub.execute_input":"2022-05-03T18:07:39.036028Z","iopub.status.idle":"2022-05-03T18:07:39.042024Z","shell.execute_reply.started":"2022-05-03T18:07:39.035998Z","shell.execute_reply":"2022-05-03T18:07:39.041128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model requires 3 inputs: query id (quids), X, y","metadata":{}},{"cell_type":"code","source":"qids_train = train.groupby(['week', 'customer_id'])['article_id'].count().values\n\nX_train = train.drop([\"y\", 'customer_id', 'week'], axis=1)\ny_train = train[\"y\"]","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:07:39.043671Z","iopub.execute_input":"2022-05-03T18:07:39.044008Z","iopub.status.idle":"2022-05-03T18:07:41.268792Z","shell.execute_reply.started":"2022-05-03T18:07:39.043975Z","shell.execute_reply":"2022-05-03T18:07:41.267941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(qids_train)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:07:41.269921Z","iopub.execute_input":"2022-05-03T18:07:41.270291Z","iopub.status.idle":"2022-05-03T18:07:41.27545Z","shell.execute_reply.started":"2022-05-03T18:07:41.27026Z","shell.execute_reply":"2022-05-03T18:07:41.274553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Basic model","metadata":{}},{"cell_type":"code","source":"# n_estimators is recommended as default to 100 by documentation\n\nmodel = LGBMRanker(\n    objective=\"lambdarank\",\n    metric=\"ndcg\",\n    boosting_type=\"dart\",\n    n_estimators=100,\n    importance_type='gain',\n    verbose=10,\n    random_state = 17\n)\n\nmodel.fit(\n    X=X_train,\n    y=y_train,\n    group=qids_train,\n)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:07:41.277045Z","iopub.execute_input":"2022-05-03T18:07:41.277386Z","iopub.status.idle":"2022-05-03T18:10:54.885022Z","shell.execute_reply.started":"2022-05-03T18:07:41.277332Z","shell.execute_reply":"2022-05-03T18:10:54.884136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# feature importance\nx_train_cols = list(X_train.columns)\n\nfor i in model.feature_importances_.argsort()[::-1]:\n    print(x_train_cols[i], model.feature_importances_[i]/model.feature_importances_.sum())","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:10:54.886509Z","iopub.execute_input":"2022-05-03T18:10:54.887077Z","iopub.status.idle":"2022-05-03T18:10:54.89875Z","shell.execute_reply.started":"2022-05-03T18:10:54.887039Z","shell.execute_reply":"2022-05-03T18:10:54.897716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Application","metadata":{}},{"cell_type":"code","source":"print(os.listdir(\"../\"))\nprint(os.listdir(\"../working/application\"))","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:10:54.899944Z","iopub.execute_input":"2022-05-03T18:10:54.900631Z","iopub.status.idle":"2022-05-03T18:10:54.908712Z","shell.execute_reply.started":"2022-05-03T18:10:54.900599Z","shell.execute_reply":"2022-05-03T18:10:54.908088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before loading application I save its path to app_path variable, since pyspark saves the file name everytime with a different name in the kaggle/working/application folder which I specified earlier","metadata":{}},{"cell_type":"code","source":"path = os.listdir(\"../working/application\")\ntrim = [x for x in path if x.startswith('part')]\nstringa = ''.join(trim)\napp_path = '../working/application/'+stringa\n\nprint(app_path)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:10:54.909525Z","iopub.execute_input":"2022-05-03T18:10:54.909734Z","iopub.status.idle":"2022-05-03T18:10:54.922482Z","shell.execute_reply.started":"2022-05-03T18:10:54.909707Z","shell.execute_reply":"2022-05-03T18:10:54.921514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"application = pd.read_csv(app_path)\napplication.sort_values('customer_id', inplace=True)\napplication.reset_index(drop=True, inplace=True)\n\napplication_x = application.drop('customer_id', axis = 1)\nprint('application_x:', application_x.shape)\napp_cols = list(application_x.columns)\nprint(app_cols)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:10:54.923792Z","iopub.execute_input":"2022-05-03T18:10:54.924092Z","iopub.status.idle":"2022-05-03T18:11:53.702425Z","shell.execute_reply.started":"2022-05-03T18:10:54.924063Z","shell.execute_reply":"2022-05-03T18:11:53.70142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"application['prediction'] = model.predict(application_x)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:11:53.704161Z","iopub.execute_input":"2022-05-03T18:11:53.704771Z","iopub.status.idle":"2022-05-03T18:12:16.914708Z","shell.execute_reply.started":"2022-05-03T18:11:53.704716Z","shell.execute_reply":"2022-05-03T18:12:16.913998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_dict = application \\\n    .sort_values(['customer_id', 'prediction'], ascending=False) \\\n    .groupby('customer_id')['article_id'].apply(list).to_dict()","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:12:16.915739Z","iopub.execute_input":"2022-05-03T18:12:16.915946Z","iopub.status.idle":"2022-05-03T18:13:14.248558Z","shell.execute_reply.started":"2022-05-03T18:12:16.915915Z","shell.execute_reply":"2022-05-03T18:13:14.24477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:13:14.253674Z","iopub.execute_input":"2022-05-03T18:13:14.255472Z","iopub.status.idle":"2022-05-03T18:13:20.018585Z","shell.execute_reply.started":"2022-05-03T18:13:14.25539Z","shell.execute_reply":"2022-05-03T18:13:20.017724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = []\nfor c_id in sub.customer_id:\n    pred = pred_dict.get(c_id, [])\n    pred = pred + top_12_lw\n    preds.append(pred[:12])","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:13:20.019834Z","iopub.execute_input":"2022-05-03T18:13:20.020065Z","iopub.status.idle":"2022-05-03T18:13:25.373325Z","shell.execute_reply.started":"2022-05-03T18:13:20.020037Z","shell.execute_reply":"2022-05-03T18:13:25.372631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = [' '.join(['0' + str(p) for p in ps]) for ps in preds]\nsub.prediction = preds","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:13:25.37423Z","iopub.execute_input":"2022-05-03T18:13:25.374581Z","iopub.status.idle":"2022-05-03T18:13:31.241093Z","shell.execute_reply.started":"2022-05-03T18:13:25.374549Z","shell.execute_reply":"2022-05-03T18:13:31.239966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_name = 'submission'\nsub.to_csv(f'{sub_name}.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-03T18:13:31.242507Z","iopub.execute_input":"2022-05-03T18:13:31.242818Z","iopub.status.idle":"2022-05-03T18:13:37.848698Z","shell.execute_reply.started":"2022-05-03T18:13:31.242778Z","shell.execute_reply":"2022-05-03T18:13:37.84775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Empty for comments\n* keep same features on application once finished -> **done**\n* out of memory on application set -> **solved**\n* to be tested: customers that buy the same items each week / month ?","metadata":{}},{"cell_type":"markdown","source":"# Credits:\n* Ideas for preprocessing and ALS model https://www.kaggle.com/code/nadianizam/h-m-fashion-recommendation-with-pyspark\n* Ranker model: https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/307288\n* Submission functions: https://www.kaggle.com/code/marcogorelli/radek-s-lgbmranker-starter-pack/notebook\n* Huge thanks to: Radek, Paweł Jankiewicz, Hao, everyone who took time to comment","metadata":{}},{"cell_type":"markdown","source":"# Changelog","metadata":{}},{"cell_type":"markdown","source":"**Changelog**\n\nFirst release (0,005): \n* basic pipeline, added comments to the workbook \n\nVer 33 (0,0083): \n* Updated strategy. Now the workbook covers two weeks and looks at previously bought items in week -3. \n* I generate 17 negative observations based on top sold items,and 4 negative based on previously bought items from the customer in week -3.\n\nVer 47: \n* Added some week -4 observations to generate candidates. \n* Added combo items (testing): who bought this also bought that. \n* Fixed typo in model setup: qids_train = train_df.groupby(\"customer_id\")['article_id'].count().to_numpy() used to be *qids_train = train_df.groupby(\"customer_id\")[\"customer_id\"].count().to_numpy()*\n\nVer 48 (0,0098): \n* Reduced top sold items to a list of 12 instead of 17 based on the assumptions that this is my least customized strategy and acts as a filler\n\nVer 54 (0,011):\n* Removed combo items because the strategy is not solid enough, I only found a few 100s of purchased together\n* Basic fine tuning of model params, n_estimators = 100\n\nVer 55 (0,0106):\n* Major notebook rework. Now the model works on week-rolling style.\n* Candidates are generated in a more coherent way with respect of the given problem, since they are created by looking at previous week\n\nVer 68 (0,0167):\n* To avoid entropy I now keep only Rank feature up to 12 and fill na with 999\n* Reworked weeks to better match the periods of observation, now I consider 7 days rather than calendar week\n* Added to final prediction function a fill part to make sure I submit 12 elements of the list. I fill by appending the top 12 most sold items from last week","metadata":{}}]}