{"cells": [{"cell_type": "markdown", "id": "247666fb", "metadata": {}, "source": ["[NB 03](https://github.com/radekosmulski/personalized_fashion_recs/blob/messing_around/03_Basic_lgbm_with_idxs_restart.ipynb) didn't quite work.\n", "\n", "The two likely reasons for that are:\n", " - issues with how I am generating train data\n", " - false assumption that you can toss whatever at a ranking model and it will do the rest\n", "\n", "In this notebook, I want to put the groundwork needed for growing a good solution. First of all, that will require a robust and fast local validation scheme. We know that using the last week of train data for validation work and tracks the leaderboard nicely.\n", "\n", "Secondly, we want to start from a kernel of a solution that we can extend. This [notebook](https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2) on kaggle seems to me like a great starting point.\n", "\n", "The plan is to develop the functionality needed for a nice setup of a solution that we will reuse in NB 05. Along the way I hope to learn a bit more about the data, about some of the trends that I might want to model through the features I will engineer.\n", "\n", "The plan is:\n", "* implement a quick training pipeline leading to good validation\n", "* train a ranking model on candidates we know to be good\n", "* only generate new training data / candidates while validating whether we are moving in the right direction using local CV\n", "* start with building sensible features and see whether they move the needle on the score\n", "\n", "The truth is I do not know what will work. These RecSys models are a completely new breed of models to me. But I can set the problem up in a way as to help me learn. And that is what I am going to do :).\n", "\n", "Once I get this working I will breathe a sigh of relief and will jump into reading papers and drawing inspiration from there.\n", "\n", "Let's get started.\n", "\n", "NOTE: You are welcome to check out the earlier code that I wrote which can be found on [this branch](https://github.com/radekosmulski/personalized_fashion_recs/tree/messing_around). I learned a lot about RecSys models and this particular problem through it. But it has quite a few bugs and a couple of issues I now know off with regards to the approach. **You should not need any of the earlier code to run the notebooks in the main branch of this repo.**"]}, {"cell_type": "code", "execution_count": 2, "id": "2cd6646a", "metadata": {}, "outputs": [], "source": ["!wget https://raw.githubusercontent.com/benhamner/Metrics/master/Python/ml_metrics/average_precision.py"]}, {"cell_type": "code", "execution_count": 1, "id": "5d3ac989", "metadata": {}, "outputs": [], "source": ["# helper functions\n", "from sklearn.base import BaseEstimator, TransformerMixin\n", "import numpy as np\n", "from average_precision import apk\n", "\n", "# https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635\n", "def customer_hex_id_to_int(series):\n", "    return series.str[-16:].apply(hex_id_to_int)\n", "\n", "def hex_id_to_int(str):\n", "    return int(str[-16:], 16)\n", "\n", "def article_id_str_to_int(series):\n", "    return series.astype('int32')\n", "\n", "def article_id_int_to_str(series):\n", "    return '0' + series.astype('str')\n", "\n", "class Categorize(BaseEstimator, TransformerMixin):\n", "    def __init__(self, min_examples=0):\n", "        self.min_examples = min_examples\n", "        self.categories = []\n", "        \n", "    def fit(self, X):\n", "        for i in range(X.shape[1]):\n", "            vc = X.iloc[:, i].value_counts()\n", "            self.categories.append(vc[vc > self.min_examples].index.tolist())\n", "        return self\n", "\n", "    def transform(self, X):\n", "        data = {X.columns[i]: pd.Categorical(X.iloc[:, i], categories=self.categories[i]).codes for i in range(X.shape[1])}\n", "        return pd.DataFrame(data=data)"]}, {"cell_type": "markdown", "id": "14db50ed", "metadata": {}, "source": ["We want this to be fast. I can get as much RAM as I will ever need through VMs on GCP, but that is not the point. I want to see how far I can push my local hardware, but this goes even beyond that.\n", "\n", "I need the speed to make a good use of my time as I continue to build my understanding of what RecSys models are about. And the path to this leads through making the data I will work on smaller."]}, {"cell_type": "code", "execution_count": 3, "id": "94cd3410", "metadata": {}, "outputs": [], "source": ["%%time\n", "import pandas as pd\n", "\n", "transactions = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/transactions_train.csv', dtype={\"article_id\": \"str\"})\n", "customers = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/customers.csv')\n", "articles = pd.read_csv('/kaggle/input/h-and-m-personalized-fashion-recommendations/articles.csv', dtype={\"article_id\": \"str\"})"]}, {"cell_type": "code", "execution_count": 4, "id": "aa75de3a", "metadata": {}, "outputs": [], "source": ["transactions.memory_usage(deep=True)"]}, {"cell_type": "code", "execution_count": 5, "id": "bf0b35ed", "metadata": {}, "outputs": [], "source": ["transactions.info(memory_usage='deep')"]}, {"cell_type": "code", "execution_count": 6, "id": "4d85685a", "metadata": {}, "outputs": [], "source": ["%%time\n", "transactions['customer_id'].nunique()"]}, {"cell_type": "code", "execution_count": 7, "id": "b0ef4c75", "metadata": {}, "outputs": [], "source": ["%%time\n", "transactions['customer_id'] = customer_hex_id_to_int(transactions['customer_id'])\n", "transactions['customer_id'].nunique()"]}, {"cell_type": "code", "execution_count": 8, "id": "3733aeb2", "metadata": {}, "outputs": [], "source": ["transactions.memory_usage(deep=True)"]}, {"cell_type": "code", "execution_count": 9, "id": "fe293c79", "metadata": {}, "outputs": [], "source": ["transactions.info(memory_usage='deep')"]}, {"cell_type": "markdown", "id": "de972b0b", "metadata": {}, "source": ["Nice!\n", "\n", "Initially, I wanted to get rid of the `t_dat` column but on second thought I am not a fan.\n", "\n", "I am all for speed and reducing weight, but the main purpose of this activity is to increase developer productivity.\n", "\n", "If I fall back down to ints representing year, week, day I will be certainly trading developer productivity for fewer CPU cycles that are needed (and I want to go in the exact opposite direction! developer productivity > (nearly) anything else)"]}, {"cell_type": "code", "execution_count": 10, "id": "0d9a0cca", "metadata": {}, "outputs": [], "source": ["%%time\n", "\n", "transactions.t_dat = pd.to_datetime(transactions.t_dat, format='%Y-%m-%d')"]}, {"cell_type": "code", "execution_count": 11, "id": "2bfae010", "metadata": {}, "outputs": [], "source": ["transactions['week'] = 104 - (transactions.t_dat.max() - transactions.t_dat).dt.days // 7"]}, {"cell_type": "code", "execution_count": 12, "id": "90457b8f", "metadata": {}, "outputs": [], "source": ["transactions.info(memory_usage='deep')"]}, {"cell_type": "markdown", "id": "b094879d", "metadata": {}, "source": ["Let's do something about the `article_id` (both here and on `articles`) and let's take a closer look at `price`, `sales_channel_id` and `week`."]}, {"cell_type": "code", "execution_count": 13, "id": "7390ff52", "metadata": {}, "outputs": [], "source": ["transactions.article_id = article_id_str_to_int(transactions.article_id)\n", "articles.article_id = article_id_str_to_int(articles.article_id)\n", "\n", "transactions.week = transactions.week.astype('int8')\n", "transactions.sales_channel_id = transactions.sales_channel_id.astype('int8')\n", "transactions.price = transactions.price.astype('float32')"]}, {"cell_type": "code", "execution_count": 14, "id": "4c28d40b", "metadata": {}, "outputs": [], "source": ["transactions.info(memory_usage='deep')"]}, {"cell_type": "code", "execution_count": 15, "id": "10623914", "metadata": {}, "outputs": [], "source": ["transactions.drop(columns='t_dat').info(memory_usage='deep')"]}, {"cell_type": "markdown", "id": "f874dac4", "metadata": {}, "source": ["Well, this is interesting. There are very few unique `t_dat` values hence despite it being a scary `datetime64` it takes up very little memory!\n", "\n", "Keeping it for convenience is definitely the way to go.\n", "\n", "Let's take a brief look at the `customers` and `articles` dfs."]}, {"cell_type": "code", "execution_count": 16, "id": "bb06d877", "metadata": {}, "outputs": [], "source": ["customers.info(memory_usage='deep')"]}, {"cell_type": "code", "execution_count": 17, "id": "55f885f6", "metadata": {}, "outputs": [], "source": ["articles.info(memory_usage='deep')"]}, {"cell_type": "markdown", "id": "1caf9c76", "metadata": {}, "source": ["Well, this stuff will be getting merged with our transactions df at some point, so I guess we can also make this smaller and easier to work with down the road."]}, {"cell_type": "code", "execution_count": 18, "id": "efd82f6e", "metadata": {}, "outputs": [], "source": ["customers['club_member_status'].unique()"]}, {"cell_type": "code", "execution_count": 19, "id": "61bf9df5", "metadata": {}, "outputs": [], "source": ["customers.customer_id = customer_hex_id_to_int(customers.customer_id)\n", "for col in ['FN', 'Active', 'age']:\n", "    customers[col].fillna(-1, inplace=True)\n", "    customers[col] = customers[col].astype('int8')"]}, {"cell_type": "code", "execution_count": 20, "id": "758411dd", "metadata": {}, "outputs": [], "source": ["customers.club_member_status = Categorize().fit_transform(customers[['club_member_status']]).club_member_status\n", "customers.postal_code = Categorize().fit_transform(customers[['postal_code']]).postal_code\n", "customers.fashion_news_frequency = Categorize().fit_transform(customers[['fashion_news_frequency']]).fashion_news_frequency"]}, {"cell_type": "code", "execution_count": 21, "id": "7761a45d", "metadata": {}, "outputs": [], "source": ["customers.info(memory_usage='deep')"]}, {"cell_type": "code", "execution_count": 22, "id": "6cb4fc65", "metadata": {}, "outputs": [], "source": ["for col in articles.columns:\n", "    if articles[col].dtype == 'object':\n", "        articles[col] = Categorize().fit_transform(articles[[col]])[col]"]}, {"cell_type": "code", "execution_count": 23, "id": "4a8e33bd", "metadata": {}, "outputs": [], "source": ["articles.info(memory_usage='deep')"]}, {"cell_type": "code", "execution_count": 24, "id": "b3596527", "metadata": {}, "outputs": [], "source": ["for col in articles.columns:\n", "    if articles[col].dtype == 'int64':\n", "        articles[col] = articles[col].astype('int32')"]}, {"cell_type": "markdown", "id": "dc411fb5", "metadata": {}, "source": ["And this concludes our raw data preparation step! Let's now write everything back to disk."]}, {"cell_type": "code", "execution_count": 25, "id": "86f4e1b3", "metadata": {}, "outputs": [], "source": ["transactions.sort_values(['t_dat', 'customer_id'], inplace=True)"]}, {"cell_type": "code", "execution_count": 26, "id": "682b1125", "metadata": {}, "outputs": [], "source": ["%%time\n", "\n", "transactions.to_parquet('transactions_train.parquet')\n", "customers.to_parquet('customers.parquet')\n", "articles.to_parquet('articles.parquet')"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7"}}, "nbformat": 4, "nbformat_minor": 5}