{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M - Implicit ALS model\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/31254/logos/header.png)\n\n## Implicit ALS base model for the competition [H&M Personalized Fashion Recommendations](https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations).\n\n\n[Implicit](https://github.com/benfred/implicit/) is a library for recommender models. In theory, it supports GPU out-of-the-box, but I haven't tried it yet.\n\nIn this notebook we use ALS (Alternating Least Squares), but the library supports a lot of other models with not many changes.\n\nALS is one of the most used ML models for recommender systems. It's a matrix factorization method based on SVD (it's actually an approximated, numerical version of SVD). Basically, ALS factorizes the interaction matrix (user x items) into two smaller matrices, one for item embeddings and one for user embeddings. These new matrices are built in a manner such that the multiplication of a user and an item gives (approximately) it's interaction score. This build embeddings for items and for users that live in the same vector space, allowing the implementation of recommendations as simple cosine distances between users and items. This is, the 12 items we recommend for a given user are the 12 items with their embedding vectors closer to the user embedding vector.\n\nThere are a lot of online resources explaining it. For example, [here](https://towardsdatascience.com/prototyping-a-recommender-system-step-by-step-part-2-alternating-least-square-als-matrix-4a76c58714a1).\n\n\nBe aware that there was a breaking API change in a recent release of implicit (11 days ago): https://github.com/benfred/implicit/releases/tag/v0.5.0 so some thing in the documentation are off if you use the version that comes installed in the Kaggle environments. Anyway, this competition doesn't forbid Internet usage, so upgrading the package to its latest version fixes all.\n\n\n---\n\n**I have reverted the kernet to version 14 with a score of `0.014` for now. The scores above `0.014` are using the `0.02` model by Heng Zheng as the fallback strategy for cold-start users. Therefore, a `0.018` score is actually bad. I did those version with the hope that they would work nicely together, obtaining `>0.02`. Since it was not the case, reporting a `0.018` score is not accurate at all.**\n\nIf I can obtain a score that surpasses the `0.02`, using ALS + Heng's baseline, I will roll back to those versions again.\n\n\n# Please, _DO_ upvote if you find this kernel useful or interesting!","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"# FYI:\n# This pip command takes a lot with GPU enabled (~15 min)\n# It works though. And GPU accelerates the process *a lot*.\n# I am developing with GPU turned off and submitting with GPU turned on\n!pip install --upgrade implicit","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:03:41.915352Z","iopub.execute_input":"2022-02-08T23:03:41.915739Z","iopub.status.idle":"2022-02-08T23:06:12.366321Z","shell.execute_reply.started":"2022-02-08T23:03:41.915645Z","shell.execute_reply":"2022-02-08T23:06:12.364617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os; os.environ['OPENBLAS_NUM_THREADS']='1'\nimport numpy as np\nimport pandas as pd\nimport implicit\nfrom scipy.sparse import coo_matrix\nfrom implicit.evaluation import mean_average_precision_at_k","metadata":{"execution":{"iopub.status.busy":"2022-02-09T14:22:41.426785Z","iopub.execute_input":"2022-02-09T14:22:41.427496Z","iopub.status.idle":"2022-02-09T14:22:41.769505Z","shell.execute_reply.started":"2022-02-09T14:22:41.427391Z","shell.execute_reply":"2022-02-09T14:22:41.768679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load dataframes","metadata":{}},{"cell_type":"code","source":"%%time\n\nbase_path = '../input/h-and-m-personalized-fashion-recommendations/'\ncsv_train = f'{base_path}transactions_train.csv'\ncsv_sub = f'{base_path}sample_submission.csv'\ncsv_users = f'{base_path}customers.csv'\ncsv_items = f'{base_path}articles.csv'\n\ndf = pd.read_csv(csv_train, dtype={'article_id': str}, parse_dates=['t_dat'])\ndf_sub = pd.read_csv(csv_sub)\ndfu = pd.read_csv(csv_users)\ndfi = pd.read_csv(csv_items, dtype={'article_id': str})","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-09T14:22:41.771261Z","iopub.execute_input":"2022-02-09T14:22:41.771693Z","iopub.status.idle":"2022-02-09T14:23:58.179973Z","shell.execute_reply.started":"2022-02-09T14:22:41.771655Z","shell.execute_reply":"2022-02-09T14:23:58.179057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Trying with less data:\n# https://www.kaggle.com/tomooinubushi/folk-of-time-is-our-best-friend/notebook\ndf = df[df['t_dat'] > '2020-08-21']\ndf.shape","metadata":{"execution":{"iopub.status.busy":"2022-02-09T14:24:33.178743Z","iopub.execute_input":"2022-02-09T14:24:33.179007Z","iopub.status.idle":"2022-02-09T14:24:33.226669Z","shell.execute_reply.started":"2022-02-09T14:24:33.178977Z","shell.execute_reply":"2022-02-09T14:24:33.225950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For validation this means 3 weeks of training and 1 week for validation\n# For submission, it means 4 weeks of training\ndf['t_dat'].max()","metadata":{"execution":{"iopub.status.busy":"2022-02-09T14:25:03.709766Z","iopub.execute_input":"2022-02-09T14:25:03.710329Z","iopub.status.idle":"2022-02-09T14:25:03.720059Z","shell.execute_reply.started":"2022-02-09T14:25:03.710286Z","shell.execute_reply":"2022-02-09T14:25:03.719059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assign autoincrementing ids starting from 0 to both users and items","metadata":{}},{"cell_type":"code","source":"ALL_USERS = dfu['customer_id'].unique().tolist()\nALL_ITEMS = dfi['article_id'].unique().tolist()\n\nuser_ids = dict(list(enumerate(ALL_USERS)))\nitem_ids = dict(list(enumerate(ALL_ITEMS)))\n\nuser_map = {u: uidx for uidx, u in user_ids.items()}\nitem_map = {i: iidx for iidx, i in item_ids.items()}\n\ndf['user_id'] = df['customer_id'].map(user_map)\ndf['item_id'] = df['article_id'].map(item_map)\n\ndel dfu, dfi","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:07:41.918576Z","iopub.execute_input":"2022-02-08T23:07:41.919179Z","iopub.status.idle":"2022-02-08T23:07:59.273467Z","shell.execute_reply.started":"2022-02-08T23:07:41.919131Z","shell.execute_reply":"2022-02-08T23:07:59.272662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create coo_matrix (user x item) and csr matrix (user x item)\n\nIt is common to use scipy sparse matrices in recommender systems, because the main core of the problem is typically modeled as a matrix with users and items, with the values representing whether the user purchased (or liked) an items. Since each user purchases only a small fraction of the catalog of products, this matrix is full of zero (aka: it's sparse).\n\nIn a very recent release they did an API breaking change, so be aware of that: https://github.com/benfred/implicit/releases\nIn this notebook we are using the latest version, so everything is aligned with (user x item)\n\n**We are using (user x item) matrices, both for training and for evaluating/recommender.**\n\nIn the previous versions the training procedure required a COO item x user\n\nFor evaluation and prediction, on the other hand, CSR matrices with users x items format should be provided.\n\n\n### About COO matrices\nCOO matrices are a kind of sparse matrix.\nThey store their values as tuples of `(row, column, value)` (the coordinates)\n\nYou can read more about them here: \n* https://en.wikipedia.org/wiki/Sparse_matrix#Coordinate_list_(COO)\n* https://scipy-lectures.org/advanced/scipy_sparse/coo_matrix.html\n\nFrom https://het.as.utexas.edu/HET/Software/Scipy/generated/scipy.sparse.coo_matrix.html\n\n```python\n>>> row  = np.array([0,3,1,0]) # user_ids\n>>> col  = np.array([0,3,1,2]) # item_ids\n>>> data = np.array([4,5,7,9]) # a bunch of ones of lenght unique(user) x unique(items)\n>>> coo_matrix((data,(row,col)), shape=(4,4)).todense()\nmatrix([[4, 0, 9, 0],\n        [0, 7, 0, 0],\n        [0, 0, 0, 0],\n        [0, 0, 0, 5]])\n```\n\n## About CSR matrices\n* https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)\n","metadata":{}},{"cell_type":"code","source":"row = df['user_id'].values\ncol = df['item_id'].values\ndata = np.ones(df.shape[0])\ncoo_train = coo_matrix((data, (row, col)), shape=(len(ALL_USERS), len(ALL_ITEMS)))\ncoo_train","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:07:59.274475Z","iopub.execute_input":"2022-02-08T23:07:59.275048Z","iopub.status.idle":"2022-02-08T23:07:59.568032Z","shell.execute_reply.started":"2022-02-08T23:07:59.275017Z","shell.execute_reply":"2022-02-08T23:07:59.567441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check that model works ok with data","metadata":{}},{"cell_type":"code","source":"%%time\nmodel = implicit.als.AlternatingLeastSquares(factors=10, iterations=2)\nmodel.fit(coo_train)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:07:59.569085Z","iopub.execute_input":"2022-02-08T23:07:59.56941Z","iopub.status.idle":"2022-02-08T23:08:18.796116Z","shell.execute_reply.started":"2022-02-08T23:07:59.569383Z","shell.execute_reply":"2022-02-08T23:08:18.795433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation","metadata":{}},{"cell_type":"markdown","source":"## Functions required for validation","metadata":{}},{"cell_type":"code","source":"def to_user_item_coo(df):\n    \"\"\" Turn a dataframe with transactions into a COO sparse items x users matrix\"\"\"\n    row = df['user_id'].values\n    col = df['item_id'].values\n    data = np.ones(df.shape[0])\n    coo = coo_matrix((data, (row, col)), shape=(len(ALL_USERS), len(ALL_ITEMS)))\n    return coo\n\n\ndef split_data(df, validation_days=7):\n    \"\"\" Split a pandas dataframe into training and validation data, using <<validation_days>>\n    \"\"\"\n    validation_cut = df['t_dat'].max() - pd.Timedelta(validation_days)\n\n    df_train = df[df['t_dat'] < validation_cut]\n    df_val = df[df['t_dat'] >= validation_cut]\n    return df_train, df_val\n\ndef get_val_matrices(df, validation_days=7):\n    \"\"\" Split into training and validation and create various matrices\n        \n        Returns a dictionary with the following keys:\n            coo_train: training data in COO sparse format and as (users x items)\n            csr_train: training data in CSR sparse format and as (users x items)\n            csr_val:  validation data in CSR sparse format and as (users x items)\n    \n    \"\"\"\n    df_train, df_val = split_data(df, validation_days=validation_days)\n    coo_train = to_user_item_coo(df_train)\n    coo_val = to_user_item_coo(df_val)\n\n    csr_train = coo_train.tocsr()\n    csr_val = coo_val.tocsr()\n    \n    return {'coo_train': coo_train,\n            'csr_train': csr_train,\n            'csr_val': csr_val\n          }\n\n\ndef validate(matrices, factors=200, iterations=20, regularization=0.01, show_progress=True):\n    \"\"\" Train an ALS model with <<factors>> (embeddings dimension) \n    for <<iterations>> over matrices and validate with MAP@12\n    \"\"\"\n    coo_train, csr_train, csr_val = matrices['coo_train'], matrices['csr_train'], matrices['csr_val']\n    \n    model = implicit.als.AlternatingLeastSquares(factors=factors, \n                                                 iterations=iterations, \n                                                 regularization=regularization, \n                                                 random_state=42)\n    model.fit(coo_train, show_progress=show_progress)\n    \n    # The MAPK by implicit doesn't allow to calculate allowing repeated items, which is the case.\n    # TODO: change MAP@12 to a library that allows repeated items in prediction\n    map12 = mean_average_precision_at_k(model, csr_train, csr_val, K=12, show_progress=show_progress, num_threads=4)\n    print(f\"Factors: {factors:>3} - Iterations: {iterations:>2} - Regularization: {regularization:4.3f} ==> MAP@12: {map12:6.5f}\")\n    return map12","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:11:16.951529Z","iopub.execute_input":"2022-02-08T23:11:16.952282Z","iopub.status.idle":"2022-02-08T23:11:16.966032Z","shell.execute_reply.started":"2022-02-08T23:11:16.952239Z","shell.execute_reply":"2022-02-08T23:11:16.965106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matrices = get_val_matrices(df)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:08:18.813222Z","iopub.execute_input":"2022-02-08T23:08:18.813482Z","iopub.status.idle":"2022-02-08T23:08:25.146694Z","shell.execute_reply.started":"2022-02-08T23:08:18.81345Z","shell.execute_reply":"2022-02-08T23:08:25.145781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nbest_map12 = 0\nfor factors in [40, 50, 60, 100, 200, 500, 1000]:\n    for iterations in [3, 12, 14, 15, 20]:\n        for regularization in [0.01]:\n            map12 = validate(matrices, factors, iterations, regularization, show_progress=False)\n            if map12 > best_map12:\n                best_map12 = map12\n                best_params = {'factors': factors, 'iterations': iterations, 'regularization': regularization}\n                print(f\"Best MAP@12 found. Updating: {best_params}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del matrices","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training over the full dataset","metadata":{}},{"cell_type":"code","source":"coo_train = to_user_item_coo(df)\ncsr_train = coo_train.tocsr()","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:13:56.676132Z","iopub.execute_input":"2022-02-08T23:13:56.676914Z","iopub.status.idle":"2022-02-08T23:14:00.48247Z","shell.execute_reply.started":"2022-02-08T23:13:56.676873Z","shell.execute_reply":"2022-02-08T23:14:00.481464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train(coo_train, factors=200, iterations=15, regularization=0.01, show_progress=True):\n    model = implicit.als.AlternatingLeastSquares(factors=factors, \n                                                 iterations=iterations, \n                                                 regularization=regularization, \n                                                 random_state=42)\n    model.fit(coo_train, show_progress=show_progress)\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:14:00.484275Z","iopub.execute_input":"2022-02-08T23:14:00.484602Z","iopub.status.idle":"2022-02-08T23:14:00.490861Z","shell.execute_reply.started":"2022-02-08T23:14:00.484562Z","shell.execute_reply":"2022-02-08T23:14:00.48987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_params","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:14:14.063305Z","iopub.execute_input":"2022-02-08T23:14:14.063667Z","iopub.status.idle":"2022-02-08T23:14:14.068211Z","shell.execute_reply.started":"2022-02-08T23:14:14.063633Z","shell.execute_reply":"2022-02-08T23:14:14.067212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = train(coo_train, **best_params)","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:14:17.223055Z","iopub.execute_input":"2022-02-08T23:14:17.223849Z","iopub.status.idle":"2022-02-08T23:14:38.799466Z","shell.execute_reply.started":"2022-02-08T23:14:17.223808Z","shell.execute_reply":"2022-02-08T23:14:38.798424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"markdown","source":"## Submission function","metadata":{}},{"cell_type":"code","source":"def submit(model, csr_train, submission_name=\"submissions.csv\"):\n    preds = []\n    batch_size = 2000\n    to_generate = np.arange(len(ALL_USERS))\n    for startidx in range(0, len(to_generate), batch_size):\n        batch = to_generate[startidx : startidx + batch_size]\n        ids, scores = model.recommend(batch, csr_train[batch], N=12, filter_already_liked_items=False)\n        for i, userid in enumerate(batch):\n            customer_id = user_ids[userid]\n            user_items = ids[i]\n            article_ids = [item_ids[item_id] for item_id in user_items]\n            preds.append((customer_id, ' '.join(article_ids)))\n\n    df_preds = pd.DataFrame(preds, columns=['customer_id', 'prediction'])\n    df_preds.to_csv(submission_name, index=False)\n    \n    display(df_preds.head())\n    print(df_preds.shape)\n    \n    return df_preds","metadata":{"execution":{"iopub.status.busy":"2022-02-08T23:15:20.277717Z","iopub.execute_input":"2022-02-08T23:15:20.278095Z","iopub.status.idle":"2022-02-08T23:15:20.290182Z","shell.execute_reply.started":"2022-02-08T23:15:20.278055Z","shell.execute_reply":"2022-02-08T23:15:20.289215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf_preds = submit(model, csr_train);","metadata":{},"execution_count":null,"outputs":[]}]}