{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Abstract","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we aim to solve the H&M Recommendations Challenge. We approach this problem by building an Item-based collaborative sytem and combining it with a time decaying popularity bechmark model as a fallback when we do not have any purchase history for a user.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom tqdm import tqdm","metadata":{"papermill":{"duration":0.025331,"end_time":"2022-03-02T12:42:44.603122","exception":false,"start_time":"2022-03-02T12:42:44.577791","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Import data","metadata":{}},{"cell_type":"markdown","source":"First, we will import the data. We will just be using the transactions data for training, and the submission sample to compute our predictions.","metadata":{}},{"cell_type":"code","source":"# Load data from different CSV files\ntransactions_df = pd.read_csv(\"./data/transactions_train.csv\")\nsubmission_df = pd.read_csv('./data/sample_submission.csv')","metadata":{"papermill":{"duration":65.954103,"end_time":"2022-03-02T12:43:50.570303","exception":false,"start_time":"2022-03-02T12:42:44.616200","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train dataset","metadata":{}},{"cell_type":"markdown","source":"We use this function to filter out articles that have not been bought frequently since they provide little to no value to our model.","metadata":{}},{"cell_type":"code","source":"def get_most_bought_articles(data, num_articles=5):\n    # Create dataframe that contains the number of times each article has been bought\n    articles_counts = data[['article_id', 't_dat']].groupby('article_id').count().reset_index().rename(columns={'t_dat': 'count'})\n    articles_counts = articles_counts.sort_values(by='count', ascending=False)\n        \n    most_bought_articles = articles_counts.loc[articles_counts['count'] >= num_articles]['article_id'].values\n    \n    return most_bought_articles","metadata":{"papermill":{"duration":0.020148,"end_time":"2022-03-02T12:43:50.602322","exception":false,"start_time":"2022-03-02T12:43:50.582174","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this part, we will filter the transactions by date, since we only want to use recent items for our prediction. We have fixed the range to around 2,5 months.","metadata":{}},{"cell_type":"code","source":"# Create training dataset with positive examples.\n# The training data will contain all transactions starting from 08/07/2020.\n# Only items that have been bought at least 10 times will be kept. Also, we\n# are only going to compute the information for the customers that appear\n# in these transactions.\nstart_date = pd.to_datetime('2020-07-08')\nend_date = pd.to_datetime('2020-09-23')\n\nfiltered_transactions_df = transactions_df.copy()\nfiltered_transactions_df.t_dat = pd.to_datetime(filtered_transactions_df.t_dat)\nfiltered_transactions_df = filtered_transactions_df.loc[filtered_transactions_df.t_dat >= start_date]\nfiltered_transactions_df = filtered_transactions_df.loc[filtered_transactions_df.t_dat < end_date]\n\ntrain_df = filtered_transactions_df.copy()\n\nmost_bought_articles = get_most_bought_articles(train_df, num_articles=10)\nmost_bought_articles = np.sort(most_bought_articles)\n\ntrain_df = train_df.drop(train_df.loc[~train_df.article_id.isin(most_bought_articles)].index)\nfiltered_transactions_df = filtered_transactions_df.drop(filtered_transactions_df.loc[~filtered_transactions_df.article_id.isin(most_bought_articles)].index)\n\nrecent_customers = train_df.loc[train_df.article_id.isin(most_bought_articles)].customer_id.unique()\nrecent_customers = np.sort(recent_customers)\n\nnum_articles = len(most_bought_articles)\nnum_customers = len(recent_customers)\n\n# Create dictionaries with mapping keys\narticles_id_to_idx = dict(zip(most_bought_articles, range(num_articles)))\ncustomers_id_to_idx = dict(zip(recent_customers, range(num_customers)))\n\ntrain_df = train_df.loc[train_df['article_id'].isin(most_bought_articles)]\ntrain_df = train_df[['customer_id', 'article_id']]\n\ntrain_df['article_id'] = train_df['article_id'].apply(lambda x: articles_id_to_idx[x])\ntrain_df['customer_id'] = train_df['customer_id'].apply(lambda x: customers_id_to_idx[x])\ntrain_df['bought'] = np.ones(train_df.shape[0])\n\ntrain_df","metadata":{"papermill":{"duration":17.257146,"end_time":"2022-03-02T12:44:07.872189","exception":false,"start_time":"2022-03-02T12:43:50.615043","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we simply generate negative examples that we add to our exisiting dataset.","metadata":{}},{"cell_type":"code","source":"# Generate negative examples\nnp.random.seed(47)\n\nnum_transactions = train_df.shape[0]\nnegative_data = pd.DataFrame(\n    {\n        'article_id': np.random.choice(num_articles, num_transactions),\n        'customer_id': np.random.choice(num_customers, num_transactions),\n        'bought': np.zeros(num_transactions)\n    }\n)\n\ntrain_df = pd.concat([train_df, negative_data])\ntrain_df = train_df.sample(frac=1).reset_index(drop=True)\n\ntrain_df","metadata":{"papermill":{"duration":1.293051,"end_time":"2022-03-02T12:44:09.177966","exception":false,"start_time":"2022-03-02T12:44:07.884915","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Validation dataset","metadata":{}},{"cell_type":"markdown","source":"In this part, we add the most recent data that has not been used for training to our validation dataset. Since this is our final notebook that is being trained with all of the most recent data, we do not have any validation data.","metadata":{}},{"cell_type":"code","source":"import datetime\nval = transactions_df.copy()\nval.t_dat = pd.to_datetime(val.t_dat)\nval = val.loc[val[\"t_dat\"] >= end_date]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"positive_items_val = val.groupby(['customer_id'])['article_id'].apply(list)\n# creating validation set for metrics use case\nval_users = positive_items_val.keys()\nval_items = []\n\nfor i,user in tqdm(enumerate(val_users)):\n    val_items.append(positive_items_val[user])\n    \nprint(\"Total users in validation:\", len(val_users))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Time decaying baseline model for default recommendations","metadata":{}},{"cell_type":"markdown","source":"As mentioned above, we will use time decaying popularity benchmark model as a fallback for users that we do not have any trasnactions data on. The model is based on this: https://www.kaggle.com/mayukh18/time-decaying-popularity-benchmark-0-0216.\n\nIt is using the last 4 weeks of transactions data and computes the popularity based on the date.","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(\"./data/transactions_train.csv\", dtype={'article_id':str})","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[\"t_dat\"] = pd.to_datetime(data[\"t_dat\"])\ntrain1 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,16)) & (data['t_dat'] < datetime.datetime(2020,9,23))]\ntrain2 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,8)) & (data['t_dat'] < datetime.datetime(2020,9,16))]\ntrain3 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,31)) & (data['t_dat'] < datetime.datetime(2020,9,8))]\ntrain4 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,23)) & (data['t_dat'] < datetime.datetime(2020,8,31))]\n\n# List of all purchases per user (has repetitions)\npositive_items_per_user1 = train1.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user2 = train2.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user3 = train3.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user4 = train4.groupby(['customer_id'])['article_id'].apply(list)\n\ntrain = pd.concat([train1, train2], axis=0)\ntrain['pop_factor'] = train['t_dat'].apply(lambda x: 1/(datetime.datetime(2020,9,23) - x).days)\npopular_items_group = train.groupby(['article_id'])['pop_factor'].sum()\n\n_, popular_items = zip(*sorted(zip(popular_items_group, popular_items_group.keys()))[::-1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from collections import Counter\n\npopular_items = list(popular_items)\n\ndef get_popularity_based_prediction(user):\n    user_output = []\n    if user in positive_items_per_user1.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user2.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user3.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user4.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    \n    user_output += list(popular_items[:12 - len(user_output)])\n    return user_output","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our model","metadata":{}},{"cell_type":"markdown","source":"Our main model is an Item-Based Collaborative Filtering (IBCF) model.\n\nWe have stored the transactions into a dataframe that contains the index of the user that has bought an item, the index of the item that has been bought, and a label $1$ or $0$ that tells whether the item has been bought or not, respectively. We decompose this pseudo-matrix $T$ into the product of two matrices $P$, which contains the information of the users, and $Q$, which contains information of the articles. Both of these matrices are in a latent space of dimension $d$.\n\nWe use SGD in order to compute the matrices, minimizing the squared error of the actual label and the dot product of the user $p_i$ and the item $q_j$, which is an estimation of whether the user is going to buy the product or not.\n\nSince the prediction of similar items is only based on other single items, we have implemented an approach which allows us to base the final prediction on more than one product. For that, we compute a relevance score for the last few bought items by a user which is based on the date when it was bought: $1 / (end_date - date bought)$ which is normalized with the softmax function. We multiply the score with each similarity prediction vector and sum them up. After that we take the 12 items with the highest score. \nIn case we do not have any information about the customer's purchase history, we use the fallback model to get the 12 most popular items.","metadata":{}},{"cell_type":"code","source":"import scipy\nfrom scipy.special import softmax\n\ndef apk(actual, predicted, k=12):\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=12):\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\n    \n    \n# Recommender\nclass ItemBasedRecommender:\n    \"\"\" Collaborative filtering using a custom sim(u,u'). \"\"\"\n    def __init__(self, df, num_articles, num_customers, num_components=10, num_last_items=5):\n        \"\"\" Constructor \"\"\"\n        self.num_components = num_components\n        self.num_articles = num_articles\n        self.num_customers = num_customers\n        self.num_last_items = num_last_items\n        \n        self.train = df\n        \n        self.articles = self.train.article_id.values\n        self.customers = self.train.customer_id.values\n        self.bought = self.train.bought.values\n\n\n    def __sdg__(self):\n        for idx in tqdm(self.training_indices):\n            customer_idx = self.customers[idx]\n            article_idx = self.articles[idx]\n            real_bought = self.bought[idx]\n            \n            prediction = self.__predict_train(customer_idx, article_idx)\n            error = (real_bought - prediction) # error\n            \n            #Update latent factors\n            self.customers_lat_mat[customer_idx] += self.learning_rate * \\\n                                    (error * self.articles_lat_mat[article_idx] - \\\n                                     self.lmbda * self.customers_lat_mat[customer_idx])\n            self.articles_lat_mat[article_idx] += self.learning_rate * \\\n                                    (error * self.customers_lat_mat[customer_idx] - \\\n                                     self.lmbda * self.articles_lat_mat[article_idx])\n                \n                \n    def fit(self, n_epochs=10, learning_rate=0.001, lmbda=0.1):\n        \"\"\"Compute the matrix factorization R = P \\times Q\"\"\"\n        self.learning_rate = learning_rate\n        self.lmbda = lmbda\n        self.n_samples = self.train.shape[0]\n        \n        self.train_rmse =[]\n        self.test_rmse = []\n        \n        # Initialize latent matrices\n        self.customers_lat_mat = np.random.normal(scale=1., size=(self.num_customers, self.num_components))\n        self.articles_lat_mat = np.random.normal(scale=1., size=(self.num_articles, self.num_components))\n\n        for epoch in range(n_epochs):\n            print('Epoch: {}'.format(epoch))\n            \n            self.training_indices = np.random.permutation(self.n_samples)\n            self.__sdg__()\n            # self.evaluate(num_samples=10000)\n        \n        del self.customers_lat_mat\n            \n        \n    def __predict_train(self, customer_idx, article_idx):\n        \"\"\" Single user and item prediction.\"\"\"\n        prediction = np.dot(self.customers_lat_mat[customer_idx], self.articles_lat_mat[article_idx])\n        prediction = np.clip(prediction, 0, 1)\n        \n        return prediction\n    \n    def predict(self, transactions_df, customers, most_bought_articles, articles_id_to_idx, customers_id_to_idx):\n        recommendations = []\n\n        # Compute similarity matrix\n        similarity_matrix = np.dot(self.articles_lat_mat, self.articles_lat_mat.T)\n        norms = np.sqrt(np.sum(self.articles_lat_mat ** 2, axis=1)).reshape(-1, 1)\n        similarity_matrix = similarity_matrix / norms\n        similarity_matrix = similarity_matrix / norms.T\n\n        last_transactions = filtered_transactions_df.groupby('customer_id').tail(5).sort_values(by=['customer_id', 't_dat'])\n        last_transactions['relevance'] = np.array([1 / (end_date - day).days for day in last_transactions.t_dat.values])\n        last_transactions['relevance'] = last_transactions.groupby('customer_id').relevance.transform(softmax).values\n        last_transactions['article_idx'] = last_transactions.article_id.apply(lambda x: articles_id_to_idx[x])\n\n\n        # Create dictionary of transactions\n        transactions_dict = last_transactions.groupby('customer_id').apply(lambda x: [(article_idx, relevance) for article_idx, relevance in zip(x.article_idx, x.relevance)])\n        transactions_dict = {user: trans for user, trans in zip(transactions_dict.index, transactions_dict.values)}\n\n        for customer in tqdm(customers):\n            try:\n                customer_idx = customers_id_to_idx[customer]\n                customer_transactions = transactions_dict[customer]\n                similarity_cum = np.zeros(similarity_matrix.shape[1])\n\n                for article_idx, relevance in customer_transactions:\n                    similarity_cum += relevance * similarity_matrix[article_idx]\n\n                similar_articles_idx = np.argsort(similarity_cum)[::-1][:12]\n                recommended_articles = most_bought_articles[similar_articles_idx]\n\n                recommendation = \"0\" + \" 0\".join([str(article) for article in recommended_articles])\n            except KeyError as kerr:\n                popular_articles_customer = get_popularity_based_prediction(customer)\n                recommendation = \"0\" + \" 0\".join([str(article) for article in popular_articles_customer])\n\n            recommendations.append(recommendation)\n\n        predictions_df = pd.DataFrame({'customer_id': customers, 'prediction': recommendations})\n\n        return predictions_df\n    \n    def evaluate(self, num_samples=None):\n        if num_samples is None:\n            train_items = self.predict(filtered_transactions_df, val_users.values, most_bought_articles, articles_id_to_idx, customers_id_to_idx)\n        else:\n            train_items = self.predict(filtered_transactions_df, val_users.values[:num_samples], most_bought_articles, articles_id_to_idx, customers_id_to_idx)\n        train_items = train_items.prediction.values\n        train_items = [prediction.split(\" \") for prediction in train_items]\n        train_items = [list(map(int, item)) for item in train_items]\n        score = mapk(val_items, train_items)\n        print(f\"Mapk score of {score}\")\n                \n    ","metadata":{"papermill":{"duration":0.04553,"end_time":"2022-03-02T12:44:09.236331","exception":false,"start_time":"2022-03-02T12:44:09.190801","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{}},{"cell_type":"markdown","source":"Note: The final model that has been submitted to Kaggle has been trained with 1000 components for 10 epochs with a learning rate of 0.001.","metadata":{}},{"cell_type":"code","source":"recommender = ItemBasedRecommender(train_df, num_articles, num_customers, num_components=2000, num_last_items=6)\nrecommender.fit(n_epochs=1, learning_rate=0.01)","metadata":{"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Submission","metadata":{}},{"cell_type":"code","source":"customers = submission_df.customer_id.values\nsubmission = recommender.predict(filtered_transactions_df, customers, most_bought_articles, articles_id_to_idx, customers_id_to_idx)","metadata":{"papermill":{"duration":129.979329,"end_time":"2022-03-02T14:42:14.429863","exception":false,"start_time":"2022-03-02T14:40:04.450534","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","metadata":{"papermill":{"duration":37.94884,"end_time":"2022-03-02T14:43:17.664800","exception":false,"start_time":"2022-03-02T14:42:39.715960","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}