{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M Recommendation - Artur Xarles & Enric Azuara","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport datetime\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:47:40.312077Z","iopub.execute_input":"2022-03-04T21:47:40.312362Z","iopub.status.idle":"2022-03-04T21:47:40.317601Z","shell.execute_reply.started":"2022-03-04T21:47:40.312329Z","shell.execute_reply":"2022-03-04T21:47:40.316611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read data and small modifications","metadata":{}},{"cell_type":"code","source":"#Read the data\n\npath = '../input/h-and-m-personalized-fashion-recommendations/'\narticles = pd.read_csv(path + 'articles.csv')\ntransactions = pd.read_csv(path + 'transactions_train.csv', dtype = {'article_id': 'str'})\ncustomers = pd.read_csv(path + 'customers.csv')\n\n#Transform t_dat to date format\ntransactions['t_dat'] = pd.to_datetime(transactions['t_dat'])","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:47:40.329317Z","iopub.execute_input":"2022-03-04T21:47:40.329951Z","iopub.status.idle":"2022-03-04T21:49:39.115023Z","shell.execute_reply.started":"2022-03-04T21:47:40.329889Z","shell.execute_reply":"2022-03-04T21:49:39.114272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook shows step by step the thought process and the final algorithm selection. First of all we will try a set of non personalized recommender systems and after that, we will use personalized methods such User and Item based Collaborative Filtering (only to show how it would work with an small dataset) and Matrix factorization methods.","metadata":{}},{"cell_type":"markdown","source":"## Non personalized recommender systems","metadata":{}},{"cell_type":"markdown","source":"The first approach we are going to follow is non personalized recommender systems. There are 2 main types:","metadata":{}},{"cell_type":"markdown","source":"   - Aggregated opinions (i.e. ranking)\n   - Basic product associations","metadata":{}},{"cell_type":"markdown","source":"### Aggregated opinions","metadata":{}},{"cell_type":"markdown","source":"Since there is not a measure to rank the items, we will look out for the most bought products.","metadata":{}},{"cell_type":"code","source":"transactions[\"article_id\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:49:39.116381Z","iopub.execute_input":"2022-03-04T21:49:39.117170Z","iopub.status.idle":"2022-03-04T21:49:49.487796Z","shell.execute_reply.started":"2022-03-04T21:49:39.117118Z","shell.execute_reply":"2022-03-04T21:49:49.486854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we are treating with clothes, this first approach is too naive, so we will take into account seasonality looking at the time when it was bought.","metadata":{}},{"cell_type":"markdown","source":"If we look at the last 4 weeks, the most bought products are the following ones:","metadata":{}},{"cell_type":"code","source":"print(transactions['t_dat'].max())\nprint(transactions['t_dat'].min())\ntransactions[(transactions['t_dat'] > '2020-08-26') & (transactions['t_dat'] < '2020-09-22')][\"article_id\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:49:49.489873Z","iopub.execute_input":"2022-03-04T21:49:49.490692Z","iopub.status.idle":"2022-03-04T21:49:51.142237Z","shell.execute_reply.started":"2022-03-04T21:49:49.490635Z","shell.execute_reply":"2022-03-04T21:49:51.141401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we look at the most bought items during september...","metadata":{}},{"cell_type":"code","source":"transactions[transactions['t_dat'].dt.month == 9][\"article_id\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:49:51.144011Z","iopub.execute_input":"2022-03-04T21:49:51.144326Z","iopub.status.idle":"2022-03-04T21:49:58.377047Z","shell.execute_reply.started":"2022-03-04T21:49:51.144295Z","shell.execute_reply":"2022-03-04T21:49:58.376398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And last, the most purchased product during last september","metadata":{}},{"cell_type":"code","source":"transactions[(transactions['t_dat'].dt.month == 9) & (transactions[\"t_dat\"].dt.year == 2020)][\"article_id\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:49:58.378144Z","iopub.execute_input":"2022-03-04T21:49:58.378782Z","iopub.status.idle":"2022-03-04T21:50:04.645266Z","shell.execute_reply.started":"2022-03-04T21:49:58.378749Z","shell.execute_reply":"2022-03-04T21:50:04.644214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we have seen, the most popular items vary depending which span times we take. We will consider this factor in order to make the predictions with this approach","metadata":{}},{"cell_type":"markdown","source":"### Basic products association","metadata":{}},{"cell_type":"markdown","source":"We will compute the most common products purchased together","metadata":{}},{"cell_type":"code","source":"transactions_sample = transactions[(transactions['t_dat'] > '2020-08-26')]\ndf_train = transactions[(transactions['t_dat'].dt.month == 9) & (transactions[\"t_dat\"].dt.year == 2020) & (transactions['t_dat'] < '2020-09-15')]\nvc = df_train.article_id.value_counts()\nvc = transactions.article_id.value_counts()\npairs = {}\nfor j,i in tqdm(enumerate(vc.index.values[1000:1032])):\n    #if j%10==0: print(j,', ',end='')\n    USERS = transactions.loc[transactions.article_id==i,'customer_id'].unique()\n    vc2 = transactions.loc[(transactions.customer_id.isin(USERS))&(transactions.article_id!=i),'article_id'].value_counts()\n    pairs[i] = [vc2.index[0], vc2.index[1], vc2.index[2]]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:50:04.646496Z","iopub.execute_input":"2022-03-04T21:50:04.646727Z","iopub.status.idle":"2022-03-04T21:58:05.072845Z","shell.execute_reply.started":"2022-03-04T21:50:04.646698Z","shell.execute_reply":"2022-03-04T21:58:05.071504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pairs","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:05.074262Z","iopub.execute_input":"2022-03-04T21:58:05.074559Z","iopub.status.idle":"2022-03-04T21:58:05.083368Z","shell.execute_reply.started":"2022-03-04T21:58:05.074487Z","shell.execute_reply":"2022-03-04T21:58:05.082502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that this method is a bit computationally slow so we only show an example of how it would be with an small subset of 32 articles.","metadata":{}},{"cell_type":"markdown","source":"### Comparing the different approaches","metadata":{}},{"cell_type":"markdown","source":"#### Aggregated rankings (count)","metadata":{}},{"cell_type":"markdown","source":"We will split the data for testing purposes. (Test data will be after 15 Sep)","metadata":{}},{"cell_type":"code","source":"last_week_start = datetime.datetime.strptime(\"16/09/20 00:00:00\", '%d/%m/%y %H:%M:%S')\ntransactions[\"days_distance\"] = (last_week_start - transactions[\"t_dat\"]).dt.days\ntransactions[\"weight\"] = 1\ntransactions[\"weight\"] *= np.exp(-(transactions[\"days_distance\"] / 3))\ntransactions['pop_factor'] = 1\ndf_train = transactions[(transactions['t_dat'].dt.month == 9) & (transactions[\"t_dat\"].dt.year == 2020) & (transactions['t_dat'] < '2020-09-15')]\ntest = transactions[(transactions['t_dat'] >= '2020-09-15') & (transactions['t_dat'] <= '2020-09-22')].groupby(['customer_id'])['article_id'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:05.084632Z","iopub.execute_input":"2022-03-04T21:58:05.084851Z","iopub.status.idle":"2022-03-04T21:58:16.561045Z","shell.execute_reply.started":"2022-03-04T21:58:05.084821Z","shell.execute_reply":"2022-03-04T21:58:16.559858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We define the first weight by time","metadata":{}},{"cell_type":"code","source":"df_train_g = df_train.groupby(\"article_id\").sum().reset_index()\ndf_train_sorted = df_train_g.sort_values(by=\"weight\",ascending=False)\nweigh1 = df_train_sorted[\"article_id\"].to_numpy()[:12]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:16.562630Z","iopub.execute_input":"2022-03-04T21:58:16.563156Z","iopub.status.idle":"2022-03-04T21:58:16.748765Z","shell.execute_reply.started":"2022-03-04T21:58:16.563103Z","shell.execute_reply":"2022-03-04T21:58:16.748111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Second way of weighting by time","metadata":{}},{"cell_type":"code","source":"df_train['pop_factor'] = df_train['t_dat'].apply(lambda x: 1/(datetime.datetime(2020,9,16) - x).days)\ndf_train_h = df_train.groupby(\"article_id\").sum().reset_index()\ndf_train_sorted2 = df_train_h.sort_values(by=\"pop_factor\",ascending=False)\nweigh2 = df_train_sorted2[\"article_id\"].to_numpy()[:12]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:16.752288Z","iopub.execute_input":"2022-03-04T21:58:16.752842Z","iopub.status.idle":"2022-03-04T21:58:26.940862Z","shell.execute_reply.started":"2022-03-04T21:58:16.752784Z","shell.execute_reply":"2022-03-04T21:58:26.940203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not weighting and just taking some time spans that can make sense","metadata":{}},{"cell_type":"code","source":"Last4W = transactions[(transactions['t_dat'] > '2020-08-26') & (transactions['t_dat'] < '2020-09-15')][\"article_id\"].value_counts().index[0:12].values\nAllSep = transactions[(transactions['t_dat'].dt.month == 9) & (transactions['t_dat'] < '2020-09-15')][\"article_id\"].value_counts().index[0:12].values\nSep2020 = transactions[(transactions['t_dat'].dt.month == 9) & (transactions[\"t_dat\"].dt.year == 2020) & (transactions['t_dat'] < '2020-09-15')][\"article_id\"].value_counts().index[0:12].values\nAll = transactions[(transactions['t_dat'] < '2020-09-15')][\"article_id\"].value_counts().index[0:12].values","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:26.942328Z","iopub.execute_input":"2022-03-04T21:58:26.942846Z","iopub.status.idle":"2022-03-04T21:58:49.409016Z","shell.execute_reply.started":"2022-03-04T21:58:26.942803Z","shell.execute_reply":"2022-03-04T21:58:49.408009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define metric to evaluate (MAP):","metadata":{}},{"cell_type":"code","source":"def apk(actual, predicted, k=12):\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=12):\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:49.410563Z","iopub.execute_input":"2022-03-04T21:58:49.410874Z","iopub.status.idle":"2022-03-04T21:58:49.420797Z","shell.execute_reply.started":"2022-03-04T21:58:49.410834Z","shell.execute_reply":"2022-03-04T21:58:49.418770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"outputsAll = []\noutputsAllSep = []\noutputsSep2020 = []\noutputs4W = []\noutputsWeigh1 = []\noutputsWeigh2 = []\n\nfor i in range(test.values.shape[0]):\n    outputsAll.append(list(All))\n    outputsAllSep.append(list(AllSep))\n    outputsSep2020.append(list(Sep2020))\n    outputs4W.append(list(Last4W))\n    outputsWeigh1.append(list(weigh1))\n    outputsWeigh2.append(list(weigh2))\n    \nprint(mapk(list(test.values), outputsAll))\nprint(mapk(list(test.values), outputsAllSep))\nprint(mapk(list(test.values), outputsSep2020))\nprint(mapk(list(test.values), outputs4W))\nprint(mapk(list(test.values), outputsWeigh1))\nprint(mapk(list(test.values), outputsWeigh2))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:49.422279Z","iopub.execute_input":"2022-03-04T21:58:49.422781Z","iopub.status.idle":"2022-03-04T21:58:52.580271Z","shell.execute_reply.started":"2022-03-04T21:58:49.422735Z","shell.execute_reply":"2022-03-04T21:58:52.579303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, we get very diverse results, where the weighted by time methods acquire a better mAP score.","metadata":{}},{"cell_type":"markdown","source":"#### Combined algorithm","metadata":{}},{"cell_type":"markdown","source":"As we have seen, taking weights from the time or not has different results, but it looks like it is still not enough. We will take as insipiration one of the notebooks provided to improve this result and combine weights with repetition.\n\nWe will make a recommendation that will have 3 scenarios:\n\n    1) The user has bought before 12 or more items: We will recommend the most recent 12 items bought by the user.\n    2) The user has bought before less than 12 items: We will recommend all the items bought by the user and the ones more popular weighted by time up to 12 items.\n    3) The user has not bought before: We will recommend the top12 popular items weighted by time.\n    \nTo test this algorithm, we will separate the data within the last 4 weeks in order to priorize the most recent ones.","metadata":{}},{"cell_type":"code","source":"train = transactions[(transactions['t_dat'] >= '2020-08-15') & (transactions['t_dat'] < '2020-09-16')]\ntrain_week_1 = transactions[(transactions['t_dat'] >= '2020-09-08') & (transactions['t_dat'] < '2020-09-16')]\ntrain_week_2 = transactions[(transactions['t_dat'] >= '2020-09-01') & (transactions['t_dat'] < '2020-09-08')]\ntrain_week_3 = transactions[(transactions['t_dat'] >= '2020-08-23') & (transactions['t_dat'] < '2020-09-01')]\ntrain_week_4 = transactions[(transactions['t_dat'] >= '2020-08-15') & (transactions['t_dat'] < '2020-08-23')]\ntest2 = transactions[(transactions['t_dat'] >= '2020-09-16')]\n\npositive_items_whole = train.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user1 = train_week_1.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user2 = train_week_2.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user3 = train_week_3.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user4 = train_week_4.groupby(['customer_id'])['article_id'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:58:52.581452Z","iopub.execute_input":"2022-03-04T21:58:52.581678Z","iopub.status.idle":"2022-03-04T21:59:08.373288Z","shell.execute_reply.started":"2022-03-04T21:58:52.581643Z","shell.execute_reply":"2022-03-04T21:59:08.372452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We issolate the most popular items the last two weeks before the test dataset to determine them as the most popular ones taking into account the most bought ones weighted by time.","metadata":{}},{"cell_type":"code","source":"popular_items_week_1_2_aux = pd.concat([train_week_1, train_week_2], axis=0).groupby(['article_id'])['pop_factor'].value_counts()\npopular_items_week_1_2 = list(popular_items_week_1_2_aux.droplevel(level=1).sort_values(ascending=False).index.values)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:59:08.374987Z","iopub.execute_input":"2022-03-04T21:59:08.375328Z","iopub.status.idle":"2022-03-04T21:59:08.635533Z","shell.execute_reply.started":"2022-03-04T21:59:08.375283Z","shell.execute_reply":"2022-03-04T21:59:08.634563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check how it performs on the test dataset","metadata":{}},{"cell_type":"code","source":"positive_items_val = test2.groupby(['customer_id'])['article_id'].apply(list)\n\nval_users = positive_items_val.keys()\nval_items = []\n\nfor i,user in enumerate(val_users):\n    val_items.append(positive_items_val[user])","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:59:08.636959Z","iopub.execute_input":"2022-03-04T21:59:08.637285Z","iopub.status.idle":"2022-03-04T21:59:10.964405Z","shell.execute_reply.started":"2022-03-04T21:59:08.637246Z","shell.execute_reply":"2022-03-04T21:59:10.963647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from collections import Counter\n\npositive_items_val = test2.groupby(['customer_id'])['article_id'].apply(list)\nval_users = positive_items_val.keys()\nval_items = []\noutputs = []\n\nfor i,user in enumerate(val_users):\n    val_items.append(positive_items_val[user])\n\nfor user in tqdm(val_users):\n    user_output = []\n    if user in positive_items_per_user1.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user2.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user3.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user4.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    \n    user_output += list(popular_items_week_1_2[:12 - len(user_output)])\n    outputs.append(user_output)\n    \nprint(\"mAP Score on Validation set:\", mapk(val_items, outputs))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T21:59:10.965819Z","iopub.execute_input":"2022-03-04T21:59:10.966572Z","iopub.status.idle":"2022-03-04T21:59:17.695673Z","shell.execute_reply.started":"2022-03-04T21:59:10.966491Z","shell.execute_reply":"2022-03-04T21:59:17.694815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, with this approach we improved the result by a large amount.","metadata":{}},{"cell_type":"markdown","source":"### Comparing with Basic products Association","metadata":{}},{"cell_type":"code","source":"train['article_id2'] = train.article_id.map(pairs)\ntrain2 = train.loc[:, ['customer_id','article_id2']].copy()\ntrain2 = train2.loc[train2.article_id2.notnull()]","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.370763Z","iopub.status.idle":"2022-03-04T18:10:24.3711Z","shell.execute_reply.started":"2022-03-04T18:10:24.370907Z","shell.execute_reply":"2022-03-04T18:10:24.370922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train2 = train[['customer_id','article_id2']].copy()\ntrain2 = train2.loc[train2.article_id2.notnull()]\ntrain2 = train2.rename({'article_id2':'article_id'},axis=1)\ntrain = train[['customer_id','article_id']]\ntrain = pd.concat([train,train2],axis=0,ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.372082Z","iopub.status.idle":"2022-03-04T18:10:24.372367Z","shell.execute_reply.started":"2022-03-04T18:10:24.372214Z","shell.execute_reply":"2022-03-04T18:10:24.372229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To compute this, we will select a subset of the users from the validation set, since finding the previous purchases for each of them is computationally exigent.\n\nFor the users that we do not have data before we will use the method proposed on the previous section.","metadata":{}},{"cell_type":"code","source":"from collections import Counter\n\npositive_items_val = test2.groupby(['customer_id'])['article_id'].apply(list)\nval_users = positive_items_val.keys()\nval_items = []\noutputs2 = []\nsubset = val_users[0:5000]\n\nfor i,user in enumerate(subset):\n    val_items.append(positive_items_val[user])\n\nfor user in tqdm(subset):\n    user_output2 = []\n    if user in train[\"customer_id\"].values:\n        user_output2 += list(train[train[\"customer_id\"]==user][\"article_id\"].values)\n\n    else:\n        if user in positive_items_per_user1.keys():\n            most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n            user_output += list(most_common_items_of_user.keys())[:12]\n        if user in positive_items_per_user2.keys():\n            most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n            user_output += list(most_common_items_of_user.keys())[:12]\n        if user in positive_items_per_user3.keys():\n            most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n            user_output += list(most_common_items_of_user.keys())[:12]\n        if user in positive_items_per_user4.keys():\n            most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n            user_output += list(most_common_items_of_user.keys())[:12]\n    \n    user_output2 += list(popular_items_week_1_2[:12 - len(user_output)])\n    outputs2.append(user_output2)\n    \nprint(\"mAP Score on Validation set:\", mapk(val_items, outputs2))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.373411Z","iopub.status.idle":"2022-03-04T18:10:24.373702Z","shell.execute_reply.started":"2022-03-04T18:10:24.373547Z","shell.execute_reply":"2022-03-04T18:10:24.373563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, we get a worse result (Map: 0.014). With more time, it would be good to explore which results we would get when trying different combined algorithms between weighting by time and commonly bought together products.","metadata":{}},{"cell_type":"markdown","source":"## Personalized recommender systems - Collaborative Filtering","metadata":{}},{"cell_type":"markdown","source":"### User and Item based Collaborative Filtering","metadata":{}},{"cell_type":"markdown","source":"After seeing the personalized methods, we will show how we would implement a User-Based and Item-Based CF recommender system to this dataset. As these methods are really slow when dealing with large datasets, we will do an example using only a really small dataset. At the end, this will not be used to make predictions and it will only be shown as an example.","metadata":{}},{"cell_type":"markdown","source":"We use a function to reduce the data and introduce a rating for each customer and article combination being the number of times that the article has been bought by a customer (we introduce 0 ratings if we do not have a transaction for these).","metadata":{}},{"cell_type":"code","source":"'''\nFunction to create a subset of the data for CF. We only use the transactions made after a certain date and the users with a \nminimum number of purchases after that day\n'''\ndef subset_transactions(transactions, init_date, min_purchases_customer, min_purchases_article, add_0rat = True, final_date = datetime.datetime(2022, 9, 20)):\n    #Get only recent transactions\n    transactions_small = transactions[(transactions['t_dat'] >= init_date) & (transactions['t_dat'] < final_date)]\n    #Get the customers with at least min_purchases purchases\n    customers = transactions_small['customer_id'].value_counts()[transactions_small['customer_id'].value_counts() > min_purchases_customer].index.to_list()\n    articles = transactions_small['article_id'].value_counts()[transactions_small['article_id'].value_counts() > min_purchases_article].index.to_list()\n    #Get transactions for these customers\n    transactions_small = transactions_small[transactions_small['customer_id'].isin(customers) & transactions_small['article_id'].isin(articles)]\n    #Each combination of customer-article one transaction with rating the number of times that this combination happens\n    transactions_small = transactions_small.groupby(['customer_id', 'article_id']).size().reset_index().rename(columns={0:'rating'})\n    #Add transactions with rating 0 for transaction combinations that has not been done\n    if add_0rat:\n        for customer in tqdm(transactions_small['customer_id'].unique()):\n            for article in transactions_small['article_id'].unique():\n                if len(transactions_small[(transactions_small['customer_id'] == customer) & (transactions_small['article_id'] == article)]) == 0:\n                    transactions_small = transactions_small.append({'customer_id': customer, 'article_id': article, 'rating': 0}, ignore_index = True)\n\n\n    print('New data has shape: ' + str(transactions_small.shape))\n    return transactions_small\n\ninitial_date = datetime.datetime(2020, 9, 20) #We use last two weeks\nmin_purchases_customer = 20\nmin_purchases_article = 70\n\n#20 and 50 correct\n\ntransactions_small = subset_transactions(transactions = transactions, init_date = initial_date, min_purchases_customer = min_purchases_customer, min_purchases_article = min_purchases_article)\nprint('Number of customers: ' + str(len(transactions_small['customer_id'].unique())))\nprint('Number of articles: ' + str(len(transactions_small['article_id'].unique())))\nprint(transactions_small.head())\n","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.374593Z","iopub.status.idle":"2022-03-04T18:10:24.374893Z","shell.execute_reply.started":"2022-03-04T18:10:24.37474Z","shell.execute_reply":"2022-03-04T18:10:24.374756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once we have the reduced dataset, we split the data into train and test to be able to compare different CF methodologies.","metadata":{}},{"cell_type":"code","source":"def assign_to_set(df):\n    sampled_ids = np.random.choice(df.index,\n                                   size=np.int64(np.ceil(df.index.size * 0.2)),\n                                   replace=False)\n    df.loc[sampled_ids, 'for_testing'] = True\n    return df\n\ntransactions_small['for_testing'] = False\ngrouped = transactions_small.groupby('customer_id', group_keys=False).apply(assign_to_set)\ntransactions_small_train = transactions_small[grouped.for_testing == False]\ntransactions_small_test = transactions_small[grouped.for_testing == True]\n\nprint(transactions_small_train.shape)\nprint(transactions_small_test.shape)\n\n\nprint(\"Training data_set has \"+ str(transactions_small_train.shape[0]) +\" ratings\")\nprint(\"Test data set has \"+ str(transactions_small_test.shape[0]) +\" ratings\")\nprint(\"The database has \", transactions_small.article_id.nunique(), \" articles\")\n","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.375714Z","iopub.status.idle":"2022-03-04T18:10:24.376013Z","shell.execute_reply.started":"2022-03-04T18:10:24.375849Z","shell.execute_reply":"2022-03-04T18:10:24.375864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the following code we create some functions for different similarities and classes to implement Collaborative Filtering in a user-based and item-based way. These classes contain a fit function, predict function (for customer and article specification) and a predict function for a customer that gives the top k articles respect to the rating. We also define a function to evaluate the recommendations based on RMSE. \nFor the User-Based we will use the following function to make the prediction:\n$$pred(a,p) = \\bar{r_a} + \\frac{\\sum_{b \\in N}{sim(a,b)*(r_{b,p}-\\bar{r_b})}}{\\sum_{b \\in N}{sim(a,b)}}$$\nas different users can have different mean of articles bought. Moreover, we will only use the subset of the top-20 most similar users in order to make the predictions.\nFor the Item-Based, we will also use the previous formula modificated to use the mean for the article and the top-20 most similar users.","metadata":{}},{"cell_type":"code","source":"from scipy.stats import pearsonr\nfrom scipy.spatial.distance import euclidean\nfrom tqdm import tqdm\n\n'''\nFunction to compute Euclidean Similarity\n'''\ndef SimEuclid(DataFrame,Customer1,Customer2,min_common_items=1):\n    # GET MOVIES OF USER1\n    movies_user1=DataFrame[DataFrame['customer_id'] ==Customer1 ]\n    # GET MOVIES OF USER2\n    movies_user2=DataFrame[DataFrame['customer_id'] ==Customer2 ]\n    \n    # FIND SHARED FILMS\n    rep=pd.merge(movies_user1 ,movies_user2,on='article_id')    \n    if len(rep)<2:\n        return 0\n    if(len(rep)<min_common_items):\n        return 0\n    #return distEuclid(rep['rating_x'],rep['rating_y']) \n    return 1.0/(1.0+euclidean(rep['rating_x'],rep['rating_y'])) \n\n'''\nFunction to compute Pearson Similarity\n'''\ndef SimPearson(DataFrame,Customer1,Customer2,min_common_items=1):\n    # GET MOVIES OF USER1\n    movies_user1=DataFrame[DataFrame['customer_id'] ==Customer1 ]\n    # GET MOVIES OF USER2\n    movies_user2=DataFrame[DataFrame['customer_id'] ==Customer2 ]\n    \n    # FIND SHARED FILMS\n    rep=pd.merge(movies_user1 ,movies_user2,on='article_id',)\n    if len(rep)<2:\n        return 0    \n    if(len(rep)<min_common_items):\n        return 0    \n    res=pearsonr(rep['rating_x'],rep['rating_y'])[0]\n    if(np.isnan(res)):\n        return 0\n    return res\n\n\n'''\nClass for Collaborative Filtering (user based)\n'''\nclass CollaborativeFilteringUB:\n    \"\"\" Collaborative filtering using a custom sim(u,u'). \"\"\"\n    \n    def __init__(self,DataFrame, similarity=SimPearson):\n        \"\"\" Constructor \"\"\"\n        self.sim_method=similarity# Gets recommendations for a person by using a weighted average\n        self.df=DataFrame\n        self.sim = {}\n        \n    def fit(self):\n        \"\"\" Prepare data structures for estimation. Similarity matrix for users \"\"\"\n        allUsers=set(self.df['customer_id'])\n        for person1 in tqdm(allUsers):\n            self.sim.setdefault(person1, {})\n            a=transactions_small_train[transactions_small_train['customer_id']==person1][['article_id']]\n            data_reduced=pd.merge(transactions_small_train,a,on='article_id')\n            for person2 in allUsers:\n                if person1==person2: continue\n                self.sim.setdefault(person2, {})\n                if(person1 in self.sim[person2]):continue # since is a simetric matrix\n                sim=self.sim_method(data_reduced,person1,person2)\n                if(sim<0):\n                    self.sim[person1][person2]=0\n                    self.sim[person2][person1]=0\n                else:\n                    self.sim[person1][person2]=sim\n                    self.sim[person2][person1]=sim\n    #Prediction for a given customer and article            \n    def predict(self, customer_id, article_id, topK = 20):\n        totals={}\n        movie_users=self.df[self.df['article_id'] ==article_id]\n        rating_num=0.0\n        rating_den=0.0\n        allUsers=set(movie_users['customer_id'])\n        user_mean = self.df[self.df['customer_id'] == customer_id].rating.mean()\n        top_similar_customers = sorted([(self.sim[customer_id][other], other) if (customer_id != other) else (0, other) for other in allUsers])[:topK]\n        for _, other in top_similar_customers:\n            if customer_id==other: continue \n            rating_num += self.sim[customer_id][other] * float(movie_users[movie_users['customer_id']==other]['rating'] - self.df[self.df['customer_id'] == other]['rating'].mean())\n            rating_den += self.sim[customer_id][other]\n        if rating_den==0: \n            if self.df.rating[self.df['article_id']==article_id].mean()>0:\n                # return the mean movie rating if there is no similar for the computation\n                return self.df.rating[self.df['article_id']==article_id].mean()\n            else:\n                # else return mean user rating \n                return self.df.rating[self.df['customer_id']==customer_id].mean()\n        return user_mean + rating_num / rating_den\n    \n    #Prediction of the top k articles for a customer\n    def predict2(self, customer_id, k):\n        totals={}\n        allArticles=set(self.df['article_id'])\n        articles_rating = []\n        for article2 in (allArticles):\n            rating = self.predict(customer_id, article2)\n            articles_rating.append([article2, rating])\n        preds = np.array(articles_rating)\n        return preds[np.argsort(preds[:, 1])][-k:, 0].astype('int').astype('str').tolist()\n\n    \n'''\nClass for Collaborative Filtering (item based)\n'''\nclass CollaborativeFilteringIB:\n    \"\"\" Collaborative filtering using a custom sim(i,i'). \"\"\"\n    \n    def __init__(self,DataFrame, similarity=SimPearson):\n        \"\"\" Constructor \"\"\"\n        self.sim_method=similarity# Gets recommendations for a person by using a weighted average\n        self.df=DataFrame\n        self.sim = {}\n        \n    def fit(self):\n        \"\"\" Prepare data structures for estimation. Similarity matrix for items \"\"\"\n        allItems=set(self.df['article_id'])\n        for item1 in tqdm(allItems):\n            self.sim.setdefault(item1, {})\n            a=transactions_small_train[transactions_small_train['article_id']==item1][['customer_id']]\n            data_reduced=pd.merge(transactions_small_train,a,on='customer_id')\n            for item2 in allItems:\n                if item1==item2: continue\n                self.sim.setdefault(item2, {})\n                if(item1 in self.sim[item2]):continue # since is a simetric matrix\n                sim=self.sim_method(data_reduced,item1,item2)\n                if(sim<0):\n                    self.sim[item1][item2]=0\n                    self.sim[item2][item1]=0\n                else:\n                    self.sim[item1][item2]=sim\n                    self.sim[item2][item1]=sim\n    #Prediction for a given customer and article            \n    def predict(self, customer_id, article_id, topK = 20):\n        totals={}\n        movie_users=self.df[self.df['customer_id'] == customer_id]\n        rating_num=0.0\n        rating_den=0.0\n        allItems=set(movie_users['article_id'])\n        article_mean = self.df[self.df['article_id'] == article_id].rating.mean()\n        top_similar_articles = sorted([(self.sim[article_id][other], other) if (article_id != other) else (0, other) for other in allItems])[:topK]\n        for _, other in top_similar_articles:\n            if article_id==other: continue \n            rating_num += self.sim[article_id][other] * float(movie_users[movie_users['article_id']==other]['rating'] - self.df[self.df['article_id'] == other]['rating'].mean())\n            rating_num += self.sim[article_id][other] * float(movie_users[movie_users['article_id']==other]['rating'])\n            rating_den += self.sim[article_id][other]\n        if rating_den==0: \n            if self.df.rating[self.df['customer_id']==customer_id].mean()>0:\n                # return the mean movie rating if there is no similar for the computation\n                return self.df.rating[self.df['customer_id']==customer_id].mean()\n            else:\n                # else return mean user rating \n                return self.df.rating[self.df['article_id']==article_id].mean()\n        return article_mean + rating_num/rating_den\n    \n    #Prediction of the top k articles for a customer\n    def predict2(self, customer_id):\n        totals={}\n        allArticles=set(self.df['article_id'])\n        articles_rating = []\n        for article2 in (allArticles):\n            rating = self.predict(customer_id, article2)\n            articles_rating.append([article2, rating])\n        preds = np.array(articles_rating)\n        return preds[np.argsort(preds[:, 1])][-k:, 0].astype('int').astype('str').tolist()\n\n    \ndef compute_rmse(y_pred, y_true):\n    \"\"\" Compute Root Mean Squared Error. \"\"\"\n    return np.sqrt(np.mean(np.power(y_pred - y_true, 2)))\n    \ndef evaluate(predict_f,data_train,data_test):\n    \"\"\" RMSE-based predictive performance evaluation with pandas. \"\"\"\n    ids_to_estimate = zip(data_test.customer_id, data_test.article_id)\n    list_customers = set(data_train.customer_id)\n    estimated = np.array([predict_f(u,i) if u in list_customers else 3 for (u,i) in ids_to_estimate ])\n    real = data_test.rating.values\n    return compute_rmse(estimated, real)\n           \n        ","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.379955Z","iopub.status.idle":"2022-03-04T18:10:24.380658Z","shell.execute_reply.started":"2022-03-04T18:10:24.380379Z","shell.execute_reply":"2022-03-04T18:10:24.380406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will fit the IBCF and UBCF using both distances, Euclidean and Pearson, and we will compare them using the test data and the RMSE as metric.","metadata":{}},{"cell_type":"code","source":"'''\nUser-Based Collaborative Filtering\n'''\nreco1 = CollaborativeFilteringUB(transactions_small)\nreco1.fit()\nids_to_estimate = zip(transactions_small_test.customer_id, transactions_small_test.article_id)\nestimated = np.array([reco1.predict(u,i) if u in transactions_small_train.customer_id else 3 for (u,i) in ids_to_estimate ])\nprint('RMSE for User-Based Collaborative Recomender: %s' % evaluate(reco1.predict,transactions_small_train,transactions_small_test))","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-03-04T18:10:24.381861Z","iopub.status.idle":"2022-03-04T18:10:24.382616Z","shell.execute_reply.started":"2022-03-04T18:10:24.382318Z","shell.execute_reply":"2022-03-04T18:10:24.382345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nUser-Based Collaborative Filtering\n'''\nreco2 = CollaborativeFilteringUB(transactions_small, similarity = SimEuclid)\nreco2.fit()\nids_to_estimate = zip(transactions_small_test.customer_id, transactions_small_test.article_id)\nestimated = np.array([reco2.predict(u,i) if u in transactions_small_train.customer_id else 3 for (u,i) in ids_to_estimate ])\nprint('RMSE for User-Based Collaborative Recomender: %s' % evaluate(reco2.predict,transactions_small_train,transactions_small_test))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.384141Z","iopub.status.idle":"2022-03-04T18:10:24.384606Z","shell.execute_reply.started":"2022-03-04T18:10:24.384348Z","shell.execute_reply":"2022-03-04T18:10:24.384372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nItem-Based Collaborative Filtering\n'''\nreco3 = CollaborativeFilteringIB(transactions_small)\nreco3.fit()\nids_to_estimate = zip(transactions_small_test.customer_id, transactions_small_test.article_id)\nestimated = np.array([reco3.predict(u,i) if u in transactions_small_train.customer_id else 3 for (u,i) in ids_to_estimate ])\nprint('RMSE for Item-Based Collaborative Recomender: %s' % evaluate(reco3.predict,transactions_small_train,transactions_small_test))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.386265Z","iopub.status.idle":"2022-03-04T18:10:24.387055Z","shell.execute_reply.started":"2022-03-04T18:10:24.386773Z","shell.execute_reply":"2022-03-04T18:10:24.386799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nItem-Based Collaborative Filtering\n'''\nreco4 = CollaborativeFilteringIB(transactions_small, similarity = SimEuclid)\nreco4.fit()\nids_to_estimate = zip(transactions_small_test.customer_id, transactions_small_test.article_id)\nestimated = np.array([reco4.predict(u,i) if u in transactions_small_train.customer_id else 3 for (u,i) in ids_to_estimate ])\nprint('RMSE for Item-Based Collaborative Recomender: %s' % evaluate(reco4.predict,transactions_small_train,transactions_small_test))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.388668Z","iopub.status.idle":"2022-03-04T18:10:24.389146Z","shell.execute_reply.started":"2022-03-04T18:10:24.388877Z","shell.execute_reply":"2022-03-04T18:10:24.388901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see the recommender system that has better results is Item Based using any of the similarities, so we would use this mehtod. However, as commented before this methods cannot be used for the whole dataset or at least for a relevant part of it as it takes too much time. In the following code we will use a more efficient recommender system to perform the recommendations:","metadata":{}},{"cell_type":"markdown","source":"### Factorization Models","metadata":{}},{"cell_type":"markdown","source":"In this case we will use a much larger dataset as this methods have less computational cost. However, we will also only use a part of the data as we believe that for users and articles for which we don't have much information (have not bought anything in a long period or only a few articles) it is better to use a non personalized recommender system.","metadata":{}},{"cell_type":"markdown","source":"First of all, we select the data from the last month and customers with at least 3 transactions and articles with at least 3 transactions. We also split in train and test data, taking as train the first two weeks and test the last one with available data.","metadata":{}},{"cell_type":"code","source":"initial_date = datetime.datetime(2020, 9, 1) #We use last two weeks\nmin_purchases_customer = 3\nmin_purchases_article = 3\ntrain_date = datetime.datetime(2020, 9, 15)\n\ntransactions_reduced_train = subset_transactions(transactions = transactions, init_date = initial_date, min_purchases_customer = min_purchases_customer, min_purchases_article = min_purchases_article, add_0rat = False, final_date = train_date)\ntransactions_reduced_test = subset_transactions(transactions = transactions, init_date = train_date, min_purchases_customer = min_purchases_customer, min_purchases_article = min_purchases_article, add_0rat = False)\n\nprint('Train data has shape: ' + str(transactions_reduced_train.shape))\nprint('Test data has shape: ' + str(transactions_reduced_test.shape))\n","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.39064Z","iopub.status.idle":"2022-03-04T18:10:24.39147Z","shell.execute_reply.started":"2022-03-04T18:10:24.391178Z","shell.execute_reply":"2022-03-04T18:10:24.391207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### SVD","metadata":{}},{"cell_type":"markdown","source":"The factorization method that we will use is the SVD factorization. The matrix that will be factorized contains the number of times that an article has been bought in our data for a customer. We also use only a part of the data to fit the model (from first two weeks of september) and another part to evaluate it through MAP (last week with data).","metadata":{}},{"cell_type":"code","source":"from scipy import sparse\nfrom scipy.linalg import sqrtm\n\nclass RecSys_mf():\n    \"\"\" Collaborative filtering using SVD. \"\"\"\n    \n    def __init__(self,df, num_components=10):\n        \"\"\" Constructor \"\"\"\n        self.num_components=num_components\n        self.train = pd.pivot_table(df[['customer_id','article_id','rating']],columns='article_id',index='customer_id',values='rating')\n        # We create a dictionary where we will store the user_id and movie_id which correspond \n        # to each index in the Rating matrix\n        \n        customer_index = np.arange(len(self.train.index))\n        self.customers = dict(zip(customer_index,self.train.index ))\n        self.customers_id2index = dict(zip(self.train.index,customer_index)) \n        \n        article_index = np.arange(len(self.train.columns))\n        self.articles = dict(zip(article_index,self.train.columns )) \n        self.articles_id2index= dict(zip(self.train.columns, article_index))\n        self.articles_index2id= dict(zip(article_index,self.train.columns))\n\n        \n        \n    def fit(self):\n        \"\"\" We decompose the R matrix into to submatrices using the training data \"\"\"\n        print('Fitting the model...')\n        train_matrix = np.array(self.train)\n        # we fill the nan values with 0 as have not been bought and remove the item average\n        train_matrix[np.isnan(train_matrix)] = 0\n        item_means = np.mean(train_matrix, axis=0)\n        x = np.tile(item_means, (train_matrix.shape[0],1))         \n        \n        # we remove the per item average from all entries.\n        # the above mentioned nan entries will be essentially zero now\n        train_matrix = train_matrix - x\n        print('Factorizing matrix...')\n        U, s, V = np.linalg.svd(train_matrix, full_matrices=False)\n\n        # reconstruct rating matix\n        S = np.diag(s[0:self.num_components])\n        U = U[:,0:self.num_components]\n        V = V[0:self.num_components,:]\n        S_root = sqrtm(S)\n        \n        print('Computing the predictions...')\n        USk=np.dot(U,S_root)\n        SkV=np.dot(S_root,V)\n        Y_hat = np.dot(USk, SkV)\n        self.Y_hat = Y_hat + x\n        \n\n        \n    def predict(self, customer_id, article_id):\n        if article_id in self.articles:\n            return self.Y_hat[self.customers_id2index[customer_id],self.articles_index2id[article_id]]\n        else: #in case it is a new article \n            return 0\n    def predict2(self, customer_id, k=12):\n        if customer_id in self.customers.values():\n            a = (np.argsort(-self.Y_hat[self.customers_id2index[customer_id]])<k)\n            b = ([i for i, x in enumerate(a) if x])\n            rec = []\n            for element in b:\n                rec.append(self.articles_index2id[element])\n        else:\n            return preds\n        return rec","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.392878Z","iopub.status.idle":"2022-03-04T18:10:24.39364Z","shell.execute_reply.started":"2022-03-04T18:10:24.393421Z","shell.execute_reply":"2022-03-04T18:10:24.393444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We fit the model with the train data.","metadata":{}},{"cell_type":"code","source":"reco = RecSys_mf(transactions_reduced_train,num_components=200)\nreco.fit()\n#print('RMSE for SVD: %s' % evaluate(reco.predict,data_train,data_test))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.394847Z","iopub.status.idle":"2022-03-04T18:10:24.395324Z","shell.execute_reply.started":"2022-03-04T18:10:24.395071Z","shell.execute_reply":"2022-03-04T18:10:24.395097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After fitting the model we evaluate it with the test dataset. (we only evaluate users that are in both datasets, train and test.","metadata":{}},{"cell_type":"code","source":"apks = []\nfor customer in tqdm(transactions_reduced_test['customer_id'].unique()):\n    if customer in transactions_reduced_train['customer_id'].unique():\n        ground_truth = list(transactions_reduced_test[transactions_reduced_test['customer_id'] == customer]['article_id'])\n        pred = reco.predict2(customer)\n        apks.append(apk(ground_truth, pred))\n        \nprint('Map achieved with SVD: ' + str(np.array(apks).mean()))","metadata":{"execution":{"iopub.status.busy":"2022-03-04T18:10:24.396759Z","iopub.status.idle":"2022-03-04T18:10:24.397228Z","shell.execute_reply.started":"2022-03-04T18:10:24.396962Z","shell.execute_reply":"2022-03-04T18:10:24.396986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, this method achieves a MAP of 0.00025 which is clearly worse than previous results.","metadata":{}},{"cell_type":"markdown","source":"## So, after trying several methods and evaluating them, we have seen that the one that achieves better results is the one that recommends the most recent articles bought by the user (if there are) or the most popular ones.","metadata":{}},{"cell_type":"markdown","source":"Following code create the predictions for the new data","metadata":{}},{"cell_type":"code","source":"transactions['pop_factor'] = 1\ntrain_week_1 = transactions[(transactions['t_dat'] >= '2020-09-16')]\ntrain_week_2 = transactions[(transactions['t_dat'] >= '2020-09-08') & (transactions['t_dat'] < '2020-09-16')]\ntrain_week_3 = transactions[(transactions['t_dat'] >= '2020-09-01') & (transactions['t_dat'] < '2020-09-08')]\ntrain_week_4 = transactions[(transactions['t_dat'] >= '2020-08-23') & (transactions['t_dat'] < '2020-09-01')]\n\n\n#Issolate users + their items purchased\npositive_items_per_user1 = train_week_1.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user2 = train_week_2.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user3 = train_week_3.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user4 = train_week_4.groupby(['customer_id'])['article_id'].apply(list)\n\n#Determine most popular items\npopular_items_week_1_2_aux = pd.concat([train_week_1, train_week_2], axis=0).groupby(['article_id'])['pop_factor'].value_counts()\npopular_items_week_1_2 = list(popular_items_week_1_2_aux.droplevel(level=1).sort_values(ascending=False).index.values)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T22:02:25.571780Z","iopub.execute_input":"2022-03-04T22:02:25.572157Z","iopub.status.idle":"2022-03-04T22:02:35.828819Z","shell.execute_reply.started":"2022-03-04T22:02:25.572125Z","shell.execute_reply":"2022-03-04T22:02:35.827768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from collections import Counter\noutputs = []\n\nfor user in tqdm(customers[\"customer_id\"].values):\n    user_output = []\n    if user in positive_items_per_user1.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user2.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user3.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user4.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    \n    user_output += list(popular_items_week_1_2[:12 - len(user_output)])\n    outputs.append(\" \".join(map(str,user_output)))\n\n","metadata":{"execution":{"iopub.status.busy":"2022-03-04T22:02:35.830448Z","iopub.execute_input":"2022-03-04T22:02:35.830667Z","iopub.status.idle":"2022-03-04T22:03:38.140864Z","shell.execute_reply.started":"2022-03-04T22:02:35.830640Z","shell.execute_reply":"2022-03-04T22:03:38.138834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'customer_id': customers['customer_id'], 'prediction': outputs})\nsubmission.to_csv('submissions.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2022-03-04T22:03:38.142024Z","iopub.execute_input":"2022-03-04T22:03:38.142259Z","iopub.status.idle":"2022-03-04T22:05:09.784289Z","shell.execute_reply.started":"2022-03-04T22:03:38.142228Z","shell.execute_reply":"2022-03-04T22:05:09.783509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The result achieved in the kaggle competition with these predictions for the 1% of evaluation data is a MAP of 0.0206","metadata":{}}]}