{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# H&M Recommendation: Time Decaying Popularity Baseline\n\n**Version[3] Notes**: I studied the differences between my code and (https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2) and made some adjustments accordingly. Also, I found that this version of heuristic algo gives a little worse(~0.00015) performance on validation if popularity is weighted down exponentially by days. Hence, I omiited the exponential weightage from the previous versions.\n\n#### **If you found this notebook helpful please leave an upvote.**\n\n\n\n## What we'll be doing here:\nThis is a heuristics based notebook. As we have already found out, popularity and repetition is king in this competition. We'll combine these two to create a good enough baseline.\n1. Recommend most bought items from last 4 weeks.\n2. Recommend popular items from last 2 weeks weighted down by time.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport glob\n#import reco\nfrom tqdm import tqdm\nimport datetime","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:41:42.463003Z","iopub.execute_input":"2022-02-24T05:41:42.463618Z","iopub.status.idle":"2022-02-24T05:41:42.469139Z","shell.execute_reply.started":"2022-02-24T05:41:42.463566Z","shell.execute_reply":"2022-02-24T05:41:42.468073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Forming Train Set","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv\", dtype={'article_id':str})\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:41:43.140818Z","iopub.execute_input":"2022-02-24T05:41:43.141141Z","iopub.status.idle":"2022-02-24T05:42:56.049051Z","shell.execute_reply.started":"2022-02-24T05:41:43.141105Z","shell.execute_reply":"2022-02-24T05:42:56.04835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll drop everything except the last few(up for experimentation) days. The info from previous months are not coming of much use. \nWe'll keep 4 weeks as train and the last week as validation.","metadata":{}},{"cell_type":"code","source":"print(\"All Transactions Date Range: {} to {}\".format(data['t_dat'].min(), data['t_dat'].max()))\n\ndata[\"t_dat\"] = pd.to_datetime(data[\"t_dat\"])\ntrain1 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,8)) & (data['t_dat'] < datetime.datetime(2020,9,16))]\ntrain2 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,1)) & (data['t_dat'] < datetime.datetime(2020,9,8))]\ntrain3 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,23)) & (data['t_dat'] < datetime.datetime(2020,9,1))]\ntrain4 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,15)) & (data['t_dat'] < datetime.datetime(2020,8,23))]\n\nval = data.loc[data[\"t_dat\"] >= datetime.datetime(2020,9,16)]","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:42:56.050519Z","iopub.execute_input":"2022-02-24T05:42:56.050745Z","iopub.status.idle":"2022-02-24T05:43:12.932047Z","shell.execute_reply.started":"2022-02-24T05:42:56.050719Z","shell.execute_reply":"2022-02-24T05:43:12.931216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Items which an user has bought in our train set time.","metadata":{}},{"cell_type":"code","source":"# List of all purchases per user (has repetitions)\npositive_items_per_user1 = train1.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user2 = train2.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user3 = train3.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user4 = train4.groupby(['customer_id'])['article_id'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:43:12.933364Z","iopub.execute_input":"2022-02-24T05:43:12.933993Z","iopub.status.idle":"2022-02-24T05:43:21.452873Z","shell.execute_reply.started":"2022-02-24T05:43:12.933955Z","shell.execute_reply":"2022-02-24T05:43:21.451884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next we do time decay based popularity for items. This leads to items bought more recently having more weight in the popularity list. In simple words, item A bought 5 times on the first day of the train period is inferior than item B bought 4 times on the last day of the train period.","metadata":{}},{"cell_type":"code","source":"train = pd.concat([train1, train2], axis=0)\ntrain['pop_factor'] = train['t_dat'].apply(lambda x: 1/(datetime.datetime(2020,9,16) - x).days)\npopular_items_group = train.groupby(['article_id'])['pop_factor'].sum()\n\n_, popular_items = zip(*sorted(zip(popular_items_group, popular_items_group.keys()))[::-1])\n\ntrain['pop_factor'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:44:46.345454Z","iopub.execute_input":"2022-02-24T05:44:46.34573Z","iopub.status.idle":"2022-02-24T05:44:57.133476Z","shell.execute_reply.started":"2022-02-24T05:44:46.345701Z","shell.execute_reply":"2022-02-24T05:44:57.132663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation: Evaluating the Idea","metadata":{}},{"cell_type":"code","source":"def apk(actual, predicted, k=12):\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=12):\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:45:32.863029Z","iopub.execute_input":"2022-02-24T05:45:32.863971Z","iopub.status.idle":"2022-02-24T05:45:32.871235Z","shell.execute_reply.started":"2022-02-24T05:45:32.863916Z","shell.execute_reply":"2022-02-24T05:45:32.870348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Items bought by users in the validation period. Similar as the one for train set.","metadata":{}},{"cell_type":"code","source":"positive_items_val = val.groupby(['customer_id'])['article_id'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:45:33.563148Z","iopub.execute_input":"2022-02-24T05:45:33.563418Z","iopub.status.idle":"2022-02-24T05:45:35.219409Z","shell.execute_reply.started":"2022-02-24T05:45:33.56339Z","shell.execute_reply":"2022-02-24T05:45:35.218574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# creating validation set for metrics use case\nval_users = positive_items_val.keys()\nval_items = []\n\nfor i,user in tqdm(enumerate(val_users)):\n    val_items.append(positive_items_val[user])\n    \nprint(\"Total users in validation:\", len(val_users))","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:45:35.220679Z","iopub.execute_input":"2022-02-24T05:45:35.220867Z","iopub.status.idle":"2022-02-24T05:45:35.725475Z","shell.execute_reply.started":"2022-02-24T05:45:35.220844Z","shell.execute_reply":"2022-02-24T05:45:35.72454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll now validate our algo on the validation set.","metadata":{}},{"cell_type":"code","source":"from collections import Counter\noutputs = []\ncnt = 0\n\npopular_items = list(popular_items)\n\nfor user in tqdm(val_users):\n    user_output = []\n    if user in positive_items_per_user1.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user2.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user3.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user4.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    \n    user_output += list(popular_items[:12 - len(user_output)])\n    outputs.append(user_output)\n    \nprint(\"mAP Score on Validation set:\", mapk(val_items, outputs))","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:45:42.451094Z","iopub.execute_input":"2022-02-24T05:45:42.451719Z","iopub.status.idle":"2022-02-24T05:45:46.965533Z","shell.execute_reply.started":"2022-02-24T05:45:42.451676Z","shell.execute_reply":"2022-02-24T05:45:46.96448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"0.02434 mAP. Not bad! We'll now create submissions by adjusting the date ranges of the train set.","metadata":{}},{"cell_type":"markdown","source":"# Prediction on Test Set: Submission","metadata":{}},{"cell_type":"code","source":"train1 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,16)) & (data['t_dat'] < datetime.datetime(2020,9,23))]\ntrain2 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,9,8)) & (data['t_dat'] < datetime.datetime(2020,9,16))]\ntrain3 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,31)) & (data['t_dat'] < datetime.datetime(2020,9,8))]\ntrain4 = data.loc[(data[\"t_dat\"] >= datetime.datetime(2020,8,23)) & (data['t_dat'] < datetime.datetime(2020,8,31))]\n\npositive_items_per_user1 = train1.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user2 = train2.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user3 = train3.groupby(['customer_id'])['article_id'].apply(list)\npositive_items_per_user4 = train4.groupby(['customer_id'])['article_id'].apply(list)\n\ntrain = pd.concat([train1, train2], axis=0)\ntrain['pop_factor'] = train['t_dat'].apply(lambda x: 1/(datetime.datetime(2020,9,23) - x).days)\npopular_items_group = train.groupby(['article_id'])['pop_factor'].sum()\n\n_, popular_items = zip(*sorted(zip(popular_items_group, popular_items_group.keys()))[::-1])\n\nuser_group = pd.concat([train1, train2, train3, train4], axis=0).groupby(['customer_id'])['article_id'].apply(list)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:49:41.46594Z","iopub.execute_input":"2022-02-24T05:49:41.466515Z","iopub.status.idle":"2022-02-24T05:50:12.544675Z","shell.execute_reply.started":"2022-02-24T05:49:41.466461Z","shell.execute_reply":"2022-02-24T05:50:12.543725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv(\"../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv\")\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:50:12.546253Z","iopub.execute_input":"2022-02-24T05:50:12.546554Z","iopub.status.idle":"2022-02-24T05:50:18.140667Z","shell.execute_reply.started":"2022-02-24T05:50:12.54652Z","shell.execute_reply":"2022-02-24T05:50:18.140011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from collections import Counter\noutputs = []\ncnt = 0\n\nfor user in tqdm(submission['customer_id']):\n    user_output = []\n    if user in positive_items_per_user1.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user1[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12]\n    if user in positive_items_per_user2.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user2[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12 - len(user_output)]\n    if user in positive_items_per_user3.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user3[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12 - len(user_output)]\n    if user in positive_items_per_user4.keys():\n        most_common_items_of_user = {k:v for k, v in Counter(positive_items_per_user4[user]).most_common()}\n        user_output += list(most_common_items_of_user.keys())[:12 - len(user_output)]\n    \n    user_output += list(popular_items[:12 - len(user_output)])\n    outputs.append(user_output)\n    \nstr_outputs = []\nfor output in outputs:\n    str_outputs.append(\" \".join([str(x) for x in output]))","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:50:18.141619Z","iopub.execute_input":"2022-02-24T05:50:18.141978Z","iopub.status.idle":"2022-02-24T05:50:40.679026Z","shell.execute_reply.started":"2022-02-24T05:50:18.141949Z","shell.execute_reply":"2022-02-24T05:50:40.677965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['prediction'] = str_outputs\nsubmission.to_csv(\"submissions.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:57:52.017137Z","iopub.execute_input":"2022-02-24T05:57:52.017577Z","iopub.status.idle":"2022-02-24T05:58:05.306309Z","shell.execute_reply.started":"2022-02-24T05:57:52.017542Z","shell.execute_reply":"2022-02-24T05:58:05.305468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T05:58:05.307953Z","iopub.execute_input":"2022-02-24T05:58:05.308205Z","iopub.status.idle":"2022-02-24T05:58:05.318824Z","shell.execute_reply.started":"2022-02-24T05:58:05.308175Z","shell.execute_reply":"2022-02-24T05:58:05.317958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}