{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n## **<span style=\"color:#3b51e3;font-size:150%\"><center>Welcome and Enjoy</center></span>**","metadata":{}},{"cell_type":"markdown","source":"In this notebook we will calculate MAP@12 using formula by Kaggle staff [average_precision.py](https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py)\n\nFrom kaggler public notebook: \n1. julian3833: https://www.kaggle.com/julian3833/h-m-content-based-12-most-popular-items-0-007?scriptVersionId=87501760\n2. gpreda: https://www.kaggle.com/gpreda/h-m-eda-and-prediction?scriptVersionId=87584685\n3. hengzheng: https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2?scriptVersionId=87521274\n4. julian3833: https://www.kaggle.com/julian3833/h-m-collaborative-filtering-user-user?scriptVersionId=88178355\n5. cdeotte: https://www.kaggle.com/cdeotte/recommend-items-purchased-together-0-021?scriptVersionId=88348330\n\nTrain-Valid Split Strategy: \n* Eval split start from '2020-09-16', Train split on the rest \n\nSome Tricks used: \n* Memory reduction by Chris Deotte : https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635\n* How to calculate MAP@12 by kaerururu : https://www.kaggle.com/kaerunantoka/h-m-how-to-calculate-map-12 \n","metadata":{}},{"cell_type":"markdown","source":"# **<a id=\"Content\" style=\"color:#3b51e3;\">Table of Content</a>**\n* [**<span style=\"color:#3b51e3;\">Results</span>**](#Results)  \n* [**<span style=\"color:#3b51e3;\">Data Preparation</span>**](#DataPrep)  \n* [**<span style=\"color:#3b51e3;\">julian3833</span>**](#julian3833)  \n* [**<span style=\"color:#3b51e3;\">gpreda</span>**](#gpreda)  \n* [**<span style=\"color:#3b51e3;\">hengzheng</span>**](#hengzheng)  \n* [**<span style=\"color:#3b51e3;\">julian3833 CF user-user</span>**](#julian3833UUCF)  \n* [**<span style=\"color:#3b51e3;\">cdeotte</span>**](#cdeotte)  \n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"## **<span id=\"Results\" style=\"color:#3b51e3;\">Results</span>**","metadata":{}},{"cell_type":"markdown","source":"\n|Notebook Owner|Notebook Link|Valid Score|Public Score|\n|---|---|---|---|\n|julian3833|[link](https://www.kaggle.com/julian3833/h-m-content-based-12-most-popular-items-0-007?scriptVersionId=87501760)|0.0067|0.0071|\n|gpreda|[link](https://www.kaggle.com/gpreda/h-m-eda-and-prediction?scriptVersionId=87584685)|0.0211|0.0195|\n|hengzheng|[link](https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2?scriptVersionId=87521274)|0.0237|0.0204|\n|julian3833|[link](https://www.kaggle.com/julian3833/h-m-collaborative-filtering-user-user?scriptVersionId=88178355)|0.0224|0.0193|\n|cdeotte|[link](https://www.kaggle.com/cdeotte/recommend-items-purchased-together-0-021)|0.0245|0.0214|\n","metadata":{}},{"cell_type":"markdown","source":"# **<span id=\"DataPrep\" style=\"color:#3b51e3;\">Data Preparation</span>**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nfrom datetime import datetime, timedelta\nimport gc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-28T06:59:18.047882Z","iopub.execute_input":"2022-02-28T06:59:18.048745Z","iopub.status.idle":"2022-02-28T06:59:18.055447Z","shell.execute_reply.started":"2022-02-28T06:59:18.048692Z","shell.execute_reply":"2022-02-28T06:59:18.054414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read dataset \n## memory reduction in importing dtype={'article_id': 'int32'}\ntransactions = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv',\n                          usecols= ['t_dat', 'customer_id', 'article_id'], dtype={'article_id': 'int32'})\n\ncustomers = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/customers.csv',\n                        usecols=['customer_id'])","metadata":{"execution":{"iopub.status.busy":"2022-02-28T06:59:20.686573Z","iopub.execute_input":"2022-02-28T06:59:20.686841Z","iopub.status.idle":"2022-02-28T07:00:20.817349Z","shell.execute_reply.started":"2022-02-28T06:59:20.686813Z","shell.execute_reply":"2022-02-28T07:00:20.816328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# memory reduction \ntransactions['t_dat'] = transactions['t_dat'].map(lambda x: datetime.strptime(x, '%Y-%m-%d'))\ntransactions['customer_id'] = transactions['customer_id'].apply(lambda x: int(x[-16:],16) ).astype('int64')\ncustomers['customer_id'] = customers['customer_id'].apply(lambda x: int(x[-16:],16) ).astype('int64')\n\n# Splitting\nval_start_date = '2020-09-16'\ntrain_df = transactions.query(f\"t_dat < '{val_start_date}'\").reset_index(drop=True)\nvalid_df = transactions.query(f\"t_dat >= '{val_start_date}'\").reset_index(drop=True)\n\n# Sorting\ntrain_df = train_df.sort_values([\"customer_id\", \"t_dat\"], ascending=False)\nvalid_df = valid_df.sort_values([\"customer_id\", \"t_dat\"], ascending=False)\n\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:00:20.819422Z","iopub.execute_input":"2022-02-28T07:00:20.819792Z","iopub.status.idle":"2022-02-28T07:07:57.791271Z","shell.execute_reply.started":"2022-02-28T07:00:20.819747Z","shell.execute_reply":"2022-02-28T07:07:57.790042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:07:57.793152Z","iopub.execute_input":"2022-02-28T07:07:57.79413Z","iopub.status.idle":"2022-02-28T07:07:57.814664Z","shell.execute_reply.started":"2022-02-28T07:07:57.794064Z","shell.execute_reply":"2022-02-28T07:07:57.81362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:07:57.817822Z","iopub.execute_input":"2022-02-28T07:07:57.818196Z","iopub.status.idle":"2022-02-28T07:07:57.832335Z","shell.execute_reply.started":"2022-02-28T07:07:57.818151Z","shell.execute_reply":"2022-02-28T07:07:57.830879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_df = valid_df.sort_values(['customer_id', 't_dat'], ascending = [True, True]) \nvalid_cust = valid_df.groupby('customer_id')['article_id'].apply(list).reset_index()\nvalid_cust['valid_true'] = valid_cust['article_id'].map(lambda x: '0'+' 0'.join(str(x)[1:-1].split(', ')))\ndel valid_df, valid_cust['article_id']\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:07:57.834508Z","iopub.execute_input":"2022-02-28T07:07:57.835155Z","iopub.status.idle":"2022-02-28T07:07:59.620611Z","shell.execute_reply.started":"2022-02-28T07:07:57.835109Z","shell.execute_reply":"2022-02-28T07:07:59.619641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_cust.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:07:59.622112Z","iopub.execute_input":"2022-02-28T07:07:59.622474Z","iopub.status.idle":"2022-02-28T07:07:59.633772Z","shell.execute_reply.started":"2022-02-28T07:07:59.622432Z","shell.execute_reply":"2022-02-28T07:07:59.632069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Util functions","metadata":{}},{"cell_type":"code","source":"# Source: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\ndef apk(actual, predicted, k=10):\n    \"\"\"\n    Computes the average precision at k.\n    This function computes the average prescision at k between two lists of\n    items.\n    Parameters\n    ----------\n    actual : list\n             A list of elements that are to be predicted (order doesn't matter)\n    predicted : list\n                A list of predicted elements (order does matter)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The average precision at k over the input lists\n    \"\"\"\n    if len(predicted)>k:\n        predicted = predicted[:k]\n\n    score = 0.0\n    num_hits = 0.0\n\n    for i,p in enumerate(predicted):\n        if p in actual and p not in predicted[:i]:\n            num_hits += 1.0\n            score += num_hits / (i+1.0)\n\n    if not actual:\n        return 0.0\n\n    return score / min(len(actual), k)\n\ndef mapk(actual, predicted, k=12):\n    \"\"\"\n    Computes the mean average precision at k.\n    This function computes the mean average prescision at k between two lists\n    of lists of items.\n    Parameters\n    ----------\n    actual : list\n             A list of lists of elements that are to be predicted \n             (order doesn't matter in the lists)\n    predicted : list\n                A list of lists of predicted elements\n                (order matters in the lists)\n    k : int, optional\n        The maximum number of predicted elements\n    Returns\n    -------\n    score : double\n            The mean average precision at k over the input lists\n    \"\"\"\n    return np.mean([apk(a,p,k) for a,p in zip(actual, predicted) if a]) # CHANGES: ignore null actual (variable=a)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-28T07:07:59.635935Z","iopub.execute_input":"2022-02-28T07:07:59.636592Z","iopub.status.idle":"2022-02-28T07:07:59.648899Z","shell.execute_reply.started":"2022-02-28T07:07:59.636551Z","shell.execute_reply":"2022-02-28T07:07:59.647741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span id=\"julian3833\" style=\"color:#3b51e3;\">julian3833</span>**","metadata":{}},{"cell_type":"markdown","source":"https://www.kaggle.com/julian3833/h-m-content-based-12-most-popular-items-0-007?scriptVersionId=87501760 Version 4","metadata":{}},{"cell_type":"code","source":"eval_julian3833 = customers.copy() \n\ntop_12_items = train_df[train_df['t_dat'] > '2020-09-01'].groupby('article_id')['customer_id'].nunique().sort_values(ascending=False).head(12).index.tolist()\ntop_12_items = ['0' + str(item) for item in top_12_items]\neval_julian3833['prediction'] =  ' '.join(top_12_items)\neval_julian3833 = valid_cust.merge(eval_julian3833, on ='customer_id', how ='left')\n","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:38:56.704335Z","iopub.execute_input":"2022-02-26T02:38:56.705038Z","iopub.status.idle":"2022-02-26T02:38:57.293576Z","shell.execute_reply.started":"2022-02-26T02:38:56.704999Z","shell.execute_reply":"2022-02-26T02:38:57.292435Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval_julian3833.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:38:57.296572Z","iopub.execute_input":"2022-02-26T02:38:57.297154Z","iopub.status.idle":"2022-02-26T02:38:57.308229Z","shell.execute_reply.started":"2022-02-26T02:38:57.297114Z","shell.execute_reply":"2022-02-26T02:38:57.307455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mapk(\n    eval_julian3833['valid_true'].map(lambda x: x.split()), \n    eval_julian3833['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:38:57.30951Z","iopub.execute_input":"2022-02-26T02:38:57.309817Z","iopub.status.idle":"2022-02-26T02:38:57.883803Z","shell.execute_reply.started":"2022-02-26T02:38:57.309778Z","shell.execute_reply":"2022-02-26T02:38:57.882831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eval_julian3833, top_12_items\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:38:57.885335Z","iopub.execute_input":"2022-02-26T02:38:57.885616Z","iopub.status.idle":"2022-02-26T02:38:58.005249Z","shell.execute_reply.started":"2022-02-26T02:38:57.885578Z","shell.execute_reply":"2022-02-26T02:38:58.004123Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span id=\"gpreda\" style=\"color:#3b51e3;\">gpreda</span>**","metadata":{"execution":{"iopub.status.busy":"2022-02-16T03:30:39.883324Z","iopub.execute_input":"2022-02-16T03:30:39.884001Z","iopub.status.idle":"2022-02-16T03:30:39.889123Z","shell.execute_reply.started":"2022-02-16T03:30:39.883961Z","shell.execute_reply":"2022-02-16T03:30:39.88834Z"}}},{"cell_type":"markdown","source":"https://www.kaggle.com/gpreda/h-m-eda-and-prediction?scriptVersionId=87584685 version 12","metadata":{}},{"cell_type":"code","source":"eval_gpreda = customers.copy() \n\nmost_frequent_articles = list(train_df.loc[train_df.t_dat==train_df.t_dat.max()].article_id.value_counts()[0:12].index)\nart_list = []\nfor art in most_frequent_articles:\n    art = \"0\"+str(art)\n    art_list.append(art)\nart_str = \" \".join(art_list)\n\ndef padding_articles(x):\n    if x:\n        xl = x.split()\n        x = []\n        for xi in xl:\n            x.append(\"0\"+xi)\n        dimm_x = len(x)\n        if dimm_x < 12:\n            x.extend(art_list[:12-dimm_x])\n        return(\" \".join(x))\n    \n\neval_gpreda = train_df.groupby([\"customer_id\"])[\"article_id\"].agg(lambda x: str(x.values[0:12])[1:-1]).reset_index()\neval_gpreda[\"prediction\"] = eval_gpreda[\"article_id\"].apply(lambda x: padding_articles(x))\neval_gpreda = valid_cust.merge(eval_gpreda, on ='customer_id', how ='left')\n\neval_gpreda['prediction'] = eval_gpreda['prediction'].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:38:58.007225Z","iopub.execute_input":"2022-02-26T02:38:58.00779Z","iopub.status.idle":"2022-02-26T02:40:36.146601Z","shell.execute_reply.started":"2022-02-26T02:38:58.007745Z","shell.execute_reply":"2022-02-26T02:40:36.145622Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval_gpreda.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:40:36.147974Z","iopub.execute_input":"2022-02-26T02:40:36.148252Z","iopub.status.idle":"2022-02-26T02:40:36.1603Z","shell.execute_reply.started":"2022-02-26T02:40:36.148219Z","shell.execute_reply":"2022-02-26T02:40:36.159638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mapk(\n    eval_gpreda['valid_true'].map(lambda x: x.split()), \n    eval_gpreda['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:40:36.161798Z","iopub.execute_input":"2022-02-26T02:40:36.162189Z","iopub.status.idle":"2022-02-26T02:40:36.716656Z","shell.execute_reply.started":"2022-02-26T02:40:36.162147Z","shell.execute_reply":"2022-02-26T02:40:36.715929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eval_gpreda, most_frequent_articles, art_list\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:40:36.718142Z","iopub.execute_input":"2022-02-26T02:40:36.718664Z","iopub.status.idle":"2022-02-26T02:40:36.839451Z","shell.execute_reply.started":"2022-02-26T02:40:36.718624Z","shell.execute_reply":"2022-02-26T02:40:36.837987Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span id=\"hengzheng\" style=\"color:#3b51e3;\">hengzheng</span>**","metadata":{"execution":{"iopub.status.busy":"2022-02-26T00:32:41.022719Z","iopub.execute_input":"2022-02-26T00:32:41.023025Z","iopub.status.idle":"2022-02-26T00:32:41.183003Z","shell.execute_reply.started":"2022-02-26T00:32:41.022988Z","shell.execute_reply":"2022-02-26T00:32:41.182029Z"}}},{"cell_type":"markdown","source":"https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2?scriptVersionId=87521274 version 5","metadata":{}},{"cell_type":"code","source":"eval_hengzheng = customers[['customer_id']]\n\ntransactions_3w = train_df[train_df['t_dat'] >= pd.to_datetime('2020-08-24')].copy()\ntransactions_2w = train_df[train_df['t_dat'] >= pd.to_datetime('2020-08-31')].copy()\ntransactions_1w = train_df[train_df['t_dat'] >= pd.to_datetime('2020-09-07')].copy()\n\npurchase_dict_3w = {}\n\nfor i,x in enumerate(zip(transactions_3w['customer_id'], transactions_3w['article_id'])):\n    cust_id, art_id = x\n    if cust_id not in purchase_dict_3w:\n        purchase_dict_3w[cust_id] = {}\n    \n    if art_id not in purchase_dict_3w[cust_id]:\n        purchase_dict_3w[cust_id][art_id] = 0\n    \n    purchase_dict_3w[cust_id][art_id] += 1\n    \ndummy_list_3w = list((transactions_3w['article_id'].value_counts()).index)[:12]\ndummy_list_3w = ['0' + str(item) for item in dummy_list_3w]\n\n\npurchase_dict_2w = {}\n\nfor i,x in enumerate(zip(transactions_2w['customer_id'], transactions_2w['article_id'])):\n    cust_id, art_id = x\n    if cust_id not in purchase_dict_2w:\n        purchase_dict_2w[cust_id] = {}\n    \n    if art_id not in purchase_dict_2w[cust_id]:\n        purchase_dict_2w[cust_id][art_id] = 0\n    \n    purchase_dict_2w[cust_id][art_id] += 1\n    \n\ndummy_list_2w = list((transactions_2w['article_id'].value_counts()).index)[:12]\ndummy_list_2w = ['0' + str(item) for item in dummy_list_2w]\n\n\npurchase_dict_1w = {}\n\nfor i,x in enumerate(zip(transactions_1w['customer_id'], transactions_1w['article_id'])):\n    cust_id, art_id = x\n    if cust_id not in purchase_dict_1w:\n        purchase_dict_1w[cust_id] = {}\n    \n    if art_id not in purchase_dict_1w[cust_id]:\n        purchase_dict_1w[cust_id][art_id] = 0\n    \n    purchase_dict_1w[cust_id][art_id] += 1\n    \ndummy_list_1w = list((transactions_1w['article_id'].value_counts()).index)[:12]\ndummy_list_1w = ['0' + str(item) for item in dummy_list_1w]\n\n\nprediction_list = []\n\ndummy_list = list((transactions_1w['article_id'].value_counts()).index)[:12]\ndummy_list = ['0' + str(item) for item in dummy_list]\ndummy_pred = ' '.join(dummy_list)\n\nfor i, cust_id in enumerate(eval_hengzheng['customer_id'].values.reshape((-1,))):\n    if cust_id in purchase_dict_1w:\n        l = sorted((purchase_dict_1w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n        l = ['0' + str(y[0]) for y in l]\n        if len(l)>12:\n            s = ' '.join(l[:12])\n        else:\n            s = ' '.join(l+dummy_list_1w[:(12-len(l))])\n    elif cust_id in purchase_dict_2w:\n        l = sorted((purchase_dict_2w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n        l = ['0' + str(y[0]) for y in l]\n        if len(l)>12:\n            s = ' '.join(l[:12])\n        else:\n            s = ' '.join(l+dummy_list_2w[:(12-len(l))])\n    elif cust_id in purchase_dict_3w:\n        l = sorted((purchase_dict_3w[cust_id]).items(), key=lambda x: x[1], reverse=True)\n        l = ['0' + str(y[0]) for y in l]\n        if len(l)>12:\n            s = ' '.join(l[:12])\n        else:\n            s = ' '.join(l+dummy_list_3w[:(12-len(l))])\n    else:\n        s = dummy_pred\n    prediction_list.append(s)\n\neval_hengzheng['prediction'] = prediction_list\neval_hengzheng = valid_cust.merge(eval_hengzheng, on ='customer_id', how ='left')\neval_hengzheng['prediction'] = eval_hengzheng['prediction'].astype(str)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-27T03:25:43.305867Z","iopub.execute_input":"2022-02-27T03:25:43.306175Z","iopub.status.idle":"2022-02-27T03:25:50.010488Z","shell.execute_reply.started":"2022-02-27T03:25:43.306144Z","shell.execute_reply":"2022-02-27T03:25:50.009418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval_hengzheng.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-27T03:25:50.012076Z","iopub.execute_input":"2022-02-27T03:25:50.012334Z","iopub.status.idle":"2022-02-27T03:25:50.023429Z","shell.execute_reply.started":"2022-02-27T03:25:50.012304Z","shell.execute_reply":"2022-02-27T03:25:50.022605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mapk(\n    eval_hengzheng['valid_true'].map(lambda x: x.split()), \n    eval_hengzheng['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-27T03:25:53.284071Z","iopub.execute_input":"2022-02-27T03:25:53.284365Z","iopub.status.idle":"2022-02-27T03:25:54.057111Z","shell.execute_reply.started":"2022-02-27T03:25:53.28433Z","shell.execute_reply":"2022-02-27T03:25:54.056129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del prediction_list, transactions_3w, transactions_2w, transactions_1w\ndel purchase_dict_3w, purchase_dict_2w, purchase_dict_1w, dummy_list_3w, dummy_list_2w,dummy_list_1w\ndel dummy_list, dummy_pred\n\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-02-26T02:40:44.458696Z","iopub.execute_input":"2022-02-26T02:40:44.45898Z","iopub.status.idle":"2022-02-26T02:40:44.660373Z","shell.execute_reply.started":"2022-02-26T02:40:44.458946Z","shell.execute_reply":"2022-02-26T02:40:44.659649Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span id=\"julian3833UUCF\" style=\"color:#3b51e3;\">julian3833 CF user-user</span>**","metadata":{}},{"cell_type":"markdown","source":"https://www.kaggle.com/julian3833/h-m-collaborative-filtering-user-user?scriptVersionId=88178355 version 19 ","metadata":{}},{"cell_type":"code","source":"import time\nimport multiprocessing as mp\nfrom multiprocessing import Pool\nfrom functools import partial\n\ndfu = customers.copy()\ndfi = pd.read_csv('../input/h-and-m-personalized-fashion-recommendations/articles.csv',\n                        usecols=['article_id'], dtype={'article_id': 'int32'})\n\nALL_USERS = dfu['customer_id'].unique().tolist()\nALL_ITEMS = dfi['article_id'].unique().tolist()\n\nuser_to_customer_map = {user_id: customer_id for user_id, customer_id in enumerate(ALL_USERS)}\ncustomer_to_user_map = {customer_id: user_id for user_id, customer_id in enumerate(ALL_USERS)}\n\nitem_to_article_map = {item_id: article_id for item_id, article_id in enumerate(ALL_ITEMS)}\narticle_to_item_map = {article_id: item_id for item_id, article_id in enumerate(ALL_ITEMS)}\n\ndel dfu, dfi\n\ndf = train_df.copy() \n\ndf['user_id'] = df['customer_id'].map(customer_to_user_map)\ndf['item_id'] = df['article_id'].map(article_to_item_map)\n\n#Configuration parameters\nN_SIMILAR_USERS = 20\nMINIMUM_PURCHASES = 10\nSTART_DATE = '2020-08-31' # '2020-09-07'\nDROP_PURCHASED_ITEMS = False\nDROP_USER_FROM_HIS_NEIGHBORHOOD = False\nTEST_RUN = False\nTEST_SIZE = 1000\n\ndef flatten(l):\n    \"\"\" Flatten a list of lists\"\"\"\n    return [item for sublist in l for item in sublist]\n\ndef compare_vectors(v1, v2):\n    \"\"\"Compare lists of purchased product for two given users\n    v1 stands for the \"vector representation for user 1\", which is a list of the purchases of u1\n    \n    Returns:\n        A value between 0 and 1 (similarity)\n    \"\"\"\n    intersection = len(set(v1) & set(v2))\n    denominator = np.sqrt(len(v1) * len(v2))\n    return intersection / denominator\n\ndef get_similar_users(u, v, dfh):\n    \"\"\"\n    Get the N_SIMILAR_USERS most similar users to the given one with their similarity score\n    Arguments:\n        u: the user_id, \n        v:  the \"vector\" representation of the user (list of item_id)\n        dfh : the \"history of transaccions\" dataframe\n        \n    Returns:\n        tuple of lists ([similar user_id], [similarity scores])\n    \"\"\"\n    similar_users = dfh.apply(lambda v_other: compare_vectors(v, v_other)).sort_values(ascending=False).head(N_SIMILAR_USERS + 1)\n    \n    if DROP_USER_FROM_HIS_NEIGHBORHOOD:\n        similar_users = similar_users[similar_users.index != u]\n        \n    return similar_users.index.tolist(), similar_users.tolist()\n\ndef get_items(u, v, dfh):\n    \"\"\" Get the recommend items for a given users\n    \n    It will:\n        1) Get similar users for the given user\n        2) Obtain all the items those users purchased\n        3) Rank them using the similarity scores of the user that purchased them\n        4) Return the 12 best ranked\n    \n    Arguments:\n        u: the user_id, \n        v:  the \"vector\" representation of the user (list of item_id)\n        dfh : the \"history of transaccions\" dataframe\n        \n    Returns:\n        list of item_id of lenght at most 12\n    \"\"\"\n    global i, n\n    \n    users, scores = get_similar_users(u, v, dfh)\n    df_nn = pd.DataFrame({'user': users, 'score': scores})\n    df_nn['items'] = df_nn.apply(lambda row: dfh.loc[row.user], axis=1)\n    df_nn['weighted_items'] = df_nn.apply(lambda row: [(item, row.score) for item in row['items']], axis=1)\n\n    recs = pd.DataFrame(flatten(df_nn['weighted_items'].tolist()), columns=['item', 'score']).groupby('item')['score'].sum().sort_values(ascending=False)\n    if DROP_PURCHASED_ITEMS:\n        recs = recs[~recs.index.isin(v)]\n    # Keep the first 12 and get the item_ids\n    i +=1\n    if i % 200 == 0:\n        pid = mp.current_process().pid\n        print(f\"[PID {pid:>2d}] Finished {i:3d} / {n:5d} - {i/n*100:3.0f}%\")\n    return recs.head(12).index.tolist()\n\ndef get_items_chunk(user_ids: np.array, dfh: pd.DataFrame):\n    \"\"\" Call get_item for a list of user_ids\n    \n    Arguments:\n        user_ids: list of user_id, \n        dfh: the \"history of transaccions\" dataframe\n        \n    Returns:\n        pd.Series with index user_id and list of item_id (recommendations) as value\n    \"\"\"\n    global i, n\n    i = 0\n    \n    n = len(user_ids)\n    pid = mp.current_process().pid\n    print(f\"[PID {pid:>2d}] Started working with {n:5d} users\")\n    \n    df_user_vectors = pd.DataFrame(dfh.loc[user_ids]).reset_index()\n    df_user_vectors['recs'] = df_user_vectors.apply(lambda row: get_items(row.user_id, row.item_id, dfh), axis=1)\n    return df_user_vectors.set_index('user_id')['recs']\n\ndef get_recommendations(users: list, dfh: pd.DataFrame):\n    \"\"\"\n    Obtained recommendation for the users using transaccion dfh in a parallelized manner\n    \n    Call get_items_chunk in a \"smart\" multiprocessing fashion\n    \n    Arguments:\n        users: list of user_id\n        dfh: the \"history of transaccions\" dataframe\n    \n    Returns:\n        pd.DataFrame with index user_id and list of item_id (recommendations) as value\n    \n    \"\"\"\n    time_start = time.time()\n    \n    # Split into approximately evenly sized chunks\n    # We will send just one batch to each CPU \n    user_chunks = np.array_split(users, mp.cpu_count())\n    \n    f = partial(get_items_chunk, dfh=dfh)\n    with Pool(mp.cpu_count()) as p:\n        res = p.map(f, user_chunks)\n    \n    df_rec = pd.DataFrame(pd.concat(res))\n\n    elapsed = (time.time() - time_start) / 60\n    print(f\"Finished get_recommendations({len(users)}). It took {elapsed:5.2f} mins\")\n    return df_rec\n\n\ndef uucf(df, start_date=START_DATE):\n    \"\"\" Entry point for the UUCF model. \n    \n    Receive the original transactions_train.csv and a start_date and gets UUCF recommendations\n    \n    The model will not cover the full list of users, but just a subset of them.\n    \n    It will provide recommendations for users with at least MINIMUM_PURCHASES after start_date.\n    It might return less than 12 recs per user.\n    \n    An ad-hoc function for filling these gaps should be used downstream.\n    (See fill functionality right below)\n    \n    \n    Arguments:\n        df: The raw dataframe from transactions_train.csv\n        start_date: a date\n        \n    Returns:\n        a submission-like pd.DataFrame with columns [customer_id, prediction]\n        'prediction' is a list and not a string though\n    \n    \"\"\"\n    df_small = df[df['t_dat'] > start_date]\n    print(f\"Kept data from {start_date} on. Total rows: {len(df_small)}\")\n    \n    # H stands for \"Transaction history\"\n    # dfh is a series of user_id => list of item_id (the list of purchases in order)\n    dfh = df_small.groupby(\"user_id\")['item_id'].apply(lambda items: list(set(items)))\n    dfh = dfh[dfh.str.len() >= MINIMUM_PURCHASES]\n    if TEST_RUN:\n        print(\"WARNING: TEST_RUN is True. It will be a toy execution.\")\n        dfh = dfh.head(TEST_SIZE)\n    \n    users = dfh.index.tolist()\n    n_users = len(users)\n    print(f\"Total users in the time frame with at least {MINIMUM_PURCHASES}: {n_users}\")\n    \n    df_rec = get_recommendations(users, dfh)\n    df_rec['customer_id'] = df_rec.index.map(user_to_customer_map)\n    df_rec['prediction'] = df_rec['recs'].map(lambda l: [item_to_article_map[i] for i in l])\n    \n    # Submission ready dataframe\n    df_rec.reset_index(drop=True)[['customer_id', 'prediction']]\n    return df_rec \n\ndf_recs = uucf(df)\n\ndf_fill = eval_hengzheng.copy()[['customer_id', 'prediction']]\n\ndef drop_duplicates(seq):\n    \"\"\" Remove duplicates of a given sequence keeping order\"\"\"\n    seen = set()\n    seen_add = seen.add\n    return [x for x in seq if not (x in seen or seen_add(x))]\n\ndef fill_row(row):\n    uucf = row['prediction_uucf']\n    fill = row['prediction_fill'].split()\n    new_list = drop_duplicates(uucf + fill)[:12]\n    return ' '.join(new_list)\n\n\ndef fill(df_recs, df_fill):\n    df_recs['len'] = df_recs['prediction'].str.len()\n    df_recs = pd.merge(df_fill, df_recs, how='left', on='customer_id', suffixes=('_fill', '_uucf'))\n    \n    \n    # No recs from UUCF at all: use the fallback model \n    df_recs.loc[df_recs['prediction_uucf'].isnull(), 'prediction'] = df_recs['prediction_fill']\n\n\n    # Full UUCF recommendation\n    mask = df_recs['prediction_uucf'].notnull() & (df_recs['len'] == 12)\n    df_recs.loc[mask, 'prediction'] = df_recs['prediction_uucf']\n\n\n    # Fill with another model. Not enough recs from UUCF\n    fill_mask = df_recs['prediction_uucf'].notnull() & (df_recs['len'] < 12)\n    df_recs.loc[fill_mask, 'prediction'] = df_recs[fill_mask].apply(fill_row, axis=1)\n    return df_recs.drop(['prediction_uucf', 'prediction_fill', 'len', 'recs'], axis=1)\n\neval_julian3833uucf = fill(df_recs, df_fill)\neval_julian3833uucf = valid_cust.merge(eval_julian3833uucf, on ='customer_id', how ='left')\neval_julian3833uucf['prediction'] = eval_julian3833uucf['prediction'].astype(str)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-27T03:35:34.709056Z","iopub.execute_input":"2022-02-27T03:35:34.709385Z","iopub.status.idle":"2022-02-27T03:38:21.61978Z","shell.execute_reply.started":"2022-02-27T03:35:34.709352Z","shell.execute_reply":"2022-02-27T03:38:21.618494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval_julian3833uucf.tail() ","metadata":{"execution":{"iopub.status.busy":"2022-02-27T03:40:04.608111Z","iopub.execute_input":"2022-02-27T03:40:04.608445Z","iopub.status.idle":"2022-02-27T03:40:04.621177Z","shell.execute_reply.started":"2022-02-27T03:40:04.60841Z","shell.execute_reply":"2022-02-27T03:40:04.620003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mapk(\n    eval_julian3833uucf['valid_true'].map(lambda x: x.split()), \n    eval_julian3833uucf['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-27T03:40:06.107614Z","iopub.execute_input":"2022-02-27T03:40:06.108064Z","iopub.status.idle":"2022-02-27T03:40:06.93347Z","shell.execute_reply.started":"2022-02-27T03:40:06.10801Z","shell.execute_reply":"2022-02-27T03:40:06.932679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eval_julian3833uucf, df_recs, df_fill\n\n_ = gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-27T03:40:09.570492Z","iopub.execute_input":"2022-02-27T03:40:09.570972Z","iopub.status.idle":"2022-02-27T03:40:09.828019Z","shell.execute_reply.started":"2022-02-27T03:40:09.570922Z","shell.execute_reply":"2022-02-27T03:40:09.826774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span id=\"cdeotte\" style=\"color:#3b51e3;\">cdeotte</span>**","metadata":{}},{"cell_type":"markdown","source":"https://www.kaggle.com/cdeotte/recommend-items-purchased-together-0-021\n\nnote: for item pairs, I re calculate it again to avoid data leakage. the note book is in https://www.kaggle.com/hervind/h-m-generate-item-pairs ","metadata":{}},{"cell_type":"code","source":"import cudf\nprint('RAPIDS version',cudf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:07:59.650964Z","iopub.execute_input":"2022-02-28T07:07:59.65175Z","iopub.status.idle":"2022-02-28T07:08:03.507577Z","shell.execute_reply.started":"2022-02-28T07:07:59.65171Z","shell.execute_reply":"2022-02-28T07:08:03.50664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\ntrain['customer_id'] = train['customer_id'].str[-16:].str.hex_to_int().astype('int64')\ntrain['article_id'] = train.article_id.astype('int32')\n\nval_start_date = '2020-09-16'\ntrain = train.loc[train['t_dat'] < val_start_date].reset_index(drop=True)\n\ntrain.t_dat = cudf.to_datetime(train.t_dat)\ntrain = train[['t_dat','customer_id','article_id']]\ntrain.to_parquet('train.pqt',index=False)\n\ntmp = train.groupby('customer_id').t_dat.max().reset_index()\ntmp.columns = ['customer_id','max_dat']\ntrain = train.merge(tmp,on=['customer_id'],how='left')\ntrain['diff_dat'] = (train.max_dat - train.t_dat).dt.days\ntrain = train.loc[train['diff_dat']<=6]\n\ntmp = train.groupby(['customer_id','article_id'])['t_dat'].agg('count').reset_index()\ntmp.columns = ['customer_id','article_id','ct']\ntrain = train.merge(tmp,on=['customer_id','article_id'],how='left')\ntrain = train.sort_values(['ct','t_dat'],ascending=False)\ntrain = train.drop_duplicates(['customer_id','article_id'])\ntrain = train.sort_values(['ct','t_dat'],ascending=False)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-28T07:08:18.521409Z","iopub.execute_input":"2022-02-28T07:08:18.521708Z","iopub.status.idle":"2022-02-28T07:08:24.781644Z","shell.execute_reply.started":"2022-02-28T07:08:18.521675Z","shell.execute_reply":"2022-02-28T07:08:24.780755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.to_pandas()\npairs = np.load('../input/h-m-generate-item-pairs/item_pair_3_months.npy',allow_pickle=True).item()\ntrain['article_id2'] = train.article_id.map(pairs)\ntrain2 = train[['customer_id','article_id2']].copy()\ntrain2 = train2.loc[train2.article_id2.notnull()]\ntrain2 = train2.drop_duplicates(['customer_id','article_id2'])\ntrain2 = train2.rename({'article_id2':'article_id'},axis=1)\n\ntrain = train[['customer_id','article_id']]\ntrain = pd.concat([train,train2],axis=0,ignore_index=True)\ntrain.article_id = train.article_id.astype('int32')\ntrain = train.drop_duplicates(['customer_id','article_id'])\n\ntrain.article_id = ' 0' + train.article_id.astype('str')\npreds = cudf.DataFrame( train.groupby('customer_id').article_id.sum().reset_index() )\npreds.columns = ['customer_id','prediction']\npreds.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:08:24.783443Z","iopub.execute_input":"2022-02-28T07:08:24.783731Z","iopub.status.idle":"2022-02-28T07:08:42.102025Z","shell.execute_reply.started":"2022-02-28T07:08:24.783682Z","shell.execute_reply":"2022-02-28T07:08:42.101047Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = cudf.read_parquet('train.pqt')\ntrain.t_dat = cudf.to_datetime(train.t_dat)\n# train = train.loc[train.t_dat >= cudf.to_datetime('2020-09-16')]\ntrain = train.loc[train.t_dat >= cudf.to_datetime('2020-09-09')]\ntop12 = ' 0' + ' 0'.join(train.article_id.value_counts().to_pandas().index.astype('str')[:12])\nprint(\"Last week's top 12 popular items:\")\nprint( top12 )","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:08:42.104029Z","iopub.execute_input":"2022-02-28T07:08:42.104961Z","iopub.status.idle":"2022-02-28T07:08:42.559006Z","shell.execute_reply.started":"2022-02-28T07:08:42.104917Z","shell.execute_reply":"2022-02-28T07:08:42.558049Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')\nsub = sub[['customer_id']]\nsub['customer_id_2'] = sub['customer_id'].str[-16:].str.hex_to_int().astype('int64')\nsub = sub.merge(preds.rename({'customer_id':'customer_id_2'},axis=1),\\\n    on='customer_id_2', how='left').fillna('')\n# del sub['customer_id_2']\nsub.prediction = sub.prediction + top12\nsub.prediction = sub.prediction.str.strip()\nsub.prediction = sub.prediction.str[:131]\n\nsub = sub.to_pandas()\neval_cdeotte = valid_cust.merge(sub[['customer_id_2', 'prediction']], left_on ='customer_id', right_on = 'customer_id_2', how ='left')\neval_cdeotte['prediction'] = eval_cdeotte['prediction'].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:08:42.560413Z","iopub.execute_input":"2022-02-28T07:08:42.561249Z","iopub.status.idle":"2022-02-28T07:08:47.903526Z","shell.execute_reply.started":"2022-02-28T07:08:42.561174Z","shell.execute_reply":"2022-02-28T07:08:47.90254Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eval_cdeotte.tail()","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:08:47.905845Z","iopub.execute_input":"2022-02-28T07:08:47.906166Z","iopub.status.idle":"2022-02-28T07:08:47.919251Z","shell.execute_reply.started":"2022-02-28T07:08:47.906124Z","shell.execute_reply":"2022-02-28T07:08:47.918302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mapk(\n    eval_cdeotte['valid_true'].map(lambda x: x.split()), \n    eval_cdeotte['prediction'].map(lambda x: x.split()), \n    k=12\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-28T07:08:51.142262Z","iopub.execute_input":"2022-02-28T07:08:51.142558Z","iopub.status.idle":"2022-02-28T07:08:51.866924Z","shell.execute_reply.started":"2022-02-28T07:08:51.142529Z","shell.execute_reply":"2022-02-28T07:08:51.866028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Thank you for reading so far\n## Please Upvote if you find this notebook helpful \n## Also upvote the sources notebook as well  \n","metadata":{}},{"cell_type":"markdown","source":"# Source\n* https://www.kaggle.com/kaerunantoka/h-m-how-to-calculate-map-12 \n* https://www.kaggle.com/julian3833/h-m-content-based-12-most-popular-items-0-007\n* https://www.kaggle.com/gpreda/h-m-eda-and-prediction\n* https://www.kaggle.com/hengzheng/time-is-our-best-friend-v2\n* https://www.kaggle.com/cdeotte/recommend-items-purchased-together-0-021","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}