{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\n\nThe notebook is based on CHRIS DEOTTE's notebook [Recommend Items Purchased Together],score is 0.0214 \n这个notebook参考了CHRIS DEOTTE的《Recommend Items Purchased Together》，他的分数是0.0214\n\nI have make some change in Part2, I use \"Purchased After\" instead of \"Purchased Together\", and let's see whether score would get higher.\n我在第二部分做了一些改变，使用了\"后来购买\"来代替\"一起购买\"，一起来看看分数是否会有改善吧～\n\n使用原来第二步：0.0214  <br>\n不使用第二步：0.0212  <br>\n使用新第二步（优化前，全不命中）：0.0205  <br>\n使用新第二步（优化前）：0.0211  <br>\n使用新第二步（优化后，全不命中）：0.0205  <br>\n使用新第二步（优化后）：0.0217  <br>\n\n优化：使用了相对日期代替绝对日期，先计算购买次数与平均购买日期，并提高推荐的门槛","metadata":{}},{"cell_type":"markdown","source":"# RAPIDS cuDF\nWe will use RAPIDS cuDF for fast dataframe operations","metadata":{}},{"cell_type":"code","source":"import cudf\nprint('RAPIDS version',cudf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:37:27.861785Z","iopub.execute_input":"2022-04-18T15:37:27.86255Z","iopub.status.idle":"2022-04-18T15:37:31.322779Z","shell.execute_reply.started":"2022-04-18T15:37:27.862456Z","shell.execute_reply":"2022-04-18T15:37:31.321393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Transactions, Reduce Memory\nDiscussion about reducing memory is [here][1]\n\n[1]: https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/308635","metadata":{}},{"cell_type":"code","source":"train = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/transactions_train.csv')\ntrain['customer_id'] = train['customer_id'].str[-16:].str.hex_to_int().astype('int64')\ntrain['article_id'] = train.article_id.astype('int64')\ntrain.t_dat = cudf.to_datetime(train.t_dat)\ntrain = train[['t_dat','customer_id','article_id']]\ntrain.to_parquet('train.pqt',index=False)\nprint( train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:43:03.530053Z","iopub.execute_input":"2022-04-18T15:43:03.530728Z","iopub.status.idle":"2022-04-18T15:43:07.020823Z","shell.execute_reply.started":"2022-04-18T15:43:03.530689Z","shell.execute_reply":"2022-04-18T15:43:07.020045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Find Each Customer's Last Week of Purchases\nOur final predictions will have the row order from of our dataframe. Each row of our dataframe will be a prediction. We will create the `predictionstring` later by `train.groupby('customer_id').article_id.sum()`. Since `article_id` is a string, when we groupby sum, it will concatenate all the customer predictions into a single string. It will also create the string in the order of the dataframe. So as we proceed in this notebook, we will order the dataframe how we want our predictions ordered.","metadata":{}},{"cell_type":"code","source":"tmp = train.groupby('customer_id').t_dat.max().reset_index()\ntmp.columns = ['customer_id','max_dat']\ntrain = train.merge(tmp,on=['customer_id'],how='left')\ntrain['diff_dat'] = (train.max_dat - train.t_dat).dt.days\ntrain = train.loc[train['diff_dat']<=6]\nprint('Train shape:',train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:43:07.645575Z","iopub.execute_input":"2022-04-18T15:43:07.646109Z","iopub.status.idle":"2022-04-18T15:43:07.817389Z","shell.execute_reply.started":"2022-04-18T15:43:07.64607Z","shell.execute_reply":"2022-04-18T15:43:07.816663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (1) Recommend Most Often Previously Purchased Items\nNote that many operations in cuDF will shuffle the order of the dataframe rows. Therefore we need to sort afterward because we want the most often previously purchased items first. Because this will be the order of our predictons. Since we sort by `ct` and then `t_dat` will will recommend items that have been purchased more frequently first followed by items purchased more recently second.","metadata":{}},{"cell_type":"code","source":"tmp = train.groupby(['customer_id','article_id'])['t_dat'].agg('count').reset_index()\ntmp.columns = ['customer_id','article_id','ct']\ntrain = train.merge(tmp,on=['customer_id','article_id'],how='left')\ntrain = train.sort_values(['ct','t_dat'],ascending=False)\ntrain = train.drop_duplicates(['customer_id','article_id'])\ntrain = train.sort_values(['ct','t_dat'],ascending=False)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:43:18.097429Z","iopub.execute_input":"2022-04-18T15:43:18.097691Z","iopub.status.idle":"2022-04-18T15:43:18.375779Z","shell.execute_reply.started":"2022-04-18T15:43:18.097662Z","shell.execute_reply":"2022-04-18T15:43:18.375118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (2) Recommend Items Purchased After","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom datetime import datetime\nfrom math import sqrt\nfrom pathlib import Path\nfrom tqdm import tqdm\ntqdm.pandas()\n\ndata_path = Path('../input/h-and-m-personalized-fashion-recommendations/')\ndata_output_path = Path('/kaggle/working/')\nN = 12\n\nprint(datetime.now())","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:39:57.785323Z","iopub.execute_input":"2022-04-18T15:39:57.785596Z","iopub.status.idle":"2022-04-18T15:39:57.792706Z","shell.execute_reply.started":"2022-04-18T15:39:57.785566Z","shell.execute_reply":"2022-04-18T15:39:57.791708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#读取transactions\n#t_dat范围：'2018-09-20'到'2020-09-22'\ndf_tran = pd.read_csv(data_path / 'transactions_train.csv',\n                 usecols = ['t_dat', 'customer_id', 'article_id'],\n                 dtype={'article_id': int})\nprint(df_tran['t_dat'].min(),'  ',df_tran['t_dat'].max())\nprint(datetime.now(),'  ',df_tran.shape)\ndisplay(df_tran.head())","metadata":{"execution":{"iopub.status.busy":"2022-04-18T15:41:27.959103Z","iopub.execute_input":"2022-04-18T15:41:27.959648Z","iopub.status.idle":"2022-04-18T15:42:03.895117Z","shell.execute_reply.started":"2022-04-18T15:41:27.959609Z","shell.execute_reply":"2022-04-18T15:42:03.894375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#先聚合计算平均购买日期和购买次数，以便后面计算联购中发现是否因为某一方交易特别多\ndat_ref = \"2018-09-20\"\ndf_tran['t_dat'] = pd.to_datetime(df_tran['t_dat'])-datetime.strptime(dat_ref, \"%Y-%m-%d\")\ndf_tran['solo_count'] = 1\ndf_tran = pd.DataFrame(df_tran.groupby(['article_id','customer_id'])\n                       .agg({'t_dat': 'sum', 'solo_count': 'sum'}))\ndf_tran['t_dat'] = df_tran['t_dat']/df_tran['solo_count']\ndf_tran = df_tran.reset_index()\nprint(datetime.now(),'  ',df_tran.shape)\ndisplay(df_tran.head())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Recommend Items always Purchased After\n#每次只计算3个月内的buy together的article，筛选buy after且发生超过1次的关联，计算1年结果约13分钟\n#16g极限，2020-03-22到2020-09-22半年纪录共7365109，通过customer_id自己inner join后189641831，join后再操作也是爆内存\ncom_year = [('2018-09-22','2018-12-22'),('2018-12-22','2019-03-22')\n            ,('2019-03-22','2019-05-22'),('2019-05-22','2019-07-22'),('2019-07-22','2019-09-22')\n            ,('2019-09-22','2019-12-22'),('2019-12-22','2020-03-22')\n            ,('2020-03-22','2020-06-22'),('2020-06-22','2020-09-22')]\nfor dates in com_year:\n    dat_start = datetime.strptime(dates[0], \"%Y-%m-%d\")-datetime.strptime(dat_ref, \"%Y-%m-%d\")\n    dat_end = datetime.strptime(dates[1], \"%Y-%m-%d\")-datetime.strptime(dat_ref, \"%Y-%m-%d\")\n    print('当前计算',dat_start,'到',dat_end)\n    #筛选3个月内与buy together关联\n    print('step1:',datetime.now())\n    df_ac = df_tran.copy()\n    df_ac = df_ac.loc[(df_ac['t_dat']>dat_start)&(df_ac['t_dat']<dat_end)]\n    df_ca = df_ac\n    df_aa = pd.merge(df_ac,df_ca,on = [\"customer_id\"],how=\"inner\")\n    #display(df_aa.head())\n    \n    #筛选buy after\n    print('step2:',datetime.now(),'  ',df_ac.shape,df_aa.shape) #(3981458, 3) (74325888, 5)\n    df_aa = df_aa.drop(columns=['customer_id']).loc[df_aa['t_dat_y']>df_aa['t_dat_x']]\n    #display(df_aa.head())\n    \n    #准备聚合的度量\n    print('step3:',datetime.now(),'  ',df_aa.shape)\n    df_aa['t_dat_diff'] = df_aa['t_dat_y']-df_aa['t_dat_x']\n    df_aa['article_count'] = 1\n    df_aa = df_aa.drop(columns=['t_dat_y','t_dat_x'])\n    #display(df_aa.head())\n    \n    #聚合计算buy了多少次，然后平均after多久后buy，1分钟+3分钟\n    print('step4:',datetime.now(),'  ',df_aa.shape)\n    df_aa = pd.DataFrame(df_aa.groupby(['article_id_x','article_id_y'])\n                         .agg({'t_dat_diff': 'sum', 'article_count': 'sum'\n                               , 'solo_count_x': 'sum', 'solo_count_y': 'sum'}))\n    #设置推荐的最小门槛\n    df_aa = df_aa.loc[(df_aa['article_count']>10)\n                      &(df_aa['article_count']/df_aa['solo_count_x']>0.2)\n                      &(df_aa['article_count']/df_aa['solo_count_y']>0.2)]\n    df_aa['avg_dat_diff'] = df_aa.apply(\n        lambda x: 0 if x['t_dat_diff'].days is np.nan else int(x['t_dat_diff'].days/x['article_count']) \n        ,axis=1)\n    df_aa = df_aa.drop(columns=['t_dat_diff'])\n    #display(df_aa.head())\n    \n    print('step5:',datetime.now(),'  ',df_aa.shape)\n    file_tmp = \"df_aa\"+dates[0]+\"_\"+dates[1]+\".csv\"\n    df_aa.to_csv(file_tmp, index=True)\n    print('-----------------')\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#读取合并\nfeatures = ['article_id_x', 'article_id_y','article_count']\ndf_f = pd.DataFrame(columns=features)\nfor dates in com_year:\n    dat_start = dates[0]\n    dat_end = dates[1]\n    file_tmp = \"df_aa\"+dat_start+\"_\"+dat_end+\".csv\"\n    df_temp = pd.read_csv(data_output_path / file_tmp,\n                       usecols = features,\n                       dtype={'article_id_x': int,'article_id_y': int})\n    print('df_temp:',df_temp.shape)\n    #取联购数量最多的配搭\n    #df_temp = df_temp.groupby('article_id_x').apply(lambda t: t[t.article_count==t.article_count.max()])\n    #print('df_temp:',df_temp.shape)\n    if(len(df_f)<1):\n        df_f = df_temp.copy()\n    else:\n        df_f = pd.concat([df_f,df_temp],ignore_index=True)\nprint('df_f:',df_f.shape)\ndisplay(df_f.head())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#取联购数量最多的配搭，约18分钟\ndf_f = df_f.groupby('article_id_x').apply(lambda t: t[t.article_count==t.article_count.max()])\ndf_f.drop_duplicates(subset=['article_id_x'], keep='first', inplace=True)\ndf_f = df_f.drop(columns=['article_count'])\nprint(datetime.now(),'   ',df_f.shape)\ndisplay(df_f.head())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_f.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#pandas转dict\npairs = {}\nfor i in tqdm(df_f.index):\n    article_id_x = df_f.at[i, 'article_id_x']\n    article_id_y = df_f.at[i, 'article_id_y']\n    #article_id_y = 0   #测试全不命中\n    pairs[article_id_x] = article_id_y\nprint(type(pairs))\n#display(pairs)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"下面的逻辑跟 [Recommend Items Purchased Together]一样","metadata":{}},{"cell_type":"code","source":"# USE PANDAS TO MAP COLUMN WITH DICTIONARY\nimport pandas as pd, numpy as np\ntrain = train.to_pandas()\ntrain['article_id2'] = train.article_id.map(pairs)\n\n# RECOMMENDATION OF PAIRED ITEMS\ntrain2 = train[['customer_id','article_id2']].copy()\ntrain2 = train2.loc[train2.article_id2.notnull()]\ntrain2 = train2.drop_duplicates(['customer_id','article_id2'])\ntrain2 = train2.rename({'article_id2':'article_id'},axis=1)\n\n# CONCATENATE PAIRED ITEM RECOMMENDATION AFTER PREVIOUS PURCHASED RECOMMENDATIONS\ntrain = train[['customer_id','article_id']]\ntrain = pd.concat([train,train2],axis=0,ignore_index=True)\ntrain.article_id = train.article_id.astype('int64')\ntrain = train.drop_duplicates(['customer_id','article_id'])\n\n# CONVERT RECOMMENDATIONS INTO SINGLE STRING\ntrain.article_id = ' 0' + train.article_id.astype('str')\npreds = cudf.DataFrame( train.groupby('customer_id').article_id.sum().reset_index() )\npreds.columns = ['customer_id','prediction']\npreds.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# (3) Recommend Last Week's Most Popular Items\nAfter recommending previous purchases and items purchased together we will then recommend the 12 most popular items. Therefore if our previous recommendations did not fill up a customer's 12 recommendations, then it will be filled by popular items.","metadata":{}},{"cell_type":"code","source":"train = cudf.read_parquet('train.pqt')\ntrain.t_dat = cudf.to_datetime(train.t_dat)\ntrain = train.loc[train.t_dat >= cudf.to_datetime('2020-09-16')]\ntop12 = ' 0' + ' 0'.join(train.article_id.value_counts().to_pandas().index.astype('str')[:12])\nprint(\"Last week's top 12 popular items:\")\nprint( top12 )","metadata":{"execution":{"iopub.status.busy":"2022-02-20T02:55:42.705242Z","iopub.execute_input":"2022-02-20T02:55:42.705714Z","iopub.status.idle":"2022-02-20T02:55:43.170621Z","shell.execute_reply.started":"2022-02-20T02:55:42.705679Z","shell.execute_reply":"2022-02-20T02:55:43.169177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Write Submission CSV\nWe will merge our predictions onto `sample_submission.csv` and submit to Kaggle.","metadata":{}},{"cell_type":"code","source":"sub = cudf.read_csv('../input/h-and-m-personalized-fashion-recommendations/sample_submission.csv')\nsub = sub[['customer_id']]\nsub['customer_id_2'] = sub['customer_id'].str[-16:].str.hex_to_int().astype('int64')\nsub = sub.merge(preds.rename({'customer_id':'customer_id_2'},axis=1),\\\n    on='customer_id_2', how='left').fillna('')\ndel sub['customer_id_2']\nsub.prediction = sub.prediction + top12\nsub.prediction = sub.prediction.str.strip()\nsub.prediction = sub.prediction.str[:131]\nsub.to_csv(f'submission.csv',index=False)\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-20T02:55:43.172957Z","iopub.execute_input":"2022-02-20T02:55:43.173205Z","iopub.status.idle":"2022-02-20T02:55:47.517952Z","shell.execute_reply.started":"2022-02-20T02:55:43.173171Z","shell.execute_reply":"2022-02-20T02:55:47.517241Z"},"trusted":true},"execution_count":null,"outputs":[]}]}